Apache Camel Tutorial - Idempotent Consumer using Memory IdempotentRepository and File IdempotentRepository

In Apache Camel we use the Idempotent Consumer pattern to filter out duplicate messages. Consider a scenario where we have to process files only once. If there are any duplicates they should be skipped. Using Apache Camel we can use Idempotent Consumer directly within the component so it will skip files that are processed once. This feature is be enabled by setting the idempotent=true option.
In order to achieve this Apache Camel keeps track of the consumed files using a message id which is stored in the repository called Idempotent Repository.
Apache Camel provides the following types of IdempotentRepository.

• MemoryIdempotentRepository
• FileIdempotentRepository
• HazelcastIdempotentRepository
• JdbcMessageIdRepository
• JpaMessageIdRepository
• InfinispanIdempotentRepository
• JCacheIdempotentRepository
• SpringCacheIdempotentRepository
• EhcacheIdempotentRepository
• KafkaIdempotentRepository

We will be implementing examples using MemoryIdempotentRepository and FileIdempotentRepository

Apache Camel - Table of Contents
File Transfer Using Java DSL Apache Camel
Apache Camel Java DSL + Spring Integration Hello World Example
Apache Camel Exception Handling Using Simple Example
Apache Camel Redelivery policy using example
Integrate Apache Camel and ActiveMQ
EIP patterns using Apache Camel
Apache Camel Tutorial- Integrate Spring Boot+ Apache Camel
Apache Camel Tutorial- Integrate with MySQL DB using SQL query
Apache Camel EIP - Splitter and Aggregator pattern
Apache Camel Unit Testing 
Apache Camel + Spring + Quartz Hello World Example 
Camel application deployment on JBoss Fuse 
Apache Camel + Apache CXF SOAP Webservices 
Apache Camel + JAX-RS REST Webservice
Apache Camel + CXFRS REST Webservice
Apache Camel Routing Slip EIP Pattern
Apache Camel Dynamic Router Pattern
Apache Camel Load Balancer EIP Pattern
Apache Camel Interceptors
Apache Camel + Kafka Hello World Example
Apache Camel - Marshalling/Unmarshalling XML/JSON Data Example
Calling and Consuming Webservices using Apache Camel
Apache Camel Tutorial - Send SMTP Email Using Gmail
Apache Camel Tutorial - SEDA component Hello World Example
Apache Camel Tutorial - Idempotent Consumer using MemoryIdempotentRepository and FileIdempotentRepository
Spring Boot + Apache Camel JDBC component + MySQL - Hello World Example 
Spring Boot + Apache Camel SQL component + MySQL - Hello World Example
Spring Boot + Apache Camel SQL component + Transaction Management Example

Implement File Transfer using Apache Camel File Component

We will first be implementing a Simple Apache Camel Program which transfers/copies file from one folder to another using the File Component. We will then be modifying this project to implement Memory Idempotent Repository and File Idempotent Repository.

The Maven project will be as follows-

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.javainuse</groupId>
	<artifactId>camel-java-dsl</artifactId>
	<version>0.0.1-SNAPSHOT</version>

	<dependencies>
		<dependency>
			<groupId>org.apache.camel</groupId>
			<artifactId>camel-core</artifactId>
			<version>3.0.0-M3</version>
		</dependency>
	</dependencies>
</project>

package com.javainuse;

import org.apache.camel.builder.RouteBuilder;

public class SimpleRouteBuilder extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		from("file:C:/inputFolder").to("file:C:/outputFolder");
	}

}

package com.javainuse;

import org.apache.camel.CamelContext;
import org.apache.camel.impl.DefaultCamelContext;

public class MainApp {

	public static void main(String[] args) {
		SimpleRouteBuilder routeBuilder = new SimpleRouteBuilder();
		CamelContext ctx = new DefaultCamelContext();
		try {
			ctx.addRoutes(routeBuilder);
			ctx.start();
			Thread.sleep(5 * 60 * 1000);
			ctx.stop();
		} catch (Exception e) {
			e.printStackTrace();
		}

	}
}

We will now be implementing the Idempotent Respository which will filter out duplicate messages. Also we will be using the File Name as the id which is stored in the Idempotent Repository for keeping track of the files processed.

MemoryIdempotentRepository Implementation

package com.javainuse;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.support.processor.idempotent.MemoryIdempotentRepository;

public class SimpleRouteBuilder extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		from("file:C://inputFolder")
				.idempotentConsumer(header("CamelFileName"), MemoryIdempotentRepository.memoryIdempotentRepository(200))
				.process(new Processor() {

					public void process(Exchange exchange) throws Exception {
						System.out.println("This file is being processed the first time -- "
								+ exchange.getIn().getHeader("CamelFileName"));

					}
				}).to("file:C://outputFolder");
	}

}

Run this project we get output as-

FileIdempotentRepository Implementation

package com.javainuse;

import java.io.File;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.support.processor.idempotent.FileIdempotentRepository;
import org.apache.camel.support.processor.idempotent.MemoryIdempotentRepository;

public class SimpleRouteBuilder extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		from("file:C://inputFolder")
				.idempotentConsumer(header("CamelFileName"),
						FileIdempotentRepository.fileIdempotentRepository(new File("C://imp//track.txt")))
				.process(new Processor() {

					public void process(Exchange exchange) throws Exception {
						System.out.println("This file is being processed the first time -- "
								+ exchange.getIn().getHeader("CamelFileName"));

					}
				}).to("file:C://outputFolder");
	}

}

Also the filestore is created as follows-

Download Source Code

Download it -
Apache Camel + File Idempotent Repository Example Application Apache Camel + Memory Idempotent Repository Example Application

See Also
Spring Boot Hello World Application- Create simple controller and jsp view using Maven
Spring Boot Tutorial-Spring Data JPA
Spring Boot + Simple Security Configuration
Pagination using Spring Boot Simple Example
Spring Boot + ActiveMQ Hello world Example
Spring Boot + Swagger Example Hello World Example
Spring Boot + Swagger- Understanding the various Swagger Annotations
Spring Boot Main Menu
Spring Boot Interview Questions