Search Tutorials


Apache Camel Tutorial - Idempotent Consumer using MemoryIdempotentRepository and FileIdempotentRepository | JavaInUse



Apache Camel Tutorial - Idempotent Consumer using MemoryIdempotentRepository and FileIdempotentRepository

In Apache Camel we use the Idempotent Consumer pattern to filter out duplicate messages. Consider a scenario where we have to process files only once. If there are any duplicates they should be skipped. Using Apache Camel we can use Idempotent Consumer directly within the component so it will skip files that are processed once. This feature is be enabled by setting the idempotent=true option.
In order to achieve this Apache Camel keeps track of the consumed files using a message id which is stored in the repository called Idempotent Repository.
Apache Camel provides the following types of IdempotentRepository.
apache camel IdempotentRepository
  • MemoryIdempotentRepository
  • FileIdempotentRepository
  • HazelcastIdempotentRepository
  • JdbcMessageIdRepository
  • JpaMessageIdRepository
  • InfinispanIdempotentRepository
  • JCacheIdempotentRepository
  • SpringCacheIdempotentRepository
  • EhcacheIdempotentRepository
  • KafkaIdempotentRepository
We will be implementing examples using MemoryIdempotentRepository and FileIdempotentRepository

Apache Camel - Table of Contents

File Transfer Using Java DSL Apache Camel Apache Camel Java DSL + Spring Integration Hello World Example Apache Camel Exception Handling Using Simple Example Apache Camel Redelivery policy using example Integrate Apache Camel and ActiveMQ EIP patterns using Apache Camel Apache Camel Tutorial- Integrate Spring Boot+ Apache Camel Apache Camel Tutorial- Integrate with MySQL DB using SQL query Apache Camel EIP - Splitter and Aggregator pattern Apache Camel Unit Testing Apache Camel + Spring + Quartz Hello World Example Camel application deployment on JBoss Fuse Apache Camel + Apache CXF SOAP Webservices Apache Camel + JAX-RS REST Webservice Apache Camel + CXFRS REST Webservice Apache Camel Routing Slip EIP Pattern Apache Camel Dynamic Router Pattern Apache Camel Load Balancer EIP Pattern Apache Camel Interceptors Apache Camel + Kafka Hello World Example Apache Camel - Marshalling/Unmarshalling XML/JSON Data Example Calling and Consuming Webservices using Apache Camel Apache Camel Tutorial - Send SMTP Email Using Gmail Apache Camel Tutorial - SEDA component Hello World Example

Implement File Transfer using Apache Camel File Component

We will first be implementing a Simple Apache Camel Program which transfers/copies file from one folder to another using the File Component. We will then be modifying this project to implement Memory Idempotent Repository and File Idempotent Repository.
apache camel Memory IdempotentRepository
The Maven project will be as follows-
Apache Camel File Copy Tutorial
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.javainuse</groupId>
	<artifactId>camel-java-dsl</artifactId>
	<version>0.0.1-SNAPSHOT</version>

	<dependencies>
		<dependency>
			<groupId>org.apache.camel</groupId>
			<artifactId>camel-core</artifactId>
			<version>3.0.0-M3</version>
		</dependency>
	</dependencies>
</project>
package com.javainuse;

import org.apache.camel.builder.RouteBuilder;

public class SimpleRouteBuilder extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		from("file:C:/inputFolder").to("file:C:/outputFolder");
	}

}
package com.javainuse;

import org.apache.camel.CamelContext;
import org.apache.camel.impl.DefaultCamelContext;

public class MainApp {

	public static void main(String[] args) {
		SimpleRouteBuilder routeBuilder = new SimpleRouteBuilder();
		CamelContext ctx = new DefaultCamelContext();
		try {
			ctx.addRoutes(routeBuilder);
			ctx.start();
			Thread.sleep(5 * 60 * 1000);
			ctx.stop();
		} catch (Exception e) {
			e.printStackTrace();
		}

	}
}
We will now be implementing the Idempotent Respository which will filter out duplicate messages. Also we will be using the File Name as the id which is stored in the Idempotent Repository for keeping track of the files processed.
apache camel File IdempotentRepository

MemoryIdempotentRepository Implementation

package com.javainuse;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.support.processor.idempotent.MemoryIdempotentRepository;

public class SimpleRouteBuilder extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		from("file:C://inputFolder")
				.idempotentConsumer(header("CamelFileName"), MemoryIdempotentRepository.memoryIdempotentRepository(200))
				.process(new Processor() {

					public void process(Exchange exchange) throws Exception {
						System.out.println("This file is being processed the first time -- "
								+ exchange.getIn().getHeader("CamelFileName"));

					}
				}).to("file:C://outputFolder");
	}

}
Run this project we get output as-
camel idempotentConsumer example

FileIdempotentRepository Implementation

package com.javainuse;

import java.io.File;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.support.processor.idempotent.FileIdempotentRepository;
import org.apache.camel.support.processor.idempotent.MemoryIdempotentRepository;

public class SimpleRouteBuilder extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		from("file:C://inputFolder")
				.idempotentConsumer(header("CamelFileName"),
						FileIdempotentRepository.fileIdempotentRepository(new File("C://imp//track.txt")))
				.process(new Processor() {

					public void process(Exchange exchange) throws Exception {
						System.out.println("This file is being processed the first time -- "
								+ exchange.getIn().getHeader("CamelFileName"));

					}
				}).to("file:C://outputFolder");
	}

}

camel idempotentConsumer example
Also the filestore is created as follows-
camel idempotentConsumer File Store

Download Source Code

Download it -
Apache Camel + File Idempotent Repository Example Application Apache Camel + Memory Idempotent Repository Example Application

See Also

Spring Boot Hello World Application- Create simple controller and jsp view using Maven Spring Boot Tutorial-Spring Data JPA Spring Boot + Simple Security Configuration Pagination using Spring Boot Simple Example Spring Boot + ActiveMQ Hello world Example Spring Boot + Swagger Example Hello World Example Spring Boot + Swagger- Understanding the various Swagger Annotations Spring Boot Main Menu Spring Boot Interview Questions