Issue
I have a requirement where an application sends n asynchronous messages for 1 batch in Kafka topic for some processing. I want to integrate the Spring aggregator at Kafka consumers to generate an event after all messages of that batch have been processed. My question is, How would I integrate aggregator in a distributed environment as there are multiple Kafka consumers for the same batch? Should we integrate this with any in-memory Databases like Hazelcast?
Solution
That's correct. You need to configure that aggregator with some shared persistent store: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#reaper.
There is not MessageGroupStore implementation for Hazelcast: https://github.com/spring-projects/spring-integration-extensions/tree/main/spring-integration-hazelcast. But you can consider to use whatever is available for you from our implementations: https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#message-store
The contribution on the matter for Hazelcast extension is welcome!
Answered By - Artem Bilan
