[FIXED] Kafka Retention.ms on specific topic is not working as expected

Issue

I'm creating topic like this in spring boot with retention time is 10 sec (10000ms).

@Configuration
public class TopicConfiguration {

    @Bean
    public NewTopic countersTopic() {
        return TopicBuilder.name("COUNTERS")
                .partitions(1)
                .replicas(1)
                .config(TopicConfig.RETENTION_MS_CONFIG, "10000")
                .build();
    }
}

After creating topic,you can see that retention time is 10000ms

I send the message to kafka every 30sec and I expect that each message should be delete after 10sec.

But still all the messages are reside in the kafka and Idk but after certain random time, all messages will be deleted.

Can anyone help me with this ?

Solution

Kafka only deletes closed segments (assuming the LogCleaner is enabled, and running) that are older than the retention period. Notice that you have segment.bytes value in your topic config, which defaults to 1GB. So, if your records are very very small, then you'd need to send a lot of them for a new segment to get created, and only then will the retention apply.

In the meantime, the broker will keep all the data.

Answered By - OneCricketeer
Answer Checked By - Willingham (JavaFixing Volunteer)