[FIXED] Is it possible to add partitions to an existing topic in Kafka 0.8.2

Issue

I have a Kafka cluster running with 2 partitions. I was looking for a way to increase the partition count to 3. However, I don't want to lose existing messages on the topic. I tried stopping href="https://kafka.apache.org/" rel="noreferrer">Kafka, modifying the server.properties file to increase the number of partitions to 3 and restart Kafka. However, that does not seem to change anything. Using Kafka ConsumerOffsetChecker, I still see it is using only 2 partitions. The Kafka version I am using is 0.8.2.2. In version 0.8.1, there used to be a script called kafka-add-partitions.sh, which I guess might do the trick. However, I don't see any such script in 0.8.2.

• Is there any way of accomplishing this?

I did experiment with creating a whole new topic and for that one, it does seem to use 3 partitions as per the change in the server.properties file. However, for existing topics, it doesn't seem to care.

Solution

Looks like you can use this script instead:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name 
   --partitions 40 

In the code it looks like they do same thing:

 AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(topic, partitionReplicaList, zkClient, true)

kafka-topics.sh executes this piece of code as well as AddPartitionsCommand used by kafka-add-partition script.

However you have to be aware of re-partitioning when using key:

Be aware that one use case for partitions is to semantically partition data, and adding partitions doesn't change the partitioning of existing data so this may disturb consumers if they rely on that partition. That is if data is partitioned by hash(key) % number_of_partitions then this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way.

Answered By - blockR
Answer Checked By - Mary Flores (JavaFixing Volunteer)