4

Can you explain what kafka-topics.sh --describe is showing? me I am following a tutorial video and also was reading the Apache documentation but I need a little more clarify as to what I'm looking at for the following columns in this graphic.

Leader: Is this pointing to the 3rd broker or is this pointing to the 3rd partition [2]?

Replicas: Is this pointing to brokers:partitions?

Isr: Is this pointing to brokers:partitions?

I would greatly appreciate it if someone explains what the columns A, B , C, D are.description of topic

1
  • I'm not sure it the graphic went through as I'm having firewall issues here at work. If it is not there I'll post it later when I get home.
    – edjm
    Commented Jan 6, 2020 at 18:45

1 Answer 1

10

Topic name: "install_test2"

4 partitions (partition 0, partition 1, partition 2, partition 3) and your replication factor for this topic is 2. It means that data in your topic will be stored (replicated) in 2 brokers for redundancy. In Kafka every partition has a leader and all the requests from producers and consumers are sent to the leader.

Leader column (column B in your image) shows broker ids of the leader for each partition. (Kafka evenly distributes partition leadership between brokers for load balancing)

Replicas column (column C in your image) shows ids of brokers that replicates data for each partition. The first id represents preferred leader. It means Kafka will try to make this broker leader of partition.

ISR (column D in your image) means in-sync-replica. In Kafka when a message is sent to a topic-partition (firstly message is received and stored in leader) and if you have replication factor greater than 1, then replica broker(s) send fetch request and this data is replicated to other broker(s). A follower (replica) broker is in-sync if it is not far behind the leader (explained in below). If a partition leader fails, Kafka chooses an ISR as the new leader for failover.

From Kafka docs:

Configuration parameter replica.lag.time.max.ms now refers not just to the time passed since last fetch request from replica, but also to time since the replica last caught up. Replicas that are still fetching messages from leaders but did not catch up to the latest messages in replica.lag.time.max.ms will be considered out of sync.

9
  • So, the "Leader" column value, for example the 1st line has the value 2 which coincides with the 1st value of "Replicas" which is 2 from the 2,1. This value is the broker.id in the config file. Question: So, if there were some broker failure with #2 then 1 would pick up the leader position && the Leader column would update to show #1 and the Replicas would show 1,2 or just 1?
    – edjm
    Commented Jan 6, 2020 at 20:07
  • Next question then is if the auto.leader.rebalance.enable is not set to true then when #2 broker is brought back online nothing would change and if set to true then it would be as originally shown in the picture with 2,1?
    – edjm
    Commented Jan 6, 2020 at 20:08
  • @Elijah There can only be a single leader at any time Commented Jan 6, 2020 at 20:12
  • 1
    @Elijah For the first question: yes. broker #1 would be the leader and leader column would be updated as 1. ISR would be just 1. Replicas would be still 2,1 because it is stored in zookeeper and there is no reason to change it. For the second question: When the #2 broker come back, possibly it would be out of sync, so it takes some time to become in-sync again. When it catch up the leader then you will see it in ISR list again. At that point leader election process starts and everything turn back to the original config shown in picture. (if you set auto.leader.rebalance.enable = true)
    – H.Ç.T
    Commented Jan 6, 2020 at 20:58
  • 1
    @AkmalSalikhov ISR list in the output of kafka-topics.sh --describe shows current in-sync replicas and it can change over time in case one of the brokers become out of sync or become in-sync again. On the other hand, the minimum number of replicas that must acknowledge a write for the write to be considered successful when acks=all in producer config is min.insync.replicas config which is defined as broker config in server.properties file.
    – H.Ç.T
    Commented Jul 8, 2021 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.