Installing Kafka Command-Line Tools using Docker

To run Confluent Kafka using Docker, use the following command:

docker run -d --name zookeeper -p 2181:2181 confluentinc/cp-zookeeper:latest
docker run -d --name kafka -p 9092:9092 --link zookeeper:zookeeper confluentinc/cp-kafka:latest

Starting and Stopping Kafka Brokers

docker start kafka
docker stop kafka

Creating Topics using Command-Line

Execute a shell in the running Kafka Docker container:

docker exec -it kafka /bin/bash
kafka-topics --create --topic my_topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Producing and Consuming Messages from CLI

Open a shell inside the Kafka container:

docker exec -it kafka /bin/bash
kafka-console-producer --topic my_topic --broker-list localhost:9092
kafka-console-consumer --topic my_topic --bootstrap-server localhost:9092

Commonly Used CLI Commands for Administration

  • List topics:

    kafka-topics --list --bootstrap-server localhost:9092

  • Describe a topic:

    kafka-topics --describe --topic my_topic --bootstrap-server localhost:9092

  • Delete a topic:

    kafka-topics --delete --topic my_topic --bootstrap-server localhost:9092

25 Questions

  1. How do you start a Kafka broker with a custom configuration file in Docker?
  2. What command changes the number of partitions for an existing topic?
  3. How to consume messages from a specific partition?
  4. How to produce messages with a specific key?
  5. Steps to stop all running Kafka and ZooKeeper containers?
  6. How to list consumer groups?
  7. How to reset the offset of a consumer group for a specific topic?
  8. How to alter the configuration of an existing topic?
  9. What command describes consumer groups?
  10. How to check the Kafka broker status?
  11. How to produce messages in a round-robin fashion to different partitions?
  12. How to set retention policies for a topic?
  13. How to change the replication factor of an existing topic?
  14. How to list all the brokers that are part of the Kafka cluster?
  15. How to purge all messages from a Kafka topic?
  16. How to read messages from a Kafka topic starting from the beginning?
  17. How to check the last offset for each partition of a topic?
  18. How to monitor the lag of a Kafka consumer group?
  19. How to set a quota for a Kafka client?
  20. How to authenticate against a Kafka broker?
  21. How to encrypt communication between Kafka brokers and clients?
  22. How to add or remove a broker to/from a Kafka cluster?
  23. How to perform a rolling restart of a Kafka cluster?
  24. How to configure ACLs for a Kafka topic?
  25. How to configure log segments in Kafka?

Solutions

  1. Use docker run with a mounted custom server.properties: docker run -v /path/to/custom/server.properties:/etc/kafka/server.properties ...
  2. kafka-topics --alter --topic my_topic --partitions 5 --bootstrap-server localhost:9092
  3. kafka-console-consumer --topic my_topic --partition 0 --bootstrap-server localhost:9092
  4. echo "key,value" | kafka-console-producer --topic my_topic --property "key.separator=," --broker-list localhost:9092
  5. docker stop kafka zookeeper && docker rm kafka zookeeper
  6. kafka-consumer-groups --list --bootstrap-server localhost:9092
  7. kafka-consumer-groups --reset-offsets --group my_group --topic my_topic --to-earliest --execute --bootstrap-server localhost:9092
  8. kafka-configs --alter --entity-type topics --entity-name my_topic --add-config max.message.bytes=128000 --bootstrap-server localhost:9092
  9. kafka-consumer-groups --describe --group my_group --bootstrap-server localhost:9092
  10. Use docker logs kafka or monitoring tools.
  11. Use kafka-console-producer without specifying keys.
  12. Use kafka-configs --alter with retention.ms and retention.bytes parameters.
  13. You can’t directly change the replication factor; create a new topic with the desired replication factor and copy the data.
  14. Use ZooKeeper CLI or tools like kafkacat.
  15. Use kafka-delete-records.
  16. kafka-console-consumer --from-beginning --topic my_topic --bootstrap-server localhost:9092
  17. Use kafka-run-class kafka.tools.GetOffsetShell
  18. kafka-consumer-groups --describe --group my_group --bootstrap-server localhost:9092
  19. Use kafka-configs --alter with client entity type.
  20. Through SASL mechanisms like PLAIN, SCRAM, etc.
  21. By enabling SSL/TLS.
  22. Add or remove a broker and then use partition reassignment.
  23. Use kafka-rolling-restart script or similar tools.
  24. Use kafka-acls.
  25. Configure log.segment.bytes and log.segment.ms