Broker Configurations

Broker configurations are set in the server.properties file and apply to the entire Kafka broker.

Important broker configurations:

  • broker.id: Unique identifier for the broker.
  • listeners: Specifies the network interfaces and ports the broker listens on.
  • log.dirs: Directories where the broker stores log segments.
  • zookeeper.connect: ZooKeeper connection string for the Kafka cluster.

Example broker configuration:

# server.properties
broker.id=1
listeners=PLAINTEXT://localhost:9092
log.dirs=/var/lib/kafka/data
zookeeper.connect=localhost:2181

Topic Configurations

Topic configurations are set when creating a topic or can be altered later. They control the behavior and characteristics of a specific topic.

Important topic configurations:

  • replication.factor: Number of replicas for each partition.
  • partitions: Number of partitions for the topic.
  • retention.ms: Retention period for messages in milliseconds.
  • cleanup.policy: Policy for message cleanup (delete or compact).

Example topic configuration:

# Create a topic with specific configurations
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 5 --topic my-topic --config retention.ms=86400000 --config cleanup.policy=delete

Producer Configurations

Producer configurations are set when creating a producer and control how the producer behaves when sending messages to Kafka.

Important producer configurations:

  • bootstrap.servers: List of broker addresses for the producer to connect to.
  • acks: Acknowledgment level required from the broker (0, 1, or all).
  • retries: Number of retry attempts for failed send requests.
  • batch.size: Maximum size of a batch of messages sent to a partition.

Example producer configuration:

# Configure a Kafka producer
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("acks", "all");
properties.setProperty("retries", "3");
properties.setProperty("batch.size", "16384");

Consumer Configurations

Consumer configurations are set when creating a consumer and control how the consumer behaves when receiving messages from Kafka.

Important consumer configurations:

  • bootstrap.servers: List of broker addresses for the consumer to connect to.
  • group.id: Consumer group identifier.
  • auto.offset.reset: Behavior for initializing offsets (earliest, latest, or none).
  • enable.auto.commit: Whether to automatically commit offsets.

Example consumer configuration:

# Configure a Kafka consumer
properties.setProperty("bootstrap.servers", "localhost:9092");
properties.setProperty("group.id", "my-consumer-group");
properties.setProperty("auto.offset.reset", "earliest");
properties.setProperty("enable.auto.commit", "true");

Understanding and properly configuring Kafka at different levels is crucial for optimizing its performance and ensuring the desired behavior. It’s important to consider factors such as scalability, reliability, and data retention when setting these configurations.

Refer to the official Kafka documentation for a complete list of available configurations and their descriptions.