Header Ads Widget

Responsive Advertisement

Kafka and ZooKeeper install and set up a topic

Here's a step-by-step guide to install Kafka and ZooKeeper, set up a topic named user-activity, create partitions, configure a consumer group named activity-consumers, and assign consumers (Consumer 1, Consumer 2, Consumer 3) with load balancing :


1. Install Kafka and ZooKeeper

Step 1: Download Kafka

Ø  Download the latest Kafka release from the Apache Kafka website.

bash

wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz

 

Step 2: Extract Kafka

bash

tar -xzf kafka_2.13-3.5.1.tgz

cd kafka_2.13-3.5.1

 

Step 3: Start ZooKeeper

Ø  Kafka (before 3.3.0) requires ZooKeeper for broker management.

bash

bin/zookeeper-server-start.sh config/zookeeper.properties

 

Step 4: Start Kafka Broker

Ø  Start the default Kafka broker.

bash

bin/kafka-server-start.sh config/server.properties

 


2. Create the Topic user-activity

Command to Create a Topic

Ø  Create the user-activity topic with 3 partitions and a replication factor of 1.

bash

bin/kafka-topics.sh --create --topic user-activity --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

 

Verify the Topic

Ø  List all topics:

bash

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

 

Ø  Describe the user-activity topic:

bash

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

 

Example output:

bash

Topic: user-activity PartitionCount: 3 ReplicationFactor: 1 Configs:

    Partition: 0 Leader: 1 Replicas: 1 Isr: 1

    Partition: 1 Leader: 1 Replicas: 1 Isr: 1

    Partition: 2 Leader: 1 Replicas: 1 Isr: 1

 


3. Load Balancing in Kafka

Ø  Partitions: Kafka achieves load balancing by distributing topic partitions across brokers.

Ø  Producers: Producers use keys or Kafka's partitioner logic to send data to specific partitions.

Ø  Consumers: Consumer groups ensure each partition is consumed by one consumer at a time.

Add More Brokers

Ø  Start additional Kafka brokers to distribute partitions. Use unique configuration files (e.g., server-1.properties, server-2.properties).

Example:

bash

# Copy the server properties file and modify it for each broker.

cp config/server.properties config/server-1.properties

cp config/server.properties config/server-2.properties

 

# Modify the broker.id, log directories, and ports.

sed -i 's/broker.id=0/broker.id=1/' config/server-1.properties

sed -i 's/9092/9093/' config/server-1.properties

sed -i 's#log.dirs=/tmp/kafka-logs#log.dirs=/tmp/kafka-logs-1#' config/server-1.properties

 

sed -i 's/broker.id=0/broker.id=2/' config/server-2.properties

sed -i 's/9092/9094/' config/server-2.properties

sed -i 's#log.dirs=/tmp/kafka-logs#log.dirs=/tmp/kafka-logs-2#' config/server-2.properties

 

# Start additional brokers.

bin/kafka-server-start.sh config/server-1.properties

bin/kafka-server-start.sh config/server-2.properties

 

Reassign Partitions

Ø  If you add brokers, you may want to reassign partitions to achieve load balancing.

  1. Generate the reassignment plan:

bash

bin/kafka-reassign-partitions.sh --generate --bootstrap-server localhost:9092 --topics-to-move-json-file topics.json --broker-list "0,1,2"

 

  1. Execute the reassignment:

bash

bin/kafka-reassign-partitions.sh --execute --bootstrap-server localhost:9092 --reassignment-json-file reassignment.json

 


4. Add Consumers to Consumer Group activity-consumers

Start Consumer 1

bash

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user-activity --group activity-consumers

 

Start Consumer 2

Open a new terminal and run:

bash

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user-activity --group activity-consumers

 

Start Consumer 3

Open another terminal and run:

bash

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user-activity --group activity-consumers

 


5. Verify Consumer Group and Load Balancing

Describe the Consumer Group

Check the details of the activity-consumers group:

bash

bin/kafka-consumer-groups.sh --describe --group activity-consumers --bootstrap-server localhost:9092

 

Example output:

bash

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID

activity-consumers  user-activity    0           10              15             5               consumer-1

activity-consumers  user-activity    1           20              25             5               consumer-2

activity-consumers  user-activity    2           30              35             5               consumer-3

 

Ø  Partitions are distributed among consumers in the group (Consumer 1, Consumer 2, Consumer 3).

Ø  Kafka automatically balances the load by assigning each partition to a single consumer in the group.


6. Produce Messages to user-activity

Start a Producer

Send messages to the user-activity topic:

bash

bin/kafka-console-producer.sh --topic user-activity --bootstrap-server localhost:9092

 

Ø  Type messages in the producer console. Press Enter to send each message.

Example:

bash

User1 logged in

User2 viewed a page

User3 logged out

 

Observe Load Balancing

Check the consumer consoles (Consumer 1, Consumer 2, Consumer 3). Messages will be distributed across partitions, and each consumer will receive data from its assigned partition.


7. Scale Kafka for Load Balancing

Add More Brokers

  1. Copy the server.properties file for additional brokers:

bash

cp config/server.properties config/server-1.properties

 

  1. Edit config/server-1.properties:

properties

broker.id=1

log.dirs=/tmp/kafka-logs-1

port=9093

zookeeper.connect=localhost:2181

 

  1. Start the second broker:

bash

bin/kafka-server-start.sh config/server-1.properties

 

Reassign Partitions

Reassign partitions to include the new broker for load balancing.

  1. Generate reassignment plan:

bash

bin/kafka-reassign-partitions.sh --execute --bootstrap-server localhost:9092 --reassignment-json-file reassignment.json

 

  1. Execute reassignment:

bash

bin/kafka-reassign-partitions.sh --execute --bootstrap-server localhost:9092 --reassignment-json-file reassignment.json

 


8. Monitor Kafka and ZooKeeper

Check ZooKeeper Status

Ø  Connect to ZooKeeper CLI:

bash

bin/zookeeper-shell.sh localhost:2181

 

Ø  List Kafka brokers:

bash

ls /brokers/ids

 

Monitor Kafka Metrics

Ø  Describe the user-activity topic to check partition distribution:

bash

bin/kafka-topics.sh --describe --topic user-activity --bootstrap-server localhost:9092

 

Ø  List all consumer groups:

bash

bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

 

Ø  Describe a specific consumer group:

bash

bin/kafka-consumer-groups.sh --describe --group activity-consumers --bootstrap-server localhost:9092

 


This setup ensures a distributed system with Kafka and ZooKeeper, a topic with multiple partitions, and consumer groups handling load balancing efficiently. Let me know if you need additional configurations! 

Kafka and ZooKeeper insatll and set up a topic
Kafka and ZooKeeper insatll and set up a topic

For More Related information, visit

Ø  Key Components of Apache Kafka for Scalable Messaging

Ø  Build a Video Stream Microservice with Kafka & REST API in Java

Ø  Kafka general questions and answers

 

For Other information, visit

Ø  How to get the neighbor of binary tree

Ø  OWASP (Open Web Application Security Project)

Ø  Mastering Debounce, Throttle, Rate Limit & Backoff in Java

Ø  How to draw sequence diagram and other diagrams using plantuml

Ø  Pascal Triangle

Ø  Molecular weight of chemistry in Java code

Ø  String to xml or html Beautifier

Post a Comment

0 Comments