Here's a step-by-step guide to install Kafka and ZooKeeper,
set up a topic named user-activity, create partitions, configure a consumer
group named activity-consumers, and assign consumers (Consumer 1, Consumer 2, Consumer
3) with load balancing :
1. Install Kafka and ZooKeeper
Step 1: Download Kafka
Ø Download
the latest Kafka release from the Apache Kafka website.
bash
wget
https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz |
Step 2: Extract Kafka
bash
tar -xzf
kafka_2.13-3.5.1.tgz cd
kafka_2.13-3.5.1 |
Step 3: Start ZooKeeper
Ø Kafka
(before 3.3.0) requires ZooKeeper for broker management.
bash
bin/zookeeper-server-start.sh
config/zookeeper.properties |
Step 4: Start Kafka Broker
Ø Start
the default Kafka broker.
bash
bin/kafka-server-start.sh
config/server.properties |
2. Create the Topic user-activity
Command to Create a Topic
Ø Create
the user-activity topic with 3 partitions and a replication factor of 1.
bash
bin/kafka-topics.sh
--create --topic user-activity --bootstrap-server localhost:9092 --partitions
3 --replication-factor 1 |
Verify the Topic
Ø List
all topics:
bash
bin/kafka-topics.sh
--list --bootstrap-server localhost:9092 |
Ø Describe
the user-activity topic:
bash
bin/kafka-topics.sh
--list --bootstrap-server localhost:9092 |
Example output:
bash
Topic:
user-activity PartitionCount: 3 ReplicationFactor: 1 Configs: Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Partition: 1 Leader: 1 Replicas: 1 Isr: 1 Partition: 2 Leader: 1 Replicas: 1 Isr: 1 |
3. Load Balancing in Kafka
Ø Partitions:
Kafka achieves load balancing by distributing topic partitions across brokers.
Ø Producers:
Producers use keys or Kafka's partitioner logic to send data to specific
partitions.
Ø Consumers:
Consumer groups ensure each partition is consumed by one consumer at a time.
Add More Brokers
Ø Start
additional Kafka brokers to distribute partitions. Use unique configuration
files (e.g., server-1.properties, server-2.properties).
Example:
bash
# Copy the
server properties file and modify it for each broker. cp
config/server.properties config/server-1.properties cp
config/server.properties config/server-2.properties # Modify the
broker.id, log directories, and ports. sed -i
's/broker.id=0/broker.id=1/' config/server-1.properties sed -i
's/9092/9093/' config/server-1.properties sed -i
's#log.dirs=/tmp/kafka-logs#log.dirs=/tmp/kafka-logs-1#'
config/server-1.properties sed -i
's/broker.id=0/broker.id=2/' config/server-2.properties sed -i
's/9092/9094/' config/server-2.properties sed -i
's#log.dirs=/tmp/kafka-logs#log.dirs=/tmp/kafka-logs-2#'
config/server-2.properties # Start
additional brokers. bin/kafka-server-start.sh
config/server-1.properties bin/kafka-server-start.sh
config/server-2.properties |
Reassign Partitions
Ø If
you add brokers, you may want to reassign partitions to achieve load balancing.
- Generate
the reassignment plan:
bash
bin/kafka-reassign-partitions.sh
--generate --bootstrap-server localhost:9092 --topics-to-move-json-file
topics.json --broker-list "0,1,2" |
- Execute
the reassignment:
bash
bin/kafka-reassign-partitions.sh
--execute --bootstrap-server localhost:9092 --reassignment-json-file
reassignment.json |
4. Add Consumers to Consumer Group activity-consumers
Start Consumer 1
bash
bin/kafka-console-consumer.sh
--bootstrap-server localhost:9092 --topic user-activity --group
activity-consumers |
Start Consumer 2
Open a new terminal and run:
bash
bin/kafka-console-consumer.sh
--bootstrap-server localhost:9092 --topic user-activity --group
activity-consumers |
Start Consumer 3
Open another terminal and run:
bash
bin/kafka-console-consumer.sh
--bootstrap-server localhost:9092 --topic user-activity --group
activity-consumers |
5. Verify Consumer Group and Load Balancing
Describe the Consumer Group
Check the details of the activity-consumers group:
bash
bin/kafka-consumer-groups.sh
--describe --group activity-consumers --bootstrap-server localhost:9092 |
Example output:
bash
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID activity-consumers user-activity 0 10 15 5 consumer-1 activity-consumers user-activity 1 20 25 5 consumer-2 activity-consumers user-activity 2 30 35 5 consumer-3 |
Ø Partitions
are distributed among consumers in the group (Consumer 1, Consumer 2, Consumer
3).
Ø Kafka
automatically balances the load by assigning each partition to a single
consumer in the group.
6. Produce Messages to user-activity
Start a Producer
Send messages to the user-activity topic:
bash
bin/kafka-console-producer.sh
--topic user-activity --bootstrap-server localhost:9092 |
Ø Type
messages in the producer console. Press Enter to send each message.
Example:
bash
User1 logged
in User2 viewed
a page User3 logged
out |
Observe Load Balancing
Check the consumer consoles (Consumer 1, Consumer 2, Consumer
3). Messages will be distributed across partitions, and each consumer will
receive data from its assigned partition.
7. Scale Kafka for Load Balancing
Add More Brokers
- Copy
the server.properties file for additional brokers:
bash
cp
config/server.properties config/server-1.properties |
- Edit config/server-1.properties:
properties
broker.id=1 log.dirs=/tmp/kafka-logs-1 port=9093 zookeeper.connect=localhost:2181 |
- Start
the second broker:
bash
bin/kafka-server-start.sh
config/server-1.properties |
Reassign Partitions
Reassign partitions to include the new broker for load
balancing.
- Generate
reassignment plan:
bash
bin/kafka-reassign-partitions.sh
--execute --bootstrap-server localhost:9092 --reassignment-json-file
reassignment.json |
- Execute
reassignment:
bash
bin/kafka-reassign-partitions.sh
--execute --bootstrap-server localhost:9092 --reassignment-json-file
reassignment.json |
8. Monitor Kafka and ZooKeeper
Check ZooKeeper Status
Ø Connect
to ZooKeeper CLI:
bash
bin/zookeeper-shell.sh
localhost:2181 |
Ø List
Kafka brokers:
bash
ls
/brokers/ids |
Monitor Kafka Metrics
Ø Describe
the user-activity topic to check partition distribution:
bash
bin/kafka-topics.sh
--describe --topic user-activity --bootstrap-server localhost:9092 |
Ø List
all consumer groups:
bash
bin/kafka-consumer-groups.sh
--list --bootstrap-server localhost:9092 |
Ø Describe
a specific consumer group:
bash
bin/kafka-consumer-groups.sh
--describe --group activity-consumers --bootstrap-server localhost:9092 |
This setup ensures a distributed system with Kafka and
ZooKeeper, a topic with multiple partitions, and consumer groups handling load
balancing efficiently. Let me know if you need additional configurations!
Kafka and ZooKeeper insatll and set up a topic |
For More Related information, visit
Ø
Key
Components of Apache Kafka for Scalable Messaging
Ø
Build
a Video Stream Microservice with Kafka & REST API in Java
Ø
Kafka
general questions and answers
For Other information, visit
Ø
How
to get the neighbor of binary tree
Ø OWASP
(Open Web Application Security Project)
Ø Mastering
Debounce, Throttle, Rate Limit & Backoff in Java
Ø How
to draw sequence diagram and other diagrams using plantuml
0 Comments