Kafka and ZooKeeper install and set up a topic

Here's a step-by-step guide to install Kafka and ZooKeeper, set up a topic named user-activity, create partitions, configure a consumer group named activity-consumers, and assign consumers (Consumer 1, Consumer 2, Consumer 3) with load balancing :

1. Install Kafka and ZooKeeper

Step 1: Download Kafka

Ø Download the latest Kafka release from the Apache Kafka website.

bash

wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz

Step 2: Extract Kafka

bash

tar -xzf kafka_2.13-3.5.1.tgz

cd kafka_2.13-3.5.1

Step 3: Start ZooKeeper

Ø Kafka (before 3.3.0) requires ZooKeeper for broker management.

bash

bin/zookeeper-server-start.sh config/zookeeper.properties

Step 4: Start Kafka Broker

Ø Start the default Kafka broker.

bash

bin/kafka-server-start.sh config/server.properties

2. Create the Topic user-activity

Command to Create a Topic

Ø Create the user-activity topic with 3 partitions and a replication factor of 1.

bash

bin/kafka-topics.sh --create --topic user-activity --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Verify the Topic

Ø List all topics:

bash

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Ø Describe the user-activity topic:

bash

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Example output:

bash

Topic: user-activity PartitionCount: 3 ReplicationFactor: 1 Configs:

Partition: 0 Leader: 1 Replicas: 1 Isr: 1

Partition: 1 Leader: 1 Replicas: 1 Isr: 1

Partition: 2 Leader: 1 Replicas: 1 Isr: 1

3. Load Balancing in Kafka

Ø Partitions: Kafka achieves load balancing by distributing topic partitions across brokers.

Ø Producers: Producers use keys or Kafka's partitioner logic to send data to specific partitions.

Ø Consumers: Consumer groups ensure each partition is consumed by one consumer at a time.

Add More Brokers

Ø Start additional Kafka brokers to distribute partitions. Use unique configuration files (e.g., server-1.properties, server-2.properties).

Example:

bash

# Copy the server properties file and modify it for each broker.

cp config/server.properties config/server-1.properties

cp config/server.properties config/server-2.properties

# Modify the broker.id, log directories, and ports.

sed -i 's/broker.id=0/broker.id=1/' config/server-1.properties

sed -i 's/9092/9093/' config/server-1.properties

sed -i 's#log.dirs=/tmp/kafka-logs#log.dirs=/tmp/kafka-logs-1#' config/server-1.properties

sed -i 's/broker.id=0/broker.id=2/' config/server-2.properties

sed -i 's/9092/9094/' config/server-2.properties

sed -i 's#log.dirs=/tmp/kafka-logs#log.dirs=/tmp/kafka-logs-2#' config/server-2.properties

# Start additional brokers.

bin/kafka-server-start.sh config/server-1.properties

bin/kafka-server-start.sh config/server-2.properties

Reassign Partitions

Ø If you add brokers, you may want to reassign partitions to achieve load balancing.

Generate the reassignment plan:

bash

bin/kafka-reassign-partitions.sh --generate --bootstrap-server localhost:9092 --topics-to-move-json-file topics.json --broker-list "0,1,2"

Execute the reassignment:

bash

bin/kafka-reassign-partitions.sh --execute --bootstrap-server localhost:9092 --reassignment-json-file reassignment.json

4. Add Consumers to Consumer Group activity-consumers

Start Consumer 1

bash

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user-activity --group activity-consumers

Start Consumer 2

Open a new terminal and run:

bash

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user-activity --group activity-consumers

Start Consumer 3

Open another terminal and run:

bash

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user-activity --group activity-consumers

5. Verify Consumer Group and Load Balancing

Describe the Consumer Group

Check the details of the activity-consumers group:

bash

bin/kafka-consumer-groups.sh --describe --group activity-consumers --bootstrap-server localhost:9092

Example output:

bash

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID

activity-consumers user-activity 0 10 15 5 consumer-1

activity-consumers user-activity 1 20 25 5 consumer-2

activity-consumers user-activity 2 30 35 5 consumer-3

Ø Partitions are distributed among consumers in the group (Consumer 1, Consumer 2, Consumer 3).

Ø Kafka automatically balances the load by assigning each partition to a single consumer in the group.

6. Produce Messages to user-activity

Start a Producer

Send messages to the user-activity topic:

bash

bin/kafka-console-producer.sh --topic user-activity --bootstrap-server localhost:9092

Ø Type messages in the producer console. Press Enter to send each message.

Example:

bash

User1 logged in

User2 viewed a page

User3 logged out

Observe Load Balancing

Check the consumer consoles (Consumer 1, Consumer 2, Consumer 3). Messages will be distributed across partitions, and each consumer will receive data from its assigned partition.

7. Scale Kafka for Load Balancing

Add More Brokers

Copy the server.properties file for additional brokers:

bash

cp config/server.properties config/server-1.properties

Edit config/server-1.properties:

properties

broker.id=1

log.dirs=/tmp/kafka-logs-1

port=9093

zookeeper.connect=localhost:2181

Start the second broker:

bash

bin/kafka-server-start.sh config/server-1.properties

Reassign Partitions

Reassign partitions to include the new broker for load balancing.

Generate reassignment plan:

bash

bin/kafka-reassign-partitions.sh --execute --bootstrap-server localhost:9092 --reassignment-json-file reassignment.json

Execute reassignment:

bash

bin/kafka-reassign-partitions.sh --execute --bootstrap-server localhost:9092 --reassignment-json-file reassignment.json

8. Monitor Kafka and ZooKeeper

Check ZooKeeper Status

Ø Connect to ZooKeeper CLI:

bash

bin/zookeeper-shell.sh localhost:2181

Ø List Kafka brokers:

bash

ls /brokers/ids

Monitor Kafka Metrics

Ø Describe the user-activity topic to check partition distribution:

bash

bin/kafka-topics.sh --describe --topic user-activity --bootstrap-server localhost:9092

Ø List all consumer groups:

bash

bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

Ø Describe a specific consumer group:

bash

bin/kafka-consumer-groups.sh --describe --group activity-consumers --bootstrap-server localhost:9092

This setup ensures a distributed system with Kafka and ZooKeeper, a topic with multiple partitions, and consumer groups handling load balancing efficiently. Let me know if you need additional configurations!