kafka-confluentinc-multi-01

Mark
docker-compose.yaml
version: '3'
services:
  zookeeper-1:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper-1
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  broker-1:
    image: confluentinc/cp-kafka:latest
    container_name: broker-1
    ports:
      - "8097:8097"
    depends_on:
      - zookeeper-1
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8097,INTERNAL://broker-1:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  broker-2:
    image: confluentinc/cp-kafka:latest
    container_name: broker-2
    ports:
      - "8098:8098"
    depends_on:
      - zookeeper-1
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8098,INTERNAL://broker-2:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  broker-3:
    image: confluentinc/cp-kafka:latest
    container_name: broker-3
    ports:
      - "8099:8099"
    depends_on:
      - zookeeper-1
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:8099,INTERNAL://broker-3:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

Start Containers

docker compose up

Verify All Containers Running

docker ps

CONTAINER ID   IMAGE                              COMMAND                  CREATED         STATUS         PORTS                              NAMES
b10ba52bc8a5   confluentinc/cp-kafka:latest       "/etc/confluent/dock…"   4 minutes ago   Up 4 minutes   0.0.0.0:8098->8098/tcp, 9092/tcp   broker-2
3d80f47ce72f   confluentinc/cp-kafka:latest       "/etc/confluent/dock…"   4 minutes ago   Up 4 minutes   0.0.0.0:8097->8097/tcp, 9092/tcp   broker-1
5d2358751e2d   confluentinc/cp-kafka:latest       "/etc/confluent/dock…"   4 minutes ago   Up 4 minutes   0.0.0.0:8099->8099/tcp, 9092/tcp   broker-3
2892cedcf094   confluentinc/cp-zookeeper:latest   "/etc/confluent/dock…"   4 minutes ago   Up 4 minutes   2181/tcp, 2888/tcp, 3888/tcp       zookeeper-1

Verify Active Kafka Brokers Using zookeeper-shell

docker exec -it zookeeper-1 /bin/zookeeper-shell localhost:2181

Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

Run bash from a randomly picked Kafka broker-2

docker exec -it broker-2 /bin/bash

[appuser@b10ba52bc8a5 ~]$

Create a new topic called randomTopic

[appuser@b10ba52bc8a5 ~]$ kafka-topics --bootstrap-server localhost:9092 --create --topic randomTopic

Created topic randomTopic.

Produce some messages to our new topic – randomTopic

kafka-console-producer --bootstrap-server localhost:9092 --topic randomTopic

>Hi
>Hello
>How are you? 
>[appuser@b10ba52bc8a5 ~]$

Read previously sent messages

exit
docker exec -it broker-3 /bin/bash
[appuser@5d2358751e2d ~]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic randomTopic --from-beginning
Hi
Hello
How are you?

–

Mark