docker 환경에 kafka cluster 구축하기

kafka를 사용하기 전에, docker compose를 사용해 로컬에 kafka cluster를 구축하려고 한다.

1. docker-compose.yml 생성

# docker-compose.yml 
version: '3.7'
services: 

  zk1:
    container_name: zookeeper1
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk1
    ports:
      - "2181:2181"
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk1/data:/data"
  zk2:
    ...
  zk3:
    ...
  kafka1
    container_name: kafka1
    image: wurstmeister/kafka:latest
    restart: on-failure
    depends_on:
      - zk1
      - zk2
      - zk3
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_HOST_NAME: host.docker.internal
      BOOTSTRAP_SERVERS: host.docker.internal:9092, host.docker.internal:9093, host.docker.internal:9094
      KAFKA_ZOOKEEPER_CONNECT: "zk1:2181,zk2:2182,zk3:2183"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
  kafka2
    ..
  kafka3
    ..
  kafka-ui
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "10000:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=host.docker.internal:9092,host.docker.internal:9093,host.docker.internal:9094
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zk1:2181,zk2:2182,zk1:2183

• zookeeper
  • ZOO_MY_ID
    • 클러스터 내에서 사용할 식별자이다. 유일해야 한다.
  • ZOO_SERVERS
    • 이부분은 무슨 의미인지 모르겠다.
• kafka
  • KAFKA_BROKER_ID
    • 클러스터 내에서 사용할 식별자이다. 유일해야 한다.
  • KAFKA_ADVERTISED_HOST_NAME
    • 클러스터에 연결 시 반환할 kafka의 host name이다.
    • host.docker.internal을 사용해 컨테이너가 설치된 host의 ip를 반환한다.
  • BOOTSTRAP_SERVERS
    • 클러스터에 등록된 kafka 서버의 목록이다.
  • KAFKA_ZOOKEEPER_CONNECT
    • 클러스터에 등록된 zookeeper들에 대한 정보이다.
  • KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
    • 기본적으로 설정할 파티션의 복사본의 갯수이다.
• kafka-ui
  • 브라우저로 접근할 수 있는 kafka cluster ui이다.
  • KAFKA_CLUSTERS_0_NAME
    • 표시할 클러스터의 이름이다.
  • KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
    • kafka 클러스터의 목록이다.
  • KAFKA_CLUSTERS_0_ZOOKEEPER
    - zookeeper 클러스터의 목록이다.
    전체 yml

# docker-compose.yml 
version: '3.7'
services: 

  zk1:
    container_name: zookeeper1
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk1
    ports:
      - "2181:2181"
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk1/data:/data"

  zk2:
    container_name: zookeeper2
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk2
    ports:
      - "2182:2181"
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk2/data:/data"

  zk3:
    container_name: zookeeper3
    image: wurstmeister/zookeeper:latest
    restart: always
    hostname: zk3
    ports:
      - "2183:2181"
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181
    volumes:
      - "~/zk-cluster/zk3/data:/data"

  kafka1:
    container_name: kafka1
    image: wurstmeister/kafka:latest
    restart: on-failure
    depends_on:
      - zk1
      - zk2
      - zk3
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_HOST_NAME: host.docker.internal
      BOOTSTRAP_SERVERS: host.docker.internal:9092, host.docker.internal:9093, host.docker.internal:9094
      KAFKA_ZOOKEEPER_CONNECT: "zk1:2181,zk2:2182,zk3:2183"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2

  kafka2:
    container_name: kafka2
    image: wurstmeister/kafka:latest
    restart: on-failure
    depends_on:
      - zk1
      - zk2
      - zk3
    ports:
      - "9093:9092"
    volumes:
       - /var/run/docker.sock:/var/run/docker.sock
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ADVERTISED_HOST_NAME: host.docker.internal
      BOOTSTRAP_SERVERS: host.docker.internal:9092, host.docker.internal:9093, host.docker.internal:9094
      KAFKA_ZOOKEEPER_CONNECT: "zk1:2181,zk2:2182,zk3:2183"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2

  kafka3:
    container_name: kafka3
    image: wurstmeister/kafka:latest
    restart: on-failure
    depends_on:
      - zk1
      - zk2
      - zk3
    ports:
      - "9094:9092"
    volumes:
       - /var/run/docker.sock:/var/run/docker.sock
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ADVERTISED_HOST_NAME: host.docker.internal
      BOOTSTRAP_SERVERS: host.docker.internal:9092, host.docker.internal:9093, host.docker.internal:9094
      KAFKA_ZOOKEEPER_CONNECT: "zk1:2181,zk2:2182,zk3:2183"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2

  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "10000:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=host.docker.internal:9092,host.docker.internal:9093,host.docker.internal:9094
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zk1:2181,zk2:2182,zk1:2183

2. docker compose 실행

• docker-compose.yml이 있는 위치에서 docker-compose up -d를 실행한다.
  

실행 결과

docker desktop에서 7개의 컨테이너가 생성된 모습을 확인할 수 있다.