[Kafka] Docker로 kafka 클러스터 구축

[Kafka Cluster 구성]

• Zookeeper 1개
• Broker 3개
  
  
  

[docker-compose.yml 설정]

 # zookeeper
 zookeeper:
   image: wurstmeister/zookeeper
   container_name: zookeeper
   ports:
     - "2181:2181"
 # kafka
 kafka1:
   hostname: kafka1
   image: wurstmeister/kafka
   container_name: kafka1
   ports:
     - "9092:9092"
   environment:
     KAFKA_LISTENERS: LISTENER_DOCKER_INTERNAL://:19092,LISTENER_DOCKER_EXTERNAL://:9092
     KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${docker_host_ip}:9092
     KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
     KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
     KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
     KAFKA_BROKER_ID: 1
     KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
   depends_on:
     - zookeeper  
 kafka2:
   hostname: kafka2
   image: wurstmeister/kafka
   container_name: kafka2
   ports:
     - "9093:9093"
   environment:
     KAFKA_LISTENERS: LISTENER_DOCKER_INTERNAL://:19093,LISTENER_DOCKER_EXTERNAL://:9093
     KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka2:19093,LISTENER_DOCKER_EXTERNAL://${docker_host_ip}:9093
     KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
     KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
     KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
     KAFKA_BROKER_ID: 2
     KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
   depends_on:
     - zookeeper  
 kafka3:
   hostname: kafka3
   image: wurstmeister/kafka
   container_name: kafka3
   ports:
     - "9094:9094"
   environment:
     KAFKA_LISTENERS: LISTENER_DOCKER_INTERNAL://:19094,LISTENER_DOCKER_EXTERNAL://:9094
     KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka3:19094,LISTENER_DOCKER_EXTERNAL://${docker_host_ip}:9094
     KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
     KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
     KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
     KAFKA_BROKER_ID: 3
     KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock    
   depends_on:
     - zookeeper

여러 블로그를 참고해가며 설정을 잡는데 꽤 어려움을 겪었지만 이것저것 종합한 결과 설정을 잡을 수 있었습니다. 참고로 127.0.0.1로 하면 리스너가 인식을 하지 못해 에러가 발생할 수 있기 때문에 host ip를 확인하여 설정해야 합니다.