Kafka connect with docker

김영재·2024년 2월 27일

[프로젝트] Spring Cloud-MSA

목록 보기

2/2

Kafka Connect란 더 간편하게 효율적으로 데이터파이프라인을 구축하는 방법이다.
Kafka Connect는 Microservice에서 직접 DB에 대한 커넥션과 처리작업을 하지 않고 관련 작업은 Kafka에 일임 하는 역활을 한다.
Kafka Connect를 실행하기 위해서는 zookeeper와 kafka서버가 필요하다.

kafka connect의 Dockerfile

FROM confluentinc/cp-kafka-connect-base
 
USER root
 
# Kafka Connect JDBC 커넥터 설치
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:latest
 
# MariaDB 접속 client 다운로드
RUN curl -k -SL "https://downloads.mariadb.com/Connectors/java/connector-java-2.7.4/mariadb-java-client-2.7.4.jar" -o /usr/share/java/kafka/mariadb-java-client-2.7.4.jar

해당 Dockerfile를 이용하여, my-kafka-connect image파일 생성

docker build -t my-kafka-connect .

docker-compose 파일 생성

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: my-zookeeper
    ports:
      - "2181:2181"
    networks:
      my-network:
        ipv4_address: 172.18.0.100
  kafka:
    image: wurstmeister/kafka
    container_name: my-kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: {HOST_PC_IP_ADDRESS}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper
    networks:
      my-network:
        ipv4_address: 172.18.0.101
  kafka-connect:
    image: my-kafka-connect
    container_name: my-kafka-connect
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:9092
      CONNECT_REST_ADVERTISED_HOST_NAME: 172.18.0.102
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: "quickstart"
      CONNECT_CONFIG_STORAGE_TOPIC: "quickstart-config"
      CONNECT_OFFSET_STORAGE_TOPIC: "quickstart-offsets"
      CONNECT_STATUS_STORAGE_TOPIC: "quickstart-status"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE: "true"
      CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE: "true"
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
    depends_on:
      - kafka
    networks:
      my-network:
        ipv4_address: 172.18.0.102
 
networks:
  my-network:
    name: ecommerce-network
    external: true

주의 할점은 Kafka에서
KAFKA_ADVERTISED_HOST_NAME: {HOST_PC_IP_ADDRESS}를
127.0.0.1로 하면 안된다는것이다
host pc의 ip주소를 넣거나 docker container의 ip를 설정하여 넣어야 한다.