statefulset kafka 클러스터 구축

1. pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka-0
  namespace: kafka-ns
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: "kafka"
  hostPath:
    path: /data/
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka-1
  namespace: kafka-ns
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: "kafka"
  hostPath:
    path: /data/
    type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka-2
  namespace: kafka-ns
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: "kafka"
  hostPath:
    path: /data/
    type: DirectoryOrCreate

2. statefulset.yaml

KAFKA_ADVERTISED_LISTENERS에서 막혀서 일주일을 고생했는데 쿠버네티스 자체에서 podIP를 받을 수 있다. 환경 변수를 삽입해주면 해결!!

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
spec:
  selector:
    matchLabels:
      app: kafka
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-broker
spec:
  selector:
    matchLabels:
      app: kafka
  serviceName: kafka-service
  replicas: 3 
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: kafka
    spec:
      hostname: kafka
      restartPolicy: Always
      containers:
      - name: kafka
        image: confluentinc/cp-kafka
        ports:
        - containerPort: 9092
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KAFKA_ADVERTISED_LISTENERS
          value: PLAINTEXT://$(MY_POD_NAME).kafka-service.default.svc.cluster.local:9092
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zk-0.zk-hs.default.svc.cluster.local:2181,zk-1.zk-hs.default.svc.cluster.local:2181,zk-1.zk-hs.default.svc.cluster.local:2181
        - name: KAFKA_LISTENERS
          value: PLAINTEXT://:9092
        - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
          value: "1"
  volumeClaimTemplates:
  - metadata:
      name: kafkadir
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "kafka"
      resources:
        requests:
          storage: 10Gi