[Kubernetes] kube-controller CrashLoopBackOff

현상

kube-controller pod가 CrashLoopBackOff이 발생하며 구동 실패

# kube-controller CrashLoopBackOff
controlplane ➜  k get po -n kube-system 
NAME                                   READY   STATUS             RESTARTS      AGE
coredns-768b85b76f-84wwg               1/1     Running            0             15m
coredns-768b85b76f-fj7tn               1/1     Running            0             15m
etcd-controlplane                      1/1     Running            0             15m
kube-apiserver-controlplane            1/1     Running            0             15m
kube-controller-manager-controlplane   0/1     CrashLoopBackOff   5 (82s ago)   4m43s
kube-proxy-lxwpx                       1/1     Running            0             15m
kube-scheduler-controlplane            1/1     Running            0             12m

원인분석

# kube-controller pod log에서 ca.crt 미존재확인
controlplane ➜  k logs -n kube-system kube-controller-manager-controlplane 
I0820 13:07:58.402537       1 serving.go:380] Generated self-signed cert in-memory
E0820 13:07:58.776169       1 run.go:74] "command failed" err="unable to load client CA provider: open /etc/kubernetes/pki/ca.crt: no such file or directory"

# 그러나 실제 해당 경로에는 파일이 존재
controlplane  ➜  ls /etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.crt

# kube-controller pod의 구성파일 확인
spec:
  containers:
  - command:
    - kube-controller-manager
    ...
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
...
  volumeMounts:
      ...
      - mountPath: /etc/kubernetes/pki
        name: k8s-certs
        readOnly: true

  volumes:
    ...
    - hostPath:
        path: /etc/kubernetes/WRONG-PKI-PATH
        type: DirectoryOrCreate
      name: k8s-certs
    ...

해결

volume의 hostpath 를 올바른 경로로 수정 후 static pod 재시작