FailedCreatePodSandBox

TroubleShooting

Error

• kubernetes 실습 중 ContainerCreating 상태에서 멈춤 현상 발생

vagrant@kube-control1:~/controller$ kubectl get pod
NAME             READY   STATUS              RESTARTS   AGE
basic-rs-4vnxs   0/1     ContainerCreating   0          6m50s
basic-rs-8sp4p   0/1     ContainerCreating   0          6m50s
basic-rs-gvzz2   0/1     ContainerCreating   0          6m50s

• describe 명령어를 통해 오류 메세지 확인

vagrant@kube-control1:~/controller$ kubectl describe pod
------------------------------------------
...
  Type     Reason                  Age                 From               Message
  ----     ------                  ----                ----               -------
  Normal   Scheduled               16m                 default-scheduler  Successfully assigned default/basic-rs-gvzz2 to kube-node3
  Warning  FailedCreatePodSandBox  16m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "50b0bc3e36e4d87898557fce3d63656a536793a59548276c36e91f749bd6bc67": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
  Normal   SandboxChanged          82s (x71 over 16m)  kubelet            Pod sandbox changed, it will be killed and re-created.
------------------------------------------

• 찾아보니 Calico 를 사용하는 경우 이와 같은 현상이 종종 발생
  -> kube-system namespace 에서 Calico 정상 동작 확인
  

vagrant@kube-control1:~/controller$ kubectl get pod --namespace kube-system
-------------------------------------------------
NAME                                      READY   STATUS    RESTARTS        AGE
calico-kube-controllers-6dfcdfb99-z8zqt   1/1     Running   1 (2d2h ago)    2d3h
calico-node-7mtn9                         1/1     Running   2 (2d2h ago)    2d21h
calico-node-7rg47                         1/1     Running   4 (2d2h ago)    2d21h
calico-node-gqggj                         1/1     Running   4 (2d2h ago)    2d21h
calico-node-t68gh                         1/1     Running   4 (2d2h ago)    2d21h
coredns-645b46f4b6-gmnz5                  1/1     Running   4 (2d2h ago)    2d21h
coredns-645b46f4b6-vkhlp                  1/1     Running   3 (2d2h ago)    2d21h
dns-autoscaler-659b8c48cb-gl9z7           1/1     Running   4 (2d2h ago)    2d21h
kube-apiserver-kube-control1              1/1     Running   5 (2d2h ago)    2d21h
kube-controller-manager-kube-control1     1/1     Running   10 (2d2h ago)   2d21h
kube-proxy-8bds7                          1/1     Running   4 (2d2h ago)    2d21h
kube-proxy-j97pj                          1/1     Running   4 (2d2h ago)    2d21h
...
-------------------------------------------------

vagrant@kube-control1:~/controller$ kubectl get pods -n kube-system -o wide | grep calico
-------------------------------------------------
calico-kube-controllers-6dfcdfb99-z8zqt   1/1     Running   1 (2d3h ago)    2d4h    10.233.74.25    kube-node2      <none>           <none>
calico-node-7mtn9                         1/1     Running   2 (2d3h ago)    2d22h   192.168.56.23   kube-node3      <none>           <none>
calico-node-7rg47                         1/1     Running   4 (2d3h ago)    2d22h   192.168.56.22   kube-node2      <none>           <none>
calico-node-gqggj                         1/1     Running   4 (2d3h ago)    2d22h   192.168.56.21   kube-node1      <none>           <none>
calico-node-t68gh                         1/1     Running   4 (2d3h ago)    2d22h   192.168.56.11   kube-control1   <none>           <none>
-------------------------------------------------

해결

• Calico pod 제거 후 재생성
  or
• Kubernetes 재기동

Kubernetes 재기동으로 해결

varant halt
vagrant up

vagrant@kube-control1:~/controller$ kubectl get pod
NAME             READY   STATUS    RESTARTS   AGE
basic-ds-bn6rt   1/1     Running   0          4s
basic-ds-kvp9r   1/1     Running   0          4s
basic-ds-xdlrs   1/1     Running   0          4s
ns-ds-gbq95      1/1     Running   0          3m56s
ns-ds-h5hth      1/1     Running   0          3m54s
ns-ds-htm8m      1/1     Running   0          3m58s

-> STATUS Running 확인

참고

Calico 제거

kubectl delete -f https://docs.projectcalico.org/manifests/calico.yaml

Calico 설치

kubectl delete -f https://docs.projectcalico.org/manifests/calico.yaml

or

Calico Pod 는 DaemonSet 으로 Running 중 이기 때문에 kubectl delete pod -n kube-system [Calico Pod Name] 으로 파드를 하나씩 제거해주면 자동으로 재생성된다