[Kubernetes] failed (add): unable to allocate IP address

현상

pod 가 CNI로부터 IP 할당을 받지못해 container 생성 불가

controlplane ~ ➜  k get po
NAME   READY   STATUS              RESTARTS   AGE
app    0/1     ContainerCreating   0          2m8s


# Pod Describe
Events:
  Type     Reason                  Age               From               Message
  ----     ------                  ----              ----               -------
  Normal   Scheduled               58s               default-scheduler  Successfully assigned default/app to controlplane
  Warning  FailedCreatePodSandBox  57s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2a99d671b8a99efdee8f11aa30997b5780e2c221e504b084e66bfcc09eb4b25e": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/2a911aa30997b5780e2c221e504b084e66bfcc09eb4b25e": dial tcp 127.0.0.1:6784: connect: connection refused
  Normal   SandboxChanged          5s (x5 over 56s)  kubelet            Pod sandbox changed, it will be killed and re-created.

# kubelet log
Aug 19 13:47:42 controlplane kubelet[4304]: E0819 13:47:42.470639    4304 kuberuntime_manager.go:1075] "killPodWithSyncResult failed" err="failed to \"KillPodSandbox\" for \"1d322870-9be6-4f44-8de0-4664d7b2acf6\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"2a99d671b8a99efdee8f11aa30997b5780e2c221e504b084e66bfcc09eb4b25e\\\": plugin type=\\\"weave-net\\\" name=\\\"weave\\\" failed (delete): Delete \\\"http://127.0.0.1:6784/ip/2a99d671b8a99efdee8f11aa30997b5780e2c221e504b084e66bfcc09eb4b25e\\\": dial tcp 127.0.0.1:6784: connect: connection refused\""
Aug 19 13:47:42 controlplane kubelet[4304]: E0819 13:47:42.470659    4304 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"1d322870-9be6-4f44-8de0-4664d7b2acf6\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"2a99d671b8a99efdee8f11aa30997b5780e2c221e504b084e66bfcc09eb4b25e\\\": plugin type=\\\"weave-net\\\" name=\\\"weave\\\" failed (delete): Delete \\\"http://127.0.0.1:6784/ip/2a99d671b8a99efdee8f11aa30997b5780e2c221e504b084e66bfcc09eb4b25e\\\": dial tcp 127.0.0.1:6784: connect: connection refused\"" pod="default/app" podUID="1d322870-9be6-4f44-8de0-4664d7b2acf6"

원인 분석

weave 가 kubelet에 설정되어 있으나, 실질적으로 weave pod가 구동되어 있지 않은 것으로 확인

## cni binary check
controlplane /opt/cni/bin ➜  ls
bandwidth  dummy        host-local  macvlan  sbr     tuning  weave-ipam
bridge     firewall     ipvlan      portmap  static  vlan    weave-net
dhcp       host-device  loopback    ptp      tap     vrf     weave-plugin-2.8.1

## cni config check
controlplane /etc/cni/net.d ➜  ls
10-weave.conflist

## pod status check
controlplane ~ ➜  k get po -n kube-system | grep weave
(none)

해결

weave net install 가이드를 통해 daemonset file 다운로드 후 배포

controlplane ~/weave ➜  k apply -f weave-daemonset-k8s.yaml 
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.apps/weave-net created

# 배포 완료
controlplane ~/weave ➜  k get po -A | grep  weave
kube-system   weave-net-4q25r                        2/2     Running   0          14s

## 배포 이후 pod 정상 생성 확인
controlplane ~/weave ➜  k get po
NAME   READY   STATUS    RESTARTS   AGE
app    1/1     Running   0          17m

참고

https://github.com/weaveworks/weave/blob/master/prog/weave-kube/weave-daemonset-k8s-1.8.yaml