[Kubernetes] Eviction manager: failed to get summary stats" err="failed to get node info: node \"node01\" not found"

현상

worker node 가 NotReady 상태

# Worker node NotReady
controlplane ~ ➜  k get no
NAME           STATUS     ROLES           AGE   VERSION
controlplane   Ready      control-plane   14m   v1.30.0
node01         NotReady   <none>          14m   v1.30.0


# 그러나 kubelet 은 구동 상태는 running
node01 ~ ➜  systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Tue 2024-08-20 13:30:25 UTC; 5min ago
       Docs: https://kubernetes.io/docs/
   Main PID: 10054 (kubelet)
      Tasks: 33 (limit: 251379)
     Memory: 51.2M
     CGroup: /system.slice/kubelet.service

원인분석

# kubelet log 확인: node 등록을 시도하지만 차단
## controlplane:6553 과 통신을 시도하는 것으로 확인
Aug 20 13:38:29 node01 kubelet[14205]: I0820 13:38:29.209319   14205 kubelet_node_status.go:73] "Attempting to register node" node="node01"
Aug 20 13:38:29 node01 kubelet[14205]: E0820 13:38:29.211454   14205 kubelet_node_status.go:96] "Unable to register node with API server" err="Post \"https://controlplane:6553/api/v1/nodes\": dial tcp 192.30.61.6:6553: connect: connection refused" node="node01"
Aug 20 13:38:29 node01 kubelet[14205]: W0820 13:38:29.360028   14205 reflector.go:547] k8s.io/client-go/informers/factory.go:160: failed to list *v1.Service: Get "https://controlplane:6553/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.30.61.5:6553: connect: connection refused

# controlplane에서는 6443 포트로 서비스 중으로 확인 (=kubelet 설정 변경 필요)
controlplane ~ ✖ kubectl cluster-info
Kubernetes control plane is running at https://controlplane:6443
CoreDNS is running at https://controlplane:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

해결

kubelet 서비스의 설정파일 중 kubeconfig 관련 파일에서 cluster 접속 정보 변경 후 kubelet 재시작

node01 /etc/kubernetes ✖ cat kubelet.conf 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://controlplane:6553		# 6443으로 변경
  name: default-cluster


# 조치 이후 클러스터 정상화
controlplane ~ ➜  k get no
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   28m   v1.30.0
node01         Ready    <none>          28m   v1.30.0