쿠버네티스 클러스터 초기화

EC2 인스턴스를 비용 관리를 위해서 중지 할 일이 자주 있는데 클러스터를 초기화해서 다시 연결해야 하는 필요성이 생겼다.
본 개시글은 kubeadm init 을 통해 구축한 클러스터에 한 해 작동한다.

Control-Plane

kubeadm reset

[root@ip-*********** ~] kubeadm reset

#########

[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0620 15:22:07.272492   20394 preflight.go:56] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
W0620 15:22:11.205493   20394 cleanupnode.go:99] [reset] Failed to remove containers: [failed to stop running pod 72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6: output: E0620 15:22:11.045385   20819 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6\": cni plugin not initialized" podSandboxID="72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6"
time="2024-06-20T15:22:11Z" level=fatal msg="stopping the pod sandbox \"72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6\": cni plugin not initialized"
: exit status 1, failed to stop running pod 09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f: output: E0620 15:22:11.204251   20842 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f\": cni plugin not initialized" podSandboxID="09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f"
time="2024-06-20T15:22:11Z" level=fatal msg="stopping the pod sandbox \"09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f\": cni plugin not initialized"
: exit status 1]
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

이후 로그에 나온 대로 파일들을 삭제해준다.

[root@ip-******** ~] rm -r /etc/cni/net.d
rm: remove directory ‘/etc/cni/net.d’? y
[root@ip-******** ~] rm -r ~/.kube/config
rm: remove regular file ‘/root/.kube/config’? y

이후 kubelet을 재시작해준다.

systemctl restart kubelet

다시 kubeadm init 을 통해 컨트롤 플레인을 활성화 시켜준다.

kubeadm init
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

다음으로 kubernets 설정파일들을 다시 설치해준다.

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system

swap 옵션도 꺼주자

swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab

이후에 중요한 것은 CNI를 반드시 설정해주고 데이터 플레인에 토큰을 통한 연결을 진행해야한다.

CNI 설정

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

만일 이래도 노드의 상태가 notReady 라면 containerd 를 재시동하자

systemctl restart containerd

Data-Plane

데이터 플레인에서도 마찬가지로 kubeadm을 통해 reset해주어야 한다.

kubeadm reset

[root@ip-*********** ~] kubeadm reset

#########

[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0620 15:22:07.272492   20394 preflight.go:56] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
W0620 15:22:11.205493   20394 cleanupnode.go:99] [reset] Failed to remove containers: [failed to stop running pod 72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6: output: E0620 15:22:11.045385   20819 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6\": cni plugin not initialized" podSandboxID="72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6"
time="2024-06-20T15:22:11Z" level=fatal msg="stopping the pod sandbox \"72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"72a43b78f9998fd99316a606959519c73d0496054b0f5e79520df82a1b27a2c6\": cni plugin not initialized"
: exit status 1, failed to stop running pod 09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f: output: E0620 15:22:11.204251   20842 remote_runtime.go:205] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f\": cni plugin not initialized" podSandboxID="09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f"
time="2024-06-20T15:22:11Z" level=fatal msg="stopping the pod sandbox \"09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"09424a7a0b954f8cffcbe20278cbf4f1ecfc24c90582027658ac12103d09547f\": cni plugin not initialized"
: exit status 1]
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

이후 로그에 나온 대로 파일들을 삭제해준다.

[root@ip-******** ~] rm -r /etc/cni/net.d
rm: remove directory ‘/etc/cni/net.d’? y
[root@ip-******** ~] rm -r ~/.kube/config
rm: remove regular file ‘/root/.kube/config’? y

만일 노드의 상태가 notReady 라면 containerd 를 재시동하자

systemctl restart containerd