주의 사항
- Rocky Linux 9버전으로 진행하였으므로, 공식 문서에서
CentOS, RHEL or Fedora탭에 적힌 명령어를 사용하였다. - 컨트롤 플레인은 메이저 버전을 건너뛰어 업그레이드할 수 없다. 즉, 1.27 → 1.28 → 1.29 → ... 이런 식으로 단계적으로 업그레이드 해야 한다.
- 워커 노드의 업그레이드 시
kubectl drain으로 파드를 노드에서 제거하고 다른 노드에 옮긴 후 진행해야 한다.
공식 문서
0. etcd 백업
쿠버네티스는 모든 작업을 API 서버를 통해 수행하고, 수행된 결과는 etcd 서버에 저장된다.
즉, etcd에는 쿠버네티스 클러스터의 모든 상태가 저장되어 있기 때문에 만약 컨트롤 플레인이 모두 망가져버리면 etcd 백업본을 통해 복구할 수 있다.
etcdctl snapshot save [SNAPSHOT_NAME] \
--ca-file=/etc/kubernetes/ca.crt \
--cert-file=/etc/kubernetes/server.crt \
--key-file=/etc/kubernetes/server.key클라이언트 측에서 etcd의 CA 인증서와 클라이언트 키를 제시해야 하는 이유?
→ 서버 측에서는 클라이언트 측의 인증서를 받고 소유하고 있는 CA 인증서를 통해 해당 클라이언트가 인증된 사용자임을 알 수 있지만, 클라이언트 측에서는 해당 etcd 서버가 인증된 서버인지 알 수 없기 때문에, 상호 TLS 인증의 목적으로 제시하는 것이며, 실제 etcd 서버에 전달되는 파일은 오직--cert-file에 명시한 클라이언트 측 인증서 뿐이고--ca-file에 명시된 파일과--key-file에 명시된 파일은 etcd로부터 수신한 인증서를 해독하기 위해 쓰인다.
1. kubeadm Repo 업데이트
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v[VERSION]/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v[VERSION]/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF[VERSION]에 업데이트하고자 하는 버전을 기입한다.
그리고 yum list --showduplicates kubeadm --disableexcludes=kubernetes 명령어로 원하는 버전이 맞는지와 마이너 버전까지 확인한다.
$ yum list --showduplicates kubeadm --disableexcludes=kubernetes
마지막 메타자료 만료확인(0:05:44 이전): 2024년 12월 23일 (월) 오후 01시 08분 36초.
설치된 꾸러미
kubeadm.x86_64 1.28.15-150500.1.1 @kubernetes
사용 가능한 꾸러미
kubeadm.aarch64 1.28.0-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.0-150500.1.1 kubernetes
kubeadm.s390x 1.28.0-150500.1.1 kubernetes
kubeadm.src 1.28.0-150500.1.1 kubernetes
kubeadm.x86_64 1.28.0-150500.1.1 kubernetes
kubeadm.aarch64 1.28.1-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.1-150500.1.1 kubernetes
kubeadm.s390x 1.28.1-150500.1.1 kubernetes
kubeadm.src 1.28.1-150500.1.1 kubernetes
kubeadm.x86_64 1.28.1-150500.1.1 kubernetes
kubeadm.aarch64 1.28.2-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.2-150500.1.1 kubernetes
kubeadm.s390x 1.28.2-150500.1.1 kubernetes
kubeadm.src 1.28.2-150500.1.1 kubernetes
kubeadm.x86_64 1.28.2-150500.1.1 kubernetes
kubeadm.aarch64 1.28.3-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.3-150500.1.1 kubernetes
kubeadm.s390x 1.28.3-150500.1.1 kubernetes
kubeadm.src 1.28.3-150500.1.1 kubernetes
kubeadm.x86_64 1.28.3-150500.1.1 kubernetes
kubeadm.aarch64 1.28.4-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.4-150500.1.1 kubernetes
kubeadm.s390x 1.28.4-150500.1.1 kubernetes
kubeadm.src 1.28.4-150500.1.1 kubernetes
kubeadm.x86_64 1.28.4-150500.1.1 kubernetes
kubeadm.aarch64 1.28.5-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.5-150500.1.1 kubernetes
kubeadm.s390x 1.28.5-150500.1.1 kubernetes
kubeadm.src 1.28.5-150500.1.1 kubernetes
kubeadm.x86_64 1.28.5-150500.1.1 kubernetes
kubeadm.aarch64 1.28.6-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.6-150500.1.1 kubernetes
kubeadm.s390x 1.28.6-150500.1.1 kubernetes
kubeadm.src 1.28.6-150500.1.1 kubernetes
kubeadm.x86_64 1.28.6-150500.1.1 kubernetes
kubeadm.aarch64 1.28.7-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.7-150500.1.1 kubernetes
kubeadm.s390x 1.28.7-150500.1.1 kubernetes
kubeadm.src 1.28.7-150500.1.1 kubernetes
kubeadm.x86_64 1.28.7-150500.1.1 kubernetes
kubeadm.aarch64 1.28.8-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.8-150500.1.1 kubernetes
kubeadm.s390x 1.28.8-150500.1.1 kubernetes
kubeadm.src 1.28.8-150500.1.1 kubernetes
kubeadm.x86_64 1.28.8-150500.1.1 kubernetes
kubeadm.aarch64 1.28.9-150500.2.1 kubernetes
kubeadm.ppc64le 1.28.9-150500.2.1 kubernetes
kubeadm.s390x 1.28.9-150500.2.1 kubernetes
kubeadm.src 1.28.9-150500.2.1 kubernetes
kubeadm.x86_64 1.28.9-150500.2.1 kubernetes
kubeadm.aarch64 1.28.10-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.10-150500.1.1 kubernetes
kubeadm.s390x 1.28.10-150500.1.1 kubernetes
kubeadm.src 1.28.10-150500.1.1 kubernetes
kubeadm.x86_64 1.28.10-150500.1.1 kubernetes
kubeadm.aarch64 1.28.11-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.11-150500.1.1 kubernetes
kubeadm.s390x 1.28.11-150500.1.1 kubernetes
kubeadm.src 1.28.11-150500.1.1 kubernetes
kubeadm.x86_64 1.28.11-150500.1.1 kubernetes
kubeadm.aarch64 1.28.12-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.12-150500.1.1 kubernetes
kubeadm.s390x 1.28.12-150500.1.1 kubernetes
kubeadm.src 1.28.12-150500.1.1 kubernetes
kubeadm.x86_64 1.28.12-150500.1.1 kubernetes
kubeadm.aarch64 1.28.13-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.13-150500.1.1 kubernetes
kubeadm.s390x 1.28.13-150500.1.1 kubernetes
kubeadm.src 1.28.13-150500.1.1 kubernetes
kubeadm.x86_64 1.28.13-150500.1.1 kubernetes
kubeadm.aarch64 1.28.14-150500.2.1 kubernetes
kubeadm.ppc64le 1.28.14-150500.2.1 kubernetes
kubeadm.s390x 1.28.14-150500.2.1 kubernetes
kubeadm.src 1.28.14-150500.2.1 kubernetes
kubeadm.x86_64 1.28.14-150500.2.1 kubernetes
kubeadm.aarch64 1.28.15-150500.1.1 kubernetes
kubeadm.ppc64le 1.28.15-150500.1.1 kubernetes
kubeadm.s390x 1.28.15-150500.1.1 kubernetes
kubeadm.src 1.28.15-150500.1.1 kubernetes
kubeadm.x86_64 1.28.15-150500.1.1 kubernetes2. Install kubeadm
yum install -y kubeadm-1.28.15 --disableexcludes=kubernetes3. Kubeadm Upgrade
먼저 kubeadm upgrade plan 으로 업그레이드 계획을 확인한 뒤,
$ kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.27.3
[upgrade/versions] kubeadm version: v1.28.15
I1223 13:15:22.418561 4164243 version.go:256] remote version is much newer: v1.32.0; falling back to: stable-1.28
[upgrade/versions] Target version: v1.28.15
[upgrade/versions] Latest version in the v1.27 series: v1.27.16
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 2 x v1.27.3 v1.27.16
Upgrade to the latest version in the v1.27 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.27.3 v1.27.16
kube-controller-manager v1.27.3 v1.27.16
kube-scheduler v1.27.3 v1.27.16
kube-proxy v1.27.3 v1.27.16
CoreDNS v1.10.1 v1.10.1
etcd 3.5.7-0 3.5.15-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.27.16
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 2 x v1.27.3 v1.28.15
Upgrade to the latest stable version:
COMPONENT CURRENT TARGET
kube-apiserver v1.27.3 v1.28.15
kube-controller-manager v1.27.3 v1.28.15
kube-scheduler v1.27.3 v1.28.15
kube-proxy v1.27.3 v1.28.15
CoreDNS v1.10.1 v1.10.1
etcd 3.5.7-0 3.5.15-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.28.15
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________문제없다면 kubeadm upgrade apply v1.28.15
$ kubeadm upgrade apply v1.28.15
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.28.15"
[upgrade/versions] Cluster version: v1.27.3
[upgrade/versions] kubeadm version: v1.28.15
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
W1223 13:16:22.963095 4164968 checks.go:835] detected that the sandbox image "time=\"2024-12-23T13:16:22+09:00\" level=warning msg=\"Config \\\"/etc/crictl.yaml\\\" does not exist, trying next: \\\"/usr/bin/crictl.yaml\\\"\"\nregistry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.28.15" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-12-23-13-16-26/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests151519974"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-12-23-13-16-26/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-12-23-13-16-26/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-12-23-13-16-26/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config23595869/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.28.15". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.로그에 출력되듯이, 컨트롤 플레인의 업그레이드 과정에는 Static Pod(kube-apiserver, etcd, kube-controller-manager, kube-scheduler) Yaml을 수정하는 과정이 있기 때문에 메이저 버전을 하나 건너 뛰어 업그레이드가 불가능한 것이다.
4. Kubelet, Kubectl Upgrade
kubectl, kubelet은 수동으로 업그레이드 해주어야 한다.
yum install -y kubelet kubectl --disableexcludes=kubernetes
systemctl restart kubelet5. 버전 확인
$ kubectl get no
NAME STATUS ROLES AGE VERSION
kube-master Ready control-plane 553d v1.28.15
kube-worker1 Ready <none> 552d v1.27.3
kube-worker2 Ready <none> 2d3h v1.27.36. 워커 노드
워커 노드 업데이트도 1, 2 까지 동일하고 3에서 kubeadm upgrade apply 가 아닌 kubeadm upgrade node 를 입력한다.
$ kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config507561166/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.그리고 4를 진행하면 된다.
yum install -y kubelet kubectl --disableexcludes=kubernetes
systemctl restart kubelet7. 버전 확인
$ kubectl get no
NAME STATUS ROLES AGE VERSION
kube-master Ready control-plane 553d v1.28.15
kube-worker1 Ready <none> 552d v1.28.15
kube-worker2 Ready <none> 2d3h v1.28.158. 1 ~ 7 반복
컨트롤 플레인은 1 ~ 4 과정을 반복해 v1.31 까지 업그레이드해주면 된다.
워커 노드는 메이저 버전을 건너뛰어서 업그레이드할 수 있으므로 컨트롤 플레인 업그레이드 종료 이후에 해도 괜찮으나, 혹시 모를 버전 간 차이 오류 때문에 컨트롤 플레인과 같이 업그레이드하는 것을 권장한다.
$ kubectl get no
NAME STATUS ROLES AGE VERSION
kube-master Ready control-plane 553d v1.31.4
kube-worker1 Ready <none> 552d v1.31.4
kube-worker2 Ready <none> 2d3h v1.31.4참고
안녕하세요. 방문해주셔서 감사합니다.