k8s deploy 3주차-2

dev_suha·2026년 1월 24일

k8s upgrade by kubeadm

K8S Version Skew Policy

NEW 버전 사용 가능 여부는 항상 kube-apiserver 기준
HA 환경에서는 “가장 낮은(apiserver OLD)” 버전을 기준으로 판단

1️⃣ kube-apiserver

환경	허용
HA	NEW(1.32) + OLD 1개(1.31)

2️⃣ kubelet

기준	허용
kube-apiserver 대비	NEW 불가 / OLD 최대 3개
apiserver 1.32	1.32, 1.31, 1.30, 1.29
apiserver HA (1.32, 1.31)	1.31, 1.30, 1.29

👉 HA에서는 최소 apiserver 버전(1.31) 기준

3️⃣ kube-controller-manager / kube-scheduler / cloud-controller-manager

기준	허용
kube-apiserver 대비	NEW 불가 / OLD 1개만 허용
apiserver 1.32	1.32, 1.31
apiserver HA (1.32, 1.31)	1.31만 가능

👉 그래서 HA 환경에서는 apiserver 전체를 먼저 1.32로 올린 후 업그레이드 필요

4️⃣ kube-proxy

기준	허용
kube-apiserver 대비	NEW 불가 / OLD 최대 3개
kubelet 대비	±3 마이너 허용
apiserver 1.32	1.32, 1.31, 1.30, 1.29
apiserver HA (1.32, 1.31)	1.31, 1.30, 1.29

5️⃣ kubectl

기준	허용
kube-apiserver 대비	NEW/OLD 1개 허용
apiserver 1.32	1.33, 1.32, 1.31
apiserver HA (1.32, 1.31)	1.32, 1.31

k8s upgrade 방식

in-place upgrade , blue-geen deployment

vagrantfile

# Base Image  https://portal.cloud.hashicorp.com/vagrant/discover/bento/rockylinux-10.0
BOX_IMAGE = "bento/rockylinux-10.0" # "bento/rockylinux-9"
BOX_VERSION = "202510.26.0"
N = 2 # max number of Worker Nodes

Vagrant.configure("2") do |config|
# ControlPlane Nodes 
    config.vm.define "k8s-ctr" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/K8S-Upgrade-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-ctr"
        vb.cpus = 4
        vb.memory = 3072 # 2048 2560 3072 4096
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-ctr"
      subconfig.vm.network "private_network", ip: "192.168.10.100"
      subconfig.vm.network "forwarded_port", guest: 22, host: "60000", auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
      subconfig.vm.provision "shell", path: "init_cfg.sh"
      subconfig.vm.provision "shell", path: "k8s-ctr.sh"
    end

# Worker Nodes
  (1..N).each do |i|
    config.vm.define "k8s-w#{i}" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/K8S-Upgrade-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-w#{i}"
        vb.cpus = 2
        vb.memory = 2048
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-w#{i}"
      subconfig.vm.network "private_network", ip: "192.168.10.10#{i}"
      subconfig.vm.network "forwarded_port", guest: 22, host: "6000#{i}", auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
      subconfig.vm.provision "shell", path: "init_cfg.sh"
      subconfig.vm.provision "shell", path: "k8s-w.sh"
    end
  end

end

init_cfg.sh

#!/usr/bin/env bash

echo ">>>> Initial Config Start <<<<"

echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul

echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

echo "[TASK 3] Disable and turn off SWAP & Delete swap partitions"
swapoff -a
sed -i '/swap/d' /etc/fstab
sfdisk --delete /dev/sda 2
partprobe /dev/sda

echo "[TASK 4] Config kernel & module"
cat << EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay >/dev/null 2>&1
modprobe br_netfilter >/dev/null 2>&1

cat << EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system >/dev/null 2>&1

echo "[TASK 5] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
cat << EOF >> /etc/hosts
192.168.10.100 k8s-ctr
192.168.10.101 k8s-w1
192.168.10.102 k8s-w2
EOF

echo "[TASK 6] Delete default routing - enp0s8 NIC" # setenforce 0 설정 필요
nmcli connection modify enp0s8 ipv4.never-default yes
nmcli connection up enp0s8 >/dev/null 2>&1

echo "[TASK 7] Install Containerd"
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo >/dev/null 2>&1
dnf install -y -q containerd.io-2.1.5-1.el10
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
systemctl daemon-reload
systemctl enable --now containerd >/dev/null 2>&1

echo "[TASK 8] Install kubeadm kubelet kubectl"
cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
dnf install -y -q kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable --now kubelet >/dev/null 2>&1

cat << EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
EOF

echo ">>>> Initial Config End <<<<"

k8s-ctr.sh

#!/usr/bin/env bash

echo ">>>> K8S Controlplane config Start <<<<"

echo "[TASK 1] Initial Kubernetes"
cat << EOF > kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
bootstrapTokens:
- token: "123456.1234567890123456"
  ttl: "0s"
  usages:
  - signing
  - authentication
nodeRegistration:
  kubeletExtraArgs:
    - name: node-ip
      value: "192.168.10.100"
  criSocket: "unix:///run/containerd/containerd.sock"
localAPIEndpoint:
  advertiseAddress: "192.168.10.100"
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: "1.32.11"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/16"
controllerManager:
  extraArgs:
    - name: "bind-address"
      value: "0.0.0.0"
scheduler:
  extraArgs:
    - name: "bind-address"
      value: "0.0.0.0"
etcd:
  local:
    extraArgs:
      - name: "listen-metrics-urls"
        value: "http://127.0.0.1:2381,http://192.168.10.100:2381"
EOF
kubeadm init --config="kubeadm-init.yaml"

echo "[TASK 2] Setting kube config file"
mkdir -p /root/.kube
cp -i /etc/kubernetes/admin.conf /root/.kube/config
chown $(id -u):$(id -g) /root/.kube/config

echo "[TASK 3] Install Helm"
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | DESIRED_VERSION=v3.18.6 bash >/dev/null 2>&1

echo "[TASK 4] Install kubecolor"
dnf install -y -q 'dnf-command(config-manager)' >/dev/null 2>&1
dnf config-manager --add-repo https://kubecolor.github.io/packages/rpm/kubecolor.repo >/dev/null 2>&1
dnf install -y -q kubecolor >/dev/null 2>&1

echo "[TASK 5] Install Kubectx & Kubens"
dnf install -y -q git >/dev/null 2>&1
git clone https://github.com/ahmetb/kubectx /opt/kubectx >/dev/null 2>&1
ln -s /opt/kubectx/kubens /usr/local/bin/kubens
ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx

echo "[TASK 6] Install Kubeps & Setting PS1"
git clone https://github.com/jonmosco/kube-ps1.git /root/kube-ps1 >/dev/null 2>&1
cat << "EOT" >> /root/.bash_profile
source /root/kube-ps1/kube-ps1.sh
KUBE_PS1_SYMBOL_ENABLE=true
function get_cluster_short() {
  echo "$1" | cut -d . -f1
}
KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
KUBE_PS1_SUFFIX=') '
PS1='$(kube_ps1)'$PS1
EOT
kubectl config rename-context "kubernetes-admin@kubernetes" "HomeLab" >/dev/null 2>&1

echo "[TASK 7] Install Flannel CNI"
/usr/local/bin/helm repo add flannel https://flannel-io.github.io/flannel >/dev/null 2>&1
kubectl create namespace kube-flannel >/dev/null 2>&1
cat << EOF > flannel.yaml
podCidr: "10.244.0.0/16"
flannel:
  cniBinDir: "/opt/cni/bin"
  cniConfDir: "/etc/cni/net.d"
  args:
  - "--ip-masq"
  - "--kube-subnet-mgr"
  - "--iface=enp0s8"  
  backend: "vxlan"
EOF
/usr/local/bin/helm install flannel flannel/flannel --namespace kube-flannel --version 0.27.3 -f flannel.yaml >/dev/null 2>&1

echo "[TASK 8] Source the completion"
echo 'source <(kubectl completion bash)' >> /etc/profile
echo 'source <(kubeadm completion bash)' >> /etc/profile

echo "[TASK 9] Alias kubectl to k"
echo 'alias k=kubectl' >> /etc/profile
echo 'alias kc=kubecolor' >> /etc/profile
echo 'complete -o default -F __start_kubectl k' >> /etc/profile

echo "[TASK 10] Install Metrics-server"
/usr/local/bin/helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/  >/dev/null 2>&1
/usr/local/bin/helm upgrade --install metrics-server metrics-server/metrics-server --set 'args[0]=--kubelet-insecure-tls' -n kube-system  >/dev/null 2>&1

echo "sudo su -" >> /home/vagrant/.bashrc

echo ">>>> K8S Controlplane Config End <<<<"

k8s-w.sh

#!/usr/bin/env bash

echo ">>>> K8S Node config Start <<<<"

echo "[TASK 1] K8S Controlplane Join"
NODEIP=$(ip -4 addr show enp0s8 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
cat << EOF > kubeadm-join.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: "123456.1234567890123456"
    apiServerEndpoint: "192.168.10.100:6443"
    unsafeSkipCAVerification: true
nodeRegistration:
  criSocket: "unix:///run/containerd/containerd.sock"
  kubeletExtraArgs:
    - name: node-ip
      value: "$NODEIP"
EOF
kubeadm join --config="kubeadm-join.yaml"

echo "sudo su -" >> /home/vagrant/.bashrc

echo ">>>> K8S Node config End <<<<"

enp0s9 → enp0s8 로 변경

upgrade 절차

실습 초기 버전 정보 : k8s 1.32.11

항목	버전	버전 호환성
Rocky Linux	10.0-1.6	version (kernel 6.12 - k8s_kernel) - kubespary
containerd	v2.1.5	CRI Version(v1), k8s 1.32~1.35 지원 - Link
runc	v1.3.3	정보 조사 필요 https://github.com/opencontainers/runc
kubelet, kubeadm, kubectl	v1.32.11	k8s 버전 정책 문서 참고 - Docs
helm	v3.18.6	k8s 1.30.x ~ 1.33.x 지원 - Docs
flannel cni	v0.27.3	k8s 1.28~ 이후 - Release

목표 버전 정보 : k8s 1.34.11 ⇒ 1.35(도전 과제)

항목	버전	버전 호환성
Rocky Linux	10.1-1.4	version (kernel 6.12 - k8s_kernel) - kubespary
containerd	v.2.1.5	CRI Version(v1), k8s 1.32~1.35 지원 - Link
runc	v1.x.y	정보 조사 필요 https://github.com/opencontainers/runc
kubelet, kubeadm, kubectl	v1.34.x	k8s 버전 정책 문서 참고 - Docs
helm	v3.19.x	k8s 1.31.x ~ 1.34.x 지원 - Docs
flannel cni	v0.28.0	k8s 1.28~ 이후 - Release , Upgrade

사전 준비 & etcd 백업
CNI Upgrade
OS Upgrade
Containerd Upgrade (skip)
Kubeadm , Kubelet, Kubectl Upgrade → 목표 버전까지 단계별 반복 작업
admin 용 kubeconfig 업데이트
worker node Upgrdade → 목표 버전까지 단계별 반복 작업

사전준비

kube-prometheus-stack 설치

vagrant ssh k8s-ctr

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

cat <<EOT > monitor-values.yaml
**prometheus**:
  prometheusSpec:
    **scrapeInterval**: "**20s**"
    **evaluationInterval**: "**20s**"
    externalLabels:
      cluster: "myk8s-cluster"
  **service**:
    type: NodePort
    nodePort: **30001**

**grafana**:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: **prom-operator**
  **service**:
    type: NodePort
    nodePort: **30002**

**alertmanager:
  enabled: true
defaultRules:
  create: true**

kubeProxy:
  enabled: false
****prometheus-windows-exporter:
  prometheus:
    monitor:
      enabled: false
EOT
cat monitor-values.yaml

# 배포
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version **80.13.3** \
-f **monitor-values.yaml** --create-namespace --namespace **monitoring

open http://192.168.10.100:30001** # prometheus
**open http://192.168.10.100:30002** # grafana : 접속 계정 admin / prom-operator

**## version 확인
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it sts/prometheus-kube-prometheus-stack-prometheus -n monitoring -c prometheus -- prometheus --version
prometheus, version 3.9.1 (branch: HEAD, revision: 9ec59baffb547e24f1468a53eb82901e58feabd8)
  build user:       root@61c3a9212c9e
  build date:       20260107-16:08:09
  go version:       go1.25.5
  platform:         linux/amd64
  tags:             netgo,builtinassets
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it -n monitoring deploy/kube-prometheus-stack-grafana -- grafana --version
grafana version 12.3.1

K8S Dashboard 추가**] Dashboard → New → Import → **15661, 15757**

grafana : 접속 계정 admin / prom-operator

샘플 애플리케이션 배포

# 샘플 애플리케이션 배포
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
EOF

# k8s-ctr 노드에 curl-pod 파드 배포

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: **Pod**
metadata:
  name: **curl-pod**
  labels:
    app: curl
spec:
  **nodeName: k8s-ctr**
  containers:
  - name: curl
    image: **nicolaka/netshoot**
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF

샘플 애플리케이션 확인 & 반복 호출

kubectl get deploy,svc,ep webpod -owide
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES           SELECTOR
deployment.apps/webpod   2/2     2            2           37s   webpod       traefik/whoami   app=webpod

NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/webpod   ClusterIP   10.96.78.109   <none>        80/TCP    37s   app=webpod

NAME               ENDPOINTS                     AGE
endpoints/webpod   10.244.1.6:80,10.244.3.6:80   37s

SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
**while true; do curl -s $SVCIP | grep Hostname; sleep 1; done**

ETCD 백업 (snapshot 저장)

(⎈|HomeLab:N/A) root@k8s-ctr:~# crictl images | grep etcd
registry.k8s.io/etcd                      3.5.24-0            8cb12dd0c3e42       23.7MB
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system etcd-k8s-ctr -- etcdctl version
E0122 23:49:02.481436   16408 websocket.go:296] Unknown stream id 1, discarding message
etcdctl version: 3.5.24
API version: 3.5
(⎈|HomeLab:N/A) root@k8s-ctr:~# ETCD_VER=3.5.24
(⎈|HomeLab:N/A) root@k8s-ctr:~# ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then ARCH=arm64; fi
echo $ARCH
amd64
(⎈|HomeLab:N/A) root@k8s-ctr:~# curl -L https://github.com/etcd-io/etcd/releases/download/v${ETCD_VER}/etcd-v${ETCD_VER}-linux-${ARCH}.tar.gz -o /tmp/etcd-v${ETCD_VER}.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 21.3M  100 21.3M    0     0  5090k      0  0:00:04  0:00:04 --:--:-- 6669k
(⎈|HomeLab:N/A) root@k8s-ctr:~# ls /tmp
etcd-v3.5.24.tar.gz                                                          systemd-private-4818c66b347c43c6a95a1487237756c7-irqbalance.service-qwgZPk      vagrant-shell
systemd-private-4818c66b347c43c6a95a1487237756c7-chronyd.service-uRYBkX      systemd-private-4818c66b347c43c6a95a1487237756c7-polkit.service-lAm05f
systemd-private-4818c66b347c43c6a95a1487237756c7-dbus-broker.service-rtEQ7F  systemd-private-4818c66b347c43c6a95a1487237756c7-systemd-logind.service-dPN91g
(⎈|HomeLab:N/A) root@k8s-ctr:~# mkdir -p /tmp/etcd-download
tar xzvf /tmp/etcd-v${ETCD_VER}.tar.gz -C /tmp/etcd-download --strip-components=1
etcd-v3.5.24-linux-amd64/Documentation/
etcd-v3.5.24-linux-amd64/Documentation/dev-guide/
etcd-v3.5.24-linux-amd64/Documentation/dev-guide/apispec/
etcd-v3.5.24-linux-amd64/Documentation/dev-guide/apispec/swagger/
etcd-v3.5.24-linux-amd64/Documentation/dev-guide/apispec/swagger/v3lock.swagger.json
etcd-v3.5.24-linux-amd64/Documentation/dev-guide/apispec/swagger/v3election.swagger.json
etcd-v3.5.24-linux-amd64/Documentation/dev-guide/apispec/swagger/rpc.swagger.json
etcd-v3.5.24-linux-amd64/Documentation/README.md
etcd-v3.5.24-linux-amd64/README-etcdutl.md
etcd-v3.5.24-linux-amd64/READMEv2-etcdctl.md
etcd-v3.5.24-linux-amd64/README-etcdctl.md
etcd-v3.5.24-linux-amd64/README.md
etcd-v3.5.24-linux-amd64/etcdutl
etcd-v3.5.24-linux-amd64/etcdctl
etcd-v3.5.24-linux-amd64/etcd
(⎈|HomeLab:N/A) root@k8s-ctr:~# mv /tmp/etcd-download/etcdctl /usr/local/bin/
mv /tmp/etcd-download/etcdutl /usr/local/bin/
chown root:root /usr/local/bin/etcdctl
chown root:root /usr/local/bin/etcdutl
(⎈|HomeLab:N/A) root@k8s-ctr:~# etcdctl version
etcdctl version: 3.5.24
API version: 3.5

etcdctl 기본 사용

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe pod -n kube-system etcd-k8s-ctr | grep isten-client-urls
      --listen-client-urls=https://127.0.0.1:2379,https://192.168.10.100:2379
(⎈|HomeLab:N/A) root@k8s-ctr:~# export ETCDCTL_API=3  # etcd v3 API를 사용
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379  # listen-client-urls
(⎈|HomeLab:N/A) root@k8s-ctr:~# etcdctl endpoint status -w table
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://127.0.0.1:2379 | f330bec74ce6cc42 |  3.5.24 |   20 MB |      true |      false |         2 |       4196 |               4196 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

etcd 백업

(⎈|HomeLab:N/A) root@k8s-ctr:~# mkdir /backup
(⎈|HomeLab:N/A) root@k8s-ctr:~# etcdctl snapshot save /backup/etcd-snapshot-$(date +%F).db
{"level":"info","ts":"2026-01-22T23:51:06.842975+0900","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/backup/etcd-snapshot-2026-01-22.db.part"}
{"level":"info","ts":"2026-01-22T23:51:06.923828+0900","logger":"client","caller":"v3@v3.5.24/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-01-22T23:51:06.924622+0900","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-01-22T23:51:10.044443+0900","logger":"client","caller":"v3@v3.5.24/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-01-22T23:51:10.351063+0900","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"20 MB","took":"3 seconds ago"}
{"level":"info","ts":"2026-01-22T23:51:10.351195+0900","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/backup/etcd-snapshot-2026-01-22.db"}
Snapshot saved at /backup/etcd-snapshot-2026-01-22.db
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /backup/
/backup/
└── etcd-snapshot-2026-01-22.db

1 directory, 1 file

(⎈|HomeLab:N/A) root@k8s-ctr:~# etcdutl snapshot status /backup/etcd-snapshot-2026-01-22.db
ee0cfba4, 3836, 1337, 20 MB

Flannel CNI 업그레이드

고려사항

노드에 미리 이미지 다운로드 하기
재기동시 배포된 app pod 영항도 파악

노드에 미리 이미지 다운로드 해두기

**crictl pull ghcr.io/flannel-io/flannel:v0.27.4
crictl pull ghcr.io/flannel-io/flannel:v1.8.0-flannel1

(⎈|HomeLab:N/A) root@k8s-ctr:~# crictl images | grep flannel
ghcr.io/flannel-io/flannel-cni-plugin     v1.7.1-flannel1     cca2af40a4a9e       4.88MB
ghcr.io/flannel-io/flannel                v0.27.3             5de71980e553f       34MB
ghcr.io/flannel-io/flannel                v0.27.4             e83704a177312       34.1MB

## k8s-w1, k8s-w2 
crictl pull ghcr.io/flannel-io/flannel:v0.27.4
crictl pull ghcr.io/flannel-io/flannel:v1.8.0-flannel1**
**crictl images | grep flannel

root@k8s-w1:~# crictl pull ghcr.io/flannel-io/flannel:v0.27.4
crictl pull ghcr.io/flannel-io/flannel:v1.8.0-flannel1
crictl images | grep flannel
Image is up to date for sha256:e83704a1773124a27dda6471ae26962cd6fac4fa8df3dac15034f8729a9ed3ab
E0125 00:29:34.481871   15087 log.go:32] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"ghcr.io/flannel-io/flannel:v1.8.0-flannel1\": failed to resolve image: ghcr.io/flannel-io/flannel:v1.8.0-flannel1: not found" image="ghcr.io/flannel-io/flannel:v1.8.0-flannel1"
FATA[0001] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/flannel-io/flannel:v1.8.0-flannel1": failed to resolve image: ghcr.io/flannel-io/flannel:v1.8.0-flannel1: not found
ghcr.io/flannel-io/flannel-cni-plugin                    v1.7.1-flannel1     cca2af40a4a9e       4.88MB
ghcr.io/flannel-io/flannel                               v0.27.3             5de71980e553f       34MB
ghcr.io/flannel-io/flannel                               v0.27.4             e83704a177312       34.1MB**

Flannel CNI 업그레이드 by helm

cat << EOF > flannel.yaml
podCidr: "10.244.0.0/16"
flannel:
  cniBinDir: "/opt/cni/bin"
  cniConfDir: "/etc/cni/net.d"
  args:
  - "--ip-masq"
  - "--kube-subnet-mgr"
  - "--iface=enp0s8"  
  backend: "vxlan"
**image:
  tag: v0.27.4**
EOF

**helm upgrade flannel flannel/flannel -n kube-flannel -f flannel.yaml --version 0.27.4**
kubectl -n kube-flannel rollout status ds/kube-flannel-ds

(⎈|HomeLab:N/A) root@k8s-ctr:~# helm list -n kube-flannel
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
flannel kube-flannel    2               2026-01-25 00:31:42.215814865 +0900 KST deployed        flannel-v0.27.4 v0.27.4

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ds -n kube-flannel -owide
NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE    CONTAINERS     IMAGES                               SELECTOR
kube-flannel-ds   3         3         3       3            3           <none>          2d1h   kube-flannel   ghcr.io/flannel-io/flannel:v0.27.4   app=flannel

enp0s8 로 변경

[k8s-ctr] Rocky Linux OS 마이너 버전 Upgrade : v10.0 → v10.1 ⇒ 노드 재기동

[k8s-ctr] 작업 전 정보 확인 : reboot 시 영향도 파악

 root@k8s-ctr:~# kubectl get pod -A -owide |grep k8s-ctr
default        curl-pod                                                    1/1     Running   0          2d      10.244.0.4       k8s-ctr   <none>          
 <none>
kube-flannel   kube-flannel-ds-kctf2                                       1/1     Running   0          94s     192.168.10.100   k8s-ctr   <none>          
 <none>
kube-system    coredns-668d6bf9bc-5msp4                                    1/1     Running   0          2d1h    10.244.0.3       k8s-ctr   <none>          
 <none>
kube-system    coredns-668d6bf9bc-p2nqr                                    1/1     Running   0          2d1h    10.244.0.2       k8s-ctr   <none>          
 <none>
kube-system    etcd-k8s-ctr                                                1/1     Running   0          2d1h    192.168.10.100   k8s-ctr   <none>          
 <none>
kube-system    kube-apiserver-k8s-ctr                                      1/1     Running   0          2d1h    192.168.10.100   k8s-ctr   <none>          
 <none>
kube-system    kube-controller-manager-k8s-ctr                             1/1     Running   0          2d1h    192.168.10.100   k8s-ctr   <none>          
 <none>
kube-system    kube-proxy-77s5q                                            1/1     Running   0          2d1h    192.168.10.100   k8s-ctr   <none>          
 <none>
kube-system    kube-scheduler-k8s-ctr                                      1/1     Running   0          2d1h    192.168.10.100   k8s-ctr   <none>          
 <none>
monitoring     kube-prometheus-stack-prometheus-node-exporter-r2kgt        1/1     Running   0          2d1h    192.168.10.100   k8s-ctr   <none>          
 <none>

k scale deployment -n kube-system coredns --replicas 1
**kubectl get pod -A -owide |grep k8s-ctr**
kubectl scale deployment -n kube-system coredns --replicas 2
kubectl get pod -n kube-system -owide | grep coredns

[k8s-ctr] Rocky Linux 마이너 버전 Upgrade : v10.0 → v10.1 ⇒ 노드 재기동 - Link

## Rocky Linux 10 커널 6.12 + VirtualBox SATA 호환성 버그로 인해 스터디내용 참고만 진행 
# 모니터링
## 터미널1 : 아래 호출 중단 발생 확인!
SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
[k8s-w1] SVCIP=<IP 직접 입력>
[k8s-w1] while true; do curl -s $SVCIP | grep Hostname; sleep 1; done # 이유는 k8s-ctr reboot 시에도 app 워크로드 통신 확인을 위함
혹은
kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'

# Rocky Linux 버전 확인
rpm -aq | grep release # rocky-release-10.0-1.6.el10.noarch
uname -r # 6.12.0-55.39.1.el10_0.aarch64

-------------------------------------------------------------
# (옵션) containerd.io 버전 업그레이드 되지 않게, 현재 설치 버전 고정 설정
rpm -q containerd.io  # 현재 설치 버전 정보 확인
containerd.io-2.1.5-1.el10.aarch64

## 버전 잠금 플러그인 설치
dnf install -y 'dnf-command(versionlock)'

## containerd.io 버전 잠금
dnf versionlock add containerd.io
Adding versionlock on: containerd.io-0:2.1.5-1.el10.*

## 버전 잠금 목록 확인
dnf versionlock list
containerd.io-0:2.1.5-1.el10.*
-------------------------------------------------------------

# update 수행 : 해당 과정에서 containerd 가 업그레이드 된것으로 보임. 정확한 버전 관리가 필요 시, containerd/runc 예외 설정 할 것
dnf -y update # 6.12.0-124.21.1.el10_1.aarch64
## 커널 업데이트 완료 : 새로운 리눅스 커널 버전인 6.12.0-124.27.1.el10_1.aarch64이 시스템에 성공적으로 설치됨.
## 부트로더 갱신 : 다음 재부팅 시 시스템은 가장 최신 버전인 6.12.0-124 커널로 부팅됨.

# 재기동 : 단일 Controlplane node 환경이므로, reboot 후 부팅 정상화 완료 기간에는 k8s api 통신 불가!
reboot
ping 192.168.10.100
...

# vagrant ssh k8s-ctr
rpm -aq | grep release # rocky-release-10.1-1.4.el10.noarch
uname -r               # 6.12.0-124.27.1.el10_1.aarch64

# 파드 기동 확인
kubectl get pod -A -owide |grep k8s-ctr
default        curl-pod                                                    1/1     Running   1 (2m25s ago)   61m    10.244.0.3       k8s-ctr   <none>           <none>
kube-flannel   kube-flannel-ds-f2572                                       1/1     Running   1 (2m25s ago)   15m    192.168.10.100   k8s-ctr   <none>           <none>
kube-system    coredns-668d6bf9bc-ctkmb                                    1/1     Running   1 (2m25s ago)   142m   10.244.0.2       k8s-ctr   <none>           <none>
kube-system    etcd-k8s-ctr                                                1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
kube-system    kube-apiserver-k8s-ctr                                      1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
kube-system    kube-controller-manager-k8s-ctr                             1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
kube-system    kube-proxy-wwd9c                                            1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
kube-system    kube-scheduler-k8s-ctr                                      1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
monitoring     kube-prometheus-stack-prometheus-node-exporter-6k5rk        1/1     Running   1 (2m25s ago)   36m    192.168.10.100   k8s-ctr   <none>           <none>

[k8s-ctr] Kubeadm , Kubelet, Kubectl Upgrade : v1.32.11 → v1.33.11

고려사항

Kubeadm 업그레이드 → kubelet,kubectl 업그레이드 → kubelet 재시작

kubeadm 신규 버전 설치 & kubeadm upgrade plan : kubelet/kubectl은 아직 업그레이드하지 않음

**cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF**
dnf makecache

## 설치 가능 버전 확인 
**dnf list --showduplicates kubeadm --disableexcludes=kubernetes**

## 설치 
dnf install -y --disableexcludes=kubernetes kubeadm-**1.33.7-150500.1.1

## 업그레이드 계획 확인 

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.32.11
[upgrade/versions] kubeadm version: v1.33.7
I0125 00:40:47.948913   24179 version.go:261] remote version is much newer: v1.35.0; falling back to: stable-1.33
[upgrade/versions] Target version: v1.33.7
[upgrade/versions] Latest version in the v1.32 series: v1.32.11

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT    TARGET
kubelet     k8s-ctr   v1.32.11   v1.33.7
kubelet     k8s-w1    v1.32.11   v1.33.7
kubelet     k8s-w2    v1.32.11   v1.33.7

Upgrade to the latest stable version:

COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            k8s-ctr   v1.32.11   v1.33.7
kube-controller-manager   k8s-ctr   v1.32.11   v1.33.7
kube-scheduler            k8s-ctr   v1.32.11   v1.33.7
kube-proxy                          1.32.11    v1.33.7
CoreDNS                             v1.11.3    v1.12.0
etcd                      k8s-ctr   3.5.24-0   3.5.24-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.33.7

_____________________________________________________________________

The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PR**

current, target 확인

kubeadm upgrade apply 실행

**# k8s-w1 k8s-w2 
kubeadm config images pull
crictl pull registry.k8s.io/kube-proxy:v1.33.7
crictl pull registry.k8s.io/coredns/coredns:v1.12.0

#k8s-ctr
kubeadm upgrade apply v1.33.7**

****

k8s 상태 확인

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get node -owide
NAME      STATUS   ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                 CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   2d1h   v1.32.11   192.168.10.100   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-w1    Ready    <none>          2d1h   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-w2    Ready    <none>          2d1h   v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.32.11
(⎈|HomeLab:N/A) root@k8s-ctr:~# crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
docker.io/nicolaka/netshoot               latest              9d3fca2526e3e       213MB
ghcr.io/flannel-io/flannel-cni-plugin     v1.7.1-flannel1     cca2af40a4a9e       4.88MB
ghcr.io/flannel-io/flannel-cni-plugin     v1.8.0-flannel1     bb28ded63816e       4.92MB
ghcr.io/flannel-io/flannel                v0.27.3             5de71980e553f       34MB
ghcr.io/flannel-io/flannel                v0.27.4             e83704a177312       34.1MB
quay.io/prometheus/node-exporter          v1.10.2             2715d223fd463       13.1MB
registry.k8s.io/coredns/coredns           v1.11.3             c69fa2e9cbf5f       18.6MB
registry.k8s.io/coredns/coredns           v1.12.0             1cf5f116067c6       20.9MB
registry.k8s.io/etcd                      3.5.24-0            8cb12dd0c3e42       23.7MB
registry.k8s.io/kube-apiserver            v1.32.11            7757c58248a29       29.1MB
registry.k8s.io/kube-apiserver            v1.33.7             021d1ceeffb11       30.1MB
registry.k8s.io/kube-controller-manager   v1.32.11            0175d0a8243db       26.7MB
registry.k8s.io/kube-controller-manager   v1.33.7             29c7cab9d8e68       27.7MB
registry.k8s.io/kube-proxy                v1.32.11            4d8fb2dc57519       31.2MB
registry.k8s.io/kube-proxy                v1.33.7             0929027b17fc3       31.9MB
registry.k8s.io/kube-scheduler            v1.32.11            23d6a1fb92fda       21.1MB
registry.k8s.io/kube-scheduler            v1.33.7             f457f6fcd712a       21.8MB
registry.k8s.io/pause                     3.10                873ed75102791       320kB

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pods -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}  - {.name}: {.image}{"\n"}{end}{"\n"}{end}'
coredns-674b8bbfcf-kjnfb
  - coredns: registry.k8s.io/coredns/coredns:v1.12.0

coredns-674b8bbfcf-mnfms
  - coredns: registry.k8s.io/coredns/coredns:v1.12.0

etcd-k8s-ctr
  - etcd: registry.k8s.io/etcd:3.5.24-0

kube-apiserver-k8s-ctr
  - kube-apiserver: registry.k8s.io/kube-apiserver:v1.33.7

kube-controller-manager-k8s-ctr
  - kube-controller-manager: registry.k8s.io/kube-controller-manager:v1.33.7

kube-proxy-6lp5l
  - kube-proxy: registry.k8s.io/kube-proxy:v1.33.7

kube-proxy-czzlz
  - kube-proxy: registry.k8s.io/kube-proxy:v1.33.7

kube-proxy-hlk87
  - kube-proxy: registry.k8s.io/kube-proxy:v1.33.7

kube-scheduler-k8s-ctr
  - kube-scheduler: registry.k8s.io/kube-scheduler:v1.33.7

metrics-server-5dd7b49d79-b426s
  - metrics-server: registry.k8s.io/metrics-server/metrics-server:v0.8.0

kubelet/kubectl 업그레이드

# Upgrade 설치 
dnf install -y --disableexcludes=kubernetes **kubelet**-**1.33.7-150500.1.1 kubectl**-**1.33.7-150500.1.1**

# 재시작
**systemctl daemon-reload
systemctl restart kubelet (수초간 단절) 

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe node k8s-ctr | grep 'Kubelet Version:'
The connection to the server 192.168.10.100:6443 was refused - did you specify the right host or port?
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe node k8s-ctr | grep 'Kubelet Version:'
Unable to connect to the server: net/http: TLS handshake timeout
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.33.7**

프로메테우스 알람 : 버전 관련 KubeVersionMismatch

[k8s-ctr] Kubeadm , Kubelet, Kubectl Upgrade : v1.33.11 → v1.34.3

kubeadm 신규 버전 설치 & kubeadm upgrade plan : kubelet/kubectl은 아직 업그레이드하지 않음

**cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF**
dnf makecache

**dnf list --showduplicates kubeadm --disableexcludes=kubernetes**

dnf install -y --disableexcludes=kubernetes kubeadm-**1.34.3-150500.1.1

Last metadata expiration check: 0:00:11 ago on Sun 25 Jan 2026 01:06:32 AM KST.
No match for argument: kubeadm-1.34.3-150500.1.1
Error: Unable to find a match: kubeadm-1.34.3-150500.1.1

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.33.7
[upgrade/versions] kubeadm version: v1.34.3
I0125 01:08:18.150127   33465 version.go:260] remote version is much newer: v1.35.0; falling back to: stable-1.34
[upgrade/versions] Target version: v1.34.3
[upgrade/versions] Latest version in the v1.33 series: v1.33.7

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT    TARGET
kubelet     k8s-w1    v1.32.11   v1.34.3
kubelet     k8s-w2    v1.32.11   v1.34.3
kubelet     k8s-ctr   v1.33.7    v1.34.3

Upgrade to the latest stable version:

COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            k8s-ctr   v1.33.7    v1.34.3
kube-controller-manager   k8s-ctr   v1.33.7    v1.34.3
kube-scheduler            k8s-ctr   v1.33.7    v1.34.3
kube-proxy                          1.33.7     v1.34.3
CoreDNS                             v1.12.0    v1.12.1
etcd                      k8s-ctr   3.5.24-0   3.6.5-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.34.3

_____________________________________________________________________

The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________**

kubeadm upgrade apply 실행 : etcd 신규 버전 기동

**kubeadm config images pull

crictl pull registry.k8s.io/kube-proxy:v1.34.3
crictl pull registry.k8s.io/coredns/coredns:v1.12.1
crictl pull registry.k8s.io/pause:3.10.1**
**crictl images

kubeadm upgrade apply v1.34.3 --yes

[upgrade/bootstrap-token] Configuring bootstrap token and cluster-info RBAC rules
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.34.3".**

k8s 상태 확인

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get node -owide
NAME      STATUS     ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                 CONTAINER-RUNTIME
k8s-ctr   Ready      control-plane   2d2h   v1.33.7    192.168.10.100   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-w1    NotReady   <none>          2d1h   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-w2    Ready      <none>          2d1h   v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.33.7
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pods -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}  - {.name}: {.image}{"\n"}{end}{"\n"}{end}'
coredns-66bc5c9577-jljxv
  - coredns: registry.k8s.io/coredns/coredns:v1.12.1

coredns-66bc5c9577-lzhjc
  - coredns: registry.k8s.io/coredns/coredns:v1.12.1

coredns-674b8bbfcf-kjnfb
  - coredns: registry.k8s.io/coredns/coredns:v1.12.0

coredns-674b8bbfcf-mnfms
  - coredns: registry.k8s.io/coredns/coredns:v1.12.0

etcd-k8s-ctr
  - etcd: registry.k8s.io/etcd:3.6.5-0

kube-apiserver-k8s-ctr
  - kube-apiserver: registry.k8s.io/kube-apiserver:v1.34.3

kube-controller-manager-k8s-ctr
  - kube-controller-manager: registry.k8s.io/kube-controller-manager:v1.34.3

kube-proxy-6lp5l
  - kube-proxy: registry.k8s.io/kube-proxy:v1.33.7

kube-proxy-czzlz
  - kube-proxy: registry.k8s.io/kube-proxy:v1.33.7

kube-proxy-hlk87
  - kube-proxy: registry.k8s.io/kube-proxy:v1.33.7

kube-scheduler-k8s-ctr
  - kube-scheduler: registry.k8s.io/kube-scheduler:v1.34.3

metrics-server-5dd7b49d79-b426s
  - metrics-server: registry.k8s.io/metrics-server/metrics-server:v0.8.0

metrics-server-5dd7b49d79-vllsd
  - metrics-server: registry.k8s.io/metrics-server/metrics-server:v0.8.0

kubelet/kubectl 업그레이드

dnf install -y --disableexcludes=kubernetes **kubelet-1.34.3-150500.1.1 kubectl-1.34.3-150500.1.1

systemctl daemon-reload 
systemctl restart kubelet

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get nodes -o wide
NAME      STATUS     ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                 CONTAINER-RUNTIME
k8s-ctr   Ready      control-plane   2d2h   v1.34.3    192.168.10.100   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-w1    NotReady   <none>          2d2h   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-w2    NotReady   <none>          2d1h   v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.34.3**

admin 용 kubeconfig 업데이트

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.34.3
(⎈|HomeLab:N/A) root@k8s-ctr:~# ls -l ~/.kube/config
-rw-------. 1 root root 5614 Jan 22 23:20 /root/.kube/config
(⎈|HomeLab:N/A) root@k8s-ctr:~# ls -l /etc/kubernetes/admin.conf 
-rw-------. 1 root root 5638 Jan 25 01:21 /etc/kubernetes/admin.conf
(⎈|HomeLab:N/A) root@k8s-ctr:~# yes | cp  /etc/kubernetes/admin.conf ~/.kube/config ; echo
chown $(id -u):$(id -g) ~/.kube/config
kubectl config rename-context "kubernetes-admin@kubernetes" "HomeLab"
kubens default
cp: overwrite '/root/.kube/config'? 
Context "kubernetes-admin@kubernetes" renamed to "HomeLab".
Context "HomeLab" modified.
Active namespace is "default".

worker node 작업 : v1.32.11 → v1.33.11 ⇒ v1.34.3

사전 정보 확인 & pdb 설정 추가

cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: **PodDisruptionBudget**
metadata:
  name: webpod
  namespace: default
spec:
  **maxUnavailable: 0**
  selector:
    matchLabels:
      app: webpod
EOF

작업 대상 워커 노드 1대 drain 후 상태 확인 : 파드 Evicted, 서비스 중단 모니터링

**kubectl drain k8s-w2

error: unable to drain node "k8s-w2" due to error: [cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-p2ncz, kube-system/kube-proxy-72ttq, monitoring/kube-prometheus-stack-prometheus-node-exporter-vc562, cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-5dd7b49d79-vllsd, monitoring/kube-prometheus-stack-grafana-5cb7c586f9-tsjsq, monitoring/prometheus-kube-prometheus-stack-prometheus-0], continuing command...
There are pending nodes to be drained:
 k8s-w2**

(⎈|HomeLab:default) root@k8s-ctr:~# kubectl get node
NAME      STATUS                     ROLES           AGE    VERSION
k8s-ctr   Ready                      control-plane   2d2h   v1.34.3
k8s-w1    NotReady(OOM)                   <none>          2d2h   v1.32.11
k8s-w2    Ready,SchedulingDisabled   <none>          2d2h   v1.32.11

(⎈|HomeLab:default) root@k8s-ctr:~# kubectl delete pdb webpod
poddisruptionbudget.policy "webpod" deleted from default namespace
(⎈|HomeLab:default) root@k8s-ctr:~# kubectl drain k8s-w2 --ignore-daemonsets --delete-emptydir-data
node/k8s-w2 already cordoned
.....
pod/kube-prometheus-stack-grafana-5cb7c586f9-tsjsq evicted
pod/coredns-66bc5c9577-jljxv evicted
node/k8s-w2 drained

Worker Node Drain : 특정 워커 노드에서 기존 파드를 안전하게 다른 노드로 이동(eviction) 시키고 해당 노드를 유지보수 가능 상태로 만드는 작업

작업 대상 워커 노드 1대 kubeadm v.1.33 업그레이드 :

vagrant ssh k8s-w1/w2

**cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF**
dnf makecache

dnf install -y --disableexcludes=kubernetes kubeadm-**1.33.7-150500.1.1**

# 노드 업그레이드 수행 : Upgrade commands for a node in the cluster
**kubeadm upgrade node

[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade/preflight] Running pre-flight checks
[upgrade/preflight] Skipping prepull. Not a control plane node.
[upgrade/control-plane] Skipping phase. Not a control plane node.
[upgrade/kubeconfig] Skipping phase. Not a control plane node.
W0125 01:42:43.697061   33741 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config3890165222 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config3890165222/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!**

작업 대상 워커 노드 1대 kubelet v.1.33 업그레이드 설치 → kubelet 재시작

**dnf list --showduplicates kubelet --disableexcludes=kubernetes
dnf list --showduplicates kubectl --disableexcludes=kubernetes**

dnf install -y --disableexcludes=kubernetes **kubelet**-**1.33.7-150500.1.1 kubectl**-**1.33.7-150500.1.1

systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet --no-pager

root@k8s-w2:~# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                                                    NAMESPACE
b4d18e496c6c8       36eef8e07bdd6       15 minutes ago      Running             kube-proxy          0                   fb1149abfd9b7       kube-proxy-72ttq                                       kube-system
599b558074ef2       2715d223fd463       15 minutes ago      Running             node-exporter       1                   2bc5bcbc30d5a       kube-prometheus-stack-prometheus-node-exporter-vc562   monitoring
62a8ece9bdc59       e83704a177312       About an hour ago   Running             kube-flannel        0                   4354ec868d1b0       kube-flannel-ds-p2ncz                                  kube-flannel**

작업 대상 워커 노드 1대 v.1.34 업그레이드

**cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF**
dnf makecache

root@k8s-w2:~# dnf install -y --disableexcludes=kubernetes kubeadm-1.34.3-150500.1.1
Last metadata expiration check: 0:00:04 ago on Sun 25 Jan 2026 01:45:12 AM KST.
Dependencies resolved.
=========================================================================================================================================================================================================================================
 Package                                              Architecture                                        Version                                                          Repository                                               Size 
=========================================================================================================================================================================================================================================
Upgrading:
 kubeadm                                              x86_64                                              1.34.3-150500.1.1                                                kubernetes                                               12 M 

Transaction Summary
=========================================================================================================================================================================================================================================
Upgrade  1 Package

Total download size: 12 M
Downloading Packages:
kubeadm-1.34.3-150500.1.1.x86_64.rpm                                                                                                                                                                     3.1 MB/s |  12 MB     00:03     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                    3.1 MB/s |  12 MB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                 1/1 
  Upgrading        : kubeadm-1.34.3-150500.1.1.x86_64                                                                                                                                                                                1/2 
  Cleanup          : kubeadm-1.33.7-150500.1.1.x86_64                                                                                                                                                                                2/2 
  Running scriptlet: kubeadm-1.33.7-150500.1.1.x86_64                                                                                                                                                                                2/2 

Upgraded:
  kubeadm-1.34.3-150500.1.1.x86_64

Complete!
root@k8s-w2:~# kubeadm upgrade node
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade/preflight] Running pre-flight checks
[upgrade/preflight] Skipping prepull. Not a control plane node.
[upgrade/control-plane] Skipping phase. Not a control plane node.
[upgrade/kubeconfig] Skipping phase. Not a control plane node.
W0125 01:45:28.412211   34716 postupgrade.go:116] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config1677569860 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1677569860/config.yaml
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/addon] Skipping the addon/coredns phase. Not a control plane node.
[upgrade/addon] Skipping the addon/kube-proxy phase. Not a control plane node.
root@k8s-w2:~# dnf install -y --disableexcludes=kubernetes kubelet-1.34.3-150500.1.1 kubectl-1.34.3-150500.1.1
Last metadata expiration check: 0:00:21 ago on Sun 25 Jan 2026 01:45:12 AM KST.
Dependencies resolved.
=========================================================================================================================================================================================================================================
 Package                                              Architecture                                        Version                                                          Repository                                               Size 
=========================================================================================================================================================================================================================================
Upgrading:
 kubectl                                              x86_64                                              1.34.3-150500.1.1                                                kubernetes                                               11 M 
 kubelet                                              x86_64                                              1.34.3-150500.1.1                                                kubernetes                                               12 M 

Transaction Summary
=========================================================================================================================================================================================================================================
Upgrade  2 Packages

Total download size: 24 M
Downloading Packages:
(1/2): kubelet-1.34.3-150500.1.1.x86_64.rpm                                                                                                                                                              2.7 MB/s |  12 MB     00:04    
(2/2): kubectl-1.34.3-150500.1.1.x86_64.rpm                                                                                                                                                              2.2 MB/s |  11 MB     00:05     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                    4.6 MB/s |  24 MB     00:05     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                 1/1 
  Upgrading        : kubelet-1.34.3-150500.1.1.x86_64                                                                                                                                                                                1/4 
  Running scriptlet: kubelet-1.34.3-150500.1.1.x86_64                                                                                                                                                                                1/4 
  Upgrading        : kubectl-1.34.3-150500.1.1.x86_64                                                                                                                                                                                2/4 
  Running scriptlet: kubelet-1.33.7-150500.1.1.x86_64                                                                                                                                                                                3/4 
  Cleanup          : kubelet-1.33.7-150500.1.1.x86_64                                                                                                                                                                                3/4 
  Running scriptlet: kubelet-1.33.7-150500.1.1.x86_64                                                                                                                                                                                3/4 
  Cleanup          : kubectl-1.33.7-150500.1.1.x86_64                                                                                                                                                                                4/4 
  Running scriptlet: kubectl-1.33.7-150500.1.1.x86_64                                                                                                                                                                                4/4 

Upgraded:
  kubectl-1.34.3-150500.1.1.x86_64                                                                                    kubelet-1.34.3-150500.1.1.x86_64

Complete!
root@k8s-w2:~# systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet --no-pager
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Sun 2026-01-25 01:46:08 KST; 85ms ago
 Invocation: 2cbcb24ea7544317a615d9b18877d4b9
       Docs: https://kubernetes.io/docs/
   Main PID: 35098 (kubelet)
      Tasks: 5 (limit: 12342)
     Memory: 7.4M (peak: 9.7M)
        CPU: 45ms
     CGroup: /system.slice/kubelet.service
             └─35098 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --node-ip=192.168.10.102 --pod-infra-container-image…

Jan 25 01:46:08 k8s-w2 systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
****

작업 대상 워커 노드 1대 uncordon 으로 정상화 → 파드 배치 확인

(⎈|HomeLab:default) root@k8s-ctr:~# kubectl uncordon k8s-w2
node/k8s-w2 uncordoned
(⎈|HomeLab:default) root@k8s-ctr:~# kubectl get node -owide
NAME      STATUS     ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                 CONTAINER-RUNTIME
k8s-ctr   Ready      control-plane   2d2h   v1.34.3    192.168.10.100   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-w1    NotReady   <none>          2d2h   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-w2    Ready      <none>          2d2h   v1.34.3    192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5

**kubectl scale deployment webpod --replicas 1**
kubectl get pod -owide
**kubectl scale deployment webpod --replicas 2**
**kubectl get pod -owide**

나머지 워커 노드 1대도 동일하게 작업 진행

워커 노드 1개 제거(reset)

kubeadm reset workflow internal design - Docs

kubeadm reset은 kubeadm으로 초기화되었거나 join된 노드에서 사용되는 명령으로, kubelet을 중지하고 실행 중인 Kubernetes 컨테이너를 종료한 뒤 /etc/kubernetes와 /var/lib/kubelet 등 kubeadm이 관리하던 파일과 설정만 정리하여 노드를 재사용 가능한 상태로 되돌리는 부분 초기화 작업입니다.

다만 CNI 설정, iptables·IPVS·nftables 규칙, containerd 이미지, 사용자 홈 디렉터리의 .kube/ 등은 정리하지 않으므로 완전 초기화를 원할 경우에는 반드시 수동 정리가 추가로 필요합니다.

[k8s-ctr] control 에서 노드 제거

**kubectl drain k8s-w2 --ignore-daemonsets --delete-emptydir-data

pod/prometheus-kube-prometheus-stack-prometheus-0 evicted
pod/kube-prometheus-stack-grafana-5cb7c586f9-v9htw evicted
node/k8s-w2 drained

kubectl delete node k8s-w2

node "k8s-w2" deleted**

(⎈|HomeLab:default) root@k8s-ctr:~# kubectl get node
NAME      STATUS     ROLES           AGE    VERSION
k8s-ctr   Ready      control-plane   2d2h   v1.34.3
k8s-w1    NotReady   <none>          2d2h   v1.32.11
(⎈|HomeLab:default) root@k8s-ctr:~#

제거 대상 노드에서 kubeadm reset 실행 : `vagrant ssh k8s-w1/w2`

**kubeadm reset -f

systemctl status containerd --no-pager 
systemctl status kubelet --no-pager

# kubelet 서비스 중지
systemctl stop kubelet && systemctl disable kubelet

# (옵션) contrainrd 서비스 중지
systemctl stop containerd && systemctl disable containerd

# reboot
reboot
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; preset: disabled)
     Active: active (running) since Sun 2026-01-25 01:58:44 KST; 2s ago
 Invocation: 759f496c596845a1ac10373a9af3df36
       Docs: https://containerd.io
    Process: 38300 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 38301 (containerd)
      Tasks: 9
     Memory: 32.4M (peak: 32.6M)
        CPU: 829ms
     CGroup: /system.slice/containerd.service
             └─38301 /usr/bin/containerd

Jan 25 01:58:43 k8s-w2 containerd[38301]: time="2026-01-25T01:58:43.879652079+09:00" level=info msg="Start recovering state"
Jan 25 01:58:44 k8s-w2 containerd[38301]: time="2026-01-25T01:58:44.282004361+09:00" level=info msg="Start event monitor"
Jan 25 01:58:44 k8s-w2 containerd[38301]: time="2026-01-25T01:58:44.282302485+09:00" level=info msg="Start cni network conf syncer for default"
Jan 25 01:58:44 k8s-w2 containerd[38301]: time="2026-01-25T01:58:44.282324913+09:00" level=info msg="Start streaming server"
Jan 25 01:58:44 k8s-w2 containerd[38301]: time="2026-01-25T01:58:44.282338599+09:00" level=info msg="Registered namespace \"k8s.io\" with NRI"
Jan 25 01:58:44 k8s-w2 containerd[38301]: time="2026-01-25T01:58:44.282350642+09:00" level=info msg="runtime interface starting up..."
Jan 25 01:58:44 k8s-w2 containerd[38301]: time="2026-01-25T01:58:44.282409547+09:00" level=info msg="starting plugins..."
Jan 25 01:58:44 k8s-w2 containerd[38301]: time="2026-01-25T01:58:44.282680102+09:00" level=info msg="Synchronizing NRI (plugin) with current runtime state"
Jan 25 01:58:44 k8s-w2 containerd[38301]: time="2026-01-25T01:58:44.283525771+09:00" level=info msg="containerd successfully booted in 0.599441s"
Jan 25 01:58:44 k8s-w2 systemd[1]: Started containerd.service - containerd container runtime.
○ kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: inactive (dead) since Sun 2026-01-25 01:49:44 KST; 9min ago
   Duration: 3min 36.455s
 Invocation: 2cbcb24ea7544317a615d9b18877d4b9
       Docs: https://kubernetes.io/docs/
    Process: 35098 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=0/SUCCESS)
   Main PID: 35098 (code=exited, status=0/SUCCESS)
   Mem peak: 46.2M
        CPU: 16.606s

Jan 25 01:49:41 k8s-w2 kubelet[35098]: E0125 01:49:41.555939   35098 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"k8s-w2\" not found" node="k8s-w2"
Jan 25 01:49:41 k8s-w2 kubelet[35098]: E0125 01:49:41.568484   35098 kubelet_node_status.go:486] "Error updating node status, will retry" err="error getting node \"k8s-w2\": nodes \"k8s-w2\" not found"
Jan 25 01:49:41 k8s-w2 kubelet[35098]: E0125 01:49:41.604815   35098 kubelet_node_status.go:486] "Error updating node status, will retry" err="error getting node \"k8s-w2\": nodes \"k8s-w2\" not found"
Jan 25 01:49:41 k8s-w2 kubelet[35098]: E0125 01:49:41.628618   35098 kubelet_node_status.go:486] "Error updating node status, will retry" err="error getting node \"k8s-w2\": nodes \"k8s-w2\" not found"
Jan 25 01:49:41 k8s-w2 kubelet[35098]: E0125 01:49:41.652056   35098 kubelet_node_status.go:486] "Error updating node status, will retry" err="error getting node \"k8s-w2\": nodes \"k8s-w2\" not found"
Jan 25 01:49:41 k8s-w2 kubelet[35098]: E0125 01:49:41.652123   35098 kubelet_node_status.go:473] "Unable to update node status" err="update node status exceeds retry count"
Jan 25 01:49:44 k8s-w2 systemd[1]: Stopping kubelet.service - kubelet: The Kubernetes Node Agent...
Jan 25 01:49:44 k8s-w2 systemd[1]: kubelet.service: Deactivated successfully.
Jan 25 01:49:44 k8s-w2 systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Jan 25 01:49:44 k8s-w2 systemd[1]: kubelet.service: Consumed 16.606s CPU time, 46.2M memory peak.
Removed '/etc/systemd/system/multi-user.target.wants/kubelet.service'.
Removed '/etc/systemd/system/multi-user.target.wants/containerd.service'.

Broadcast message from root@k8s-w2 on pts/2 (Sun 2026-01-25 01:58:48 KST):

The system will reboot now!

Broadcast message from root@k8s-w2 on pts/2 (Sun 2026-01-25 01:58:48 KST):

The system will reboot now!

root@k8s-w2:~# Connection to 127.0.0.1 closed by remote host**

모든 실습 완료 후 가상 머신 삭제 :

vagrant destroy -f && rm -rf .vagrant

dev_suha

이전 포스트

k8s deploy 3주차-1

다음 포스트

k8s deploy 3주차-2

k8s upgrade by kubeadm

K8S Version Skew Policy

1️⃣ kube-apiserver

2️⃣ kubelet

3️⃣ kube-controller-manager / kube-scheduler / cloud-controller-manager

4️⃣ kube-proxy

5️⃣ kubectl

k8s upgrade 방식

in-place upgrade , blue-geen deployment

vagrantfile

init_cfg.sh

k8s-ctr.sh

k8s-w.sh

upgrade 절차

사전준비

kube-prometheus-stack 설치

샘플 애플리케이션 배포

샘플 애플리케이션 확인 & 반복 호출

ETCD 백업 (snapshot 저장)

Flannel CNI 업그레이드

고려사항

노드에 미리 이미지 다운로드 해두기

Flannel CNI 업그레이드 by helm

[k8s-ctr] Rocky Linux OS 마이너 버전 Upgrade : v10.0 → v10.1 ⇒ 노드 재기동

[k8s-ctr] 작업 전 정보 확인 : reboot 시 영향도 파악

[k8s-ctr] Rocky Linux 마이너 버전 Upgrade : v10.0 → v10.1 ⇒ 노드 재기동 - Link

[k8s-ctr] Kubeadm , Kubelet, Kubectl Upgrade : v1.32.11 → v1.33.11

고려사항

kubeadm 신규 버전 설치 & kubeadm upgrade plan : kubelet/kubectl은 아직 업그레이드하지 않음

kubeadm upgrade apply 실행

k8s 상태 확인

kubelet/kubectl 업그레이드

프로메테우스 알람 : 버전 관련 KubeVersionMismatch

[k8s-ctr] Kubeadm , Kubelet, Kubectl Upgrade : v1.33.11 → v1.34.3

kubeadm 신규 버전 설치 & kubeadm upgrade plan : kubelet/kubectl은 아직 업그레이드하지 않음

kubeadm upgrade apply 실행 : etcd 신규 버전 기동

k8s 상태 확인

kubelet/kubectl 업그레이드

admin 용 kubeconfig 업데이트

worker node 작업 : v1.32.11 → v1.33.11 ⇒ v1.34.3

사전 정보 확인 & pdb 설정 추가

작업 대상 워커 노드 1대 drain 후 상태 확인 : 파드 Evicted, 서비스 중단 모니터링

작업 대상 워커 노드 1대 kubeadm v.1.33 업그레이드 :

작업 대상 워커 노드 1대 kubelet v.1.33 업그레이드 설치 → kubelet 재시작

작업 대상 워커 노드 1대 v.1.34 업그레이드

작업 대상 워커 노드 1대 uncordon 으로 정상화 → 파드 배치 확인

나머지 워커 노드 1대도 동일하게 작업 진행

워커 노드 1개 제거(reset)

kubeadm reset workflow internal design - Docs

[k8s-ctr] control 에서 노드 제거

제거 대상 노드에서 kubeadm reset 실행 : vagrant ssh k8s-w1/w2

모든 실습 완료 후 가상 머신 삭제 :

k8s deploy 3주차-1

k8s deploy 4주차

0개의 댓글

제거 대상 노드에서 kubeadm reset 실행 : `vagrant ssh k8s-w1/w2`