가시다님의 KANS [3기] 스터디 내용을 정리한 포스트 입니다.
1. Kind 소개 및 설치
Kind를 통해서 도커 IN 도커 docker in docker’로 쿠버네티스 클러스터 환경을 구성
- kind or kubernetes in docker is a suite of tooling for local Kubernetes “clusters” where each “node” is a Docker container
- kind is targeted at testing Kubernetes , kind supports multi-node (including HA) clusters
- kind uses kubeadm to configure cluster nodes.
[실습1] Kind 설치(macOS)
1) OrbStack 설치
‘Docker Desktop’ 대체제로 OrbStack 설치
참고 링크: https://mokpolar.tistory.com/61
brew install orbstack
2) Kind 및 툴 설치
- 필수 툴 설치
# Install Kind
brew install kind
kind --version
kind version 0.24.0
# Install kubectl
brew install kubernetes-cli
kubectl version --client=true
Client Version: v1.31.0
Kustomize Version: v5.4.2
# Install Helm
brew install helm
helm version
version.BuildInfo{Version:"v3.15.4", GitCommit:"fa9efb07d9d8debbb4306d72af76a383895aa8c4", GitTreeState:"clean", GoVersion:"go1.22.6"}
# Install Wireshark : 캡처된 패킷 확인
brew install --cask wireshark3) Kind 기본 사용
# 클러스터 배포 전 확인
docker ps
# Create a cluster with kind
kind create cluster
# 클러스터 배포 확인
kind get clusters
kind get nodes
kubectl cluster-info
# 노드 정보 확인
kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kind-control-plane Ready control-plane 2m48s v1.31.0 172.17.0.2 <none> Debian GNU/Linux 12 (bookworm) 6.10.7-orbstack-00280-gd3b7ec68d3d4 containerd://1.7.18
# 파드 정보 확인
kubectl get pod -A
kubectl get componentstatuses
# 컨트롤플레인 (컨테이너) 노드 1대가 실행
docker ps
docker images
# kube config 파일 확인
cat ~/.kube/config
혹은
cat $KUBECONFIG # KUBECONFIG 변수 지정 사용 시
# nginx 파드 배포 및 확인 : 컨트롤플레인 노드인데 파드가 배포 될까요?
kubectl run nginx --image=nginx:alpine
kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 12s 10.244.0.5 kind-control-plane <none> <none>
# 노드에 Taints 정보 확인
kubectl describe node | grep Taints
Taints: <none>4) 클러스터 삭제
# 클러스터 삭제
kind delete cluster
# kube config 삭제 확인
cat ~/.kube/config
혹은
cat $KUBECONFIG # KUBECONFIG 변수 지정 사용 시[실습2] Multi-Node Cluster (Control-plane, Nodes) with kube-ops-view & Mapping ports(macOS)
1) Multi-Node Cluster 생성
# '컨트롤플레인, 워커 노드 1대' 클러스터 배포 : 파드에 접속하기 위한 포트 맵핑 설정
cat <<EOT> kind-2node.yaml
# two node (one workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
extraPortMappings:
- containerPort: 31000
hostPort: 31000
listenAddress: "0.0.0.0" # Optional, defaults to "0.0.0.0"
protocol: tcp # Optional, defaults to tcp
- containerPort: 31001
hostPort: 31001
EOT
CLUSTERNAME=myk8s
kind create cluster --config kind-2node.yaml --name $CLUSTERNAME
# 배포 확인
kind get clusters
kind get nodes --name $CLUSTERNAME
myk8s-worker
myk8s-control-plane
# 노드 확인
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
myk8s-control-plane Ready control-plane 7m24s v1.31.0 172.17.0.3 <none> Debian GNU/Linux 12 (bookworm) 6.10.7-orbstack-00280-gd3b7ec68d3d4 containerd://1.7.18
myk8s-worker Ready <none> 7m13s v1.31.0 172.17.0.2 <none> Debian GNU/Linux 12 (bookworm) 6.10.7-orbstack-00280-gd3b7ec68d3d4 containerd://1.7.18
# 노드에 Taints 정보 확인
kubectl describe node $CLUSTERNAME-control-plane | grep Taints
Taints: node-role.kubernetes.io/control-plane:NoSchedule
kubectl describe node $CLUSTERNAME-worker | grep Taints
Taints: <none>
# 컨테이너 확인 : 컨테이너 갯수, 컨테이너 이름 확인
# kind yaml 에 포트 맵핑 정보 처럼, 자신의 PC 호스트에 31000 포트 접속 시, 워커노드(실제로는 컨테이너)에 TCP 31000 포트로 연결
# 즉, 워커노드에 NodePort TCP 31000 설정 시 자신의 PC 호스트에서 접속 가능!
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ea203f027f42 kindest/node:v1.31.0 "/usr/local/bin/entr…" 8 minutes ago Up 8 minutes 0.0.0.0:31000-31001->31000-31001/tcp myk8s-worker
35dbcbf231d5 kindest/node:v1.31.0 "/usr/local/bin/entr…" 8 minutes ago Up 8 minutes 127.0.0.1:63925->6443/tcp myk8s-control-plane
docker port $CLUSTERNAME-worker
31000/tcp -> 0.0.0.0:31000
31001/tcp -> 0.0.0.0:31001
# 컨테이너 내부 정보 확인 : 필요 시 각각의 노드(?)들에 bash로 접속하여 사용 가능
docker exec -it $CLUSTERNAME-control-plane ip -br -c -4 addr
lo UNKNOWN 127.0.0.1/8
veth09020b8b@if4 UP 10.244.0.1/32
vethc9ffbb53@if4 UP 10.244.0.1/32
vethef0a4f3f@if4 UP 10.244.0.1/32
eth0@if13 UP 172.17.0.3/16
docker exec -it $CLUSTERNAME-worker ip -br -c -4 addr
lo UNKNOWN 127.0.0.1/8
eth0@if11 UP 172.17.0.2/162) kube-ops-view 설치 및 실행
k8s cluster를 웹기반으로 간단하게 모니터링 할 수 있는 kube-ops-view를 설치한다.
# kube-ops-view
# helm show values geek-cookbook/kube-ops-view
helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=NodePort,service.main.ports.http.nodePort=31000 --set env.TZ="Asia/Seoul" --namespace kube-system
# 설치 확인
kubectl get deploy,pod,svc,ep -n kube-system -l app.kubernetes.io/instance=kube-ops-view
# kube-ops-view 접속 URL 확인 (1.5 , 2 배율)
echo -e "KUBE-OPS-VIEW URL = http://localhost:31000/#scale=1.5"
echo -e "KUBE-OPS-VIEW URL = http://localhost:31000/#scale=2"
3) nginx 배포 - NodePort 31001
# 디플로이먼트와 서비스 배포
cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-websrv
spec:
replicas: 2
selector:
matchLabels:
app: deploy-websrv
template:
metadata:
labels:
app: deploy-websrv
spec:
terminationGracePeriodSeconds: 0
containers:
- name: deploy-websrv
image: nginx:alpine
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: deploy-websrv
spec:
ports:
- name: svc-webport
port: 80
targetPort: 80
nodePort: 31001
selector:
app: deploy-websrv
type: NodePort
EOF
# 확인
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
117a1145a676 kindest/node:v1.29.2 "/usr/local/bin/entr…" 7 minutes ago Up 7 minutes 0.0.0.0:31000-31001->31000-31001/tcp myk8s-worker
...
kubectl get deploy,svc,ep deploy-websrv
...
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/deploy-websrv NodePort 10.96.204.112 <none> 80:31001/TCP 55s
...
# 자신의 PC에 호스트 포트 31001 접속 시 쿠버네티스 서비스에 접속 확인
open http://localhost:31001
curl -s localhost:31001 | grep -o "<title>.*</title>"
<title>Welcome to nginx!</title>
# 디플로이먼트와 서비스 삭제
kubectl delete deploy,svc deploy-websrv2. 파드 & PAUSE 컨테이너
파드는 1개 이상의 컨테이너로 구성된 컨테이너의 집합이며, PAUSE 컨테이너가 Network/IPC/UTS 네임스페이스를 생성하고 유지/공유함
- K8S CRI
CRI (Container Runtime Interface)는 Kubernetes에서 컨테이너 런타임과의 통신을 위한 표준 인터페이스입니다. 이를 통해 Kubernetes는 다양한 컨테이너 런타임(예: containerd, CRI-O)과 상호작용할 수 있습니다.
아래 요소를 통해 Kubernetes는 다양한 컨테이너 런타임과 일관되게 동작할 수 있으며, 플러그인 방식으로 런타임을 교체할 수 있습니다.
- Kubelet: Kubernetes의 주요 컴포넌트로, 노드에서 컨테이너를 실행, 관리합니다.
- CRI 인터페이스: Kubelet과 컨테이너 런타임 사이의 API로, 컨테이너 실행, 네트워킹, 스토리지 관리 등 다양한 기능을 수행합니다.
- 컨테이너 런타임: 실제로 컨테이너를 생성하고 관리하는 소프트웨어(예: containerd, CRI-O)
- Pause 컨테이너
Pause 컨테이너는 Kubernetes에서 Pod의 네트워크 네임스페이스를 유지하기 위해 사용되는 특별한 컨테이너입니다. 각 Pod는 하나의 IP 주소를 가지며, 이 IP는 Pause 컨테이너가 생성한 네트워크 네임스페이스에 연결됩니다. Pod 내의 다른 모든 컨테이너는 Pause 컨테이너와 같은 네트워크 네임스페이스를 공유하여 서로 통신할 수 있습니다.
- 네트워크 네임스페이스 유지: Pause 컨테이너는 Pod 내 다른 컨테이너의 네트워크 환경을 관리하고, 중지되지 않는 한 네트워크 구성을 유지합니다.
- 리소스 공유: Pod 내 모든 컨테이너가 동일한 네트워크 네임스페이스에서 실행되어 서로 IP 및 포트를 공유하게 합니다.
- 경량화: Pause 컨테이너는 매우 경량이며, 실제로 별도의 작업을 하지 않고 네트워크 환경만 유지하는 역할을 합니다.
[실습1] 파드 & PAUSE 컨테이너 확인
1) 실습 환경 구성하기
# '컨트롤플레인, 워커 노드 1대' 클러스터 배포 : 파드에 접속하기 위한 포트 맵핑 설정
cat <<EOT> kind-2node.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
extraPortMappings:
- containerPort: 30000
hostPort: 30000
- containerPort: 30001
hostPort: 30001
EOT
kind create cluster --config kind-2node.yaml --name myk8s
# 툴 설치
docker exec -it myk8s-control-plane sh -c 'apt update && apt install tree jq psmisc lsof wget bridge-utils tcpdump htop git nano -y'
docker exec -it myk8s-worker sh -c 'apt update && apt install tree jq psmisc lsof wget bridge-utils tcpdump htop -y'
# 확인
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
myk8s-control-plane Ready control-plane 75s v1.31.0 172.17.0.2 <none> Debian GNU/Linux 12 (bookworm) 6.10.7-orbstack-00280-gd3b7ec68d3d4 containerd://1.7.18
myk8s-worker Ready <none> 63s v1.31.0 172.17.0.3 <none> Debian GNU/Linux 12 (bookworm) 6.10.7-orbstack-00280-gd3b7ec68d3d4 containerd://1.7.18
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1f39ef4112ac kindest/node:v1.31.0 "/usr/local/bin/entr…" About a minute ago Up About a minute 0.0.0.0:30000-30001->30000-30001/tcp myk8s-worker
9470d135ec64 kindest/node:v1.31.0 "/usr/local/bin/entr…" About a minute ago Up About a minute 127.0.0.1:52263->6443/tcp myk8s-control-plane
docker port myk8s-worker
30000/tcp -> 0.0.0.0:30000
30001/tcp -> 0.0.0.0:30001
docker exec -it myk8s-control-plane ip -br -c -4 addr
lo UNKNOWN 127.0.0.1/8
veth662cc78d@if4 UP 10.244.0.1/32
vethec7eb0e6@if4 UP 10.244.0.1/32
vethe544038f@if4 UP 10.244.0.1/32
eth0@if16 UP 172.17.0.2/16
docker exec -it myk8s-worker ip -br -c -4 addr
lo UNKNOWN 127.0.0.1/8
eth0@if18 UP 172.17.0.3/16
# kube-ops-view
helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=NodePort,service.main.ports.http.nodePort=30000 --set env.TZ="Asia/Seoul" --namespace kube-system
# 설치 확인
kubectl get deploy,pod,svc,ep -n kube-system -l app.kubernetes.io/instance=kube-ops-view
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kube-ops-view 0/1 1 0 5s
NAME READY STATUS RESTARTS AGE
pod/kube-ops-view-657dbc6cd8-7dtnb 0/1 ContainerCreating 0 5s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-ops-view NodePort 10.96.77.138 <none> 8080:30000/TCP 5s
NAME ENDPOINTS AGE
endpoints/kube-ops-view <none> 5s
# kube-ops-view 접속 URL 확인 (1.5 : macOS 사용자
echo -e "KUBE-OPS-VIEW URL = http://localhost:30000/#scale=1.5"
2) Pod 배포 및 격리 확인
# [터미널1] myk8s-worker bash 진입 후 실행 및 확인
docker exec -it myk8s-worker bash
----------------------------------
containerd.service enabled enabled
kubelet.service enabled enabled
open-iscsi.service enabled enabled
undo-mount-hacks.service enabled enabled
#
crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
9fa592ac1033d a645de6a07a3d 2 minutes ago Running kube-ops-view 0 e8648bb52e3d9 kube-ops-view-657dbc6cd8-7dtnb
60ea3ddcafc6d 6a23fa8fd2b78 5 minutes ago Running kindnet-cni 0 a472c9bab082f kindnet-2c28v
0a8d27f288ac0 c573e1357a14e 5 minutes ago Running kube-proxy 0 7f08089cddf88 kube-proxy-fffrk
# 확인 : kubelet에 --container-runtime-endpoint=unix:///run/containerd/containerd.sock
pstree -aln
systemd
|-systemd-journal
|-containerd
| `-16*[{containerd}]
|-kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip=172.17.0.3 --node-labels= --pod-infra-container-image=registry.k8s.io/pause:3.10 --provider-id=kind://docker/myk8s/myk8s-worker --runtime-cgroups=/system.slice/containerd.service
| `-14*[{kubelet}]
|-containerd-shim -namespace k8s.io -id 7f08089cddf88604fa56c09813164c464593863133e279b758923ec5928b279c -address /run/containerd/containerd.sock
| |-12*[{containerd-shim}]
| |-pause
| `-kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=myk8s-worker
| `-8*[{kube-proxy}]
|-containerd-shim -namespace k8s.io -id a472c9bab082fd517666a17723ea8fc17a2cf3727056d49c0a1b58ffe06fbd63 -address /run/containerd/containerd.sock
| |-12*[{containerd-shim}]
| |-pause
| `-kindnetd
| `-12*[{kindnetd}]
`-containerd-shim -namespace k8s.io -id e8648bb52e3d9c7a2ea2e6520357978bed04c8ace13759b75f8171d147021694 -address /run/containerd/containerd.sock
|-12*[{containerd-shim}]
|-pause
`-python3 /usr/local/bin/python3 python3 -m kube_ops_view
`-2*[{python3}]
# 확인 : 파드내에 pause 컨테이너와 metrics-server 컨테이너, 네임스페이스 정보
pstree -aclnpsS
systemd,1
|-systemd-journal,88
|-containerd,102
| |-{containerd},103
| |-{containerd},104
| |-{containerd},105
| |-{containerd},106
| |-{containerd},107
| |-{containerd},108
| |-{containerd},109
| |-{containerd},110
| |-{containerd},111
| |-{containerd},112
| |-{containerd},113
| |-{containerd},114
| |-{containerd},115
| |-{containerd},349
| |-{containerd},351
| `-{containerd},352
|-kubelet,226 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip=172.17.0.3 --node-labels= --pod-infra-container-image=registry.k8s.io/pause:3.10 --provider-id=kind://docker/myk8s/myk8s-worker --runtime-cgroups=/system.slice/containerd.service
| |-{kubelet},227
| |-{kubelet},228
| |-{kubelet},229
| |-{kubelet},230
| |-{kubelet},231
| |-{kubelet},232
| |-{kubelet},233
| |-{kubelet},234
| |-{kubelet},235
| |-{kubelet},237
| |-{kubelet},238
| |-{kubelet},239
| |-{kubelet},254
| `-{kubelet},257
|-containerd-shim,273 -namespace k8s.io -id 7f08089cddf88604fa56c09813164c464593863133e279b758923ec5928b279c -address /run/containerd/containerd.sock
| |-{containerd-shim},278
| |-{containerd-shim},279
| |-{containerd-shim},280
| |-{containerd-shim},281
| |-{containerd-shim},282
| |-{containerd-shim},284
| |-{containerd-shim},286
| |-{containerd-shim},287
| |-{containerd-shim},289
| |-pause,320,ipc,mnt,pid
| |-{containerd-shim},335
| |-kube-proxy,379,ipc,mnt,pid --config=/var/lib/kube-proxy/config.conf --hostname-override=myk8s-worker
| | |-{kube-proxy},394
| | |-{kube-proxy},395
| | |-{kube-proxy},396
| | |-{kube-proxy},397
| | |-{kube-proxy},398
| | |-{kube-proxy},399
| | |-{kube-proxy},400
| | `-{kube-proxy},401
| |-{containerd-shim},607
| `-{containerd-shim},615
|-containerd-shim,290 -namespace k8s.io -id a472c9bab082fd517666a17723ea8fc17a2cf3727056d49c0a1b58ffe06fbd63 -address /run/containerd/containerd.sock
| |-{containerd-shim},291
| |-{containerd-shim},292
| |-{containerd-shim},293
| |-{containerd-shim},295
| |-{containerd-shim},297
| |-{containerd-shim},298
| |-{containerd-shim},299
| |-{containerd-shim},300
| |-{containerd-shim},308
| |-pause,329,ipc,mnt,pid
| |-{containerd-shim},342
| |-kindnetd,549,cgroup,ipc,mnt,pid
| | |-{kindnetd},564
| | |-{kindnetd},565
| | |-{kindnetd},566
| | |-{kindnetd},567
| | |-{kindnetd},568
| | |-{kindnetd},569
| | |-{kindnetd},570
| | |-{kindnetd},571
| | |-{kindnetd},572
| | |-{kindnetd},573
| | |-{kindnetd},597
| | `-{kindnetd},598
| |-{containerd-shim},606
| `-{containerd-shim},1033
`-containerd-shim,1078 -namespace k8s.io -id e8648bb52e3d9c7a2ea2e6520357978bed04c8ace13759b75f8171d147021694 -address /run/containerd/containerd.sock
|-{containerd-shim},1079
|-{containerd-shim},1080
|-{containerd-shim},1081
|-{containerd-shim},1082
|-{containerd-shim},1083
|-{containerd-shim},1084
|-{containerd-shim},1085
|-{containerd-shim},1086
|-pause,1097,ipc,mnt,net,pid,uts
|-{containerd-shim},1109
|-{containerd-shim},1110
|-python3,1162,cgroup,ipc,mnt,net,pid,uts /usr/local/bin/python3 python3 -m kube_ops_view
| |-{python3},1178
| `-{python3},1179
|-{containerd-shim},1184
`-{containerd-shim},1195
# 네임스페이스 확인 : lsns - List system namespaces
lsns -p 1
4026531834 time 15 1 root /sbin/init
4026531837 user 15 1 root /sbin/init
4026532619 mnt 9 1 root /sbin/init
4026532620 uts 13 1 root /sbin/init
4026532621 ipc 9 1 root /sbin/init
4026532622 pid 9 1 root /sbin/init
4026532623 net 13 1 root /sbin/init
4026532865 cgroup 13 1 root /sbin/init
4026532903 mnt 1 320 65535 /pause
4026532904 ipc 2 320 65535 /pause
4026532905 pid 1 320 65535 /pause
4026532906 mnt 1 329 65535 /pause
4026532907 ipc 2 329 65535 /pause
4026532908 pid 1 329 65535 /pause
4026532909 mnt 1 379 root /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=myk8s-worker
4026532910 pid 1 379 root /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=myk8s-worker
4026532911 mnt 1 549 root /bin/kindnetd
4026532912 pid 1 549 root /bin/kindnetd
4026532913 cgroup 1 549 root /bin/kindnetd
4026533298 net 2 1097 65535 /pause
4026533419 mnt 1 1097 65535 /pause
4026533420 uts 2 1097 65535 /pause
4026533421 ipc 2 1097 65535 /pause
4026533422 pid 1 1097 65535 /pause
4026533423 mnt 1 1162 1000 [rosetta] /usr/local/bin/python3 python3 -m kube_ops_view
4026533424 pid 1 1162 1000 [rosetta] /usr/local/bin/python3 python3 -m kube_ops_view
4026533425 cgroup 1 1162 1000 [rosetta] /usr/local/bin/python3 python3 -m kube_ops_view
lsns -p $$
NS TYPE NPROCS PID USER COMMAND
4026531834 time 15 1 root /sbin/init
4026531837 user 15 1 root /sbin/init
4026532619 mnt 9 1 root /sbin/init
4026532620 uts 13 1 root /sbin/init
4026532621 ipc 9 1 root /sbin/init
4026532622 pid 9 1 root /sbin/init
4026532623 net 13 1 root /sbin/init
4026532865 cgroup 13 1 root /sbin/init
# 해당 파드에 pause 컨테이너는 호스트NS와 다른 5개의 NS를 가짐 : mnt/pid 는 pasue 자신만 사용, net/uts/ipc는 app 컨테이너를 위해서 먼저 생성해둠
lsns -p 1097
NS TYPE NPROCS PID USER COMMAND
4026531834 time 15 1 root /sbin/init
4026531837 user 15 1 root /sbin/init
4026532865 cgroup 13 1 root /sbin/init
4026533298 net 2 1097 65535 /pause
4026533419 mnt 1 1097 65535 /pause
4026533420 uts 2 1097 65535 /pause
4026533421 ipc 2 1097 65535 /pause
4026533422 pid 1 1097 65535 /pause
# app 컨테이너(metrics-server)는 호스트NS와 다른 6개의 NS를 가짐 : mnt/pid/cgroup 는 자신만 사용, net/uts/ipc는 pause 컨테이너가 생성한 것을 공유 사용함
pgrep python3
lsns -p $(pgrep python3)
pgrep metrics-server
1896
lsns -p $(pgrep metrics-server)
NS TYPE NPROCS PID USER COMMAND
4026531834 time 15 1 root /sbin/init
4026531837 user 15 1 root /sbin/init
4026533298 net 2 1097 65535 /pause
4026533420 uts 2 1097 65535 /pause
4026533421 ipc 2 1097 65535 /pause
4026533423 mnt 1 1162 1000 [rosetta] /usr/local/bin/python3 python3 -m kube_ops_view
4026533424 pid 1 1162 1000 [rosetta] /usr/local/bin/python3 python3 -m kube_ops_view
4026533425 cgroup 1 1162 1000 [rosetta] /usr/local/bin/python3 python3 -m kube_ops_view
#
ls -l /run/containerd/containerd.sock
srw-rw---- 1 root root 0 Sep 7 06:45 /run/containerd/containerd.sock
# 특정 소켓 파일을 사용하는 프로세스 확인
lsof /run/containerd/containerd.sock
root@myk8s-worker:/# lsof /run/containerd/containerd.sock
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
container 102 root 9u unix 0x000000003e5ba4c2 0t0 470753 /run/containerd/containerd.sock type=STREAM (LISTEN)
container 102 root 11u unix 0x000000006e64e129 0t0 477106 /run/containerd/containerd.sock type=STREAM (CONNECTED)
container 102 root 12u unix 0x000000003df4d372 0t0 483375 /run/containerd/containerd.sock type=STREAM (CONNECTED)
container 102 root 13u unix 0x000000008acd89d7 0t0 477675 /run/containerd/containerd.sock type=STREAM (CONNECTED)
#
ss -xl | egrep 'Netid|containerd'
Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
u_str LISTEN 0 4096 /run/containerd/s/f37eb0715030a0cc0c98bfa2e71d4d76ee26ea27eab716e4e3ac15199e0567ec 505052 * 0
u_str LISTEN 0 4096 /run/containerd/containerd.sock.ttrpc 470750 * 0
u_str LISTEN 0 4096 /run/containerd/containerd.sock 470753 * 0
u_str LISTEN 0 4096 /run/containerd/s/ecd661332499220fd7737949a448d50549750becc21b04ee043f3ad22c399a69 477795 * 0
u_str LISTEN 0 4096 /run/containerd/s/e2728ff238b8f809ddb6c8289015d311b2d7540b4a1d5a987b4126504f141a54 479022 * 0
#
findmnt -A
TARGET SOURCE FSTYPE OPTIONS
/ overlay overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/HW4BGGJ4LV6M5
...
|-/sys sysfs sysfs ro,nosuid,nodev,noexec,relatime
| |-/sys/kernel/debug debugfs debugfs rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/tracing tracefs tracefs rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/fuse/connections fusectl fusectl rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/config configfs configfs rw,nosuid,nodev,noexec,relatime
| \-/sys/fs/cgroup cgroup cgroup2 rw,nosuid,nodev,noexec,relatime
findmnt -t cgroup2
TARGET SOURCE FSTYPE OPTIONS
/sys/fs/cgroup cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate
grep cgroup /proc/filesystems
nodev cgroup
nodev cgroup2
stat -fc %T /sys/fs/cgroup/
cgroup2fs
----------------------------------3) 신규 파드를 배포하고 확인 - myweb 파드 생성
# [터미널2] kubectl 명령 실행 및 확인
# Pod 생성 : YAML 파일에 컨테이너가 사용할 포트(TCP 80)을 설정
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: myweb
spec:
containers:
- image: nginx:alpine
name: myweb-container
ports:
- containerPort: 80
protocol: TCP
terminationGracePeriodSeconds: 0
EOF
# Pod 정보 확인 : pause 컨테이너 정보가 보이는가?
kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myweb 1/1 Running 0 12s 10.244.1.3 myk8s-worker <none> <none>
kubectl describe pod myweb
kubectl get pod myweb -o json # status.conditions 에 Type 정보 확인 : 시간 정렬은 안되어 있음.
# [터미널1] myk8s-worker bash 진입 후 실행 및 확인
docker exec -it myk8s-worker bash
----------------------------------
crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
715c69079c26d 70594c812316a 4 minutes ago Running myweb-container 0 3d86f7cd72bbd myweb
9fa592ac1033d a645de6a07a3d 15 minutes ago Running kube-ops-view 0 e8648bb52e3d9 kube-ops-view-657dbc6cd8-7dtnb
60ea3ddcafc6d 6a23fa8fd2b78 18 minutes ago Running kindnet-cni 0 a472c9bab082f kindnet-2c28v
0a8d27f288ac0 c573e1357a14e 18 minutes ago Running kube-proxy 0 7f08089cddf88 kube-proxy-fffrk
pstree -aln
pstree -aclnpsS # 파드내에 pause 컨테이너와 app 컨테이너, 네임스페이스 정보
# 네임스페이스 확인 : lsns - List system namespaces
lsns -p 1
NS TYPE NPROCS PID USER COMMAND
4026531834 time 26 1 root /sbin/init
4026531837 user 26 1 root /sbin/init
4026532619 mnt 10 1 root /sbin/init
4026532620 uts 14 1 root /sbin/init
4026532621 ipc 10 1 root /sbin/init
4026532622 pid 10 1 root /sbin/init
4026532623 net 14 1 root /sbin/init
4026532865 cgroup 15 1 root /sbin/init
lsns -p $$
NS TYPE NPROCS PID USER COMMAND
4026531834 time 26 1 root /sbin/init
4026531837 user 26 1 root /sbin/init
4026532619 mnt 10 1 root /sbin/init
4026532620 uts 14 1 root /sbin/init
4026532621 ipc 10 1 root /sbin/init
4026532622 pid 10 1 root /sbin/init
4026532623 net 14 1 root /sbin/init
4026532865 cgroup 15 1 root /sbin/init
lsns -p 1388 #<pstree -aclnpsS에서 출력된 pause 컨테이너 PID>
lsns -p $(pgrep nginx) # app 컨테이너(metrics-server)
----------------------------------
# [터미널2] kubectl 명령 실행 및 확인
kubectl delete pod myweb[실습2] 파드(2개의 컨테이너 동작) & PAUSE 컨테이너 확인
1) 신규 파드를 추가 배포하고 확인 - myweb2 파드에 2개의 컨테이너가 동작
- myweb2.yaml 생성
apiVersion: v1
kind: Pod
metadata:
name: myweb2
spec:
containers:
- name: myweb2-nginx
image: nginx
ports:
- containerPort: 80
protocol: TCP
- name: myweb2-netshoot
image: nicolaka/netshoot
command: ["/bin/bash"]
args: ["-c", "while true; do sleep 5; curl localhost; done"] # 포드가 종료되지 않도록 유지합니다
terminationGracePeriodSeconds: 02) Pod 배포 후 확인(NET IPC UTS 공유 확인)
# [터미널1] 파드 생성
kubectl apply -f ./myweb2.yaml
# 확인
# pod 정보 READY 에 2/2 를 확인 : pod 내 모든 컨테이너가 정상이여야지 status 가 Running 가 됨
kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myweb 1/1 Running 0 63m 10.244.1.3 myk8s-worker <none> <none>
myweb2 2/2 Running 0 34m 10.244.1.4 myk8s-worker <none> <none>
# Pod 상세 정보에 컨테이너 2개 정보가 보인다
kubectl describe pod myweb2
root@k8s-m:~# kubectl describe pod myweb2
Name: myweb2
...
Containers:
myweb2-nginx:
Container ID: docker://2717dd093ee5c69a918c6c52461f47cf5f0c0330378730ce717d1fcabb0fc748
Image: nginx
...
myweb2-netshoot:
Container ID: docker://e3e3aef9ee53ef805336d4b6e0986f63e23c767b1648d18ff09948815c5f06a9
Image: nicolaka/netshoot
...
# 파드의 각각 컨테이너 IP 확인 >> IP가 같다!
kubectl exec myweb2 -c myweb2-netshoot -- ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host proto kernel_lo
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
link/tunnel6 :: brd :: permaddr e7b:6696:6eba::
4: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether aa:b7:f4:30:1a:4e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.1.4/24 brd 10.244.1.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a8b7:f4ff:fe30:1a4e/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
kubectl exec myweb2 -c myweb2-nginx -- apt update
kubectl exec myweb2 -c myweb2-nginx -- apt install -y net-tools
kubectl exec myweb2 -c myweb2-nginx -- ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.244.1.4 netmask 255.255.255.0 broadcast 10.244.1.255
inet6 fe80::a8b7:f4ff:fe30:1a4e prefixlen 64 scopeid 0x20<link>
ether aa:b7:f4:30:1a:4e txqueuelen 0 (Ethernet)
RX packets 588 bytes 9419708 (8.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 556 bytes 38554 (37.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 3572 bytes 537602 (525.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3572 bytes 537602 (525.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# myweb2-netshoot 컨테이너 zsh 진입
kubectl exec myweb2 -c myweb2-netshoot -it -- zsh
----------------------------------
ifconfig
ss -tnlp
curl localhost # nginx 컨테이너가 아닌데, 로컬 접속 되고 tcp 80 listen 이다. 왜그럴까?
ps -ef # nginx 프로세스 정보가 안보이는데...
exit
----------------------------------
# 터미널3 : nginx 컨테이너 웹 접속 로그 출력 : 접속자(myweb2-netshoot)의 IP 가 ::1(ipv6) 혹은 127.0.0.1(ipv4) 이닷!
kubectl logs -f myweb2 -c myweb2-nginx
::1 - - [01/Sep/2024:06:33:26 +0000] "GET / HTTP/1.1" 200 615 "-" "curl/8.7.1" "-"
혹은
127.0.0.1 - - [16/Jun/2021:06:22:24 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.77.0" "-"
# [터미널2]
docker exec -it myk8s-worker bash
----------------------------------
# 컨테이너 정보 확인 : POD 와 POD ID가 같음을 확인
crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
9ee5bd26f3707 e286c635d1232 38 minutes ago Running myweb2-netshoot 0 1ead0ddecd433 myweb2
a343e08591e68 195245f0c7927 38 minutes ago Running myweb2-nginx 0 1ead0ddecd433 myweb2
715c69079c26d 70594c812316a About an hour ago Running
...
# 워커 노드에서 컨테이너 프로세스 정보 확인
ps -ef | grep 'nginx -g' | grep -v grep
root 1450 1368 0 07:11 ? 00:00:00 nginx: master process nginx -g daemon off;
root 1979 1901 0 07:40 ? 00:00:00 nginx: master process nginx -g daemon off;
ps -ef | grep 'curl' | grep -v grep
root 2113 1901 0 07:40 ? 00:00:00 /bin/bash -c while true; do sleep 5; curl localhost; done
# 각각 프로세스를 변수에 지정
NGINXPID=$(ps -ef | grep 'nginx -g' | grep -v grep | awk '{print $2}')
echo $NGINXPID
1450 1979
NETSHPID=$(ps -ef | grep 'curl' | grep -v grep | awk '{print $2}')
echo $NETSHPID
2113
# 한 파드 내의 각 컨테이너의 네임스페이스 정보 확인
## time, user 네임스페이스는 호스트와 같음, 격리하지 않음
## mnt, uts, pid 네임스페이스는 컨테이너별로 격리
## ipc, uts, net 네임스페이스는 파드 내의 컨테이너 간 공유 (IPC : 컨테이너 프로세스간 공유 - signal, socket, pipe 등)
## Pause 컨테이너는 IPC, Network, UTS 네임스페이스를 생성하고 유지 -> 나머지 컨테이너들은 해당 네임스페이스를 공유하여 사용
## 유저가 실행한 특정 컨테이너가 비정상 종료되어 컨터이너 전체에서 공유되는 네임스페이스에 문제가 발생하는 것을 방지
lsns -p $NGINXPID
NS TYPE NPROCS PID USER COMMAND
4026531834 time 28 1 root /sbin/init
4026531837 user 28 1 root /sbin/init
4026533482 net 12 2112 65535 /pause
4026533611 uts 12 2112 65535 /pause
4026533612 ipc 12 2112 65535 /pause
4026533614 mnt 9 2172 root nginx: master process nginx -g daemon off;
4026533615 pid 9 2172 root nginx: master process nginx -g daemon off;
4026533616 cgroup 9 2172 root nginx: master process nginx -g daemon off;
lsns -p $NETSHPID
NS TYPE NPROCS PID USER COMMAND
4026531834 time 28 1 root /sbin/init
4026531837 user 28 1 root /sbin/init
4026533482 net 12 2112 65535 /pause
4026533611 uts 12 2112 65535 /pause
4026533612 ipc 12 2112 65535 /pause
4026533617 mnt 2 2296 root /bin/bash -c while true; do sleep 5; curl localhost; done
4026533618 pid 2 2296 root /bin/bash -c while true; do sleep 5; curl localhost; done
4026533619 cgroup 2 2296 root /bin/bash -c while true; do sleep 5; curl localhost; done
# pause 정보 확인 :
PAUSEPID=<각자 자신의 pause PID>
PAUSEPID=2113
lsns -p $PAUSEPID
NS TYPE NPROCS PID USER COMMAND
4026531834 time 40 1 root /sbin/init
4026531837 user 40 1 root /sbin/init
4026533554 net 13 1922 65535 /pause
4026533676 uts 13 1922 65535 /pause
4026533677 ipc 13 1922 65535 /pause
4026533682 mnt 3 2113 root /bin/bash -c while true; do sleep 5; curl localhost; done
4026533683 pid 3 2113 root /bin/bash -c while true; do sleep 5; curl localhost; done
4026533684 cgroup 3 2113 root /bin/bash -c while true; do sleep 5; curl localhost; done
# 개별 컨테이너에 명령 실행 : IP 동일 확인
crictl ps
crictl ps -q
crictl exec -its 904e43d8fca65 ifconfig
root@myk8s-worker:/# crictl exec -its 9ee5bd26f3707 ifconfig
eth0 Link encap:Ethernet HWaddr AA:B7:F4:30:1A:4E
inet addr:10.244.1.4 Bcast:10.244.1.255 Mask:255.255.255.0
inet6 addr: fe80::a8b7:f4ff:fe30:1a4e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:588 errors:0 dropped:0 overruns:0 frame:0
TX packets:557 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9419708 (8.9 MiB) TX bytes:38624 (37.7 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:4844 errors:0 dropped:0 overruns:0 frame:0
TX packets:4844 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:728932 (711.8 KiB) TX bytes:728932 (711.8 KiB)
crictl exec -its 63f82edc9caa6 ifconfig
root@myk8s-worker:/# crictl exec -its a343e08591e68 ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.244.1.4 netmask 255.255.255.0 broadcast 10.244.1.255
inet6 fe80::a8b7:f4ff:fe30:1a4e prefixlen 64 scopeid 0x20<link>
ether aa:b7:f4:30:1a:4e txqueuelen 0 (Ethernet)
RX packets 588 bytes 9419708 (8.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 557 bytes 38624 (37.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 4940 bytes 743372 (725.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4940 bytes 743372 (725.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# PAUSE 의 NET 네임스페이스 PID 확인 및 IP 정보 확인
lsns -t net
NS TYPE NPROCS PID USER NETNSID NSFS COMMAND
4026532623 net 15 1 root unassigned /sbin/init
4026533298 net 2 1097 65535 1 /run/netns/cni-811662b7-d95f-5e7b-c45d-fbbe25f99d79 /pause
4026533426 net 10 1388 65535 2 /run/netns/cni-13c7dca7-bf61-f60c-e2af-5408a6a4cca2 /pause
4026533554 net 13 1922 65535 3 /run/netns/cni-76292e91-fcf1-cf36-1023-8a47ccb986fb /pause
nsenter -t $PAUSEPID -n ip -c addr
nsenter -t $NGINXPID -n ip -c addr
nsenter -t $NETSHPID -n ip -c addr
# 2개의 네임스페이스 비교 , 아래 1922 프로세스의 정제는? => NET IPC UTS
crictl inspect a343e08591e68 | jq #<myweb2-nginx 컨테이너ID>
crictl inspect 9ee5bd26f3707 | jq #<myweb2-netshoot 컨테이너ID>
...
"namespaces": [
{
"type": "pid"
},
{
"type": "ipc",
"path": "/proc/1922/ns/ipc"
},
{
"type": "uts",
"path": "/proc/1922/ns/uts"
},
{
"type": "mount"
},
{
"type": "network",
"path": "/proc/1922/ns/net"
},
{
"type": "cgroup"
}
],
...3) 파드 및 클러스터 삭제
#파드 삭제
kubectl delete pod myweb2
#클러스터 삭제
kind delete cluster --name myk8s3. Flannel CNI
쿠버네티스는 네트워크 모델의 요건을 만족하는 CNI 플러그인이 있고 Flannel은 주로 간단하고 가벼운 네트워킹을 제공하는데 사용되는 CNI Plugin입니다.
- 간단한 네트워킹 구성: Flannel은 Kubernetes에서 Pod 간 네트워킹을 쉽게 설정할 수 있는 CNI 플러그인입니다.
- 오버레이 네트워크: Flannel은 각 노드에 고유한 서브넷을 할당하고 이를 통해 Pod 간 통신을 가능하게 합니다.
- 경량 솔루션: 복잡한 네트워킹 요구가 없는 경우, Flannel은 가벼운 대안으로 많이 사용됩니다.
- 네트워킹 환경 지원 (Backends) : VXLAN, host-gw, UDP, 그외에는 아직 실험적임 - 링크 링크2
- VXLAN (권장) : Port(UDP 8472), DirecRouting 지원(같은 서브넷 노드와는 host-gw 처럼 동작)
- 단, 네트워크 엔지니어분들이 알고 있는 ‘L2 확장’ 이 아니라, 각 노드마다 별도의 서브넷이 있고, 해당 서브넷 대역끼리 NAT 없이 라우팅 처리됨
- host-gw : 호스트 L2 모드?, 일반적으로 퍼블릭 클라우드 환경에서는 동작하지 않는다
- UDP (비권장) : VXLAN 지원하지 않는 오래된 커널 사용 시, Port(UDP 8285)
- 노드마다 flannel.1 생성 : VXLAN VTEP 역할 , 뒷에 숫자는 VXLAN id 1 에 1을 의미
- 노드마다 cni0 생성 : bridge 역할
[실습1] kind & Flannel 배포
1) k8s cluster 배포
#
cat <<EOF> kind-cni.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
labels:
mynode: control-plane
extraPortMappings:
- containerPort: 30000
hostPort: 30000
- containerPort: 30001
hostPort: 30001
- containerPort: 30002
hostPort: 30002
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
controllerManager:
extraArgs:
bind-address: 0.0.0.0
etcd:
local:
extraArgs:
listen-metrics-urls: http://0.0.0.0:2381
scheduler:
extraArgs:
bind-address: 0.0.0.0
- |
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0
- role: worker
labels:
mynode: worker
- role: worker
labels:
mynode: worker2
networking:
disableDefaultCNI: true
EOF
kind create cluster --config kind-cni.yaml --name myk8s --image kindest/node:v1.30.4
# 배포 확인
kind get clusters
kind get nodes --name myk8s
kubectl cluster-info
# 네트워크 확인
kubectl cluster-info dump | grep -m 2 -E "cluster-cidr|service-cluster-ip-range"
kubectl cluster-info dump | grep -m 2 -E "cluster-cidr|service-cluster-ip-range"
"--service-cluster-ip-range=10.96.0.0/16",
"--cluster-cidr=10.244.0.0/16",
# 노드 확인 : CRI
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
myk8s-control-plane NotReady control-plane 96s v1.30.4 172.17.0.3 <none> Debian GNU/Linux 12 (bookworm) 6.10.7-orbstack-00280-gd3b7ec68d3d4 containerd://1.7.18
myk8s-worker NotReady <none> 76s v1.30.4 172.17.0.2 <none> Debian GNU/Linux 12 (bookworm) 6.10.7-orbstack-00280-gd3b7ec68d3d4 containerd://1.7.18
myk8s-worker2 NotReady <none> 76s v1.30.4 172.17.0.4 <none> Debian GNU/Linux 12 (bookworm) 6.10.7-orbstack-00280-gd3b7ec68d3d4 containerd://1.7.18
# 노드 라벨 확인
kubectl get nodes myk8s-control-plane -o jsonpath={.metadata.labels} | jq
{
"beta.kubernetes.io/arch": "arm64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/arch": "arm64",
"kubernetes.io/hostname": "myk8s-control-plane",
"kubernetes.io/os": "linux",
"mynode": "control-plane",
"node-role.kubernetes.io/control-plane": "",
"node.kubernetes.io/exclude-from-external-load-balancers": ""
}
kubectl get nodes myk8s-worker -o jsonpath={.metadata.labels} | jq
kubectl get nodes myk8s-worker -o jsonpath={.metadata.labels} | jq
{
"beta.kubernetes.io/arch": "arm64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/arch": "arm64",
"kubernetes.io/hostname": "myk8s-worker",
"kubernetes.io/os": "linux",
"mynode": "worker"
}
kubectl get nodes myk8s-worker2 -o jsonpath={.metadata.labels} | jq
kubectl get nodes myk8s-worker2 -o jsonpath={.metadata.labels} | jq
{
"beta.kubernetes.io/arch": "arm64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/arch": "arm64",
"kubernetes.io/hostname": "myk8s-worker2",
"kubernetes.io/os": "linux",
"mynode": "worker2"
}
# 컨테이너 확인 : 컨테이너 갯수, 컨테이너 이름 확인
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4f96197f8600 kindest/node:v1.30.4 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes myk8s-worker
1d776ac46f30 kindest/node:v1.30.4 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes 0.0.0.0:30000-30002->30000-30002/tcp, 127.0.0.1:55359->6443/tcp myk8s-control-plane
cf586ca2e028 kindest/node:v1.30.4 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes myk8s-worker2
docker port myk8s-control-plane
6443/tcp -> 127.0.0.1:55359
30000/tcp -> 0.0.0.0:30000
30001/tcp -> 0.0.0.0:30001
30002/tcp -> 0.0.0.0:30002
docker port myk8s-worker
docker port myk8s-worker2
# 컨테이너 내부 정보 확인
docker exec -it myk8s-control-plane ip -br -c -4 addr
lo UNKNOWN 127.0.0.1/8
eth0@if23 UP 172.17.0.3/16
docker exec -it myk8s-worker ip -br -c -4 addr
lo UNKNOWN 127.0.0.1/8
eth0@if21 UP 172.17.0.2/16
docker exec -it myk8s-worker2 ip -br -c -4 addr
lo UNKNOWN 127.0.0.1/8
eth0@if25 UP 172.17.0.4/16
#
docker exec -it myk8s-control-plane sh -c 'apt update && apt install tree jq psmisc lsof wget bridge-utils tcpdump iputils-ping htop git nano -y'
docker exec -it myk8s-worker sh -c 'apt update && apt install tree jq psmisc lsof wget bridge-utils tcpdump iputils-ping -y'
docker exec -it myk8s-worker2 sh -c 'apt update && apt install tree jq psmisc lsof wget bridge-utils tcpdump iputils-ping -y'2) bridge 실행 파일 생성
#
docker exec -it myk8s-control-plane bash
apt install golang -y
git clone https://github.com/containernetworking/plugins
cd plugins
chmod +x build_linux.sh
#
./build_linux.sh
Building plugins
bandwidth
firewall
portmap
sbr
tuning
vrf
bridge
dummy
host-device
ipvlan
loopback
macvlan
ptp
tap
vlan
dhcp
host-local
static
# 파일 권한 확인 755
root@myk8s-control-plane:/plugins# ls -l bin | grep bridge
-rwxr-xr-x 1 root root 4471145 Sep 7 09:41 bridge
exit
-------
#mac local로 복사
docker cp bridge myk8s-control-plane:/opt/cni/bin/bridge .
ls -l bridge
#로컬에 복사한 bridge 파일을 kind 노드에 배포
docker cp bridge myk8s-control-plane:/opt/cni/bin/bridge
docker cp bridge myk8s-worker:/opt/cni/bin/bridge
docker cp bridge myk8s-worker2:/opt/cni/bin/bridge
docker exec -it myk8s-control-plane chmod 755 /opt/cni/bin/bridge
docker exec -it myk8s-worker chmod 755 /opt/cni/bin/bridge
docker exec -it myk8s-worker2 chmod 755 /opt/cni/bin/bridge
3) Flannel 배포
#
watch -d kubectl get pod -A -owide
#
kubectl describe pod -n kube-system -l k8s-app=kube-dns
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 57s default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
# Flannel cni 설치
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
# namespace 에 pod-security.kubernetes.io/enforce=privileged Label 확인
kubectl get ns --show-labels
NAME STATUS AGE LABELS
kube-flannel Active 2m49s k8s-app=flannel,kubernetes.io/metadata.name=kube-flannel,pod-security.kubernetes.io/enforce=privileged
kubectl get ds,pod,cm -n kube-flannel
kubectl describe cm -n kube-flannel kube-flannel-cfg
cni-conf.json:
----
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json:
----
{
"Network": "10.244.0.0/16",
"EnableNFTables": false,
"Backend": {
"Type": "vxlan"
}
}
kubectl describe ds -n kube-flannel kube-flannel-ds
Name: kube-flannel-ds
Selector: app=flannel
Node-Selector: <none>
Labels: app=flannel
k8s-app=flannel
tier=node
Annotations: deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 3
Number of Nodes Misscheduled: 0
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=flannel
tier=node
Service Account: flannel
Init Containers:
install-cni-plugin:
Image: docker.io/flannel/flannel-cni-plugin:v1.5.1-flannel2
Port: <none>
Host Port: <none>
Command:
cp
Args:
-f
/flannel
/opt/cni/bin/flannel
Environment: <none>
Mounts:
/opt/cni/bin from cni-plugin (rw)
install-cni:
Image: docker.io/flannel/flannel:v0.25.6
Port: <none>
Host Port: <none>
Command:
cp
Args:
-f
/etc/kube-flannel/cni-conf.json
/etc/cni/net.d/10-flannel.conflist
Environment: <none>
Mounts:
/etc/cni/net.d from cni (rw)
/etc/kube-flannel/ from flannel-cfg (rw)
Containers:
kube-flannel:
Image: docker.io/flannel/flannel:v0.25.6
Port: <none>
Host Port: <none>
Command:
/opt/bin/flanneld
Args:
--ip-masq
--kube-subnet-mgr
Requests:
cpu: 100m
memory: 50Mi
Environment:
POD_NAME: (v1:metadata.name)
POD_NAMESPACE: (v1:metadata.namespace)
EVENT_QUEUE_DEPTH: 5000
Mounts:
/etc/kube-flannel/ from flannel-cfg (rw)
/run/flannel from run (rw)
/run/xtables.lock from xtables-lock (rw)
Volumes:
run:
Type: HostPath (bare host directory volume)
Path: /run/flannel
HostPathType:
cni-plugin:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
flannel-cfg:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-flannel-cfg
Optional: false
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
Priority Class Name: system-node-critical
Node-Selectors: <none>
Tolerations: :NoSchedule op=Exists
kubectl exec -it ds/kube-flannel-ds -n kube-flannel -c kube-flannel -- ls -l /etc/kube-flannel
lrwxrwxrwx 1 root root 20 Sep 7 09:27 cni-conf.json -> ..data/cni-conf.json
lrwxrwxrwx 1 root root 20 Sep 7 09:27 net-conf.json -> ..data/net-conf.json
# failed to find plugin "bridge" in path [/opt/cni/bin] => step2에서 bridge 복사 후 해결
kubectl get pod -A -owide
#
docker exec -it myk8s-control-plane ls -l /opt/cni/bin/
docker exec -it myk8s-worker ls -l /opt/cni/bin/
docker exec -it myk8s-worker2 ls -l /opt/cni/bin/
for i in myk8s-control-plane myk8s-worker myk8s-worker2; do echo ">> node $i <<"; docker exec -it $i ls /opt/cni/bin/; echo; done
>> node myk8s-control-plane <<
bridge flannel host-local loopback portmap ptp
>> node myk8s-worker <<
bridge flannel host-local loopback portmap ptp
>> node myk8s-worker2 <<
bridge flannel host-local loopback portmap ptp
#
kubectl get pod -A -owide
4) Flannel 정보 확인
#
kubectl get ds,pod,cm -n kube-flannel -owide
kubectl describe cm -n kube-flannel kube-flannel-cfg
...
cni-conf.json:
----
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json:
----
{
"Network": "10.244.0.0/16",
"EnableNFTables": false,
"Backend": {
"Type": "vxlan"
}
}
...
# iptables 정보 확인
for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-control-plane iptables -t $i -S ; echo; done
for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-worker iptables -t $i -S ; echo; done
for i in filter nat mangle raw ; do echo ">> IPTables Type : $i <<"; docker exec -it myk8s-worker2 iptables -t $i -S ; echo; done
# flannel 정보 확인 : 대역, MTU
for i in myk8s-control-plane myk8s-worker myk8s-worker2; do echo ">> node $i <<"; docker exec -it $i cat /run/flannel/subnet.env ; echo; done
>> node myk8s-control-plane <<
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24 # 해당 노드에 파드가 배포 시 할당 할 수 있는 네트워크 대역
FLANNEL_MTU=1450 # MTU 지정
FLANNEL_IPMASQ=true # 파드가 외부(인터넷) 통신 시 해당 노드의 마스커레이딩을 사용
...
# 노드마다 할당된 dedicated subnet (podCIDR) 확인
kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}' ;echo
10.244.0.0/24 10.244.2.0/24 10.244.1.0/24
# 노드 정보 중 flannel 관련 정보 확인 : VXLAN 모드 정보와, VTEP 정보(노드 IP, VtepMac) 를 확인
kubectl describe node | grep -A3 Annotations
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"5a:38:be:34:b9:16"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 172.17.0.3
--
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"3e:18:4f:26:2f:06"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 172.17.0.2
--
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"96:68:ec:3b:76:b0"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 172.17.0.4
# 각 노드 마다 bash 진입 후 아래 기본 정보 확인 : 먼저 worker 부터 bash 진입 후 확인하자
docker exec -it myk8s-worker bash
docker exec -it myk8s-worker2 bash
docker exec -it myk8s-control-plane bash
----------------------------------------
# 호스트 네트워크 NS와 flannel, kube-proxy 컨테이너의 네트워크 NS 비교 : 파드의 IP와 호스트(서버)의 IP를 비교해보자!
lsns -p 1
lsns -p $(pgrep flanneld)
lsns -p $(pgrep kube-proxy)
# 기본 네트워크 정보 확인
ip -c -br addr
ip -c link | grep -E 'flannel|cni|veth' -A1
ip -c addr
ip -c -d addr show cni0 # 네트워크 네임스페이스 격리 파드가 1개 이상 배치 시 확인됨
ip -c -d addr show flannel.1
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 1e:81:fc:d5:8a:42 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
vxlan id 1 local 192.168.100.10 dev enp0s8 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 172.16.0.0/32 brd 172.16.0.0 scope global flannel.1
brctl show
bridge name bridge id STP enabled interfaces
cni0 8000.663bf746b6a8 no vethbce1591c
vethc17ba51b
vethe6540260
# 라우팅 정보 확인 : 다른 노드의 파드 대역(podCIDR)의 라우팅 정보가 업데이트되어 있음을 확인
ip -c route
default via 172.17.0.1 dev eth0
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.3
# flannel.1 인터페이스를 통한 ARP 테이블 정보 확인 : 다른 노드의 flannel.1 IP와 MAC 정보를 확인
ip -c neigh show dev flannel.1
10.244.2.0 lladdr 3e:18:4f:26:2f:06 PERMANENT
10.244.1.0 lladdr 96:68:ec:3b:76:b0 PERMANENT
# 브리지 fdb 정보에서 해당 MAC 주소와 통신 시 각 노드의 enp0s8
bridge fdb show dev flannel.1
96:68:ec:3b:76:b0 dst 172.17.0.4 self permanent
3e:18:4f:26:2f:06 dst 172.17.0.2 self permanent
# 다른 노드의 flannel.1 인터페이스로 ping 통신 : VXLAN 오버레이를 통해서 통신
ping -c 1 10.244.0.0
ping -c 1 10.244.1.0
ping -c 1 10.244.2.0
# iptables 필터 테이블 정보 확인 : 파드의 10.244.0.0/16 대역 끼리는 모든 노드에서 전달이 가능
iptables -t filter -S | grep 10.244.0.0
-A FLANNEL-FWD -s 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A FLANNEL-FWD -d 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
# iptables NAT 테이블 정보 확인 : 10.244.0.0/16 대역 끼리 통신은 마스커레이딩 없이 통신을 하며,
# 10.244.0.0/16 대역에서 동일 대역(10.244.0.0/16)과 멀티캐스트 대역(224.0.0.0/4) 를 제외한 나머지 (외부) 통신 시에는 마스커레이딩을 수행
iptables -t nat -S | grep 'flanneld masq' | grep -v '! -s'
-A POSTROUTING -m comment --comment "flanneld masq" -j FLANNEL-POSTRTG
-A FLANNEL-POSTRTG -m mark --mark 0x4000/0x4000 -m comment --comment "flanneld masq" -j RETURN
-A FLANNEL-POSTRTG -s 10.244.0.0/24 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j RETURN
-A FLANNEL-POSTRTG -s 10.244.0.0/16 -d 10.244.0.0/24 -m comment --comment "flanneld masq" -j RETURN
-A FLANNEL-POSTRTG -s 10.244.0.0/16 ! -d 224.0.0.0/4 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
----------------------------------------[실습2] Pod간 통신 확인
1) Pod 2개 생성
# [터미널1,2] 워커 노드1,2 - 모니터링
docker exec -it myk8s-worker bash
docker exec -it myk8s-worker2 bash
-----------------------------
watch -d "ip link | egrep 'cni|veth' ;echo; brctl show cni0"
-----------------------------
# [터미널3] cat & here document 명령 조합으로 즉석(?) 리소스 생성
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: pod-1
labels:
app: pod
spec:
nodeSelector:
kubernetes.io/hostname: myk8s-worker
containers:
- name: netshoot-pod
image: nicolaka/netshoot
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
---
apiVersion: v1
kind: Pod
metadata:
name: pod-2
labels:
app: pod
spec:
nodeSelector:
kubernetes.io/hostname: myk8s-worker2
containers:
- name: netshoot-pod
image: nicolaka/netshoot
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
# 파드 확인 : IP 확인
kubectl get pod -o wide
2) Pod 2개 생성후 정보 확인
# [터미널1,2] 워커 노드1,2
docker exec -it myk8s-worker bash
docker exec -it myk8s-worker2 bash
-----------------------------
# 브리지 정보 확인
brctl show cni0
# 브리지 연결 링크(veth) 확인
bridge link
# 브리지 VLAN 정보 확인
bridge vlan
# cbr(custom bridge) 정보 : kubenet CNI의 bridge - 링크
tree /var/lib/cni/networks/cbr0
/var/lib/cni/networks/cbr0
├── 10.244.2.4.
├── last_reserved_ip.0
└── lock
# 네트워크 관련 정보들 확인
ip -c addr | grep veth -A3
8: veth3060489e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether ea:26:f0:2a:c9:cb brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::e826:f0ff:fe2a:c9cb/64 scope link
valid_lft forever preferred_lft forever
...
-----------------------------
통신 흐름 이해: 동일 노드, 다른 노드 간
- 동일 노드
- 다른 노드 간
3) 파드 Shell 접속 후 확인 & 패킷 캡처
kubectl exec -it pod-1 -- zsh
-----------------------------
ip -c addr show eth0
# GW IP는 어떤 인터페이스인가? (1) flannel.1 (2) cni0
ip -c route
route -n
ping -c 1 <GW IP>
ping -c 1 <pod-2 IP> # 다른 노드에 배포된 파드 통신 확인
ping -c 8.8.8.8 # 외부 인터넷 IP 접속 확인
curl -s wttr.in/Seoul # 외부 인터넷 도메인 접속 확인
ip -c neigh
exit
-----------------------------
- 패킷 캡처 후 확인
# [터미널1,2] 워커 노드1,2
docker exec -it myk8s-worker bash
docker exec -it myk8s-worker2 bash
-----------------------------
tcpdump -i cni0 -nn icmp
tcpdump -i flannel.1 -nn icmp
tcpdump -i eth0 -nn icmp
tcpdump -i eth0 -nn udp port 8472 -w /root/vxlan.pcap
# CTRL+C 취소 후 확인 : ls -l /root/vxlan.pcap
conntrack -L | grep -i icmp
-----------------------------
# [터미널3]
docker cp myk8s-worker:/root/vxlan.pcap .
wireshark vxlan.pcap
