0. CloudFormation Stack 배포
- 만약 CloudFormation 스택 배포를 Default 계정이 아닌 다른 AWS 계정(Profile)을 사용할 경우
--profile <profile Name>옵션을 추가 해줍시다.
# YAML 파일 다운로드
curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/K8S/eks-oneclick2.yaml
# CloudFormation 스택 배포
예시) aws cloudformation deploy --template-file eks-oneclick2.yaml --stack-name myeks --parameter-overrides KeyName=kp-gasida SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 MyIamUserAccessKeyID='~' MyIamUserSecretAccessKey='~' ClusterBaseName=myeks --region ap-northeast-2
# CloudFormation 스택 배포 완료 후 작업용 EC2 IP 출력
aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text
# 작업용 EC2 SSH 접속
ssh -i ~/.ssh/kp-gasida.pem ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text)
예시) ssh -i ejl-eks.pem ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text --profile ejl-personal) ✔ 14:40:08
The authenticity of host '3.38.115.229 (3.38.115.229)' can't be established.
ED25519 key fingerprint is SHA256:tQwPk327qdcRbXZ1vtHT4d1W09OhFyDZR/ULkgGPMLA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '3.38.115.229' (ED25519) to the list of known hosts.
, #_
~\_ ####_ Amazon Linux 2
~~ \_#####\
~~ \###| AL2 End of Life is 2025-06-30.
~~ \#/ ___
~~ V~' '->
~~~ / A newer version of Amazon Linux is available!
~~._. _/
_/ _/ Amazon Linux 2023, GA and supported until 2028-03-15.
_/m/' https://aws.amazon.com/linux/amazon-linux-2023/
7 package(s) needed for security, out of 12 available
Run "sudo yum update" to apply all updates.- 기본 설정 및 EFS 확인
- 이번 실습은 Storage와 관련된 내용이므로 이전 포스팅과는 다르게 AWS EFS 서비스가 프로비저닝 됩니다.
# EFS 서비스 ID 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# echo $EfsFsId
fs-024b43fbf2d8c3b93
# EFS 시스템에 마운트 설정
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# mount -t efs -o tls $EfsFsId:/ /mnt/myefs
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# df -hT --type nfs4
Filesystem Type Size Used Avail Use% Mounted on
127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /mnt/myefs
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# echo "efs file test" > /mnt/myefs/memo.txt
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat /mnt/myefs/memo.txt
efs file test
# 스토리지 클래스 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 23m
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# clear
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 27m
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get sc gp2 -o yaml | yh
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2024-03-21T05:28:41Z"
name: gp2
resourceVersion: "281"
uid: 19c4debb-c6bf-4135-8f69-0da412605537
parameters:
fsType: ext4
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get csinodes
NAME DRIVERS AGE
ip-192-168-1-166.ap-northeast-2.compute.internal 0 18m
ip-192-168-2-138.ap-northeast-2.compute.internal 0 18m
ip-192-168-3-184.ap-northeast-2.compute.internal 0 18m
# 노드의 정보 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE CAPACITYTYPE ZONE
ip-192-168-1-166.ap-northeast-2.compute.internal Ready <none> 19m v1.28.5-eks-5e0fdde t3.medium ON_DEMAND ap-northeast-2a
ip-192-168-2-138.ap-northeast-2.compute.internal Ready <none> 19m v1.28.5-eks-5e0fdde t3.medium ON_DEMAND ap-northeast-2b
ip-192-168-3-184.ap-northeast-2.compute.internal Ready <none> 19m v1.28.5-eks-5e0fdde t3.medium ON_DEMAND ap-northeast-2c
# 실습을 위한 노드 IP 변수 지정 (private)
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2b -o jsonpath={.items[0].status.addresses[0].address})
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# N3=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# echo "export N1=$N1" >> /etc/profile
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# echo "export N2=$N2" >> /etc/profile
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# echo "export N3=$N3" >> /etc/profile
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# echo $N1, $N2, $N3
192.168.1.166, 192.168.2.138, 192.168.3.184
# 노드 보안그룹 ID 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# NGSGID=$(aws ec2 describe-security-groups --filters Name=group-name,Values=*ng1* --query "SecurityGroups[*].[GroupId]" --output text)
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 authorize-security-group-ingress --group-id $NGSGID --protocol '-1' --cidr 192.168.1.100/32
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-07f45e0d4a0d62cc5",
"GroupId": "sg-0d1dcd192d8d6bcf6",
"GroupOwnerId": "236747833953",
"IsEgress": false,
"IpProtocol": "-1",
"FromPort": -1,
"ToPort": -1,
"CidrIpv4": "192.168.1.100/32"
}
]
}
# Worker 노드 SSH 접속 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# for node in $N1 $N2 $N3; do ssh ec2-user@$node hostname; done
yes
yes
yes- AWS LB/ExternalDNS, kube-ops-view Install
# LB Controller Install
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# helm repo add eks https://aws.github.io/eks-charts
"eks" has been added to your repositories
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "eks" chart repository
Update Complete. ⎈Happy Helming!⎈
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME \
> --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
NAME: aws-load-balancer-controller
LAST DEPLOYED: Thu Mar 21 15:02:28 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get pod -n kube-system | grep load
aws-load-balancer-controller-75c55f7bd-b9gxv 1/1 Running 0 58s
aws-load-balancer-controller-75c55f7bd-zpq4n 1/1 Running 0 58s
# ExternalDNS Install
# 도메인 등록
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# MyDomain=22joo.shop
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --output text)
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# echo $MyDomain, $MyDnzHostedZoneId
22joo.shop, /hostedzone/Z07798463AFECYTX1ODP4
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# sed -i "s/0.13.4/0.14.0/g" externaldns.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst < externaldns.yaml | kubectl apply -f -
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get pod -n kube-system | grep external
external-dns-7fd77dcbc-9gpll 1/1 Running 0 26s
# kube-ops-view 설치
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# helm repo add geek-cookbook https://geek-cookbook.github.io/charts/
"geek-cookbook" has been added to your repositories
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set env.TZ="Asia/Seoul" --namespace kube-system
NAME: kube-ops-view
LAST DEPLOYED: Thu Mar 21 15:28:31 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace kube-system -l "app.kubernetes.io/name=kube-ops-view,app.kubernetes.io/instance=kube-ops-view" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl port-forward $POD_NAME 8080:8080
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl patch svc -n kube-system kube-ops-view -p '{"spec":{"type":"LoadBalancer"}}'
service/kube-ops-view patched
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl annotate service kube-ops-view -n kube-system "external-dns.alpha.kubernetes.io/hostname=kubeopsview.$MyDomain"
service/kube-ops-view annotated
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# echo -e "Kube Ops View URL = http://kubeopsview.$MyDomain:8080/#scale=1.5"
Kube Ops View URL = http://kubeopsview.22joo.shop:8080/#scale=1.5
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
aws-load-balancer-webhook-service ClusterIP 10.100.128.136 <none> 443/TCP 28m
kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP,9153/TCP 60m
kube-ops-view LoadBalancer 10.100.13.209 a45ef58292b404fe7a8f305185be4be8-1700233.ap-northeast-2.elb.amazonaws.com 8080:31678/TCP 2m3s- A Record 로 Loadbalancer type으로 설정한 kube-ops-view 서비스를 먹고있음을 확인 할 수 있습니다.
- 도메인으로 접속
- 설치 정보 확인
# 이미지 정보 확인
kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c
# eksctl 설치/업데이트 addon 확인
eksctl get addon --cluster $CLUSTER_NAME
# IRSA 확인
eksctl get iamserviceaccount --cluster $CLUSTER_NAME1. Storage의 이해
- 실행되고 있는 Pod 가 정지되면 해당 Pod 내의 데이터는 모두 삭제됩니다. 즉 일시적인 파일시스템을 사용하고 있습니다.
- 따라서, 데이터의 보존이 필요하기 때문에 k8s 에서는 PV, PVC 의 개념을 도입합니다.
- 로컬 Volume (Host path)에 PV(Persist Volume)을 정의 한뒤, PVC(Persist Volume Claim)을 통해 Bound 시켜 사용합니다.
- 또한, 파드가 생성될 때 자동으로 볼륨을 마운트하여 파드에 연결하는 기능을 동적 프로비저닝(Dynamic Provisioning)이라고 합니다.
- 만약, PV 의 사용이 끝났을 때 해당 볼륨을 어떻게 할 것인지 별도로 설정할 수 있으며 이것을 k8s 에서 Reclaim Policy 라고 칭합니다. 아래는 Reclaim Policy 방식입니다.
- Retain (보존)
- Delete (삭제, EBS 볼륨도 삭제되게 됩니다.)
CSI Driver
- 간단히 요약하자면, k8s 에 내장된 provisioner를 모두 삭제하고, 별도의 Controller Pod 를 통해 Dynamic Provisioning 을 사용할 수 있게 만든 Driver 입니다.
-
오른쪽 StatefulSet 또는 Deployment로 배포된 controller Pod이 AWS API를 사용하여 실제 EBS volume을 생성하는 역할을 합니다.
왼쪽 DaemonSet으로 배포된 node Pod은 AWS API를 사용하여 Kubernetes node (EC2 instance)에 EBS volume을 attach 해줍니다. -
그러면 기본 컨테이너 환경에서는 임시 파일 시스템이 어떻게 사용되는지 확인해봅시다.
- Pod 를 삭제했다가 다시 띄우는 시나리오를 통해 알아봅시다.
# date 명령어로 현재 시간을 10초 간격으로 /home/pod-out.txt 파일에 저장하는 Pod 배포
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/3/date-busybox-pod.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat date-busybox-pod.yaml | yh
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
terminationGracePeriodSeconds: 3
containers:
- name: busybox
image: busybox
command:
- "/bin/sh"
- "-c"
- "while true; do date >> /home/pod-out.txt; cd /home; sync; sync; sleep 10; done"
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f date-busybox-pod.yaml
pod/busybox created
# file 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 27s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl exec busybox -- tail -f /home/pod-out.txt
Thu Mar 21 07:28:26 UTC 2024
Thu Mar 21 07:28:36 UTC 2024
Thu Mar 21 07:28:46 UTC 2024
# 파드 삭제 후 다시 생성 후 파일 정보 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pod busybox
pod "busybox" deleted
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f date-busybox-pod.yaml
pod/busybox created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl exec busybox -- tail -f /home/pod-out.txt
Thu Mar 21 07:29:35 UTC 2024-
확인 결과
Thu Mar 21 07:28:26 UTC 2024 Thu Mar 21 07:28:36 UTC 2024 Thu Mar 21 07:28:46 UTC 2024에 대한 데이터들은 보이지 않습니다. 임시적이란 것입니다. -
그러면, 호스트 Path 를 사용하는 PV/PVC : local-path-provisioner 스트리지 클래스 배포해보겠습니다.
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat local-path-storage.yaml
apiVersion: v1
kind: Namespace
metadata:
name: local-path-storage
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: local-path-provisioner-service-account
namespace: local-path-storage
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: local-path-provisioner-role
namespace: local-path-storage
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: local-path-provisioner-role
rules:
- apiGroups: [""]
resources: ["nodes", "persistentvolumeclaims", "configmaps", "pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: local-path-provisioner-bind
namespace: local-path-storage
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: local-path-provisioner-role
subjects:
- kind: ServiceAccount
name: local-path-provisioner-service-account
namespace: local-path-storage
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: local-path-provisioner-bind
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: local-path-provisioner-role
subjects:
- kind: ServiceAccount
name: local-path-provisioner-service-account
namespace: local-path-storage
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: local-path-provisioner
namespace: local-path-storage
spec:
replicas: 1
selector:
matchLabels:
app: local-path-provisioner
template:
metadata:
labels:
app: local-path-provisioner
spec:
serviceAccountName: local-path-provisioner-service-account
containers:
- name: local-path-provisioner
image: rancher/local-path-provisioner:master-head
imagePullPolicy: IfNotPresent
command:
- local-path-provisioner
- --debug
- start
- --config
- /etc/config/config.json
volumeMounts:
- name: config-volume
mountPath: /etc/config/
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumes:
- name: config-volume
configMap:
name: local-path-config
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-path
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
kind: ConfigMap
apiVersion: v1
metadata:
name: local-path-config
namespace: local-path-storage
data:
config.json: |-
{
"nodePathMap":[
{
"node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
"paths":["/opt/local-path-provisioner"]
}
]
}
setup: |-
#!/bin/sh
set -eu
mkdir -m 0777 -p "$VOL_DIR"
teardown: |-
#!/bin/sh
set -eu
rm -rf "$VOL_DIR"
helperPod.yaml: |-
apiVersion: v1
kind: Pod
metadata:
name: helper-pod
spec:
priorityClassName: system-node-critical
tolerations:
- key: node.kubernetes.io/disk-pressure
operator: Exists
effect: NoSchedule
containers:
- name: helper-pod
image: busybox
imagePullPolicy: IfNotPresent
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f local-path-storage.yaml
namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
role.rbac.authorization.k8s.io/local-path-provisioner-role created
clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
rolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get-all -n local-path-storage
NAME NAMESPACE AGE
configmap/kube-root-ca.crt local-path-storage 5s
configmap/local-path-config local-path-storage 5s
pod/local-path-provisioner-5d854bc5c4-bj55s local-path-storage 5s
serviceaccount/default local-path-storage 5s
serviceaccount/local-path-provisioner-service-account local-path-storage 5s
deployment.apps/local-path-provisioner local-path-storage 5s
replicaset.apps/local-path-provisioner-5d854bc5c4 local-path-storage 5s
rolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind local-path-storage 5s
role.rbac.authorization.k8s.io/local-path-provisioner-role local-path-storage 5s
# 스토리지 클래스 생성 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 125m
local-path rancher.io/local-path Delete WaitForFirstConsumer false 34s- PV/PVC 를 사용하는 파드를 생성해보겠습니다.
# PVC 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/3/localpath1.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat localpath1.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: localpath-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
# 현재 Pending 상태 입니다.
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f localpath1.yaml
persistentvolumeclaim/localpath-claim created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
localpath-claim Pending local-path 7s
# Pod 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/3/localpath2.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat localpath2.yaml | yh
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
terminationGracePeriodSeconds: 3
containers:
- name: app
image: centos
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
volumeMounts:
- name: persistent-storage
mountPath: /data
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: localpath-claim
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f localpath2.yaml
pod/app created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
localpath-claim Bound pvc-47595f15-4808-4084-98e8-067c9d15284e 1Gi RWO local-path 86s- Data 보존 확인
- Pod 삭제 이후 어떻게 되는지 변화 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl exec -it app -- tail -f /data/out.txt
Thu Mar 21 07:37:41 UTC 2024
Thu Mar 21 07:37:46 UTC 2024
Thu Mar 21 07:37:51 UTC 2024
Thu Mar 21 07:37:56 UTC 2024
Thu Mar 21 07:38:01 UTC 2024
Thu Mar 21 07:38:06 UTC 2024
Thu Mar 21 07:38:11 UTC 2024
Thu Mar 21 07:38:16 UTC 2024
Thu Mar 21 07:38:21 UTC 2024
Thu Mar 21 07:38:26 UTC 2024
Thu Mar 21 07:38:31 UTC 2024
# 각 Node 에 볼륨이 연결되어 저장되어 있는 파일 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# for node in $N1 $N2 $N3; do ssh ec2-user@$node tree /opt/local-path-provisioner; done
/opt/local-path-provisioner [error opening dir]
0 directories, 0 files
/opt/local-path-provisioner [error opening dir]
0 directories, 0 files
/opt/local-path-provisioner
└── pvc-47595f15-4808-4084-98e8-067c9d15284e_default_localpath-claim
└── out.txt
1 directory, 1 file
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pod app
pod "app" deleted
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-47595f15-4808-4084-98e8-067c9d15284e 1Gi RWO Delete Bound default/localpath-claim local-path 3m56s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/localpath-claim Bound pvc-47595f15-4808-4084-98e8-067c9d15284e 1Gi RWO local-path 4m55s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# for node in $N1 $N2 $N3; do ssh ec2-user@$node tree /opt/local-path-provisioner; done
/opt/local-path-provisioner [error opening dir]
0 directories, 0 files
/opt/local-path-provisioner [error opening dir]
0 directories, 0 files
/opt/local-path-provisioner
└── pvc-47595f15-4808-4084-98e8-067c9d15284e_default_localpath-claim
└── out.txt
1 directory, 1 file
# Pod 다시 실행 이전의 time travel 도 남아 있는 것을 확인할 수 있습니다.
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl exec -it app -- head /data/out.txt
Thu Mar 21 07:37:06 UTC 2024
Thu Mar 21 07:37:11 UTC 2024
Thu Mar 21 07:37:16 UTC 2024
Thu Mar 21 07:37:21 UTC 2024
Thu Mar 21 07:37:26 UTC 2024
Thu Mar 21 07:37:31 UTC 2024
Thu Mar 21 07:37:36 UTC 2024
Thu Mar 21 07:37:41 UTC 2024
Thu Mar 21 07:37:46 UTC 2024
Thu Mar 21 07:37:51 UTC 2024
# 삭제
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pod app
pod "app" deleted
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-47595f15-4808-4084-98e8-067c9d15284e 1Gi RWO Delete Bound default/localpath-claim local-path 7m49s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/localpath-claim Bound pvc-47595f15-4808-4084-98e8-067c9d15284e 1Gi RWO local-path 8m48s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pvc localpath-claim
persistentvolumeclaim "localpath-claim" deleted
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-47595f15-4808-4084-98e8-067c9d15284e 1Gi RWO Delete Released default/localpath-claim local-path 7m51s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# for node in $N1 $N2 $N3; do ssh ec2-user@$node tree /opt/local-path-provisioner; done
/opt/local-path-provisioner [error opening dir]
0 directories, 0 files
/opt/local-path-provisioner [error opening dir]
0 directories, 0 files
/opt/local-path-provisioner
0 directories, 0 files2. AWS EBS Controller
-
EBS CSI 드라이버는 동작이 단순합니다.
-
k8s API 서버가 csi-controller 를 지켜보고 있다가. EBS 볼륨 요청이 들어오면 AWS API (볼륨 생성 API)를 사용하여 노드에 볼륨을 탑재시킵니다.
-
다음으로 k8s API 서버의 Mount 요청은 kubelet 이 받아 csi-node에게 전달하며 결과적으로 Mount 요청에 따라 Node 와 Pod의 EBS 볼륨 마운트가 형성됩니다.
- EBS Driver 는 EKS 의 Add-on 기능으로 분리되어 있으므로 이것을 설치해보겠습니다.
# aws-ebs-csi-driver 전체 버전 정보와 기본 설치 버전(True) 정보 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws eks describe-addon-versions \
> --addon-name aws-ebs-csi-driver \
> --kubernetes-version 1.28 \
> --query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" \
> --output text
v1.28.0-eksbuild.1
True # 가능
v1.27.0-eksbuild.1
False
v1.26.1-eksbuild.1
False
v1.26.0-eksbuild.1
False
v1.25.0-eksbuild.1
False
v1.24.1-eksbuild.1
False
v1.24.0-eksbuild.1
False
# ISRA 설정 : AWS관리형 정책 AmazonEBSCSIDriverPolicy 사용
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl create iamserviceaccount \
> --name ebs-csi-controller-sa \
> --namespace kube-system \
> --cluster ${CLUSTER_NAME} \
> --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
> --approve \
> --role-only \
> --role-name AmazonEKS_EBS_CSI_DriverRole
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl get iamserviceaccount --cluster myeks
NAMESPACE NAME ROLE ARN
kube-system aws-load-balancer-controller arn:aws:iam::236747833953:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-zr2D0yV8p1DD
kube-system ebs-csi-controller-sa arn:aws:iam::236747833953:role/AmazonEKS_EBS_CSI_DriverRole
# Add on 기능 추가
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl create addon --name aws-ebs-csi-driver --cluster ${CLUSTER_NAME} --service-account-role-arn arn:aws:iam::${ACCOUNT_ID}:role/AmazonEKS_EBS_CSI_DriverRole --force
2024-03-21 17:03:24 [ℹ] Kubernetes version "1.28" in use by cluster "myeks"
2024-03-21 17:03:24 [ℹ] using provided ServiceAccountRoleARN "arn:aws:iam::236747833953:role/AmazonEKS_EBS_CSI_DriverRole"
2024-03-21 17:03:24 [ℹ] creating addon
# 애드온 추가 됐는지 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl get addon --cluster ${CLUSTER_NAME}
2024-03-21 17:03:55 [ℹ] Kubernetes version "1.28" in use by cluster "myeks"
2024-03-21 17:03:55 [ℹ] getting all addons
2024-03-21 17:03:57 [ℹ] to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>`
NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES
aws-ebs-csi-driver v1.28.0-eksbuild.1 CREATING 0 arn:aws:iam::236747833953:role/AmazonEKS_EBS_CSI_DriverRole
coredns v1.10.1-eksbuild.7 ACTIVE 0
kube-proxy v1.28.6-eksbuild.2 ACTIVE 0
vpc-cni v1.17.1-eksbuild.1 ACTIVE 0 arn:aws:iam::236747833953:role/eksctl-myeks-addon-vpc-cni-Role1-kjRgjssq7Ee4 enableNetworkPolicy: "true"
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get deploy,ds -l=app.kubernetes.io/name=aws-ebs-csi-driver -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ebs-csi-controller 2/2 2 2 30s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/ebs-csi-node 3 3 3 3 3 kubernetes.io/os=linux 30s
daemonset.apps/ebs-csi-node-windows 0 0 0 0 0 kubernetes.io/os=windows 30s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pod -n kube-system -l 'app in (ebs-csi-controller,ebs-csi-node)'
NAME READY STATUS RESTARTS AGE
ebs-csi-controller-765cf7cf9-8pk4z 6/6 Running 0 31s
ebs-csi-controller-765cf7cf9-9ljpn 6/6 Running 0 31s
ebs-csi-node-45t6q 3/3 Running 0 31s
ebs-csi-node-6zk7p 3/3 Running 0 31s
ebs-csi-node-tdbqz 3/3 Running 0 31s
- gp3 스토리지 클래스 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 157m
local-path rancher.io/local-path Delete WaitForFirstConsumer false 32m
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat <<EOT > gp3-sc.yaml
> kind: StorageClass
> apiVersion: storage.k8s.io/v1
> metadata:
> name: gp3
> allowVolumeExpansion: true
> provisioner: ebs.csi.aws.com
> volumeBindingMode: WaitForFirstConsumer
> parameters:
> type: gp3
> #iops: "5000"
> #throughput: "250"
> allowAutoIOPSPerGBIncrease: 'true'
> encrypted: 'true'
> fsType: xfs # 기본값이 ext4
> EOT
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f gp3-sc.yaml
storageclass.storage.k8s.io/gp3 created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 158m
gp3 ebs.csi.aws.com Delete WaitForFirstConsumer true 2s
local-path rancher.io/local-path Delete WaitForFirstConsumer false 33m
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl describe sc gp3 | grep Parameters
Parameters: allowAutoIOPSPerGBIncrease=true,encrypted=true,fsType=xfs,type=gp3- Worker Node 의 EBS 볼륨 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --output table
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| DescribeVolumes |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|| Volumes ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
|| AvailabilityZone | CreateTime | Encrypted | Iops | MultiAttachEnabled | Size | SnapshotId | State | Throughput | VolumeId | VolumeType ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
|| ap-northeast-2a | 2024-03-21T05:36:54.520000+00:00 | False | 3000 | False | 30 | snap-0143f910a7ab68752 | in-use | 125 | vol-0ad80fc82498caf7a | gp3 ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
||| Attachments |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| AttachTime | DeleteOnTermination | Device | InstanceId | State | VolumeId |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| 2024-03-21T05:36:54+00:00 | True | /dev/xvda | i-099bf63e828715bb4 | attached | vol-0ad80fc82498caf7a |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| Tags |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
||| Key | Value |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
||| eks:cluster-name | myeks |||
||| Name | myeks-ng1-Node |||
||| eks:nodegroup-name | ng1 |||
||| alpha.eksctl.io/nodegroup-type | managed |||
||| alpha.eksctl.io/nodegroup-name | ng1 |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
|| Volumes ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
|| AvailabilityZone | CreateTime | Encrypted | Iops | MultiAttachEnabled | Size | SnapshotId | State | Throughput | VolumeId | VolumeType ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
|| ap-northeast-2b | 2024-03-21T05:36:54.510000+00:00 | False | 3000 | False | 30 | snap-0143f910a7ab68752 | in-use | 125 | vol-04d0478c63af9f20f | gp3 ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
||| Attachments |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| AttachTime | DeleteOnTermination | Device | InstanceId | State | VolumeId |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| 2024-03-21T05:36:54+00:00 | True | /dev/xvda | i-07d8c1dd603a90f38 | attached | vol-04d0478c63af9f20f |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| Tags |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
||| Key | Value |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
||| Name | myeks-ng1-Node |||
||| eks:nodegroup-name | ng1 |||
||| alpha.eksctl.io/nodegroup-type | managed |||
||| eks:cluster-name | myeks |||
||| alpha.eksctl.io/nodegroup-name | ng1 |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
|| Volumes ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
|| AvailabilityZone | CreateTime | Encrypted | Iops | MultiAttachEnabled | Size | SnapshotId | State | Throughput | VolumeId | VolumeType ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
|| ap-northeast-2c | 2024-03-21T05:36:54.597000+00:00 | False | 3000 | False | 30 | snap-0143f910a7ab68752 | in-use | 125 | vol-0c18634befa65b660 | gp3 ||
|+------------------+-----------------------------------+------------+-------+---------------------+-------+-------------------------+---------+-------------+-------------------------+--------------+|
||| Attachments |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| AttachTime | DeleteOnTermination | Device | InstanceId | State | VolumeId |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| 2024-03-21T05:36:54+00:00 | True | /dev/xvda | i-0be872382bf8d6c78 | attached | vol-0c18634befa65b660 |||
||+-------------------------------------------+-----------------------------------+-------------------+-----------------------------------+------------------+---------------------------------------+||
||| Tags |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
||| Key | Value |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
||| alpha.eksctl.io/nodegroup-name | ng1 |||
||| eks:cluster-name | myeks |||
||| eks:nodegroup-name | ng1 |||
||| Name | myeks-ng1-Node |||
||| alpha.eksctl.io/nodegroup-type | managed |||
||+------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------+||
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Volumes[*].Attachments" | jq
[
[
{
"AttachTime": "2024-03-21T05:36:54+00:00",
"Device": "/dev/xvda",
"InstanceId": "i-099bf63e828715bb4",
"State": "attached",
"VolumeId": "vol-0ad80fc82498caf7a",
"DeleteOnTermination": true
}
],
[
{
"AttachTime": "2024-03-21T05:36:54+00:00",
"Device": "/dev/xvda",
"InstanceId": "i-07d8c1dd603a90f38",
"State": "attached",
"VolumeId": "vol-04d0478c63af9f20f",
"DeleteOnTermination": true
}
],
[
{
"AttachTime": "2024-03-21T05:36:54+00:00",
"Device": "/dev/xvda",
"InstanceId": "i-0be872382bf8d6c78",
"State": "attached",
"VolumeId": "vol-0c18634befa65b660",
"DeleteOnTermination": true
}
]
]
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Volumes[*].{ID:VolumeId,Tag:Tags}" | jq
[
{
"ID": "vol-0ad80fc82498caf7a",
"Tag": [
{
"Key": "eks:cluster-name",
"Value": "myeks"
},
{
"Key": "Name",
"Value": "myeks-ng1-Node"
},
{
"Key": "eks:nodegroup-name",
"Value": "ng1"
},
{
"Key": "alpha.eksctl.io/nodegroup-type",
"Value": "managed"
},
{
"Key": "alpha.eksctl.io/nodegroup-name",
"Value": "ng1"
}
]
},
{
"ID": "vol-04d0478c63af9f20f",
"Tag": [
{
"Key": "Name",
"Value": "myeks-ng1-Node"
},
{
"Key": "eks:nodegroup-name",
"Value": "ng1"
},
{
"Key": "alpha.eksctl.io/nodegroup-type",
"Value": "managed"
},
{
"Key": "eks:cluster-name",
"Value": "myeks"
},
{
"Key": "alpha.eksctl.io/nodegroup-name",
"Value": "ng1"
}
]
},
{
"ID": "vol-0c18634befa65b660",
"Tag": [
{
"Key": "alpha.eksctl.io/nodegroup-name",
"Value": "ng1"
},
{
"Key": "eks:cluster-name",
"Value": "myeks"
},
{
"Key": "eks:nodegroup-name",
"Value": "ng1"
},
{
"Key": "Name",
"Value": "myeks-ng1-Node"
},
{
"Key": "alpha.eksctl.io/nodegroup-type",
"Value": "managed"
}
]
}
]
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Volumes[].[VolumeId, VolumeType, Attachments[].[InstanceId, State][]][]" | jq
[
"vol-0ad80fc82498caf7a",
"gp3",
[
"i-099bf63e828715bb4",
"attached"
],
"vol-04d0478c63af9f20f",
"gp3",
[
"i-07d8c1dd603a90f38",
"attached"
],
"vol-0c18634befa65b660",
"gp3",
[
"i-0be872382bf8d6c78",
"attached"
]
]
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Volumes[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}" | jq
[
{
"VolumeId": "vol-0ad80fc82498caf7a",
"VolumeType": "gp3",
"InstanceId": "i-099bf63e828715bb4",
"State": "attached"
},
{
"VolumeId": "vol-04d0478c63af9f20f",
"VolumeType": "gp3",
"InstanceId": "i-07d8c1dd603a90f38",
"State": "attached"
},
{
"VolumeId": "vol-0c18634befa65b660",
"VolumeType": "gp3",
"InstanceId": "i-0be872382bf8d6c78",
"State": "attached"
}
]- Worker Node 에서 Pod 에 추가한 EBS 볼륨 확인
- 아직 Volume Claim 을 걸지 않았기 때문에 요소가 없습니다.
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --filters Name=tag:ebs.csi.aws.com/cluster,Values=true --output table
-----------------
|DescribeVolumes|
+---------------+
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --filters Name=tag:ebs.csi.aws.com/cluster,Values=true --query "Volumes[*].{ID:VolumeId,Tag:Tags}" | jq
[]
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --filters Name=tag:ebs.csi.aws.com/cluster,Values=true --query "Volumes[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}" | jq
[]
# 워커노드에서 파드에 추가한 EBS 볼륨 모니터링 - 아직 안뜸
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# while true; do aws ec2 describe-volumes --filters Name=tag:ebs.csi.aws.com/cluster,Values=true --query "Volumes[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}" --output text; date; sleep 1; done
Thu Mar 21 17:14:05 KST 2024
Thu Mar 21 17:14:07 KST 2024
Thu Mar 21 17:14:09 KST 2024
Thu Mar 21 17:14:11 KST 2024
Thu Mar 21 17:14:13 KST 2024
Thu Mar 21 17:14:15 KST 2024- PVC 및 Pod 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat <<EOT > awsebs-pvc.yaml
> apiVersion: v1
> kind: PersistentVolumeClaim
> metadata:
> name: ebs-claim
> spec:
> accessModes:
> - ReadWriteOnce
> resources:
> requests:
> storage: 4Gi
> storageClassName: gp3
> EOT
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f awsebs-pvc.yaml
persistentvolumeclaim/ebs-claim created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pvc,pv
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/ebs-claim Pending gp3 4s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat <<EOT > awsebs-pod.yaml
> apiVersion: v1
> kind: Pod
> metadata:
> name: app
> spec:
> terminationGracePeriodSeconds: 3
> containers:
> - name: app
> image: centos
> command: ["/bin/sh"]
> args: ["-c", "while true; do echo \$(date -u) >> /data/out.txt; sleep 5; done"]
> volumeMounts:
> - name: persistent-storage
> mountPath: /data
> volumes:
> - name: persistent-storage
> persistentVolumeClaim:
> claimName: ebs-claim
> EOT
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f awsebs-pod.yaml
pod/app created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pvc,pv,pod
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/ebs-claim Bound pvc-88219b6e-b07a-4e4c-9c44-3beb085ac338 4Gi RWO gp3 43s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-88219b6e-b07a-4e4c-9c44-3beb085ac338 4Gi RWO Delete Bound default/ebs-claim gp3 3s
NAME READY STATUS RESTARTS AGE
pod/app 0/1 ContainerCreating 0 7s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get VolumeAttachment
NAME ATTACHER PV NODE ATTACHED AGE
csi-601ea645c59ddf481c1194dc41417b87955f266107106b73a87c1755a59c4d16 ebs.csi.aws.com pvc-88219b6e-b07a-4e4c-9c44-3beb085ac338 ip-192-168-3-184.ap-northeast-2.compute.internal true 23s
# 모니터링
Thu Mar 21 17:15:10 KST 2024
Thu Mar 21 17:15:12 KST 2024
Thu Mar 21 17:15:14 KST 2024
Thu Mar 21 17:15:17 KST 2024
Thu Mar 21 17:15:19 KST 2024
Thu Mar 21 17:15:21 KST 2024
None None vol-0dfadc62a6f23d034 gp3
Thu Mar 21 17:15:24 KST 2024
None None vol-0dfadc62a6f23d034 gp3
Thu Mar 21 17:15:26 KST 2024
i-0be872382bf8d6c78 attaching vol-0dfadc62a6f23d034 gp3 ## 탑재되기 시작합니다.
Thu Mar 21 17:15:28 KST 2024
i-0be872382bf8d6c78 attached vol-0dfadc62a6f23d034 gp3
Thu Mar 21 17:15:30 KST 2024
i-0be872382bf8d6c78 attached vol-0dfadc62a6f23d034 gp3
Thu Mar 21 17:15:32 KST 2024
i-0be872382bf8d6c78 attached vol-0dfadc62a6f23d034 gp3
Thu Mar 21 17:15:35 KST 2024
i-0be872382bf8d6c78 attached vol-0dfadc62a6f23d034 gp3
Thu Mar 21 17:15:37 KST 2024
i-0be872382bf8d6c78 attached vol-0dfadc62a6f23d034 gp3
# 추가된 EBS 볼륨 상세 정보 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-volumes --volume-ids $(kubectl get pv -o jsonpath="{.items[0].spec.csi.volumeHandle}") | jq
{
"Volumes": [
{
"Attachments": [
{
"AttachTime": "2024-03-21T08:15:27+00:00",
"Device": "/dev/xvdaa",
"InstanceId": "i-0be872382bf8d6c78",
"State": "attached",
"VolumeId": "vol-0dfadc62a6f23d034",
"DeleteOnTermination": false
}
],
"AvailabilityZone": "ap-northeast-2c",
"CreateTime": "2024-03-21T08:15:22.720000+00:00",
"Encrypted": true,
"KmsKeyId": "arn:aws:kms:ap-northeast-2:236747833953:key/817c75a3-9bb7-42d2-a2c8-4a4ec6dccaa1",
"Size": 4,
"SnapshotId": "",
"State": "in-use",
"VolumeId": "vol-0dfadc62a6f23d034",
"Iops": 3000,
"Tags": [
{
"Key": "KubernetesCluster",
"Value": "myeks"
},
{
"Key": "CSIVolumeName",
"Value": "pvc-88219b6e-b07a-4e4c-9c44-3beb085ac338"
},
{
"Key": "ebs.csi.aws.com/cluster",
"Value": "true"
},
{
"Key": "kubernetes.io/created-for/pvc/name",
"Value": "ebs-claim"
},
{
"Key": "Name",
"Value": "myeks-dynamic-pvc-88219b6e-b07a-4e4c-9c44-3beb085ac338"
},
{
"Key": "kubernetes.io/cluster/myeks",
"Value": "owned"
},
{
"Key": "kubernetes.io/created-for/pvc/namespace",
"Value": "default"
},
{
"Key": "kubernetes.io/created-for/pv/name",
"Value": "pvc-88219b6e-b07a-4e4c-9c44-3beb085ac338"
}
],
"VolumeType": "gp3",
"MultiAttachEnabled": false,
"Throughput": 125
}
]
}
- PV 에서 nodeAffinity 는 무엇일까
- 볼륨에 접근할 수 있는 노드를 제한하는 제약 조건을 정의하는 것입니다.
- PV를 사용하는 파드는 Node affinity에 의해 선택된 노드로만 스케줄링됩니다.
- 즉, 아래의 경우는 ap-northeast-2c 에 프로비저닝 되어 있는 Node 만이 볼륨에 접근할 수 있는 것입니다.
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pv -o yaml | yh
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
volume.kubernetes.io/provisioner-deletion-secret-name: ""
volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
creationTimestamp: "2024-03-21T08:15:26Z"
finalizers:
- kubernetes.io/pv-protection
- external-attacher/ebs-csi-aws-com
name: pvc-88219b6e-b07a-4e4c-9c44-3beb085ac338
resourceVersion: "40040"
uid: b83c3fdc-2750-4435-a6ec-364ebfdf2f97
spec:
.
.
.## 아래 부분
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: topology.ebs.csi.aws.com/zone
operator: In
values:
- ap-northeast-2c
persistentVolumeReclaimPolicy: Delete
storageClassName: gp3
volumeMode: Filesystem
status:
phase: Bound
kind: List
metadata:
resourceVersion: ""- 아래는 ap-northeast-2c 의 Node 의 EBS 탑재 상태입니다.
- 다른 AZ의 노드에서 확인헤봅시다.
- 아래는 AZ : b 의 노트의 EBS 볼륨 정보입니다. nodeAffinity 에 의해 스케줄링 된 것을 볼 수 있습니다.
- 방금 탑재한 EBS 볼륨의 사이즈를 증가시켜보겠습니다. 4GiB - 10 GiB
- 당연히, 현재 상태에서 사이즈를 줄이는 것은 불가능 합니다.
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pvc ebs-claim -o jsonpath={.status.capacity.storage} ; echo
4Gi
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl patch pvc ebs-claim -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}'
persistentvolumeclaim/ebs-claim patched
- Resizing 하는데 시간이 소요됩니다.
- 삭제
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pod app && kubectl delete pvc ebs-claim
pod "app" deleted
persistentvolumeclaim "ebs-claim" deleted3. AWS Volume SnapShots Controller
- AWS 의 볼륨 스냅샷 컨트롤러를 설치를 통해 스냅샷으로 Pod 를 복원해보겠습니다.
- Controller 설치
# Snapshot CRD 설치
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml,snapshot.storage.k8s.io_volumesnapshotclasses.yaml,snapshot.storage.k8s.io_volumesnapshotcontents.yaml
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get crd | grep snapshot
volumesnapshotclasses.snapshot.storage.k8s.io 2024-03-21T08:37:39Z
volumesnapshotcontents.snapshot.storage.k8s.io 2024-03-21T08:37:39Z
volumesnapshots.snapshot.storage.k8s.io 2024-03-21T08:37:39Z
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl api-resources | grep snapshot
volumesnapshotclasses vsclass,vsclasses snapshot.storage.k8s.io/v1 false VolumeSnapshotClass
volumesnapshotcontents vsc,vscs snapshot.storage.k8s.io/v1 false VolumeSnapshotContent
volumesnapshots vs snapshot.storage.k8s.io/v1 true VolumeSnapshot
# Snapshot controller 설치
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f rbac-snapshot-controller.yaml,setup-snapshot-controller.yaml
serviceaccount/snapshot-controller created
clusterrole.rbac.authorization.k8s.io/snapshot-controller-runner created
clusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role created
role.rbac.authorization.k8s.io/snapshot-controller-leaderelection created
rolebinding.rbac.authorization.k8s.io/snapshot-controller-leaderelection created
deployment.apps/snapshot-controller created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get deploy -n kube-system snapshot-controller
NAME READY UP-TO-DATE AVAILABLE AGE
snapshot-controller 2/2 2 0 11s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pod -n kube-system | grep snapshot
snapshot-controller-749cb47b55-jsrpp 1/1 Running 0 55s
snapshot-controller-749cb47b55-zkhd6 1/1 Running 0 55s
# Snapshotclass 설치
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/examples/kubernetes/snapshot/manifests/classes/snapshotclass.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat snapshotclass.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-aws-vsc
driver: ebs.csi.aws.com
deletionPolicy: Delete
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f snapshotclass.yaml
volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get vsclass
NAME DRIVER DELETIONPOLICY AGE
csi-aws-vsc ebs.csi.aws.com Delete 8s- PVC 및 Pod 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat awsebs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
storageClassName: gp3
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f awsebs-pvc.yaml
persistentvolumeclaim/ebs-claim created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f awsebs-pod.yaml
pod/app created
# 파일 내용 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl exec app -- tail -f /data/out.txt
Thu Mar 21 08:43:44 UTC 2024
Thu Mar 21 08:43:49 UTC 2024
Thu Mar 21 08:43:54 UTC 2024
Thu Mar 21 08:43:59 UTC 2024
# Volume Snapshot 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# clear
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/3/ebs-volume-snapshot.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat ebs-volume-snapshot.yaml | yh
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: ebs-volume-snapshot
spec:
volumeSnapshotClassName: csi-aws-vsc
source:
persistentVolumeClaimName: ebs-claim
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f ebs-volume-snapshot.yaml
volumesnapshot.snapshot.storage.k8s.io/ebs-volume-snapshot created
# 스냅샷 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get volumesnapshot
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
ebs-volume-snapshot true ebs-claim 4Gi csi-aws-vsc snapcontent-e2dc48b5-db52-4d9f-972b-7dd2bf42f814 19s 20s
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get volumesnapshot ebs-volume-snapshot -o jsonpath={.status.boundVolumeSnapshotContentName} ; echo
snapcontent-e2dc48b5-db52-4d9f-972b-7dd2bf42f814
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl describe volumesnapshot.snapshot.storage.k8s.io ebs-volume-snapshot
Name: ebs-volume-snapshot
Namespace: default
Labels: <none>
Annotations: <none>
API Version: snapshot.storage.k8s.io/v1
Kind: VolumeSnapshot
Metadata:
Creation Timestamp: 2024-03-21T08:44:38Z
Finalizers:
snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
Generation: 1
Resource Version: 49020
UID: e2dc48b5-db52-4d9f-972b-7dd2bf42f814
Spec:
Source:
Persistent Volume Claim Name: ebs-claim
Volume Snapshot Class Name: csi-aws-vsc
Status:
Bound Volume Snapshot Content Name: snapcontent-e2dc48b5-db52-4d9f-972b-7dd2bf42f814
Creation Time: 2024-03-21T08:44:39Z
Ready To Use: true
Restore Size: 4Gi
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreatingSnapshot 23s snapshot-controller Waiting for a snapshot default/ebs-volume-snapshot to be created by the CSI driver.
Normal SnapshotCreated 22s snapshot-controller Snapshot default/ebs-volume-snapshot was successfully created by the CSI driver.
Normal SnapshotReady 12s snapshot-controller Snapshot default/ebs-volume-snapshot is ready to use.
# VS ID 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get volumesnapshotcontents -o jsonpath='{.items[*].status.snapshotHandle}' ; echo
snap-066b8006c377cf428
# AWS EBS 스냅샷 확인 - snapshot ID 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws ec2 describe-snapshots --owner-ids self --query 'Snapshots[]' --output table
-----------------------------------------------------------------------------------------------------
| DescribeSnapshots |
+-------------+-------------------------------------------------------------------------------------+
| Description| Created by AWS EBS CSI driver for volume vol-099cac72524c097fd |
| Encrypted | True |
| KmsKeyId | arn:aws:kms:ap-northeast-2:236747833953:key/817c75a3-9bb7-42d2-a2c8-4a4ec6dccaa1 |
| OwnerId | 236747833953 |
| Progress | 100% |
| SnapshotId | snap-066b8006c377cf428 |
| StartTime | 2024-03-21T08:44:39.180000+00:00 |
| State | completed |
| StorageTier| standard |
| VolumeId | vol-099cac72524c097fd |
| VolumeSize | 4 |
+-------------+-------------------------------------------------------------------------------------+
|| Tags ||
|+-------------------------------+-----------------------------------------------------------------+|
|| Key | Value ||
|+-------------------------------+-----------------------------------------------------------------+|
|| ebs.csi.aws.com/cluster | true ||
|| kubernetes.io/cluster/myeks | owned ||
|| Name | myeks-dynamic-snapshot-e2dc48b5-db52-4d9f-972b-7dd2bf42f814 ||
|| CSIVolumeSnapshotName | snapshot-e2dc48b5-db52-4d9f-972b-7dd2bf42f814 ||
|+-------------------------------+-----------------------------------------------------------------+|- 복원이 되는지 강제로 Pod, PVC 를 삭제해보겠습니다.
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pod app && kubectl delete pvc ebs-claim
pod "app" deleted
persistentvolumeclaim "ebs-claim" deleted
# 스냅샷에서 PVC 로 복원
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat <<EOT > ebs-snapshot-restored-claim.yaml
> apiVersion: v1
> kind: PersistentVolumeClaim
> metadata:
> name: ebs-snapshot-restored-claim
> spec:
> storageClassName: gp3
> accessModes:
> - ReadWriteOnce
> resources:
> requests:
> storage: 4Gi
> dataSource:
> name: ebs-volume-snapshot
> kind: VolumeSnapshot
> apiGroup: snapshot.storage.k8s.io
> EOT
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat ebs-snapshot-restored-claim.yaml | yh
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-snapshot-restored-claim
spec:
storageClassName: gp3
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
dataSource:
name: ebs-volume-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f ebs-snapshot-restored-claim.yaml
persistentvolumeclaim/ebs-snapshot-restored-claim created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# k get pv,pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/ebs-snapshot-restored-claim Pending gp3 20s
# Pod 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/3/ebs-snapshot-restored-pod.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat ebs-snapshot-restored-pod.yaml | yh
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: centos
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
volumeMounts:
- name: persistent-storage
mountPath: /data
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: ebs-snapshot-restored-claim
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f ebs-snapshot-restored-pod.yaml
pod/app created
# 파일 내용 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl exec app -- cat /data/out.txt
Thu Mar 21 08:43:44 UTC 2024
Thu Mar 21 08:43:49 UTC 2024
Thu Mar 21 08:43:54 UTC 2024
Thu Mar 21 08:43:59 UTC 2024
Thu Mar 21 08:44:04 UTC 2024
Thu Mar 21 08:44:09 UTC 2024
Thu Mar 21 08:44:14 UTC 2024 # 작업시간 5분의 텀이 있으며 이전의 데이터가 저장되어 있음
Thu Mar 21 08:49:34 UTC 2024
Thu Mar 21 08:49:39 UTC 2024
Thu Mar 21 08:49:44 UTC 2024
Thu Mar 21 08:49:49 UTC 2024
# 자원 삭제
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pod app && kubectl delete pvc ebs-snapshot-restored-claim && kubectl delete volumesnapshots ebs-volume-snapshot
pod "app" deleted
persistentvolumeclaim "ebs-snapshot-restored-claim" deleted
volumesnapshot.snapshot.storage.k8s.io "ebs-volume-snapshot" deleted- 참고로 정기적으로 스냅샷을 찍어내는 Volume SnapShot Scheduler 도 제공됩니다.
참고 문서 : https://popappend.tistory.com/113
참고 문서 : https://jerryljh.tistory.com/42
참고 문서 : https://backube.github.io/snapscheduler/
4. AWS EFS Controller
- AWS EFS Controller 소개 및 아키텍처 구성
- EKS Add on 으로 설치하겠습니다.
# EFS 정보 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text
fs-024b43fbf2d8c3b93
# IAM Policy 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/docs/iam-policy-example.json
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws iam create-policy --policy-name AmazonEKS_EFS_CSI_Driver_Policy --policy-document file://iam-policy-example.json
{
"Policy": {
"PolicyName": "AmazonEKS_EFS_CSI_Driver_Policy",
"PolicyId": "ANPATOH2FIJQ2OICO52VI",
"Arn": "arn:aws:iam::236747833953:policy/AmazonEKS_EFS_CSI_Driver_Policy",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 0,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"CreateDate": "2024-03-21T09:01:10+00:00",
"UpdateDate": "2024-03-21T09:01:10+00:00"
}
}
# ISRA 설정 : 고객관리형 정책 AmazonEKS_EFS_CSI_Driver_Policy 사용
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl create iamserviceaccount \
> --name efs-csi-controller-sa \
> --namespace kube-system \
> --cluster ${CLUSTER_NAME} \
> --attach-policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/AmazonEKS_EFS_CSI_Driver_Policy \
> --approve
# ISRA 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get sa -n kube-system efs-csi-controller-sa -o yaml | head -5
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::236747833953:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-vIJLW0PS5gHs
# 애드온 설치 Using Helm
'(leeeuijoo@myeks:default) [root@myeks-bastion ~]# helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver/
"aws-efs-csi-driver" has been added to your repositories
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "aws-efs-csi-driver" chart repository
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "geek-cookbook" chart repository
Update Complete. ⎈Happy Helming!⎈
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# helm upgrade -i aws-efs-csi-driver aws-efs-csi-driver/aws-efs-csi-driver \
> --namespace kube-system \
> --set image.repository=602401143452.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/eks/aws-efs-csi-driver \
> --set controller.serviceAccount.create=false \
> --set controller.serviceAccount.name=efs-csi-controller-sa
Release "aws-efs-csi-driver" does not exist. Installing it now.
NAME: aws-efs-csi-driver
LAST DEPLOYED: Thu Mar 21 18:03:08 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that aws-efs-csi-driver has started, run:
kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-efs-csi-driver,app.kubernetes.io/instance=aws-efs-csi-driver"
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-efs-csi-driver,app.kubernetes.io/instance=aws-efs-csi-driver"
NAME READY STATUS RESTARTS AGE
efs-csi-controller-789c8bf7bf-5c8tg 3/3 Running 0 51s
efs-csi-controller-789c8bf7bf-dqdfw 3/3 Running 0 51s
efs-csi-node-ht75x 3/3 Running 0 51s
efs-csi-node-rhwj2 3/3 Running 0 51s
efs-csi-node-w4ftz 3/3 Running 0 51s- AWS EFS 파일 시스템의 네트워크 탭에서 탑재 대상의 ID 를 확인 합니다.
시나리오 1 : EFS 파일시스템을 다수의 파드가 사용하게 설정 : Add empty StorageClasses from static provisioning
# 모니터링 걸어주기
watch 'kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod'
Every 2.0s: kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod Thu Mar 21 18:06:06 2024
Error from server (NotFound): storageclasses.storage.k8s.io "efs-sc" not found
No resources found
# 실습 code clone
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# git clone https://github.com/kubernetes-sigs/aws-efs-csi-driver.git /root/efs-csi
Cloning into '/root/efs-csi'...
remote: Enumerating objects: 22887, done.
remote: Counting objects: 100% (4784/4784), done.
remote: Compressing objects: 100% (1191/1191), done.
remote: Total 22887 (delta 3911), reused 3713 (delta 3549), pack-reused 18103
Receiving objects: 100% (22887/22887), 20.08 MiB | 18.40 MiB/s, done.
Resolving deltas: 100% (12597/12597), done.
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cd /root/efs-csi/examples/kubernetes/multiple_pods/specs && tree
.
├── claim.yaml
├── pod1.yaml
├── pod2.yaml
├── pv.yaml
└── storageclass.yaml
0 directories, 5 files
# EFS 스토리지클래스 생성 및 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat storageclass.yaml | yh
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl apply -f storageclass.yaml
storageclass.storage.k8s.io/efs-sc created
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl get sc efs-sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
efs-sc efs.csi.aws.com Delete Immediate false 2s
# 모니터링
Every 2.0s: kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod Thu Mar 21 18:07:20 2024
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
efs-sc efs.csi.aws.com Delete Immediate false 18s
No resources found
# PV 생성 및 확인 : volumeHandle을 자신의 EFS 파일시스템ID로 변경
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# EfsFsId=$(aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text)
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# sed -i "s/fs-4af69aab/$EfsFsId/g" pv.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat pv.yaml | yh
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-pv
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
csi:
driver: efs.csi.aws.com
volumeHandle: fs-024b43fbf2d8c3b93
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl apply -f pv.yaml
persistentvolume/efs-pv created
# PVC 생성 및 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat claim.yaml | yh
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-sc
resources:
requests:
storage: 5Gi
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl apply -f claim.yaml
persistentvolumeclaim/efs-claim created
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
efs-claim Pending efs-sc 34s
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat pod1.yaml pod2.yaml | yh
apiVersion: v1
kind: Pod
metadata:
name: app1
spec:
containers:
- name: app1
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /data/out1.txt; sleep 5; done"]
volumeMounts:
- name: persistent-storage
mountPath: /data
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: efs-claim
apiVersion: v1
kind: Pod
metadata:
name: app2
spec:
containers:
- name: app2
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /data/out2.txt; sleep 5; done"]
volumeMounts:
- name: persistent-storage
mountPath: /data
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: efs-claim
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl apply -f pod1.yaml,pod2.yaml
pod/app1 created
pod/app2 created
# 모니터링
Every 2.0s: kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod Thu Mar 21 18:12:37 2024
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
efs-sc efs.csi.aws.com Delete Immediate false 5m35s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASO
N AGE
persistentvolume/efs-pv 5Gi RWX Retain Bound default/efs-claim efs-sc
87s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/efs-claim Bound efs-pv 5Gi RWX efs-sc 3m54s
NAME READY STATUS RESTARTS AGE
pod/app1 1/1 Running 0 2m59s
pod/app2 1/1 Running 0 2m59s
# 파드 생성 및 연동 : 파드 내에 /data 데이터는 EFS를 사용
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl exec -ti app2 -- sh -c "df -hT -t nfs4"
Filesystem Type Size Used Available Use% Mounted on
127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /data
# 공유 저장소 저장 동작 확인 - bastion Host 에서 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# tree /mnt/myefs
/mnt/myefs
├── memo.txt
├── out1.txt
└── out2.txt
0 directories, 3 files
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# tail -f /mnt/myefs/out1.txt
Thu Mar 21 09:14:24 UTC 2024
Thu Mar 21 09:14:29 UTC 2024
Thu Mar 21 09:14:34 UTC 2024
Thu Mar 21 09:14:39 UTC 2024
Thu Mar 21 09:14:45 UTC 2024
Thu Mar 21 09:14:50 UTC 2024
Thu Mar 21 09:14:55 UTC 2024
Thu Mar 21 09:15:00 UTC 2024
Thu Mar 21 09:15:05 UTC 2024
Thu Mar 21 09:15:10 UTC 2024
# Pod 데이터 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl exec -ti app1 -- tail -f /data/out1.txt
Thu Mar 21 09:14:55 UTC 2024
Thu Mar 21 09:15:00 UTC 2024
Thu Mar 21 09:15:05 UTC 2024
Thu Mar 21 09:15:10 UTC 2024
Thu Mar 21 09:15:15 UTC 2024
Thu Mar 21 09:15:20 UTC 2024
Thu Mar 21 09:15:25 UTC 2024
Thu Mar 21 09:15:30 UTC 2024
Thu Mar 21 09:15:35 UTC 2024
Thu Mar 21 09:15:40 UTC 2024
^Ccommand terminated with exit code 130
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl exec -ti app2 -- tail -f /data/out2.txt
Thu Mar 21 09:15:04 UTC 2024
Thu Mar 21 09:15:09 UTC 2024
Thu Mar 21 09:15:14 UTC 2024
Thu Mar 21 09:15:19 UTC 2024
Thu Mar 21 09:15:24 UTC 2024
Thu Mar 21 09:15:29 UTC 2024
Thu Mar 21 09:15:34 UTC 2024
Thu Mar 21 09:15:39 UTC 2024
Thu Mar 21 09:15:44 UTC 2024
Thu Mar 21 09:15:49 UTC 2024
# 자원 삭제
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl delete pod app1 app2
pod "app1" deleted
pod "app2" deleted
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl delete pvc efs-claim && kubectl delete pv efs-pv && kubectl delete sc efs-sc
persistentvolumeclaim "efs-claim" deleted
persistentvolume "efs-pv" deleted
storageclass.storage.k8s.io "efs-sc" deleted시나리오 2 : EFS 파일시스템을 다수의 파드가 사용하게 설정 : Dynamic provisioning
# 모니터링 걸어 줍니다.
watch 'kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod'
Every 2.0s: kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod Thu Mar 21 18:24:48 2024
Error from server (NotFound): storageclasses.storage.k8s.io "efs-sc" not found
No resources found
# EFS 스토리지클래스 생성 및 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# curl -s -O https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/storageclass.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat storageclass.yaml | yh
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
parameters:
provisioningMode: efs-ap
fileSystemId: fs-92107410
directoryPerms: "700"
gidRangeStart: "1000" # optional
gidRangeEnd: "2000" # optional
basePath: "/dynamic_provisioning" # optional
subPathPattern: "${.PVC.namespace}/${.PVC.name}" # optional
ensureUniqueDirectory: "true" # optional
reuseAccessPoint: "false" # optional
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# sed -i "s/fs-92107410/$EfsFsId/g" storageclass.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl apply -f storageclass.yaml
storageclass.storage.k8s.io/efs-sc created
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl get sc efs-sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
efs-sc efs.csi.aws.com Delete Immediate false
# PVC/파드 생성 및 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# curl -s -O https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/pod.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat pod.yaml | yh
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-sc
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
name: efs-app
spec:
containers:
- name: app
image: centos
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /data/out; sleep 5; done"]
volumeMounts:
- name: persistent-storage
mountPath: /data
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: efs-claim
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl apply -f pod.yaml
persistentvolumeclaim/efs-claim created
pod/efs-app created
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl get pvc,pv,pod
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/efs-claim Bound pvc-aca31297-91a1-4ec1-a699-3c0b7bae8e76 5Gi RWX efs-sc 2s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-aca31297-91a1-4ec1-a699-3c0b7bae8e76 5Gi RWX Delete Bound default/efs-claim efs-sc 2s
NAME READY STATUS RESTARTS AGE
pod/efs-app 0/1 ContainerCreating 0 2s
# PVC/PV 생성 로그 확인 - successfully provisioned
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl logs -n kube-system -l app=efs-csi-controller -c csi-provisioner -f
W0321 09:03:17.015921 1 feature_gate.go:241] Setting GA feature gate Topology=true. It will be removed in a future release.
I0321 09:03:17.015973 1 feature_gate.go:249] feature gates: &{map[Topology:true]}
I0321 09:03:17.015994 1 csi-provisioner.go:154] Version: v3.6.3
I0321 09:03:17.016012 1 csi-provisioner.go:177] Building kube configs for running in cluster...
I0321 09:03:17.023790 1 common.go:138] Probing CSI driver for readiness
I0321 09:03:17.033580 1 csi-provisioner.go:230] Detected CSI driver efs.csi.aws.com
I0321 09:03:17.035101 1 csi-provisioner.go:302] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0321 09:03:17.035639 1 controller.go:732] Using saving PVs to API server in background
I0321 09:03:17.036002 1 leaderelection.go:250] attempting to acquire leader lease kube-system/efs-csi-aws-com...
I0321 09:10:52.745362 1 controller.go:1075] Final error received, removing PVC 168f7d90-2939-4b2c-aa41-8cd875f8c9a2 from claims in progress
W0321 09:10:52.745388 1 controller.go:934] Retrying syncing claim "168f7d90-2939-4b2c-aa41-8cd875f8c9a2", failure 7
E0321 09:10:52.745410 1 controller.go:957] error syncing claim "168f7d90-2939-4b2c-aa41-8cd875f8c9a2": failed to provision volume with StorageClass "efs-sc": rpc error: code = InvalidArgument desc = Missing provisioningMode parameter
I0321 09:10:52.745426 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"efs-claim", UID:"168f7d90-2939-4b2c-aa41-8cd875f8c9a2", APIVersion:"v1", ResourceVersion:"56567", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "efs-sc": rpc error: code = InvalidArgument desc = Missing provisioningMode parameter
I0321 09:26:01.356513 1 controller.go:1366] provision "default/efs-claim" class "efs-sc": started
I0321 09:26:01.357115 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"efs-claim", UID:"aca31297-91a1-4ec1-a699-3c0b7bae8e76", APIVersion:"v1", ResourceVersion:"62090", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/efs-claim"
I0321 09:26:01.682336 1 controller.go:923] successfully created PV pvc-aca31297-91a1-4ec1-a699-3c0b7bae8e76 for PVC efs-claim and csi volume name fs-024b43fbf2d8c3b93::fsap-0bfaed4b44c757359
I0321 09:26:01.682513 1 controller.go:1449] provision "default/efs-claim" class "efs-sc": volume "pvc-aca31297-91a1-4ec1-a699-3c0b7bae8e76" provisioned
I0321 09:26:01.682539 1 controller.go:1462] provision "default/efs-claim" class "efs-sc": succeeded
I0321 09:26:01.692090 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"efs-claim", UID:"aca31297-91a1-4ec1-a699-3c0b7bae8e76", APIVersion:"v1", ResourceVersion:"62090", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-aca31297-91a1-4ec1-a699-3c0b7bae8e76
# 파드 정보 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl exec -it efs-app -- sh -c "df -hT -t nfs4"
Filesystem Type Size Used Avail Use% Mounted on
127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /data
# 공유 저장소 저장 동작 확인
# bastion Host 에서 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# tree /mnt/myefs
/mnt/myefs
├── dynamic_provisioning
│ └── default
│ └── efs-claim-90a5a9ab-9f94-448f-b48a-df56940e36b2
│ └── out
├── memo.txt
├── out1.txt
└── out2.txt
3 directories, 4 files
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl exec efs-app -- bash -c "cat data/out"
Thu Mar 21 09:26:06 UTC 2024
Thu Mar 21 09:26:11 UTC 2024
Thu Mar 21 09:26:16 UTC 2024
Thu Mar 21 09:26:21 UTC 2024
Thu Mar 21 09:26:26 UTC 2024
Thu Mar 21 09:26:31 UTC 2024
Thu Mar 21 09:26:36 UTC 2024- EFS Access Point 확인
- k8s 자원 삭제
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl delete -f pod.yaml
persistentvolumeclaim "efs-claim" deleted
pod "efs-app" deleted
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl delete -f storageclass.yaml
storageclass.storage.k8s.io "efs-sc" deleted
# 모니터링
Every 2.0s: kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod Thu Mar 21 18:31:00 2024
Error from server (NotFound): storageclasses.storage.k8s.io "efs-sc" not found
No resources found
5. MountPoint for Amazon S3 CSI Driver
-
Mountpoint for Amazon S3 CSI 드라이버는 정적 프로비저닝만 지원합니다.
-
동적 프로비저닝 또는 새 버킷의 생성은 지원되지 않습니다.
-
Mountpoint for Amazon S3 CSI 드라이버는 AWS Fargate를 지원하지 않습니다. 하지만 Amazon EC2에서 실행되는 컨테이너(Amazon EKS 또는 사용자 지정 Kubernetes 설치)는 지원됩니다.
-
IAM 정책 생성
- DOC-EXAMPLE-BUCKET1 의 이름을 자신의 버킷 이름으로 수정합니다.
- Bucket_name : ejl-s3-bucket
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "MountpointFullBucketAccess",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::ejl-s3-bucket"
]
},
{
"Sid": "MountpointFullObjectAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::ejl-s3-bucket/*"
]
}
]
}- Policy Name : AmazonS3CSIDriverPolicy-ejl
- IAM 역할(Role) 생성
- eksctl 로 생성하며, 변수를 자신의 환경과 맞게 수정해줍니다.
- CLUSTER_NAME
- REGION
- ROLE_NAME
- POLICY_ARN
- eksctl 로 생성하며, 변수를 자신의 환경과 맞게 수정해줍니다.
# 변수 지정
ROLE_NAME=AmazonEKS_S3_CSI_DriverRole-ejl
POLICY_ARN=arn:aws:iam::759320821027:policy/AmazonS3CSIDriverPolicy-ejl
# IRSA 생성
eksctl create iamserviceaccount \
--name s3-csi-driver-sa \
--namespace kube-system \
--cluster $CLUSTER_NAME \
--attach-policy-arn $POLICY_ARN \
--approve \
--role-name $ROLE_NAME \
--region $AWS_DEFAULT_REGION \
--role-only-
위의 명령어를 수행하면 CloudFormation stack 으로 IAM Role 이 생성 되게 됩니다.
-
Mountpoing for Amazon S3 CSI Driver 설치
- eksctl 로 설치하겠습니다.
- my-cluster를 클러스터 이름으로, 111122223333을 Account ID로, AmazonEKS_S3_CSI_DriverRole을 이전에 생성한 IAM 역할의 이름으로 바꿉니다.
eksctl create addon --name aws-mountpoint-s3-csi-driver --cluster ${CLUSTER_NAME} --service-account-role-arn arn:aws:iam::${ACCOUNT_ID}:role/AmazonEKS_S3_CSI_DriverRole-ejl --force- s3-csi-node Demonset , Pod 가 생성됐는지 확인합니다.
kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/aws-node-9nhr2 2/2 Running 0 40m
pod/aws-node-hzk44 2/2 Running 0 40m
pod/coredns-56dfff779f-kgcb2 1/1 Running 0 46m
pod/coredns-56dfff779f-vftj5 1/1 Running 0 46m
pod/kube-proxy-m62wh 1/1 Running 0 40m
pod/kube-proxy-tlshm 1/1 Running 0 40m
pod/s3-csi-node-hgw4r 3/3 Running 0 64s
pod/s3-csi-node-z8p8b 3/3 Running 0 64s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 46m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/aws-node 2 2 2 2 2 <none> 46m
daemonset.apps/kube-proxy 2 2 2 2 2 <none> 46m
daemonset.apps/s3-csi-node 2 2 2 2 2 kubernetes.io/os=linux 64s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 2/2 2 2 46m
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-56dfff779f 2 2 2 46m-
PV 에 대한 정적 프로비저닝 테스트
- 샘플 앱은 파일 이름이 현재 날짜/시간인 파일을 생성하는 Pod 를 생성할 것입니다.
- Root 가 아닌 사용자도 S3 Bucket 내용을 볼 수 있는 옵션 추가
mountOptions:
- uid=1000
- gid=2000
- allow-other-
3개의 단계를 확인하면서 테스트 해보겠습니다.
-
PV(Persist Volume) 생성
-
PVC(Persist Volume Claim) 생성
-
PV, PVC 를 이용한 Pod 생성
-
-
PV 생성
# test-s3-pv.yml / pv name : s3-pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: s3-pv
spec:
capacity:
storage: 1200Gi # ignored, required
accessModes:
- ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
mountOptions:
- uid=1000
- gid=2000
- allow-other
csi:
driver: s3.csi.aws.com # required
volumeHandle: s3-csi-driver-volume
volumeAttributes:
bucketName: ejl-s3-bucket
# 적용
kubectl apply -f test-s3-pv.yaml
persistentvolume/s3-pv created
# 확인
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
s3-pv 1200Gi RWX Retain Available- PVC 생성
# test-s3-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: s3-claim
spec:
accessModes:
- ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
storageClassName: "" # required for static provisioning
resources:
requests:
storage: 1200Gi # ignored, required
volumeName: s3-pv
# 적용
kubectl apply -f test-s3-pvc.yaml
persistentvolumeclaim/s3-claim created
# 확인
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
s3-claim Bound s3-pv 1200Gi RWX 16s- PV, PVC 를 사용한 샘플 Pod 생성
# test-s3-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: s3-app
spec:
securityContext:
runAsUser: 1000
runAsGroup: 2000
containers:
- name: app
image: centos
command: ["/bin/sh"]
args: ["-c", "echo 'Hello from the container!' >> /data/$(date -u).txt; tail -f /dev/null"]
volumeMounts:
- name: persistent-storage
mountPath: /data
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: s3-claim
# 적용
kubectl apply -f test-s3-pod.yaml
pod/s3-app created
kubectl get pod
NAME READY STATUS RESTARTS AGE
s3-app 1/1 Running 0 34s- S3 에서도 확인
- 정상적으로 현재 날짜와 시간을 이름으로하는 File 이 생성된 것을 확인 할 수 있습니다.
- Pod 로컬의 마운트 포인트는 /data
- Pod 에 직접 접속하여 데이터 확인
kubectl exec -it s3-app /bin/bash
bash-4.4$ cd /data/
bash-4.4$ ls
'Mon Mar 11 06:51:33 UTC 2024.txt'
bash-4.4$ cat 'Mon Mar 11 06:51:33 UTC 2024.txt'
Hello from the container!- k8s 자원 삭제
kubectl delete -f test-s3-pod.yaml
kubectl delete -f test-s3-pvc.yaml
kubectl delete -f test-s3-pv.yaml
6. EKS Persistent Volumes for Instance Store & Add NodeGroup
- 신규 노드 그룹 ng2 생성 : c5d.large 의 EC2 인스턴스 스토어(임시 블록 스토리지) 설정 작업
- 데이터 손실의 케이스 : 기본 디스크 드라이브 오류 - 인스턴스가 중지됨 - 인스턴스가 최대 절전 모드로 전환됨 - 인스턴스가 종료됨
- 인스턴스 스토어는 EC2 EBS 정보에 출력되지 않습니다.
# 인스턴스 스토어 볼륨이 있는 c5 모든 타입의 스토리지 크기
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# aws ec2 describe-instance-types \
> --filters "Name=instance-type,Values=c5*" "Name=instance-storage-supported,Values=true" \
> --query "InstanceTypes[].[InstanceType, InstanceStorageInfo.TotalSizeInGB]" \
> --output table
--------------------------
| DescribeInstanceTypes |
+---------------+--------+
| c5d.12xlarge | 1800 |
| c5d.large | 50 |
| c5d.24xlarge | 3600 |
| c5d.2xlarge | 200 |
| c5d.18xlarge | 1800 |
| c5d.4xlarge | 400 |
| c5d.xlarge | 100 |
| c5d.metal | 3600 |
| c5d.9xlarge | 900 |
+---------------+--------+
# 신규 노드 그룹 생성
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# eksctl create nodegroup -c $CLUSTER_NAME -r $AWS_DEFAULT_REGION --subnet-ids "$PubSubnet1","$PubSubnet2","$PubSubnet3" --ssh-access \
> -n ng2 -t c5d.large -N 1 -m 1 -M 1 --node-volume-size=30 --node-labels disk=nvme --max-pods-per-node 100 --dry-run > myng2.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat myng2.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
managedNodeGroups:
- amiFamily: AmazonLinux2
desiredCapacity: 1
disableIMDSv1: true
disablePodIMDS: false
iam:
withAddonPolicies:
albIngress: false
appMesh: false
appMeshPreview: false
autoScaler: false
awsLoadBalancerController: false
certManager: false
cloudWatch: false
ebs: false
efs: false
externalDNS: false
fsx: false
imageBuilder: false
xRay: false
instanceSelector: {}
instanceType: c5d.large
labels:
alpha.eksctl.io/cluster-name: myeks
alpha.eksctl.io/nodegroup-name: ng2
disk: nvme
maxPodsPerNode: 100
maxSize: 1
minSize: 1
name: ng2
privateNetworking: false
releaseVersion: ""
securityGroups:
withLocal: null
withShared: null
ssh:
allow: true
publicKeyPath: ~/.ssh/id_rsa.pub
subnets:
- subnet-041ac1b03304e2253
- subnet-078624b25732697eb
- subnet-07d1173d4d997121d
tags:
alpha.eksctl.io/nodegroup-name: ng2
alpha.eksctl.io/nodegroup-type: managed
volumeIOPS: 3000
volumeSize: 30
volumeThroughput: 125
volumeType: gp3
metadata:
name: myeks
region: ap-northeast-2
version: "1.28"
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat <<EOT > nvme.yaml
> preBootstrapCommands:
> - |
> # Install Tools
> yum install nvme-cli links tree jq tcpdump sysstat -y
>
> # Filesystem & Mount
> mkfs -t xfs /dev/nvme1n1
> mkdir /data
> mount /dev/nvme1n1 /data
>
> # Get disk UUID
> uuid=\$(blkid -o value -s UUID mount /dev/nvme1n1 /data)
>
> # Mount the disk during a reboot
> echo /dev/nvme1n1 /data xfs defaults,noatime 0 2 >> /etc/fstab
> EOT
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# sed -i -n -e '/volumeType/r nvme.yaml' -e '1,$p' myng2.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# cat myng2.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
managedNodeGroups:
- amiFamily: AmazonLinux2
desiredCapacity: 1
disableIMDSv1: true
disablePodIMDS: false
iam:
withAddonPolicies:
albIngress: false
appMesh: false
appMeshPreview: false
autoScaler: false
awsLoadBalancerController: false
certManager: false
cloudWatch: false
ebs: false
efs: false
externalDNS: false
fsx: false
imageBuilder: false
xRay: false
instanceSelector: {}
instanceType: c5d.large
labels:
alpha.eksctl.io/cluster-name: myeks
alpha.eksctl.io/nodegroup-name: ng2
disk: nvme
maxPodsPerNode: 100
maxSize: 1
minSize: 1
name: ng2
privateNetworking: false
releaseVersion: ""
securityGroups:
withLocal: null
withShared: null
ssh:
allow: true
publicKeyPath: ~/.ssh/id_rsa.pub
subnets:
- subnet-041ac1b03304e2253
- subnet-078624b25732697eb
- subnet-07d1173d4d997121d
tags:
alpha.eksctl.io/nodegroup-name: ng2
alpha.eksctl.io/nodegroup-type: managed
volumeIOPS: 3000
volumeSize: 30
volumeThroughput: 125
volumeType: gp3
preBootstrapCommands:
- |
# Install Tools
yum install nvme-cli links tree jq tcpdump sysstat -y
# Filesystem & Mount
mkfs -t xfs /dev/nvme1n1
mkdir /data
mount /dev/nvme1n1 /data
# Get disk UUID
uuid=$(blkid -o value -s UUID mount /dev/nvme1n1 /data)
# Mount the disk during a reboot
echo /dev/nvme1n1 /data xfs defaults,noatime 0 2 >> /etc/fstab
metadata:
name: myeks
region: ap-northeast-2
version: "1.28"
# 배포
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# eksctl create nodegroup -f myng2.yaml
2024-03-21 19:26:43 [ℹ] nodegroup "ng2" will use "" [AmazonLinux2/1.28]
2024-03-21 19:26:43 [ℹ] using SSH public key "/root/.ssh/id_rsa.pub" as "eksctl-myeks-nodegroup-ng2-08:bb:62:c9:87:5e:e7:82:43:b3:cd:10:0c:96:a8:92"
2024-03-21 19:26:44 [ℹ] 1 existing nodegroup(s) (ng1) will be excluded
.
.
.
- Node 보안 그룹에 Bastion Host를 whitelist 로 추가해줍니다.
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# NG2SGID=$(aws ec2 describe-security-groups --filters Name=group-name,Values=*ng2* --query "SecurityGroups[*].[GroupId]" --output text)
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# aws ec2 authorize-security-group-ingress --group-id $NG2SGID --protocol '-1' --cidr 192.168.1.100/32
{
"Return": true,
"SecurityGroupRules": [
{
"SecurityGroupRuleId": "sgr-063e3a45630d4740a",
"GroupId": "sg-0f8e84a5c70c7c4e3",
"GroupOwnerId": "236747833953",
"IsEgress": false,
"IpProtocol": "-1",
"FromPort": -1,
"ToPort": -1,
"CidrIpv4": "192.168.1.100/32"
}
]
}
# Worker Node SSH 접속
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# k get nodes -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-192-168-1-166.ap-northeast-2.compute.internal Ready <none> 4h55m v1.28.5-eks-5e0fdde 192.168.1.166 43.203.210.160 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11
ip-192-168-2-138.ap-northeast-2.compute.internal Ready <none> 4h55m v1.28.5-eks-5e0fdde 192.168.2.138 15.165.208.143 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11
ip-192-168-3-184.ap-northeast-2.compute.internal Ready <none> 4h55m v1.28.5-eks-5e0fdde 192.168.3.184 3.37.129.253 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11
ip-192-168-3-204.ap-northeast-2.compute.internal Ready <none> 4m8s v1.28.5-eks-5e0fdde 192.168.3.204 13.125.213.72 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11
# 변수 설정
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# N4=192.168.3.204
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# ssh ec2-user@$N4
Last login: Thu Mar 7 07:15:20 2024 from 54.240.230.190
, #_
~\_ ####_ Amazon Linux 2
~~ \_#####\
~~ \###| AL2 End of Life is 2025-06-30.
~~ \#/ ___
~~ V~' '->
~~~ / A newer version of Amazon Linux is available!
~~._. _/
_/ _/ Amazon Linux 2023, GA and supported until 2028-03-15.
_/m/' https://aws.amazon.com/linux/amazon-linux-2023/
7 package(s) needed for security, out of 10 available
Run "sudo yum update" to apply all updates.
exit
# 정보 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# ssh ec2-user@$N4 sudo nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 vol001590a302c9e9970 Amazon Elastic Block Store 1 32.21 GB / 32.21 GB 512 B + 0 B 1.0
/dev/nvme1n1 AWS2BE5936503A0BEDD3 Amazon EC2 NVMe Instance Storage 1 50.00 GB / 50.00 GB 512 B + 0 B 0
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# ssh ec2-user@$N4 sudo lsblk -e 7 -d
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 30G 0 disk
nvme1n1 259:1 0 46.6G 0 disk /data
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# ssh ec2-user@$N4 sudo df -hT -t xfs
Filesystem Type Size Used Avail Use% Mounted on
/dev/nvme0n1p1 xfs 30G 3.4G 27G 12% /
/dev/nvme1n1 xfs 47G 365M 47G 1% /data
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# ssh ec2-user@$N4 sudo tree /data
/data
0 directories, 0 files
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# ssh ec2-user@$N4 sudo cat /etc/fstab
#
UUID=7b79f154-ffb0-4c3f-a104-87c589eb01ec / xfs defaults,noatime 1 1
/dev/nvme1n1 /data xfs defaults,noatime 0 2
# max-pod 확인
(leeeuijoo@myeks:default) [root@myeks-bastion specs]# kubectl describe node -l disk=nvme | grep Allocatable: -A7
Allocatable:
cpu: 1930m
ephemeral-storage: 27905944324
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3068868Ki
pods: 100
System Info:
- 스토리지 클래스 재생성
# 가존의 스토리지 클래스 삭제
cd ~
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete -f local-path-storage.yaml
# local-path-storage.yml 수정
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# sed -i 's/opt/data/g' local-path-storage.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f local-path-storage.yaml
namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
role.rbac.authorization.k8s.io/local-path-provisioner-role created
clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
rolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
# 모니터링
watch 'kubectl get pod -owide;echo;kubectl get pv,pvc'
ssh ec2-user@$N4 iostat -xmdz 1 -p nvme1n1
# 측정 : Read
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -s -O https://raw.githubusercontent.com/wikibook/kubepractice/main/ch10/fio-read.fio
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubestr fio -f fio-read.fio -s local-path --size 10G --nodeselector disk=nvme
PVC created kubestr-fio-pvc-6nq49
Pod created kubestr-fio-pod-jx652
Running FIO test (fio-read.fio) on StorageClass (local-path) with a PVC of Size (10G)
....
FIO test results:
FIO version - fio-3.36
Global options - ioengine=libaio verify= direct=1 gtod_reduce=
JobName:
blocksize= filesize= iodepth= rw=
read:
IOPS=20309.548828 BW(KiB/s)=81238
iops: min=16216 max=93966 avg=20319.347656
bw(KiB/s): min=64864 max=375866 avg=81277.398438
Disk stats (read/write):
nvme1n1: ios=2433480/10 merge=0/3 ticks=7646794/26 in_queue=7646819, util=99.959122%
- OK
- 결과 - IOPS MAX : 548828
- 참고) 일반 EBS(기본값 3000IOPS) vs 인스턴스 스토어 평균 IOPS 속도 비교를 했을떄 ← 인스턴스 스토어가 7배 빠르다고 합니다.
출처 : https://sharing-for-us.tistory.com/34
- k8s 자원 삭제 및 노드 그룹 삭제
kubectl delete -f local-path-storage.yaml
eksctl delete nodegroup -c $CLUSTER_NAME -n ng27. NodeGroup
- Graviton Instannce (ARM 아키텍처) 노드그룹
- arm 기반의 아키텍처는 비교적 다른 아키텍처보다 저렴하며 아래와 같은 방식으로 따로 분리하여 노드 그룹을 구성하며 비용효율적으로 구성할 수 있습니다.
# 현재 노드들의 아키텍처 정보 amd64
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get nodes -L kubernetes.io/arch
NAME STATUS ROLES AGE VERSION ARCH
ip-192-168-1-166.ap-northeast-2.compute.internal Ready <none> 5h12m v1.28.5-eks-5e0fdde amd64
ip-192-168-2-138.ap-northeast-2.compute.internal Ready <none> 5h12m v1.28.5-eks-5e0fdde amd64
ip-192-168-3-184.ap-northeast-2.compute.internal Ready <none> 5h12m v1.28.5-eks-5e0fdde amd64
ip-192-168-3-204.ap-northeast-2.compute.internal Ready <none> 21m v1.28.5-eks-5e0fdde amd64
# graviton 타입을 사용하는 신규 노드 그룹 생성
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl create nodegroup -c $CLUSTER_NAME -r $AWS_DEFAULT_REGION --subnet-ids "$PubSubnet1","$PubSubnet2","$PubSubnet3" --ssh-access \
> -n ng3 -t t4g.medium -N 1 -m 1 -M 1 --node-volume-size=30 --node-labels family=graviton --dry-run > myng3.yaml
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl create nodegroup -f myng3.yaml
# 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get nodes --label-columns eks.amazonaws.com/nodegroup,kubernetes.io/arch
NAME STATUS ROLES AGE VERSION NODEGROUP ARCH
ip-192-168-1-166.ap-northeast-2.compute.internal Ready <none> 5h16m v1.28.5-eks-5e0fdde ng1 amd64
ip-192-168-1-192.ap-northeast-2.compute.internal Ready <none> 2m14s v1.28.5-eks-5e0fdde ng3 arm64
ip-192-168-2-138.ap-northeast-2.compute.internal Ready <none> 5h16m v1.28.5-eks-5e0fdde ng1 amd64
ip-192-168-3-184.ap-northeast-2.compute.internal Ready <none> 5h16m v1.28.5-eks-5e0fdde ng1 amd64
ip-192-168-3-204.ap-northeast-2.compute.internal Ready <none> 25m v1.28.5-eks-5e0fdde ng2 amd64
# taints 셋팅 - 적용에 2~3분 정도 시간 소요
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws eks update-nodegroup-config --cluster-name $CLUSTER_NAME --nodegroup-name ng3 --taints "addOrUpdateTaints=[{key=frontend, value=true, effect=NO_EXECUTE}]"
{
"update": {
"id": "68052798-3ded-3701-b43f-1cc493135666",
"status": "InProgress",
"type": "ConfigUpdate",
"params": [
{
"type": "TaintsToAdd",
"value": "[{\"effect\":\"NO_EXECUTE\",\"value\":\"true\",\"key\":\"frontend\"}]"
}
],
"createdAt": "2024-03-21T19:54:41.017000+09:00",
"errors": []
}
}
# 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl describe nodes --selector family=graviton | grep Taints
Taints: frontend=true:NoExecute
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws eks describe-nodegroup --cluster-name $CLUSTER_NAME --nodegroup-name ng3 | jq .nodegroup.taints
[
{
"key": "frontend",
"value": "true",
"effect": "NO_EXECUTE"
}
]
- Node IP : 192.168.1.192
- Graviton Type 의 인스턴스 노드로 Pod 를 띄어 보겠습니다.
- 매니페스트 파일의 tolerations 부분을 주목
- Pod 가 어디에 배치됐는지 주목
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat << EOT > busybox.yaml
> apiVersion: v1
> kind: Pod
> metadata:
> name: busybox
> spec:
> terminationGracePeriodSeconds: 3
> containers:
> - name: busybox
> image: busybox
> command:
> - "/bin/sh"
> - "-c"
> - "while true; do date >> /home/pod-out.txt; cd /home; sync; sync; sleep 10; done"
> tolerations:
> - effect: NoExecute
> key: frontend
> operator: Exists
> EOT
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f busybox.yaml
pod/busybox created
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 17s 192.168.1.4 ip-192-168-1-192.ap-northeast-2.compute.internal <none> <none>
# 삭제
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pod busybox
pod "busybox" deleted
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl delete nodegroup -c $CLUSTER_NAME -n ng3
2024-03-21 20:01:48 [ℹ] 1 nodegroup (ng3) was included (based on the include/exclude rules)
2024-03-21 20:01:48 [ℹ] will drain 1 nodegroup(s) in cluster "myeks"
2024-03-21 20:01:48 [ℹ] starting parallel draining, max in-flight of 1
2024-03-21 20:01:48 [ℹ] cordon node "ip-192-168-1-192.ap-northeast-2.compute.internal"
2024-03-21 20:01:48 [✔] drained all nodes: [ip-192-168-1-192.ap-northeast-2.compute.internal]
2024-03-21 20:01:48 [ℹ] will delete 1 nodegroups from cluster "myeks"
2024-03-21 20:01:48 [ℹ] 1 task: { 1 task: { delete nodegroup "ng3" [async] } }
2024-03-21 20:01:49 [ℹ] will delete stack "eksctl-myeks-nodegroup-ng3"
2024-03-21 20:01:49 [✔] deleted 1 nodegroup(s) from cluster "myeks"
-
Spot Instance 노드그룹 구성
-
ec2-instance-selectort 설치
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# curl -Lo ec2-instance-selector https://github.com/aws/amazon-ec2-instance-selector/releases/download/v2.4.1/ec2-instance-selector-`uname | tr '[:upper:]' '[:lower:]'`-amd64 && chmod +x ec2-instance-selector
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 22.0M 100 22.0M 0 0 16.3M 0 0:00:01 0:00:01 --:--:-- 191M
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# mv ec2-instance-selector /usr/local/bin/
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# ec2-instance-selector --version
v2.4.1
# 사용
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# ec2-instance-selector --vcpus 2 --memory 4 --gpus 0 --current-generation -a x86_64 --deny-list 't.*' --output table-wide
NOTE: Could not retrieve 30 day avg hourly spot price for instance type x2idn.metal
NOTE: Could not retrieve 30 day avg hourly spot price for instance type p2.16xlarge
NOTE: Could not retrieve 30 day avg hourly spot price for instance type i2.xlarge
NOTE: Could not retrieve 30 day avg hourly spot price for instance type x2iedn.metal
NOTE: Could not retrieve 30 day avg hourly spot price for instance type r5dn.metal
Instance Type VCPUs Mem (GiB) Hypervisor Current Gen Hibernation Support CPU Arch Network Performance ENIs GPUs GPU Mem (GiB) GPU Info On-Demand Price/Hr Spot Price/Hr (30d avg)
------------- ----- --------- ---------- ----------- ------------------- -------- ------------------- ---- ---- ------------- -------- ------------------ -----------------------
c5.large 2 4 nitro true true x86_64 Up to 10 Gigabit 3 0 0 none $0.096 $0.04285
c5a.large 2 4 nitro true false x86_64 Up to 10 Gigabit 3 0 0 none $0.086 $0.03231
c5d.large 2 4 nitro true true x86_64 Up to 10 Gigabit 3 0 0 none $0.11 $0.03281
c6i.large 2 4 nitro true true x86_64 Up to 12.5 Gigabit 3 0 0 none $0.096 $0.0322
c6id.large 2 4 nitro true true x86_64 Up to 12.5 Gigabit 3 0 0 none $0.1155 $0.02737
c6in.large 2 4 nitro true false x86_64 Up to 25 Gigabit 3 0 0 none $0.1281 $0.02739
c7i.large 2 4 nitro true true x86_64 Up to 12.5 Gigabit 3 0 0 none $0.1008 $0.02866
- spot 공급자 생성
- Node IAM Role 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get nodes -l eks.amazonaws.com/capacityType=ON_DEMAND
NAME STATUS ROLES AGE VERSION
ip-192-168-1-166.ap-northeast-2.compute.internal Ready <none> 5h31m v1.28.5-eks-5e0fdde
ip-192-168-2-138.ap-northeast-2.compute.internal Ready <none> 5h31m v1.28.5-eks-5e0fdde
ip-192-168-3-184.ap-northeast-2.compute.internal Ready <none> 5h31m v1.28.5-eks-5e0fdde
# 생성 : 아래 node-role 은 각자 자신의 노드롤 ARN을 입력
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# aws eks create-nodegroup \
> --cluster-name $CLUSTER_NAME \
> --nodegroup-name managed-spot \
> --subnets $PubSubnet1 $PubSubnet2 $PubSubnet3 \
> --node-role arn:aws:iam::236747833953:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-gAT1cVbSa8Yc \
> --instance-types c5.large c5d.large c5a.large \
> --capacity-type SPOT \
> --scaling-config minSize=2,maxSize=3,desiredSize=2 \
> --disk-size 20
{
"nodegroup": {
"nodegroupName": "managed-spot",
"nodegroupArn": "arn:aws:eks:ap-northeast-2:236747833953:nodegroup/myeks/managed-spot/b0c7305a-c0e3-634c-24e7-21953de80bf2",
"clusterName": "myeks",
"version": "1.28",
"releaseVersion": "1.28.5-20240307",
"createdAt": "2024-03-21T20:11:35.954000+09:00",
"modifiedAt": "2024-03-21T20:11:35.954000+09:00",
"status": "CREATING",
"capacityType": "SPOT",
"scalingConfig": {
"minSize": 2,
"maxSize": 3,
"desiredSize": 2
},
"instanceTypes": [
"c5.large",
"c5d.large",
"c5a.large"
],
"subnets": [
"subnet-041ac1b03304e2253",
"subnet-078624b25732697eb",
"subnet-07d1173d4d997121d"
],
"amiType": "AL2_x86_64",
"nodeRole": "arn:aws:iam::236747833953:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-gAT1cVbSa8Yc",
"diskSize": 20,
"health": {
"issues": []
},
"updateConfig": {
"maxUnavailable": 1
},
"tags": {}
}
}
# 확인
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get nodes -L eks.amazonaws.com/capacityType,eks.amazonaws.com/nodegroup
NAME STATUS ROLES AGE VERSION CAPACITYTYPE NODEGROUP
ip-192-168-1-166.ap-northeast-2.compute.internal Ready <none> 5h36m v1.28.5-eks-5e0fdde ON_DEMAND ng1
ip-192-168-1-225.ap-northeast-2.compute.internal Ready <none> 53s v1.28.5-eks-5e0fdde SPOT managed-spot
ip-192-168-2-138.ap-northeast-2.compute.internal Ready <none> 5h36m v1.28.5-eks-5e0fdde ON_DEMAND ng1
ip-192-168-2-208.ap-northeast-2.compute.internal Ready <none> 55s v1.28.5-eks-5e0fdde SPOT managed-spot
ip-192-168-3-184.ap-northeast-2.compute.internal Ready <none> 5h36m v1.28.5-eks-5e0fdde ON_DEMAND ng1
- 인스턴스 목록
- 스팟 인스턴스 목록
- 스팟 인스턴스 타입인 노드 그룹을 사용하여 Test Pod 를 프로비저닝 해보겠습니다.
nodeSelector부분에 주목
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# cat << EOT > busybox.yaml
> apiVersion: v1
> kind: Pod
> metadata:
> name: busybox
> spec:
> terminationGracePeriodSeconds: 3
> containers:
> - name: busybox
> image: busybox
> command:
> - "/bin/sh"
> - "-c"
> - "while true; do date >> /home/pod-out.txt; cd /home; sync; sync; sleep 10; done"
> nodeSelector:
> eks.amazonaws.com/capacityType: SPOT
> EOT
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl apply -f busybox.yaml
pod/busybox created
# 스팟 타입의 노드 IP : 192.168.1.225, 192.168.2.208
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl get nodes -L eks.amazonaws.com/capacityType,eks.amazonaws.com/nodegroup -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME CAPACITYTYPE NODEGROUP
ip-192-168-1-166.ap-northeast-2.compute.internal Ready <none> 5h40m v1.28.5-eks-5e0fdde 192.168.1.166 43.203.210.160 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11 ON_DEMAND ng1
ip-192-168-1-225.ap-northeast-2.compute.internal Ready <none> 4m50s v1.28.5-eks-5e0fdde 192.168.1.225 52.78.201.182 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11 SPOT managed-spot
ip-192-168-2-138.ap-northeast-2.compute.internal Ready <none> 5h40m v1.28.5-eks-5e0fdde 192.168.2.138 15.165.208.143 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11 ON_DEMAND ng1
ip-192-168-2-208.ap-northeast-2.compute.internal Ready <none> 4m52s v1.28.5-eks-5e0fdde 192.168.2.208 3.35.15.143 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11 SPOT managed-spot
ip-192-168-3-184.ap-northeast-2.compute.internal Ready <none> 5h40m v1.28.5-eks-5e0fdde 192.168.3.184 3.37.129.253 Amazon Linux 2 5.10.210-201.852.amzn2.x86_64 containerd://1.7.11 ON_DEMAND ng1
- 어떤 노드에 Pod 가 띄어 졌는지 확인
- 스팟 타입의 노드 IP
- 192.168.1.225
- 192.168.2.208
- 스팟 타입의 노드 IP
- busybox 애플리케이션 확인
- k8s 및 노드그룹 삭제
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# kubectl delete pod busybox
pod "busybox" deleted
(leeeuijoo@myeks:default) [root@myeks-bastion ~]# eksctl delete nodegroup -c $CLUSTER_NAME -n managed-spot
2024-03-21 20:21:11 [ℹ] 1 nodegroup (managed-spot) was included (based on the include/exclude rules)
2024-03-21 20:21:11 [ℹ] will drain 1 nodegroup(s) in cluster "myeks"
2024-03-21 20:21:11 [ℹ] starting parallel draining, max in-flight of 1
2024-03-21 20:21:11 [ℹ] cordon node "ip-192-168-1-225.ap-northeast-2.compute.internal"
2024-03-21 20:21:11 [ℹ] cordon node "ip-192-168-2-208.ap-northeast-2.compute.internal"
2024-03-21 20:21:11 [✔] drained all nodes: [ip-192-168-1-225.ap-northeast-2.compute.internal ip-192-168-2-208.ap-northeast-2.compute.internal]
2024-03-21 20:21:11 [ℹ] will delete 1 nodegroups from cluster "myeks"
2024-03-21 20:21:11 [ℹ]
2 parallel tasks: { delete unowned nodegroup managed-spot, no tasks
}
2024-03-21 20:21:12 [✔] deleted 1 nodegroup(s) from cluster "myeks"8. 실습 자원 삭제
eksctl delete cluster --name $CLUSTER_NAME && aws cloudformation delete-stack --stack-name $CLUSTER_NAME
무럭무럭 자라볼까