가시다(gasida) 님이 진행하는 AEWS(Amazon EKS Workshop Study) 3기 과정으로 학습한 내용을 정리 또는 실습한 내용을 정리한 게시글입니다. 2주차는 K8S와 EKS의 Network 관련 전반적인 주제들을 4시간에 걸쳐 Study 하였습니다. 많은 학습내용 중 제가 느낀 핵심 부분들에 실습위주로 정리하고자 합니다.
1.실습 환경 배포
1.1 아키텍처
- 2개의 VPC(EKS 배포용/운영용 구분)
- 운영서버 EC2 + EKS 제어부
- EKS 데이터부(관리형 노드그룹으로 생성) * AL2023 - 페도라 기반, dnf로 패키지 설치Docs
- 비용 절감을 위해 NAT Gateway를 사용하지 않음, WorkerNode를 Public Subnet에 위치함(실운영 환경엔 부적합)
- 구성도
1.2 실습 순서
- 자신의 PC에 실습을 위한 툴 및 설정 : macOS or Windows(WSL2 사용 권장)
- AWS CloudFormation 을 통해 기본 실습 환경 배포 : 2개의 VPC, 운영 서버 EC2 1대
- eksctl 을 통해 EKS 배포
- 관리형 노드 그룹(EC2) 접속
1.2.1 Tool 설치 : macOS
-
필수
# Install awscli brew install awscli aws --version # Install eksctl brew install eksctl eksctl version # Install kubectl brew install kubernetes-cli kubectl version --client=true # Install Helm brew install helm helm version # krew 툴 및 플러그인 설치 brew install krew kubectl krew version kubectl krew install neat get-all df-pv stern kubectl krew list # 편리성 툴 설치 brew install kube-ps1 brew install kubectx # kubectl 단축 및 하이라이트 설정 brew install kubecolor echo "alias k=kubectl" >> ~/.zshrc echo "alias kubectl=kubecolor" >> ~/.zshrc echo "compdef kubecolor=kubectl" >> ~/.zshrc -
옵션
# AWS 세션매니저로 관리 노드 EC2 접속 시 사용 brew install --cask session-manager-plugin # Install sshpass brew install sshpass # Install Wireshark : 패킷 캡쳐 및 캡쳐된 파일에서 패킷 내용 확인 brew install --cask wireshark
1.2.2 AWS Configure 자격증명 설정
-
profile을 aews로 명시, aws default profile로 함
# IAM User 자격 구성 : 실습 편리를 위해 administrator 권한을 가진 IAM User 의 자격 증명 입력 aws configure --profile aews AWS Access Key ID [None]: AKIA5... AWS Secret Access Key [None]: CVNa2... Default region name [None]: ap-northeast-2 Default output format [None]: json export AWS_DEFAULT_PROFILE=aews -
caller-identity 확인
aws sts get-caller-identity { "UserId": "A***X", "Account": "1***3", "Arn": "arn:aws:iam::1***3:user/sejkim" }
1.2.3 AWS CloudFormation을 통해 기본 실습 환경 배포
- 소스 다운로드
# yaml 파일 다운로드
curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/K8S/myeks-2week.yaml
- 실습환경을 팀원들과 사용하여서 CloudFormation 소스 일부를 수정하여 개인ID 추가
```bash
ClusterBaseName:
Type: String
Default: myeks-sejkim
AllowedPattern: "[a-zA-Z][-a-zA-Z0-9]*"
Description: must be a valid Allowed Pattern '[a-zA-Z][-a-zA-Z0-9]*'
ConstraintDescription: ClusterBaseName - must be a valid Allowed Pattern
OperatorBaseName:
Type: String
Default: operator-sejkim- CloudFormation 배포
# 배포
# aws cloudformation deploy --template-file myeks-2week.yaml --stack-name mykops --parameter-overrides KeyName=<My SSH Keyname> SgIngressSshCidr=<My Home Public IP Address>/32 --region <리전>
예시) aws cloudformation deploy --template-file myeks-2week.yaml \
--stack-name myeks-sejkim --parameter-overrides KeyName=sejkim SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 --region ap-northeast-2
# CloudFormation 스택 배포 완료 후 운영서버 EC2 IP 출력
aws cloudformation describe-stacks --stack-name myeks-sejkim --query 'Stacks[*].Outputs[*].OutputValue' --output text
예시) 43.201.31.121
# 운영서버 EC2 에 SSH 접속
예시) ssh ec2-user@43.201.31.121
ssh -i <ssh 키파일> ec2-user@$(aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text)- operator-host를 쉽게 접속할 수 있도록 ~/.ssh/config 수정
cat <<EOF>> ~/.ssh/config
Host operator
Hostname 43.201.31.121
User ec2-user
Port 22
IdentityFile ~/.ssh/sejkim.pem
EOF- VPC 배포된 모습 & EC2접속
- DNS 호스트 이름, DNS 확인 활성화 됨
- VPC 리소스 맵
- EC2 접속
- DNS 호스트 이름, DNS 확인 활성화 됨
1.2.4 eksctl을 통해 EKS 배포
- 배포할 YAML 파일 작성
# Cluster_Name 설정
export CLUSTER_NAME=myeks-sejkim
# myeks-VPC/Subnet 정보 확인 및 변수 지정
export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)
echo $VPCID
export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)
echo $PubSubnet1 $PubSubnet2 $PubSubnet3
# 출력된 내용 참고 : 아래 yaml 파일 참고해서 vpc/subnet id, ssh key 경로 수정
eksctl create cluster --name $CLUSTER_NAME --region=ap-northeast-2 --nodegroup-name=ng1-sejkim --node-type=t3.medium --nodes 3 --node-volume-size=30 --vpc-public-subnets "$PubSubnet1","$PubSubnet2","$PubSubnet3" --version 1.31 --with-oidc --external-dns-access --full-ecr-access --alb-ingress-access --node-ami-family AmazonLinux2023 --ssh-access --dry-run > myeks-sejkim.yaml
# ssh 퍼블릭 키 경로 지정
SshPublic=<각자 자신의 ssh 퍼블릭 키 경로>
SshPublic=~/.ssh/sejkim.pem
echo $SshPublic
eksctl create cluster --name $CLUSTER_NAME --region=ap-northeast-2 --nodegroup-name=ng1-sejkim --node-type=t3.medium --nodes 3 --node-volume-size=30 --vpc-public-subnets "$PubSubnet1","$PubSubnet2","$PubSubnet3" --version 1.31 --with-oidc --external-dns-access --full-ecr-access --alb-ingress-access --node-ami-family AmazonLinux2023 --ssh-access --ssh-public-key $SshPublic --dry-run > myeks-sejkim.yaml- myeks-sejkim.yaml 확인
cat myeks-sejkim.yaml
accessConfig:
authenticationMode: API_AND_CONFIG_MAP
addonsConfig: {}
apiVersion: eksctl.io/v1alpha5
cloudWatch:
clusterLogging: {}
iam:
vpcResourceControllerPolicy: true
withOIDC: true
kind: ClusterConfig
kubernetesNetworkConfig:
ipFamily: IPv4
managedNodeGroups:
- amiFamily: AmazonLinux2023
desiredCapacity: 3
disableIMDSv1: true
disablePodIMDS: false
iam:
withAddonPolicies:
albIngress: false
appMesh: false
appMeshPreview: false
autoScaler: false
awsLoadBalancerController: true
certManager: false
cloudWatch: false
ebs: false
efs: false
externalDNS: true
fsx: false
imageBuilder: true
xRay: false
instanceSelector: {}
instanceType: t3.medium
labels:
alpha.eksctl.io/cluster-name: myeks-sejkim
alpha.eksctl.io/nodegroup-name: ng1-sejkim
maxSize: 3
minSize: 3
name: ng1-sejkim
privateNetworking: false
releaseVersion: ""
securityGroups:
withLocal: null
withShared: null
ssh:
allow: true
publicKeyPath: /Users/sjkim/.ssh/sejkim.pem
tags:
alpha.eksctl.io/nodegroup-name: ng1-sejkim
alpha.eksctl.io/nodegroup-type: managed
volumeIOPS: 3000
volumeSize: 30
volumeThroughput: 125
volumeType: gp3
metadata:
name: myeks-sejkim
region: ap-northeast-2
version: "1.31"
privateCluster:
enabled: false
skipEndpointCreation: false
vpc:
autoAllocateIPv6: false
cidr: 192.168.0.0/16
clusterEndpoints:
privateAccess: false
publicAccess: true
id: vpc-0e42196e1e9d25c75
manageSharedNodeSecurityGroupRules: true
nat:
gateway: Disable
subnets:
public:
ap-northeast-2a:
az: ap-northeast-2a
cidr: 192.168.1.0/24
id: subnet-073719b47fcc01727
ap-northeast-2b:
az: ap-northeast-2b
cidr: 192.168.2.0/24
id: subnet-0b55f4d3498fb87cd
ap-northeast-2c:
az: ap-northeast-2c
cidr: 192.168.3.0/24
id: subnet-049f7327bce2fabc6- myeks-sejkim.yaml 파일 작성 : ssm 접속은 기본값 적용됨
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: myeks-sejkim
region: ap-northeast-2
version: "1.31"
kubernetesNetworkConfig:
ipFamily: IPv4
iam:
vpcResourceControllerPolicy: true
withOIDC: true
accessConfig:
authenticationMode: API_AND_CONFIG_MAP
vpc:
autoAllocateIPv6: false
cidr: 192.168.0.0/16
clusterEndpoints:
privateAccess: true # if you only want to allow private access to the cluster
publicAccess: true # if you want to allow public access to the cluster
id: vpc-0e42196e1e9d25c75 # 각자 환경 정보로 수정
manageSharedNodeSecurityGroupRules: true # if you want to manage the rules of the shared node security group
nat:
gateway: Disable
subnets:
public:
ap-northeast-2a:
az: ap-northeast-2a
cidr: 192.168.1.0/24
id: subnet-073719b47fcc01727 # 각자 환경 정보로 수정
ap-northeast-2b:
az: ap-northeast-2b
cidr: 192.168.2.0/24
id: subnet-0b55f4d3498fb87cd # 각자 환경 정보로 수정
ap-northeast-2c:
az: ap-northeast-2c
cidr: 192.168.3.0/24
id: subnet-049f7327bce2fabc6 # 각자 환경 정보로 수정
addons:
- name: vpc-cni # no version is specified so it deploys the default version
version: latest # auto discovers the latest available
attachPolicyARNs: # attach IAM policies to the add-on's service account
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
configurationValues: |-
enableNetworkPolicy: "true"
- name: kube-proxy
version: latest
- name: coredns
version: latest
- name: metrics-server
version: latest
privateCluster:
enabled: false
skipEndpointCreation: false
managedNodeGroups:
- amiFamily: AmazonLinux2023
desiredCapacity: 3
disableIMDSv1: true
disablePodIMDS: false
iam:
withAddonPolicies:
albIngress: false # Disable ALB Ingress Controller
appMesh: false
appMeshPreview: false
autoScaler: false
awsLoadBalancerController: true # Enable AWS Load Balancer Controller
certManager: true # Enable cert-manager
cloudWatch: false
ebs: false
efs: false
externalDNS: true # Enable ExternalDNS
fsx: false
imageBuilder: true
xRay: false
instanceSelector: {}
instanceType: t3.medium
preBootstrapCommands:
# install additional packages
- "dnf install nvme-cli links tree tcpdump sysstat ipvsadm ipset bind-utils htop -y"
# disable hyperthreading
- "for n in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | cut -s -d, -f2- | tr ',' '\n' | sort -un); do echo 0 > /sys/devices/system/cpu/cpu${n}/online; done"
labels:
alpha.eksctl.io/cluster-name: myeks-sejkim
alpha.eksctl.io/nodegroup-name: ng1-sejkim
maxSize: 3
minSize: 3
name: ng1-sejkim
privateNetworking: false
releaseVersion: ""
securityGroups:
withLocal: null
withShared: null
ssh:
allow: true
publicKeyName: sejkim # 각자 환경 정보로 수정
tags:
alpha.eksctl.io/nodegroup-name: ng1-sejkim
alpha.eksctl.io/nodegroup-type: managed
volumeIOPS: 3000
volumeSize: 30
volumeThroughput: 125
volumeType: gp3- 최종 yaml로 eks 배포
# kubeconfig 파일 경로 위치 지정 :
export KUBECONFIG=$HOME/kubeconfig
혹은 각자 편한 경로 위치에 파일 지정
export KUBECONFIG=~/Downloads/kubeconfig
# 배포
eksctl create cluster -f myeks-sejkim.yaml --verbose 4-
배포 후 기본 정보 확인
-
EKS 관리 콘솔 확인
- Overview : API server endpoint, Open ID Connect provider URL기본 정보(oidc)
- Compute : Node groups 클릭 → AMI(AL2023)..
- Networking : access(public and private)..
- Add-ons : VPC CNI 클릭 → edit 후 권한 설정 확인(IRSA) ⇒ 해당 IAM Role 확인
- Access : IAM access entries (설치 시 사용한 자격증명 username 확인)
-
EKS 정보 확인
# cluster 정보 확인 kubectl cluster-info eksctl get cluster # 네임스페이스 default 변경 적용 kubens default # kubectl ctx cat $KUBECONFIG | grep current-context kubectl config rename-context "<각자 자신의 IAM User>@myeks.ap-northeast-2.eksctl.io" "eksworkshop" kubectl config rename-context "sejkim@myeks-sejkim.ap-northeast-2.eksctl.io" "eksworkshop" cat $KUBECONFIG | grep current-context current-context: eksworkshop # kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone kubectl get node -v=6
# pdb coredns, metrics-server 확인 kubectl get pod -A kubectl get pdb -n kube-system NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE coredns N/A 1 1 28m metrics-server N/A 1 1 28m # 관리형 노드 그룹 확인 eksctl get nodegroup --cluster $CLUSTER_NAME aws eks describe-nodegroup --cluster-name $CLUSTER_NAME --nodegroup-name ng1 | jq # eks addon 확인 eksctl get addon --cluster $CLUSTER_NAME NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES POD IDENTITY ASSOCIATION ROLES coredns v1.11.4-eksbuild.2 ACTIVE 0 kube-proxy v1.31.3-eksbuild.2 ACTIVE 0 metrics-server v0.7.2-eksbuild.2 ACTIVE 0 vpc-cni v1.19.2-eksbuild.5 ACTIVE 0 arn:aws:iam::1**********3:role/eksctl-myeks-sejkim-addon-vpc-cni-Role1-u0nEgbdcdaOH enableNetworkPolicy: "true"
-
1.2.5 관리형 노드 그룹(EC2) 접속 및 노드 정보 확인
- 관리 콘솔 EC2 서비스 : 관리형 노드 그룹(EC2) 에 보안그룹 ID 확인
- 해당 보안그룹 inbound 에 자신의 집 공인 IP 추가 후 접속 확인
# 인스턴스 공인 IP 확인
aws ec2 describe-instances --query "Reservations[*].Instances[*].{InstanceID:InstanceId, PublicIPAdd:PublicIpAddress, PrivateIPAdd:PrivateIpAddress, InstanceName:Tags[?Key=='Name']|[0].Value, Status:State.Name}" --filters Name=instance-state-name,Values=running --output table | greo sejkim
| i-066526b2d9dcc5ccc| operator-sejkim-host | 172.20.1.100 | 43.201.31.121 | running |
| i-0b7a4239537a4fd6b| myeks-sejkim-ng1-sejkim-Node | 192.168.1.222 | 3.35.207.105 | running |
| i-04f367bed56a0451e| myeks-sejkim-ng1-sejkim-Node | 192.168.2.252 | 43.203.145.220 | running |
| i-0c5625c5990d90930| myeks-sejkim-ng1-sejkim-Node | 192.168.3.206 | 13.209.76.129 | running |
# 인스턴스 공인 IP 변수 지정
export N1=3.35.207.105
export N2=43.203.145.220
export N3=13.209.76.129
echo $N1, $N2, $N3
# ping 테스트
ping -c 2 $N1
ping -c 2 $N2
ping -c 2 $N3
# *nodegroup-ng1* 포함된 보안그룹 ID
export MNSGID=<각자 자신의 관리형 노드 그룹(EC2) 에 보안그룹 ID>
export MNSGID=sg-0a620caa15a87a604
# 해당 보안그룹 inbound 에 자신의 집 공인 IP 룰 추가
aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr $(curl -s ipinfo.io/ip)/32
# 해당 보안그룹 inbound 에 운영서버 내부 IP 룰 추가
aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr 172.20.1.100/32
# AWS EC2 관리 콘솔에서 EC2에 보안 그룹에 inbound rule 에 추가된 규칙 정보 확인
# ping 테스트
ping -c 2 $N1
ping -c 2 $N2
# 워커 노드 SSH 접속
ssh -i ~/.ssh/sejkim.pem -o StrictHostKeyChecking=no ec2-user@$N1 hostname
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -o StrictHostKeyChecking=no ec2-user@$i hostname; echo; done
ssh ec2-user@$N1
exit
ssh ec2-user@$N2
exit
ssh ec2-user@$N2
exit
------------------
# 운영서버 EC2에서 접속 시
## 인스턴스 내부 IP 변수 지정
export N1=<az1 배치된 EC2 내부 IP>
export N2=<az2 배치된 EC2 내부 IP>
export N3=<az3 배치된 EC2 내부 IP>
export N1=192.168.1.222
export N2=192.168.2.252
export N3=192.168.3.206
echo $N1, $N2, $N3
## ping 테스트
ping -c 2 $N1
ping -c 2 $N2
ping -c 2 $N3- (옵션) AWS EC2 System Manager - Session Manager로 관리형 노드 그룹(EC2) 접속
- 방안1. Terminal에서 접속
# 인스턴스 ID 확인
aws ec2 describe-instances --query "Reservations[*].Instances[*].{InstanceID:InstanceId, PublicIPAdd:PublicIpAddress, PrivateIPAdd:PrivateIpAddress, InstanceName:Tags[?Key=='Name']|[0].Value, Status:State.Name}" --filters Name=instance-state-name,Values=running --output text
# Session Manager 를 통한 접속
aws ssm start-session --target i-066526b2d9dcc5ccc
--------------------------------------------------
# 기본 사용자 정보 확인
whoami
pwd
# bash shell 적용
bash
whoami
pwd
# 기본 정보 확인
hostnamectl
# sudo 권한 사용 확인 >> 가능한 이유는? ChatGPT 등에 물어보시라!
sudo cat /etc/passwd
# 빠져나오기
exit
exit
--------------------------------------------------- 방안2. 관리 콘솔 AWS EC2 System Manager - Session Manager 에서 접속 - Link
- 노드 정보 확인
# 노드 정보 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i hostnamectl; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo iptables -t nat -S; echo; done
# Node cgroup version : v1(tmpfs), v2(cgroup2fs) - Link
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i stat -fc %T /sys/fs/cgroup/; echo; done
/sys/fs/cgroup/; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i findmnt -t cgroup2; echo; done
# kubelet 상태 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo systemctl status kubelet; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i ps axf |grep /usr/bin/containerd; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo tree /etc/kubernetes/kubelet/; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo cat /etc/kubernetes/kubelet/config.json | jq; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo cat /etc/kubernetes/kubelet/config.json.d/00-nodeadm.conf | jq; echo; done
# ebs 볼륨목록과 상태 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i lsblk; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i df -hT /; echo; done
# 컨테이너 리스트 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ctr -n k8s.io container list; echo; done
>> node 13.209.76.129 <<
CONTAINER IMAGE RUNTIME
313ecc1795072a5bd4328c440b623d470a90cc4fb664712d9289a7eb5be2cbf2 602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/metrics-server:v0.7.2-eksbuild.2 io.containerd.runc.v2
3b461022b25c8bbf14561a8778c788564e34a14bce13074dc43a7435de902ac6 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.10 io.containerd.runc.v2
653b20dde80ceeb1fc2d48b00b32a439de224e00dba055d37415e3c68f1ca5e5 nvcr.io/nvidia/k8s-device-plugin:v0.17.0
...
# 컨테이너 이미지 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ctr -n k8s.io image list --quiet; echo; done
...
# 태스크 리스트 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ctr -n k8s.io task list; echo; done1.2.6 운영서버 EC2에서 eks 를 사용 할 수 있게 설정
# eks 설치한 iam 자격증명을 설정하기
aws configure
...
# get-caller-identity 확인
aws sts get-caller-identity --query Arn
# kubeconfig 생성
cat ~/.kube/config
aws eks update-kubeconfig --name myeks-sejkim --user-alias arn:aws:iam::170698194833:user/sejkim@lgcns.com
aws eks update-kubeconfig --name myeks-sejkim --user-alias admin
# 추가된 kubeconfig 정보 확인
cat ~/.kube/config
# eks api dig 조회 : VPC 내부에서 질의하는데 왜 그럴까? private hosted zone 의 특징을 알아보자
APIDNS=$(aws eks describe-cluster --name myeks-sejkim | jq -r .cluster.endpoint | cut -d '/' -f 3)
dig +short $APIDNS
# cluster 정보, default ns 지정
kubectl cluster-info
kubectl ns default
kubectl get node -v61.2.7 실습에서 자주 사용하는 변수
#
export KUBECONFIG=~/Downloads/kubeconfig
export CLUSTER_NAME=myeks-sejkim
# myeks-VPC/Subnet 정보 확인 및 변수 지정
export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)
echo $VPCID
export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)
echo $PubSubnet1 $PubSubnet2 $PubSubnet3
# 인스턴스 IP 확인
aws ec2 describe-instances --query "Reservations[*].Instances[*].{InstanceID:InstanceId, PublicIPAdd:PublicIpAddress, PrivateIPAdd:PrivateIpAddress, InstanceName:Tags[?Key=='Name']|[0].Value, Status:State.Name}" --filters Name=instance-state-name,Values=running --output table | grep sejkim
# 인스턴스 공인 IP 변수 지정
#export N1=<az1 배치된 EC2 공인 IP>
#export N2=<az2 배치된 EC2 공인 IP>
#export N3=<az3 배치된 EC2 공인 IP>
export N1=3.35.207.105
export N2=43.203.145.220
export N3=13.209.76.129
echo $N1, $N2, $N3
# 노드 정보 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i hostnamectl; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo iptables -t nat -S; echo; done
# 파드 이름 변수 지정
PODNAME1=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[0].metadata.name}')
PODNAME2=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[1].metadata.name}')
PODNAME3=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[2].metadata.name}')
# 파드 IP 변수 지정
PODIP1=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[0].status.podIP}')
PODIP2=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[1].status.podIP}')
PODIP3=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[2].status.podIP}')
# 자신의 도메인 변수 지정 : 소유하고 있는 자신의 도메인을 입력하시면 됩니다
MyDomain=<자신의 도메인>
MyDomain=ksj7279.click
MyDnzHostedZoneId=`aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --profile devops --output text`
echo $MyDnzHostedZoneId
/hostedzone/Z0703281374FZDJKGMOKK
# A 레코드 값 반복 조회
while true; do aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A']" --profile devops | jq ; date ; echo ; sleep 1; done2. AWS VPC CNI 소개
- KANS (Kubernetes Advanced Network Study) 중 본인이 작성한 글도 참고해 주세요 - 링크
- K8S CNI : Container Network Interface 는 k8s 네트워크 환경을 구성해준다 - 링크, 다양한 플러그인이 존재 - 링크
- AWS VPC CNI : 파드의 IP를 할당해준다, 파드의 IP 네트워크 대역과 노드(워커)의 IP 대역이 같아서 직접 통신이 가능하다 - Github Proposal
AWS VPC CNI(Amazon VPC Container Network Interface, amazon-vpc-cni-k8s)는 Amazon EKS(Elastic Kubernetes Service) 및 자체 관리형 Kubernetes 클러스터에서 Pod가 AWS VPC 네트워크에서 직접 IP 주소를 할당받고 네트워크를 관리할 수 있도록 해주는 CNI 플러그인
2.1 주요 특징
- VPC 네이티브 네트워킹
- Pod가 직접 VPC 서브넷에서 IP 주소를 받아 AWS 네트워크와 NAT 없이 통신 가능
- 다른 AWS 서비스 (예: RDS, ElastiCache)와의 통신 시 성능 향상
- 고성능 네트워크
- AWS ENI(Elastic Network Interface)를 활용하여 Pod 간 고속 네트워크 제공
- SR-IOV 및 DPDK 같은 네트워크 최적화 기능 활용 가능
- Security Group 적용 가능
- Pod 단위로 보안 그룹을 설정하여 세밀한 네트워크 보안 정책 적용
- IP 주소 관리 최적화
- IP 주소를 효율적으로 관리하는 Prefix Mode, Warm IP Mode 등의 모드 제공
- IP 부족 문제를 해결하기 위한 ENI Trunking 기능 지원
2.2 주요 구성 요소
- aws-node DaemonSet: 각 노드에서 실행되며 VPC CNI 플러그인을 관리
- ENI(Elastic Network Interface): Pod에 할당되는 IP 주소를 제공하는 네트워크 인터페이스
- IPAM (IP Address Management): VPC 서브넷 내에서 Pod의 IP 주소를 자동으로 관리
2.3 관련 명령어
- VPC CNI 버전 확인:
kubectl describe daemonset aws-node -n kube-system | grep Image
Image: 602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/amazon-k8s-cni-init:v1.19.2-eksbuild.5
Image: 602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/amazon-k8s-cni:v1.19.2-eksbuild.5
Image: 602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/amazon/aws-network-policy-agent:v1.1.6-eksbuild.2- VPC CNI 설정 확인:
kubectl neat get -- configmap -n kube-system amazon-vpc-cni -o yaml
apiVersion: v1
data:
branch-eni-cooldown: "60"
enable-network-policy-controller: "true"
enable-windows-ipam: "false"
enable-windows-prefix-delegation: "false"
minimum-ip-target: "3"
warm-ip-target: "1"
warm-prefix-target: "0"
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: aws-vpc-cni
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: aws-node
app.kubernetes.io/version: v1.19.2
helm.sh/chart: aws-vpc-cni-1.19.2
k8s-app: aws-node
name: amazon-vpc-cni
namespace: kube-system2.4 AWS VPC CNI 아키텍처
-
EC2 인스턴스 유형에 따라 최대 네트워크 인터페이스 수와 사용할 수 있는 슬롯 수가 다릅니다. 각 Pod는 슬롯에서 IP 주소를 사용하므로, 특정 EC2 인스턴스에서 실행할 수 있는 Pod의 수는 해당 인스턴스에 연결할 수 있는 ENI(Elastic Network Interface) 개수와 각 ENI가 지원하는 슬롯 수에 따라 결정됩니다.
인스턴스의 CPU 및 메모리 리소스 고갈을 방지하기 위해 EKS 사용자 가이드에서 권장하는 최대 Pod 수를 설정하는 것이 좋습니다.
단,hostNetwork를 사용하는 Pod는 이 계산에서 제외됩니다.
또한, 특정 인스턴스 유형에 대한 EKS의 권장 최대 Pod 수를 계산하려면max-pod-calculator.sh스크립트를 사용할 수 있습니다.
-
https://aws.github.io/aws-eks-best-practices/networking/vpc-cni/
- supports native VPC networking with the Amazon VPC Container Network Interface (CNI) plugin for Kubernetes.
- VPC 와 통합 : VPC Flow logs , VPC 라우팅 정책, 보안 그룹(Security group) 을 사용 가능함
- This plugin assigns an IP address from your VPC to each pod.
- VPC ENI 에 미리 할당된 IP(=Local-IPAM Warm IP Pool)를 파드에서 사용할 수 있음 ← 파드의 빠른 시작을 위해서
-
Calico CNI와 AWS VPC CNI 비교
- 일반적인 K8S CNI 플러그인(Calico)의 경우 노드 네트워크 대역과 Pod 네트워크 대역이 상이한 반면에
- AWS VPC CNI는 노드와 Pod의 네트워크 대역이 동일하다
- 파드간 통신 시 일반적으로 K8S CNI는 오버레이(VXLAN, IP-IP 등) 통신을 하고, AWS VPC CNI는 동일 대역으로 직접 통신을 한다
2.5 워커노드에 생성 가능한 최대 파드 갯수
- 여러가지 최대 파드 배치 제한 조건(cpu/mem 리소스, disk/pid 부족 등) 중 파드 IP 관련 살펴 보기 - 링크
- Secondary IPv4 addresses : 인스턴스 유형에 최대 ENI 갯수와 할당 가능 IP 수를 조합하여 선정
- 본인이 앞전 KANS에서 작성한 Secondary CIDR에서 적용방법을 참고 바랍니다.
- IPv4 Prefix Delegation : IPv4 28bit 서브넷(prefix)를 위임하여 할당 가능 IP 수와 인스턴스 유형에 권장하는 최대 갯수로 선정
- AWS VPC CNI Custom Networking : 노드와 파드 대역 분리, 파드에 별도 서브넷 부여 후 사용 - Docs
- Secondary IPv4 addresses : 인스턴스 유형에 최대 ENI 갯수와 할당 가능 IP 수를 조합하여 선정
- Amazon VPC CNI 플러그인을 통한 최대 파드 갯수는?
2.6 [실습] 네트워크 기본 정보 확인
- CNI 정보 확인
# CNI 정보 확인
kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.19.2-eksbuild.5
amazon-k8s-cni:v1.19.2-eksbuild.5
amazon/aws-network-policy-agent:v1.1.6-eksbuild.2
# kube-proxy config 확인 : 모드 iptables 사용 >> ipvs 모드로 변경 해보자!
kubectl describe cm -n kube-system kube-proxy-config
...
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
acceptContentTypes: ""
burst: 10
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /var/lib/kube-proxy/kubeconfig
qps: 5
clusterCIDR: ""
configSyncPeriod: 15m0s
conntrack:
maxPerCore: 32768
min: 131072
tcpCloseWaitTimeout: 1h0m0s
tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 0s
syncPeriod: 30s
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: ""
syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:10249
mode: "iptables"
nodePortAddresses: null
oomScoreAdj: -998
portRange: ""
...
# 노드 IP 확인
aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output table | grep sejkim
| operator-sejkim-host | 172.20.1.100 | 43.201.31.121 | running |
| myeks-sejkim-ng1-sejkim-Node | 192.168.1.222 | 3.35.207.105 | running |
| myeks-sejkim-ng1-sejkim-Node | 192.168.2.252 | 43.203.145.220 | running |
| myeks-sejkim-ng1-sejkim-Node | 192.168.3.206 | 13.209.76.129 | running |
# 파드 IP 확인
kubectl get pod -n kube-system -o=custom-columns=NAME:.metadata.name,IP:.status.podIP,STATUS:.status.phase
NAME IP STATUS
aws-node-2j5jk 192.168.1.222 Running
aws-node-4wv5x 192.168.2.252 Running
aws-node-h297m 192.168.3.206 Running
coredns-86f5954566-d7tm2 192.168.1.245 Running
coredns-86f5954566-rxdgh 192.168.3.59 Running
kube-proxy-q4d4t 192.168.1.222 Running
kube-proxy-rpxnw 192.168.2.252 Running
kube-proxy-vwk6x 192.168.3.206 Running
metrics-server-6bf5998d9c-74cgj 192.168.1.168 Running
metrics-server-6bf5998d9c-qscp7 192.168.3.132 Running
nvidia-device-plugin-daemonset-2nrmh 192.168.1.140 Running
nvidia-device-plugin-daemonset-fwrdw 192.168.2.176 Running
nvidia-device-plugin-daemonset-sq226 192.168.3.254 Running
# 파드 이름 확인
kubectl get pod -A -o name
# 파드 갯수 확인
kubectl get pod -A -o name | wc -l- 노드에 네트워크 정보 확인
# CNI 정보 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i tree /var/log/aws-routed-eni; echo; done
>> node 3.35.207.105 <<
/var/log/aws-routed-eni
├── ebpf-sdk.log
├── egress-v6-plugin.log
├── ipamd.log
├── network-policy-agent.log
└── plugin.log
0 directories, 5 files
>> node 43.203.145.220 <<
/var/log/aws-routed-eni
├── ebpf-sdk.log
├── egress-v6-plugin.log
├── ipamd.log
├── network-policy-agent.log
└── plugin.log
0 directories, 5 files
>> node 13.209.76.129 <<
/var/log/aws-routed-eni
├── ebpf-sdk.log
├── egress-v6-plugin.log
├── ipamd.log
├── network-policy-agent.log
└── plugin.log
0 directories, 5 files
ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/plugin.log | jq
ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/ipamd.log | jq
ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/egress-v6-plugin.log | jq
ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/ebpf-sdk.log | jq
ssh ec2-user@$N1 sudo cat /var/log/aws-routed-eni/network-policy-agent.log | jq
# 네트워크 정보 확인 : eniY는 pod network 네임스페이스와 veth pair
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -br -c addr; echo; done
>> node 3.35.207.105 <<
lo UNKNOWN 127.0.0.1/8 ::1/128
ens5 UP 192.168.1.222/24 metric 1024 fe80::e7:3eff:fe00:8fa7/64
enie5441997f0c@if3 UP fe80::74a9:d9ff:fe26:f2c1/64
eniff6a1715926@if3 UP fe80::3cb9:d7ff:feb1:835d/64
ens6 UP 192.168.1.73/24 fe80::98:cbff:fe60:e09b/64
enid369d8e5c92@if3 UP fe80::5441:16ff:fe46:1a0f/64
eni41053b8470e@if3 UP fe80::9083:d6ff:fe32:66bb/64
>> node 43.203.145.220 <<
lo UNKNOWN 127.0.0.1/8 ::1/128
ens5 UP 192.168.2.252/24 metric 1024 fe80::4b7:4dff:feb5:12c7/64
eni0101774b7b2@if3 UP fe80::a4d9:70ff:fe4d:c2c/64
ens6 UP 192.168.2.77/24 fe80::42f:66ff:fe4a:9531/64
eni60490a956e1@if3 UP fe80::a414:b9ff:fe90:5ec2/64
>> node 13.209.76.129 <<
lo UNKNOWN 127.0.0.1/8 ::1/128
ens5 UP 192.168.3.206/24 metric 1024 fe80::8ee:4fff:fe01:22ff/64
enied0d50eb716@if3 UP fe80::5c5f:c2ff:fee2:e94b/64
enif80566e7c70@if3 UP fe80::a408:83ff:fe8e:96f0/64
ens6 UP 192.168.3.56/24 fe80::8fb:eaff:fe13:1df9/64
eni924c15bed5d@if3 UP fe80::4be:b1ff:fefc:b4b/64
eniea0e4bc19e5@if3 UP fe80::3482:4bff:fe08:ebd6/64
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c addr; echo; done
>> node 3.35.207.105 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 02:e7:3e:00:8f:a7 brd ff:ff:ff:ff:ff:ff
altname enp0s5
inet 192.168.1.222/24 metric 1024 brd 192.168.1.255 scope global dynamic ens5
valid_lft 3099sec preferred_lft 3099sec
inet6 fe80::e7:3eff:fe00:8fa7/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
3: enie5441997f0c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 76:a9:d9:26:f2:c1 brd ff:ff:ff:ff:ff:ff link-netns cni-2d70cd07-5c1b-6d44-525d-d5b01d9827a0
inet6 fe80::74a9:d9ff:fe26:f2c1/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
4: eniff6a1715926@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 3e:b9:d7:b1:83:5d brd ff:ff:ff:ff:ff:ff link-netns cni-d552f3d7-e8d7-a0e2-1419-4b91e4f12e9c
inet6 fe80::3cb9:d7ff:feb1:835d/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
5: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 02:98:cb:60:e0:9b brd ff:ff:ff:ff:ff:ff
altname enp0s6
inet 192.168.1.73/24 brd 192.168.1.255 scope global ens6
valid_lft forever preferred_lft forever
inet6 fe80::98:cbff:fe60:e09b/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
6: enid369d8e5c92@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 56:41:16:46:1a:0f brd ff:ff:ff:ff:ff:ff link-netns cni-1b94e79d-fff0-c921-f5dd-c59da122055b
inet6 fe80::5441:16ff:fe46:1a0f/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
7: eni41053b8470e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 92:83:d6:32:66:bb brd ff:ff:ff:ff:ff:ff link-netns cni-82370977-555e-c748-0d41-ef41628be146
inet6 fe80::9083:d6ff:fe32:66bb/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
>> node 43.203.145.220 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 06:b7:4d:b5:12:c7 brd ff:ff:ff:ff:ff:ff
altname enp0s5
inet 192.168.2.252/24 metric 1024 brd 192.168.2.255 scope global dynamic ens5
valid_lft 3103sec preferred_lft 3103sec
inet6 fe80::4b7:4dff:feb5:12c7/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
3: eni0101774b7b2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether a6:d9:70:4d:0c:2c brd ff:ff:ff:ff:ff:ff link-netns cni-1c329921-0be9-d301-6edd-c1f549bd1e24
inet6 fe80::a4d9:70ff:fe4d:c2c/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 06:2f:66:4a:95:31 brd ff:ff:ff:ff:ff:ff
altname enp0s6
inet 192.168.2.77/24 brd 192.168.2.255 scope global ens6
valid_lft forever preferred_lft forever
inet6 fe80::42f:66ff:fe4a:9531/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
5: eni60490a956e1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether a6:14:b9:90:5e:c2 brd ff:ff:ff:ff:ff:ff link-netns cni-c6c98ba8-7447-3930-93e3-20ad165abce1
inet6 fe80::a414:b9ff:fe90:5ec2/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
>> node 13.209.76.129 <<
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 0a:ee:4f:01:22:ff brd ff:ff:ff:ff:ff:ff
altname enp0s5
inet 192.168.3.206/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5
valid_lft 3097sec preferred_lft 3097sec
inet6 fe80::8ee:4fff:fe01:22ff/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
3: enied0d50eb716@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 5e:5f:c2:e2:e9:4b brd ff:ff:ff:ff:ff:ff link-netns cni-ca357156-5369-e3b0-405e-733e33ed35c3
inet6 fe80::5c5f:c2ff:fee2:e94b/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
4: enif80566e7c70@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether a6:08:83:8e:96:f0 brd ff:ff:ff:ff:ff:ff link-netns cni-76b9be9c-9dee-3075-e09b-5227df9d373c
inet6 fe80::a408:83ff:fe8e:96f0/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
5: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 0a:fb:ea:13:1d:f9 brd ff:ff:ff:ff:ff:ff
altname enp0s6
inet 192.168.3.56/24 brd 192.168.3.255 scope global ens6
valid_lft forever preferred_lft forever
inet6 fe80::8fb:eaff:fe13:1df9/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
6: eni924c15bed5d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 06:be:b1:fc:0b:4b brd ff:ff:ff:ff:ff:ff link-netns cni-0e78002c-2aaa-c2a9-51b4-f6ae03492798
inet6 fe80::4be:b1ff:fefc:b4b/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
7: eniea0e4bc19e5@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 36:82:4b:08:eb:d6 brd ff:ff:ff:ff:ff:ff link-netns cni-2c8a2c55-70ac-14c1-1103-28d5d655ab6d
inet6 fe80::3482:4bff:fe08:ebd6/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done
>> node 3.35.207.105 <<
default via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.222 metric 1024
192.168.0.2 via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.222 metric 1024
192.168.1.0/24 dev ens5 proto kernel scope link src 192.168.1.222 metric 1024
192.168.1.1 dev ens5 proto dhcp scope link src 192.168.1.222 metric 1024
192.168.1.138 dev eni41053b8470e scope link
192.168.1.140 dev enid369d8e5c92 scope link
192.168.1.168 dev eniff6a1715926 scope link
192.168.1.245 dev enie5441997f0c scope link
>> node 43.203.145.220 <<
default via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.252 metric 1024
192.168.0.2 via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.252 metric 1024
192.168.2.0/24 dev ens5 proto kernel scope link src 192.168.2.252 metric 1024
192.168.2.1 dev ens5 proto dhcp scope link src 192.168.2.252 metric 1024
192.168.2.27 dev eni60490a956e1 scope link
192.168.2.176 dev eni0101774b7b2 scope link
>> node 13.209.76.129 <<
default via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.206 metric 1024
192.168.0.2 via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.206 metric 1024
192.168.3.0/24 dev ens5 proto kernel scope link src 192.168.3.206 metric 1024
192.168.3.1 dev ens5 proto dhcp scope link src 192.168.3.206 metric 1024
192.168.3.59 dev enied0d50eb716 scope link
192.168.3.132 dev enif80566e7c70 scope link
192.168.3.140 dev eniea0e4bc19e5 scope link
192.168.3.254 dev eni924c15bed5d scope link
ssh ec2-user@$N1 sudo iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N AWS-CONNMARK-CHAIN-0
-N AWS-SNAT-CHAIN-0
-N KUBE-KUBELET-CANARY
-N KUBE-MARK-MASQ
-N KUBE-NODEPORTS
-N KUBE-POSTROUTING
-N KUBE-PROXY-CANARY
-N KUBE-SEP-5AISLEFJL7N622O2
-N KUBE-SEP-5GKHB2LRNADIY4S6
-N KUBE-SEP-6FUUAO2HHLOXAINV
-N KUBE-SEP-BNHITNQJBUB2VZEE
-N KUBE-SEP-CYJUIAJ4MIICQ6MF
-N KUBE-SEP-FE2UE3WSS6ORRRVB
-N KUBE-SEP-JNCJZ43LL7RYNKNT
-N KUBE-SEP-MHBUK33ULMCCHTPI
-N KUBE-SEP-THNVEMPUVVBQ5EXO
-N KUBE-SEP-USLTIBDFD52R3Q2J
-N KUBE-SEP-XYDDOFWXZXQGZRSQ
-N KUBE-SERVICES
-N KUBE-SVC-ERIFXISQEP7F7OF4
-N KUBE-SVC-I7SKRZYQ7PWYV5X7
-N KUBE-SVC-JD5MR3NA4I4DYORP
-N KUBE-SVC-NPX46M4PTMTKRN6Y
-N KUBE-SVC-TCOU7JCQXEZGVUNU
-N KUBE-SVC-Z4ANX4WAEWEBLCTM
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -i eni+ -m comment --comment "AWS, outbound connections" -j AWS-CONNMARK-CHAIN-0
-A PREROUTING -m comment --comment "AWS, CONNMARK" -j CONNMARK --restore-mark --nfmask 0x80 --ctmask 0x80
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -m comment --comment "AWS SNAT CHAIN" -j AWS-SNAT-CHAIN-0
-A AWS-CONNMARK-CHAIN-0 -d 192.168.0.0/16 -m comment --comment "AWS CONNMARK CHAIN, VPC CIDR" -j RETURN
-A AWS-CONNMARK-CHAIN-0 -m comment --comment "AWS, CONNMARK" -j CONNMARK --set-xmark 0x80/0x80
-A AWS-SNAT-CHAIN-0 -d 192.168.0.0/16 -m comment --comment "AWS SNAT CHAIN" -j RETURN
-A AWS-SNAT-CHAIN-0 ! -o vlan+ -m comment --comment "AWS, SNAT" -m addrtype ! --dst-type LOCAL -j SNAT --to-source 192.168.1.222 --random-fully
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SEP-5AISLEFJL7N622O2 -s 192.168.1.100/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-5AISLEFJL7N622O2 -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 192.168.1.100:443
-A KUBE-SEP-5GKHB2LRNADIY4S6 -s 192.168.1.245/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-5GKHB2LRNADIY4S6 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 192.168.1.245:53
-A KUBE-SEP-6FUUAO2HHLOXAINV -s 192.168.3.132/32 -m comment --comment "kube-system/metrics-server:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-6FUUAO2HHLOXAINV -p tcp -m comment --comment "kube-system/metrics-server:https" -m tcp -j DNAT --to-destination 192.168.3.132:10251
-A KUBE-SEP-BNHITNQJBUB2VZEE -s 192.168.1.168/32 -m comment --comment "kube-system/metrics-server:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-BNHITNQJBUB2VZEE -p tcp -m comment --comment "kube-system/metrics-server:https" -m tcp -j DNAT --to-destination 192.168.1.168:10251
-A KUBE-SEP-CYJUIAJ4MIICQ6MF -s 192.168.3.31/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-CYJUIAJ4MIICQ6MF -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 192.168.3.31:443
-A KUBE-SEP-FE2UE3WSS6ORRRVB -s 192.168.3.59/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-FE2UE3WSS6ORRRVB -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 192.168.3.59:53
-A KUBE-SEP-JNCJZ43LL7RYNKNT -s 192.168.1.245/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-JNCJZ43LL7RYNKNT -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 192.168.1.245:53
-A KUBE-SEP-MHBUK33ULMCCHTPI -s 192.168.3.59/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-MHBUK33ULMCCHTPI -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 192.168.3.59:53
-A KUBE-SEP-THNVEMPUVVBQ5EXO -s 192.168.3.59/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-THNVEMPUVVBQ5EXO -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 192.168.3.59:9153
-A KUBE-SEP-USLTIBDFD52R3Q2J -s 192.168.1.245/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-USLTIBDFD52R3Q2J -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 192.168.1.245:9153
-A KUBE-SEP-XYDDOFWXZXQGZRSQ -s 172.0.32.0/32 -m comment --comment "kube-system/eks-extension-metrics-api:metrics-api" -j KUBE-MARK-MASQ
-A KUBE-SEP-XYDDOFWXZXQGZRSQ -p tcp -m comment --comment "kube-system/eks-extension-metrics-api:metrics-api" -m tcp -j DNAT --to-destination 172.0.32.0:10443
-A KUBE-SERVICES -d 10.100.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -d 10.100.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.100.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.100.244.127/32 -p tcp -m comment --comment "kube-system/metrics-server:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-Z4ANX4WAEWEBLCTM
-A KUBE-SERVICES -d 10.100.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.100.166.249/32 -p tcp -m comment --comment "kube-system/eks-extension-metrics-api:metrics-api cluster IP" -m tcp --dport 443 -j KUBE-SVC-I7SKRZYQ7PWYV5X7
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp -> 192.168.1.245:53" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-5GKHB2LRNADIY4S6
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp -> 192.168.3.59:53" -j KUBE-SEP-MHBUK33ULMCCHTPI
-A KUBE-SVC-I7SKRZYQ7PWYV5X7 -m comment --comment "kube-system/eks-extension-metrics-api:metrics-api -> 172.0.32.0:10443" -j KUBE-SEP-XYDDOFWXZXQGZRSQ
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics -> 192.168.1.245:9153" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-USLTIBDFD52R3Q2J
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics -> 192.168.3.59:9153" -j KUBE-SEP-THNVEMPUVVBQ5EXO
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https -> 192.168.1.100:443" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-5AISLEFJL7N622O2
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https -> 192.168.3.31:443" -j KUBE-SEP-CYJUIAJ4MIICQ6MF
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 192.168.1.245:53" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-JNCJZ43LL7RYNKNT
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 192.168.3.59:53" -j KUBE-SEP-FE2UE3WSS6ORRRVB
-A KUBE-SVC-Z4ANX4WAEWEBLCTM -m comment --comment "kube-system/metrics-server:https -> 192.168.1.168:10251" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BNHITNQJBUB2VZEE
-A KUBE-SVC-Z4ANX4WAEWEBLCTM -m comment --comment "kube-system/metrics-server:https -> 192.168.3.132:10251" -j KUBE-SEP-6FUUAO2HHLOXAINV
ssh ec2-user@$N1 sudo iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
201 12286 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
7 445 AWS-CONNMARK-CHAIN-0 all -- eni+ * 0.0.0.0/0 0.0.0.0/0 /* AWS, outbound connections */
197 12046 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* AWS, CONNMARK */ CONNMARK restore mask 0x80
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
16813 1040K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
16884 1044K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
16836 1040K AWS-SNAT-CHAIN-0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* AWS SNAT CHAIN */
Chain AWS-CONNMARK-CHAIN-0 (1 references)
pkts bytes target prot opt in out source destination
7 445 RETURN all -- * * 0.0.0.0/0 192.168.0.0/16 /* AWS CONNMARK CHAIN, VPC CIDR */
0 0 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* AWS, CONNMARK */ CONNMARK or 0x80
Chain AWS-SNAT-CHAIN-0 (1 references)
pkts bytes target prot opt in out source destination
6844 423K RETURN all -- * * 0.0.0.0/0 192.168.0.0/16 /* AWS SNAT CHAIN */
6990 437K SNAT all -- * !vlan+ 0.0.0.0/0 0.0.0.0/0 /* AWS, SNAT */ ADDRTYPE match dst-type !LOCAL to:192.168.1.222 random-fully
Chain KUBE-KUBELET-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-MARK-MASQ (11 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
Chain KUBE-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
605 37150 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000/0x4000
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK xor 0x4000
0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ random-fully
Chain KUBE-PROXY-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-SEP-5AISLEFJL7N622O2 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.1.100 0.0.0.0/0 /* default/kubernetes:https */
2 120 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ tcp to:192.168.1.100:443
Chain KUBE-SEP-5GKHB2LRNADIY4S6 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.1.245 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.1.245:53
Chain KUBE-SEP-6FUUAO2HHLOXAINV (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.3.132 0.0.0.0/0 /* kube-system/metrics-server:https */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/metrics-server:https */ tcp to:192.168.3.132:10251
Chain KUBE-SEP-BNHITNQJBUB2VZEE (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.1.168 0.0.0.0/0 /* kube-system/metrics-server:https */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/metrics-server:https */ tcp to:192.168.1.168:10251
Chain KUBE-SEP-CYJUIAJ4MIICQ6MF (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.3.31 0.0.0.0/0 /* default/kubernetes:https */
4 240 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ tcp to:192.168.3.31:443
Chain KUBE-SEP-FE2UE3WSS6ORRRVB (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.3.59 0.0.0.0/0 /* kube-system/kube-dns:dns */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.3.59:53
Chain KUBE-SEP-JNCJZ43LL7RYNKNT (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.1.245 0.0.0.0/0 /* kube-system/kube-dns:dns */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.1.245:53
Chain KUBE-SEP-MHBUK33ULMCCHTPI (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.3.59 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.3.59:53
Chain KUBE-SEP-THNVEMPUVVBQ5EXO (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.3.59 0.0.0.0/0 /* kube-system/kube-dns:metrics */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:192.168.3.59:9153
Chain KUBE-SEP-USLTIBDFD52R3Q2J (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 192.168.1.245 0.0.0.0/0 /* kube-system/kube-dns:metrics */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:192.168.1.245:9153
Chain KUBE-SEP-XYDDOFWXZXQGZRSQ (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 172.0.32.0 0.0.0.0/0 /* kube-system/eks-extension-metrics-api:metrics-api */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/eks-extension-metrics-api:metrics-api */ tcp to:172.0.32.0:10443
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SVC-JD5MR3NA4I4DYORP tcp -- * * 0.0.0.0/0 10.100.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.100.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.100.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
0 0 KUBE-SVC-Z4ANX4WAEWEBLCTM tcp -- * * 0.0.0.0/0 10.100.244.127 /* kube-system/metrics-server:https cluster IP */ tcp dpt:443
0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.100.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
0 0 KUBE-SVC-I7SKRZYQ7PWYV5X7 tcp -- * * 0.0.0.0/0 10.100.166.249 /* kube-system/eks-extension-metrics-api:metrics-api cluster IP */ tcp dpt:443
126 7588 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-5GKHB2LRNADIY4S6 all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp -> 192.168.1.245:53 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-MHBUK33ULMCCHTPI all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp -> 192.168.3.59:53 */
Chain KUBE-SVC-I7SKRZYQ7PWYV5X7 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-XYDDOFWXZXQGZRSQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/eks-extension-metrics-api:metrics-api -> 172.0.32.0:10443 */
Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-USLTIBDFD52R3Q2J all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics -> 192.168.1.245:9153 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-THNVEMPUVVBQ5EXO all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics -> 192.168.3.59:9153 */
Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
pkts bytes target prot opt in out source destination
2 120 KUBE-SEP-5AISLEFJL7N622O2 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https -> 192.168.1.100:443 */ statistic mode random probability 0.50000000000
4 240 KUBE-SEP-CYJUIAJ4MIICQ6MF all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https -> 192.168.3.31:443 */
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-JNCJZ43LL7RYNKNT all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns -> 192.168.1.245:53 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-FE2UE3WSS6ORRRVB all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns -> 192.168.3.59:53 */
Chain KUBE-SVC-Z4ANX4WAEWEBLCTM (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-BNHITNQJBUB2VZEE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/metrics-server:https -> 192.168.1.168:10251 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-6FUUAO2HHLOXAINV all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/metrics-server:https -> 192.168.3.132:10251 */3. 노드에서 기본 네트워크 정보 확인
3.1 노드 기본 네트워크 구성
- Network 네임스페이스는 호스트(Root)와 파드 별(Per Pod)로 구분된다
- 특정한 파드(kube-proxy, aws-node)는 호스트(Root)의 IP를 그대로 사용한다 ⇒ 파드의 Host Network 옵션
[Kubernetes] Pod 관련 Host Network 옵션과 동작원리 - t3.medium 의 경우 ENI 마다 최대 6개의 IP를 가질 수 있다
- ENI0, ENI1로 2개의 ENI는 자신의 IP 이외에 추가적으로 5개의 보조 프라이빗 IP를 가질수 있다
- coredns 파드는 veth 으로 호스트에는 eniY@ifN 인터페이스와 파드에 eth0 과 연결되어 있다
워커노드1예시
- 나의
워커노드1인스턴스의 네트워크 정보 확인 : 프라이빗 IP와 보조 프라이빗 IP 확인
3.2 보조 IPv4 주소를 Pod가 사용하는지 확인
# coredns 파드 IP 정보 확인
kubectl get pod -n kube-system -l k8s-app=kube-dns -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-86f5954566-d7tm2 1/1 Running 0 5h10m 192.168.1.245 ip-192-168-1-222.ap-northeast-2.compute.internal <none> <none>
coredns-86f5954566-rxdgh 1/1 Running 0 5h10m 192.168.3.59 ip-192-168-3-206.ap-northeast-2.compute.internal <none> <none>
# 노드의 라우팅 정보 확인 >> EC2 네트워크 정보의 '보조 프라이빗 IPv4 주소'와 비교해보자
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done
>> node 3.35.207.105 <<
default via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.222 metric 1024
192.168.0.2 via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.222 metric 1024
192.168.1.0/24 dev ens5 proto kernel scope link src 192.168.1.222 metric 1024
192.168.1.1 dev ens5 proto dhcp scope link src 192.168.1.222 metric 1024
192.168.1.138 dev eni41053b8470e scope link
192.168.1.140 dev enid369d8e5c92 scope link
192.168.1.168 dev eniff6a1715926 scope link
192.168.1.245 dev enie5441997f0c scope link
>> node 43.203.145.220 <<
default via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.252 metric 1024
192.168.0.2 via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.252 metric 1024
192.168.2.0/24 dev ens5 proto kernel scope link src 192.168.2.252 metric 1024
192.168.2.1 dev ens5 proto dhcp scope link src 192.168.2.252 metric 1024
192.168.2.27 dev eni60490a956e1 scope link
192.168.2.176 dev eni0101774b7b2 scope link
>> node 13.209.76.129 <<
default via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.206 metric 1024
192.168.0.2 via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.206 metric 1024
192.168.3.0/24 dev ens5 proto kernel scope link src 192.168.3.206 metric 1024
192.168.3.1 dev ens5 proto dhcp scope link src 192.168.3.206 metric 1024
192.168.3.59 dev enied0d50eb716 scope link
192.168.3.132 dev enif80566e7c70 scope link
192.168.3.140 dev eniea0e4bc19e5 scope link
192.168.3.254 dev eni924c15bed5d scope link3.3 테스트용 netshoot-pod 디플로이먼트 생성 - nicolaka/netshoot
# [터미널1~3] 노드 모니터링
ssh ec2-user@$N1
watch -d "ip link | egrep 'ens|eni' ;echo;echo "[ROUTE TABLE]"; route -n | grep eni"
ssh ec2-user@$N2
watch -d "ip link | egrep 'ens|eni' ;echo;echo "[ROUTE TABLE]"; route -n | grep eni"
ssh ec2-user@$N3
watch -d "ip link | egrep 'ens|eni' ;echo;echo "[ROUTE TABLE]"; route -n | grep eni"
# 테스트용 netshoot-pod 디플로이먼트 생성
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: netshoot-pod
spec:
replicas: 3
selector:
matchLabels:
app: netshoot-pod
template:
metadata:
labels:
app: netshoot-pod
spec:
containers:
- name: netshoot-pod
image: nicolaka/netshoot
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
# 파드 이름 변수 지정
PODNAME1=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[0].metadata.name}')
PODNAME2=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[1].metadata.name}')
PODNAME3=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[2].metadata.name}')
# 파드 확인
kubectl get pod -o wide
kubectl get pod -o=custom-columns=NAME:.metadata.name,IP:.status.podIP
netshoot-pod-744bd84b46-24tlb 192.168.3.82
netshoot-pod-744bd84b46-88vfb 192.168.1.138
netshoot-pod-744bd84b46-f5qwj 192.168.2.135
# 노드에 라우팅 정보 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo ip -c route; echo; done
>> node 3.35.207.105 <<
default via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.222 metric 1024
192.168.0.2 via 192.168.1.1 dev ens5 proto dhcp src 192.168.1.222 metric 1024
192.168.1.0/24 dev ens5 proto kernel scope link src 192.168.1.222 metric 1024
192.168.1.1 dev ens5 proto dhcp scope link src 192.168.1.222 metric 1024
192.168.1.138 dev eni1c67c247bd8 scope link
192.168.1.140 dev enid369d8e5c92 scope link
192.168.1.168 dev eniff6a1715926 scope link
192.168.1.245 dev enie5441997f0c scope link
>> node 43.203.145.220 <<
default via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.252 metric 1024
192.168.0.2 via 192.168.2.1 dev ens5 proto dhcp src 192.168.2.252 metric 1024
192.168.2.0/24 dev ens5 proto kernel scope link src 192.168.2.252 metric 1024
192.168.2.1 dev ens5 proto dhcp scope link src 192.168.2.252 metric 1024
192.168.2.135 dev eni5b90a216fda scope link
192.168.2.176 dev eni0101774b7b2 scope link
>> node 13.209.76.129 <<
default via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.206 metric 1024
192.168.0.2 via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.206 metric 1024
192.168.3.0/24 dev ens5 proto kernel scope link src 192.168.3.206 metric 1024
192.168.3.1 dev ens5 proto dhcp scope link src 192.168.3.206 metric 1024
192.168.3.59 dev enied0d50eb716 scope link
192.168.3.82 dev eni6da4127c930 scope link
192.168.3.132 dev enif80566e7c70 scope link
192.168.3.254 dev eni924c15bed5d scope link- 파드가 생성되면, 워커 노드에 eniY@ifN 추가되고 라우팅 테이블에도 정보가 추가된다
- 테스트용 파드 eniY 정보 확인 - 워커 노드 EC2
# 노드3에서 네트워크 인터페이스 정보 확인
ssh ec2-user@$N3
----------------
[ec2-user@ip-192-168-3-206 ~]$ ip -br -c addr show
lo UNKNOWN 127.0.0.1/8 ::1/128
ens5 UP 192.168.3.206/24 metric 1024 fe80::8ee:4fff:fe01:22ff/64
enied0d50eb716@if3 UP fe80::5c5f:c2ff:fee2:e94b/64
enif80566e7c70@if3 UP fe80::a408:83ff:fe8e:96f0/64
ens6 UP 192.168.3.56/24 fe80::8fb:eaff:fe13:1df9/64
eni924c15bed5d@if3 UP fe80::4be:b1ff:fefc:b4b/64
eni6da4127c930@if3 UP fe80::e02f:17ff:fee9:39dc/64
[ec2-user@ip-192-168-3-206 ~]$ ip -c link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 0a:ee:4f:01:22:ff brd ff:ff:ff:ff:ff:ff
altname enp0s5
3: enied0d50eb716@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP mode DEFAULT group default
link/ether 5e:5f:c2:e2:e9:4b brd ff:ff:ff:ff:ff:ff link-netns cni-ca357156-5369-e3b0-405e-733e33ed35c3
4: enif80566e7c70@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP mode DEFAULT group default
link/ether a6:08:83:8e:96:f0 brd ff:ff:ff:ff:ff:ff link-netns cni-76b9be9c-9dee-3075-e09b-5227df9d373c
5: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 0a:fb:ea:13:1d:f9 brd ff:ff:ff:ff:ff:ff
altname enp0s6
6: eni924c15bed5d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP mode DEFAULT group default
link/ether 06:be:b1:fc:0b:4b brd ff:ff:ff:ff:ff:ff link-netns cni-0e78002c-2aaa-c2a9-51b4-f6ae03492798
8: eni6da4127c930@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP mode DEFAULT group default
link/ether e2:2f:17:e9:39:dc brd ff:ff:ff:ff:ff:ff link-netns cni-e7c1d5ec-6fa3-c5ae-506a-2ec339b97412
[ec2-user@ip-192-168-3-206 ~]$ ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 0a:ee:4f:01:22:ff brd ff:ff:ff:ff:ff:ff
altname enp0s5
inet 192.168.3.206/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5
valid_lft 1935sec preferred_lft 1935sec
inet6 fe80::8ee:4fff:fe01:22ff/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
3: enied0d50eb716@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 5e:5f:c2:e2:e9:4b brd ff:ff:ff:ff:ff:ff link-netns cni-ca357156-5369-e3b0-405e-733e33ed35c3
inet6 fe80::5c5f:c2ff:fee2:e94b/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
4: enif80566e7c70@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether a6:08:83:8e:96:f0 brd ff:ff:ff:ff:ff:ff link-netns cni-76b9be9c-9dee-3075-e09b-5227df9d373c
inet6 fe80::a408:83ff:fe8e:96f0/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
5: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
link/ether 0a:fb:ea:13:1d:f9 brd ff:ff:ff:ff:ff:ff
altname enp0s6
inet 192.168.3.56/24 brd 192.168.3.255 scope global ens6
valid_lft forever preferred_lft forever
inet6 fe80::8fb:eaff:fe13:1df9/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
6: eni924c15bed5d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 06:be:b1:fc:0b:4b brd ff:ff:ff:ff:ff:ff link-netns cni-0e78002c-2aaa-c2a9-51b4-f6ae03492798
inet6 fe80::4be:b1ff:fefc:b4b/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
8: eni6da4127c930@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether e2:2f:17:e9:39:dc brd ff:ff:ff:ff:ff:ff link-netns cni-e7c1d5ec-6fa3-c5ae-506a-2ec339b97412
inet6 fe80::e02f:17ff:fee9:39dc/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
[ec2-user@ip-192-168-3-206 ~]$ ip route # 혹은 route -n
default via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.206 metric 1024
192.168.0.2 via 192.168.3.1 dev ens5 proto dhcp src 192.168.3.206 metric 1024
192.168.3.0/24 dev ens5 proto kernel scope link src 192.168.3.206 metric 1024
192.168.3.1 dev ens5 proto dhcp scope link src 192.168.3.206 metric 1024
192.168.3.59 dev enied0d50eb716 scope link
192.168.3.82 dev eni6da4127c930 scope link
192.168.3.132 dev enif80566e7c70 scope link
192.168.3.254 dev eni924c15bed5d scope link
# 네임스페이스 정보 출력 -t net(네트워크 타입)
sudo lsns -t net
NS TYPE NPROCS PID USER NETNSID NSFS COMMAND
4026531840 net 117 1 root unassigned /usr/lib/systemd/systemd --switched-root --system --deserialize=32
4026532205 net 2 3214 65535 0 /run/netns/cni-ca357156-5369-e3b0-405e-733e33ed35c3 /pause
4026532291 net 2 3303 65535 1 /run/netns/cni-76b9be9c-9dee-3075-e09b-5227df9d373c /pause
4026532401 net 2 3779 65535 2 /run/netns/cni-0e78002c-2aaa-c2a9-51b4-f6ae03492798 /pause
4026532481 net 2 100712 65535 3 /run/netns/cni-e7c1d5ec-6fa3-c5ae-506a-2ec339b97412 /pause
# PID 정보로 파드 정보 확인
PID=<PID> # PID 높은 것 중 COMMAND가 pause 인것
PID=3214
[ec2-user@ip-192-168-3-206 ~]$ PID=3214
[ec2-user@ip-192-168-3-206 ~]$ sudo nsenter -t $PID -n ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host proto kernel_lo
valid_lft forever preferred_lft forever
3: eth0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 36:b0:37:75:e0:6a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.3.59/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::34b0:37ff:fe75:e06a/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
[ec2-user@ip-192-168-3-206 ~]$ sudo nsenter -t $PID -n ip -c route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
exit
----------------- 테스트용 파드 접속(exec) 후 확인
# 테스트용 파드 접속(exec) 후 Shell 실행
kubectl exec -it $PODNAME1 -- zsh
dP dP dP
88 88 88
88d888b. .d8888b. d8888P .d8888b. 88d888b. .d8888b. .d8888b. d8888P
88' `88 88ooood8 88 Y8ooooo. 88' `88 88' `88 88' `88 88
88 88 88. ... 88 88 88 88 88. .88 88. .88 88
dP dP `88888P' dP `88888P' dP dP `88888P' `88888P' dP
Welcome to Netshoot! (github.com/nicolaka/netshoot)
Version: 0.13
# 아래부터는 pod-1 Shell 에서 실행 : 네트워크 정보 확인
----------------------------
netshoot-pod-744bd84b46-24tlb ~ ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host proto kernel_lo
valid_lft forever preferred_lft forever
3: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether f2:4a:18:df:96:76 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.3.82/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::f04a:18ff:fedf:9676/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
netshoot-pod-744bd84b46-24tlb ~ ip -c route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
netshoot-pod-744bd84b46-24tlb ~ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 169.254.1.1 0.0.0.0 UG 0 0 0 eth0
169.254.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
netshoot-pod-744bd84b46-24tlb ~ ping -c 1 192.168.2.135 <pod2 ip>
PING 192.168.2.135 (192.168.2.135) 56(84) bytes of data.
64 bytes from 192.168.2.135: icmp_seq=1 ttl=125 time=1.36 ms
--- 192.168.2.135 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.356/1.356/1.356/0.000 ms
netshoot-pod-744bd84b46-24tlb ~ ps
PID USER TIME COMMAND
1 root 0:00 tail -f /dev/null
7 root 0:01 zsh
93 root 0:00 ps
netshoot-pod-744bd84b46-24tlb ~ cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local ap-northeast-2.compute.internal
nameserver 10.100.0.10
options ndots:5
exit
----------------------------
# 파드2 Shell 실행
kubectl exec -it $PODNAME2 -- ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host proto kernel_lo
valid_lft forever preferred_lft forever
3: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default
link/ether 42:7b:64:12:b8:31 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.1.138/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::407b:64ff:fe12:b831/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
# 파드3 Shell 실행
kubectl exec -it $PODNAME3 -- ip -br -c addr
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0@if6 UP 192.168.2.135/32 fe80::287d:60ff:feb9:7922/644. 노드 간 파드 통신
목표: 파드간 통신 시 tcpdump 내용을 확인하고 통신 과정을 알아본다
4.1 파드간 통신 흐름
- AWS VPC CNI 경우 별도의 오버레이(Overlay) 통신 기술 없이, VPC Native 하게 파드간 직접 통신이 가능하다
- 파드간 통신 시 과정 참고 - 링크
4.2 [실습] 파드간 통신 테스트 및 확인
- 별도의 NAT 동작 없이 통신 가능!
# 파드 IP 변수 지정
PODIP1=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[0].status.podIP}')
PODIP2=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[1].status.podIP}')
PODIP3=$(kubectl get pod -l app=netshoot-pod -o jsonpath='{.items[2].status.podIP}')
# 파드1 Shell 에서 파드2로 ping 테스트
kubectl exec -it $PODNAME1 -- ping -c 2 $PODIP2
# 파드2 Shell 에서 파드3로 ping 테스트
kubectl exec -it $PODNAME2 -- ping -c 2 $PODIP3
# 파드3 Shell 에서 파드1로 ping 테스트
kubectl exec -it $PODNAME3 -- ping -c 2 $PODIP1
# 워커 노드 EC2 : TCPDUMP 확인
## For Pod to external (outside VPC) traffic, we will program iptables to SNAT using Primary IP address on the Primary ENI.
sudo tcpdump -i any -nn icmp
sudo tcpdump -i ens5 -nn icmp
sudo tcpdump -i ens6 -nn icmp
sudo tcpdump -i eniYYYYYYYY -nn icmp
[워커 노드1]
# routing policy database management 확인
ip rule
# routing table management 확인
ip route show table local
# 디폴트 네트워크 정보를 ens5 을 통해서 빠져나간다
ip route show table main
default via 192.168.1.1 dev ens5
...5. 파드에서 외부 통신
파드에서 외부 통신 흐름 : iptable 에 SNAT 을 통하여 노드의 eth0(ens5) IP로 변경되어서 외부와 통신됨
5.1 통신 흐름
- VPC CNI 의 External source network address translation (SNAT) 설정에 따라, 외부(인터넷) 통신 시 SNAT 하거나 혹은 SNAT 없이 통신을 할 수 있다 - 링크
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/cni-proposal.md
5.2 [실습] 파드에서 외부 통신 테스트 및 확인
# pod-1 Shell 에서 외부로 ping
kubectl exec -it $PODNAME1 -- ping -c 1 www.google.com
kubectl exec -it $PODNAME1 -- ping -i 0.1 www.google.com
kubectl exec -it $PODNAME1 -- ping -i 0.1 8.8.8.8
# 워커 노드 EC2 : TCPDUMP 확인
sudo tcpdump -i any -nn icmp
sudo tcpdump -i ens5 -nn icmp
# 퍼블릭IP 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i curl -s ipinfo.io/ip; echo; echo; done
# 작업용 EC2 : pod-1 Shell 에서 외부 접속 확인 - 공인IP는 어떤 주소인가?
## The right way to check the weather - 링크
for i in $PODNAME1 $PODNAME2 $PODNAME3; do echo ">> Pod : $i <<"; kubectl exec -it $i -- curl -s ipinfo.io/ip; echo; echo; done
kubectl exec -it $PODNAME1 -- curl -s wttr.in/seoul
kubectl exec -it $PODNAME1 -- curl -s wttr.in/seoul?format=3
kubectl exec -it $PODNAME1 -- curl -s wttr.in/Moon
kubectl exec -it $PODNAME1 -- curl -s wttr.in/:help
# 워커 노드 EC2
## 출력된 결과를 보고 어떻게 빠져나가는지 고민해보자!
ip rule
ip route show table main
sudo iptables -L -n -v -t nat
sudo iptables -t nat -S
# 파드가 외부와 통신시에는 아래 처럼 'AWS-SNAT-CHAIN-0' 룰(rule)에 의해서 SNAT 되어서 외부와 통신!
# 참고로 뒤 IP는 eth0(ENI 첫번째)의 IP 주소이다
# --random-fully 동작 - 링크1 링크2
sudo iptables -t nat -S | grep 'A AWS-SNAT-CHAIN'
-A AWS-SNAT-CHAIN-0 ! -d 192.168.0.0/16 -m comment --comment "AWS SNAT CHAIN" -j RETURN
-A AWS-SNAT-CHAIN-0 ! -o vlan+ -m comment --comment "AWS, SNAT" -m addrtype ! --dst-type LOCAL -j SNAT --to-source 192.168.1.251 --random-fully
## 아래 'mark 0x4000/0x4000' 매칭되지 않아서 RETURN 됨!
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
...
# 카운트 확인 시 AWS-SNAT-CHAIN-0에 매칭되어, 목적지가 192.168.0.0/16 아니고 외부 빠져나갈때 SNAT 192.168.1.251(EC2 노드1 IP) 변경되어 나간다!
sudo iptables -t filter --zero; sudo iptables -t nat --zero; sudo iptables -t mangle --zero; sudo iptables -t raw --zero
watch -d 'sudo iptables -v --numeric --table nat --list AWS-SNAT-CHAIN-0; echo ; sudo iptables -v --numeric --table nat --list KUBE-POSTROUTING; echo ; sudo iptables -v --numeric --table nat --list POSTROUTING'
# conntrack 확인 : EC2 메타데이터 주소(169.254.169.254) 제외 출력
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo conntrack -L -n |grep -v '169.254.169'; echo; done
conntrack v1.4.5 (conntrack-tools):
icmp 1 28 src=172.30.66.58 dst=8.8.8.8 type=8 code=0 id=34392 src=8.8.8.8 dst=172.30.85.242 type=0 code=0 id=50705 mark=128 use=1
tcp 6 23 TIME_WAIT src=172.30.66.58 dst=34.117.59.81 sport=58144 dport=80 src=34.117.59.81 dst=172.30.85.242 sport=80 dport=44768 [ASSURED] mark=128 use=15.3 [실습] 파드 ↔ 운영서버 EC2 간 통신 확인
- 운영서버 EC2 → 파드 IP 통신 : 통신이 가능한 이유는? 통신 경로를 알아보자
# 운영서버 EC2 SSH 접속
ssh <운영서버 EC2 공인 IP>
-----------------------
POD1IP=<파드1 IP 지정>
POD1IP=192.168.1.101
ping -c 1 $POD1IP
exit
-----------------------
# 워커노드1 에서 tcpdump 확인 : NAT 동작 적용 여유 확인
sudo tcpdump -i any -nn icmp- 파드1 → 운영서버 EC2 통신 : 통신이 가능한 이유는? 통신 경로를 알아보자 - Docs
# vpc cni env 정보 확인
kubectl get ds aws-node -n kube-system -o json | jq '.spec.template.spec.containers[0].env'
...
{
"name": "AWS_VPC_K8S_CNI_EXTERNALSNAT",
"value": "false"
},
...
# 운영서버 EC2 SSH 접속
kubectl exec -it $PODNAME1 -- ping 172.20.1.100
# 파드1 배치 워커노드에서 tcpdump 확인 : NAT 동작 적용 여유 확인
sudo tcpdump -i any -nn icmp
# 운영서버 EC2 에서 tcpdump 확인 : NAT 동작 적용 여유 확인
sudo tcpdump -i any -nn icmp
-----------------------------------------------------
# 파드1 배치 워커노드 : NAT 적용 정책 확인
sudo iptables -t filter --zero; sudo iptables -t nat --zero; sudo iptables -t mangle --zero; sudo iptables -t raw --zero
watch -d 'sudo iptables -v --numeric --table nat --list AWS-SNAT-CHAIN-0; echo ; sudo iptables -v --numeric --table nat --list KUBE-POSTROUTING; echo ; sudo iptables -v --numeric --table nat --list POSTROUTING'
# 파드 상태 모니터링
# kubectl set env 명령어는 내부적으로 kubectl patch를 실행하여 PodSpec을 변경 → 이로 인해 aws-node 데몬셋이 자동으로 롤링 업데이트
watch -d kubectl get pod -n kube-system
# 파드1 배치 워커노드 iptables rule 모니터링 : iptables rule 추가됨
watch -d 'sudo iptables -v --numeric --table nat --list AWS-SNAT-CHAIN-0; echo ; sudo iptables -v --numeric --table nat --list KUBE-POSTROUTING; echo ; sudo iptables -v --numeric --table nat --list POSTROUTING'
# 사내 내부에 연결 확장된 네트워크 대역과 SNAT 없이 통신 가능하게 설정
kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS=172.20.0.0/16
#
kubectl get ds aws-node -n kube-system -o json | jq '.spec.template.spec.containers[0].env'
...
{
"name": "AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS",
"value": "172.20.0.0/16"
}
# 운영서버 EC2 SSH 접속
kubectl exec -it $PODNAME1 -- ping 172.20.1.100
# 파드1 배치 워커노드 : NAT 적용 정책 확인
sudo iptables -t filter --zero; sudo iptables -t nat --zero; sudo iptables -t mangle --zero; sudo iptables -t raw --zero
watch -d 'sudo iptables -v --numeric --table nat --list AWS-SNAT-CHAIN-0; echo ; sudo iptables -v --numeric --table nat --list KUBE-POSTROUTING; echo ; sudo iptables -v --numeric --table nat --list POSTROUTING'
Chain AWS-SNAT-CHAIN-0 (1 references)
pkts bytes target prot opt in out source destination
1 84 RETURN all -- * * 0.0.0.0/0 172.20.0.0/16 /* AWS SNAT CHAIN EXCLUSION */
730 45228 RETURN all -- * * 0.0.0.0/0 192.168.0.0/16 /* AWS SNAT CHAIN */
...- env 에 설정을 영구 유지하려면 어떻게 해야 될까요? - Link
- 다음 실습을 위해서 디플로이먼트 삭제: kubectl delete deploy netshoot-pod
6. Service & AWS LoadBalancer Controller
6.1 k8s Service
6.2 Service 종류
- ClusterIP 타입
- NodePort 타입
- LoadBalancer 타입 (기본 모드) : NLB 인스턴스 유형 ⇒ 노드IP:노드포트
- Cloud Controller Manager 를 통해 K8S NodePort 정보를 사용하는 CLB/NLB 프로비저닝
- Service (LoadBalancer Controller) : AWS Load Balancer Controller + NLB (파드) IP 모드 동작 with AWS VPC CNI
- iptables 정책 적용 순서 : PREROUTING → KUBE-SERVICES → KUBE-SVC-### → KUBE-SEP-#<파드1> , KUBE-SEP-#<파드2> , KUBE-SEP-#<파드3>
6.3 AWS LoadBalancer Controller 배포 - Link
# 설치 전 CRD 확인
kubectl get crd
# Helm Chart 설치
helm repo add eks https://aws.github.io/eks-charts
helm repo update
helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME
## 설치 확인
kubectl get crd
kubectl explain ingressclassparams.elbv2.k8s.aws
kubectl explain targetgroupbindings.elbv2.k8s.aws
kubectl get deployment -n kube-system aws-load-balancer-controller
kubectl describe deploy -n kube-system aws-load-balancer-controller
kubectl describe deploy -n kube-system aws-load-balancer-controller | grep 'Service Account'
Service Account: aws-load-balancer-controller
# 클러스터롤, 롤 확인
kubectl describe clusterrolebindings.rbac.authorization.k8s.io aws-load-balancer-controller-rolebinding
kubectl describe clusterroles.rbac.authorization.k8s.io aws-load-balancer-controller-role
...
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
targetgroupbindings.elbv2.k8s.aws [] [] [create delete get list patch update watch]
events [] [] [create patch]
ingresses [] [] [get list patch update watch]
services [] [] [get list patch update watch]
ingresses.extensions [] [] [get list patch update watch]
services.extensions [] [] [get list patch update watch]
ingresses.networking.k8s.io [] [] [get list patch update watch]
services.networking.k8s.io [] [] [get list patch update watch]
endpoints [] [] [get list watch]
namespaces [] [] [get list watch]
nodes [] [] [get list watch]
pods [] [] [get list watch]
endpointslices.discovery.k8s.io [] [] [get list watch]
ingressclassparams.elbv2.k8s.aws [] [] [get list watch]
ingressclasses.networking.k8s.io [] [] [get list watch]
ingresses/status [] [] [update patch]
pods/status [] [] [update patch]
services/status [] [] [update patch]
targetgroupbindings/status [] [] [update patch]
ingresses.elbv2.k8s.aws/status [] [] [update patch]
pods.elbv2.k8s.aws/status [] [] [update patch]
services.elbv2.k8s.aws/status [] [] [update patch]
targetgroupbindings.elbv2.k8s.aws/status [] [] [update patch]
ingresses.extensions/status [] [] [update patch]
pods.extensions/status [] [] [update patch]
services.extensions/status [] [] [update patch]
targetgroupbindings.extensions/status [] [] [update patch]
ingresses.networking.k8s.io/status [] [] [update patch]
pods.networking.k8s.io/status [] [] [update patch]
services.networking.k8s.io/status [] [] [update patch]
targetgroupbindings.networking.k8s.io/status [] [] [update patch]
6.4 서비스/파드 배포 테스트 with NLB - Docs, NLB
- Network Load Balancer - AWS Load Balancer Controller : NLB 보안그룹 지원에 따른 기본 설정 내용 참고
# 모니터링
watch -d kubectl get pod,svc,ep,endpointslices
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 6h59m
NAME ENDPOINTS AGE
endpoints/kubernetes 192.168.1.100:443,192.168.3.31:443 6h59m
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
endpointslice.discovery.k8s.io/kubernetes IPv4 443 192.168.1.100,192.168.3.31 6h59m
# 디플로이먼트 & 서비스 생성
cat << EOF > echo-service-nlb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploy-echo
spec:
replicas: 2
selector:
matchLabels:
app: deploy-websrv
template:
metadata:
labels:
app: deploy-websrv
spec:
terminationGracePeriodSeconds: 0
containers:
- name: aews-websrv
image: k8s.gcr.io/echoserver:1.5
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: svc-nlb-ip-type
annotations:
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8080"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
type: LoadBalancer
loadBalancerClass: service.k8s.aws/nlb
selector:
app: deploy-websrv
EOF
kubectl apply -f echo-service-nlb.yaml
# 확인
aws elbv2 describe-load-balancers --query 'LoadBalancers[*].State.Code' --output text
kubectl get deploy,pod
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/deploy-echo 2/2 2 2 33s
NAME READY STATUS RESTARTS AGE
pod/deploy-echo-bf9bdb8bc-2tqls 1/1 Running 0 33s
pod/deploy-echo-bf9bdb8bc-b67dt 1/1 Running 0 33s
kubectl get svc,ep,ingressclassparams,targetgroupbindings
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 7h1m
service/svc-nlb-ip-type LoadBalancer 10.100.103.26 k8s-default-svcnlbip-b9f4a7d045-df3b0a43de1e7e78.elb.ap-northeast-2.amazonaws.com 80:30438/TCP 50s
NAME ENDPOINTS AGE
endpoints/kubernetes 192.168.1.100:443,192.168.3.31:443 7h1m
endpoints/svc-nlb-ip-type 192.168.2.27:8080,192.168.3.82:8080 50s
NAME GROUP-NAME SCHEME IP-ADDRESS-TYPE AGE
ingressclassparams.elbv2.k8s.aws/alb 14m
NAME SERVICE-NAME SERVICE-PORT TARGET-TYPE AGE
targetgroupbinding.elbv2.k8s.aws/k8s-default-svcnlbip-9aa7b9b4dc svc-nlb-ip-type 80 ip 46
kubectl get targetgroupbindings -o json | jq
{
"apiVersion": "v1",
"items": [
{
"apiVersion": "elbv2.k8s.aws/v1beta1",
"kind": "TargetGroupBinding",
"metadata": {
"annotations": {
"elbv2.k8s.aws/checkpoint": "gFbmrl063ATAAuEPNUrOR6pgIJNo3Ugx3hXo9jGkfuY/X9jowi32V4q5d2TFj2GPZLCE5Xsujs6lS4Zx1eql5mk",
"elbv2.k8s.aws/checkpoint-timestamp": "1739626660"
},
"creationTimestamp": "2025-02-15T13:37:36Z",
"finalizers": [
"elbv2.k8s.aws/resources"
],
"generation": 1,
"labels": {
"service.k8s.aws/stack-name": "svc-nlb-ip-type",
"service.k8s.aws/stack-namespace": "default"
},
"name": "k8s-default-svcnlbip-9aa7b9b4dc",
"namespace": "default",
"resourceVersion": "85609",
"uid": "bbec03e2-8b74-44f3-8ccc-d3ebfbb0c832"
},
"spec": {
"ipAddressType": "ipv4",
"networking": {
"ingress": [
{
"from": [
{
"securityGroup": {
"groupID": "sg-0b4259c1f6ca841a9"
}
}
],
"ports": [
{
"port": 8080,
"protocol": "TCP"
}
]
}
]
},
"serviceRef": {
"name": "svc-nlb-ip-type",
"port": 80
},
"targetGroupARN": "arn:aws:elasticloadbalancing:ap-northeast-2:170698194833:targetgroup/k8s-default-svcnlbip-9aa7b9b4dc/4871f3eb4b95333e",
"targetType": "ip",
"vpcID": "vpc-0e42196e1e9d25c75"
},
"status": {
"observedGeneration": 1
}
}
],
"kind": "List",
"metadata": {
"resourceVersion": ""
}
}
# AWS 관리콘솔에서 NLB 정보 확인
# 빠른 실습을 위해서 등록 취소 지연(드레이닝 간격) 수정 : 기본값 300초
echo-service-nlb.yaml 파일 IDE(VS code)에서 수정
..
apiVersion: v1
kind: Service
metadata:
name: svc-nlb-ip-type
annotations:
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8080"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=60
...
kubectl apply -f echo-service-nlb.yaml
# AWS ELB(NLB) 정보 확인
aws elbv2 describe-load-balancers | jq
aws elbv2 describe-load-balancers --query 'LoadBalancers[*].State.Code' --output text
ALB_ARN=$(aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-default-svcnlbip`) == `true`].LoadBalancerArn' | jq -r '.[0]')
aws elbv2 describe-target-groups --load-balancer-arn $ALB_ARN | jq
TARGET_GROUP_ARN=$(aws elbv2 describe-target-groups --load-balancer-arn $ALB_ARN | jq -r '.TargetGroups[0].TargetGroupArn')
aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN | jq
{
"TargetHealthDescriptions": [
{
"Target": {
"Id": "192.168.2.27",
"Port": 8080,
"AvailabilityZone": "ap-northeast-2b"
},
"HealthCheckPort": "8080",
"TargetHealth": {
"State": "healthy"
},
"AdministrativeOverride": {
"State": "no_override",
"Reason": "AdministrativeOverride.NoOverride",
"Description": "No override is currently active on target"
}
},
...
# 웹 접속 주소 확인
kubectl get svc svc-nlb-ip-type -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "Pod Web URL = http://"$1 }'
# 파드 로깅 모니터링
kubectl logs -l app=deploy-websrv -f
192.168.1.97 - - [15/Feb/2025:13:49:48 +0000] "GET / HTTP/1.1" 200 1119 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
192.168.1.97 - - [15/Feb/2025:13:49:48 +0000] "GET /favicon.ico HTTP/1.1" 200 1110 "http://k8s-default-svcnlbip-b9f4a7d045-df3b0a43de1e7e78.elb.ap-northeast-2.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
kubectl stern -l app=deploy-websrv
+ deploy-echo-bf9bdb8bc-2tqls › aews-websrv
+ deploy-echo-bf9bdb8bc-b67dt › aews-websrv
deploy-echo-bf9bdb8bc-b67dt aews-websrv 192.168.1.97 - - [15/Feb/2025:13:49:48 +0000] "GET / HTTP/1.1" 200 1119 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
deploy-echo-bf9bdb8bc-b67dt aews-websrv 192.168.1.97 - - [15/Feb/2025:13:49:48 +0000] "GET /favicon.ico HTTP/1.1" 200 1110 "http://k8s-default-svcnlbip-b9f4a7d045-df3b0a43de1e7e78.elb.ap-northeast-2.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
# 분산 접속 확인
NLB=$(kubectl get svc svc-nlb-ip-type -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl -s $NLB
for i in {1..100}; do curl -s $NLB | grep Hostname ; done | sort | uniq -c | sort -nr
61 Hostname: deploy-echo-bf9bdb8bc-b67dt
39 Hostname: deploy-echo-bf9bdb8bc-2tqls
# 지속적인 접속 시도 : 아래 상세 동작 확인 시 유용(패킷 덤프 등)
while true; do curl -s --connect-timeout 1 $NLB | egrep 'Hostname|client_address'; echo "----------" ; date "+%Y-%m-%d %H:%M:%S" ; sleep 1; done- AWS NLB의 대상 그룹 확인 : IP를 확인해보자
- 파드 2개 → 1개 → 3개 설정 시 동작 : auto discovery ← 어떻게 가능할까?
# (신규 터미널) 모니터링
while true; do aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN --output text; echo; done
# 작업용 EC2 - 파드 1개 설정
kubectl scale deployment deploy-echo --replicas=1
# 확인
kubectl get deploy,pod,svc,ep
NLB=$(kubectl get svc svc-nlb-ip-type -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl -s $NLB
for i in {1..100}; do curl -s --connect-timeout 1 $NLB | grep Hostname ; done | sort | uniq -c | sort -nr
# 파드 3개 설정
kubectl scale deployment deploy-echo --replicas=3
# 확인 : NLB 대상 타켓이 아직 initial 일 때 100번 반복 접속 시 어떻게 되는지 확인해보자!
kubectl get deploy,pod,svc,ep
NLB=$(kubectl get svc svc-nlb-ip-type -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl -s $NLB
for i in {1..100}; do curl -s --connect-timeout 1 $NLB | grep Hostname ; done | sort | uniq -c | sort -nr
#
kubectl describe deploy -n kube-system aws-load-balancer-controller | grep -i 'Service Account'
Service Account: aws-load-balancer-controller
# [AWS LB Ctrl] 클러스터 롤 바인딩 정보 확인
kubectl describe clusterrolebindings.rbac.authorization.k8s.io aws-load-balancer-controller-rolebinding
Name: aws-load-balancer-controller-rolebinding
Labels: app.kubernetes.io/instance=aws-load-balancer-controller
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=aws-load-balancer-controller
app.kubernetes.io/version=v2.11.0
helm.sh/chart=aws-load-balancer-controller-1.11.0
Annotations: meta.helm.sh/release-name: aws-load-balancer-controller
meta.helm.sh/release-namespace: kube-system
Role:
Kind: ClusterRole
Name: aws-load-balancer-controller-role
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount aws-load-balancer-controller kube-system
# [AWS LB Ctrl] 클러스터롤 확인
kubectl describe clusterroles.rbac.authorization.k8s.io aws-load-balancer-controller-role- 실습 리소스 삭제
kubectl delete deploy deploy-echo; kubectl delete svc svc-nlb-ip-type
deployment.apps "deploy-echo" deleted
service "svc-nlb-ip-type" deleted
7. Ingress
7.1 Ingress 소개
인그레스 소개 : 클러스터 내부의 서비스(ClusterIP, NodePort, Loadbalancer)를 외부로 노출(HTTP/HTTPS) - Web Proxy 역할
- AWS Load Balancer Controller + Ingress (ALB) IP 모드 동작 with AWS VPC CNI
7.2 서비스/파드 배포 테스트 with Ingress(ALB) - ALB
# 게임 파드와 Service, Ingress 배포
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
name: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: game-2048
name: deployment-2048
spec:
selector:
matchLabels:
app.kubernetes.io/name: app-2048
replicas: 2
template:
metadata:
labels:
app.kubernetes.io/name: app-2048
spec:
containers:
- image: public.ecr.aws/l6m2t8p7/docker-2048:latest
imagePullPolicy: Always
name: app-2048
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
namespace: game-2048
name: service-2048
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
type: NodePort
selector:
app.kubernetes.io/name: app-2048
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: game-2048
name: ingress-2048
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service-2048
port:
number: 80
EOF
namespace/game-2048 created
deployment.apps/deployment-2048 created
service/service-2048 created
ingress.networking.k8s.io/ingress-2048 created
# 모니터링
watch -d kubectl get pod,ingress,svc,ep,endpointslices -n game-2048
# 생성 확인
kubectl get ingress,svc,ep,pod -n game-2048
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/ingress-2048 alb * k8s-game2048-ingress2-f220b97822-1069833855.ap-northeast-2.elb.amazonaws.com 80 48s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/service-2048 NodePort 10.100.21.63 <none> 80:30847/TCP 48s
NAME ENDPOINTS AGE
endpoints/service-2048 192.168.2.84:80,192.168.3.82:80 48s
NAME READY STATUS RESTARTS AGE
pod/deployment-2048-7df5f9886b-8vk62 1/1 Running 0 48s
pod/deployment-2048-7df5f9886b-fhkq2 1/1 Running 0 48s
kubectl get-all -n game-2048
NAME NAMESPACE AGE
configmap/kube-root-ca.crt game-2048 82s
endpoints/service-2048 game-2048 82s
pod/deployment-2048-7df5f9886b-8vk62 game-2048 82s
pod/deployment-2048-7df5f9886b-fhkq2 game-2048 82s
serviceaccount/default game-2048 82s
service/service-2048 game-2048 82s
deployment.apps/deployment-2048 game-2048 82s
replicaset.apps/deployment-2048-7df5f9886b game-2048 82s
endpointslice.discovery.k8s.io/service-2048-rrxvd game-2048 82s
targetgroupbinding.elbv2.k8s.aws/k8s-game2048-service2-993b066073 game-2048 78s
ingress.networking.k8s.io/ingress-2048 game-2048 82s
kubectl get targetgroupbindings -n game-2048
NAME SERVICE-NAME SERVICE-PORT TARGET-TYPE AGE
k8s-game2048-service2-993b066073 service-2048 80 ip 95s
# ALB 생성 확인
aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-game2048`) == `true`]' | jq
ALB_ARN=$(aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-game2048`) == `true`].LoadBalancerArn' | jq -r '.[0]')
aws elbv2 describe-target-groups --load-balancer-arn $ALB_ARN
TARGET_GROUP_ARN=$(aws elbv2 describe-target-groups --load-balancer-arn $ALB_ARN | jq -r '.TargetGroups[0].TargetGroupArn')
aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN | jq
# Ingress 확인
kubectl describe ingress -n game-2048 ingress-2048
kubectl get ingress -n game-2048 ingress-2048 -o jsonpath="{.status.loadBalancer.ingress[*].hostname}{'\n'}"
Name: ingress-2048
Labels: <none>
Namespace: game-2048
Address: k8s-game2048-ingress2-f220b97822-1069833855.ap-northeast-2.elb.amazonaws.com
Ingress Class: alb
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/ service-2048:80 (192.168.2.84:80,192.168.3.82:80)
Annotations: alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfullyReconciled 2m29s ingress Successfully reconciled
# 게임 접속 : ALB 주소로 웹 접속
kubectl get ingress -n game-2048 ingress-2048 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "Game URL = http://"$1 }'
Game URL = http://k8s-game2048-ingress2-f220b97822-1069833855.ap-northeast-2.elb.amazonaws.com
# 파드 IP 확인
kubectl get pod -n game-2048 -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-2048-7df5f9886b-8vk62 1/1 Running 0 3m43s 192.168.2.84 ip-192-168-2-252.ap-northeast-2.compute.internal <none> <none>
deployment-2048-7df5f9886b-fhkq2 1/1 Running 0 3m43s 192.168.3.82 ip-192-168-3-206.ap-northeast-2.compute.internal <none> <none>
- 실습 리소스 삭제
kubectl delete ingress ingress-2048 -n game-2048
kubectl delete svc service-2048 -n game-2048 && kubectl delete deploy deployment-2048 -n game-2048 && kubectl delete ns game-20487.3 Target Groups CRD를 분리 사용하여, EKS 업그레이드에 활용하기
- Ingress를 제거하고 별도로 ALB를 구성 후 파드를 TargetGroupBinding으로 직접 연결
- ALB가 Kubernetes와 독립적으로 구성되니 Route 53이나 CloudFront나 WAF에 대한 조정이 필요 없음!!!
- Ingress 구성이 아닌 TargetGroupBinding으로 생성된 ALB로 지정해서 서비스
- 두개의 클러스터의 자원 모두 기존에 생성한 ALB를 활용
- Terraform을 통해 ALB 관련 자원을 생성하고 제어
- ALB의 Target Group을 클러스터 별로 분리해서 사용
- ALB Listener에 2개의 Target Group을 등록하고 Weight를 조정해서 사용
- (일반적인 서비스) v1 TG 50% : v2 TG 50%
- (v1 클러스터 업그레이드 전) v1 TG 0% : v2 TG 100% ⇒ v1 클러스터 업그레이드 진행
- (v2 클러스터 업그레이드 전) v1 TG 100% : v2 TG 0% ⇒ v2 클러스터 업그레이드 진행
- (두 클러스터 모두 업그레이드 완료) v1 TG 50% : v2 TG 50%
- ALB Listener에 2개의 Target Group을 등록하고 Weight를 조정해서 사용
- 두개의 클러스터의 자원 모두 기존에 생성한 ALB를 활용
8. ExternalDNS
소개 : K8S 서비스/인그레스 생성 시 도메인을 설정하면, AWS(Route 53), Azure(DNS), GCP(Cloud DNS) 에 A 레코드(TXT 레코드)로 자동 생성/삭제
8.1 소개
- ExternalDNS CTRL 권한 주는 방법 3가지 : Node IAM Role, Static credentials, IRSA
8.2 AWS Route 53 정보 확인 & 변수 지정
- Public 도메인 소유를 하고 계셔야 합니다!
# 자신의 도메인 변수 지정 : 소유하고 있는 자신의 도메인을 입력하시면 됩니다
MyDomain=<자신의 도메인>
MyDomain=ksj7279.click
# 자신의 Route 53 도메인 ID 조회 및 변수 지정
aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." | jq
aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Name"
aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --output text
MyDnzHostedZoneId=`aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --output text`
echo $MyDnzHostedZoneId
# (옵션) NS 레코드 타입 첫번째 조회
aws route53 list-resource-record-sets --output json --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'NS']" | jq -r '.[0].ResourceRecords[].Value'
# (옵션) A 레코드 타입 모두 조회
aws route53 list-resource-record-sets --output json --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A']"
# A 레코드 타입 조회
aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A']" | jq
aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A'].Name" | jq
aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A'].Name" --output text
# A 레코드 값 반복 조회
while true; do aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A']" | jq ; date ; echo ; sleep 1; done8.3 ExternalDNS 설치 - 링크
# EKS 배포 시 Node IAM Role 설정되어 있음
# eksctl create cluster ... --external-dns-access ...
#
MyDomain=<자신의 도메인>
MyDomain=ksj7279.click
# 자신의 Route 53 도메인 ID 조회 및 변수 지정
MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "${MyDomain}." --query "HostedZones[0].Id" --output text)
# 변수 확인
echo $MyDomain, $MyDnzHostedZoneId
# ExternalDNS 배포
curl -s -O https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml
cat externaldns.yaml
MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst < externaldns.yaml | kubectl apply -f -
# 확인 및 로그 모니터링
kubectl get pod -l app.kubernetes.io/name=external-dns -n kube-system
kubectl logs deploy/external-dns -n kube-system -f- (참고) 기존에 ExternalDNS를 통해 사용한 A/TXT 레코드가 있는 존의 경우에 policy 정책을 upsert-only 로 설정 후 사용 하자 - Link
- #--policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization8.4 Service(NLB) + 도메인 연동(ExternalDNS) - 도메인체크
# 터미널1 (모니터링)
watch -d 'kubectl get pod,svc'
kubectl logs deploy/external-dns -n kube-system -f
혹은
kubectl stern -l app.kubernetes.io/name=external-dns -n kube-system
# 테트리스 디플로이먼트 배포
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: tetris
labels:
app: tetris
spec:
replicas: 1
selector:
matchLabels:
app: tetris
template:
metadata:
labels:
app: tetris
spec:
containers:
- name: tetris
image: bsord/tetris
---
apiVersion: v1
kind: Service
metadata:
name: tetris
annotations:
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
#service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "80"
spec:
selector:
app: tetris
ports:
- port: 80
protocol: TCP
targetPort: 80
type: LoadBalancer
loadBalancerClass: service.k8s.aws/nlb
EOF
# 배포 확인
kubectl get deploy,svc,ep tetris
# NLB에 ExternanDNS 로 도메인 연결
kubectl annotate service tetris "external-dns.alpha.kubernetes.io/hostname=tetris.$MyDomain"
while true; do aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A']" | jq ; date ; echo ; sleep 1; done
# Route53에 A레코드 확인
aws route53 list-resource-record-sets --hosted-zone-id "${MyDnzHostedZoneId}" --query "ResourceRecordSets[?Type == 'A']" | jq
# 확인
dig +short tetris.$MyDomain @8.8.8.8
dig +short tetris.$MyDomain
# 도메인 체크
echo -e "My Domain Checker Site1 = https://www.whatsmydns.net/#A/tetris.$MyDomain"
echo -e "My Domain Checker Site2 = https://dnschecker.org/#A/tetris.$MyDomain"
# 웹 접속 주소 확인 및 접속
echo -e "Tetris Game URL = http://tetris.$MyDomain"9. CoreDNS
- Kubernetes에서 DNS 다루는 방법 - 도메인을 찾아서
- Stop Leaking Kubernetes Service Information via DNS! - John Belamaric, Google & Yong Tang, Ivanti
- 쿠버네티스 DNS 쿼리 Flow - 링크
- Coredns config, 플러그인과 NodeLocal DNS 동작 정리 https://popappend.tistory.com/142
- The following diagram shows application response time with and without DNS caching:
10. Topology Aware Routing
- Topology Aware Routing: Understanding the Tradeoffs - Rob Scott, Google
- 본인이 작성한 Amazon EKS에서 Topology Aware Hint 기능을 활용하여 Cross-AZ 통신 비용 절감 자료 참고해 주세요~
11. Using AWS LoadBalancer Controller for blue/green deployment, canary deployment and A/B testing
11.1 ALB 동작 소개
- Weighted target group 가중치가 적용된 대상 그룹
- AWS 고객이 블루/그린 및 카나리아 배포와 A/B 테스트 전략을 채택할 수 있도록 돕기 위해 AWS는 2019년 11월에 애플리케이션 로드 밸런서에 대한 가중 대상 그룹을 발표했습니다. 여러 대상 그룹을 리스너 규칙 의 동일한 전달 작업 에 연결 하고 각 그룹에 대한 가중치를 지정할 수 있습니다.
- 이를 통해 개발자는 트래픽을 여러 버전의 애플리케이션에 분산하는 방법을 제어할 수 있습니다. 예를 들어, 가중치가 8과 2인 두 개의 대상 그룹이 있는 규칙을 정의하면 로드 밸런서는 트래픽의 80%를 첫 번째 대상 그룹으로, 20%를 다른 대상 그룹으로 라우팅합니다.
- Advanced request routing 고급 요청 라우팅
- AWS는 가중치가 적용된 대상 그룹 외에도 2019년에 고급 요청 라우팅 기능을 발표했습니다 . 고급 요청 라우팅은 개발자에게 표준 및 사용자 지정 HTTP 헤더와 메서드, 요청 경로, 쿼리 문자열, 소스 IP 주소를 기반으로 규칙을 작성하고 트래픽을 라우팅할 수 있는 기능을 제공합니다.
- 이 새로운 기능은 라우팅을 위한 프록시 플릿의 필요성을 없애 애플리케이션 아키텍처를 간소화하고, 로드 밸런서에서 원치 않는 트래픽을 차단하며, A/B 테스트를 구현할 수 있도록 합니다.
- AWS Load Balancer Controller AWS 로드 밸런서 컨트롤러
- AWS Load Balancer Controller 는 Kubernetes 클러스터의 Elastic Load Balancer를 관리하는 데 도움이 되는 컨트롤러입니다. 애플리케이션 로드 밸런서를 프로비저닝하여 Kubernetes 인그레스 리소스를 충족합니다.
- Kubernetes 인그레스 객체에 주석을 추가하여 프로비저닝된 애플리케이션 로드 밸런서의 동작을 사용자 지정할 수 있습니다. 이를 통해 개발자는 애플리케이션 로드 밸런서를 구성하고 Kubernetes 기본 의미 체계를 사용하여 블루/그린, 카나리아 및 A/B 배포를 실현할 수 있습니다.
- 예를 들어, 다음 인그레스 주석은 애플리케이션 로드 밸런서를 구성하여 두 버전의 애플리케이션 간에 트래픽을 분할합니다.
annotations:
...
alb.ingress.kubernetes.io/actions.blue-green: |
{
"type":"forward",
"forwardConfig":{
"targetGroups":[
{
"serviceName":"hello-kubernetes-v1",
"servicePort":"80",
"weight":50
},
{
"serviceName":"hello-kubernetes-v2",
"servicePort":"80",
"weight":50
}
]
}
}
11.2 Deploy the sample application version 1 and version 2
- Deploy the sample application version 1 and version 2
- The sample application used here is hello-kubernetes. Deploy two versions of the applications with custom messages and set the service type to
ClusterIP:
- The sample application used here is hello-kubernetes. Deploy two versions of the applications with custom messages and set the service type to
- Deploy the sample application version 1 and version 2
#
git clone https://github.com/paulbouwer/hello-kubernetes.git
tree hello-kubernetes/
# Install sample application version 1
helm install --create-namespace --namespace hello-kubernetes v1 \
./hello-kubernetes/deploy/helm/hello-kubernetes \
--set message="You are reaching hello-kubernetes version 1" \
--set ingress.configured=true \
--set service.type="ClusterIP"
# Install sample application version 2
helm install --create-namespace --namespace hello-kubernetes v2 \
./hello-kubernetes/deploy/helm/hello-kubernetes \
--set message="You are reaching hello-kubernetes version 2" \
--set ingress.configured=true \
--set service.type="ClusterIP"
# 확인
kubectl get-all -n hello-kubernetes
kubectl get pod,svc,ep -n hello-kubernetes
kubectl get pod -n hello-kubernetes --label-columns=app.kubernetes.io/instance,pod-template-hash11.3 Deploy ingress and test the blue/green deployment
- Ingress annotation alb.ingress.kubernetes.io/actions.${action-name} provides a method for configuring custom actions on the listener of an Application Load Balancer, such as redirect action, forward action. With forward action, multiple target groups with different weights can be defined in the annotation. AWS Load Balancer Controller provisions the target groups and configures the listener rules as per the annotation to direct the traffic. For example, the following ingress resource configures the Application Load Balancer to forward all traffic to hello-kubernetes-v1 service (weight: 100 vs. 0)
- Note, the action-name in the annotation must match the serviceName in the ingress rules, and servicePort must be use-annotation as in the previous code snippet.
#
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "hello-kubernetes"
namespace: "hello-kubernetes"
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/actions.blue-green: |
{
"type":"forward",
"forwardConfig":{
"targetGroups":[
{
"serviceName":"hello-kubernetes-v1",
"servicePort":"80",
"weight":100
},
{
"serviceName":"hello-kubernetes-v2",
"servicePort":"80",
"weight":0
}
]
}
}
labels:
app: hello-kubernetes
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: blue-green
port:
name: use-annotation
EOF
# 확인
kubectl get ingress -n hello-kubernetes
kubectl describe ingress -n hello-kubernetes
...
Rules:
Host Path Backends
---- ---- --------
*
/ blue-green:use-annotation (<error: endpoints "blue-green" not found>)
Annotations: alb.ingress.kubernetes.io/actions.blue-green:
{
"type":"forward",
"forwardConfig":{
"targetGroups":[
{
"serviceName":"hello-kubernetes-v1",
"servicePort":"80",
"weight":100
},
{
"serviceName":"hello-kubernetes-v2",
"servicePort":"80",
"weight":0
...
# 반복 접속 확인
ELB_URL=$(kubectl get ingress -n hello-kubernetes -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
while true; do curl -s $ELB_URL | grep version; sleep 1; done
You are reaching hello-kubernetes version 1
You are reaching hello-kubernetes version 1
...- ALB Listener rules 에서 2개의 Target group 확인(weight)
11.4 Blue/green deployment
- To perform the blue/green deployment, update the ingress annotation to move all weight to version 2:
#
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "hello-kubernetes"
namespace: "hello-kubernetes"
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/actions.blue-green: |
{
"type":"forward",
"forwardConfig":{
"targetGroups":[
{
"serviceName":"hello-kubernetes-v1",
"servicePort":"80",
"weight":0
},
{
"serviceName":"hello-kubernetes-v2",
"servicePort":"80",
"weight":100
}
]
}
}
labels:
app: hello-kubernetes
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: blue-green
port:
name: use-annotation
EOF
# 확인
kubectl describe ingress -n hello-kubernetes
# 반복 접속 확인 : 적용에 약간의 시간 소요
ELB_URL=$(kubectl get ingress -n hello-kubernetes -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
while true; do curl -s $ELB_URL | grep version; sleep 1; done
You are reaching hello-kubernetes version 2
You are reaching hello-kubernetes version 2
...11.5 Deploy Ingress and test the canary deployment
- Instead of moving all traffic to version 2, we can shift the traffic slowly towards version 2 by increasing the weight on version 2 step by step. This allows version 2 to be verified against a small portion of the production traffic before moving more traffic over. The following example shows that 10 percent of the traffic is shifted to version 2, while 90 percent of the traffic remains with version 1.
#
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "hello-kubernetes"
namespace: "hello-kubernetes"
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/actions.blue-green: |
{
"type":"forward",
"forwardConfig":{
"targetGroups":[
{
"serviceName":"hello-kubernetes-v1",
"servicePort":"80",
"weight":90
},
{
"serviceName":"hello-kubernetes-v2",
"servicePort":"80",
"weight":10
}
]
}
}
labels:
app: hello-kubernetes
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: blue-green
port:
name: use-annotation
EOF
# 확인
kubectl describe ingress -n hello-kubernetes
# 반복 접속 확인 : 적용에 약간의 시간 소요
ELB_URL=$(kubectl get ingress -n hello-kubernetes -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
while true; do curl -s $ELB_URL | grep version; sleep 1; done
# 100번 접속
for i in {1..100}; do curl -s $ELB_URL | grep version ; done | sort | uniq -c | sort -nr11.6 Argo Rollouts
- When performing a canary deployment in a production environment, typically the traffic is shifted with small increments. Usually it is done with some level of automation behind it. Various performance monitoring systems can also be integrated into this process, making sure that every step of the way there are no errors, or the errors are below an acceptable threshold. This is where progressive delivery mechanisms such as Argo Rollouts are very beneficial.When performing a canary deployment in a production environment, typically the traffic is shifted with small increments. Usually it is done with some level of automation behind it. Various performance monitoring systems can also be integrated into this process, making sure that every step of the way there are no errors, or the errors are below an acceptable threshold. This is where progressive delivery mechanisms such as Argo Rollouts are very beneficial.
- Argo Rollouts offers first class support for using the annotation-based traffic shaping abilities of AWS Load Balancer Controller to gradually shift traffic to the new version during an update. Additionally, Argo Rollouts can query and interpret metrics from various providers to verify key KPIs and drive automated promotion or rollback during an update. More information is available at Argo Rollouts integration with Application Load Balancer.
11.7 Deploy ingress and test the A/B testing
- Ingress annotation alb.ingress.kubernetes.io/conditions.${conditions-name} provides a method for specifying routing conditions in addition to original host/path condition on ingress spec. The additional routing conditions can be based on http-header, http-request-method, query-string and source-ip. This provides developers multiple advanced routing options for their A/B testing implementation, without the need for setting up and managing a separate routing system, such as service mesh.
- AWS Load Balancer Controller configures the listener rules as per the annotation to direct a portion of incoming traffic to a specific backend. In the following example, all requests are directed to version 1 by default. The following ingress resource directs the traffic to version 2 when the request contains a custom HTTP header: HeaderName=HeaderValue1.
#
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "hello-kubernetes"
namespace: "hello-kubernetes"
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/conditions.ab-testing: >
[{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "HeaderName", "values":["aews-study"]}}]
alb.ingress.kubernetes.io/actions.ab-testing: >
{"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"hello-kubernetes-v2","servicePort":80}]}}
labels:
app: hello-kubernetes
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ab-testing
port:
name: use-annotation
- path: /
pathType: Prefix
backend:
service:
name: hello-kubernetes-v1
port:
name: http
EOF
# 확인
kubectl describe ingress -n hello-kubernetes
# 반복 접속 확인 : 적용에 약간의 시간 소요
ELB_URL=$(kubectl get ingress -n hello-kubernetes -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
while true; do curl -s $ELB_URL | grep version; sleep 1; done
...
while true; do curl -s -H "HeaderName: aews-study" $ELB_URL | grep version; sleep 1; done
...
# 100번 접속
for i in {1..100}; do curl -s $ELB_URL | grep version ; done | sort | uniq -c | sort -nr
for i in {1..100}; do curl -s -H "HeaderName: aews-study" $ELB_URL | grep version ; done | sort | uniq -c | sort -nr
- 자원삭제
kubectl delete ingress -n hello-kubernetes hello-kubernetes && kubectl delete ns hello-kubernetes12. Network Policies with VPC CNI
12.1 소개
- AWS EKS fully supports the upstream Kubernetes Network Policy API, ensuring compatibility and adherence to Kubernetes standards.
동작 : eBPF로 패킷 필터링 동작 - Network Policy Controller, Node Agent, eBPF SDK
- 사전 조건 : EKS 1.25 버전 이상, AWS VPC CNI 1.14 이상, OS 커널 5.10 이상 EKS 최적화 AMI(AL2, Bottlerocket, Ubuntu)
- Network Policy Controller : v1.25 EKS 버전 이상 자동 설치, 통제 정책 모니터링 후 eBPF 프로그램을 생성 및 업데이트하도록 Node Agent에 지시
- Node Agent : AWS VPC CNI 번들로 ipamd 플러그인과 함께 설치됨(aws-node 데몬셋). eBPF 프래그램을 관리
- eBPF SDK : AWS VPC CNI에는 노드에서 eBPF 프로그램과 상호 작용할 수 있는 SDK 포함, eBPF 실행의 런타임 검사, 추적 및 분석 가능
→ AWS SG for Pods 와 함께 사용하면 더욱 좋습니다!
12.2 사전 준비 및 기본 정보 확인
# Network Policy 기본 비활성화되어 있어, 활성화 필요 : 실습 환경은 미리 활성화 설정 추가되어 있음
tail -n 11 myeks.yaml
addons:
- name: vpc-cni # no version is specified so it deploys the default version
version: latest # auto discovers the latest available
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
configurationValues: |-
enableNetworkPolicy: "true"
# Node Agent 확인 : AWS VPC CNI 1.14 이상 버전 정보 확인
kubectl get ds aws-node -n kube-system -o yaml | k neat
...
- args:
- --enable-ipv6=false
- --enable-network-policy=true
...
volumeMounts:
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
- mountPath: /sys/fs/bpf
name: bpf-pin-path
- mountPath: /var/log/aws-routed-eni
name: log-dir
- mountPath: /var/run/aws-node
name: run-dir
...
kubectl get ds aws-node -n kube-system -o yaml | grep -i image:
kubectl get pod -n kube-system -l k8s-app=aws-node
kubectl get ds -n kube-system aws-node -o jsonpath='{.spec.template.spec.containers[*].name}{"\n"}'
aws-node aws-eks-nodeagent
# EKS 1.25 버전 이상 확인
kubectl get nod
# OS 커널 5.10 이상 확인
ssh ec2-user@$N1 uname -r
5.10.210-201.852.amzn2.x86_64
# 실행 중인 eBPF 프로그램 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo /opt/cni/bin/aws-eks-na-cli ebpf progs; echo; done
...
Programs currently loaded :
Type : 26 ID : 6 Associated maps count : 1
========================================================================================
Type : 26 ID : 8 Associated maps count : 1
========================================================================================
# 각 노드에 BPF 파일 시스템을 탑재 확인
ssh ec2-user@$N1 mount | grep -i bpf
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
ssh ec2-user@$N1 df -a | grep -i bpf
none 0 0 0 - /sys/fs/bpf12.3 샘플 애플리케이션 배포 및 네트워크 정책 적용 실습 - Link
#
git clone https://github.com/aws-samples/eks-network-policy-examples.git
cd eks-network-policy-examples
tree advanced/manifests/
kubectl apply -f advanced/manifests/
# 확인
kubectl get pod,svc
kubectl get pod,svc -n another-ns
# 통신 확인
kubectl exec -it client-one -- curl demo-app
kubectl exec -it client-two -- curl demo-app
kubectl exec -it another-client-one -n another-ns -- curl demo-app
kubectl exec -it another-client-one -n another-ns -- curl demo-app.default
kubectl exec -it another-client-two -n another-ns -- curl demo-app.default.svc- 모든 트래픽 거부
# 모니터링
# kubectl exec -it client-one -- curl demo-app
while true; do kubectl exec -it client-one -- curl --connect-timeout 1 demo-app ; date; sleep 1; done
# 정책 적용
cat advanced/policies/01-deny-all-ingress.yaml
kubectl apply -f advanced/policies/01-deny-all-ingress.yaml
kubectl get networkpolicy
# 실행 중인 eBPF 프로그램 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo /opt/cni/bin/aws-eks-na-cli ebpf progs; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo /opt/cni/bin/aws-eks-na-cli ebpf loaded-ebpfdata; echo; done
...
>> node 192.168.3.201 <<
PinPath: /sys/fs/bpf/globals/aws/programs/demo-app-6fd76f694b-default_handle_ingress
Pod Identifier : demo-app-6fd76f694b-default Direction : ingress
Prog ID: 9
Associated Maps ->
Map Name: ingress_map
Map ID: 7
Map Name: policy_events
Map ID: 6
Map Name: aws_conntrack_map
Map ID: 5
========================================================================================
PinPath: /sys/fs/bpf/globals/aws/programs/demo-app-6fd76f694b-default_handle_egress
Pod Identifier : demo-app-6fd76f694b-default Direction : egress
Prog ID: 10
Associated Maps ->
Map Name: aws_conntrack_map
Map ID: 5
Map Name: egress_map
Map ID: 8
Map Name: policy_events
Map ID: 6
========================================================================================
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 5
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 9
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 10
# 정책 다시 삭제
kubectl delete -f advanced/policies/01-deny-all-ingress.yaml
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo /opt/cni/bin/aws-eks-na-cli ebpf loaded-ebpfdata; echo; done
# 다시 적용
kubectl apply -f advanced/policies/01-deny-all-ingress.yaml- 동일 네임스페이스 + 클라이언트1 로부터의 수신 허용
#
cat advanced/policies/03-allow-ingress-from-samens-client-one.yaml
kubectl apply -f advanced/policies/03-allow-ingress-from-samens-client-one.yaml
kubectl get networkpolicy
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo /opt/cni/bin/aws-eks-na-cli ebpf loaded-ebpfdata; echo; done
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 5
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 9
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 10
# 클라이언트2 수신 확인
kubectl exec -it client-two -- curl --connect-timeout 1 demo-app- another-ns 네임스페이스로부터의 수신 허용
# 모니터링
# kubectl exec -it another-client-one -n another-ns -- curl --connect-timeout 1 demo-app.default
while true; do kubectl exec -it another-client-one -n another-ns -- curl --connect-timeout 1 demo-app.default ; date; sleep 1; done
#
cat advanced/policies/04-allow-ingress-from-xns.yaml
kubectl apply -f advanced/policies/04-allow-ingress-from-xns.yaml
kubectl get networkpolicy
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo /opt/cni/bin/aws-eks-na-cli ebpf loaded-ebpfdata; echo; done
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 5
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 9
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 10
#
kubectl exec -it another-client-two -n another-ns -- curl --connect-timeout 1 demo-app.default- eBPF 관련 정보 확인
# 실행 중인 eBPF 프로그램 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo /opt/cni/bin/aws-eks-na-cli ebpf progs; echo; done
# eBPF 로그 확인
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo cat /var/log/aws-routed-eni/ebpf-sdk.log; echo; done
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo cat /var/log/aws-routed-eni/network-policy-agent; echo; done- 송신 트래픽 거부 : 기본 네임스페이스의 클라이언트-1 포드에서 모든 송신 격리를 적용
# 모니터링
while true; do kubectl exec -it client-one -- curl --connect-timeout 1 google.com ; date; sleep 1; done
#
cat advanced/policies/06-deny-egress-from-client-one.yaml
kubectl apply -f advanced/policies/06-deny-egress-from-client-one.yaml
kubectl get networkpolicy
for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh ec2-user@$i sudo /opt/cni/bin/aws-eks-na-cli ebpf loaded-ebpfdata; echo; done
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 5
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 9
ssh ec2-user@$N3 sudo /opt/cni/bin/aws-eks-na-cli ebpf dump-maps 10
#
kubectl exec -it client-one -- nslookup demo-app- 송신 트래픽 허용 : DNS 트래픽을 포함하여 여러 포트 및 네임스페이스에서의 송신을 허용
# 모니터링
while true; do kubectl exec -it client-one -- curl --connect-timeout 1 demo-app ; date; sleep 1; done
#
cat advanced/policies/08-allow-egress-to-demo-app.yaml | yh
kubectl apply -f advanced/policies/08-allow-egress-to-demo-app.yaml
kubectl get networkpolicy- 실습 후 리소스 삭제
kubectl delete networkpolicy --all
kubectl delete -f advanced/manifests/13. 실습 후 자원 삭제
- Amazon EKS 클러스터 삭제(10분 정도 소요):
eksctl delete cluster --name $CLUSTER_NAME - (클러스터 삭제 완료 확인 후) AWS CloudFormation 스택 삭제 :
aws cloudformation delete-stack --stack-name myeks-sejkim
I'm SJ