본 게시물은 CloudNet@팀 Gasida(서종호) 님이 진행하시는
AWS EKS Workshop Study 내용을 기반으로 작성되었습니다.
k8s 인증/인가
- 서비스 어카운트(Service Account)
- Pod 또는 애플리케이션이 Kubernetes API 서버에 자신을 인증하는 데 사용할 수 있는 ID
주로 Pod 내에서 실행되는 애플리케이션이 Kubernetes API 또는 클러스터 내의 다른 리소스와 상호 작용하는 데 사용된다.
- Pod 또는 애플리케이션이 Kubernetes API 서버에 자신을 인증하는 데 사용할 수 있는 ID
- API 서버 사용 : kubectl(config, 다수 클러스터 관리 가능), 서비스 어카운트, https(x.509 Client Certs)
- API 서버 접근 과정 : 인증 → 인가 → Admission Control(API 요청 검증, 필요 시 변형 - 예. ResourceQuota, LimitRange) - 참고
인증(Authentication)
- X.509 Client Certs : kubeconfig 에 CA crt(발급 기관 인증서) , Client crt(클라이언트 인증서) , Client key(클라이언트 개인키) 를 통해 인증
- kubectl : 여러 클러스터(kubeconfig)를 관리 가능 - contexts 에 클러스터와 유저 및 인증서/키 참고
- Service Account : 기본 서비스 어카운트(default) - 시크릿(CA crt 와 token)
인가(Authorization)
- 인가 방식 : RBAC(Role, RoleBinding), ABAC, Webhook, Node Authorization
- RBAC : 역할 기반의 권한 관리, 사용자와 역할을 별개로 선언 후 두가지를 조합(binding)해서 사용자에게 권한을 부여하여 kubectl or API로 관리 가능
- Namespace/Cluster - Role/ClusterRole, RoleBinding/ClusterRoleBinding, Service Account
- Role(롤) - (RoleBinding 롤 바인딩) - Service Account(서비스 어카운트) : 롤 바인딩은 롤과 서비스 어카운트를 연결
- Role(네임스페이스내 자원의 권한) vs ClusterRole(클러스터 수준의 자원의 권한)
.kube/config
cat .kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJWUhQdFdTWkJuTll3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBME1UTXhNVEUxTlRSYUZ3MHpOREEwTVRFeE1USXdOVFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUMvS2pKdE5aV0VkTjF3QmNLWGhCYU5ldlF4SlgvbitUanNMZ0djOTRsN1l1UVVKRlFXdlU4RzZ1VmIKNDZpRjBmVWVBRmwxNlFnWHZYbGpLd3VzZXZ5ZGtJZDBmeWFQZnlQdTcrcU1vQm43UjlidDl3VWd0ZWFkUytxeApQcENQTEM0a1pHYi84a2RIcjNKUTZMbjZweFVqTVNCRmRSQlRTcHJNYWZEbk0xTEw0UjlkelZQdlNXVjlWWjdWCng4dUdCRHZQYnpIcEp6MlVaSUEvUlNpaHlnK2VYaDVXcVZLTnRRNjYzYkR6eXdjN2FWZXBJeldCYjRnTkRTNFgKQ052UmpVZEw0REFjWmFid0hyVitzVGJLL3Nqc3JyYWoyU3IzL3pTNmI2Mk1nQWFPcnFYSWJDQXgwZzFwT1VJOApyZElPb1I1cDh1Qmx5UndFMVQ0T3FoM3RwL0paQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRenRVc3M3bU9DQUF2QkRGbzg4OWxMRW5DMGhEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQXFZallzOG0vSApEU1Z2dVk2Nkh1ellPbDVWRnFyS1BxbzlGaU1jOXZsMjNFS1JjeVlaZ0NNRm9TYWkrVkdrVXB2NXhvOWtOUFF3CjI1N2RqMVE2Z1BSUVBLUTExSFJXTHNTckhhRGVOaGJ4MVorem1OK21yNkNjQS9JQW5xZXE1V1JIaU5XbVBrczUKTzdLWmo2N2wyaTNjS0YvYmthNVZJRXRMVHJLY1JCdy90eXY3K2wxMWREQW5pc1dsWE11dHVRRmgveTltYTVJagpYZWFoUkg1ZWlPTm9lbXZ0cXRkMEhEbDN6WHI4VTRsOTYwaDhUaWFNQWlyWncyejY1TkoweTUzTnZUZy81TFYxCmVBNVFxTTRKb2ptUFpUU29EaE1FU1VNR0xsWTZoTjFuN3F0ZkE3UmN1eVlwN25pdUl1UHNVbFJXdlhVUVAzbHMKNFUxZDVZdi9HK3hKCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://2D3EF5FBF387DEF75B12F3C46DFC57C0.gr7.ap-northeast-2.eks.amazonaws.com
name: myeks.ap-northeast-2.eksctl.io
contexts:
- context:
cluster: myeks.ap-northeast-2.eksctl.io
namespace: default
user: admin@myeks.ap-northeast-2.eksctl.io
name: admin@myeks.ap-northeast-2.eksctl.io
current-context: admin@myeks.ap-northeast-2.eksctl.io
kind: Config
preferences: {}
users:
- name: admin@myeks.ap-northeast-2.eksctl.io
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- eks
- get-token
- --output
- json
- --cluster-name
- myeks
- --region
- ap-northeast-2
command: aws
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
interactiveMode: IfAvailable
provideClusterInfo: false
- clusters : kubectl 이 사용할 쿠버네티스 API 서버의 접속 정보 목록. 원격의 쿠버네티스 API 서버의 주소를 추가해 사용 가능
- users : 쿠버네티스의 API 서버에 접속하기 위한 사용자 인증 정보 목록. (서비스 어카운트의 토큰, 혹은 인증서의 데이터 등)
- contexts : cluster 항목과 users 항목에 정의된 값을 조합해 최종적으로 사용할 쿠버네티스 클러스터의 정보(컨텍스트)를 설정.
- 예를 들어 clusters 항목에 클러스터 A,B 가 정의돼 있고, users 항목에 사용자 a,b 가 정의돼 있다면 cluster A + user a 를 조합해,
'cluster A 에 user a 로 인증해 쿠버네티스를 사용한다' 라는 새로운 컨텍스트를 정의할 수 있습니다.
- kubectl 을 사용하려면 여러 개의 컨텍스트 중 하나를 선택.
실습
- 쿠버네티스에 사용자를 위한 서비스 어카운트(Service Account, SA)를 생성 : dev-k8s, infra-k8s
- 사용자는 각기 다른 권한(Role, 인가)을 가짐 : dev-k8s(dev-team 네임스페이스 내 모든 동작) , infra-k8s(dev-team 네임스페이스 내 모든 동작)
- 각각 별도의 kubectl 파드를 생성하고, 해당 파드에 SA 를 지정하여 권한에 대한 테스트를 진행
1. 네임스페이스와 서비스 어카운트 생성 후 확인
# 네임스페이스(Namespace, NS) 생성 및 확인
kubectl create namespace dev-team
kubectl create ns infra-team
# 네임스페이스 확인
kubectl get ns
NAME STATUS AGE
default Active 28m
dev-team Active 6s
infra-team Active 5s
kube-node-lease Active 28m
kube-public Active 28m
kube-system Active 28m
monitoring Active 12m
# 네임스페이스에 각각 서비스 어카운트 생성 : serviceaccounts 약자(=sa)
kubectl create sa dev-k8s -n dev-team
kubectl create sa infra-k8s -n infra-team
# 서비스 어카운트 정보 확인
kubectl get sa -n dev-team
kubectl get sa dev-k8s -n dev-team -o yaml | yh
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2024-04-13T11:50:28Z"
name: dev-k8s
namespace: dev-team
resourceVersion: "8164"
uid: 8404b11f-2839-4c84-9c4b-86df1d4baa2f
kubectl get sa -n infra-team
kubectl get sa infra-k8s -n infra-team -o yaml | yh
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2024-04-13T11:50:31Z"
name: infra-k8s
namespace: infra-team
resourceVersion: "8175"
uid: 1e5d7b89-1814-4655-92c1-04f30e088ced
2. 서비스 어카운트를 지정하여 파드 생성 후 권한 테스트
# 각각 네임스피이스에 kubectl 파드 생성 - 컨테이너이미지
# docker run --rm --name kubectl -v /path/to/your/kube/config:/.kube/config bitnami/kubectl:latest
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: dev-kubectl
namespace: dev-team
spec:
serviceAccountName: dev-k8s
containers:
- name: kubectl-pod
image: bitnami/kubectl:1.28.5
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: infra-kubectl
namespace: infra-team
spec:
serviceAccountName: infra-k8s
containers:
- name: kubectl-pod
image: bitnami/kubectl:1.28.5
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
# 확인
kubectl get pod -A
kubectl get pod -o dev-kubectl -n dev-team -o yaml
serviceAccount: dev-k8s
...
kubectl get pod -o infra-kubectl -n infra-team -o yaml
serviceAccount: infra-k8s
...
# 파드에 기본 적용되는 서비스 어카운트(토큰) 정보 확인
kubectl exec -it dev-kubectl -n dev-team -- ls /run/secrets/kubernetes.io/serviceaccount
ca.crt namespace token
kubectl exec -it dev-kubectl -n dev-team -- cat /run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IjU3MmNjOTMwOTU0ZmFmY2UwZjQ1Yzc5NmViZmM1YWY5ZTgxM2VmZTQifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTc0NDU0NTM1NSwiaWF0IjoxNzEzMDA5MzU1LCJpc3MiOiJodHRwczovL29pZGMuZWtzLmFwLW5vcnRoZWFzdC0yLmFtYXpvbmF3cy5jb20vaWQvMkQzRUY1RkJGMzg3REVGNzVCMTJGM0M0NkRGQzU3QzAiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRldi10ZWFtIiwicG9kIjp7Im5hbWUiOiJkZXYta3ViZWN0bCIsInVpZCI6IjliNzdiYzU5LWQ4ZTMtNGViMC04ZDIyLTIwZTAyNjg4M2Q0MCJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZGV2LWs4cyIsInVpZCI6Ijg0MDRiMTFmLTI4MzktNGM4NC05YzRiLTg2ZGYxZDRiYWEyZiJ9LCJ3YXJuYWZ0ZXIiOjE3MTMwMTI5NjJ9LCJuYmYiOjE3MTMwMDkzNTUsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZXYtdGVhbTpkZXYtazhzIn0.GVcm-wB4XIhP8T2WCL4cyL8CMxGAplCwlZWayCs08fLLy2Q_pA83LJRQ2v7gkDY0ku74Se5L0kh7oxOtZpWE-qKxkiCC6eRobLvkcwkYGmQchHgjiltrw1DXvLXIhrHwptCuReNlyHRIUBjJAgch8snS5piEZ5EkJkl-VBJJR9DZF6J3J2p2hGUTlQQsMBRClKfCDN14VhYnLqWeH2XbeU4MR6B3PSdhHNgTl-GbqSpHcKZMwXX8jLwEfUCsN10LKD-rVrA3nmmdyuH4OK47AoQb9Ei_1dL08m67GOZzy6okJxb1rFSccB-wizvfXPPf8qdPEAC93xqxnPuApT1OwQ
kubectl exec -it dev-kubectl -n dev-team -- cat /run/secrets/kubernetes.io/serviceaccount/namespace
dev-team
kubectl exec -it dev-kubectl -n dev-team -- cat /run/secrets/kubernetes.io/serviceaccount/ca.crt
-----BEGIN CERTIFICATE-----
MIIDBTCCAe2gAwIBAgIIYHPtWSZBnNYwDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
AxMKa3ViZXJuZXRlczAeFw0yNDA0MTMxMTE1NTRaFw0zNDA0MTExMTIwNTRaMBUx
EzARBgNVBAMTCmt1YmVybmV0ZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC/KjJtNZWEdN1wBcKXhBaNevQxJX/n+TjsLgGc94l7YuQUJFQWvU8G6uVb
46iF0fUeAFl16QgXvXljKwusevydkId0fyaPfyPu7+qMoBn7R9bt9wUgteadS+qx
PpCPLC4kZGb/8kdHr3JQ6Ln6pxUjMSBFdRBTSprMafDnM1LL4R9dzVPvSWV9VZ7V
x8uGBDvPbzHpJz2UZIA/RSihyg+eXh5WqVKNtQ663bDzywc7aVepIzWBb4gNDS4X
CNvRjUdL4DAcZabwHrV+sTbK/sjsrraj2Sr3/zS6b62MgAaOrqXIbCAx0g1pOUI8
rdIOoR5p8uBlyRwE1T4Oqh3tp/JZAgMBAAGjWTBXMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQztUss7mOCAAvBDFo889lLEnC0hDAV
BgNVHREEDjAMggprdWJlcm5ldGVzMA0GCSqGSIb3DQEBCwUAA4IBAQAqYjYs8m/H
DSVvuY66HuzYOl5VFqrKPqo9FiMc9vl23EKRcyYZgCMFoSai+VGkUpv5xo9kNPQw
257dj1Q6gPRQPKQ11HRWLsSrHaDeNhbx1Z+zmN+mr6CcA/IAnqeq5WRHiNWmPks5
O7KZj67l2i3cKF/bka5VIEtLTrKcRBw/tyv7+l11dDAnisWlXMutuQFh/y9ma5Ij
XeahRH5eiONoemvtqtd0HDl3zXr8U4l960h8TiaMAirZw2z65NJ0y53NvTg/5LV1
eA5QqM4JojmPZTSoDhMESUMGLlY6hN1n7qtfA7RcuyYp7niuIuPsUlRWvXUQP3ls
4U1d5Yv/G+xJ
-----END CERTIFICATE-----
# 각각 파드로 Shell 접속하여 정보 확인 : 단축 명령어(alias) 사용
alias k1='kubectl exec -it dev-kubectl -n dev-team -- kubectl'
alias k2='kubectl exec -it infra-kubectl -n infra-team -- kubectl'
# 권한 테스트
k1 get pods # kubectl exec -it dev-kubectl -n dev-team -- kubectl get pods 와 동일한 실행 명령이다!
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "pods" in API group "" in the namespace "dev-team"
command terminated with exit code 1
k1 run nginx --image nginx:1.20-alpine
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot create resource "pods" in API group "" in the namespace "dev-team"
command terminated with exit code 1
k1 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
k2 get pods # kubectl exec -it infra-kubectl -n infra-team -- kubectl get pods 와 동일한 실행 명령이다!
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "pods" in API group "" in the namespace "infra-team"
command terminated with exit code 1
k2 run nginx --image nginx:1.20-alpine
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot create resource "pods" in API group "" in the namespace "infra-team"
command terminated with exit code 1
k2 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
# (옵션) kubectl auth can-i 로 kubectl 실행 사용자가 특정 권한을 가졌는지 확인
k1 auth can-i get pods
no
3. 각각 네임스페이스에 롤(Role)를 생성 후 서비스 어카운트 바인딩
- 롤(Role) : apiGroups 와 resources 로 지정된 리소스에 대해 verbs 권한을 인가
- 실행 가능한 조작(verbs) : *(모두 처리), create(생성), delete(삭제), get(조회), list(목록조회), patch(일부업데이트), update(업데이트), watch(변경감시)
# 각각 네임스페이스내의 모든 권한에 대한 롤 생성
cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: role-dev-team
namespace: dev-team
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
EOF
cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: role-infra-team
namespace: infra-team
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
EOF
# 롤 확인
kubectl get roles -n dev-team
kubectl get roles -n infra-team
kubectl get roles -n dev-team -o yaml
kubectl describe roles role-dev-team -n dev-team
Name: role-dev-team
Labels: <none>
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
*.* [] [] [*]
# 롤바인딩 생성 : '서비스어카운트 <-> 롤' 간 서로 연동
cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: roleB-dev-team
namespace: dev-team
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: role-dev-team
subjects:
- kind: ServiceAccount
name: dev-k8s
namespace: dev-team
EOF
cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: roleB-infra-team
namespace: infra-team
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: role-infra-team
subjects:
- kind: ServiceAccount
name: infra-k8s
namespace: infra-team
EOF
# 롤바인딩 확인
kubectl get rolebindings -n dev-team
kubectl get rolebindings -n infra-team
kubectl get rolebindings -n dev-team -o yaml
kubectl describe rolebindings roleB-dev-team -n dev-team
Name: roleB-dev-team
Labels: <none>
Annotations: <none>
Role:
Kind: Role
Name: role-dev-team
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount dev-k8s dev-team
4. 서비스 어카운트를 지정하여 생성한 파드에서 다시 권한 테스트
# 각각 파드로 Shell 접속하여 정보 확인 : 단축 명령어(alias) 사용
alias k1='kubectl exec -it dev-kubectl -n dev-team -- kubectl'
alias k2='kubectl exec -it infra-kubectl -n infra-team -- kubectl'
# 권한 테스트
k1 get pods
NAME READY STATUS RESTARTS AGE
dev-kubectl 1/1 Running 0 21m
k1 run nginx --image nginx:1.20-alpine
pod/nginx created
k1 get pods
NAME READY STATUS RESTARTS AGE
dev-kubectl 1/1 Running 0 21m
nginx 1/1 Running 0 11s
k1 delete pods nginx
pod "nginx" deleted
k1 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
k1 get nodes
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "nodes" in API group "" at the cluster scope
command terminated with exit code 1
k2 get pods
NAME READY STATUS RESTARTS AGE
infra-kubectl 1/1 Running 0 23m
k2 run nginx --image nginx:1.20-alpine
pod/nginx created
k2 get pods
NAME READY STATUS RESTARTS AGE
infra-kubectl 1/1 Running 0 24m
nginx 1/1 Running 0 12s
k2 delete pods nginx
pod "nginx" deleted
k2 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
k2 get nodes
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "nodes" in API group "" at the cluster scope
command terminated with exit code 1
# (옵션) kubectl auth can-i 로 kubectl 실행 사용자가 특정 권한을 가졌는지 확인
k1 auth can-i get pods
yes
EKS 인증/인가
동작 개요
- 사용자/애플리케이션 → k8s 사용 시 ⇒ 인증은 AWS IAM, 인가는 K8S RBAC
krew (RBAC 관련 플러그인)
# 설치
kubectl krew install access-matrix rbac-tool rbac-view rolesum whoami
# k8s 인증된 주체 확인
kubectl whoami
arn:aws:iam::891377200830:user/admin
# Show an RBAC access matrix for server resources
kubectl access-matrix # Review access to cluster-scoped resources
NAME LIST CREATE UPDATE DELETE
alertmanagerconfigs.monitoring.coreos.com ✔ ✔ ✔ ✔
alertmanagers.monitoring.coreos.com ✔ ✔ ✔ ✔
apiservices.apiregistration.k8s.io ✔ ✔ ✔ ✔
bindings ✔
certificatesigningrequests.certificates.k8s.io ✔ ✔ ✔ ✔
clusterrolebindings.rbac.authorization.k8s.io ✔ ✔ ✔ ✔
clusterroles.rbac.authorization.k8s.io ✔ ✔ ✔ ✔
cninodes.vpcresources.k8s.aws ✔ ✔ ✔ ✔
componentstatuses ✔
configmaps ✔ ✔ ✔ ✔
controllerrevisions.apps ✔ ✔ ✔ ✔
cronjobs.batch ✔ ✔ ✔ ✔
csidrivers.storage.k8s.io ✔ ✔ ✔ ✔
csinodes.storage.k8s.io ✔ ✔ ✔ ✔
csistoragecapacities.storage.k8s.io ✔ ✔ ✔ ✔
customresourcedefinitions.apiextensions.k8s.io ✔ ✔ ✔ ✔
daemonsets.apps ✔ ✔ ✔ ✔
deployments.apps ✔ ✔ ✔ ✔
endpoints ✔ ✔ ✔ ✔
endpointslices.discovery.k8s.io ✔ ✔ ✔ ✔
eniconfigs.crd.k8s.amazonaws.com ✔ ✔ ✔ ✔
events ✔ ✔ ✔ ✔
events.events.k8s.io ✔ ✔ ✔ ✔
flowschemas.flowcontrol.apiserver.k8s.io ✔ ✔ ✔ ✔
horizontalpodautoscalers.autoscaling ✔ ✔ ✔ ✔
ingressclasses.networking.k8s.io ✔ ✔ ✔ ✔
ingressclassparams.elbv2.k8s.aws ✔ ✔ ✔ ✔
ingresses.networking.k8s.io ✔ ✔ ✔ ✔
jobs.batch ✔ ✔ ✔ ✔
leases.coordination.k8s.io ✔ ✔ ✔ ✔
limitranges ✔ ✔ ✔ ✔
localsubjectaccessreviews.authorization.k8s.io ✔
mutatingwebhookconfigurations.admissionregistration.k8s.io ✔ ✔ ✔ ✔
namespaces ✔ ✔ ✔ ✔
networkpolicies.networking.k8s.io ✔ ✔ ✔ ✔
nodes ✔ ✔ ✔ ✔
nodes.metrics.k8s.io ✔
persistentvolumeclaims ✔ ✔ ✔ ✔
persistentvolumes ✔ ✔ ✔ ✔
poddisruptionbudgets.policy ✔ ✔ ✔ ✔
podmonitors.monitoring.coreos.com ✔ ✔ ✔ ✔
pods ✔ ✔ ✔ ✔
pods.metrics.k8s.io ✔
podtemplates ✔ ✔ ✔ ✔
policyendpoints.networking.k8s.aws ✔ ✔ ✔ ✔
priorityclasses.scheduling.k8s.io ✔ ✔ ✔ ✔
prioritylevelconfigurations.flowcontrol.apiserver.k8s.io ✔ ✔ ✔ ✔
probes.monitoring.coreos.com ✔ ✔ ✔ ✔
prometheusagents.monitoring.coreos.com ✔ ✔ ✔ ✔
prometheuses.monitoring.coreos.com ✔ ✔ ✔ ✔
prometheusrules.monitoring.coreos.com ✔ ✔ ✔ ✔
replicasets.apps ✔ ✔ ✔ ✔
replicationcontrollers ✔ ✔ ✔ ✔
resourcequotas ✔ ✔ ✔ ✔
rolebindings.rbac.authorization.k8s.io ✔ ✔ ✔ ✔
roles.rbac.authorization.k8s.io ✔ ✔ ✔ ✔
runtimeclasses.node.k8s.io ✔ ✔ ✔ ✔
scrapeconfigs.monitoring.coreos.com ✔ ✔ ✔ ✔
secrets ✔ ✔ ✔ ✔
securitygrouppolicies.vpcresources.k8s.aws ✔ ✔ ✔ ✔
selfsubjectaccessreviews.authorization.k8s.io ✔
selfsubjectreviews.authentication.k8s.io ✔
selfsubjectrulesreviews.authorization.k8s.io ✔
serviceaccounts ✔ ✔ ✔ ✔
servicemonitors.monitoring.coreos.com ✔ ✔ ✔ ✔
services ✔ ✔ ✔ ✔
statefulsets.apps ✔ ✔ ✔ ✔
storageclasses.storage.k8s.io ✔ ✔ ✔ ✔
subjectaccessreviews.authorization.k8s.io ✔
targetgroupbindings.elbv2.k8s.aws ✔ ✔ ✔ ✔
thanosrulers.monitoring.coreos.com ✔ ✔ ✔ ✔
tokenreviews.authentication.k8s.io ✔
validatingwebhookconfigurations.admissionregistration.k8s.io ✔ ✔ ✔ ✔
volumeattachments.storage.k8s.io ✔ ✔ ✔ ✔
kubectl access-matrix --namespace default # Review access to namespaced resources in 'default'
# RBAC Lookup by subject (user/group/serviceaccount) name
kubectl rbac-tool lookup
kubectl rbac-tool lookup system:masters
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE | BINDING
+----------------+--------------+-------------+-----------+---------------+---------------+
system:masters | Group | ClusterRole | | cluster-admin | cluster-admin
kubectl rbac-tool lookup system:nodes # eks:node-bootstrapper
kubectl rbac-tool lookup system:bootstrappers # eks:node-bootstrapper
kubectl describe ClusterRole eks:node-bootstrapper
# RBAC List Policy Rules For subject (user/group/serviceaccount) name
kubectl rbac-tool policy-rules
kubectl rbac-tool policy-rules -e '^system:.*'
kubectl rbac-tool policy-rules -e '^system:authenticated'
TYPE | SUBJECT | VERBS | NAMESPACE | API GROUP | KIND | NAMES | NONRESOURCEURI | ORIGINATED FROM
+-------+----------------------+--------+-----------+-----------------------+--------------------------+-------+------------------------------------------------------------------------------------------+-----------------------------------------+
Group | system:authenticated | create | * | authentication.k8s.io | selfsubjectreviews | | | ClusterRoles>>system:basic-user
Group | system:authenticated | create | * | authorization.k8s.io | selfsubjectaccessreviews | | | ClusterRoles>>system:basic-user
Group | system:authenticated | create | * | authorization.k8s.io | selfsubjectrulesreviews | | | ClusterRoles>>system:basic-user
Group | system:authenticated | get | * | | | | /healthz,/livez,/readyz,/version,/version/ | ClusterRoles>>system:public-info-viewer
Group | system:authenticated | get | * | | | | /api,/api/*,/apis,/apis/*,/healthz,/livez,/openapi,/openapi/*,/readyz,/version,/version/ | ClusterRoles>>system:discovery
# Generate ClusterRole with all available permissions from the target cluster
kubectl rbac-tool show
# Shows the subject for the current context with which one authenticates with the cluster
kubectl rbac-tool whoami
{Username: "arn:aws:iam::911283...:user/admin", <<-- 과거 "kubernetes-admin"에서 변경됨
UID: "aws-iam-authenticator:911283.:AIDA5ILF2FJI...",
Groups: ["system:authenticated"], <<-- 과거 "system:master"는 안보임
Extra: {accessKeyId: ["AKIA5ILF2FJI....."],
arn: ["arn:aws:iam::9112834...:user/admin"],
canonicalArn: ["arn:aws:iam::9112834...:user/admin"],
principalId: ["AIDA5ILF2FJI...."],
sessionName: [""]}}
# Summarize RBAC roles for subjects : ServiceAccount(default), User, Group
kubectl rolesum -h
kubectl rolesum aws-node -n kube-system
ServiceAccount: kube-system/aws-node
Secrets:
Policies:
• [CRB] */aws-node ⟶ [CR] */aws-node
Resource Name Exclude Verbs G L W C U P D DC
cninodes.vpcresources.k8s.aws [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✔ ✖ ✖
eniconfigs.crd.k8s.amazonaws.com [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
events.[,events.k8s.io] [*] [-] [-] ✖ ✔ ✖ ✔ ✖ ✔ ✖ ✖
namespaces [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
nodes [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
pods [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
policyendpoints.networking.k8s.aws [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
policyendpoints.networking.k8s.aws/status [*] [-] [-] ✔ ✖ ✖ ✖ ✖ ✖ ✖ ✖
kubectl rolesum -k User system:kube-proxy
User: system:kube-proxy
Policies:
• [CRB] */system:node-proxier ⟶ [CR] */system:node-proxier
Resource Name Exclude Verbs G L W C U P D DC
endpoints [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
endpointslices.discovery.k8s.io [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
events.[,events.k8s.io] [*] [-] [-] ✖ ✖ ✖ ✔ ✔ ✔ ✖ ✖
nodes [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
services [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
kubectl rolesum -k Group system:masters
Group: system:masters
Policies:
• [CRB] */cluster-admin ⟶ [CR] */cluster-admin
Resource Name Exclude Verbs G L W C U P D DC
*.* [*] [-] [-] ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
kubectl rolesum -k Group system:authenticated
Group: system:authenticated
Policies:
• [CRB] */system:basic-user ⟶ [CR] */system:basic-user
Resource Name Exclude Verbs G L W C U P D DC
selfsubjectaccessreviews.authorization.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
selfsubjectreviews.authentication.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
selfsubjectrulesreviews.authorization.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
• [CRB] */system:discovery ⟶ [CR] */system:discovery
• [CRB] */system:public-info-viewer ⟶ [CR] */system:public-info-viewer
# [터미널1] A tool to visualize your RBAC permissions
kubectl rbac-view
INFO[0000] Getting K8s client
INFO[0000] serving RBAC View and http://localhost:8800
## 이후 해당 작업용PC 공인 IP:8800 웹 접속 : 최초 접속 후 정보 가져오는데 다시 시간 걸림 (2~3분 정도 후 화면 출력됨)
echo -e "RBAC View Web http://$(curl -s ipinfo.io/ip):8800"
인증/인가 완벽 분석 해보기
1. kubectl 명령 → aws eks get-token → EKS Service endpoint(STS)에 토큰 요청 ⇒ 응답값 디코드(Pre-Signed URL 이며 GetCallerIdentity..) - 링크
- STS (Security Token Service)
AWS 리소스에 대한 액세스를 제어할 수 있는 임시 보안 자격 증명(STS)을 생성하여 신뢰받는 사용자에게 제공할 수 있음
- AWS CLI 버전 1.16.156 이상에서는 별도 aws-iam-authenticator 설치 없이 aws eks get-token으로 사용 가능 - Docs
# sts caller id의 ARN 확인
aws sts get-caller-identity --query Arn
"arn:aws:iam::891377200830:user/admin"
# kubeconfig 정보 확인
cat ~/.kube/config | yh
...
- name: admin@myeks.ap-northeast-2.eksctl.io
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- eks
- get-token
- --output
- json
- --cluster-name
- myeks
- --region
- ap-northeast-2
command: aws
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
interactiveMode: IfAvailable
provideClusterInfo: false
# Get a token for authentication with an Amazon EKS cluster.
# This can be used as an alternative to the aws-iam-authenticator.
aws eks get-token help
# 임시 보안 자격 증명(토큰)을 요청 : expirationTimestamp 시간경과 시 토큰 재발급됨
aws eks get-token --cluster-name $CLUSTER_NAME | jq
{
"kind": "ExecCredential",
"apiVersion": "client.authentication.k8s.io/v1beta1",
"spec": {},
"status": {
"expirationTimestamp": "2024-04-13T13:18:01Z",
"token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuYXAtbm9ydGhlYXN0LTIuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNSZYLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUE0N0NSWVYyN0hKU1BDNVFZJTJGMjAyNDA0MTMlMkZhcC1ub3J0aGVhc3QtMiUyRnN0cyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDEzVDEzMDQwMVomWC1BbXotRXhwaXJlcz02MCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QlM0J4LWs4cy1hd3MtaWQmWC1BbXotU2lnbmF0dXJlPWFlZjVmYmNkMTEzYWM3ODcyOTQ4MTZhNDc1ZWRiODFmYjQ0ODc0YWI0ZjY3NTA5MmM0NGM1YTQ4ZmY0MjhkNzE"
}
}
aws eks get-token --cluster-name $CLUSTER_NAME | jq -r '.status.token'
k8s-aws-v1.aHR0cHM6Ly9zdHMuYXAtbm9ydGhlYXN0LTIuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNSZYLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUE0N0NSWVYyN0hKU1BDNVFZJTJGMjAyNDA0MTMlMkZhcC1ub3J0aGVhc3QtMiUyRnN0cyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDEzVDEzMDQyMVomWC1BbXotRXhwaXJlcz02MCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QlM0J4LWs4cy1hd3MtaWQmWC1BbXotU2lnbmF0dXJlPWRmNGI3M2Q0YWU2ZDUwNmE0MWEyMTc2Mzc4MDFlYTNiYzI5YjE3N2Q2OTMyZDI2ZDdhZjQyMTY3OGEzOWI5MGQ
2. kubectl의 Client-Go 라이브러리는 Pre-Signed URL을 Bearer Token으로 EKS API Cluster Endpoint로 요청을 보냄
- 토큰을 jwt 사이트에 복붙으로 디코드 정보 확인(HS384 → HS256) PAYLOAD 정보 확인 : 일반적인 AWS API 호출과 유사합니다!
- PAYLOAD의 값을 URL Decode Online 에서 DECODE로 확인
https://sts.ap-northeast-2.amazonaws.com/?Action=GetCallerIdentity&
Version=2011-06-15&
X-Amz-Algorithm=AWS4-HMAC-SHA256&
X-Amz-Credential=AKIA47CRYV27HJSPC5QY/20240413/ap-northeast-2/sts/aws4_request&
X-Amz-Date=20240413T130421Z&X-Amz-Expires=60&
X-Amz-SignedHeaders=host;x-k8s-aws-id&
X-Amz-Signature=df4b73d4ae6d506a41a217637801ea3bc29b177d6932d26d7af421678a39b90d
3. EKS API는 Token Review 를 Webhook token authenticator에 요청 ⇒ (STS GetCallerIdentity 호출) AWS IAM 해당 호출 인증 완료 후 User/Role에 대한 ARN 반환
- Webhook token authenticator 는 aws-iam-authenticator 를 사용
# tokenreviews api 리소스 확인
kubectl api-resources | grep authentication
selfsubjectreviews authentication.k8s.io/v1 false SelfSubjectReview
tokenreviews authentication.k8s.io/v1 false TokenReview
# List the fields for supported resources.
kubectl explain tokenreviews
GROUP: authentication.k8s.io
KIND: TokenReview
VERSION: v1
DESCRIPTION:
TokenReview attempts to authenticate a token to a known user. Note:
TokenReview requests may be cached by the webhook token authenticator plugin
in the kube-apiserver.
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string>
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <ObjectMeta>
Standard object's metadata. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec <TokenReviewSpec> -required-
Spec holds information about the request being evaluated
status <TokenReviewStatus>
Status is filled in by the server and indicates whether the request can be
authenticated.
4. 쿠버네티스 RBAC 인가 처리
- 해당 IAM User/Role 확인이 되면 k8s aws-auth configmap에서 mapping 정보를 확인
- aws-auth 컨피그맵에 'IAM 사용자, 역할 arm, K8S 오브젝트' 로 권한 확인 후 k8s 인가 허가가 되면 최종적으로 동작 실행
- 참고로 EKS를 생성한 IAM principal은 aws-auth 와 상관없이 kubernetes-admin Username으로 system:masters 그룹에 권한을 가짐 - 링크
# Webhook api 리소스 확인
kubectl api-resources | grep Webhook
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration
# validatingwebhookconfigurations 리소스 확인
kubectl get validatingwebhookconfigurations
NAME WEBHOOKS AGE
aws-load-balancer-webhook 3 118m
eks-aws-auth-configmap-validation-webhook 1 132m
kube-prometheus-stack-admission 1 116m
vpc-resource-validating-webhook 2 132m
kubectl get validatingwebhookconfigurations eks-aws-auth-configmap-validation-webhook -o yaml | kubectl neat | yh
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: eks-aws-auth-configmap-validation-webhook
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJWUhQdFdTWkJuTll3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBME1UTXhNVEUxTlRSYUZ3MHpOREEwTVRFeE1USXdOVFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUMvS2pKdE5aV0VkTjF3QmNLWGhCYU5ldlF4SlgvbitUanNMZ0djOTRsN1l1UVVKRlFXdlU4RzZ1VmIKNDZpRjBmVWVBRmwxNlFnWHZYbGpLd3VzZXZ5ZGtJZDBmeWFQZnlQdTcrcU1vQm43UjlidDl3VWd0ZWFkUytxeApQcENQTEM0a1pHYi84a2RIcjNKUTZMbjZweFVqTVNCRmRSQlRTcHJNYWZEbk0xTEw0UjlkelZQdlNXVjlWWjdWCng4dUdCRHZQYnpIcEp6MlVaSUEvUlNpaHlnK2VYaDVXcVZLTnRRNjYzYkR6eXdjN2FWZXBJeldCYjRnTkRTNFgKQ052UmpVZEw0REFjWmFid0hyVitzVGJLL3Nqc3JyYWoyU3IzL3pTNmI2Mk1nQWFPcnFYSWJDQXgwZzFwT1VJOApyZElPb1I1cDh1Qmx5UndFMVQ0T3FoM3RwL0paQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRenRVc3M3bU9DQUF2QkRGbzg4OWxMRW5DMGhEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQXFZallzOG0vSApEU1Z2dVk2Nkh1ellPbDVWRnFyS1BxbzlGaU1jOXZsMjNFS1JjeVlaZ0NNRm9TYWkrVkdrVXB2NXhvOWtOUFF3CjI1N2RqMVE2Z1BSUVBLUTExSFJXTHNTckhhRGVOaGJ4MVorem1OK21yNkNjQS9JQW5xZXE1V1JIaU5XbVBrczUKTzdLWmo2N2wyaTNjS0YvYmthNVZJRXRMVHJLY1JCdy90eXY3K2wxMWREQW5pc1dsWE11dHVRRmgveTltYTVJagpYZWFoUkg1ZWlPTm9lbXZ0cXRkMEhEbDN6WHI4VTRsOTYwaDhUaWFNQWlyWncyejY1TkoweTUzTnZUZy81TFYxCmVBNVFxTTRKb2ptUFpUU29EaE1FU1VNR0xsWTZoTjFuN3F0ZkE3UmN1eVlwN25pdUl1UHNVbFJXdlhVUVAzbHMKNFUxZDVZdi9HK3hKCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
url: https://127.0.0.1:21375/validate
failurePolicy: Ignore
matchConditions:
- expression: (request.name == "aws-auth")
name: aws-auth-configmap
matchPolicy: Equivalent
name: eks-aws-auth-configmap-validation-webhook.amazonaws.com
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- UPDATE
resources:
- configmaps
scope: '*'
sideEffects: None
timeoutSeconds: 5
# aws-auth 컨피그맵 확인
kubectl get cm -n kube-system aws-auth -o yaml | kubectl neat | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV
username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
# EKS 설치한 IAM User 정보 >> system:authenticated는 어떤 방식으로 추가가 되었는지 궁금???
kubectl rbac-tool whoami
{Username: "arn:aws:iam::891377200830:user/admin",
UID: "aws-iam-authenticator:891377200830:AIDA47CRYV27GNFRVL2FR",
Groups: ["system:authenticated"],
Extra: {accessKeyId: ["AKIA47CRYV27HJSPC5QY"],
arn: ["arn:aws:iam::891377200830:user/admin"],
canonicalArn: ["arn:aws:iam::891377200830:user/admin"],
principalId: ["AIDA47CRYV27GNFRVL2FR"],
sessionName: [""]}}
# system:masters , system:authenticated 그룹의 정보 확인
kubectl rbac-tool lookup system:masters
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE | BINDING
+----------------+--------------+-------------+-----------+---------------+---------------+
system:masters | Group | ClusterRole | | cluster-admin | cluster-admin
kubectl rbac-tool lookup system:authenticated
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE | BINDING
+----------------------+--------------+-------------+-----------+---------------------------+---------------------------+
system:authenticated | Group | ClusterRole | | system:discovery | system:discovery
system:authenticated | Group | ClusterRole | | system:public-info-viewer | system:public-info-viewer
system:authenticated | Group | ClusterRole | | system:basic-user | system:basic-user
kubectl rolesum -k Group system:masters
Group: system:masters
Policies:
• [CRB] */cluster-admin ⟶ [CR] */cluster-admin
Resource Name Exclude Verbs G L W C U P D DC
*.* [*] [-] [-] ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
kubectl rolesum -k Group system:authenticated
Group: system:authenticated
Policies:
• [CRB] */system:basic-user ⟶ [CR] */system:basic-user
Resource Name Exclude Verbs G L W C U P D DC
selfsubjectaccessreviews.authorization.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
selfsubjectreviews.authentication.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
selfsubjectrulesreviews.authorization.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
• [CRB] */system:discovery ⟶ [CR] */system:discovery
• [CRB] */system:public-info-viewer ⟶ [CR] */system:public-info-viewer
# system:masters 그룹이 사용 가능한 클러스터 롤 확인 : cluster-admin
kubectl describe clusterrolebindings.rbac.authorization.k8s.io cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
Role:
Kind: ClusterRole
Name: cluster-admin
Subjects:
Kind Name Namespace
---- ---- ---------
Group system:masters
# cluster-admin 의 PolicyRule 확인 : 모든 리소스 사용 가능!
kubectl describe clusterrole cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
*.* [] [] [*]
[*] [] [*]
# system:authenticated 그룹이 사용 가능한 클러스터 롤 확인
kubectl describe ClusterRole system:discovery
Name: system:discovery
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version] [] [get]
kubectl describe ClusterRole system:public-info-viewer
Name: system:public-info-viewer
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
[/healthz] [] [get]
[/livez] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version] [] [get]
kubectl describe ClusterRole system:basic-user
Name: system:basic-user
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
selfsubjectreviews.authentication.k8s.io [] [] [create]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
kubectl describe ClusterRole eks:podsecuritypolicy:privileged
데브옵스 신입 사원을 위한 myeks-bastion-2 설정 해보기
1. [myeks-bastion] testuser 사용자 생성
# testuser 사용자 생성
aws iam create-user --user-name testuser
{
"User": {
"Path": "/",
"UserName": "testuser",
"UserId": "AIDA4##",
"Arn": "arn:aws:iam::891377200830:user/testuser",
"CreateDate": "2024-04-13T13:48:07+00:00"
}
}
# 사용자에게 프로그래밍 방식 액세스 권한 부여
aws iam create-access-key --user-name testuser
{
"AccessKey": {
"UserName": "testuser",
"AccessKeyId": "AKIA4##",
"Status": "Active",
"SecretAccessKey": "wIkXy##",
"CreateDate": "2024-04-13T13:48:23+00:00"
}
}
# testuser 사용자에 정책을 추가
aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AdministratorAccess --user-name testuser
# get-caller-identity 확인
aws sts get-caller-identity --query Arn
"arn:aws:iam::891377200830:user/admin"
kubectl whoami
2. [myeks-bastion-2] testuser 자격증명 설정 및 확인
# testuser 자격증명 설정
aws configure
# get-caller-identity 확인
aws sts get-caller-identity --query Arn
"arn:aws:iam::891377200830:user/testuser"
# kubectl 시도 >> testuser도 AdministratorAccess 권한을 가지고 있는데, 실패 이유는?
kubectl get node -v6
ls ~/.kube
ls: cannot access /root/.kube: No such file or directory3. [myeks-bastion] testuser에 system:masters 그룹 부여로 EKS 관리자 수준 권한 설정
# eksctl 사용 >> iamidentitymapping 실행 시 aws-auth 컨피그맵 작성해줌
# Creates a mapping from IAM role or user to Kubernetes user and groups
eksctl get iamidentitymapping --cluster $CLUSTER_NAME
eksctl create iamidentitymapping --cluster $CLUSTER_NAME --username testuser --group system:masters --arn arn:aws:iam::$ACCOUNT_ID:user/testuser
# 확인
kubectl get cm -n kube-system aws-auth -o yaml | kubectl neat | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- groups:
- system:masters
userarn: arn:aws:iam::891377200830:user/testuser
username: testuser
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
kubectl get validatingwebhookconfigurations eks-aws-auth-configmap-validation-webhook -o yaml | kubectl neat | yh
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: eks-aws-auth-configmap-validation-webhook
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJWUhQdFdTWkJuTll3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBME1UTXhNVEUxTlRSYUZ3MHpOREEwTVRFeE1USXdOVFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUMvS2pKdE5aV0VkTjF3QmNLWGhCYU5ldlF4SlgvbitUanNMZ0djOTRsN1l1UVVKRlFXdlU4RzZ1VmIKNDZpRjBmVWVBRmwxNlFnWHZYbGpLd3VzZXZ5ZGtJZDBmeWFQZnlQdTcrcU1vQm43UjlidDl3VWd0ZWFkUytxeApQcENQTEM0a1pHYi84a2RIcjNKUTZMbjZweFVqTVNCRmRSQlRTcHJNYWZEbk0xTEw0UjlkelZQdlNXVjlWWjdWCng4dUdCRHZQYnpIcEp6MlVaSUEvUlNpaHlnK2VYaDVXcVZLTnRRNjYzYkR6eXdjN2FWZXBJeldCYjRnTkRTNFgKQ052UmpVZEw0REFjWmFid0hyVitzVGJLL3Nqc3JyYWoyU3IzL3pTNmI2Mk1nQWFPcnFYSWJDQXgwZzFwT1VJOApyZElPb1I1cDh1Qmx5UndFMVQ0T3FoM3RwL0paQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRenRVc3M3bU9DQUF2QkRGbzg4OWxMRW5DMGhEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQXFZallzOG0vSApEU1Z2dVk2Nkh1ellPbDVWRnFyS1BxbzlGaU1jOXZsMjNFS1JjeVlaZ0NNRm9TYWkrVkdrVXB2NXhvOWtOUFF3CjI1N2RqMVE2Z1BSUVBLUTExSFJXTHNTckhhRGVOaGJ4MVorem1OK21yNkNjQS9JQW5xZXE1V1JIaU5XbVBrczUKTzdLWmo2N2wyaTNjS0YvYmthNVZJRXRMVHJLY1JCdy90eXY3K2wxMWREQW5pc1dsWE11dHVRRmgveTltYTVJagpYZWFoUkg1ZWlPTm9lbXZ0cXRkMEhEbDN6WHI4VTRsOTYwaDhUaWFNQWlyWncyejY1TkoweTUzTnZUZy81TFYxCmVBNVFxTTRKb2ptUFpUU29EaE1FU1VNR0xsWTZoTjFuN3F0ZkE3UmN1eVlwN25pdUl1UHNVbFJXdlhVUVAzbHMKNFUxZDVZdi9HK3hKCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
url: https://127.0.0.1:21375/validate
failurePolicy: Ignore
matchConditions:
- expression: (request.name == "aws-auth")
name: aws-auth-configmap
matchPolicy: Equivalent
name: eks-aws-auth-configmap-validation-webhook.amazonaws.com
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- UPDATE
resources:
- configmaps
scope: '*'
sideEffects: None
timeoutSeconds: 5
# 확인 : 기존에 있는 role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-YYYYY 는 어떤 역할/동작을 하는 걸까요?
eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
arn:aws:iam::891377200830:user/testuser testuser system:masters4. [myeks-bastion-2] testuser kubeconfig 생성 및 kubectl 사용 확인
# testuser kubeconfig 생성 >> aws eks update-kubeconfig 실행이 가능한 이유는?, 3번 설정 후 약간의 적용 시간 필요
aws eks update-kubeconfig --name $CLUSTER_NAME --user-alias testuser
# 첫번째 bastic ec2의 config와 비교해보자
cat ~/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJWUhQdFdTWkJuTll3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBME1UTXhNVEUxTlRSYUZ3MHpOREEwTVRFeE1USXdOVFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUMvS2pKdE5aV0VkTjF3QmNLWGhCYU5ldlF4SlgvbitUanNMZ0djOTRsN1l1UVVKRlFXdlU4RzZ1VmIKNDZpRjBmVWVBRmwxNlFnWHZYbGpLd3VzZXZ5ZGtJZDBmeWFQZnlQdTcrcU1vQm43UjlidDl3VWd0ZWFkUytxeApQcENQTEM0a1pHYi84a2RIcjNKUTZMbjZweFVqTVNCRmRSQlRTcHJNYWZEbk0xTEw0UjlkelZQdlNXVjlWWjdWCng4dUdCRHZQYnpIcEp6MlVaSUEvUlNpaHlnK2VYaDVXcVZLTnRRNjYzYkR6eXdjN2FWZXBJeldCYjRnTkRTNFgKQ052UmpVZEw0REFjWmFid0hyVitzVGJLL3Nqc3JyYWoyU3IzL3pTNmI2Mk1nQWFPcnFYSWJDQXgwZzFwT1VJOApyZElPb1I1cDh1Qmx5UndFMVQ0T3FoM3RwL0paQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRenRVc3M3bU9DQUF2QkRGbzg4OWxMRW5DMGhEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQXFZallzOG0vSApEU1Z2dVk2Nkh1ellPbDVWRnFyS1BxbzlGaU1jOXZsMjNFS1JjeVlaZ0NNRm9TYWkrVkdrVXB2NXhvOWtOUFF3CjI1N2RqMVE2Z1BSUVBLUTExSFJXTHNTckhhRGVOaGJ4MVorem1OK21yNkNjQS9JQW5xZXE1V1JIaU5XbVBrczUKTzdLWmo2N2wyaTNjS0YvYmthNVZJRXRMVHJLY1JCdy90eXY3K2wxMWREQW5pc1dsWE11dHVRRmgveTltYTVJagpYZWFoUkg1ZWlPTm9lbXZ0cXRkMEhEbDN6WHI4VTRsOTYwaDhUaWFNQWlyWncyejY1TkoweTUzTnZUZy81TFYxCmVBNVFxTTRKb2ptUFpUU29EaE1FU1VNR0xsWTZoTjFuN3F0ZkE3UmN1eVlwN25pdUl1UHNVbFJXdlhVUVAzbHMKNFUxZDVZdi9HK3hKCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://2D3EF5FBF387DEF75B12F3C46DFC57C0.gr7.ap-northeast-2.eks.amazonaws.com
name: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
contexts:
- context:
cluster: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
user: testuser
name: testuser
current-context: testuser
kind: Config
preferences: {}
users:
- name: testuser
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- ap-northeast-2
- eks
- get-token
- --cluster-name
- myeks
- --output
- json
command: aws
# kubectl 사용 확인
kubectl ns default
kubectl get node -v6
I0413 22:59:26.361717 2013 loader.go:395] Config loaded from file: /root/.kube/config
I0413 22:59:27.135798 2013 round_trippers.go:553] GET https://2D3EF5FBF387DEF75B12F3C46DFC57C0.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 764 milliseconds
NAME STATUS ROLES AGE VERSION
ip-192-168-1-23.ap-northeast-2.compute.internal Ready <none> 148m v1.28.5-eks-5e0fdde
ip-192-168-2-241.ap-northeast-2.compute.internal Ready <none> 149m v1.28.5-eks-5e0fdde
ip-192-168-3-140.ap-northeast-2.compute.internal Ready <none> 149m v1.28.5-eks-5e0fdde
# rbac-tool 후 확인
kubectl krew install rbac-tool && kubectl rbac-tool whoami
{Username: "testuser",
UID: "aws-iam-authenticator:891377200830:AIDA47CRYV27B5RY3SGEB",
Groups: ["system:masters",
"system:authenticated"],
Extra: {accessKeyId: ["AKIA47CRYV27HDGXPIUM"],
arn: ["arn:aws:iam::891377200830:user/testuser"],
canonicalArn: ["arn:aws:iam::891377200830:user/testuser"],
principalId: ["AIDA47CRYV27B5RY3SGEB"],
sessionName: [""]}}
...
5. [myeks-bastion] testuser 의 Group 변경(system:masters → system:authenticated)으로 RBAC 동작 확인
# 아래 edit로 mapUsers 내용 직접 수정 system:authenticated
kubectl edit cm -n kube-system aws-auth
...
# 확인
eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
arn:aws:iam::891377200830:user/testuser testuser system:authenticated
6. [myeks-bastion-2] testuser kubectl 사용 확인
# 시도
kubectl get node -v6
I0413 23:03:49.308545 2213 loader.go:395] Config loaded from file: /root/.kube/config
I0413 23:03:50.233520 2213 round_trippers.go:553] GET https://2D3EF5FBF387DEF75B12F3C46DFC57C0.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 403 Forbidden in 908 milliseconds
I0413 23:03:50.233851 2213 helpers.go:246] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "nodes is forbidden: User \"testuser\" cannot list resource \"nodes\" in API group \"\" at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "nodes"
},
"code": 403
}]
Error from server (Forbidden): nodes is forbidden: User "testuser" cannot list resource "nodes" in API group "" at the cluster scope
kubectl api-resources -v5
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
endpoints ep v1 true Endpoints
events ev v1 true Event
limitranges limits v1 true LimitRange
namespaces ns v1 false Namespace
nodes no v1 false Node
persistentvolumeclaims pvc v1 true PersistentVolumeClaim
persistentvolumes pv v1 false PersistentVolume
pods po v1 true Pod
podtemplates v1 true PodTemplate
replicationcontrollers rc v1 true ReplicationController
resourcequotas quota v1 true ResourceQuota
secrets v1 true Secret
serviceaccounts sa v1 true ServiceAccount
services svc v1 true Service
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration
customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition
apiservices apiregistration.k8s.io/v1 false APIService
controllerrevisions apps/v1 true ControllerRevision
daemonsets ds apps/v1 true DaemonSet
deployments deploy apps/v1 true Deployment
replicasets rs apps/v1 true ReplicaSet
statefulsets sts apps/v1 true StatefulSet
selfsubjectreviews authentication.k8s.io/v1 false SelfSubjectReview
tokenreviews authentication.k8s.io/v1 false TokenReview
localsubjectaccessreviews authorization.k8s.io/v1 true LocalSubjectAccessReview
selfsubjectaccessreviews authorization.k8s.io/v1 false SelfSubjectAccessReview
selfsubjectrulesreviews authorization.k8s.io/v1 false SelfSubjectRulesReview
subjectaccessreviews authorization.k8s.io/v1 false SubjectAccessReview
horizontalpodautoscalers hpa autoscaling/v2 true HorizontalPodAutoscaler
cronjobs cj batch/v1 true CronJob
jobs batch/v1 true Job
certificatesigningrequests csr certificates.k8s.io/v1 false CertificateSigningRequest
leases coordination.k8s.io/v1 true Lease
eniconfigs crd.k8s.amazonaws.com/v1alpha1 false ENIConfig
endpointslices discovery.k8s.io/v1 true EndpointSlice
ingressclassparams elbv2.k8s.aws/v1beta1 false IngressClassParams
targetgroupbindings elbv2.k8s.aws/v1beta1 true TargetGroupBinding
events ev events.k8s.io/v1 true Event
flowschemas flowcontrol.apiserver.k8s.io/v1beta3 false FlowSchema
prioritylevelconfigurations flowcontrol.apiserver.k8s.io/v1beta3 false PriorityLevelConfiguration
nodes metrics.k8s.io/v1beta1 false NodeMetrics
pods metrics.k8s.io/v1beta1 true PodMetrics
alertmanagerconfigs amcfg monitoring.coreos.com/v1alpha1 true AlertmanagerConfig
alertmanagers am monitoring.coreos.com/v1 true Alertmanager
podmonitors pmon monitoring.coreos.com/v1 true PodMonitor
probes prb monitoring.coreos.com/v1 true Probe
prometheusagents promagent monitoring.coreos.com/v1alpha1 true PrometheusAgent
prometheuses prom monitoring.coreos.com/v1 true Prometheus
prometheusrules promrule monitoring.coreos.com/v1 true PrometheusRule
scrapeconfigs scfg monitoring.coreos.com/v1alpha1 true ScrapeConfig
servicemonitors smon monitoring.coreos.com/v1 true ServiceMonitor
thanosrulers ruler monitoring.coreos.com/v1 true ThanosRuler
policyendpoints networking.k8s.aws/v1alpha1 true PolicyEndpoint
ingressclasses networking.k8s.io/v1 false IngressClass
ingresses ing networking.k8s.io/v1 true Ingress
networkpolicies netpol networking.k8s.io/v1 true NetworkPolicy
runtimeclasses node.k8s.io/v1 false RuntimeClass
poddisruptionbudgets pdb policy/v1 true PodDisruptionBudget
clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding
clusterroles rbac.authorization.k8s.io/v1 false ClusterRole
rolebindings rbac.authorization.k8s.io/v1 true RoleBinding
roles rbac.authorization.k8s.io/v1 true Role
priorityclasses pc scheduling.k8s.io/v1 false PriorityClass
csidrivers storage.k8s.io/v1 false CSIDriver
csinodes storage.k8s.io/v1 false CSINode
csistoragecapacities storage.k8s.io/v1 true CSIStorageCapacity
storageclasses sc storage.k8s.io/v1 false StorageClass
volumeattachments storage.k8s.io/v1 false VolumeAttachment
cninodes cnd vpcresources.k8s.aws/v1alpha1 false CNINode
securitygrouppolicies sgp vpcresources.k8s.aws/v1beta1 true SecurityGroupPolicy7. [myeks-bastion]에서 testuser IAM 맵핑 삭제
# testuser IAM 맵핑 삭제
eksctl delete iamidentitymapping --cluster $CLUSTER_NAME --arn arn:aws:iam::$ACCOUNT_ID:user/testuser
# Get IAM identity mapping(s)
eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
kubectl get cm -n kube-system aws-auth -o yaml | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
[]
kind: ConfigMap
metadata:
creationTimestamp: "2024-04-13T11:29:40Z"
name: aws-auth
namespace: kube-system
resourceVersion: "47862"
uid: 8ba018df-2d1d-4fb7-9b6a-3fca80256c99
8. [myeks-bastion-2] testuser kubectl 사용 확인
# 시도
kubectl get node -v6
I0413 23:06:48.012808 2318 loader.go:395] Config loaded from file: /root/.kube/config
I0413 23:06:49.576758 2318 round_trippers.go:553] GET https://2D3EF5FBF387DEF75B12F3C46DFC57C0.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 401 Unauthorized in 1554 milliseconds
I0413 23:06:49.577044 2318 helpers.go:246] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}]
error: You must be logged in to the server (Unauthorized)
kubectl api-resources -v5
E0413 23:07:08.357669 2387 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
I0413 23:07:08.357702 2387 cached_discovery.go:120] skipped caching discovery info due to the server has asked for the client to provide credentials
NAME SHORTNAMES APIVERSION NAMESPACED KIND
I0413 23:07:08.358018 2387 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials",
"reason": "Unauthorized",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "unknown"
}
]
},
"code": 401
}]
error: You must be logged in to the server (the server has asked for the client to provide credentials)EC2 Instance Profile(IAM Role)에 맵핑된 k8s rbac 확인 해보기
1. 노드 mapRoles 확인
# 노드에 STS ARN 정보 확인 : Role 뒤에 인스턴스 ID!
for node in $N1 $N2 $N3; do ssh ec2-user@$node aws sts get-caller-identity --query Arn; done
"arn:aws:sts::891377200830:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV/i-00540f30eb497c689"
"arn:aws:sts::891377200830:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV/i-0d695b153aad2fdfc"
"arn:aws:sts::891377200830:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV/i-0b410ce447000ecf1"
# username 확인! 인스턴스 ID? EC2PrivateDNSName?
kubectl describe configmap -n kube-system aws-auth
Name: aws-auth
Namespace: kube-system
Labels: <none>
Annotations: <none>
Data
====
mapRoles:
----
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV
username: system:node:{{EC2PrivateDNSName}}
mapUsers:
----
[]
BinaryData
====
Events: <none>
# Get IAM identity mapping(s)
eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
2. awscli 파드를 추가하고, 해당 노드(EC2)의 IMDS 정보 확인 : AWS CLI v2 파드 생성
# awscli 파드 생성
cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: awscli-pod
spec:
replicas: 2
selector:
matchLabels:
app: awscli-pod
template:
metadata:
labels:
app: awscli-pod
spec:
containers:
- name: awscli-pod
image: amazon/aws-cli
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
# 파드 생성 확인
kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
awscli-pod-5bdb44b5bd-lz772 1/1 Running 0 33s 192.168.1.242 ip-192-168-1-23.ap-northeast-2.compute.internal <none> <none>
awscli-pod-5bdb44b5bd-mpzw8 1/1 Running 0 33s 192.168.2.64 ip-192-168-2-241.ap-northeast-2.compute.internal <none> <none>
# 파드 이름 변수 지정
APODNAME1=$(kubectl get pod -l app=awscli-pod -o jsonpath={.items[0].metadata.name})
APODNAME2=$(kubectl get pod -l app=awscli-pod -o jsonpath={.items[1].metadata.name})
echo $APODNAME1, $APODNAME2
awscli-pod-5bdb44b5bd-lz772, awscli-pod-5bdb44b5bd-mpzw8
# awscli 파드에서 EC2 InstanceProfile(IAM Role)의 ARN 정보 확인
kubectl exec -it $APODNAME1 -- aws sts get-caller-identity --query Arn
"arn:aws:sts::891377200830:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV/i-00540f30eb497c689"
kubectl exec -it $APODNAME2 -- aws sts get-caller-identity --query Arn
"arn:aws:sts::891377200830:assumed-role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV/i-0d695b153aad2fdfc"
# awscli 파드에서 EC2 InstanceProfile(IAM Role)을 사용하여 AWS 서비스 정보 확인 >> 별도 IAM 자격 증명이 없는데 어떻게 가능한 것일까요?
# > 최소권한부여 필요!!! >>> 보안이 허술한 아무 컨테이너나 탈취 시, IMDS로 해당 노드의 IAM Role 사용 가능!
kubectl exec -it $APODNAME1 -- aws ec2 describe-instances --region ap-northeast-2 --output table --no-cli-pager
kubectl exec -it $APODNAME2 -- aws ec2 describe-vpcs --region ap-northeast-2 --output table --no-cli-pager
# EC2 메타데이터 확인 : IDMSv1은 Disable, IDMSv2 활성화 상태, IAM Role - 링크
kubectl exec -it $APODNAME1 -- bash
-----------------------------------
아래부터는 파드에 bash shell 에서 실행
curl -s http://169.254.169.254/ -v
...
# Token 요청
curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" ; echo
AQAEAP43BMyb3S_l7o4BCp4J57hC2JVfjvW-GJ-v7uZ91WybHZdgyA==
curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" ; echo
AQAEAP43BMx5d3bGAU3tzDObvegF7kAEBIDAG7ELnH6C2E0Os3xK5w==
# Token을 이용한 IMDSv2 사용
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
echo $TOKEN
AQAEAP43BMwtm5G4qODQ-poMZkaV_6FWtOHsJ03wNJddT0PbKof1kQ==
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" –v http://169.254.169.254/ ; echo
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" –v http://169.254.169.254/latest/ ; echo
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" –v http://169.254.169.254/latest/meta-data/iam/security-credentials/ ; echo
# 위에서 출력된 IAM Role을 아래 입력 후 확인
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" –v http://169.254.169.254/latest/meta-data/iam/security-credentials/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV
{
"Code" : "Success",
"LastUpdated" : "2024-04-13T14:20:30Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIA47CRYV27F6M4X55C",
"SecretAccessKey" : "AJONq9eV7XZq2xAk2TICd+YYIOURQtZM30cHqmnk",
"Token" : "IQoJb3JpZ2luX2VjED4aDmFwLW5vcnRoZWFzdC0yIkcwRQIgK8EPCr1p+gpbgvKNHBXBmxWv/B/9XJbKcekbk0sbWlMCIQCCeECc4PlugaibD0Yc11/t06RniPf94gykIboBcg0aiCrLBQh3EAAaDDg5MTM3NzIwMDgzMCIMoB+xVGZt+hDTpKenKqgFOy9eSm7l2yRbKpWZMm3dfxLyGxFNX/rNfMP3t1YW6OcZGrYTdNI1vdq5RkDL4nUWlf+/wOJJJir2Qkh6YVZIGp5oyGNhtVjvrI5Y5i5Ymh0ZMos8hXBxCGJNb9Iipu+XfEbtzKsh6Hbkob5ERbjLOV+SLmexViTdVcmmJWvQwro9qZLbiFjFGyj/LBgWyacFy7izCj3DrhRP/pGTW/bgzW9BxTtOh1lMSjMP9IyZvIC6EAHX2jqOzsjMszbp7jqH+UKFUdcG3z3Xuqi0KV+QqnkCE+4N3AbItiZ8GykHS4FLsvWQQMShGt7Yb8R2L+VCu7qFHKFSZWAn98fpQdfG18G7f2AFwJpb3pr9JestUwT8U8pIqLSzJZ50rrTD7VFcYlTElcMmmc3otA97iLNNiSQzTahY4x3ji3w5HN8DZCdtl3dS8iOILLjj+RmtgTNTWbXOfqFJXazehYQ3PHyAYFOad0XzFrcYHcaiRmQBi5iDIiJYh/cJgaMVEPmunitOa0f1oLi3eD5lOvJlObZ3BNA3Wavin/X++qt+M37NZLN5bHi1pRTYt0bUD2DkB/S0/S1I/HI+20wKSs5TRTYQBd9hBP7IbuvAkZ/7sYgB0ukDvZZfKe7Au8Wt6nEWMjMoARAxIRqkZ26GR2SXlBY9mHW+/IKDQL2Oq6HyFfC83bNJRv8AX5vPrzLN8KEQCSNeY8nz8zkqwnUznJ3oGEfEggZohoNlg/MmFP0AEOMJQNS9Fu6D23yIBO3i27CNQAkn0J4cfFTVW3BhaMixRJcq7Rn4i36gOPx/aGrSEkiTKuESiPSz7ybJ10Y5A2wwjsOaQoXirm7E8vG9iWuPaktf1PoSikneDkqOpoWLPTyXakMAoiRcZi75Gkyl+sv0U+3bYDsU+lVSrvMwsKnqsAY6sQFjPM7KNEfhn6lVU1B+EET7cLIljAakWLt5GnyeAGOQomKSz2bEYlNwgtUYwCJcWBH07X6MT34uo0Gy+hkBI7ypEkM7NuJDTboix0HDhZ2PLTQJ68AZf9KiIy46poqcjuwfcghFK7EbKSCxhtO+OwB/UBDgLfTBSUkhFt2mbsge1l9gpoGP2GBDoyj+WOGDLtO+lDRrZSBKh8b66A/1HpZpE1lOcBVkfVMeLM0IFO7wJdg=",
"Expiration" : "2024-04-13T20:41:44Z"
}
## 출력된 정보는 AWS API를 사용할 수 있는 어느곳에서든지 Expiration 되기전까지 사용 가능
# 파드에서 나오기
exit
---
3. awscli 파드에 kubeconfig (mapRoles) 정보 생성 및 확인
# node 의 IAM Role ARN을 변수로 지정
eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::891377200830:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
NODE_ROLE=eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV
# awscli 파드에서 kubeconfig 정보 생성 및 확인 >> kubeconfig 에 정보가 기존 iam user와 차이점은?
kubectl exec -it $APODNAME1 -- aws eks update-kubeconfig --name $CLUSTER_NAME --role-arn $NODE_ROLE
kubectl exec -it $APODNAME1 -- cat /root/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJWUhQdFdTWkJuTll3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBME1UTXhNVEUxTlRSYUZ3MHpOREEwTVRFeE1USXdOVFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUMvS2pKdE5aV0VkTjF3QmNLWGhCYU5ldlF4SlgvbitUanNMZ0djOTRsN1l1UVVKRlFXdlU4RzZ1VmIKNDZpRjBmVWVBRmwxNlFnWHZYbGpLd3VzZXZ5ZGtJZDBmeWFQZnlQdTcrcU1vQm43UjlidDl3VWd0ZWFkUytxeApQcENQTEM0a1pHYi84a2RIcjNKUTZMbjZweFVqTVNCRmRSQlRTcHJNYWZEbk0xTEw0UjlkelZQdlNXVjlWWjdWCng4dUdCRHZQYnpIcEp6MlVaSUEvUlNpaHlnK2VYaDVXcVZLTnRRNjYzYkR6eXdjN2FWZXBJeldCYjRnTkRTNFgKQ052UmpVZEw0REFjWmFid0hyVitzVGJLL3Nqc3JyYWoyU3IzL3pTNmI2Mk1nQWFPcnFYSWJDQXgwZzFwT1VJOApyZElPb1I1cDh1Qmx5UndFMVQ0T3FoM3RwL0paQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRenRVc3M3bU9DQUF2QkRGbzg4OWxMRW5DMGhEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQXFZallzOG0vSApEU1Z2dVk2Nkh1ellPbDVWRnFyS1BxbzlGaU1jOXZsMjNFS1JjeVlaZ0NNRm9TYWkrVkdrVXB2NXhvOWtOUFF3CjI1N2RqMVE2Z1BSUVBLUTExSFJXTHNTckhhRGVOaGJ4MVorem1OK21yNkNjQS9JQW5xZXE1V1JIaU5XbVBrczUKTzdLWmo2N2wyaTNjS0YvYmthNVZJRXRMVHJLY1JCdy90eXY3K2wxMWREQW5pc1dsWE11dHVRRmgveTltYTVJagpYZWFoUkg1ZWlPTm9lbXZ0cXRkMEhEbDN6WHI4VTRsOTYwaDhUaWFNQWlyWncyejY1TkoweTUzTnZUZy81TFYxCmVBNVFxTTRKb2ptUFpUU29EaE1FU1VNR0xsWTZoTjFuN3F0ZkE3UmN1eVlwN25pdUl1UHNVbFJXdlhVUVAzbHMKNFUxZDVZdi9HK3hKCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://2D3EF5FBF387DEF75B12F3C46DFC57C0.gr7.ap-northeast-2.eks.amazonaws.com
name: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
contexts:
- context:
cluster: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
user: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
name: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
current-context: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
kind: Config
preferences: {}
users:
- name: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- ap-northeast-2
- eks
- get-token
- --cluster-name
- myeks
- --output
- json
- --role
- eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV
command: aws
kubectl exec -it $APODNAME2 -- aws eks update-kubeconfig --name $CLUSTER_NAME --role-arn $NODE_ROLE
kubectl exec -it $APODNAME2 -- cat /root/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJWUhQdFdTWkJuTll3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBME1UTXhNVEUxTlRSYUZ3MHpOREEwTVRFeE1USXdOVFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUMvS2pKdE5aV0VkTjF3QmNLWGhCYU5ldlF4SlgvbitUanNMZ0djOTRsN1l1UVVKRlFXdlU4RzZ1VmIKNDZpRjBmVWVBRmwxNlFnWHZYbGpLd3VzZXZ5ZGtJZDBmeWFQZnlQdTcrcU1vQm43UjlidDl3VWd0ZWFkUytxeApQcENQTEM0a1pHYi84a2RIcjNKUTZMbjZweFVqTVNCRmRSQlRTcHJNYWZEbk0xTEw0UjlkelZQdlNXVjlWWjdWCng4dUdCRHZQYnpIcEp6MlVaSUEvUlNpaHlnK2VYaDVXcVZLTnRRNjYzYkR6eXdjN2FWZXBJeldCYjRnTkRTNFgKQ052UmpVZEw0REFjWmFid0hyVitzVGJLL3Nqc3JyYWoyU3IzL3pTNmI2Mk1nQWFPcnFYSWJDQXgwZzFwT1VJOApyZElPb1I1cDh1Qmx5UndFMVQ0T3FoM3RwL0paQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRenRVc3M3bU9DQUF2QkRGbzg4OWxMRW5DMGhEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQXFZallzOG0vSApEU1Z2dVk2Nkh1ellPbDVWRnFyS1BxbzlGaU1jOXZsMjNFS1JjeVlaZ0NNRm9TYWkrVkdrVXB2NXhvOWtOUFF3CjI1N2RqMVE2Z1BSUVBLUTExSFJXTHNTckhhRGVOaGJ4MVorem1OK21yNkNjQS9JQW5xZXE1V1JIaU5XbVBrczUKTzdLWmo2N2wyaTNjS0YvYmthNVZJRXRMVHJLY1JCdy90eXY3K2wxMWREQW5pc1dsWE11dHVRRmgveTltYTVJagpYZWFoUkg1ZWlPTm9lbXZ0cXRkMEhEbDN6WHI4VTRsOTYwaDhUaWFNQWlyWncyejY1TkoweTUzTnZUZy81TFYxCmVBNVFxTTRKb2ptUFpUU29EaE1FU1VNR0xsWTZoTjFuN3F0ZkE3UmN1eVlwN25pdUl1UHNVbFJXdlhVUVAzbHMKNFUxZDVZdi9HK3hKCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://2D3EF5FBF387DEF75B12F3C46DFC57C0.gr7.ap-northeast-2.eks.amazonaws.com
name: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
contexts:
- context:
cluster: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
user: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
name: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
current-context: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
kind: Config
preferences: {}
users:
- name: arn:aws:eks:ap-northeast-2:891377200830:cluster/myeks
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- ap-northeast-2
- eks
- get-token
- --cluster-name
- myeks
- --output
- json
- --role
- eksctl-myeks-nodegroup-ng1-NodeInstanceRole-5rxDjOwEYYoV
command: aws
