[Kubernetes] Security (TLS, Certificates, CSR, KubeConfig, context)

Xabi·2025년 9월 9일

cka kubernetes

kubernetes

목록 보기

15/20

Kubernetes에서 TLS 인증서와 개인키 보안 정리

1. 개요

Kubernetes(이하 k8s)에서는 애플리케이션/서비스의 보안 강화를 위한 TLS(Transport Layer Security) 인증서를 반드시 사용하게 된다.

2. TLS 인증서란?

TLS는 네트워크 상에서 데이터가 안전하게(암호화) 전달되도록 해주는 프로토콜이다.
이 때 사용하는 인증서를 통해 "서버의 신원 보장"과 "암호화 통신"이 가능하다.

2.1 인증서의 구성요소

명칭	역할	확장자
CA(Certificate Authority)	인증서 발급을 보증하는 기관(공개키)	.crt, .pem
TLS 인증서	서버의 공개키 + 서버 정보 + CA의 서명	.crt, .pem
TLS 개인키	서버만 보관하는 비밀키 (암호화·복호화, 전자서명용)	.key, .pem

3. 각 파일 상세 설명

3.1 CA(인증기관, ca.crt)

공인된 CA가 인증서(서버용 .crt)에 전자서명을 해줌.
웹브라우저/클라이언트는 이미 대부분의 공인 CA 인증서를 내장하고 있음.
서버의 인증서가 신뢰할 수 있는지 최종적 기준 역할.

3.2 TLS 인증서(tls.crt)

서버가 발급받은 신분증(공개키, 서버정보, CA서명 포함).
k8s에서는 주로 Nginx Ingress 등에서 HTTPS 통신에 사용.

3.3 TLS 개인키(tls.key)

인증서와 한 쌍인 비밀키. 서버만 소유, 절대로 외부 유출 금지.
암호화, 복호화, 전자서명을 위해 사용.

4. TLS 파일이 실제로 사용되는 예시

4.1 HTTPS 통신 과정

클라이언트가 서버 접속 시도
서버는 자기 TLS 인증서(tls.crt)와 공개키 전송
클라이언트는 인증서에 CA의 서명이 있는지 검증
서버만 아는 개인키(tls.key)로 암호통신 진행
양측 모두 안전하게 데이터 송수신

4.2 Kubernetes에서 사용 예시

apiVersion: v1
kind: Secret
metadata:
  name: my-tls-secret
type: kubernetes.io/tls
data:
  tls.crt: <base64 인코딩된 인증서>
  tls.key: <base64 인코딩된 개인키>

Secret 이름을 Ingress 리소스에 지정하여 HTTPS 적용.

5. CA의 역할과 중요성

CA(인증기관)는 인증서의 신뢰성을 보장한다.
CA에서 전자서명된 인증서는 모든 클라이언트, 브라우저가 신뢰할 수 있다.
직접 CA를 운영할 수도 있고, 공인 CA를 사용할 수도 있다.

6. TLS 개인키(Private Key) 보안, 왜 중요한가?

6.1 해커에게 탈취 시 위험성

해커가 개인키를 소유하면 다음이 가능하다.

서버를 위장해 중간자(MITM, Man-in-the-middle) 공격
암호화된 통신 내용 탈취 및 변조
사용자의 중요한 데이터(비밀번호, 인증정보 등) 노출

6.2 실제 노출 사례

잘못된 파일 권한 설정 (예: 644, 777)
서버 해킹 후 파일 직접 복사
소스/배포 과정에서 실수로 공개 저장소(commits)에 유출
쿠버네티스 Secret 권한 부여 실수로 third-party가 조회

6.3 보안 관리 방법

권한 제한: 개인키 파일은 반드시 600으로 설정(소유자만 읽기/쓰기)
안전한 저장소 사용: HashiCorp Vault, Cloud KMS(AWS/GCP/Azure), HSM 등
k8s RBAC 최소화: Secret 사용 권한은 꼭 필요한 Pod/사용자만 부여
불분명한 사용처 제거 및 백업·로그에 남기지 않기
개인키 유출 시 빠른 인증서 교체 및 로테이션

7. 요약 및 결론

CA: 신뢰의 밑바탕, 모든 인증서의 최상위 관리자
TLS 인증서 (tls.crt): 서버의 신분증, 누구에게나 보여줘도 된다
TLS 개인키 (tls.key): 서버만 아는 비밀, 유출 시 정체성 복제/탈취 가능
개인키는 반드시 안전하게 관리하며, 여러 차단책을 마련해야 한다

👉 핵심 3줄 요약

TLS 인증서를 사용하면 안전하게 데이터 통신 및 서버 신원확인이 가능하다.
개인키가 유출되면 심각한 보안사고로 이어질 수 있으므로, 반드시 철저하게 권한 제한 및 관리해야 한다.
k8s에서는 Secret, RBAC, KMS 등 다양한 기능을 활용하여 키·인증서 보안을 유지할 수 있다.

관련 연습문제

1) Kubectl suddenly stops responding to your commands. Check it out! Someone recently modified the /etc/kubernetes/manifests/etcd.yaml file.

You are asked to investigate and fix the issue. Once you fix the issue wait for sometime for kubectl to respond. Check the logs of the ETCD container.

controlplane ~ ➜  kubectl get pods
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)

crictl ps -a | grep kube-apiserver
crictl logs container-id

controlplane /var/log/containers ➜  crictl ps -a | grep kube-apiserver
c785c7c0161d5       6ba9545b2183e       9 seconds ago        Running             kube-apiserver            6                   68f0eae2c4a26       kube-apiserver-controlplane            kube-system
623e291a0c21c       6ba9545b2183e       2 minutes ago        Exited              kube-apiserver            5                   68f0eae2c4a26       kube-apiserver-controlplane            kube-system

controlplane /var/log/containers ➜  crictl logs c785c7c0161d5
I0909 04:39:25.739014       1 options.go:249] external host was not specified, using 192.168.138.245
I0909 04:39:25.741163       1 server.go:147] Version: v1.33.0
I0909 04:39:25.741193       1 server.go:149] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
W0909 04:39:26.389211       1 logging.go:55] [core] [Channel #1 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:26.389226       1 logging.go:55] [core] [Channel #2 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
I0909 04:39:26.391363       1 shared_informer.go:350] "Waiting for caches to sync" controller="node_authorizer"
I0909 04:39:26.401472       1 shared_informer.go:350] "Waiting for caches to sync" controller="*generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]"
I0909 04:39:26.409434       1 plugins.go:157] Loaded 14 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,PodTopologyLabels,MutatingAdmissionPolicy,MutatingAdmissionWebhook.
I0909 04:39:26.409467       1 plugins.go:160] Loaded 13 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,ClusterTrustBundleAttest,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
I0909 04:39:26.409735       1 instance.go:233] Using reconciler: lease
W0909 04:39:26.411022       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:27.390468       1 logging.go:55] [core] [Channel #1 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:27.390492       1 logging.go:55] [core] [Channel #2 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:27.411878       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:29.063283       1 logging.go:55] [core] [Channel #1 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:29.114788       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:29.130175       1 logging.go:55] [core] [Channel #2 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:31.253953       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:31.435044       1 logging.go:55] [core] [Channel #2 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:31.476465       1 logging.go:55] [core] [Channel #1 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:34.977045       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:35.535661       1 logging.go:55] [core] [Channel #2 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:36.244217       1 logging.go:55] [core] [Channel #1 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:41.513983       1 logging.go:55] [core] [Channel #2 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:41.655901       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0909 04:39:43.069154       1 logging.go:55] [core] [Channel #1 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
F0909 04:39:46.410711       1 instance.go:226] Error creating leases: error creating storage factory: context deadline exceeded

-> crictl로 에러메시지 확인 : "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
-> 127.0.0.1:2379는 ETCD 서버
-> kube-apiserver가 etcd 서버에 연결되지 못한 이슈로 kubectl이 작동하지 않음

controlplane /var/log/containers ✖ crictl ps -a | grep etcd
3fce5c7b4ab1c       499038711c081       2 minutes ago       Exited              etcd                      6                   b5365a1abe03b       etcd-controlplane                      kube-system

controlplane /var/log/containers ➜  crictl logs 3fce5c7b4ab1c
{"level":"warn","ts":"2025-09-09T04:40:32.688192Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-09-09T04:40:32.688313Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--advertise-client-urls=https://192.168.138.245:2379","--cert-file=/etc/kubernetes/pki/etcd/server-certificate.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--experimental-initial-corrupt-check=true","--experimental-watch-progress-notify-interval=5s","--initial-advertise-peer-urls=https://192.168.138.245:2380","--initial-cluster=controlplane=https://192.168.138.245:2380","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://192.168.138.245:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://192.168.138.245:2380","--name=controlplane","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]}
{"level":"info","ts":"2025-09-09T04:40:32.688435Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"}
{"level":"warn","ts":"2025-09-09T04:40:32.688455Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-09-09T04:40:32.688466Z","caller":"embed/etcd.go:140","msg":"configuring peer listeners","listen-peer-urls":["https://192.168.138.245:2380"]}
{"level":"info","ts":"2025-09-09T04:40:32.688503Z","caller":"embed/etcd.go:528","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2025-09-09T04:40:32.690427Z","caller":"embed/etcd.go:148","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://192.168.138.245:2379"]}
{"level":"info","ts":"2025-09-09T04:40:32.690604Z","caller":"embed/etcd.go:323","msg":"starting an etcd server","etcd-version":"3.5.21","git-sha":"a17edfd","go-version":"go1.23.7","go-os":"linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":true,"name":"controlplane","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://192.168.138.245:2380"],"listen-peer-urls":["https://192.168.138.245:2380"],"advertise-client-urls":["https://192.168.138.245:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.138.245:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2025-09-09T04:40:32.700627Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"9.614598ms"}
{"level":"info","ts":"2025-09-09T04:40:32.707387Z","caller":"etcdserver/server.go:534","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2025-09-09T04:40:32.713441Z","caller":"etcdserver/raft.go:541","msg":"restarting local member","cluster-id":"5c709c924160fd6d","local-member-id":"590f24c3d8737b02","commit-index":1491}
{"level":"info","ts":"2025-09-09T04:40:32.714397Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 switched to configuration voters=()"}
{"level":"info","ts":"2025-09-09T04:40:32.714473Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 became follower at term 8"}
{"level":"info","ts":"2025-09-09T04:40:32.714489Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 590f24c3d8737b02 [peers: [], term: 8, commit: 1491, applied: 0, lastindex: 1491, lastterm: 8]"}
{"level":"warn","ts":"2025-09-09T04:40:32.714839Z","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2025-09-09T04:40:32.714906Z","caller":"mvcc/kvstore.go:348","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":849}
{"level":"info","ts":"2025-09-09T04:40:32.716130Z","caller":"mvcc/kvstore.go:425","msg":"kvstore restored","current-rev":1334}
{"level":"info","ts":"2025-09-09T04:40:32.716174Z","caller":"etcdserver/server.go:628","msg":"restore consistentIndex","index":1489}
{"level":"info","ts":"2025-09-09T04:40:32.716299Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2025-09-09T04:40:32.716743Z","caller":"etcdserver/corrupt.go:96","msg":"starting initial corruption check","local-member-id":"590f24c3d8737b02","timeout":"7s"}
{"level":"info","ts":"2025-09-09T04:40:32.717024Z","caller":"etcdserver/corrupt.go:177","msg":"initial corruption checking passed; no corruption","local-member-id":"590f24c3d8737b02"}
{"level":"info","ts":"2025-09-09T04:40:32.717056Z","caller":"etcdserver/server.go:875","msg":"starting etcd server","local-member-id":"590f24c3d8737b02","local-server-version":"3.5.21","cluster-version":"to_be_decided"}
{"level":"info","ts":"2025-09-09T04:40:32.717148Z","caller":"etcdserver/server.go:775","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2025-09-09T04:40:32.717209Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
{"level":"info","ts":"2025-09-09T04:40:32.717331Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
{"level":"info","ts":"2025-09-09T04:40:32.717378Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
{"level":"info","ts":"2025-09-09T04:40:32.717434Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 switched to configuration voters=(6417388417594915586)"}
{"level":"info","ts":"2025-09-09T04:40:32.717467Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","service":"","status":"SERVING"}
{"level":"info","ts":"2025-09-09T04:40:32.717509Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"5c709c924160fd6d","local-member-id":"590f24c3d8737b02","added-peer-id":"590f24c3d8737b02","added-peer-peer-urls":["https://192.168.138.245:2380"],"added-peer-is-learner":false}
{"level":"info","ts":"2025-09-09T04:40:32.717631Z","caller":"membership/cluster.go:587","msg":"set initial cluster version","cluster-id":"5c709c924160fd6d","local-member-id":"590f24c3d8737b02","cluster-version":"3.5"}
{"level":"info","ts":"2025-09-09T04:40:32.717668Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2025-09-09T04:40:32.720709Z","caller":"embed/etcd.go:762","msg":"starting with client TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/server-certificate.crt, key = /etc/kubernetes/pki/etcd/server.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2025-09-09T04:40:32.720861Z","caller":"embed/etcd.go:633","msg":"serving peer traffic","address":"192.168.138.245:2380"}
{"level":"info","ts":"2025-09-09T04:40:32.720892Z","caller":"embed/etcd.go:603","msg":"cmux::serve","address":"192.168.138.245:2380"}
{"level":"info","ts":"2025-09-09T04:40:32.721118Z","caller":"embed/etcd.go:292","msg":"now serving peer/client/metrics","local-member-id":"590f24c3d8737b02","initial-advertise-peer-urls":["https://192.168.138.245:2380"],"listen-peer-urls":["https://192.168.138.245:2380"],"advertise-client-urls":["https://192.168.138.245:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.138.245:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"]}
{"level":"info","ts":"2025-09-09T04:40:32.721172Z","caller":"embed/etcd.go:908","msg":"serving metrics","address":"http://127.0.0.1:2381"}
{"level":"info","ts":"2025-09-09T04:40:33.714981Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 is starting a new election at term 8"}
{"level":"info","ts":"2025-09-09T04:40:33.715055Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 became pre-candidate at term 8"}
{"level":"info","ts":"2025-09-09T04:40:33.715079Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 received MsgPreVoteResp from 590f24c3d8737b02 at term 8"}
{"level":"info","ts":"2025-09-09T04:40:33.715091Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 became candidate at term 9"}
{"level":"info","ts":"2025-09-09T04:40:33.715131Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 received MsgVoteResp from 590f24c3d8737b02 at term 9"}
{"level":"info","ts":"2025-09-09T04:40:33.715146Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 became leader at term 9"}
{"level":"info","ts":"2025-09-09T04:40:33.715160Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 590f24c3d8737b02 elected leader 590f24c3d8737b02 at term 9"}
{"level":"info","ts":"2025-09-09T04:40:33.715593Z","caller":"etcdserver/server.go:2144","msg":"published local member to cluster through raft","local-member-id":"590f24c3d8737b02","local-member-attributes":"{Name:controlplane ClientURLs:[https://192.168.138.245:2379]}","request-path":"/0/members/590f24c3d8737b02/attributes","cluster-id":"5c709c924160fd6d","publish-timeout":"7s"}
{"level":"info","ts":"2025-09-09T04:40:33.715630Z","caller":"embed/serve.go:124","msg":"ready to serve client requests"}
{"level":"info","ts":"2025-09-09T04:40:33.715646Z","caller":"embed/serve.go:124","msg":"ready to serve client requests"}
{"level":"error","ts":"2025-09-09T04:40:33.715991Z","caller":"embed/serve.go:142","msg":"registerGateway failed","error":"open /etc/kubernetes/pki/etcd/server-certificate.crt: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve\n\tgo.etcd.io/etcd/server/v3/embed/serve.go:142\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients.func1\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:818"}
{"level":"info","ts":"2025-09-09T04:40:33.716068Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2025-09-09T04:40:33.716194Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
{"level":"error","ts":"2025-09-09T04:40:33.716149Z","caller":"embed/serve.go:142","msg":"registerGateway failed","error":"open /etc/kubernetes/pki/etcd/server-certificate.crt: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/embed.(*serveCtx).serve\n\tgo.etcd.io/etcd/server/v3/embed/serve.go:142\ngo.etcd.io/etcd/server/v3/embed.(*Etcd).serveClients.func1\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:818"}
{"level":"fatal","ts":"2025-09-09T04:40:33.716205Z","caller":"etcdmain/etcd.go:219","msg":"listener failed","error":"open /etc/kubernetes/pki/etcd/server-certificate.crt: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:219\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:272"}

-> 에러메시지 확인 : {"level":"fatal","ts":"2025-09-09T04:40:33.716205Z","caller":"etcdmain/etcd.go:219","msg":"listener failed","error":"open /etc/kubernetes/pki/etcd/server-certificate.crt: no such file or directory","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:219\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:272"}
-> /etc/kubernetes/pki/etcd/server-certificate.crt 파일을 못 찾고 있음
-> yaml 파일에서 경로를 올바르게 수정해줘야 함

controlplane ~ ➜  cat /etc/kubernetes/manifests/etcd.yaml | grep "\-\-cert"
    - --cert-file=/etc/kubernetes/pki/etcd/server-certificate.crt

-> 현재 etcd.yaml 파일에서는 위와 같이 작성하였음 확인
-> /etc/kubernetes/pki/etcd/server-certificate.crt 는 에러 로그에 나온 못 찾겠다는 파일 명칭임

controlplane ~ ➜  ls /etc/kubernetes/pki/etcd
ca.crt  ca.key  healthcheck-client.crt  healthcheck-client.key  peer.crt  peer.key  server.crt  server.key

-> 실제 파일 etcd 인증서 파일 경로를 들어가보니 yaml 파일에 기재된 인증서 명과 다르게 server.crt 가 맞는 이름임을 확인

controlplane ~ ➜  vi /etc/kubernetes/manifests/etcd.yaml

-> vi etcd.yaml 로 server.crt 로 명칭 수정해주고 편집

controlplane ~ ➜  crictl ps -a | grep etcd
6ace31bb46baf       499038711c081       18 seconds ago      Running             etcd                      0                   05e9a899698c8       etcd-controlplane                      kube-system

controlplane ~ ➜  crictl logs 6ace31bb46baf
{"level":"warn","ts":"2025-09-09T04:58:30.190350Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-09-09T04:58:30.190431Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--advertise-client-urls=https://192.168.138.245:2379","--cert-file=/etc/kubernetes/pki/etcd/server.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--experimental-initial-corrupt-check=true","--experimental-watch-progress-notify-interval=5s","--initial-advertise-peer-urls=https://192.168.138.245:2380","--initial-cluster=controlplane=https://192.168.138.245:2380","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://192.168.138.245:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://192.168.138.245:2380","--name=controlplane","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]}
{"level":"info","ts":"2025-09-09T04:58:30.190506Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"}
{"level":"warn","ts":"2025-09-09T04:58:30.190523Z","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"info","ts":"2025-09-09T04:58:30.190533Z","caller":"embed/etcd.go:140","msg":"configuring peer listeners","listen-peer-urls":["https://192.168.138.245:2380"]}
{"level":"info","ts":"2025-09-09T04:58:30.190561Z","caller":"embed/etcd.go:528","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2025-09-09T04:58:30.192745Z","caller":"embed/etcd.go:148","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://192.168.138.245:2379"]}
{"level":"info","ts":"2025-09-09T04:58:30.192922Z","caller":"embed/etcd.go:323","msg":"starting an etcd server","etcd-version":"3.5.21","git-sha":"a17edfd","go-version":"go1.23.7","go-os":"linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":true,"name":"controlplane","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://192.168.138.245:2380"],"listen-peer-urls":["https://192.168.138.245:2380"],"advertise-client-urls":["https://192.168.138.245:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.138.245:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2025-09-09T04:58:30.197051Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"3.805703ms"}
{"level":"info","ts":"2025-09-09T04:58:30.203678Z","caller":"etcdserver/server.go:534","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2025-09-09T04:58:30.211340Z","caller":"etcdserver/raft.go:541","msg":"restarting local member","cluster-id":"5c709c924160fd6d","local-member-id":"590f24c3d8737b02","commit-index":1499}
{"level":"info","ts":"2025-09-09T04:58:30.211652Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 switched to configuration voters=()"}
{"level":"info","ts":"2025-09-09T04:58:30.211691Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 became follower at term 12"}
{"level":"info","ts":"2025-09-09T04:58:30.211704Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 590f24c3d8737b02 [peers: [], term: 12, commit: 1499, applied: 0, lastindex: 1499, lastterm: 12]"}
{"level":"warn","ts":"2025-09-09T04:58:30.212150Z","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2025-09-09T04:58:30.212220Z","caller":"mvcc/kvstore.go:348","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":849}
{"level":"info","ts":"2025-09-09T04:58:30.221630Z","caller":"mvcc/kvstore.go:425","msg":"kvstore restored","current-rev":1334}
{"level":"info","ts":"2025-09-09T04:58:30.221690Z","caller":"etcdserver/server.go:628","msg":"restore consistentIndex","index":1497}
{"level":"info","ts":"2025-09-09T04:58:30.221781Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2025-09-09T04:58:30.222165Z","caller":"etcdserver/corrupt.go:96","msg":"starting initial corruption check","local-member-id":"590f24c3d8737b02","timeout":"7s"}
{"level":"info","ts":"2025-09-09T04:58:30.222443Z","caller":"etcdserver/corrupt.go:177","msg":"initial corruption checking passed; no corruption","local-member-id":"590f24c3d8737b02"}
{"level":"info","ts":"2025-09-09T04:58:30.222486Z","caller":"etcdserver/server.go:875","msg":"starting etcd server","local-member-id":"590f24c3d8737b02","local-server-version":"3.5.21","cluster-version":"to_be_decided"}
{"level":"info","ts":"2025-09-09T04:58:30.222570Z","caller":"etcdserver/server.go:775","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2025-09-09T04:58:30.222667Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
{"level":"info","ts":"2025-09-09T04:58:30.222727Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
{"level":"info","ts":"2025-09-09T04:58:30.222736Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
{"level":"info","ts":"2025-09-09T04:58:30.222948Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 switched to configuration voters=(6417388417594915586)"}
{"level":"info","ts":"2025-09-09T04:58:30.222992Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"5c709c924160fd6d","local-member-id":"590f24c3d8737b02","added-peer-id":"590f24c3d8737b02","added-peer-peer-urls":["https://192.168.138.245:2380"],"added-peer-is-learner":false}
{"level":"info","ts":"2025-09-09T04:58:30.223052Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","service":"","status":"SERVING"}
{"level":"info","ts":"2025-09-09T04:58:30.223076Z","caller":"membership/cluster.go:587","msg":"set initial cluster version","cluster-id":"5c709c924160fd6d","local-member-id":"590f24c3d8737b02","cluster-version":"3.5"}
{"level":"info","ts":"2025-09-09T04:58:30.223105Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2025-09-09T04:58:30.225380Z","caller":"embed/etcd.go:762","msg":"starting with client TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/server.crt, key = /etc/kubernetes/pki/etcd/server.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2025-09-09T04:58:30.225531Z","caller":"embed/etcd.go:633","msg":"serving peer traffic","address":"192.168.138.245:2380"}
{"level":"info","ts":"2025-09-09T04:58:30.225574Z","caller":"embed/etcd.go:603","msg":"cmux::serve","address":"192.168.138.245:2380"}
{"level":"info","ts":"2025-09-09T04:58:30.225690Z","caller":"embed/etcd.go:292","msg":"now serving peer/client/metrics","local-member-id":"590f24c3d8737b02","initial-advertise-peer-urls":["https://192.168.138.245:2380"],"listen-peer-urls":["https://192.168.138.245:2380"],"advertise-client-urls":["https://192.168.138.245:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.138.245:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"]}
{"level":"info","ts":"2025-09-09T04:58:30.225734Z","caller":"embed/etcd.go:908","msg":"serving metrics","address":"http://127.0.0.1:2381"}
{"level":"info","ts":"2025-09-09T04:58:31.713145Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 is starting a new election at term 12"}
{"level":"info","ts":"2025-09-09T04:58:31.713222Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 became pre-candidate at term 12"}
{"level":"info","ts":"2025-09-09T04:58:31.713262Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 received MsgPreVoteResp from 590f24c3d8737b02 at term 12"}
{"level":"info","ts":"2025-09-09T04:58:31.713281Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 became candidate at term 13"}
{"level":"info","ts":"2025-09-09T04:58:31.713325Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 received MsgVoteResp from 590f24c3d8737b02 at term 13"}
{"level":"info","ts":"2025-09-09T04:58:31.713336Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"590f24c3d8737b02 became leader at term 13"}
{"level":"info","ts":"2025-09-09T04:58:31.713345Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 590f24c3d8737b02 elected leader 590f24c3d8737b02 at term 13"}
{"level":"info","ts":"2025-09-09T04:58:31.713603Z","caller":"etcdserver/server.go:2144","msg":"published local member to cluster through raft","local-member-id":"590f24c3d8737b02","local-member-attributes":"{Name:controlplane ClientURLs:[https://192.168.138.245:2379]}","request-path":"/0/members/590f24c3d8737b02/attributes","cluster-id":"5c709c924160fd6d","publish-timeout":"7s"}
{"level":"info","ts":"2025-09-09T04:58:31.713625Z","caller":"embed/serve.go:124","msg":"ready to serve client requests"}
{"level":"info","ts":"2025-09-09T04:58:31.713675Z","caller":"embed/serve.go:124","msg":"ready to serve client requests"}
{"level":"info","ts":"2025-09-09T04:58:31.713983Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2025-09-09T04:58:31.714017Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
{"level":"info","ts":"2025-09-09T04:58:31.716170Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","service":"","status":"SERVING"}
{"level":"info","ts":"2025-09-09T04:58:31.716150Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","service":"","status":"SERVING"}
{"level":"info","ts":"2025-09-09T04:58:31.717057Z","caller":"embed/serve.go:275","msg":"serving client traffic securely","traffic":"grpc+http","address":"192.168.138.245:2379"}
{"level":"info","ts":"2025-09-09T04:58:31.717057Z","caller":"embed/serve.go:275","msg":"serving client traffic securely","traffic":"grpc+http","address":"127.0.0.1:2379"}

controlplane ~ ➜  k get pods
No resources found in default namespace.

-> 이후 etcd가 static pod이므로 자동으로 반영되어 etcd 서버 정상화, kubectl 명령어를 정상적으로 사용 가능함을 확인

2) The kube-api server stopped again! Check it out. Inspect the kube-api server logs and identify the root cause and fix the issue.
Run crictl ps -a command to identify the kube-api server container. Run crictl logs container-id command to view the logs.

controlplane ~ ➜  k get pods
The connection to the server controlplane:6443 was refused - did you specify the right host or port?

-> 오류 메세지 확인

controlplane ~ ✖ crictl ps -a | grep kube-api
037030b867bfb       6ba9545b2183e       5 seconds ago        Running             kube-apiserver            2                   2244560041c77       kube-apiserver-controlplane            kube-system
d144ccb8aed5f       6ba9545b2183e       42 seconds ago       Exited              kube-apiserver            1                   2244560041c77       kube-apiserver-controlplane            kube-system

controlplane ~ ➜  crictl logs 037030b867bfb
I0909 05:02:00.831961       1 options.go:249] external host was not specified, using 192.168.138.245
I0909 05:02:00.833844       1 server.go:147] Version: v1.33.0
I0909 05:02:00.833870       1 server.go:149] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0909 05:02:01.448673       1 shared_informer.go:350] "Waiting for caches to sync" controller="node_authorizer"
W0909 05:02:01.451769       1 logging.go:55] [core] [Channel #3 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:01.452589       1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
I0909 05:02:01.459149       1 shared_informer.go:350] "Waiting for caches to sync" controller="*generic.policySource[*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicy,*k8s.io/api/admissionregistration/v1.ValidatingAdmissionPolicyBinding,k8s.io/apiserver/pkg/admission/plugin/policy/validating.Validator]"
I0909 05:02:01.467286       1 plugins.go:157] Loaded 14 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,PodTopologyLabels,MutatingAdmissionPolicy,MutatingAdmissionWebhook.
I0909 05:02:01.467314       1 plugins.go:160] Loaded 13 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,ClusterTrustBundleAttest,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
I0909 05:02:01.467603       1 instance.go:233] Using reconciler: lease
W0909 05:02:01.471717       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:02.455774       1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:02.458327       1 logging.go:55] [core] [Channel #3 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:02.474858       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:04.030069       1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:04.120255       1 logging.go:55] [core] [Channel #3 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:04.274151       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:06.617976       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:06.630857       1 logging.go:55] [core] [Channel #3 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:07.025750       1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:10.675670       1 logging.go:55] [core] [Channel #3 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:11.296970       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:11.929960       1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:17.723838       1 logging.go:55] [core] [Channel #3 SubChannel #4]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:17.868372       1 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W0909 05:02:18.135735       1 logging.go:55] [core] [Channel #5 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
F0909 05:02:21.468573       1 instance.go:226] Error creating leases: error creating storage factory: context deadline exceeded

-> 에러메시지 확인 : "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
-> 인증서 이슈

controlplane /etc/kubernetes/manifests ➜  cat kube-apiserver.yaml | grep "crt"
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --etcd-cafile=/etc/kubernetes/pki/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt

-> 현재 kube-apiserver.yaml에 기재된 인증서 목록 확인
-> --etcd-cafile=/etc/kubernetes/pki/ca.crt 값의 경로가 /etcd 하위에 있는 값으로 변경 필요함 확인

controlplane kubernetes/pki/etcd ➜  ll
total 40
drwxr-xr-x 2 root root 4096 Sep  9 04:21 ./
drwxr-xr-x 3 root root 4096 Sep  9 04:21 ../
-rw-r--r-- 1 root root 1094 Sep  9 04:21 ca.crt
-rw------- 1 root root 1675 Sep  9 04:21 ca.key
-rw-r--r-- 1 root root 1123 Sep  9 04:21 healthcheck-client.crt
-rw------- 1 root root 1675 Sep  9 04:21 healthcheck-client.key
-rw-r--r-- 1 root root 1208 Sep  9 04:21 peer.crt
-rw------- 1 root root 1675 Sep  9 04:21 peer.key
-rw-r--r-- 1 root root 1208 Sep  9 04:21 server.crt
-rw------- 1 root root 1679 Sep  9 04:21 server.key

-> 실제 파일 경로 및 이름 확인

controlplane /etc/kubernetes/manifests ➜  vi kube-apiserver.yaml 

controlplane /etc/kubernetes/manifests ➜  k get pod
The connection to the server controlplane:6443 was refused - did you specify the right host or port?

controlplane /etc/kubernetes/manifests ✖ k get pods
The connection to the server controlplane:6443 was refused - did you specify the right host or port?

controlplane /etc/kubernetes/manifests ✖ crictl ps -a | grep api-server

controlplane /etc/kubernetes/manifests ✖ crictl ps -a | grep api
4169824c8dbdf       6ba9545b2183e       12 seconds ago      Running             kube-apiserver            0                   835ad386e4060       kube-apiserver-controlplane            kube-system

controlplane /etc/kubernetes/manifests ➜  crictl logs 4169824c8dbdf

-> kube-apiserver.yaml 파일 편집 이후 로그에 에러 없음 확인

controlplane /etc/kubernetes/manifests ➜  k get pods
No resources found in default namespace.

-> kube-apiserver 서버 정상화 및 kubectl 사용 가능 확인

Kubernetes Certificates API와 CSR 사용법

Certificates API란?

Certificates API는 Kubernetes에서 인증서 발급 신청(Request), 승인(Approval), 발급(Issue) 를 원활하게 처리하기 위한 리소스(오브젝트)와 메커니즘을 제공한다.
기본적으로 사용자 또는 Pod가 인증서를 신청하면, Kubernetes가 이를 관리해서 자동으로 인증서를 만들고 교체(갱신)할 수 있도록 돕는다.

Certificates API를 왜 쓸까?

신규 인증서가 필요할 때마다 매번 수동으로 파일을 만들고 서명, 배포하는 과정이 번거롭기 때문
Pod, 사용자가 보안이 필요한 서비스를 안전하게 실행할 수 있게 해줌
인증서 인증 기반(혹은 mTLS 기반) 서비스 접근 제어나 내부 통신 암호화에 사용

주요 리소스: CertificateSigningRequest (CSR)

CertificateSigningRequest (CSR)란?

인증서 발급을 Kubernetes API에 요청하기 위한 리소스 객체
흔히 csr라고도 부름
사용자가 인증서를 발급받기 위해 CSR 오브젝트를 만들면, Kubernetes가 이를 관리함

CSR 흐름

요청: 사용자가 인증서 서명 요청(CertificateSigningRequest, CSR)을 생성하여 Kubernetes에 제출
승인: 관리자가(혹은 자동화 시스템이) 해당 CSR 오브젝트를 "승인"
인증서 발급: Kubernetes가 자체 CA(클러스터 CA)를 이용해 인증서 발급
사용: 사용자/Pod는 발급받은 인증서를 사용하여 보안 통신, 인증 처리

Certificates API와 기타 솔루션

cert-manager: Certificates API를 더 쉽게 관리하고, 자동 갱신/발급/폐기까지 맡아주는 CNCF 오픈소스 툴
Certificates API는 Kubernetes 자체 CA를 사용하지만, cert-manager를 쓰면 Let's Encrypt 같은 외부 CA 연동까지 아주 편리하게 구현 가능

CSR 파일 생성 및 신청 절차

1. 개인키와 CSR 생성

아래 명령어로 개인키와 CSR 파일을 만든다.

openssl genrsa -out my-user.key 2048
openssl req -new -key my-user.key -out my-user.csr -subj "/CN=my-user"

my-user.key : 개인키 파일
my-user.csr : 인증서 서명 요청 파일 (CSR, Certificate Signing Request)

2. CSR 파일을 base64로 인코딩

아래 명령어로 파일을 base64로 변환한다. (한 줄로 출력)

base64 < my-user.csr | tr -d '\n'

결과값(길게 출력된 한 줄)을 복사한다.

3. CertificateSigningRequest YAML 작성

복사한 base64 문자열을 아래 YAML의 spec.request 필드에 붙여넣는다.

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: my-cert-request
spec:
  request: LS0tLS1CRUdJTiBDRVJUSUZJ...   # base64로 인코딩한 CSR 한 줄
  signerName: kubernetes.io/kube-apiserver-client
  usages:
    - digital signature
    - key encipherment
    - client auth

spec.request에는 반드시 따옴표 없는 한 줄 base64 데이터를 쓴다.

4. CSR 제출 및 승인

CSR 오브젝트를 클러스터에 생성한다.

kubectl create -f my-cert-request.yaml

CSR 목록을 확인한다.

kubectl get csr

CSR을 승인한다.

kubectl certificate approve my-cert-request

5. 발급된 인증서 다운로드

승인 후 아래 명령어로 발급된 인증서를 파일로 저장한다.

kubectl get csr my-cert-request -o jsonpath='{.status.certificate}' | base64 --decode > my-user.crt

my-user.crt : 발급된 인증서 파일
my-user.key : 처음 만든 개인키 파일
두 파일을 세트로 활용한다.

Certificates API 주요 활용

파드 및 서비스 간 mTLS(상호 TLS) 통신 자동화
노드 인증서 자동 갱신 및 부트스트랩 용도
cert-manager와 연동해 외부 CA 인증서도 Kubernetes에서 발급∙관리 가능

요약

Certificates API로 Kubernetes 클러스터 내 인증서 발급, 갱신, 관리를 자동화할 수 있다.
cert-manager 등 연동 시 외부 CA 인증서 관리도 쉽다.
파일 배포, 인증서 교체를 중앙에서 일괄 관리 가능하며 보안성도 높아진다.

모든 인증서 관련 작업은 모두 kube-apiserver 내 controller manager가 담당한다!

kubeconfig

1. kubeconfig 파일이란?

쿠버네티스 클러스터에 접근하기 위한 접속 설정 파일
kubectl, kubeadm, helm 등 쿠버네티스 클라이언트 툴이 클러스터와 통신할 때 참조
기본 위치: ~/.kube/config

(환경변수 KUBECONFIG 로 다른 경로 지정 가능)

vi ~/.bashrc
export KUBECONFIG="/path/to/your/kube-config-file"
source ~/.bashrc

즉, “어떤 클러스터에, 어떤 사용자 권한으로, 어떤 context에서 접근할지”를 정의한 설정 모음.

2. kubeconfig 파일의 주요 요소

kubeconfig는 보통 아래 3가지 구성 요소를 가지고 있음

1) clusters
• 연결할 쿠버네티스 API 서버 정보
• API server의 주소(server), CA 인증서(certificate-authority) 등이 포함

clusters:

name: my-cluster
cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://192.168.0.10:6443

2) users
• 어떤 사용자 인증 정보를 쓸지 정의
• 클라이언트 인증서, 토큰, exec plugin(OIDC 등) 사용 가능

users:

name: admin-user
user:
client-certificate: /etc/kubernetes/pki/admin.crt
client-key: /etc/kubernetes/pki/admin.key

3) contexts
• “클러스터 + 사용자 + 네임스페이스”의 조합
• 여러 개의 클러스터를 관리할 때 현재 어떤 조합을 쓸지 정하는 역할

contexts:

name: admin-context
context:
cluster: my-cluster
user: admin-user
namespace: default

3. 현재 사용되는 context 확인 및 전환

현재 context 확인

kubectl config current-context

context 리스트 확인

kubectl config get-contexts

context 전환

kubectl config use-context admin-context

기본 파일 변경

mv /root/my-kube-config /root/.kube/config

여러 개의 kubeconfig 파일 합치기

때로는 여러 개의 config 파일을 써야 할 때가 있다.

export KUBECONFIG=~/.kube/config:~/.kube/dev-config
kubectl config view --merge --flatten > ~/.kube/merged-config

5. CKA 시험에서 자주 쓰이는 부분

•	시험 환경에서 기본 제공되는 ~/.kube/config 를 반드시 활용
•	여러 클러스터가 있을 때 kubectl config use-context <context-name> 으로 전환
•	인증/접속 문제 발생 시 clusters / users / contexts 세 부분을 확인

👉 요약하면, kubeconfig는 쿠버네티스 접속을 위한 “주소록 + 계정정보 + 환경설정”

kubeconfig context 전환을 하는 이유

1) context란?

kubeconfig에는 여러 개의 클러스터와 사용자 정보가 들어갈 수 있음
그런데 kubectl은 항상 “현재 context” 기준으로 동작
즉, context는 <클러스터 + 사용자 + 네임스페이스> 의 조합

2) 왜 context 전환이 필요할까?

예를 들어:

클러스터가 2개 있음
- dev-cluster (개발용)
- prod-cluster (운영용)
사용자도 다름
- dev-admin
- prod-admin

이때 kubeconfig에 두 클러스터와 두 사용자 정보가 모두 들어있음
👉 그런데 kubectl get pods 를 입력했을 때, 어떤 클러스터에서 어떤 권한으로 실행할지를 정해야 함

그걸 결정하는 게 context이고, context 전환을 통해 kubectl이 바라보는 클러스터를 바꿔주는 것임

3) 실제 예시

현재 context 확인

kubectl config current-context
-> 출력: dev-context

pod 조회 → dev-cluster 기준으로 조회됨

kubectl get pods -n default

만약 운영 클러스터로 바꾸고 싶다면, prod 클러스터로 context 전환

kubectl config use-context prod-context

이제 같은 명령어를 쳐도 prod-cluster에서 실행됨

kubectl get pods -n default

4) 정리

context 전환 = kubectl이 어떤 클러스터/사용자/네임스페이스를 쓸지 바꾸는 것
이유:
- 여러 클러스터를 한 kubeconfig에서 관리하기 위해
- 동일한 클러스터라도 사용자/네임스페이스를 달리 쓰기 위해
시험에서는 kubectl config use-context 을 거의 필수적으로 사용하게 됨

Xabi

롱런하는 개발자!

이전 포스트

[Kubernetes] Cluster Maintenance (OS/Cluster Upgrades, Backup, Restore, ETCDCTL, ETCDUTL)

다음 포스트

[Kubernetes] Security (TLS, Certificates, CSR, KubeConfig, context)

kubernetes

Kubernetes에서 TLS 인증서와 개인키 보안 정리

1. 개요

2. TLS 인증서란?

2.1 인증서의 구성요소

3. 각 파일 상세 설명

3.1 CA(인증기관, ca.crt)

3.2 TLS 인증서(tls.crt)

3.3 TLS 개인키(tls.key)

4. TLS 파일이 실제로 사용되는 예시

4.1 HTTPS 통신 과정

4.2 Kubernetes에서 사용 예시

5. CA의 역할과 중요성

6. TLS 개인키(Private Key) 보안, 왜 중요한가?

6.1 해커에게 탈취 시 위험성

6.2 실제 노출 사례

6.3 보안 관리 방법

7. 요약 및 결론

👉 핵심 3줄 요약

관련 연습문제

Kubernetes Certificates API와 CSR 사용법

Certificates API란?

Certificates API를 왜 쓸까?

주요 리소스: CertificateSigningRequest (CSR)

CertificateSigningRequest (CSR)란?

CSR 흐름

Certificates API와 기타 솔루션

CSR 파일 생성 및 신청 절차

1. 개인키와 CSR 생성

2. CSR 파일을 base64로 인코딩

3. CertificateSigningRequest YAML 작성

4. CSR 제출 및 승인

5. 발급된 인증서 다운로드

Certificates API 주요 활용

요약

관련 연습문제

kubeconfig

1. kubeconfig 파일이란?

2. kubeconfig 파일의 주요 요소

3. 현재 사용되는 context 확인 및 전환

5. CKA 시험에서 자주 쓰이는 부분

kubeconfig context 전환을 하는 이유

[Kubernetes] Cluster Maintenance (OS/Cluster Upgrades, Backup, Restore, ETCDCTL, ETCDUTL)

[Kubernetes] Security (API Groups, Authorization, Roles, RoleBinding, RBAC, Service Accounts, Image Security, Network Policy, ..)

0개의 댓글