AEWS3 - 10주차 K8s 시크릿 관리

김성중·2025년 4월 7일

AWS EKS Workshop

목록 보기

10/12

가시다(gasida) 님이 진행하는 AEWS(Amazon EKS Workshop Study) 3기 과정으로 학습한 내용을 정리 또는 실습한 내용을 정리한 게시글입니다.10주차는 하시코프 유형욱님이 K8s Secret 관련 주제로 Vault 소개와 실습(macOS환경)을 진행해 주셨고 이를 정리한 글입니다.

❤️❤️❤️ 열혈한 강의를 하여주신 하시코프 유형욱님께 깊은 감사를 드립니다 ❤️❤️❤️

1. 실습환경 구성

1.1 Jenkins 설치

컨테이터 (Jenkins) : 호스트 OS 포트 노출(expose)로 접속 및 사용
Jenkins 컨테이너 기동

# 작업 디렉토리 생성 후 이동
mkdir cicd-labs
cd cicd-labs

# cicd-labs 작업 디렉토리 IDE(VSCODE 등)로 열어두기
code .

# 
cat <<EOT > docker-compose.yaml
services:

  jenkins:
    container_name: jenkins
    image: jenkins/jenkins
    restart: unless-stopped
    networks:
      - cicd-network
    ports:
      - "8080:8080"
      - "50000:50000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - jenkins_home:/var/jenkins_home

volumes:
  jenkins_home:

networks:
  cicd-network:
    driver: bridge
EOT


# 배포
docker compose up -d
[+] Running 12/12
 ✔ jenkins Pulled                                                                                                                                               24.2s 
   ✔ e4b8da63e67a Pull complete                                                                                                                                  1.0s 
   ✔ cb6b359ad2fe Pull complete                                                                                                                                  1.0s 
   ✔ 08f6bab0cb52 Pull complete                                                                                                                                  6.5s 
   ✔ 599265d0bee4 Pull complete                                                                                                                                 20.9s 
   ✔ 8f7a49bcaa0e Pull complete                                                                                                                                 19.5s 
   ✔ ec2511b50f87 Pull complete                                                                                                                                  1.3s 
   ✔ 74901fce1a6a Pull complete                                                                                                                                  1.1s 
   ✔ d15499e83589 Pull complete                                                                                                                                  1.0s 
   ✔ f4657d27c8ae Pull complete                                                                                                                                 19.6s 
   ✔ 2a5996dba6d5 Pull complete                                                                                                                                  1.1s 
   ✔ e184c2818798 Pull complete                                                                                                                                 19.8s 
[+] Running 3/3
 ✔ Network cicd-labs_cicd-network   Created                                                                                                                      0.0s 
 ✔ Volume "cicd-labs_jenkins_home"  Created                                                                                                                      0.0s 
 ✔ Container jenkins                Started                                                                                                                      0.5s 
docker compose ps
NAME      IMAGE             COMMAND                  SERVICE   CREATED          STATUS          PORTS
jenkins   jenkins/jenkins   "/usr/bin/tini -- /u…"   jenkins   32 seconds ago   Up 32 seconds   0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp

# 기본 정보 확인
for i in jenkins ; do echo ">> container : $i <<"; docker compose exec
$i sh -c "whoami && pwd"; echo; done
>> container : jenkins <<
jenkins
/
# 도커를 이용하여 각 컨테이너로 접속
docker compose exec jenkins bash
jenkins@8dfb955f0f5a:/$ ls
bin  boot  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
exit

Jenkins 컨테이너 초기 설정

# Jenkins 초기 암호 확인
docker compose exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
d770512bf50141ef942cd7620954b80a

# Jenkins 웹 접속 주소 확인 : 계정 / 암호 입력 >> admin / qwe123
open "http://127.0.0.1:8080" # macOS
웹 브라우저에서 http://127.0.0.1:8080 접속 # Windows

# (참고) 로그 확인 : 플러그인 설치 과정 확인
docker compose logs jenkins -f
jenkins  | Running from: /usr/share/jenkins/jenkins.war
jenkins  | webroot: /var/jenkins_home/war
jenkins  | 2025-04-10 22:04:47.117+0000 [id=1]  INFO    winstone.Logger#logInternal: Beginning extraction from war file
jenkins  | 2025-04-10 22:04:47.555+0000 [id=1]  WARNING o.e.j.ee9.nested.ContextHandler#setContextPath: Empty contextPath
jenkins  | 2025-04-10 22:04:47.582+0000 [id=1]  INFO    org.eclipse.jetty.server.Server#doStart: jetty-12.0.19; built: 2025-04-01T13:28:40.005Z; git: 23436cd577b4238238eb192683dfc2b1476c6b87; jvm 17.0.14+7
jenkins  | 2025-04-10 22:04:47.783+0000 [id=1]  INFO    o.e.j.e.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.ee9.jsp.JettyJspServlet
jenkins  | 2025-04-10 22:04:47.805+0000 [id=1]  INFO    o.e.j.s.DefaultSessionIdManager#doStart: Session workerName=node0
jenkins  | 2025-04-10 22:04:48.070+0000 [id=1]  INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
jenkins  | 2025-04-10 22:04:48.135+0000 [id=1]  INFO    o.e.j.s.handler.ContextHandler#doStart: Started 
....
jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
jenkins  | 2025-04-10 22:12:01.815+0000 [id=257]        INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
jenkins  | 2025-04-10 22:12:01.816+0000 [id=93] INFO    h.m.UpdateCenter$CompleteBatchJob#run: Completed installation of 88 plugins in 1 min 58 sec

1.2 Kind로 K8S 설치

기본정보 확인

# 클러스터 배포 전 확인
docker ps

# Create a cluster with kind
MyIP=<각자 자신의 PC IP>
MyIP=192.168.0.254

# cicd-labs 디렉터리에서 아래 파일 작성
cat > kind-3node.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: "192.168.0.254" # $MyIP로 설정하셔도 됩니다.
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
  - containerPort: 30001
    hostPort: 30001
  - containerPort: 30002
    hostPort: 30002
  - containerPort: 30003
    hostPort: 30003
  - containerPort: 30004
    hostPort: 30004
  - containerPort: 30005
    hostPort: 30005
  - containerPort: 30006
    hostPort: 30006
- role: worker
- role: worker
EOF
kind create cluster --config kind-3node.yaml --name myk8s --image kindest/node:v1.32.2
Creating cluster "myk8s" ...
 ✓ Ensuring node image (kindest/node:v1.32.2) 🖼 
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-myk8s"
You can now use your cluster with:

kubectl cluster-info --context kind-myk8s

Have a nice day! 👋
# 확인
kind get nodes --name myk8s
myk8s-worker
myk8s-worker2
myk8s-control-plane

kubens default
Context "kind-myk8s" modified.
Active namespace is "default".

# kind 는 별도 도커 네트워크 생성 후 사용 : 기본값 172.18.0.0/16
docker network ls
NETWORK ID     NAME                     DRIVER    SCOPE
4cbcf5d6ed31   bridge                   bridge    local
ed7574143b6e   cicd-labs_cicd-network   bridge    local
88153fefe41f   host                     host      local
4a1bac68fb38   kind                     bridge    local
6e2fe0aabee3   none                     null      local

docker inspect kind | jq
[
  {
    "Name": "kind",
    "Id": "4a1bac68fb3814234d8ace297f7410f75b157df2ee87eb6d8b2bd8cbfac7a175",
    "Created": "2025-03-29T07:40:25.439081048Z",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv4": true,
    "EnableIPv6": true,
    "IPAM": {
      "Driver": "default",
      "Options": {},
      "Config": [
        {
          "Subnet": "172.19.0.0/16",
          "Gateway": "172.19.0.1"
        },
        {
          "Subnet": "fc00:f853:ccd:e793::/64",
          "Gateway": "fc00:f853:ccd:e793::1"
        }
      ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
      "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {
      "128365a166f5096ec30ee1eb0561e58ae6eb9ae98ebd51cba451f8267348dc1f": {
        "Name": "myk8s-control-plane",
        "EndpointID": "1eabfaf860ac8781d6de543770fc6354862632a391b820f726fffda9ddcf8712",
        "MacAddress": "ea:9e:6d:60:59:63",
        "IPv4Address": "172.19.0.4/16",
        "IPv6Address": "fc00:f853:ccd:e793::4/64"
      },
      "6cd290d85e574cf531326608203b805a4d1cf625e35539b000c8dd6672dc0b29": {
        "Name": "myk8s-worker",
        "EndpointID": "4104a3e5cb42413234ba1597ef14c8e17df1b9f1ecc08546fab0ccdd94ae6e85",
        "MacAddress": "e6:22:76:e3:cd:e4",
        "IPv4Address": "172.19.0.2/16",
        "IPv6Address": "fc00:f853:ccd:e793::2/64"
      },
      "e14b3bc36f7da01e4858b05627bbfea7d5d24b0550c132abc40ea9bb67eb1ac1": {
        "Name": "myk8s-worker2",
        "EndpointID": "a14db4663da7ce0e27804d31dd8d0cbf70cae2c97852dbc8814ea24d6dcf567a",
        "MacAddress": "8a:7b:5d:cc:7b:02",
        "IPv4Address": "172.19.0.3/16",
        "IPv6Address": "fc00:f853:ccd:e793::3/64"
      }
    },
    "Options": {
      "com.docker.network.bridge.enable_ip_masquerade": "true",
      "com.docker.network.driver.mtu": "65535"
    },
    "Labels": {}
  }
]

# k8s api 주소 확인 : 어떻게 로컬에서 접속이 되는 걸까?
kubectl cluster-info
Kubernetes control plane is running at https://192.168.0.254:51857
CoreDNS is running at https://192.168.0.254:51857/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

# 노드 정보 확인 : CRI 는 containerd 사용
kubectl get node -o wide
NAME                  STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
myk8s-control-plane   Ready    control-plane   2m44s   v1.32.2   172.19.0.4    <none>        Debian GNU/Linux 12 (bookworm)   6.10.14-linuxkit   containerd://2.0.3
myk8s-worker          Ready    <none>          2m29s   v1.32.2   172.19.0.2    <none>        Debian GNU/Linux 12 (bookworm)   6.10.14-linuxkit   containerd://2.0.3
myk8s-worker2         Ready    <none>          2m29s   v1.32.2   172.19.0.3    <none>        Debian GNU/Linux 12 (bookworm)   6.10.14-linuxkit   containerd://2.0.3

# 파드 정보 확인 : CNI 는 kindnet 사용
kubectl get pod -A -o wide
NAMESPACE            NAME                                          READY   STATUS    RESTARTS   AGE     IP           NODE                  NOMINATED NODE   READINESS GATES
kube-system          coredns-668d6bf9bc-c5hfz                      1/1     Running   0          3m12s   10.244.0.3   myk8s-control-plane   <none>           <none>
kube-system          coredns-668d6bf9bc-qcbsk                      1/1     Running   0          3m12s   10.244.0.2   myk8s-control-plane   <none>           <none>
kube-system          etcd-myk8s-control-plane                      1/1     Running   0          3m20s   172.19.0.4   myk8s-control-plane   <none>           <none>
kube-system          kindnet-2zqzf                                 1/1     Running   0          3m13s   172.19.0.4   myk8s-control-plane   <none>           <none>
kube-system          kindnet-lqjkz                                 1/1     Running   0          3m7s    172.19.0.3   myk8s-worker2         <none>           <none>
kube-system          kindnet-mhn4s                                 1/1     Running   0          3m7s    172.19.0.2   myk8s-worker          <none>           <none>
kube-system          kube-apiserver-myk8s-control-plane            1/1     Running   0          3m20s   172.19.0.4   myk8s-control-plane   <none>           <none>
kube-system          kube-controller-manager-myk8s-control-plane   1/1     Running   0          3m20s   172.19.0.4   myk8s-control-plane   <none>           <none>
kube-system          kube-proxy-kl79h                              1/1     Running   0          3m7s    172.19.0.3   myk8s-worker2         <none>           <none>
kube-system          kube-proxy-skx7l                              1/1     Running   0          3m7s    172.19.0.2   myk8s-worker          <none>           <none>
kube-system          kube-proxy-zrplw                              1/1     Running   0          3m13s   172.19.0.4   myk8s-control-plane   <none>           <none>
kube-system          kube-scheduler-myk8s-control-plane            1/1     Running   0          3m20s   172.19.0.4   myk8s-control-plane   <none>           <none>
local-path-storage   local-path-provisioner-7dc846544d-f6bl2       1/1     Running   0          3m12s   10.244.0.4   myk8s-control-plane   <none>           <none>

# 네임스페이스 확인 >> 도커 컨테이너에서 배운 네임스페이스와 다릅니다!
kubectl get namespaces
NAME                 STATUS   AGE
default              Active   3m47s
kube-node-lease      Active   3m47s
kube-public          Active   3m47s
kube-system          Active   3m47s
local-path-storage   Active   3m43s

# 컨트롤플레인/워커 노드(컨테이너) 확인 : 도커 컨테이너 이름은 myk8s-control-plane , myk8s-worker/worker-2 임을 확인
docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                                                                 NAMES
6cd290d85e57   kindest/node:v1.32.2   "/usr/local/bin/entr…"   4 minutes ago    Up 4 minutes                                                                          myk8s-worker
e14b3bc36f7d   kindest/node:v1.32.2   "/usr/local/bin/entr…"   4 minutes ago    Up 4 minutes                                                                          myk8s-worker2
128365a166f5   kindest/node:v1.32.2   "/usr/local/bin/entr…"   4 minutes ago    Up 4 minutes    0.0.0.0:30000-30006->30000-30006/tcp, 192.168.0.254:51857->6443/tcp   myk8s-control-plane
8dfb955f0f5a   jenkins/jenkins        "/usr/bin/tini -- /u…"   21 minutes ago   Up 21 minutes   0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp                      jenkins
docker images

# 디버그용 내용 출력에 ~/.kube/config 권한 인증 로드
kubectl get pod -v6
I0411 07:26:20.973604   73279 loader.go:402] Config loaded from file:  /Users/sjkim/.kube/config
I0411 07:26:20.974632   73279 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0411 07:26:20.974640   73279 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0411 07:26:20.974643   73279 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0411 07:26:20.974644   73279 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0411 07:26:20.996491   73279 round_trippers.go:560] GET https://192.168.0.254:51857/api/v1/namespaces/default/pods?limit=500 200 OK in 18 milliseconds
No resources found in default namespace.

# kube config 파일 확인
cat ~/.kube/config

1.3 ArgoCD 배포

Argo CD 설치

# 네임스페이스 생성 및 파라미터 파일 작성
cd cicd-labs

kubectl create ns argocd
cat <<EOF > argocd-values.yaml
dex:
  enabled: false

server:
  service:
    type: NodePort
    nodePortHttps: 30002
  extraArgs:
    - --insecure  # HTTPS 대신 HTTP 사용
EOF

# 설치
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd --version 7.8.13 -f argocd-values.yaml --namespace argocd
NAME: argocd
LAST DEPLOYED: Fri Apr 11 07:29:57 2025
NAMESPACE: argocd
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
In order to access the server UI you have the following options:

1. kubectl port-forward service/argocd-server -n argocd 8080:443

    and then open the browser on http://localhost:8080 and accept the certificate

2. enable ingress in the values file `server.ingress.enabled` and either
      - Add the annotation for ssl passthrough: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-1-ssl-passthrough
      - Set the `configs.params."server.insecure"` in the values file and terminate SSL at your ingress: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-2-multiple-ingress-objects-and-hosts


After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by running:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

(You should delete the initial secret afterwards as suggested by the Getting Started Guide: https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli)

# 확인
kubectl get pod,svc,ep,secret,cm -n argocd
NAME                                                   READY   STATUS    RESTARTS   AGE
pod/argocd-application-controller-0                    1/1     Running   0          93s
pod/argocd-applicationset-controller-cccb64dc8-9n4hz   1/1     Running   0          93s
pod/argocd-notifications-controller-7cd4d88cd4-28b5x   1/1     Running   0          93s
pod/argocd-redis-6c5698fc46-xp8n6                      1/1     Running   0          93s
pod/argocd-repo-server-5f6c4f4cf4-p7fdv                1/1     Running   0          93s
pod/argocd-server-7cb958f5fb-8qblv                     1/1     Running   0          93s

NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/argocd-applicationset-controller   ClusterIP   10.96.69.180    <none>        7000/TCP                     93s
service/argocd-redis                       ClusterIP   10.96.220.240   <none>        6379/TCP                     93s
service/argocd-repo-server                 ClusterIP   10.96.49.203    <none>        8081/TCP                     93s
service/argocd-server                      NodePort    10.96.72.188    <none>        80:30080/TCP,443:30002/TCP   93s

NAME                                         ENDPOINTS                         AGE
endpoints/argocd-applicationset-controller   10.244.1.3:7000                   93s
endpoints/argocd-redis                       10.244.1.2:6379                   93s
endpoints/argocd-repo-server                 10.244.2.4:8081                   93s
endpoints/argocd-server                      10.244.2.5:8080,10.244.2.5:8080   93s

NAME                                  TYPE                 DATA   AGE
secret/argocd-initial-admin-secret    Opaque               1      92s
secret/argocd-notifications-secret    Opaque               0      94s
secret/argocd-redis                   Opaque               1      96s
secret/argocd-secret                  Opaque               3      94s
secret/sh.helm.release.v1.argocd.v1   helm.sh/release.v1   1      2m3s

NAME                                      DATA   AGE
configmap/argocd-cm                       9      94s
configmap/argocd-cmd-params-cm            32     94s
configmap/argocd-gpg-keys-cm              0      94s
configmap/argocd-notifications-cm         1      94s
configmap/argocd-rbac-cm                  4      94s
configmap/argocd-redis-health-configmap   2      94s
configmap/argocd-ssh-known-hosts-cm       1      94s
configmap/argocd-tls-certs-cm             0      94s
configmap/kube-root-ca.crt                1      3m31s

kubectl get crd | grep argo
applications.argoproj.io      2025-04-10T22:30:27Z
applicationsets.argoproj.io   2025-04-10T22:30:28Z
appprojects.argoproj.io       2025-04-10T22:30:27Z

kubectl get appproject -n argocd -o yaml
apiVersion: v1
items:
- apiVersion: argoproj.io/v1alpha1
  kind: AppProject
  metadata:
    creationTimestamp: "2025-04-10T22:30:29Z"
    generation: 1
    name: default
    namespace: argocd
    resourceVersion: "1457"
    uid: b8160841-af21-421d-8fe0-3d5a926bce72
  spec:
    clusterResourceWhitelist:
    - group: '*'
      kind: '*'
    destinations:
    - namespace: '*'
      server: '*'
    sourceRepos:
    - '*'
  status: {}
kind: List
metadata:
  resourceVersion: ""

# configmap
kubectl get cm -n argocd argocd-cm -o yaml
apiVersion: v1
data:
  admin.enabled: "true"
  application.instanceLabelKey: argocd.argoproj.io/instance
  application.sync.impersonation.enabled: "false"
  exec.enabled: "false"
  server.rbac.log.enforce.enable: "false"
  statusbadge.enabled: "false"
  timeout.hard.reconciliation: 0s
  timeout.reconciliation: 180s
  url: https://argocd.example.com
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2025-04-10T22:30:27Z"
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.14.7
    helm.sh/chart: argo-cd-7.8.13
  name: argocd-cm
  namespace: argocd
  
kubectl get cm -n argocd argocd-rbac-cm -o yaml
apiVersion: v1
data:
  policy.csv: ""
  policy.default: ""
  policy.matchMode: glob
  scopes: '[groups]'
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2025-04-10T22:30:27Z"
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.14.7
    helm.sh/chart: argo-cd-7.8.13
  name: argocd-rbac-cm
  namespace: argocd

# 최초 접속 암호 확인
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ;echo
Po-147u8y9vTWqur

# Argo CD 웹 접속 주소 확인 : 초기 암호 입력 (admin 계정)
open "http://127.0.0.1:30002" # macOS
# Windows OS경우 직접 웹 브라우저에서 http://127.0.0.1:30002 접속

2. Vault 개요

2.1 Valut란

참조 : What is Vault?
HashiCorp Vault는 신원 기반(identity-based)의 시크릿 및 암호화 관리 시스템입니다. 이 시스템은 인증(authentication) 및 인가(authorization) 방법을 통해 암호화 서비스를 제공하여 비밀에 대한 안전하고 감사 가능하며 제한된 접근을 보장합니다.
시크릿(Secret)이란 접근을 철저히 통제하고자 하는 모든 것을 의미하며, 예를 들어 토큰, API 키, 비밀번호, 암호화 키 또는 인증서 등이 이에 해당합니다. Vault는 모든 시크릿에 대해 통합된 인터페이스를 제공하면서, 엄격한 접근 제어와 상세한 감사 로그 기록 기능을 제공합니다.
외부 서비스용 API 키, 서비스 지향 아키텍처 간 통신을 위한 자격 증명 등은 플랫폼에 따라 누가 어떤 비밀에 접근했는지를 파악하기 어려울 수 있습니다. 여기에 키 롤링(교체), 안전한 저장, 상세한 감사 로그까지 추가하려면 별도의 커스텀 솔루션 없이는 거의 불가능합니다. Vault는 바로 이 지점에서 해결책을 제공합니다.
Vault는 클라이언트(사용자, 기계, 애플리케이션 등)를 검증하고 인가한 후에만 비밀이나 저장된 민감한 데이터에 접근할 수 있도록 합니다.

2.2 Vault의 동작방식

참조 : How does Valult work?
Vault는 주로 토큰(Token)을 기반으로 작동하며, 이 토큰은 클라이언트의 정책(Policy)과 연결되어 있습니다. 각 정책은 경로(path) 기반으로 설정되며, 정책 규칙은 클라이언트가 해당 경로에서 수행할 수 있는 작업과 접근 가능성을 제한합니다.
Vault에서는 토큰을 수동으로 생성해 클라이언트에 할당할 수도 있고, 클라이언트가 로그인하여 토큰을 직접 획득할 수도 있습니다.
아래 그림은 Vault의 핵심 워크플로우를 보여줍니다.
Vault의 핵심 워크플로우는 다음 네 단계로 구성됩니다:
1. 인증 (Authenticate): Vault에서 인증은 클라이언트가 Vault에 자신이 누구인지 증명할 수 있는 정보를 제공하는 과정입니다. 클라이언트가 인증 메서드를 통해 인증되면, 토큰이 생성되고 정책과 연결됩니다.
2. 검증 (Validation): Vault는 Github, LDAP, AppRole 등과 같은 신뢰할 수 있는 외부 소스를 통해 클라이언트를 검증합니다.
3. 인가 (Authorize): 클라이언트는 Vault의 보안 정책과 비교됩니다. 이 정책은 Vault 토큰을 사용하여 클라이언트가 접근할 수 있는 API 엔드포인트를 정의하는 규칙의 집합입니다. 정책은 Vault 내 특정 경로나 작업에 대한 접근을 허용하거나 거부하는 선언적 방식으로 권한을 제어합니다.
4. 접근 (Access): Vault는 클라이언트의 신원에 연관된 정책을 기반으로 토큰을 발급하여 비밀, 키, 암호화 기능 등에 대한 접근을 허용합니다. 클라이언트는 이후 작업에서 해당 Vault 토큰을 사용할 수 있습니다.

2.3 왜 Vault가 필요한가요?

오늘날 대부분의 기업은 자격 증명이 조직 전반에 걸쳐 무분별하게 퍼져 있습니다.
비밀번호, API 키, 자격 증명 등이 일반 텍스트로 앱 소스 코드, 설정 파일, 기타 여러 위치에 저장되어 있습니다. 자격 증명이 이처럼 여기저기 흩어져 있으면 누가 무엇에 접근하고 권한이 있는지를 명확히 파악하기 어렵고, 그로 인해 큰 부담이 따릅니다. 일반 텍스트로 자격 증명을 저장하면 내부 공격자든 외부 공격자든 악의적인 공격 가능성이 크게 증가합니다.
Vault는 이러한 문제를 해결하기 위해 설계되었습니다.
Vault는 이러한 모든 자격 증명을 한 곳에 중앙 집중화하여 정의함으로써, 자격 증명의 불필요한 노출을 줄입니다. 하지만 Vault는 여기서 멈추지 않고, 사용자, 애플리케이션, 시스템이 인증 및 명시적으로 인가된 후에만 리소스에 접근할 수 있도록 보장하며, 클라이언트의 모든 작업 기록을 추적하고 저장하는 감사 로그 기능도 제공합니다.
Vault의 주요 기능은 다음과 같습니다:
1. 안전한 비밀 저장 (Secure Secret Storage): → Static 시크릿 : Vault는 임의의 key/value 형식의 시크릿을 저장할 수 있으며, 이 시크릿은 영구 저장소에 기록되기 전에 암호화됩니다. 따라서 저장소에 직접 접근하더라도 비밀을 열람할 수 없습니다.
Vault는 Disk, Consul 등 다양한 저장소를 지원합니다.
2. 동적 비밀 (Dynamic Secrets): Vault는 AWS나 SQL 데이터베이스와 같은 일부 시스템에 대해 요청 시 비밀을 동적으로 생성할 수 있습니다. 예를 들어, 애플리케이션이 S3 버킷에 접근해야 할 때 Vault에 자격 증명을 요청하면, Vault는 해당 권한을 가진 AWS 키쌍을 생성해줍니다. 이 동적 시크릿은 일정 시간이 지나면 자동으로 폐기됩니다.
3. 데이터 암호화 (Data Encryption): Vault는 데이터를 저장하지 않고 암호화 및 복호화를 수행할 수 있습니다.
이를 통해 보안 팀은 암호화 매개변수를 정의하고, 개발자는 암호화된 데이터를 SQL 데이터베이스 등 외부 저장소에 안전하게 저장할 수 있습니다.
4. 임대 및 갱신 (Leasing and Renewal): Vault에 저장된 모든 시크릿은 임대 기간(lease)이 설정되어 있으며, 이 기간이 끝나면 해당 비밀은 자동으로 폐기됩니다. 클라이언트는 내장된 갱신 API를 통해 임대를 연장할 수 있습니다.
5. 폐기 (Revocation): Vault는 비밀 폐기를 기본적으로 지원합니다. 단일 비밀뿐만 아니라 특정 사용자에 의해 읽힌 모든 비밀, 또는 특정 유형의 모든 비밀 등 비밀의 계층 구조 전체를 폐기할 수 있습니다. 이 기능은 키 롤링이나 침입 발생 시 시스템을 신속하게 차단하는 데 유용합니다.

3. Vault 기본 구조의 이해

3.1 Hotel Checkin == Valult 동작방식

Hashicorp의 Vault는 보안 생명주기 관리 및 시크릿 변경 자동화되어 있음
개별 ID 인증/인가를 통해 필요한 자격 증명을 동적으로 발급

3.2 Vault 동작 워크 플로우

인증(Authenticate) : 클라이언트가 누구인지 확인하고 인증이 완료되면 토큰을 생성한다.
검증(Validation) : 신뢰할 수 있는 외부 소스를 통해 클라이언트를 검증한다.
권한 부여(Authorize) : 클라이언트를 Vault 보안 정책과 일치시키고, 액세스 권한을 부여한다.
액세스(Access) : 클라이언트의 ID와 관련된 정책에 기반하여 토큰을 발급하여 비밀, 키, 암호화 기능 등에 접근할 수 있는 액세스 권한을 부여한다. 이후 클라이언트는 Vault 토큰을 사용할 수 있다.

출처: https://wlsdn3004.tistory.com/11 [IT DevOps 기록:티스토리]

3.3 Vault 내부 아키텍처

참조 : High-Level overview

4. Kubernetes에 Vault 설치

참조 : Vault on Kubernetes deployment guide, Helm chart
학습목표
- 실습용 K8s 환경(KinD)에 Vault를 Helm으로 설치
- 설치된 Vault 서버에 접속하고 UI/CLI로 기본 기능을 확인
- Vault의 동작 방식과 구조의 이해

4.1 Helm을 사용한 배포

네임스페이스 생성 및 Helm Repo 추가

# Create a Kubernetes namespace.
kubectl create namespace vault

# View all resources in a namespace.
kubectl get all --namespace vault

# Setup Helm repo
helm repo add hashicorp https://helm.releases.hashicorp.com

# Check that you have access to the chart.
helm search repo hashicorp/vault
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION                          
hashicorp/vault                         0.30.0          1.19.0          Official HashiCorp Vault Chart       
hashicorp/vault-secrets-gateway         0.0.2           0.1.0           A Helm chart for Kubernetes          
hashicorp/vault-secrets-operator        0.10.0          0.10.0          Official Vault Secrets Operator Chart

Helm Chart 설정 Values 설정 및 배포

cat <<EOF > override-values.yaml
global:
  enabled: true
  tlsDisable: true  # Disable TLS for demo purposes

server:
  image:
    repository: "hashicorp/vault"
    tag: "1.19.0"
  standalone:
    enabled: true
    replicas: 1
    config: |
      ui = true

      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_disable = 1
      }

      storage "file" {
        path = "/vault/data"
      }

  service:
    enabled: true
    type: NodePort
    port: 8200
    targetPort: 8200
    nodePort: 30000   # 🔥 Kind에서 열어둔 포트 중 하나 사용

injector:
  enabled: true
EOF

# Helm Install 실행
helm upgrade vault hashicorp/vault -n vault -f override-values.yaml --install
Release "vault" does not exist. Installing it now.
NAME: vault
LAST DEPLOYED: Fri Apr 11 09:34:52 2025
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://developer.hashicorp.com/vault/docs


Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault
  
# 네임스페이스 변경 : vault
kubens vault
Context "kind-myk8s" modified.
Active namespace is "vault".

# 배포확인
k get pods,svc,pvc

NAME                                        READY   STATUS    RESTARTS   AGE
pod/vault-0                                 0/1     Running   0          56s
pod/vault-agent-injector-56459c7545-zslsg   1/1     Running   0          56s

NAME                               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                         AGE
service/vault                      NodePort    10.96.3.62     <none>        8200:30000/TCP,8201:32187/TCP   56s
service/vault-agent-injector-svc   ClusterIP   10.96.250.85   <none>        443/TCP                         56s
service/vault-internal             ClusterIP   None           <none>        8200/TCP,8201/TCP               56s

NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/data-vault-0   Bound    pvc-20fbe519-1548-4f18-a239-aeb310c67155   10Gi       RWO            standard       <unset>                 56s

vault-0 pod 기동 불가
vault-0 pod describe 확인 시 Readiness probe failed 상태 지속 발생
log 확인결과 core: seal configuration missing, not initialized 지속 발생중

4.2 Vault 초기화 및 잠금해제

상태확인

# Vault Status 명령으로 Sealed 상태확인
kubectl exec -ti vault-0 -- vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            1.19.0
Build Date         2025-03-04T12:36:40Z
Storage Type       file
HA Enabled         false
command terminated with exit code 2

init-unseal.sh 을 사용하여 Vault Unseal 자동화 - Shamir Secret Sharing 방식이란?

cat <<EOF > init-unseal.sh
#!/bin/bash

# Vault Pod 이름
VAULT_POD="vault-0"

# Vault 명령 실행
VAULT_CMD="kubectl exec -ti \$VAULT_POD -- vault"

# 출력 저장 파일
VAULT_KEYS_FILE="./vault-keys.txt"
UNSEAL_KEY_FILE="./vault-unseal-key.txt"
ROOT_TOKEN_FILE="./vault-root-token.txt"

# Vault 초기화 (Unseal Key 1개만 생성되도록 설정)
\$VAULT_CMD operator init -key-shares=1 -key-threshold=1 | sed \$'s/\\x1b\\[[0-9;]*m//g' | tr -d '\r' > "\$VAULT_KEYS_FILE"

# Unseal Key / Root Token 추출
grep 'Unseal Key 1:' "\$VAULT_KEYS_FILE" | awk -F': ' '{print \$2}' > "\$UNSEAL_KEY_FILE"
grep 'Initial Root Token:' "\$VAULT_KEYS_FILE" | awk -F': ' '{print \$2}' > "\$ROOT_TOKEN_FILE"

# Unseal 수행
UNSEAL_KEY=\$(cat "\$UNSEAL_KEY_FILE")
\$VAULT_CMD operator unseal "\$UNSEAL_KEY"

# 결과 출력
echo "[🔓] Vault Unsealed!"
echo "[🔐] Root Token: \$(cat \$ROOT_TOKEN_FILE)"
EOF

# 실행 권한 부여
chmod +x init-unseal.sh

# 실행
./init-unseal.sh
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.19.0
Build Date      2025-03-04T12:36:40Z
Storage Type    file
Cluster Name    vault-cluster-8b16c381
Cluster ID      b35528c9-4160-9f5d-2739-e0b9ceb88088
HA Enabled      false
[🔓] Vault Unsealed!
[🔐] Root Token: hvs.lVDAL27sdG1y3c6VTCxYL8pT

vault status 명령을 사용하여 Unseal 되었는지 확인한다 → Sealed=false

kubectl exec -ti vault-0 -- vault status

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false  <-- 상태 변경 됨
Total Shares    1
Threshold       1
Version         1.19.0
Build Date      2025-03-04T12:36:40Z
Storage Type    file
Cluster Name    vault-cluster-8b16c381
Cluster ID      b35528c9-4160-9f5d-2739-e0b9ceb88088
HA Enabled      false

vault 상태 재확인

kubectl get all --namespace vault       

NAME                                        READY   STATUS    RESTARTS   AGE
pod/vault-0                                 1/1     Running   0          16m
pod/vault-agent-injector-56459c7545-zslsg   1/1     Running   0          16m

NAME                               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                         AGE
service/vault                      NodePort    10.96.3.62     <none>        8200:30000/TCP,8201:32187/TCP   16m
service/vault-agent-injector-svc   ClusterIP   10.96.250.85   <none>        443/TCP                         16m
service/vault-internal             ClusterIP   None           <none>        8200/TCP,8201/TCP               16m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           16m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-agent-injector-56459c7545   1         1         1       16m

NAME                     READY   AGE
statefulset.apps/vault   1/1     16m

(참고) UI에 접속해서 Unseal Key 입력도 가능
이제 Unseal이 마무리 되었으므로 Root Token 값을 사용하여 UI에 접속할 수 있습니다.
- 접속 : http://192.168.0.254:30000
- Root Token 입력 후 Vault UI 화면 - vault-root-token.txt 파일에서 획득

4.3 Vault CLI 설정

brew tap hashicorp/tap
brew install hashicorp/tap/vault
vault --version  # 설치 확인

# NodePort로 공개한 30000 Port로 설정
export VAULT_ADDR='http://192.168.0.254:30000'

# vault 상태확인
vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.19.0
Build Date      2025-03-04T12:36:40Z
Storage Type    file
Cluster Name    vault-cluster-8b16c381
Cluster ID      b35528c9-4160-9f5d-2739-e0b9ceb88088
HA Enabled      false

# Root Token으로 로그인
vault login

Token (will be hidden): 
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.lVDAL27sdG1y3c6VTCxYL8pT
token_accessor       D8wqAd0JtzGLjzX2SLRwbtvE
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

4.4 KV 시크릿 엔진 활성화 및 샘플 구성 → Static Secret

→ Vault KV version 2 엔진을 활성화하고 샘플 데이터를 저장합니다.
→ Version1 : KV 버전관리 불가 / Version2 : KV 버전관리 가능

Step 1. KV 엔진 활성화 및 샘플 데이터 추가

# KV v2 형태로 엔진 활성화
vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

# 샘플 시크릿 저장
vault kv put secret/sampleapp/config \
  username="demo" \
  password="p@ssw0rd"
======== Secret Path ========
secret/data/sampleapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2025-04-11T01:07:35.898436677Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
  
# 입력된 데이터 확인
vault kv get **secret/sampleapp/config**

======== Secret Path ========
secret/data/sampleapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2025-04-11T01:07:35.898436677Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
**password    p@ssw0rd
username    demo**

Step2. Vault UI : [Secrets Engine] 탭에 접속 후 [sampleapp - config] 접속하여 실제 저장된 Key / Value 확인
- username : demo
- password : p@ssw0rd
Step3. (참고) 경로 확인하는 명령 가이드

5. Vault Sidecar 연동

5.1 소개

참조 : Vault Agent Injector examples, Kubernetes에 Vault Agent(Sidecar) 수동 구성
실습 목표
- Vault Agent Sidecar 패턴을 이해하고, 앱(Pod)이 시크릿을 볼륨으로 마운트하여 사용하도록 구성한다.
- Vault Agent Injector 구성을 위해 Helm Chart(https://github.com/hashicorp/vault-helm) 에서 활성화 해야하며, https://github.com/hashicorp/vault-k8s을 활용한다.

5.2 요약(ChatGPT)

Vault Agent Injector는 Kubernetes Pod 내부에 Vault Agent를 자동으로 주입해주는 기능입니다. 이를 통해 어플리케이션이 Vault로부터 자동으로 비밀 정보를 받아올 수 있게 됩니다. 하지만 이를 사용하기 전에 몇 가지 사전 준비가 필요합니다.

1. Vault가 설치되어 있고, Kubernetes와 통합되어 있어야 합니다

Vault가 실행 중이어야 하고, Kubernetes 클러스터에 접근 가능해야 합니다.
Vault는 Kubernetes 인증 방식을 설정하고 있어야 하며, 이를 통해 서비스 어카운트를 기반으로 토큰을 발급받을 수 있습니다.

2. Vault Agent Injector가 클러스터에 배포되어 있어야 합니다

Injector는 Kubernetes에 배포되는 별도의 구성 요소입니다.
일반적으로 Helm Chart를 통해 배포하며, 이 컴포넌트가 있어야 Pod에 Vault Agent가 자동으로 주입됩니다.

3. Kubernetes 인증 방식이 활성화되어야 합니다

Vault에서 Kubernetes Auth Method를 활성화하고 구성해야 합니다.
이 설정을 통해 특정 서비스 어카운트에 Vault 접근 권한을 부여할 수 있습니다.

4. 정책과 역할이 정의되어 있어야 합니다

Vault에 접근할 수 있도록 적절한 Policy와 Role이 설정되어야 합니다.
예를 들어, 특정 서비스 어카운트가 특정 경로의 시크릿에만 접근할 수 있도록 제한할 수 있습니다.

5. 애플리케이션 Pod에 주입할 주석(annotation)을 추가해야 합니다

Vault Agent Injector는 특정 주석이 있는 Pod에 대해서만 Vault Agent를 주입합니다.

vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "example-role"

5.3 Vault Kubernetes Sidecar 아키텍처 및 워크플로우

5.4 Step1. Vault AppRole 방식 인증 구성

참조 : AppRole 인증이란?
인증 구성 및 정책 적용

# 1. AppRole 인증 방식 활성화
vault auth enable approle || echo "AppRole already enabled"
Success! Enabled approle auth method at: approle/

vault auth list
Path        Type       Accessor                 Description                Version
----        ----       --------                 -----------                -------
approle/    approle    auth_approle_5e875a9e    n/a                        n/a
token/      token      auth_token_ce15d494      token based credentials    n/a

# 2. 정책 생성
vault policy write sampleapp-policy - <<EOF
path "secret/data/sampleapp/*" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: sampleapp-policy

# 3. AppRole Role 생성
vault write auth/approle/role/sampleapp-role \
  token_policies="sampleapp-policy" \
  secret_id_ttl="1h" \
  token_ttl="1h" \
  token_max_ttl="4h"
Success! Data written to: auth/approle/role/sampleapp-role

# 4. Role ID 및 Secret ID 추출 및 저장
ROLE_ID=$(vault read -field=role_id auth/approle/role/sampleapp-role/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/sampleapp-role/secret-id)

echo "ROLE_ID: $ROLE_ID"
ROLE_ID: 6b26b5f7-9fd6-9fcb-cc98-89ff2adf2d20

echo "SECRET_ID: $SECRET_ID"
SECRET_ID: 916f2540-6bb3-020c-2c83-d1c0317f647e

# 5. 파일로 저장
mkdir -p approle-creds
echo "$ROLE_ID" > approle-creds/role_id.txt
echo "$SECRET_ID" > approle-creds/secret_id.txt

# 6. (옵션) Kubernetes Secret으로 저장
kubectl create secret generic vault-approle -n vault \
  --from-literal=role_id="${ROLE_ID}" \
  --from-literal=secret_id="${SECRET_ID}" \
  --save-config \
  --dry-run=client -o yaml | kubectl apply -f -
secret/vault-approle created

5.5 Step2. Vault Agent Sidecar 연동

Vault Agent는 vault-agent-config.hcl 설정을 통해 연결할 Vault의 정보와, Template 구성, 렌더링 주기, 참조할 Vault KV 위치정보 등을 정의한다.

1. Vault Agent 설정 파일 작성 및 생성 (vault-agent-config.hcl) - HCL

cat <<EOF | kubectl create configmap vault-agent-config -n vault --from-file=agent-config.hcl=/dev/stdin --dry-run=client -o yaml | kubectl apply -f -
vault {
  address = "http://vault.vault.svc:8200"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path = "/etc/vault/approle/role_id"
      secret_id_file_path = "/etc/vault/approle/secret_id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = {
      path = "/etc/vault-agent-token/token"
    }
  }
}

template_config {
  static_secret_render_interval = "20s"
}

template {
  destination = "/etc/secrets/index.html"
  contents = <<EOH
  <html>
  <body>
    <p>username: {{ with secret "secret/data/sampleapp/config" }}{{ .Data.data.username }}{{ end }}</p>
    <p>password: {{ with secret "secret/data/sampleapp/config" }}{{ .Data.data.password }}{{ end }}</p>
  </body>
  </html>
EOH
}
EOF
configmap/vault-agent-config created

2. 샘플 애플리케이션 + Sidecar 배포(수동방식)
- Nginx + Vault Agent 생성

kubectl apply -n vault -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-vault-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-vault-demo
  template:
    metadata:
      labels:
        app: nginx-vault-demo
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: html-volume
          mountPath: /usr/share/nginx/html
      - name: vault-agent-sidecar
        image: hashicorp/vault:latest
        args:
          - "agent"
          - "-config=/etc/vault/agent-config.hcl"
        volumeMounts:
        - name: vault-agent-config
          mountPath: /etc/vault
        - name: vault-approle
          mountPath: /etc/vault/approle
        - name: vault-token
          mountPath: /etc/vault-agent-token
        - name: html-volume
          mountPath: /etc/secrets
      volumes:
      - name: vault-agent-config
        configMap:
          name: vault-agent-config
      - name: vault-approle
        secret:
          secretName: vault-approle
      - name: vault-token
        emptyDir: {}
      - name: html-volume
        emptyDir: {}
EOF
deployment.apps/nginx-vault-demo created

3. SVC 생성

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx-vault-demo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30001 # Kind에서 설정한 Port
EOF
service/nginx-service created

4. 생성된 컨테이너 확인

# 파드 내에 사이드카 컨테이너 추가되어 2/2 확인
NAME                                READY   STATUS    RESTARTS   AGE
nginx-vault-demo-7776649597-sff46   2/2     Running   0          57s

kubectl describe pod -l app=nginx-vault-demo
...
Containers:
  nginx:
    Container ID:   containerd://067f02c254d53c94de7568437fe873aec498c157830d49447060658e9771ff1c
    Image:          nginx:latest
    Image ID:       docker.io/library/nginx@sha256:09369da6b10306312cd908661320086bf87fbae1b6b0c49a1f50ba531fef2eab
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 11 Apr 2025 12:55:24 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/nginx/html from html-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2m6sr (ro)
  vault-agent-sidecar:
    Container ID:  containerd://29a2fa378f2ec52b228264a8035c717dba1c6ba046305995f5ab95f8ac9cce6c
    Image:         hashicorp/vault:latest
    Image ID:      docker.io/hashicorp/vault@sha256:ee674e47dcf85849aadf255b5341f76c0e1a474bc5fa9be9cdfff2a2edf9a628
    Port:          <none>
    Host Port:     <none>
    Args:
      agent
      -config=/etc/vault/agent-config.hcl
    State:          Running
      Started:      Fri, 11 Apr 2025 12:55:46 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/secrets from html-volume (rw)
      /etc/vault from vault-agent-config (rw)
      /etc/vault-agent-token from vault-token (rw)
      /etc/vault/approle from vault-approle (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2m6sr (ro)
...


# 볼륨 마운트 확인
kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- ls -l /etc/vault-agent-token
total 4
-rw-r-----    1 vault    vault           95 Apr 11 03:55 token

kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- cat /etc/vault-agent-token/token ; echo
hvs.CAESIME1SLIjY5eIuIv85VlGOGwoa4osSaS3_wAr9f1TwON0Gh4KHGh2cy5BeGpGYmQ1Y3ppcHJKNlBtVUJLd2dKV1U

kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- ls -al /etc/vault
...
lrwxrwxrwx    1 root     root            23 Apr 11 03:55 agent-config.hcl -> ..data/agent-config.hcl
drwxrwxrwt    3 root     root           120 Apr 11 03:55 approle
kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- cat /etc/vault/agent-config.hcl
...

kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- ls -al /etc/vault/approle
...
lrwxrwxrwx    1 root     root            14 Apr 11 03:55 role_id -> ..data/role_id
lrwxrwxrwx    1 root     root            16 Apr 11 03:55 secret_id -> ..data/secret_id

kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- cat /etc/vault/approle/role_id ; echo
6b26b5f7-9fd6-9fcb-cc98-89ff2adf2d20

kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- cat /etc/vault/approle/secret_id ; echo
916f2540-6bb3-020c-2c83-d1c0317f647e

kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- ls -l /etc/secrets       
total 4
-rw-r--r--    1 vault    vault           94 Apr 11 03:55 index.html

kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- cat /etc/secrets/index.html
  <html>
  <body>
    <p>username: demo</p>
    <p>password: p@ssw0rd</p>
  </body>
  </html>
  
kubectl exec -it deploy/nginx-vault-demo -c nginx -- cat /usr/share/nginx/html/index.html
  <html>
  <body>
    <p>username: demo</p>
    <p>password: p@ssw0rd</p>
  </body>
  </html>


# 로그 확인
kubectl stern -l app=nginx-vault-demo -c vault-agent-sidecar
+ nginx-vault-demo-7776649597-sff46 › vault-agent-sidecar
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar Couldn't start vault with IPC_LOCK. Disabling IPC_LOCK, please use --cap-add IPC_LOCK
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar ==> Note: Vault Agent version does not match Vault server version. Vault Agent version: 1.19.1, Vault server version: 1.19.0
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar ==> Vault Agent started! Log data will stream in below:
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar ==> Vault Agent configuration:
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent.sink.file: creating file sink
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent.sink.file: file sink configured: path=/etc/vault-agent-token/token mode=-rw-r----- owner=100 group=1000
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar            Api Address 1: http://bufconn
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar                      Cgo: disabled
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar                Log Level: 
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar                  Version: Vault v1.19.1, built 2025-04-02T15:43:01Z
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar              Version Sha: aa75903ec499b2236da9e7bbbfeb7fd16fa4fd9d
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent.exec.server: starting exec server
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent.exec.server: no env templates or exec config, exiting
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent.auth.handler: starting auth handler
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent.auth.handler: authenticating
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent.sink.server: starting sink server
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent.template.server: starting template server
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.056Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.057Z [INFO]  agent: (runner) creating watcher
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.066Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.066Z [INFO]  agent.auth.handler: starting renewal process
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.067Z [INFO]  agent.template.server: template server received new token
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.067Z [INFO]  agent.sink.file: token written: path=/etc/vault-agent-token/token
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.067Z [INFO]  agent: (runner) stopping
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.067Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.067Z [INFO]  agent: (runner) creating watcher
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.067Z [INFO]  agent: (runner) starting
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.070Z [INFO]  agent.auth.handler: renewed auth token
nginx-vault-demo-7776649597-sff46 vault-agent-sidecar 2025-04-11T03:55:47.075Z [INFO]  agent: (runner) rendered "(dynamic)" => "/etc/secrets/index.html"

# mutating admission
kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io   
NAME                       WEBHOOKS   AGE
vault-agent-injector-cfg   1          3h27m

5. 실제 배포된 화면 확인
6. KV 값 변경 후 확인
- Create new 클릭 후 Passwd 값을 변경
- Secret 버전 2가 생성 됨
- Passwd가 동적으로 변경 됨
(참고) Annotation을 활용한 Vault Sidecar Injection - Docs
- Vault의 Kubernetes 인증 활성화 및 구성

# Kubernetes Auth Method 활성화
vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

# Kubernetes Auth Config 설정
vault write auth/kubernetes/config \
  token_reviewer_jwt="$(kubectl get secret $(kubectl get serviceaccount vault -n vault -o jsonpath='{.secrets[0].name}') -n vault -o jsonpath="{.data.token}" | base64 --decode)" \
  kubernetes_host="$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.server}')" \
  kubernetes_ca_cert="$(kubectl get secret $(kubectl get serviceaccount vault -n vault -o jsonpath='{.secrets[0].name}') -n vault -o jsonpath='{.data.ca\.crt}' | base64 --decode)"
Success! Data written to: auth/kubernetes/config

# 필요한 Policy 작성 (앞선 과정에서 만들었으므로 생략가능)
vault policy write sampleapp-policy - <<EOF
path "secret/data/sampleapp/*" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: sampleapp-policy

# Role 생성 (Injector가 로그인할 수 있도록)
vault write auth/kubernetes/role/sampleapp-role \
    bound_service_account_names="vault-ui-sa" \
    bound_service_account_namespaces="vault" \
    policies="sampleapp-policy" \
    ttl="24h"
Success! Data written to: auth/kubernetes/role/sampleapp-role

6. Jenkins + Vault (AppRole)

학습 목표
- Vault KV Store에 저장한 username, password을 Jenkins을 활용해서 획득하는 방안
- CI 파이프라인에서 정적(Static) 시크릿을 외부에 저장하고 관리할 경우 사용할 수 있습니다.

6.1 CI/CD 보안 고려사항

CI/CD 안티패턴
- AppRole 인증 - 보안 고려사항
  - Hard-coded secrets
  - Hard-coded Vault authentication
- AppRole 인증 - Anti Pattern

6.2 Vault - Jenkins Plugin with AppRole 인증방식

참조 : Jenkins CI/CD
공식문서 요약(ChatGPT)
- 젠킨스는 Vault에 시크릿으로 분류된 데이터를 필요로 하는 작업(job)을 실행해야 합니다. 젠킨스는 마스터 노드와 워커 노드를 가지고 있으며, 워커 노드는 짧은 시간 동안 실행되는 컨테이너 러너에서 작업을 실행합니다.
- 프로세스는 다음과 같습니다:
  1. 젠킨스 워커가 Vault에 인증
  2. Vault는 토큰을 반환
  3. 워커는 이 토큰을 사용해 작업에 해당하는 역할의 Wrapped SecretID를 요청
  4. Vault는 Wrapped SecretID를 반환
  5. 워커는 작업 러너를 생성하고, Wrapped SecretID를 변수로 전달
  6. 러너 컨테이너는 Wrapped SecretID의 unwrap을 요청
  7. Vault는 SecretID를 반환
  8. 러너는 RoleID와 SecretID를 사용해 Vault에 인증
  9. Vault는 필요한 시크릿 정보를 읽을 수 있는 정책이 포함된 토큰을 반환
  10. 러너는 이 토큰을 사용해 Vault에서 시크릿을 가져옴

6.3 실습

Step1. Jenkins에서 Vault Plugin 설치
1. Jenkins UI 접속
2. 상단 메뉴에서 Manage Jenkins → Plugins
3. Available 탭에서 Vault 검색
4. HashiCorp Vault Plugin 설치 후 Jenkins 재시작
Step2. Vault AppRole 정보 확인 ⇒ Secret ID는 1시간 만료이므로, 그냥 다시 생성해서, 해당 값을 젠킨스에 설정하고 빌드 실습 하자.
- Vault에서 발급된 ROLE_ID, SECRET_ID는 이전에 생성한 role_id.txt secret_id.txt 값을 참고하여 사용할 수 있습니다.

# Role ID 확인 및 Secret ID 신규 발급
ROLE_ID=$(vault read -field=role_id auth/approle/role/sampleapp-role/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/sampleapp-role/secret-id)

echo "ROLE_ID: $ROLE_ID"
ROLE_ID: 6b26b5f7-9fd6-9fcb-cc98-89ff2adf2d20

echo "SECRET_ID: $SECRET_ID"
SECRET_ID: f60e75ff-9e2b-2af3-05d2-355f5a2573a0

(참고) Role ID, Secret ID 획득방안

# Role ID
vault read auth/approle/role/<role-name>/role-id

# Secret ID
vault write -f auth/approle/role/<role-name>/secret-id

# 예시
vault read auth/approle/role/sampleapp-role/role-id
Key        Value
---        -----
role_id    678c0c6e-57df-bb23-1427-f6318843a514

Step3. Jenkins에서 Vault 설정 및 Credentials 추가
- Jenkins UI(admin/qwe123) → Manage Jenkins → Configure System
Step4. Jenkins Pipeline Job 생성
1. Jenkins UI → New Item → Pipeline 선택
2. jenkins-vault-kv 입력 후 생성
3. Jenkinsfile 작성
- Jenkinsfile 예시

pipeline {
  agent any

  environment {
    VAULT_ADDR = 'http://192.168.0.2:30000' // 실제 Vault 주소로 변경!!!
  }

  stages {
    stage('Read Vault Secret') {
      steps {
        withVault([
          vaultSecrets: [
            [
              path: 'secret/sampleapp/config',
              engineVersion: 2,
              secretValues: [
                [envVar: 'USERNAME', vaultKey: 'username'],
                [envVar: 'PASSWORD', vaultKey: 'password']
              ]
            ]
          ],
          configuration: [
            vaultUrl: "${VAULT_ADDR}",
            vaultCredentialId: 'vault-approle-creds'
          ]
        ]) {
          sh '''
            echo "Username from Vault: $USERNAME"
            echo "Password from Vault: $PASSWORD"
          '''
          script {
            echo "Username (env): ${env.USERNAME}"
            echo "Password (env): ${env.PASSWORD}"
          }
        }
      }
    }
  }
}

실행결과
유의사항
- KV Version1은 경로에 data을 넣고 Version2는 경로에 data을 넣지 않습니다! - 참고링크
  - Version 1 : secret/data/sampleapp/config
  - Version 2 : secret/sampleapp/config
- sh 블록 vs script 블록

7. ArgoCD + Vault Plugin

7.1 사전 준비

사전준비 : Argo CD 설치 및 기본 설정 - helm_chart → 앞에서 진행하여 생략!

7.2 Vault + ArgoCD Plugin

참조 : ArgoCD Vault Plugin,Secret Management with GitOps and Argo CD Vault Plugin
ArgoCD Vault Plugin 소개
- Argo CD에는 다양한 시크릿 관리 도구(HashiCorp Vault, IBM Cloud Secrets Manager, AWS Secrets Manager 등)플러그인을 통해 Kubernetes 리소스에 주입할 수 있도록 지원합니다.
- 플러그인을 통해 Operator 또는 CRD(Custom Resource Definition)에 의존하지 않고 GitOps와 Argo CD로 시크릿 관리 문제를 해결할 수 있습니다.
- 특히 Secret 뿐만 아니라, deployment, configMap 또는 기타 Kubernetes 리소스에도 사용할 수 있습니다.
설치/구성관련 내용 요약(ChatGPT) - Installing in Argo CD
Argo CD Vault Plugin의 공식 문서에서는 Argo CD에 플러그인을 설치하는 방법으로 네 가지를 제시하고 있습니다:

argocd-cm ConfigMap을 통한 설치:
- argocd-repo-server에 InitContainer를 추가하여 플러그인을 다운로드하고 설정합니다.
- 또는 플러그인이 사전 설치된 커스텀 이미지를 생성하여 사용합니다.
사이드카 컨테이너를 통한 설치:
- 사이드카 컨테이너를 추가하여 플러그인과 필요한 도구들을 포함시킵니다.
- 또는 플러그인이 사전 설치된 커스텀 사이드카 이미지를 생성하여 사용합니다.

이 중 사이드카 컨테이너를 활용한 방법은 Argo CD v2.4.0부터 도입된 최신 방식으로, 보안성과 유지보수 측면에서 권장됩니다.

7.3 실습

Step 1. ArgoCD Vault Plugin을 위한 Credentials 활성화 - AppRole 인증
- 이전 실습에서 획득한 Role_ID, Secret_ID활용

kubectl apply -f - <<EOF
kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
  namespace: argocd
type: Opaque
stringData:
  VAULT_ADDR: "http://vault.vault:8200"
  AVP_TYPE: "vault"
  AVP_AUTH_TYPE: "approle"
  AVP_ROLE_ID: 6b26b5f7-9fd6-9fcb-cc98-89ff2adf2d20 #Role_ID
  AVP_SECRET_ID: f60e75ff-9e2b-2af3-05d2-355f5a2573a0
#Secret_ID
EOF
secret/argocd-vault-plugin-credentials created

Step 2. ArgoCD Vault Plugin 설치 - Blog
- 참고사항
  ArgoCD Vault Plugin 설치 방법은 2가지가 있으며 현재는 Installation via a sidecar container 방식을 사용하는 것을 권장합니다.
  → 이번 스터디에서는 편의상 Helm으로 배포한 ArgoCD에 Kustomize을 활용해 기존 YAML에 대한 Patch을 적용합니다.
- ArgoCD Helm 설치시 Plugin 활성화 방안

git clone https://github.com/hyungwook0221/argocd-vault-plugin.git
cd argocd-vault-plugin/manifests/cmp-sidecar

# argocd 네임스페이스 설정
kubens argocd

# 생성될 메니페스트 파일에 대한 확인
kubectl kustomize .
apiVersion: v1
data:
  avp-helm.yaml: "---\napiVersion: argoproj.io/v1alpha1\nkind: ConfigManagementPlugin\nmetadata:\n
    \ name: argocd-vault-plugin-helm\nspec:\n  allowConcurrency: true\n\n  # Note:
    this command is run _before_ any Helm templating is done, therefore the logic
    is to check\n  # if this looks like a Helm chart\n  discover:\n    find:\n      command:\n
    \       - sh\n        - \"-c\"\n        - \"find . -name 'Chart.yaml' && find
    . -name 'values.yaml'\"\n  generate:\n    # **IMPORTANT**: passing `${ARGOCD_ENV_HELM_ARGS}`
    effectively allows users to run arbitrary code in the Argo CD \n    # repo-server
    (or, if using a sidecar, in the plugin sidecar). Only use this when the users
    are completely trusted. If\n    # possible, determine which Helm arguments are
    needed by your users and explicitly pass only those arguments.\n    command:\n
    \     - sh\n      - \"-c\"\n      - |\n        helm template $ARGOCD_APP_NAME
    -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |\n        argocd-vault-plugin
    generate -\n  lockRepo: false\n"
  avp-kustomize.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-kustomize
    spec:
      allowConcurrency: true

      # Note: this command is run _before_ anything is done, therefore the logic is to check
      # if this looks like a Kustomize bundle
      discover:
        find:
          command:
            - find
            - "."
            - -name
            - kustomization.yaml
      generate:
        command:
          - sh
          - "-c"
          - "kustomize build . | argocd-vault-plugin generate -"
      lockRepo: false
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
      generate:
        command:
          - argocd-vault-plugin
          - generate
          - "."
      lockRepo: false
kind: ConfigMap
metadata:
  name: cmp-plugin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-repo-server
  template:
    metadata:
      labels:
        app.kubernetes.io/name: argocd-repo-server
    spec:
      automountServiceAccountToken: true
      containers:
      - command:
        - /var/run/argocd/argocd-cmp-server
        envFrom:
        - secretRef:
            name: argocd-vault-plugin-credentials
        image: quay.io/argoproj/argocd:v2.7.9
        name: avp-helm
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          name: cmp-plugin
          subPath: avp-helm.yaml
        - mountPath: /usr/local/bin/argocd-vault-plugin
          name: custom-tools
          subPath: argocd-vault-plugin
      - command:
        - /var/run/argocd/argocd-cmp-server
        image: quay.io/argoproj/argocd:v2.7.9
        name: avp-kustomize
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          name: cmp-plugin
          subPath: avp-kustomize.yaml
        - mountPath: /usr/local/bin/argocd-vault-plugin
          name: custom-tools
          subPath: argocd-vault-plugin
      - command:
        - /var/run/argocd/argocd-cmp-server
        image: quay.io/argoproj/argocd:v2.7.9
        name: avp
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          name: cmp-plugin
          subPath: avp.yaml
        - mountPath: /usr/local/bin/argocd-vault-plugin
          name: custom-tools
          subPath: argocd-vault-plugin
      - image: quay.io/argoproj/argocd:v2.8.13
        name: repo-server
      initContainers:
      - args:
        - curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64
          -o argocd-vault-plugin && chmod +x argocd-vault-plugin && mv argocd-vault-plugin
          /custom-tools/
        command:
        - sh
        - -c
        env:
        - name: AVP_VERSION
          value: 1.18.0
        image: registry.access.redhat.com/ubi8
        name: download-tools
        volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
      volumes:
      - configMap:
          name: cmp-plugin
        name: cmp-plugin
      - emptyDir: {}
        name: custom-tools
        
# -k 옵션으로 kusomize 실행
kubectl apply -n argocd -k .
configmap/cmp-plugin created
Warning: resource deployments/argocd-repo-server is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
deployment.apps/argocd-repo-server configured

k exec -it -n vault vault-0 -- sh

# vault pod shell에 접속 후 root token으로 로그인
/ $ vault login
Token (will be hidden): 
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.lVDAL27sdG1y3c6VTCxYL8pT
token_accessor       D8wqAd0JtzGLjzX2SLRwbtvE
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
    
# 확인명령
/ $ vault read auth/kubernetes/role/argocd
No value found at auth/kubernetes/role/argocd

secret
deployment
argocd-repo-server
Step 3. 샘플 Application 배포하여 Vault와 동기화
- (참고) luafanti/spring-boot-debug-app:main 이미지는 amd64 CPU만 가능 (arm CPU는 기동 실패) - Link
Step 3-1) Application.yaml 작성
- GitHub에 저장된 Helm Repo을 배포하며, Helm 메니페스트 내에 변수로 치환된 값(username/password)을 CD 단계에서 Vault 통해서 읽고 렌더링하여 배포

kubectl apply -n argocd -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: infra/helm
    repoURL: https://github.com/hyungwook0221/spring-boot-debug-app
    targetRevision: main
    plugin:
      name: argocd-vault-plugin-helm
      env:
        - name: HELM_ARGS
          value: -f new-values.yaml
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

Step 3-2) Application 배포시 참조하는 new-values.yaml 확인 - GitHub

serviceAccount:
  create: true

image:
  repository: luafanti/spring-boot-debug-app
  tag: main
  pullPolicy: IfNotPresent

replicaCount: 1

resources:
  memoryRequest: 256Mi
  memoryLimit: 512Mi
  cpuRequest: 500m
  cpuLimit: 1

probes:
  liveness:
    initialDelaySeconds: 15
    path: /actuator/health/liveness
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 3
    periodSeconds: 5
  readiness:
    initialDelaySeconds: 15
    path: /actuator/health/readiness
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 3
    periodSeconds: 5

ports:
  http:
    name: http
    value: 8080
  management:
    name: management
    value: 8081

envs:
  - name: VAULT_SECRET_USER
    value: <path:secret/data/sampleapp/config#username>
  - name: VAULT_SECRET_PASSWORD
    value: <path:secret/data/sampleapp/config#password>

log:
  level:
    spring: "info"
    service: "info"

Step 3-3) 실제 배포시 적용된 화면
- ArgoCD Vault Plugin 적용된 화면 : [DETAILS] - [PARAMETERS]
- Application 배포화면
- Deployment에 적용된 env 값 확인 :
```
envs:
  - name: VAULT_SECRET_USER
    value: <path:secret/data/sampleapp/config#username>
  - name: VAULT_SECRET_PASSWORD
    value: <path:secret/data/sampleapp/config#password>
```
- ArgoCD App 삭제 kubectl delete applications demo
(참고) Unknown 에러 발생시 redis 파드 재배포 후 필요
- ArgoCD repo-server 재기동 → Redis 재기동 → Application 재생성

8. Vault Secrets Operator (VSO)

학습목표
- Vault Secrets Operator(이하, VSO)에 대한 주요 개념과 구성요소를 알아본다.
- VSO을 통해서 Vault을 통해 획득한 시크릿을 Kubernetes Secret을 자동으로 동기화하고 관리하는 방법을 알아본다.
VSO란? - Docs
Vault Secrets Operator(VSO)는 Kubernetes Secrets에서 Vault secrets 및 HCP Vault Secrets Apps를 네이티브하게 사용할 수 있도록 Pods에 제공해줍니다.
VSO 구성도 - 개발자(Developer) / 운영자(Operator)의 역할

8.1 Static Secrets (KV)

참조 : Manage Kubernetes native secrets with the Vault Secrets Operator
개요
- 운영자가 단일 Vault 정적 시크릿을 단일 Kubernetes 시크릿으로 동기화하는 데 필요한 구성입니다.
- 지원되는 시크릿 엔진: kv-v2, kv-v1
워크플로우 및 아키텍처

8.2 Dynamic Secrets

개요
- Vault의 Dynamic Secret Engine을 활용하여 동적으로 변경되는 Secrets을 K8s Secrets에 동기화
- 지원되는 시크릿 엔진: DB Credentials, Cloud Credentials(AWS, Azure, GCP 등)
시나리오
- Spring(Web Application) → DB 접근하기 위해서는 DB Credentials을 K8s Secrets으로 참조해야함
- 대상 DB에 접근하기 위한 DB Credentials을 Vault의 Dynamic Secrets 기능을 활용하여 주기적으로 변경하고 VSO을 통해 K8s Secets에 갱신(업데이트)
- Spring(Web Application)에서는 갱신된 DB Credentials 정보를 K8s Secret을 통해서 읽어오기 위해 재기동(rolloutRestartTargets 설정)
실습
Step1. VSO 배포를 위한 Chart Values 파일 작성

# vault-operator-values.yaml
defaultVaultConnection:
  enabled: true
  address: "http://vault.vault.svc.cluster.local:8200"
  skipTLSVerify: false
controller:
  manager:
    clientCache:
      persistenceModel: direct-encrypted
      storageEncryption:
        enabled: true
        mount: k8s-auth-mount
        keyName: vso-client-cache
        transitMount: demo-transit
        kubernetes:
          role: auth-role-operator
          serviceAccount: vault-secrets-operator-controller-manager
          tokenAudiences: ["vault"]

- Values 구성요소 분석
    - `defaultVaultConnection`: 기본 Vault 연결 설정을 구성
        - `address`: vault.vault.svc.cluster.local:8200 (Helm으로 설치된 Vault의 ClusterIP)
        - `skipTLSVerify: false`: TLS 검증 비활성화 안 함 (Vault에 TLS 적용된 경우 주의 필요)
    - **clientCache 설정**
        - `persistenceModel`: direct-encrypted: 암호화된 데이터를 직접 저장
        - `storageEncryption.enabled`: true: 암호화 기능 활성화
        - `transitMount: demo-transit`: 앞에서 Vault에 만든 Transit 엔진 mount path
        - `keyName: vso-client-cache`: Transit에서 사용하는 암호화 키
        - `mount: k8s-auth-mount`: Kubernetes 인증 방식에서 사용할 auth path
        - `kubernetes.role`: auth-role-operator 라는 Role에 지정된 정책으로 암복호화 가능
        - `serviceAccount`: VSO가 사용하는 ServiceAccount 이름
        - `tokenAudiences`: Vault 서버에서 사용하는 audience 설정
    - 위와 같은 설정으로 VSO는 Vault에 인증된 후, 동적/정적 시크릿을 Kubernetes 리소스로 가져오고, 가져온 시크릿 데이터를 자체적으로 **암호화해 캐시** 가능

Step2. VSO 배포

helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  -n vault-secrets-operator-system \
  --create-namespace \
  --values vault-operator-values.yaml
NAME: vault-secrets-operator
LAST DEPLOYED: Fri Apr 11 16:17:38 2025
NAMESPACE: vault-secrets-operator-system
STATUS: deployed
REVISION: 1  
#
kubectl get-all -n vault-secrets-operator-system
NAME                                                                                      NAMESPACE                      AGE
configmap/kube-root-ca.crt                                                                vault-secrets-operator-system  23s  
configmap/vault-secrets-operator-manager-config                                           vault-secrets-operator-system  23s  
endpoints/vault-secrets-operator-metrics-service                                          vault-secrets-operator-system  23s  
pod/vault-secrets-operator-controller-manager-7f67cd89fd-htprm                            vault-secrets-operator-system  23s  
secret/sh.helm.release.v1.vault-secrets-operator.v1                                       vault-secrets-operator-system  23s  
secret/vso-cc-storage-hmac-key                                                            vault-secrets-operator-system  5s   
serviceaccount/default                                                                    vault-secrets-operator-system  23s  
serviceaccount/vault-secrets-operator-controller-manager                                  vault-secrets-operator-system  23s  
service/vault-secrets-operator-metrics-service                                            vault-secrets-operator-system  23s  
deployment.apps/vault-secrets-operator-controller-manager                                 vault-secrets-operator-system  23s  
replicaset.apps/vault-secrets-operator-controller-manager-7f67cd89fd                      vault-secrets-operator-system  23s  
lease.coordination.k8s.io/b0d477c0.hashicorp.com                                          vault-secrets-operator-system  5s   
endpointslice.discovery.k8s.io/vault-secrets-operator-metrics-service-gbw8m               vault-secrets-operator-system  23s  
rolebinding.rbac.authorization.k8s.io/vault-secrets-operator-leader-election-rolebinding  vault-secrets-operator-system  23s  
role.rbac.authorization.k8s.io/vault-secrets-operator-leader-election-role                vault-secrets-operator-system  23s  
vaultauth.secrets.hashicorp.com/vault-secrets-operator-default-transit-auth               vault-secrets-operator-system  21s  
vaultconnection.secrets.hashicorp.com/default                                             vault-secrets-operator-system  21s 

#
kubectl get pod -n vault-secrets-operator-system      
NAME                                                         READY   STATUS    RESTARTS   AGE
vault-secrets-operator-controller-manager-7f67cd89fd-htprm   2/2     Running   0          42s

kubectl describe pod -n vault-secrets-operator-system                                       
...
Service Account:  vault-secrets-operator-controller-manager
...
Containers:
  kube-rbac-proxy:
    Container ID:  containerd://db3eae7b836fb4f1b4236c494c8fa96ada94769a6c602e1a150c75293a6a4162
    Image:         quay.io/brancz/kube-rbac-proxy:v0.18.1
  ...
  manager:
    Container ID:  containerd://1ab1545fb4bd86ac52d6c7609a3e962cd2d1a81daa9bbd9c82f79d9a0d8b6466
    Image:         hashicorp/vault-secrets-operator:0.10.0
...

#
kubectl rbac-tool lookup vault-secrets-operator-controller-manager
  SUBJECT                                   | SUBJECT TYPE   | SCOPE       | NAMESPACE                     | ROLE                                        | BINDING                                             
--------------------------------------------+----------------+-------------+-------------------------------+---------------------------------------------+-----------------------------------------------------
  vault-secrets-operator-controller-manager | ServiceAccount | ClusterRole |                               | vault-secrets-operator-proxy-role           | vault-secrets-operator-proxy-rolebinding            
  vault-secrets-operator-controller-manager | ServiceAccount | ClusterRole |                               | vault-secrets-operator-manager-role         | vault-secrets-operator-manager-rolebinding          
  vault-secrets-operator-controller-manager | ServiceAccount | Role        | vault-secrets-operator-system | vault-secrets-operator-leader-election-role | vault-secrets-operator-leader-election-rolebinding  
  
#
kubectl rolesum vault-secrets-operator-controller-manager -n vault-secrets-operator-system
ServiceAccount: vault-secrets-operator-system/vault-secrets-operator-controller-manager
Secrets:

Policies:
• [RB] vault-secrets-operator-system/vault-secrets-operator-leader-election-rolebinding ⟶  [R] vault-secrets-operator-system/vault-secrets-operator-leader-election-role
  Resource                    Name  Exclude  Verbs  G L W C U P D DC  
  configmaps                  [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  events                      [*]     [-]     [-]   ✖ ✖ ✖ ✔ ✖ ✔ ✖ ✖   
  leases.coordination.k8s.io  [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   


• [CRB] */vault-secrets-operator-manager-rolebinding ⟶  [CR] */vault-secrets-operator-manager-role
  Resource                                                Name  Exclude  Verbs  G L W C U P D DC  
  configmaps                                              [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖   
  daemonsets.apps                                         [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✔ ✖ ✖   
  deployments.apps                                        [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✔ ✖ ✖   
  events                                                  [*]     [-]     [-]   ✖ ✖ ✖ ✔ ✖ ✔ ✖ ✖   
  hcpauths.secrets.hashicorp.com                          [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  hcpauths.secrets.hashicorp.com/finalizers               [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  hcpauths.secrets.hashicorp.com/status                   [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   
  hcpvaultsecretsapps.secrets.hashicorp.com               [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  hcpvaultsecretsapps.secrets.hashicorp.com/finalizers    [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  hcpvaultsecretsapps.secrets.hashicorp.com/status        [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   
  rollouts.argoproj.io                                    [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✔ ✖ ✖   
  secrets                                                 [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔   
  secrettransformations.secrets.hashicorp.com             [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  secrettransformations.secrets.hashicorp.com/finalizers  [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  secrettransformations.secrets.hashicorp.com/status      [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   
  serviceaccounts                                         [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖   
  serviceaccounts/token                                   [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✖ ✖ ✖ ✖   
  statefulsets.apps                                       [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✔ ✖ ✖   
  vaultauthglobals.secrets.hashicorp.com                  [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  vaultauthglobals.secrets.hashicorp.com/finalizers       [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  vaultauthglobals.secrets.hashicorp.com/status           [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   
  vaultauths.secrets.hashicorp.com                        [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  vaultauths.secrets.hashicorp.com/finalizers             [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  vaultauths.secrets.hashicorp.com/status                 [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   
  vaultconnections.secrets.hashicorp.com                  [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  vaultconnections.secrets.hashicorp.com/finalizers       [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  vaultconnections.secrets.hashicorp.com/status           [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   
  vaultdynamicsecrets.secrets.hashicorp.com               [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  vaultdynamicsecrets.secrets.hashicorp.com/finalizers    [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  vaultdynamicsecrets.secrets.hashicorp.com/status        [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   
  vaultpkisecrets.secrets.hashicorp.com                   [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  vaultpkisecrets.secrets.hashicorp.com/finalizers        [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  vaultpkisecrets.secrets.hashicorp.com/status            [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   
  vaultstaticsecrets.secrets.hashicorp.com                [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖   
  vaultstaticsecrets.secrets.hashicorp.com/finalizers     [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✔ ✖ ✖ ✖   
  vaultstaticsecrets.secrets.hashicorp.com/status         [*]     [-]     [-]   ✔ ✖ ✖ ✖ ✔ ✔ ✖ ✖   


• [CRB] */vault-secrets-operator-proxy-rolebinding ⟶  [CR] */vault-secrets-operator-proxy-role
  Resource                                   Name  Exclude  Verbs  G L W C U P D DC  
  subjectaccessreviews.authorization.k8s.io  [*]     [-]     [-]   ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖   
  tokenreviews.authentication.k8s.io         [*]     [-]     [-]   ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖  
#
kubectl get vaultauth -n vault-secrets-operator-system vault-secrets-operator-default-transit-auth -o jsonpath='{.spec}' | jq
{
  "kubernetes": {
    "audiences": [
      "vault"
    ],
    "role": "auth-role-operator",
    "serviceAccount": "vault-secrets-operator-controller-manager",
    "tokenExpirationSeconds": 600
  },
  "method": "kubernetes",
  "mount": "k8s-auth-mount",
  "storageEncryption": {
    "keyName": "vso-client-cache",
    "mount": "demo-transit"
  },
  "vaultConnectionRef": "default"
}

kubectl get vaultconnection -n vault-secrets-operator-system default -o jsonpath='{.spec}' | jq
{
  "address": "http://vault.vault.svc.cluster.local:8200",
  "skipTLSVerify": false
}

Step3. PostgreSQL 설치 (Bitnami Helm Chart)

kubectl create ns postgres

helm repo add bitnami https://charts.bitnami.com/bitnami

helm upgrade --install postgres bitnami/postgresql \
  --namespace postgres \
  --set auth.audit.logConnections=true \
  --set auth.postgresPassword=secret-pass
Release "postgres" does not exist. Installing it now.
NAME: postgres
LAST DEPLOYED: Fri Apr 11 16:21:54 2025
NAMESPACE: postgres
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 16.6.3
APP VERSION: 17.4.0

Did you know there are enterprise versions of the Bitnami catalog? For enhanced secure software supply chain features, unlimited pulls from Docker, LTS support, or application customization, see Bitnami Premium or Tanzu Application Catalog. See https://www.arrow.com/globalecs/na/vendors/bitnami for more information.

** Please be patient while the chart is being deployed **

PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:

    postgres-postgresql.postgres.svc.cluster.local - Read/Write connection

To get the password for "postgres" run:

    export POSTGRES_PASSWORD=$(kubectl get secret --namespace postgres postgres-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

To connect to your database run the following command:

    kubectl run postgres-postgresql-client --rm --tty -i --restart='Never' --namespace postgres --image docker.io/bitnami/postgresql:17.4.0-debian-12-r15 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host postgres-postgresql -U postgres -d postgres -p 5432

    > NOTE: If you access the container using bash, make sure that you execute "/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash" in order to avoid the error "psql: local user with ID 1001} does not exist"

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace postgres svc/postgres-postgresql 5432:5432 &
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432

WARNING: The configured password will be ignored on new installation in case when previous PostgreSQL release was deleted through the helm command. In that case, old PVC will have an old password, and setting it through helm won't take effect. Deleting persistent volumes (PVs) will solve the issue.

WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
  - primary.resources
  - readReplicas.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/  
  
# psql 로그인 확인
kubectl exec -it -n postgres postgres-postgresql-0 -- sh -c 'PGPASSWORD=secret-pass psql -U postgres -h localhost'
psql (17.4)
Type "help" for help.

postgres=# 

kubectl exec -it -n postgres postgres-postgresql-0 -- sh -c "PGPASSWORD=secret-pass psql -U postgres -h localhost -c '\l'"
                                                     List of databases
   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | Locale | ICU Rules |   Access privileges   
-----------+----------+----------+-----------------+-------------+-------------+--------+-----------+-----------------------
 postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |        |           | 
 template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |        |           | =c/postgres          +
           |          |          |                 |             |             |        |           | postgres=CTc/postgres
 template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |        |           | =c/postgres          +
           |          |          |                 |             |             |        |           | postgres=CTc/postgres
(3 rows)

Step4. Kubernetes Auth Method 설정

#
kubectl exec -it vault-0 -n vault -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-----BEGIN CERTIFICATE-----
MIIDBTCCAe2gAwIBAgIIYJPloAf87sswDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
AxMKa3ViZXJuZXRlczAeFw0yNTA0MTAyMjE2MzlaFw0zNTA0MDgyMjIxMzlaMBUx
EzARBgNVBAMTCmt1YmVybmV0ZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC2LRnWiqx+Wq2BTu8WFO/gNFMvffRRW/TSMSAflUY64w0acXR+NM2XoeUW
j8tiz1LwGEGf7iSvmzN2t35x9TObLGU6NY6zEnO33ZOaqfhm1ZihHv5rle3+WiEl
6TFJIaWFmxb1MjPnCEqkMzaMQ70Nxuv5O6PbE6N7AKJ5+RWlJK/FQbyXLOVTg9qV
omCYSEDEC/+X8HSM/vEzTRunBcE7MLN7JSjF8PWy4GnPEzbG7cIhJ4xwHOMXm9/L
nMqq0+MdNuaR2W2xfDcRYUjZf7qZ7Q17spdbtTzMgX0fnKJTwLWjykch6JgK9zxf
8txrW5C276StC8WAB4GrzfskA0wVAgMBAAGjWTBXMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQk9qVRSGdZRO/jEOZGp+eOqHLymDAV
BgNVHREEDjAMggprdWJlcm5ldGVzMA0GCSqGSIb3DQEBCwUAA4IBAQAIkiGxISGX
ELIj52ZMWNlj4+bT/MqqUoG2NXMIVSDW/rjS6RwjLhm5ubeltfESSx8cQcvh2IfL
sVSXZGVQk9Cn4Ki7/AHRSGkA2yQ9VNy937/7IfcRfwUyBHUR6yC0B+0URs7kmvim
eNvEWv+ySxdpeEJX2na/lzhXurV6ftnXtGa/6YGaWqsrNG5/4v8KDLM1vbH8hwGT
JDKZQpl+U6q2fyXkei/txap4XhGc43mNCO/eFQWfhmCBuiL9xXJVjqhMb5T5CNiR
XIHqlzbAvHR4v+z8VgMOC5XrN7hlfcpNrfaV7p+jhdr2CIGRrRRUegWILF9rt0mD
YOBZ+7E4fpnu
-----END CERTIFICATE-----

# 해당 ca.crt 는 k8s 의 루트인증서 정보
kubectl exec -it vault-0 -n vault -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > vso-ca.crt
openssl x509 -in vso-ca.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6959158324709093067 (0x6093e5a007fceecb)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
...

# vault 서비스 어카운트의 token 확인
kubectl exec -it vault-0 -n vault -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IkMzZ1NValRPLWJJbDdWRU15ekk2bWdJMHhKT1RrR2M0Wk1nelR0blRWajAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzc1ODkwMjkyLCJpYXQiOjE3NDQzNTQyOTIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiNDE4NTY5YWYtNDIzYi00ZThhLTg5ZjctNzM3YjhmYzc5MjRmIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJ2YXVsdCIsIm5vZGUiOnsibmFtZSI6Im15azhzLXdvcmtlcjIiLCJ1aWQiOiJjZjkzZDFiNi00YzkxLTRjMzgtYjM1My04MjU2ODQ4NjZiZmYifSwicG9kIjp7Im5hbWUiOiJ2YXVsdC0wIiwidWlkIjoiOTVjY2NmMDUtZGYxMi00NTM1LTk4NWQtNzU5NmNiZjYyOTFkIn0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJ2YXVsdCIsInVpZCI6IjMyNjRmNDhlLWJmYTItNDZlZC1hMGE4LTg1MmY1M2E4ZWNkZSJ9LCJ3YXJuYWZ0ZXIiOjE3NDQzNTc4OTl9LCJuYmYiOjE3NDQzNTQyOTIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp2YXVsdDp2YXVsdCJ9.oPrk91-lvWHAGlIv75D9ONAag9i0Ptu7bxfKnblHCehm5tXHzuuycK78g9PJyOkD5nVjfH79izW4LIX_rS7kxrGYUpy21cD15PoEtSYK6R-xGx-XdS_O8_ChUtiniZSA9BKxbQbOj2N-rq6PnOa-LBllXYUcicZgf0-hvJD3OTEE7CaKrqxCIOaGZfnyXTU1ByrPgvGXsY-ZSGq8y7xB1v2jmOJ3wG3aGC-y2jcTPao9GOtvrmxT2CmvdgAz2HCqZeQXu5r-r0tcjuc1FflR0tTf8upjDR7DVjiA8J7sI0_NOhI7Lz1iNNnqHSj8Co92g1kJcOFzuVy44kKHEG8S2g  

# https://jwt.io/ 에서 token 디코드 확인 시 PAYLOAD 부분
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1775890292,
  "iat": 1744354292,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "jti": "418569af-423b-4e8a-89f7-737b8fc7924f",
  "kubernetes.io": {
    "namespace": "vault",
    "node": {
      "name": "myk8s-worker2",
      "uid": "cf93d1b6-4c91-4c38-b353-825684866bff"
    },
    "pod": {
      "name": "vault-0",
      "uid": "95cccf05-df12-4535-985d-7596cbf6291d"
    },
    "serviceaccount": {
      "name": "vault",
      "uid": "3264f48e-bfa2-46ed-a0a8-852f53a8ecde"
    },
    "warnafter": 1744357899
  },
  "nbf": 1744354292,
  "sub": "system:serviceaccount:vault:vault"
}

# vault SA 토큰 만료 기간 : 대략 1시간(3600초)+7초
k get pod -n vault vault-0 -o yaml
...
 - name: kube-api-access-vqkvw
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt



kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
----------------------------------------------------------------
# 최초 설치시 획득한 vault root token 사용. 
# 예시) hvs.lVDAL27sdG1y3c6VTCxYL8pT
vault login

# Kubernetes 인증 메서드 활성화
vault auth enable -path k8s-auth-mount kubernetes

#
vault auth list
Path               Type          Accessor                    Description                Version
----               ----          --------                    -----------                -------
approle/           approle       auth_approle_5e875a9e       n/a                        n/a
k8s-auth-mount/    kubernetes    auth_kubernetes_ef3a1d99    n/a                        n/a
kubernetes/        kubernetes    auth_kubernetes_65a23464    n/a                        n/a
token/             token         auth_token_ce15d494         token based credentials    n/a

# Kubernetes 클러스터 정보 구성
vault write auth/k8s-auth-mount/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
Success! Data written to: auth/k8s-auth-mount/config

# 설정 확인  
vault read auth/k8s-auth-mount/config
Key                                  Value
---                                  -----
disable_iss_validation               true
disable_local_ca_jwt                 false
issuer                               n/a
kubernetes_ca_cert                   -----BEGIN CERTIFICATE-----
MIIDBTCCAe2gAwIBAgIIYJPloAf87sswDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
AxMKa3ViZXJuZXRlczAeFw0yNTA0MTAyMjE2MzlaFw0zNTA0MDgyMjIxMzlaMBUx
EzARBgNVBAMTCmt1YmVybmV0ZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC2LRnWiqx+Wq2BTu8WFO/gNFMvffRRW/TSMSAflUY64w0acXR+NM2XoeUW
j8tiz1LwGEGf7iSvmzN2t35x9TObLGU6NY6zEnO33ZOaqfhm1ZihHv5rle3+WiEl
6TFJIaWFmxb1MjPnCEqkMzaMQ70Nxuv5O6PbE6N7AKJ5+RWlJK/FQbyXLOVTg9qV
omCYSEDEC/+X8HSM/vEzTRunBcE7MLN7JSjF8PWy4GnPEzbG7cIhJ4xwHOMXm9/L
nMqq0+MdNuaR2W2xfDcRYUjZf7qZ7Q17spdbtTzMgX0fnKJTwLWjykch6JgK9zxf
8txrW5C276StC8WAB4GrzfskA0wVAgMBAAGjWTBXMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQk9qVRSGdZRO/jEOZGp+eOqHLymDAV
BgNVHREEDjAMggprdWJlcm5ldGVzMA0GCSqGSIb3DQEBCwUAA4IBAQAIkiGxISGX
ELIj52ZMWNlj4+bT/MqqUoG2NXMIVSDW/rjS6RwjLhm5ubeltfESSx8cQcvh2IfL
sVSXZGVQk9Cn4Ki7/AHRSGkA2yQ9VNy937/7IfcRfwUyBHUR6yC0B+0URs7kmvim
eNvEWv+ySxdpeEJX2na/lzhXurV6ftnXtGa/6YGaWqsrNG5/4v8KDLM1vbH8hwGT
JDKZQpl+U6q2fyXkei/txap4XhGc43mNCO/eFQWfhmCBuiL9xXJVjqhMb5T5CNiR
XIHqlzbAvHR4v+z8VgMOC5XrN7hlfcpNrfaV7p+jhdr2CIGRrRRUegWILF9rt0mD
YOBZ+7E4fpnu
-----END CERTIFICATE-----
kubernetes_host                      https://kubernetes.default.svc:443
pem_keys                             []
token_reviewer_jwt_set               true
use_annotations_as_alias_metadata    false
----------------------------------------------------------------

vault write 명령 분석
- Vault가 K8s API 서버에 인증 요청을 보낼 수 있도록 설정
  - kubernetes_host: API 서버 주소
  - kubernetes_ca_cert: API 서버의 CA 인증서 (Pod 내 ServiceAccount에서 자동 주입됨)
  - token_reviewer_jwt: TokenReviewer 권한이 있는 토큰으로 Vault가 요청자의 토큰을 검증할 수 있도록 설정
Step5. Vault Kubernetes Auth Role 설정
실행 중인 애플리케이션이 Vault로부터 DB 자격증명(동적 사용자)을 받아올 수 있도록 권한을 연결을 위한 설정

vault write auth/k8s-auth-mount/role/auth-role \
   bound_service_account_names=demo-dynamic-app \
   bound_service_account_namespaces=demo-ns \
   token_ttl=0 \
   token_period=120 \
   token_policies=demo-auth-policy-db \
   audience=vault
Success! Data written to: auth/k8s-auth-mount/role/auth-role

# auth-role 생성
vault write auth/k8s-auth-mount/role/auth-role \
   bound_service_account_names=demo-dynamic-app \ # demo-dynamic-app 서비스 어카운트를 사용하는 Pod에서만 인증 허용, Pod 내부에서 Vault에 로그인하려면 이 서비스 어카운트를 써야 함
   bound_service_account_namespaces=demo-ns \ # 이 Role은 오직 demo-ns 네임스페이스에서 오는 요청만 허용
   token_ttl=0 \ # 발급되는 Vault 토큰의 기본 TTL 없음(0). 일반적으로 token_ttl=0이면 token_period가 사용됨
   token_period=120 \ # Lease 기간이 120초인 Renewable Token 발급. TTL이 없는 대신, 2분마다 갱신 가능한 토큰을 생성. 보통 이 설정은 "periodic token"이라 불리며, long-lived session에 적합.
   token_policies=demo-auth-policy-db \ # 인증에 성공하면 발급되는 토큰에 demo-auth-policy-db 정책이 적용
   audience=vault # JWT 토큰의 aud (Audience) claim 검증에 사용.

Step6. Vault Database Secret Engine 설정

# demo-db라는 경로로 Database Secret Engine을 활성화
vault secrets enable -path=demo-db database
Success! Enabled the database secrets engine at: demo-db/

# PostgreSQL 연결 정보 등록
# 해당 과정은 postgres가 정상적으로 동작 시 적용 가능
# allowed_roles: 이후 설정할 Role 이름 지정
vault write demo-db/config/demo-db \
   plugin_name=postgresql-database-plugin \
   allowed_roles="dev-postgres" \
   connection_url="postgresql://{{username}}:{{password}}@postgres-postgresql.postgres.svc.cluster.local:5432/postgres?sslmode=disable" \
   username="postgres" \
   password="secret-pass"
Success! Data written to: demo-db/config/demo-db   
  
# DB 사용자 동적 생성 Role 등록
# 해당 Role 사용 시 Vault가 동적으로 사용자 계정과 비밀번호를 생성 가능
# TTL은 생성된 자격증명의 유효 시간 (30초~10분)
## creation_statements=... : Vault가 계정을 자동 생성할 때 실행할 SQL 문. {{name}}, {{password}}, {{expiration}}은 Vault가 자동으로 채워줌.
## revocation_statements=... : Vault가 동적 사용자 계정을 폐기(revoke)할 때 실행할 SQL 문. 권한을 모두 회수.
## default_ttl="1m" / max_ttl="1m" : 기본 1분만 유효합니다. 갱신을 안 하면 1분 후 자동 만료 (계정도 자동 삭제). 최대 TTL도 1분이므로 연장도 최대 1분까지만.
vault write demo-db/roles/dev-postgres \
   db_name=demo-db \
   creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
      GRANT ALL PRIVILEGES ON DATABASE postgres TO \"{{name}}\";" \
   revocation_statements="REVOKE ALL ON DATABASE postgres FROM  \"{{name}}\";" \
   backend=demo-db \
   name=dev-postgres \
   default_ttl="1m" \
   max_ttl="1m"
Success! Data written to: demo-db/roles/dev-postgres

# 정책 설정: DB 자격증명 읽기 권한
# demo-db/creds/dev-postgres 경로에 대한 read 권한 부여
# 추후 Kubernetes 서비스 어카운트(demo-dynamic-app)에 이 정책을 연결해서 자격증명 요청 가능
vault policy write demo-auth-policy-db - <<EOF
path "demo-db/creds/dev-postgres" {
   capabilities = ["read"]
}
EOF
Success! Uploaded policy: demo-auth-policy-db

Step7. Transit Secret Engine 설정 + VSO 연동 Role 구성 - Docs , Client-cache

kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
----------------------------------------------------------------
# Transit Secret Engine 활성화
# transit 엔진을 demo-transit 경로로 활성화.
# 데이터를 저장하지 않고 암복호화 기능만 제공하는 Vault의 기능
# 클라이언트 캐시는 리더십 변경 시에도 Vault 토큰 및 동적 비밀 임대를 계속 추적하고 갱신할 수 있으므로 원활한 업그레이드를 지원합니다
## Vault 서버에 클라이언트 캐시를 저장하고 암호화할 수 있습니다.
## Vault 동적 비밀을 사용하는 경우 클라이언트 캐시를 영구적으로 저장하고 암호화하는 것이 좋습니다.
## 이렇게 하면 재시작 및 업그레이드를 통해 동적 비밀 임대가 유지됩니다.
vault secrets enable -path=demo-transit transit
Success! Enabled the transit secrets engine at: demo-transit/

vault secrets list -detailed
Path             Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    External Entropy Access    Options           Description                                                UUID                                    Version    Running Version          Running SHA256    Deprecation Status
----             ------       --------              -----------    -------    --------------    -----------    ---------    -----------------------    -------           -----------                                                ----                                    -------    ---------------          --------------    ------------------
cubbyhole/       cubbyhole    cubbyhole_81343c9d    n/a            n/a        false             local          false        false                      map[]             per-token private secret storage                           42c8881e-ddd4-de89-0965-d5d2322252f4    n/a        v1.19.0+builtin.vault    n/a               n/a
demo-db/         database     database_22a52598     system         system     false             replicated     false        false                      map[]             n/a                                                        7cc9dedf-df08-1fd1-6404-ab5f1a4ec34f    n/a        v1.19.0+builtin.vault    n/a               n/a
demo-transit/    transit      transit_b7eb4f67      system         system     false             replicated     false        false                      map[]             n/a                                                        c14021e1-93b4-cff9-70af-853ffebeb85f    n/a        v1.19.0+builtin.vault    n/a               supported
identity/        identity     identity_ff863c9a     system         system     false             replicated     false        false                      map[]             identity store                                             48893d8e-2a03-dd93-550d-ad95a4258806    n/a        v1.19.0+builtin.vault    n/a               n/a
secret/          kv           kv_899fcc51           system         system     false             replicated     false        false                      map[version:2]    n/a                                                        432bfbdd-040a-e190-00e1-1d17e5ee6b61    n/a        v0.21.0+builtin          n/a               supported
sys/             system       system_6c9bd623       n/a            n/a        false             replicated     true         false                      map[]             system endpoints used for control, policy and debugging    b168292f-4ff2-32c1-a1dc-8a328fa5becb    n/a        v1.19.0+builtin.vault    n/a               n/a

# vso-client-cache라는 키를 생성
# 이 키는 VSO가 암복호화 시 사용할 암호화 키 역할
vault write -force demo-transit/keys/vso-client-cache
Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1744357522]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      vso-client-cache
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96

# vso-client-cache 키에 대해 암호화(encrypt), 복호화(decrypt)를 허용하는 정책 생성
vault policy write demo-auth-policy-operator - <<EOF
path "demo-transit/encrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
path "demo-transit/decrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
EOF
Success! Uploaded policy: demo-auth-policy-operator

# Vault Secrets Operator가 사용하는 ServiceAccount에 위 정책을 바인딩
# vso가 Vault에 로그인할 때 사용할 수 있는 JWT 기반 Role 설정
# 해당 Role을 통해 Operator는 Transit 엔진을 이용한 암복호화 API 호출 가능
vault write auth/k8s-auth-mount/role/auth-role-operator \
   bound_service_account_names=vault-secrets-operator-controller-manager \
   bound_service_account_namespaces=vault-secrets-operator-system \
   token_ttl=0 \
   token_period=120 \
   token_policies=demo-auth-policy-operator \
   audience=vault
Success! Data written to: auth/k8s-auth-mount/role/auth-role-operator

vault read auth/k8s-auth-mount/role/auth-role-operator
Key                                         Value
---                                         -----
alias_name_source                           serviceaccount_uid
audience                                    vault
bound_service_account_names                 [vault-secrets-operator-controller-manager]
bound_service_account_namespace_selector    n/a
bound_service_account_namespaces            [vault-secrets-operator-system]
token_bound_cidrs                           []
token_explicit_max_ttl                      0s
token_max_ttl                               0s
token_no_default_policy                     false
token_num_uses                              0
token_period                                2m
token_policies                              [demo-auth-policy-operator]
token_ttl                                   0s
token_type                                  default

Step8. 샘플 애플리케이션 YAML 작성 및 배포
- demo-ns 네임스페이스 생성

kubectl create ns demo-ns
mkdir vso-dynamic
cd vso-dynamic

vault-auth-dynamic.yaml :
- 앱이 Vault에 인증하기 위한 ServiceAccount 및 VaultAuth 리소스
- Vault에서 발급한 동적 PostgreSQL 크레덴셜을 얻기 위해 반드시 필요

---
# vault-auth-dynamic.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: demo-ns
  name: demo-dynamic-app
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: dynamic-auth
  namespace: demo-ns
spec:
  method: kubernetes
  mount: k8s-auth-mount
  kubernetes:
    role: auth-role
    serviceAccount: demo-dynamic-app
    audiences:
      - vault

app-secret.yaml :
- Spring App에서 PostgreSQL에 접속할 때 사용할 해당 시크릿에 username/password을 동적으로 생성

---
# app-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: vso-db-demo
  namespace: demo-ns

vault-dynamic-secret.yaml
- Vault에서 동적으로 PostgreSQL 접속정보 생성하고 K8s Secret에 저장
- 생성된 Secret(vso-db-demo)은 앱에서 환경 변수(env)로 사용
- 애플리케이션에서 Dynamic Reloading을 지원하지 않을 경우 rolloutRestartTargets 을 사용하여 애플리케이션을 재배포하여 새로 업데이트된 시크릿을 사용하도록 할 수 있음
- refreshAfter 설정을 통해 VSO가 소스 시크릿 데이터를 동기화 하는 주기 설정가능 - Docs
  → Event Notification 기능을 통해 변경이 있을 경우 즉시 반영할 수도 있음 - Docs

---
# vault-dynamic-secret.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: vso-db-demo
  namespace: demo-ns
spec:
  refreshAfter: 25s
  mount: demo-db
  path: creds/dev-postgres
  destination:
    name: vso-db-demo
    create: true
    overwrite: true
  vaultAuthRef: dynamic-auth
  rolloutRestartTargets:
  - kind: Deployment
    name: vaultdemo

app-spring-deploy.yaml :
- DB 접속 테스트를 위한 Spring App →

---
# app-spring-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vaultdemo
  namespace: demo-ns
  labels:
    app: vaultdemo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vaultdemo
  template:
    metadata:
      labels:
        app: vaultdemo
    spec:
      volumes:
        - name: secrets
          secret:
            secretName: "vso-db-demo"
      containers:
        - name: vaultdemo
          image: hyungwookhub/vso-spring-demo:v5
          imagePullPolicy: IfNotPresent
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: "vso-db-demo"
                  key: password
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: "vso-db-demo"
                  key: username
            - name: DB_HOST
              value: "postgres-postgresql.postgres.svc.cluster.local"
            - name: DB_PORT
              value: "5432"
            - name: DB_NAME
              value: "postgres"
          ports:
            - containerPort: 8088
          volumeMounts:
            - name: secrets
              mountPath: /etc/secrets
              readOnly: true
---
apiVersion: v1
kind: Service
metadata:
  name: vaultdemo
  namespace: demo-ns
spec:
  ports:
    - name: vaultdemo
      port: 8088         
      targetPort: 8088 
      nodePort: 30003
  selector:
    app: vaultdemo
  type: NodePort

애플리케이션 배포 :

kubectl apply -f .`
secret/vso-db-demo created
deployment.apps/vaultdemo created
service/vaultdemo created
serviceaccount/demo-dynamic-app created
vaultauth.secrets.hashicorp.com/dynamic-auth created
vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo created

Step9. 동작확인(UI) : postgres-postgresql.postgres.svc , db는 postgres
- localhost:30003 접속 후 K8s Secrets에 생성된 username, password 값으로 연결 테스트
Step9. 동작확인(CLI)

# 스크릿이 변경되고, 적용을 위해서 파드가 42~45초 사이로 재생성됨
while true; do kubectl get pod -n demo-ns ; echo; kubectl view-secret -n demo-ns vso-db-demo --all; date; sleep 30 ; echo ; done 
NAME                         READY   STATUS    RESTARTS   AGE
vaultdemo-54466b5f8b-8gglh   1/1     Running   0          34s

_raw='{"password":"XkQ-J9PWOUxdaJqr8Ga0","username":"v-k8s-auth-dev-post-gJlLw0hjQSZNtI3vsPB0-1744358614"}'
password='XkQ-J9PWOUxdaJqr8Ga0'
username='v-k8s-auth-dev-post-gJlLw0hjQSZNtI3vsPB0-1744358614'
Fri Apr 11 17:04:09 KST 2025

NAME                         READY   STATUS    RESTARTS   AGE
vaultdemo-7d8dc56b8c-f5s4k   1/1     Running   0          21s

_raw='{"password":"dBZuGove-sLQcPvwwR3q","username":"v-k8s-auth-dev-post-vXvDAaDe9XqLvP9K04vH-1744358658"}'
password='dBZuGove-sLQcPvwwR3q'
username='v-k8s-auth-dev-post-vXvDAaDe9XqLvP9K04vH-1744358658'
Fri Apr 11 17:04:39 KST 2025

# 실제 postgresql 에 사용자 정보 확인 : 계속 추가되고 있음..
kubectl exec -it -n postgres postgres-postgresql-0 -- sh -c "PGPASSWORD=secret-pass psql -U postgres -h localhost -c '\du'"
                                                  List of roles
                      Role name                      |                         Attributes                         
-----------------------------------------------------+------------------------------------------------------------
 postgres                                            | Superuser, Create role, Create DB, Replication, Bypass RLS
 v-k8s-auth-dev-post-073kCwgaqwPD9yB4REPF-1744358701 | Password valid until 2025-04-11 08:06:06+00
 v-k8s-auth-dev-post-5YI48vzkiQo3iAAbEtfH-1744358568 | Password valid until 2025-04-11 08:03:53+00
 v-k8s-auth-dev-post-9p9X4qcFjqYS0Z1CgENM-1744358438 | Password valid until 2025-04-11 08:01:43+00
 v-k8s-auth-dev-post-CeLjNhO2M4jX9JI8C2II-1744358743 | Password valid until 2025-04-11 08:06:48+00
 v-k8s-auth-dev-post-LNgZaqarkrYdeDZRE5yA-1744358307 | Password valid until 2025-04-11 07:59:32+00
 v-k8s-auth-dev-post-T82jXw4tz8VlMNeSsAGL-1744358351 | Password valid until 2025-04-11 08:00:16+00
 v-k8s-auth-dev-post-U163tgCi4KTeh5ALoixi-1744358264 | Password valid until 2025-04-11 07:58:49+00
 v-k8s-auth-dev-post-ZKZp9hwhkpMKtMJQSITC-1744358525 | Password valid until 2025-04-11 08:03:10+00
 v-k8s-auth-dev-post-aQzuu4etveKItQQ4nU9c-1744358484 | Password valid until 2025-04-11 08:02:29+00
 v-k8s-auth-dev-post-chObYVU1zv3AbLRxu7kG-1744358183 | Password valid until 2025-04-11 07:57:28+00
 v-k8s-auth-dev-post-gJlLw0hjQSZNtI3vsPB0-1744358614 | Password valid until 2025-04-11 08:04:39+00
 v-k8s-auth-dev-post-mw3XklNZ5fDfTYpBRev1-1744358183 | Password valid until 2025-04-11 07:57:28+00
 v-k8s-auth-dev-post-nCEAnQZV5SgV0K0wzcp6-1744358395 | Password valid until 2025-04-11 08:01:00+00
 v-k8s-auth-dev-post-utfzStl1Tl2yPgpHv7ZA-1744358224 | Password valid until 2025-04-11 07:58:09+00
 v-k8s-auth-dev-post-vXvDAaDe9XqLvP9K04vH-1744358658 | Password valid until 2025-04-11 08:05:23+00
 ...


# 로그 확인
kubectl stern -n demo-ns -l app=vaultdemo
+ vaultdemo-5d77487ccd-62kbw › vaultdemo
vaultdemo-5d77487ccd-62kbw vaultdemo 
vaultdemo-5d77487ccd-62kbw vaultdemo   .   ____          _            __ _ _
vaultdemo-5d77487ccd-62kbw vaultdemo  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
vaultdemo-5d77487ccd-62kbw vaultdemo ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
vaultdemo-5d77487ccd-62kbw vaultdemo  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
vaultdemo-5d77487ccd-62kbw vaultdemo   '  |____| .__|_| |_|_| |_\__, | / / / /
vaultdemo-5d77487ccd-62kbw vaultdemo  =========|_|==============|___/=/_/_/_/
vaultdemo-5d77487ccd-62kbw vaultdemo 
vaultdemo-5d77487ccd-62kbw vaultdemo  :: Spring Boot ::                (v3.4.4)
vaultdemo-5d77487ccd-62kbw vaultdemo 
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:44.475Z  INFO 1 --- [           main] c.e.vaultdemo.VaultdemoApplicationKt     : Starting VaultdemoApplicationKt v0.0.1-SNAPSHOT using Java 17.0.14 with PID 1 (/app/app.jar started by root in /app)
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:44.477Z  INFO 1 --- [           main] c.e.vaultdemo.VaultdemoApplicationKt     : No active profile set, falling back to 1 default profile: "default"
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:45.269Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8088 (http)
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:45.277Z  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:45.277Z  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.39]
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:45.296Z  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:45.296Z  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 793 ms
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:45.512Z  INFO 1 --- [           main] o.s.b.a.w.s.WelcomePageHandlerMapping    : Adding welcome page template: index
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:45.810Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8088 (http) with context path '/'
vaultdemo-5d77487ccd-62kbw vaultdemo 2025-04-11T08:05:45.824Z  INFO 1 --- [           main] c.e.vaultdemo.VaultdemoApplicationKt     : Started VaultdemoApplicationKt in 1.631 seconds (process running for 1.922)
...

kubectl stern -n vault vault-0
+ vault-0 › vault
vault-0 vault ==> Vault server configuration:
vault-0 vault 
vault-0 vault Administrative Namespace: 
vault-0 vault              Api Address: http://10.244.2.9:8200
vault-0 vault                      Cgo: disabled
vault-0 vault          Cluster Address: https://vault-0.vault-internal:8201
vault-0 vault    Environment Variables: HOME, HOSTNAME, HOST_IP, KUBERNETES_PORT, KUBERNETES_PORT_443_TCP, KUBERNETES_PORT_443_TCP_ADDR, KUBERNETES_PORT_443_TCP_PORT, KUBERNETES_PORT_443_TCP_PROTO, KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT, KUBERNETES_SERVICE_PORT_HTTPS, NAME, NGINX_SERVICE_PORT, NGINX_SERVICE_PORT_80_TCP, NGINX_SERVICE_PORT_80_TCP_ADDR, NGINX_SERVICE_PORT_80_TCP_PORT, NGINX_SERVICE_PORT_80_TCP_PROTO, NGINX_SERVICE_SERVICE_HOST, NGINX_SERVICE_SERVICE_PORT, PATH, POD_IP, PWD, SHLVL, SKIP_CHOWN, SKIP_SETCAP, VAULT_ADDR, VAULT_AGENT_INJECTOR_SVC_PORT, VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP, VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_ADDR, VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_PORT, VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_PROTO, VAULT_AGENT_INJECTOR_SVC_SERVICE_HOST, VAULT_AGENT_INJECTOR_SVC_SERVICE_PORT, VAULT_AGENT_INJECTOR_SVC_SERVICE_PORT_HTTPS, VAULT_API_ADDR, VAULT_CLUSTER_ADDR, VAULT_K8S_NAMESPACE, VAULT_K8S_POD_NAME, VAULT_PORT, VAULT_PORT_8200_TCP, VAULT_PORT_8200_TCP_ADDR, VAULT_PORT_8200_TCP_PORT, VAULT_PORT_8200_TCP_PROTO, VAULT_PORT_8201_TCP, VAULT_PORT_8201_TCP_ADDR, VAULT_PORT_8201_TCP_PORT, VAULT_PORT_8201_TCP_PROTO, VAULT_SERVICE_HOST, VAULT_SERVICE_PORT, VAULT_SERVICE_PORT_HTTP, VAULT_SERVICE_PORT_HTTPS_INTERNAL, VERSION
vault-0 vault               Go Version: go1.23.6
vault-0 vault               Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", disable_request_limiter: "false", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault-0 vault                Log Level: 
vault-0 vault                    Mlock: supported: true, enabled: false
vault-0 vault            Recovery Mode: false
vault-0 vault                  Storage: file
vault-0 vault                  Version: Vault v1.19.0, built 2025-03-04T12:36:40Z
vault-0 vault              Version Sha: 7eeafb6160d60ede73c1d95566b0c8ea54f3cb5a
vault-0 vault 
vault-0 vault ==> Vault server started! Log data will stream in below:
vault-0 vault 
vault-0 vault 2025-04-11T06:51:33.007Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault-0 vault 2025-04-11T06:51:33.008Z [INFO]  incrementing seal generation: generation=1
vault-0 vault 2025-04-11T06:51:33.009Z [INFO]  core: Initializing version history cache for core
vault-0 vault 2025-04-11T06:51:33.009Z [INFO]  events: Starting event system
vault-0 vault 2025-04-11T06:52:03.036Z [INFO]  core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
vault-0 vault 2025-04-11T06:52:03.036Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
vault-0 vault 2025-04-11T06:52:03.037Z [INFO]  core: post-unseal setup starting
vault-0 vault 2025-04-11T06:52:03.038Z [INFO]  core: loaded wrapping token key
vault-0 vault 2025-04-11T06:52:03.038Z [INFO]  core: successfully setup plugin runtime catalog
vault-0 vault 2025-04-11T06:52:03.038Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
vault-0 vault 2025-04-11T06:52:03.043Z [INFO]  core: successfully mounted: type=system version="v1.19.0+builtin.vault" path=sys/ namespace="ID: root. Path: "
vault-0 vault 2025-04-11T06:52:03.043Z [INFO]  core: successfully mounted: type=identity version="v1.19.0+builtin.vault" path=identity/ namespace="ID: root. Path: "
vault-0 vault 2025-04-11T06:52:03.045Z [INFO]  core: successfully mounted: type=kv version="v0.21.0+builtin" path=secret/ namespace="ID: root. Path: "
vault-0 vault 2025-04-11T06:52:03.045Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.19.0+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
vault-0 vault 2025-04-11T06:52:03.047Z [INFO]  core: successfully mounted: type=token version="v1.19.0+builtin.vault" path=token/ namespace="ID: root. Path: "
vault-0 vault 2025-04-11T06:52:03.047Z [INFO]  core: successfully mounted: type=approle version="v1.19.0+builtin.vault" path=approle/ namespace="ID: root. Path: "
vault-0 vault 2025-04-11T06:52:03.047Z [INFO]  core: successfully mounted: type=kubernetes version="v0.21.0+builtin" path=kubernetes/ namespace="ID: root. Path: "
vault-0 vault 2025-04-11T06:52:03.048Z [INFO]  rollback: Starting the rollback manager with 256 workers
vault-0 vault 2025-04-11T06:52:03.048Z [INFO]  rollback: starting rollback manager
vault-0 vault 2025-04-11T06:52:03.048Z [INFO]  core: restoring leases
vault-0 vault 2025-04-11T06:52:03.050Z [INFO]  expiration: lease restore complete
vault-0 vault 2025-04-11T06:52:03.051Z [INFO]  identity: entities restored
vault-0 vault 2025-04-11T06:52:03.051Z [INFO]  identity: groups restored
vault-0 vault 2025-04-11T06:52:03.051Z [INFO]  core: usage gauge collection is disabled
vault-0 vault 2025-04-11T06:52:03.052Z [INFO]  core: post-unseal setup complete
vault-0 vault 2025-04-11T06:52:03.052Z [INFO]  core: vault is unsealed
vault-0 vault 2025-04-11T07:33:25.271Z [INFO]  core: enabled credential backend: path=k8s-auth-mount/ type=kubernetes version="v0.21.0+builtin"
vault-0 vault 2025-04-11T07:33:36.419Z [ERROR] secrets.system.system_6c9bd623: error occurred during enable credential: path=k8s-auth-mount/ error="path is already in use at k8s-auth-mount/"
vault-0 vault 2025-04-11T07:33:47.599Z [ERROR] secrets.system.system_6c9bd623: error occurred during enable credential: path=k8s-auth-mount/ error="path is already in use at k8s-auth-mount/"
vault-0 vault 2025-04-11T07:34:17.460Z [ERROR] core: login attempts exceeded, user is locked out: request_path=auth/approle/login
vault-0 vault 2025-04-11T07:34:33.433Z [ERROR] core: login attempts exceeded, user is locked out: request_path=auth/approle/login
vault-0 vault 2025-04-11T07:35:03.000Z [ERROR] core: login attempts exceeded, user is locked out: request_path=auth/approle/login
vault-0 vault 2025-04-11T07:36:01.455Z [ERROR] core: login attempts exceeded, user is locked out: request_path=auth/approle/login
vault-0 vault 2025-04-11T07:37:39.773Z [ERROR] core: login attempts exceeded, user is locked out: request_path=auth/approle/login
vault-0 vault 2025-04-11T07:40:18.149Z [ERROR] core: login attempts exceeded, user is locked out: request_path=auth/approle/login
vault-0 vault 2025-04-11T07:41:51.553Z [INFO]  secrets.database.database_22a52598: initializing database rotation queue
vault-0 vault 2025-04-11T07:41:51.554Z [INFO]  core: successful mount: namespace="" path=demo-db/ type=database version="v1.19.0+builtin.vault"
vault-0 vault 2025-04-11T07:41:51.565Z [INFO]  secrets.database.database_22a52598: populating role rotation queue
vault-0 vault 2025-04-11T07:41:51.566Z [INFO]  secrets.database.database_22a52598: starting periodic ticker
vault-0 vault 2025-04-11T07:44:26.602Z [ERROR] core: login attempts exceeded, user is locked out: request_path=auth/approle/login
vault-0 vault 2025-04-11T07:44:46.490Z [INFO]  core: successful mount: namespace="" path=demo-transit/ type=transit version="v1.19.0+builtin.vault"
vault-0 vault 2025-04-11T07:48:26.729Z [ERROR] core: login attempts exceeded, user is locked out: request_path=auth/approle/login
vault-0 vault 2025-04-11T07:55:47.253Z [INFO]  expiration: revoked lease: lease_id=auth/approle/login/h5389c0ab267a3887a96d9a199dc2c4c9c616cd2746a61627e316e51bce498333
vault-0 vault 2025-04-11T07:57:23.210Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/8ozUkFrQ10RJ0zmzIvXuVRwh
vault-0 vault 2025-04-11T07:57:23.471Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/0HFcmVcXxtDkaOJ2fRuqFOkB
vault-0 vault 2025-04-11T07:58:04.905Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/gPlKN05NCn6TqG7ALPbspmVk
vault-0 vault 2025-04-11T07:58:44.664Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/mAOO2TTlZdTLI1mWvslCdak4
vault-0 vault 2025-04-11T07:59:27.999Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/3FsgMe9msEHmC60j3JPe8kwA
vault-0 vault 2025-04-11T08:00:11.563Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/LGyhgqPakIcMJyGkbUgzOosv
vault-0 vault 2025-04-11T08:00:55.151Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/LPilSJxrbB3qjcqdFK784rc9
vault-0 vault 2025-04-11T08:01:38.338Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/PBzFA36MGRHdl9tLcLAfE2q3
vault-0 vault 2025-04-11T08:02:24.100Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/AMnQDWq8bVPKDPSYTKXPG9oQ
vault-0 vault 2025-04-11T08:03:05.878Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/QKtzcoOh78fKpWqI1fvR1Kdk
vault-0 vault 2025-04-11T08:03:48.409Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/ClVArkbXTd4cEusGbbyY1tnC
vault-0 vault 2025-04-11T08:04:34.717Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/QW89tAm8B3zBGsfkYtUO9jaQ
vault-0 vault 2025-04-11T08:05:18.441Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/2sxDNw5q1oRg8xRNiXnU34fp
vault-0 vault 2025-04-11T08:06:01.323Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/jl13xLjckRBhSktO7IDVyKr8

# vault-secrets-operator 가 clientCachePersistenceModel 설정 적용 정보 확인 : 참고로 해당 파드가 재시작되지는 않음.
kubectl stern -n vault-secrets-operator-system -l app.kubernetes.io/name=vault-secrets-operator
+ vault-secrets-operator-controller-manager-7f67cd89fd-htprm › manager
+ vault-secrets-operator-controller-manager-7f67cd89fd-htprm › kube-rbac-proxy
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","logger":"initCachingClientFactory","msg":"Initializing the CachingClientFactory"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","logger":"clientCallbackHandler","msg":"Starting client callback handler"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","logger":"setup","msg":"Starting manager","gitVersion":"0.10.0","gitCommit":"aebf0c1c59485059a9ea6c58340fd406afe4cbef","gitTreeState":"clean","buildDate":"2025-03-04T22:22:24+0000","goVersion":"go1.23.6","platform":"linux/arm64","clientCachePersistenceModel":"direct-encrypted","clientCacheSize":10000,"backoffMultiplier":1.5,"backoffMaxInterval":60,"backoffMaxElapsedTime":0,"backoffInitialInterval":5,"backoffRandomizationFactor":0.5,"globalTransformationOptions":"","globalVaultAuthOptions":"allow-default-globals"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy W0411 07:17:46.984244       1 options.go:150] 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy ==== Deprecation Warning ======================
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy Insecure listen address will be removed.
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy Using --insecure-listen-address won't be possible!
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy The ability to run kube-rbac-proxy without TLS certificates will be removed.
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy Not using --tls-cert-file and --tls-private-key-file won't be possible!
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy For more information, please go to https://github.com/brancz/kube-rbac-proxy/issues/187
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"starting server","name":"health probe","addr":"[::]:8081"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy ===============================================
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy W0411 07:17:46.984393       1 deprecated.go:66] 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager I0411 07:17:56.580456       1 leaderelection.go:257] attempting to acquire leader lease vault-secrets-operator-system/b0d477c0.hashicorp.com...
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager I0411 07:17:56.583737       1 leaderelection.go:271] successfully acquired lease vault-secrets-operator-system/b0d477c0.hashicorp.com
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultpkisecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultPKISecret","source":"kind source: *v1.PartialObjectMetadata"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","source":"channel source: 0x400024ce70"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultpkisecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultPKISecret","source":"kind source: *v1beta1.VaultPKISecret"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultconnection","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultConnection","source":"kind source: *v1beta1.VaultConnection"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultpkisecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultPKISecret","source":"kind source: *v1beta1.SecretTransformation"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","source":"kind source: *v1beta1.VaultStaticSecret"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","source":"kind source: *v1beta1.SecretTransformation"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","source":"kind source: *v1.PartialObjectMetadata"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultauth","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultAuth","source":"kind source: *v1beta1.VaultAuthGlobal"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"hcpauth","controllerGroup":"secrets.hashicorp.com","controllerKind":"HCPAuth","source":"kind source: *v1beta1.HCPAuth"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultdynamicsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultDynamicSecret","source":"channel source: 0x400024cee0"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultauth","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultAuth","source":"kind source: *v1beta1.VaultAuth"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"secrettransformation","controllerGroup":"secrets.hashicorp.com","controllerKind":"SecretTransformation","source":"kind source: *v1beta1.SecretTransformation"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"hcpvaultsecretsapp","controllerGroup":"secrets.hashicorp.com","controllerKind":"HCPVaultSecretsApp","source":"kind source: *v1.PartialObjectMetadata"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultdynamicsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultDynamicSecret","source":"kind source: *v1beta1.SecretTransformation"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultdynamicsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultDynamicSecret","source":"kind source: *v1.PartialObjectMetadata"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"hcpvaultsecretsapp","controllerGroup":"secrets.hashicorp.com","controllerKind":"HCPVaultSecretsApp","source":"kind source: *v1beta1.HCPVaultSecretsApp"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"hcpvaultsecretsapp","controllerGroup":"secrets.hashicorp.com","controllerKind":"HCPVaultSecretsApp","source":"kind source: *v1beta1.SecretTransformation"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultdynamicsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultDynamicSecret","source":"kind source: *v1beta1.VaultDynamicSecret"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting EventSource","controller":"vaultauthglobal","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultAuthGlobal","source":"kind source: *v1beta1.VaultAuthGlobal"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"secrettransformation","controllerGroup":"secrets.hashicorp.com","controllerKind":"SecretTransformation"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"vaultconnection","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultConnection"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy ==== Removed Flag Warning ======================
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"secrettransformation","controllerGroup":"secrets.hashicorp.com","controllerKind":"SecretTransformation","worker count":1}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy logtostderr is removed in the k8s upstream and has no effect any more.
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy ===============================================
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy I0411 07:17:46.984604       1 kube-rbac-proxy.go:232] Valid token audiences: 
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy I0411 07:17:46.985315       1 kube-rbac-proxy.go:326] Generating self signed cert as no cert is provided
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy I0411 07:17:47.400205       1 kube-rbac-proxy.go:390] Starting TCP socket on 0.0.0.0:8443
vault-secrets-operator-controller-manager-7f67cd89fd-htprm kube-rbac-proxy I0411 07:17:47.400487       1 kube-rbac-proxy.go:397] Listening securely on 0.0.0.0:8443
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"vaultconnection","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultConnection","worker count":1}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"vaultauth","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultAuth"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"vaultauth","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultAuth","worker count":1}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"hcpauth","controllerGroup":"secrets.hashicorp.com","controllerKind":"HCPAuth"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"hcpauth","controllerGroup":"secrets.hashicorp.com","controllerKind":"HCPAuth","worker count":1}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","worker count":100}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"vaultauthglobal","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultAuthGlobal"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"vaultauthglobal","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultAuthGlobal","worker count":1}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"hcpvaultsecretsapp","controllerGroup":"secrets.hashicorp.com","controllerKind":"HCPVaultSecretsApp"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"hcpvaultsecretsapp","controllerGroup":"secrets.hashicorp.com","controllerKind":"HCPVaultSecretsApp","worker count":100}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"vaultdynamicsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultDynamicSecret"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"vaultdynamicsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultDynamicSecret","worker count":100}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting Controller","controller":"vaultpkisecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultPKISecret"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:17:56Z","msg":"Starting workers","controller":"vaultpkisecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultPKISecret","worker count":100}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:56:23Z","logger":"lifetimeWatcher","msg":"Starting","id":"98dab696-a202-4f11-a995-8163b389efa6","entityID":"e15ab976-671e-afc8-614d-875fa192ec93","clientID":"36b9ab822375356b25e915d756d8ce6083894e25d0518911c022670f05d2f2d0","cacheKey":"kubernetes-ad55540ae5d3cf51f94232"}
vault-secrets-operator-controller-manager-7f67cd89fd-htprm manager {"level":"info","ts":"2025-04-11T07:56:23Z","logger":"lifetimeWatcher","msg":"Starting","id":"cba45aa7-b6d0-4335-978e-474ead1cb1d5","entityID":"2f5b0f00-d094-754e-1758-e0df06f4e0ee","clientID":"736b9671e3d02cdeb4c5f5ccf3f9b71420d7b537c63e4fa2f724191ae57a06f3","cacheKey":"kubernetes-0fcd6d53bcef79a3934f85"}

# 암호화된 캐시 저장소 확인 : vso-cc-<auth method>
kubectl get secret -n vault-secrets-operator-system -l app.kubernetes.io/component=client-cache-storage 
NAME                                       TYPE     DATA   AGE
vso-cc-kubernetes-0fcd6d53bcef79a3934f85   Opaque   2      11m

kubectl get secret -n vault-secrets-operator-system -l app.kubernetes.io/component=client-cache-storage -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    messageMAC: BVR67aukUNSLI6GBiYW8LumYHhUCLw2KLpG6vFn5G+0=
    secret: eyJjaXBoZXJ0ZXh0IjoidmF1bHQ6djE6eDlYTmRWSkRNSDh4RGpzSng3Sml0UDU2RDBXVUQ1YnhVU1lNK0I3RzBYU2IrMHpmOUdOVzBFU1Z5OFB5N1pQenFwak9qSEVmR2FZRmtCSG5nUkxhSWRNaE9SSVhXRFBSNk5tYWpxS2ZiR3VvdjFYM3cyd2pDVHNaSkltOVFtZnFDaS9BVXJkeTJiTFVCOVdESUJFY09xUmZsaEFla1ZSNG1PckRUeG1qbmUyWFVrT1lmMVhETEhkNC83QW9zNk4yS2xDVHNVMVNmek5zREJvNFNid04vL21zWVU2QW1VSHBzR3ZNRk9IbWFCeFIrYlZ2amROYXVzOGRkMU43TnN3M0Y4RkQ4NS9DcUkvNzZJQ0xzdURVenI4WTY2QTZqYVpUd1Y5ZHVQUG9qQUR4NVBsL2tGdGxaSXVJT2tPYWVnR2VrODZXZkhTTFJRT2w5OGY1ayszdWd2WWpMaXJUZUN3VWRNOFk4WWN0NU5jbHZaVkFzTFNXNlg5c1hMUEt6eFNoS0MxWjZZV0poLzZBd3BVKysxQU5RQ2VtcDR2b29RU0xzR3VWM29acE5PamNKYkJKMjcvc1ZpdFJJL0JuQ1kxZkRGeWdBVDdzUEZCWmVPenUvQ3lTdHJsWGh6S1phdm5KY0ZyYUQ1YjhVZEY0VGxIZ3VSbjJldWc1RjViUmpwVEt2R1krd3p1SFphdU83SVhkem5UUkI1WTJUdFE3c3FaY1pORHhGUVdtMWQ0c3YzMUtYTTlCNmJJNFlEWCtISjFmQndocUp2dUtGNmV5L3N1WU83M3R2UEwwZ2V6OG1vcktpbVBHdEUzQnJiYVJ2VVRQVlVSd1FBaE11cm5PSDB2U0N5U2RmRDFCOWxrekpvWVpKTXlWOCtKdEpXVExrZ1o4b095ejFXZ2VBYmZGRlU2L3l4TTVHc0IvV3FYNW02UFVIeVhIKzJKTEt2MWlZWnRCcHNpd2hzZlhYTlhjNHRVL1JabXNwcXplZlRvQ2pxVUhSbFA2dGlPQ1k0SGV0SjhaWDZOeGFXUVZhQ2RDVUdGcnZPaUVIRUM2VzFqMTBJS08xZmhvT3owNGExNCtDZHlBT2EydVlxelpidUx0VWMzZXc3L0tWWU1aODh4N2wwRnlhVUZnS3BEQkV2b043d1IzSmdZcW41ZExwS1RBeHpSeDMxcDR2MkNXOTY4QWJMcUhoTmI0L21SVG1aMjlzTStWUjBWSlhPTGZBSXQzb1ppdHRrTkZFZ3VhY3AwbHlncHNkRngxY2F5MmhQQmhraHlITjFWS25mRlVZRGxnY2VrOWtrM0w5K2dUV09IeDNRbTlkSHM4Mzh1N3M3QStUbEVwUEthUlVlYWZicC9NMERBWmhPYzVqYnNRNmZKN3NCb1ZrSmthOVM0STFOQUdOTk1aeU8wPSIsImtleV92ZXJzaW9uIjoxfQ==
  immutable: true
  kind: Secret
  metadata:
    creationTimestamp: "2025-04-11T07:56:23Z"
    labels:
      app.kubernetes.io/component: client-cache-storage
      app.kubernetes.io/managed-by: vso
      app.kubernetes.io/name: vault-secrets-operator
      auth/UID: 18d135b3-427e-4035-accd-c69d343faad9
      auth/generation: "1"
      auth/namespace: demo-ns
      cacheKey: kubernetes-0fcd6d53bcef79a3934f85
      connection/UID: 618370ab-737c-4b84-8a6d-1dcb551eea8d
      connection/generation: "1"
      connection/namespace: vault-secrets-operator-system
      encrypted: "true"
      provider/UID: bd812bed-70a9-489c-87f4-c0c876de3687
      provider/namespace: demo-ns
      vaultTransitRef: vault-secrets-operator-default-transit-auth
    name: vso-cc-kubernetes-0fcd6d53bcef79a3934f85
    namespace: vault-secrets-operator-system
    resourceVersion: "54126"
    uid: eeb8135d-5ee9-49dc-80db-547ea39fc562
  type: Opaque
kind: List
metadata:
  resourceVersion: ""

김성중

I'm SJ

이전 포스트

AEWS3 - 9주차 EKS Upgrade

다음 포스트