vault on k8s(kubernetes)

k8s 노드 구조

kubectl get nodes

NAME         STATUS   ROLES                       AGE    VERSION
k8smaster1   Ready    control-plane,etcd,master   3d6h   v1.25.11+rke2r1
k8sworker1   Ready    <none>                      3d3h   v1.25.11+rke2r1
k8sworker2   Ready    <none>                      3d3h   v1.25.11+rke2r1
k8sworker3   Ready    <none>                      3d3h   v1.25.11+rke2r1

yaml 파일 작성

---

sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
volumeBindingMode: WaitForFirstConsumer
provisioner: kubernetes.io/no-provisioner
parameters:
  type: local
  path: /vault/data

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-vault-0 # PV 이름
spec:
  capacity:
    storage: 10Gi # min 10G
  accessModes:
  - ReadWriteOnce 
  storageClassName: local-storage # StorageClass 이름 
  local:
    path: /vault/data # local 데이터 경로 - 각 k8s 노드에 디렉토리 만들어야 함 
  nodeAffinity: # k8s 노드 설정
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8sworker1 # k8s 노드1 이름 
---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-vault-1
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  local:
    path: /vault/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8sworker2 # k8s 노드2 이름

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-vault-2
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  local:
    path: /vault/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8sworker3 # k8s 노드3 이름

values.yaml

# 들여쓰기 주의할 것
global:
  enabled: true

server:
  image:
    repository: "hashicorp/vault"
    tag: "1.14.0"

  service:
    enabled: true
    # Port on which Vault server is listening
    # valut 서버 listen port
    port: 8200
    # Target port to which the service should be mapped to 
    # pod 내에서 서비스로 접속할 수 있는 port 
    targetPort: 8200
    
    # If type is set to "NodePort", a specific nodePort value can be configured,
    # will be random if left blank.
    #nodePort: 30000

    # When HA mode is enabled
    # If type is set to "NodePort", a specific nodePort value can be configured,
    # will be random if left blank.
    #activeNodePort: 30001

    # When HA mode is enabled
    # If type is set to "NodePort", a specific nodePort value can be configured,
    # will be random if left blank.
    #standbyNodePort: 30002


	# StorageClass 설정 
	# pvc configure 내용
	# (추정) pod내에 각 노드에 설정한 pv파일들을 그대로 옮겨오는듯
  dataStorage:
    enabled: true
    size: 10Gi
    mountPath: "/vault/data"
    storageClass: local-storage
    accessMode: ReadWriteOnce
    annotations: {}

	# HA cluster 설정
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true

apply 순서

---

1. StorageClass

kubectl apply -f sc.yaml

2. PersistentVolume (PV)

kubectl apply -f pv.yaml

3. helm install

# ubuntu helm 설치
sudo snap install helm --classic 

# Helm Repo 추가
helm repo add hashicorp https://helm.releases.hashicorp.com
# Helm Repo 확인
helm repo list | grep hashicorp

#설치 조회
helm list 

#vault 설치
helm install vault hashicorp/vault -f values.yaml

vault HA 설정

---

1. pod 확인

kubectl get pods

NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          122m
vault-1                                 0/1     Running   0          122m
vault-2                                 0/1     Running   0          122m
vault-agent-injector-7688469fb4-gnbms   1/1     Running   0          122m

2. active할 pod (vault-0) init

1) pod에 직접 접속

kubectl exec -it <pod 명> -- sh

2) pod 외부에서 명령어 입력

kubectl exec <pod 명> -- <명령어>

• init

kubectl exec -it vault-0 -- sh

# init 내용 /init.txt에 저장
vault operator init -key-shares=1 -key-threshold=1 | tee /init.txt 

Unseal Key 1: 3Qixxxxxxxxxxxxxxxxxxxxxxxxxx

Initial Root Token: hvs.xxxxxxxxx....
....

3. active pod (vault-0) unseal

• unseal

vault operator unseal

unseal Key : (Unseal Key 입력)

• login

vault login

Token : (Initial Root Token 입력)

• 확인

vault status

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.14.0
Build Date              2023-06-19T11:40:23Z
Storage Type            raft
Cluster Name            vault-cluster-8eb67b84
Cluster ID              5889d8d3-9115-cb2e-401a-0fb54cd948b5
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2023-08-04T04:48:48.813065052Z
Raft Committed Index    42
Raft Applied Index      42

---
vault operator raft list-peers

Node                                    Address                        State       Voter
----                                    -------                        -----       -----
8ea01271-59fe-396d-e572-7004ab84dadb    vault-0.vault-internal:8201    leader      true

4. 나머지 pod : raft join → unseal

• raft join

kubectl exec -it vault-1 -- sh

# active pod에 join
# 주소 http, 8200 포트 주의하기

vault operator raft join http://vault-0.vault-internal:8200 

kubectl exec -it vault-2 -- sh

vault operator raft join http://vault-0.vault-internal:8200 

• unseal

# vault-1, vault-2 둘다
# unseal 전에는 raft list-peers에 나타나지 않음

vault operator unseal
unseal key : (active pod의 unseal key 입력)

• 확인

kubectl exec vault-0 -- vault operator raft list-peers

Node                                    Address                        State       Voter
----                                    -------                        -----       -----
8ea01271-59fe-396d-e572-7004ab84dadb    vault-0.vault-internal:8201    leader      true
2aaa02dc-cf7a-b5f2-aa96-23d7a4fe87ae    vault-1.vault-internal:8201    follower    true
7392193b-68a4-1934-f908-a5482fd49792    vault-2.vault-internal:8201    follower    true