CI/CD STUDY 6주차 1

Cluster Management

kind mgmt k8s 배포 + ingress-nginx + Argo CD

kind k8s 배포

kind create cluster --name **mgmt** --image kindest/node:v1.32.8 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: **control-plane
  labels:
    ingress-ready: true**
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
  - containerPort: 30000
    hostPort: 30000
EOF

nginx ingress 배포

**kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
# ingress-nginx-controller 수정**
KUBE_EDITOR="nano" **kubectl edit -n ingress-nginx deployments/ingress-nginx-controller

    spec:
      automountServiceAccountToken: true
      containers:
      - args:
        - /nginx-ingress-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --watch-ingress-without-class=true
        - --publish-status-address=localhost
        - --enable-ssl-passthrough**

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout argocd.example.com.key \
  -out argocd.example.com.crt \
  -subj "/CN=**argocd.example.com**/O=**argocd**"

openssl req: 인증서 요청 생성

-x509 : 즉시 self-signed 인증서를 생성

-nodes: private key를 암호로 보호하지 않음

-newkey rsa:

2048: RSA 2048bit 새 private key 생성
key와 인증서를 동시에 생성

-keyout argocd.example.com.key:

생성될 private key 파일경로

-out: argocd.example.com.crt

생성될 self-sigend 인증서 파일 경로

-subj "/CN=argocd.example.com/O=argocd"

사용자 입력 없이 인증서 정보를 자동으로 넣음
CN = Common Name : 인증서의 도메인이름
O = organization

kubectl create ns **argocd**

secret 생성

**kubectl -n argocd create secret tls argocd-server-tls \
  --cert=argocd.example.com.crt \
  --key=argocd.example.com.key

kubectl get secret -n argocd
NAME                TYPE                DATA   AGE
argocd-server-tls   kubernetes.io/tls   2      16s**

argocd-values.yaml

global:
  domain: argocd.example.com

server:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    tls: true

argocd helm

helm repo add argo https://argoproj.github.io/argo-helm
helm install **argocd** argo/argo-cd --version 9.0.5 -f argocd-values.yaml --namespace argocd

password 확인

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ;echo

argocd CLI 로그인

 argocd login argocd.example.com --insecure --username admin --password "password"

password 변경

argocd account update-password --current-password "password" --new-password "newPassword"

kind dev/prd k8s 배포 & k8s 자격증명 수정

docker network 확인 : mgmt 컨테이너 IP 확인

 "Containers": {
      "ed8e067be0fde5ac19e71cab19b9429bb41e1b415e1b2d09cfc007177f3a678f": {
        "Name": "mgmt-control-plane",
        "EndpointID": "346d666b6f0064b43d84422480accb503c900191e1f1771043c31410b5767f91",
        "MacAddress": "12:9b:32:a0:10:07",
        "IPv4Address": "172.18.0.2/16",
        "IPv6Address": "fc00:f853:ccd:e793::2/64"
      }
    },

kind k8s 배포

### dev

kind create cluster --name **dev** --image kindest/node:v1.32.8 --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: **control-plane**
  extraPortMappings:
  - containerPort: 31000
    hostPort: 31000
EOF

# mgmt k8s 자격증명 변경
**kubectl config get-contexts

kubectl config use-context kind-mgmt**
kubectl config get-contexts

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get node -v=6 --context kind-mgmt
I1118 00:07:35.339827   39321 cmd.go:527] kubectl command headers turned on
I1118 00:07:35.343161   39321 loader.go:402] Config loaded from file:  /root/.kube/config
I1118 00:07:35.344046   39321 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I1118 00:07:35.344141   39321 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I1118 00:07:35.344159   39321 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1118 00:07:35.344171   39321 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1118 00:07:35.344194   39321 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I1118 00:07:35.380178   39321 round_trippers.go:632] "Response" verb="GET" url="https://127.0.0.1:33675/api/v1/nodes?limit=500" status="200 OK" milliseconds=26
NAME                 STATUS   ROLES           AGE   VERSION
mgmt-control-plane   Ready    control-plane   57m   v1.32.8
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get node -v=6 --context kind-dev
I1118 00:07:39.679771   39350 cmd.go:527] kubectl command headers turned on
I1118 00:07:39.682772   39350 loader.go:402] Config loaded from file:  /root/.kube/config
I1118 00:07:39.683083   39350 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1118 00:07:39.683111   39350 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1118 00:07:39.683115   39350 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I1118 00:07:39.683119   39350 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I1118 00:07:39.683121   39350 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I1118 00:07:39.697392   39350 round_trippers.go:632] "Response" verb="GET" url="https://127.0.0.1:42917/api/v1/nodes?limit=500" status="200 OK" milliseconds=7
NAME                STATUS   ROLES           AGE   VERSION
dev-control-plane   Ready    control-plane   29m   v1.32.8

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get pod -A --context kind-mgmt
NAMESPACE            NAME                                               READY   STATUS    RESTARTS   AGE
argocd               argocd-application-controller-0                    1/1     Running   0          38m
argocd               argocd-applicationset-controller-bbff79c6f-8qmwd   1/1     Running   0          38m
argocd               argocd-dex-server-6877ddf4f8-j2k88                 1/1     Running   0          38m
argocd               argocd-notifications-controller-7b5658fc47-phx82   1/1     Running   0          38m
argocd               argocd-redis-7d948674-z9d2n                        1/1     Running   0          38m
argocd               argocd-repo-server-7679dc55f5-stgp5                1/1     Running   0          38m
argocd               argocd-server-7d769b6f48-fldxd                     1/1     Running   0          38m
ingress-nginx        ingress-nginx-controller-5b89cb54f9-pwnn8          1/1     Running   0          52m
kube-system          coredns-668d6bf9bc-4mjkh                           1/1     Running   0          57m
kube-system          coredns-668d6bf9bc-6l4jz                           1/1     Running   0          57m
kube-system          etcd-mgmt-control-plane                            1/1     Running   0          57m
kube-system          kindnet-bvv7h                                      1/1     Running   0          57m
kube-system          kube-apiserver-mgmt-control-plane                  1/1     Running   0          57m
kube-system          kube-controller-manager-mgmt-control-plane         1/1     Running   0          57m
kube-system          kube-proxy-vh76k                                   1/1     Running   0          57m
kube-system          kube-scheduler-mgmt-control-plane                  1/1     Running   0          57m
local-path-storage   local-path-provisioner-7dc846544d-9r98k            1/1     Running   0          57m
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get pod -A --context kind-dev
NAMESPACE            NAME                                        READY   STATUS    RESTARTS   AGE
kube-system          coredns-668d6bf9bc-c7wtg                    1/1     Running   0          29m
kube-system          coredns-668d6bf9bc-q6dbn                    1/1     Running   0          29m
kube-system          etcd-dev-control-plane                      1/1     Running   0          30m
kube-system          kindnet-bmdpz                               1/1     Running   0          29m
kube-system          kube-apiserver-dev-control-plane            1/1     Running   0          30m
kube-system          kube-controller-manager-dev-control-plane   1/1     Running   0          30m
kube-system          kube-proxy-mmkct                            1/1     Running   0          29m
kube-system          kube-scheduler-dev-control-plane            1/1     Running   0          30m
local-path-storage   local-path-provisioner-7dc846544d-g4gfp     1/1     Running   0          29m

alias 설정

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# alias k8s1='kubectl --context kind-mgmt'
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# alias k8s2='kubectl --context kind-dev

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# k8s1 get node -owide
NAME                 STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                     CONTAINER-RUNTIME
mgmt-control-plane   Ready    control-plane   58m   v1.32.8   172.18.0.2    <none>        Debian GNU/Linux 12 (bookworm)   6.6.87.2-microsoft-standard-WSL2   containerd://2.1.3
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# k8s2 get node -owide
NAME                STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                     CONTAINER-RUNTIME
dev-control-plane   Ready    control-plane   31m   v1.32.8   172.18.0.3    <none>        Debian GNU/Linux 12 (bookworm)   6.6.87.2-microsoft-standard-WSL2   containerd://2.1.3

docker network 확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# docker network inspect kind | grep -E 'Name|IPv4Address'
        "Name": "kind",
                "Name": "dev-control-plane",
                "IPv4Address": "172.18.0.3/16",
                "Name": "mgmt-control-plane",
                "IPv4Address": "172.18.0.2/16",

dev/prd k8s 에 api server 주소 컨테이너 IP로 변경

**vi ~/.kube/config

    server: https://172.18.0.3:6443
  name: kind-dev
- cluster:
    certificate-authority-data: 
    server: https://172.18.0.2:6443
  name: kind-mgmt**

도메인 통신 확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# docker exec -it mgmt-control-plane curl -sk https://dev-control-plane:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.8",
  "gitCommit": "2e83bc4bf31e88b7de81d5341939d5ce2460f46f",
  "gitTreeState": "clean",
  "buildDate": "2025-08-13T14:21:22Z",
  "goVersion": "go1.23.11",
  "compiler": "gc",
  "platform": "linux/amd64"

Argo CD에 다른 K8S Cluster 등록

alias 설정

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# alias k8s1='kubectl --context kind-dev'
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# alias k8s2='kubectl --context kind-mgmt'

dev k8s 등록

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# argocd cluster add kind-dev --name dev-k8s
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `kind-dev` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0002] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0002] ClusterRole "argocd-manager-role" updated
INFO[0002] ClusterRoleBinding "argocd-manager-role-binding" updated
Cluster 'https://172.18.0.3:6443' added

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# k8s1 get sa -n kube-system argocd-manager
NAME             SECRETS   AGE
argocd-manager   1         13m

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster
NAME                            TYPE     DATA   AGE
cluster-172.18.0.3-4100004299   Opaque   3      67s

k9s 확인

설치

curl -sS https://webinstall.dev/k9s | bash

k9s -> : secret argocd -> 아래 secret 에서 d (Describe) -> x (Toggle Decode) 로 확인

cluseter 확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# argocd cluster list
SERVER                          NAME        VERSION  STATUS   MESSAGE                                                  PROJECT
https://172.18.0.3:6443         dev-k8s              Unknown  Cluster has no applications and is not being monitored.
https://kubernetes.default.svc  in-cluster           Unknown  Cluster has no applications and is not being monitored.

Argo CD로 3개의 K8S Cluster 에 각각 Nginx 배포

context 확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl config get-contexts
CURRENT   NAME        CLUSTER     AUTHINFO    NAMESPACE
          kind-dev    kind-dev    kind-dev
*         kind-mgmt   kind-mgmt   kind-mgmt

IP 확인

docker network inspect kind | grep -E 'Name|IPv4Address'
        "Name": "kind",
                "Name": "dev-control-plane",
                "IPv4Address": "172.18.0.3/16",
                "Name": "mgmt-control-plane",
                "IPv4Address": "172.18.0.2/16",

DEVK8SIP=172.18.0.3
echo $DEVK8SIP

argocd app 배포

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: **Application**
metadata:
  name: **mgmt-nginx**
  namespace: argocd
  **finalizers**:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  **source**:
    helm:
      valueFiles:
      - **values.yaml**
    path: **nginx-chart**
    repoURL: https://github.com/gasida/cicd-study
    targetRevision: HEAD
  **syncPolicy**:
    **automated**:
      **prune: true**
    syncOptions:
    - CreateNamespace=true
  **destination**:
    namespace: **mgmt-nginx**
    server: https://kubernetes.default.svc
EOF

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: **Application**
metadata:
  name: **dev-nginx**
  namespace: argocd
  **finalizers**:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  **source**:
    helm:
      valueFiles:
      - **values-dev.yaml**
    path: **nginx-chart**
    repoURL: https://github.com/gasida/cicd-study
    targetRevision: HEAD
  **syncPolicy**:
    **automated**:
      **prune: true**
    syncOptions:
    - CreateNamespace=true
  **destination**:
    namespace: **dev-nginx**
    server: https://$DEVK8SIP:6443
EOF

확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# argocd app list
NAME        CLUSTER                         NAMESPACE   PROJECT  STATUS  HEALTH       SYNCPOLICY  CONDITIONS  REPO                                  PATH         TARGET
dev-nginx   https://172.18.0.3:6443         dev-nginx   default  Synced  Progressing  Auto-Prune  <none>      https://github.com/gasida/cicd-study  nginx-chart  HEAD
mgmt-nginx  https://kubernetes.default.svc  mgmt-nginx  default  Synced  Progressing  Auto-Prune  <none>      https://github.com/gasida/cicd-study  nginx-chart  HEAD

kubectl get applications -n argocd dev-nginx -o yaml

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applications -n argocd
NAME         SYNC STATUS   HEALTH STATUS
dev-nginx    Synced        Healthy
mgmt-nginx   Synced        Healthy

nginx cluster 통신 mgmt

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get pod,svc,ep,cm -n mgmt-nginx
NAME                              READY   STATUS    RESTARTS   AGE
pod/mgmt-nginx-6fc86948bc-4nngm   1/1     Running   0          2m48s

NAME                 TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/mgmt-nginx   NodePort   10.96.232.254   <none>        80:30000/TCP   2m48s

NAME                   ENDPOINTS        AGE
endpoints/mgmt-nginx   10.244.0.13:80   2m48s

NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      2m49s
configmap/mgmt-nginx         1      2m48s
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# curl -s http://127.0.0.1:30000
<!DOCTYPE html>
<html>
<head>
  <title>Welcome to Nginx!</title>
</head>
<body>
  <h1>Hello, Kubernetes!</h1>
  <p>Nginx version 1.26.1</p>
</body>
</html>

nginx cluster 통신 dev

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get pod,svc,ep,cm -n dev-nginx --context kind-dev
NAME                            READY   STATUS    RESTARTS   AGE
pod/dev-nginx-59f4c8899-sc24c   1/1     Running   0          3m29s

NAME                TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/dev-nginx   NodePort   10.96.197.209   <none>        80:31000/TCP   3m30s

NAME                  ENDPOINTS       AGE
endpoints/dev-nginx   10.244.0.5:80   3m30s

NAME                         DATA   AGE
configmap/dev-nginx          1      3m30s
configmap/kube-root-ca.crt   1      3m30s

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# curl -s http://127.0.0.1:31000
<!DOCTYPE html>
<html>
<head>
  <title>Welcome to Nginx!</title>
</head>
<body>
  <h1>Hello, Dev - Kubernetes!</h1>
  <p>Nginx version 1.26.1</p>
</body>
</html>

ApplicationSet

app of apps 패턴

참조 : https://tech.ktcloud.com/entry/2025-05-ktcloud-kubernetes-gitops-appofapps-구현환경-전략

ArgoCD의 App of Apps 패턴은 루트(Root) Application이 여러 개의 하위(Child) Application 리소스를 관리하고 배포하는 계층적 배포 전략입니다.

루트 Application (Root Application):
◦ Git 저장소 내에 정의된 하위 Application 리소스(CRD)들을 가리키는 ArgoCD Application입니다.
◦ 이 루트 앱이 동기화되면, Git에 정의된 모든 하위 Application들이 ArgoCD에 의해 생성되고 관리됩니다.
◦ 일반적으로 클러스터 초기 구성(Bootstrapping)이나 환경별 공통 애플리케이션 관리에 사용됩니다.
하위 Application (Child Application):
◦ 실제 Kubernetes 리소스(Deployment, Service, Ingress 등)를 포함하는 Git 경로를 가리키는 개별 ArgoCD Application입니다.
◦ 각 하위 앱은 마이크로서비스나 플랫폼 컴포넌트 등 논리적인 배포 단위를 나타냅니다.

한계점

구분	한계점 (Limitation)	상세 설명 (Detail)
확장성 및 동적 생성	동적 앱 생성 불가	배포할 하위 Application 목록이 정적 YAML에 하드코딩되어 있어, 고객 테넌트 증가 등 동적인 상황에 따라 Application을 자동으로 생성할 수 없습니다.
관리 복잡성	Git/App 구조의 높은 결합도	Git 저장소의 폴더 구조 변경 시, 이를 참조하는 모든 하위 앱과 루트 앱 YAML을 수동으로 수정해야 하므로 관리 오버헤드가 큽니다.
멀티 클러스터 관리	자동 확장 및 Discovery 부재	여러 클러스터에 동일한 앱을 배포하려면, 클러스터 수만큼 App 정의를 수동으로 반복 추가해야 합니다. ArgoCD에 새 클러스터가 추가되어도 자동으로 배포하지 못합니다.
스케일링 문제	루트 앱 YAML의 비대화	테넌트/앱이 증가할수록 루트 Application YAML 파일이 비대해져서 Git Conflict 발생 가능성이 높아지고, 파일 관리 및 ArgoCD 동기화 성능이 저하될 수 있습니다.
유연성	선언적(Dynamic) 구성의 어려움	단순 정적 목록이기 때문에, Helm values의 값이나 클러스터의 특정 레이블 등 동적인 조건을 기반으로 Application을 생성하는 유연한 배포 전략을 구현하기 어렵습니다.

applicationSet

Generators - Argo CD - Declarative GitOps CD for Kubernetes

ApplicationSet은 여러 개의 ArgoCD Application 리소스를 자동으로 생성하고 관리하는 컨트롤러입니다

Application: 실제 Kubernetes 리소스(Deployment, Service 등)의 배포를 담당하는 작업 단위입니다.
ApplicationSet: 반복되는 Application들을 찍어내는 공장 또는 설계도 역할을 합니다.

핵심 구성요소

ApplicationSet 리소스는 모든 기능을 Generator와 Template의 결합으로 수행합니다.

Generator

Application을 생성하는 데 필요한 동적인 변수 데이터를 제공

Generator 종류	역할 및 생성 데이터	사용 사례
Cluster Generator	ArgoCD에 등록된 모든 클러스터의 정보(`server`, `name`, `labels`)를 변수로 생성	모든 클러스터에 공통 인프라(로그 수집기 등) 배포
Git Generator	Git 저장소 내의 특정 폴더나 파일 내용을 스캔하여 해당 정보를 변수로 생성	폴더별로 서비스/테넌트 Application을 자동 생성
List Generator	YAML 파일 내에 정의된 정적인 변수 목록을 생성	지역(Region) 코드, 환경(Dev/Prod) 이름 등 특정 값 목록을 순회하며 배포
Matrix Generator	두 개 이상의 Generator 출력을 곱집합(Cross Product)하여 새로운 변수 세트를 생성	3개 클러스터 $\times$ 5개 Git 폴더 = 총 15개의 Application 생성

Template

Template은 실제로 생성될 ArgoCD Application의 YAML 정의

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - cluster: engineering-dev
        url: https://1.2.3.4
      - cluster: engineering-staging
        url: https://2.4.6.8
      - cluster: engineering-prod
        url: https://9.8.7.6
  template:
    metadata:
      name: '{{.cluster}}-guestbook'
    spec:
      project: "my-project"
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: applicationset/examples/list-generator/guestbook/{{.cluster}}
      destination:
        server: '{{.url}}'
        namespace: guestbook

ApplicationSet List 제네레이터

ip check

docker network inspect kind | grep -E 'Name|IPv4Address'

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# docker network inspect kind | grep -E 'Name|IPv4Address'
        "Name": "kind",
                "Name": "dev-control-plane",
                "IPv4Address": "172.18.0.3/16",
                "Name": "mgmt-control-plane",
                "IPv4Address": "172.18.0.2/16",
 (⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# DEVK8SIP=172.18.0.3
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# echo $DEVK8SIP
172.18.0.3
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~#

argocd app 배포


cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: **ApplicationSet**
metadata:
  name: guestbook
  namespace: **argocd**
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  **generators:
  - list:**
      elements:
      - cluster: dev-k8s
        url: https://$DEVK8SIP:6443
  template:
    metadata:
      name: '{{.cluster}}-guestbook'
      labels:
        environment: '{{.cluster}}'
        managed-by: applicationset
    spec:
      project: default
      source:
        repoURL: https://github.com/gasida/cicd-study.git
        targetRevision: HEAD
        path: appset/list/{{.cluster}}
      destination:
        server: '{{.url}}'
        namespace: guestbook
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
EOF

확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applicationsets -n argocd guestbook -o yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"ApplicationSet","metadata":{"annotations":{},"name":"guestbook","namespace":"argocd"},"spec":{"generators":[{"list":{"elements":[{"cluster":"dev-k8s","url":"https://172.18.0.3:6443"}]}}],"goTemplate":true,"goTemplateOptions":["missingkey=error"],"template":{"metadata":{"labels":{"environment":"{{.cluster}}","managed-by":"applicationset"},"name":"{{.cluster}}-guestbook"},"spec":{"destination":{"namespace":"guestbook","server":"{{.url}}"},"project":"default","source":{"path":"appset/list/{{.cluster}}","repoURL":"https://github.com/gasida/cicd-study.git","targetRevision":"HEAD"},"syncPolicy":{"syncOptions":["CreateNamespace=true"]}}}}}
  creationTimestamp: "2025-11-19T09:55:43Z"
  generation: 1
  name: guestbook
  namespace: argocd
  resourceVersion: "20747"
  uid: e6599ef3-4dc7-4cce-8776-99db775f24fc
spec:
  generators:
  - list:
      elements:
      - cluster: dev-k8s
        url: https://172.18.0.3:6443
  goTemplate: true
  goTemplateOptions:
  - missingkey=error
  template:
    metadata:
      labels:
        environment: '{{.cluster}}'
        managed-by: applicationset
      name: '{{.cluster}}-guestbook'
    spec:
      destination:
        namespace: guestbook
        server: '{{.url}}'
      project: default
      source:
        path: appset/list/{{.cluster}}
        repoURL: https://github.com/gasida/cicd-study.git
        targetRevision: HEAD
      syncPolicy:
        syncOptions:
        - CreateNamespace=true
status:
  conditions:
  - lastTransitionTime: "2025-11-19T09:55:43Z"
    message: Successfully generated parameters for all Applications
    reason: ApplicationSetUpToDate
    status: "False"
    type: ErrorOccurred
  - lastTransitionTime: "2025-11-19T09:55:43Z"
    message: Successfully generated parameters for all Applications
    reason: ParametersGenerated
    status: "True"
    type: ParametersGenerated
  - lastTransitionTime: "2025-11-19T09:55:43Z"
    message: ApplicationSet up to date
    reason: ApplicationSetUpToDate
    status: "True"
    type: ResourcesUpToDate
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applicationsets -n argocd guestbook -o yaml | k neat | yq
{
  "apiVersion": "argoproj.io/v1alpha1",
  "kind": "ApplicationSet",
  "metadata": {
    "name": "guestbook",
    "namespace": "argocd"
  },
  "spec": {
    "generators": [
      {
        "list": {
          "elements": [
            {
              "cluster": "dev-k8s",
              "url": "https://172.18.0.3:6443"
            }
          ]
        }
      }
    ],
    "goTemplate": true,
    "goTemplateOptions": [
      "missingkey=error"
    ],
    "template": {
      "metadata": {
        "labels": {
          "environment": "{{.cluster}}",
          "managed-by": "applicationset"
        },
        "name": "{{.cluster}}-guestbook"
      },
      "spec": {
        "destination": {
          "namespace": "guestbook",
          "server": "{{.url}}"
        },
        "project": "default",
        "source": {
          "path": "appset/list/{{.cluster}}",
          "repoURL": "https://github.com/gasida/cicd-study.git",
          "targetRevision": "HEAD"
        },
        "syncPolicy": {
          "syncOptions": [
            "CreateNamespace=true"
          ]
        }
      }
    }
  }
}
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applicationsets -n argocd
NAME        AGE
guestbook   14s

troubleshooting (token 만료 )

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# argocd app list
FATA[0000] rpc error: code = Unauthenticated desc = invalid session: token has invalid claims: token is expired

💡

'admin:login' logged in successfully
Context 'argocd.example.com' updated
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# argocd app list
NAME               CLUSTER                         NAMESPACE   PROJECT  STATUS     HEALTH   SYNCPOLICY  CONDITIONS  REPO                                      PATH                 TARGET
dev-k8s-guestbook  https://172.18.0.3:6443         guestbook   default  OutOfSync  Missing  <none>      <none>      https://github.com/gasida/cicd-study.git  appset/list/dev-k8s  HEAD
dev-nginx          https://172.18.0.3:6443         dev-nginx   default  Synced     Healthy  Auto-Prune  <none>      https://github.com/gasida/cicd-study      nginx-chart          HEAD
mgmt-nginx         https://kubernetes.default.svc  mgmt-nginx  default  Synced     Healthy  Auto-Prune  <none>      https://github.com/gasida/cicd-study      nginx-chart          HEAD

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# argocd app list -l managed-by=applicationset
NAME               CLUSTER                  NAMESPACE  PROJECT  STATUS     HEALTH   SYNCPOLICY  CONDITIONS  REPO                                      PATH                 TARGET
dev-k8s-guestbook  https://172.18.0.3:6443  guestbook  default  OutOfSync  Missing  <none>      <none>      https://github.com/gasida/cicd-study.git  appset/list/dev-k8s  HEAD

sync 진행

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applications -n argocd --show-labels
NAME                SYNC STATUS   HEALTH STATUS   LABELS
dev-k8s-guestbook   OutOfSync     Missing         environment=dev-k8s,managed-by=applicationset
dev-nginx           Synced        Healthy         <none>
mgmt-nginx          Synced        Healthy         <none>

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applications -n argocd dev-k8s-guestbook -o yaml | k neat | yq
{
  "apiVersion": "argoproj.io/v1alpha1",
  "kind": "Application",
  "metadata": {
    "labels": {
      "environment": "dev-k8s",
      "managed-by": "applicationset"
    },
    "name": "dev-k8s-guestbook",
    "namespace": "argocd"
  },
  "spec": {
    "destination": {
      "namespace": "guestbook",
      "server": "https://172.18.0.3:6443"
    },
    "project": "default",
    "source": {
      "path": "appset/list/dev-k8s",
      "repoURL": "https://github.com/gasida/cicd-study.git",
      "targetRevision": "HEAD"
    },
    "syncPolicy": {
      "syncOptions": [
        "CreateNamespace=true"
      ]
    }
  }
}

applicationset 삭제

 kubectl delete applicationsets guestbook -n argocd

ApplicationSet Cluster 제네레이터

applicationSet cluster 배포

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: **ApplicationSet**
metadata:
  name: **guestbook**
  namespace: **argocd**
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  **generators:
  - clusters: {}**
  template:
    metadata:
      name: **'{{.name}}-guestbook'**
      labels:
        managed-by: applicationset
    spec:
      project: "default"
      source:
        repoURL: https://github.com/gasida/cicd-study
        targetRevision: HEAD
        path: guestbook
      destination:
        server: **'{{.server}}'**
        namespace: guestbook
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
EOF

확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applicationsets -n argocd guestbook -o yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"ApplicationSet","metadata":{"annotations":{},"name":"guestbook","namespace":"argocd"},"spec":{"generators":[{"clusters":{}}],"goTemplate":true,"goTemplateOptions":["missingkey=error"],"template":{"metadata":{"labels":{"managed-by":"applicationset"},"name":"{{.name}}-guestbook"},"spec":{"destination":{"namespace":"guestbook","server":"{{.server}}"},"project":"default","source":{"path":"guestbook","repoURL":"https://github.com/gasida/cicd-study","targetRevision":"HEAD"},"syncPolicy":{"syncOptions":["CreateNamespace=true"]}}}}}
  creationTimestamp: "2025-11-19T10:19:38Z"
  generation: 1
  name: guestbook
  namespace: argocd
  resourceVersion: "23105"
  uid: 9e699131-1f28-4181-ba71-f0b6370c0164
spec:
  generators:
  - clusters: {}
  goTemplate: true
  goTemplateOptions:
  - missingkey=error
  template:
    metadata:
      labels:
        managed-by: applicationset
      name: '{{.name}}-guestbook'
    spec:
      destination:
        namespace: guestbook
        server: '{{.server}}'
      project: default
      source:
        path: guestbook
        repoURL: https://github.com/gasida/cicd-study
        targetRevision: HEAD
      syncPolicy:
        syncOptions:
        - CreateNamespace=true
status:
  conditions:
  - lastTransitionTime: "2025-11-19T10:19:38Z"
    message: Successfully generated parameters for all Applications
    reason: ApplicationSetUpToDate
    status: "False"
    type: ErrorOccurred
  - lastTransitionTime: "2025-11-19T10:19:38Z"
    message: Successfully generated parameters for all Applications
    reason: ParametersGenerated
    status: "True"
    type: ParametersGenerated
  - lastTransitionTime: "2025-11-19T10:19:38Z"
    message: ApplicationSet up to date
    reason: ApplicationSetUpToDate
    status: "True"
    type: ResourcesUpToDate
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applicationsets -n argocd
NAME        AGE
guestbook   59s

mgmt-nginx            https://kubernetes.default.svc  mgmt-nginx  default  Synced     Healthy  Auto-Prune  <none>      https://github.com/gasida/cicd-study  nginx-chart  HEAD
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# argocd app list -l managed-by=applicationset
NAME                  CLUSTER                         NAMESPACE  PROJECT  STATUS     HEALTH   SYNCPOLICY  CONDITIONS  REPO                                  PATH       TARGET
dev-k8s-guestbook     https://172.18.0.3:6443         guestbook  default  OutOfSync  Missing  <none>      <none>      https://github.com/gasida/cicd-study  guestbook  HEAD
in-cluster-guestbook  https://kubernetes.default.svc  guestbook  default  OutOfSync  Missing  <none>      <none>      https://github.com/gasida/cicd-study  guestbook  HEAD

sync

**argocd app sync -l managed-by=applicationset**

application yaml 확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applications -n argocd in-cluster-guestbook -o yaml | k neat | yq
{
  "apiVersion": "argoproj.io/v1alpha1",
  "kind": "Application",
  "metadata": {
    "labels": {
      "managed-by": "applicationset"
    },
    "name": "in-cluster-guestbook",
    "namespace": "argocd"
  },
  "spec": {
    "destination": {
      "namespace": "guestbook",
      "server": "https://kubernetes.default.svc"
    },
    "project": "default",
    "source": {
      "path": "guestbook",
      "repoURL": "https://github.com/gasida/cicd-study",
      "targetRevision": "HEAD"
    },
    "syncPolicy": {
      "syncOptions": [
        "CreateNamespace=true"
      ]
    }
  }
}
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applications -n argocd dev-k8s-guestbook -o yaml | k neat | yq
{
  "apiVersion": "argoproj.io/v1alpha1",
  "kind": "Application",
  "metadata": {
    "labels": {
      "managed-by": "applicationset"
    },
    "name": "dev-k8s-guestbook",
    "namespace": "argocd"
  },
  "spec": {
    "destination": {
      "namespace": "guestbook",
      "server": "https://172.18.0.3:6443"
    },
    "project": "default",
    "source": {
      "path": "guestbook",
      "repoURL": "https://github.com/gasida/cicd-study",
      "targetRevision": "HEAD"
    },
    "syncPolicy": {
      "syncOptions": [
        "CreateNamespace=true"
      ]
    }
  }
}

삭제

kubectl delete applicationset guestbook -n argocd

dev k8s 만 배포 ( ApplicationSet Cluster 제네레이터)

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster
NAME                            TYPE     DATA   AGE
cluster-172.18.0.3-4100004299   Opaque   3      21h
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# docker network inspect kind | grep -E 'Name|IPv4Address'
        "Name": "kind",
                "Name": "dev-control-plane",
                "IPv4Address": "172.18.0.3/16",
                "Name": "mgmt-control-plane",
                "IPv4Address": "172.18.0.2/16",
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# DEVK8S=cluster-172.18.0.3-4100004299
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl label secrets $DEVK8S -n argocd env=stg
secret/cluster-172.18.0.3-4100004299 labeled
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get secret -n argocd -l env=stg
NAME                            TYPE     DATA   AGE
cluster-172.18.0.3-4100004299   Opaque   3      21h

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: **ApplicationSet**
metadata:
  name: **guestbook**
  namespace: **argocd**
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  **generators:
  - clusters:
      selector:
        matchLabels:
          env: "stg"**
  template:
    metadata:
      name: **'{{.name}}-guestbook'**
      labels:
        managed-by: applicationset
    spec:
      project: "default"
      source:
        repoURL: https://github.com/gasida/cicd-study
        targetRevision: HEAD
        path: guestbook
      destination:
        server: **'{{.server}}'**
        namespace: guestbook
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
        **automated:
          prune: true
          selfHeal: true**
EOF

dev만 배포된거 확인

(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applicationsets -n argocd
NAME        AGE
guestbook   12s
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# argocd app list -l managed-by=applicationset
NAME               CLUSTER                  NAMESPACE  PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                  PATH       TARGET
dev-k8s-guestbook  https://172.18.0.3:6443  guestbook  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/gasida/cicd-study  guestbook  HEAD
(⎈|kind-mgmt:N/A) root@DESKTOP-8S932ET:~# kubectl get applications -n argocd --show-labels
NAME                SYNC STATUS   HEALTH STATUS   LABELS
dev-k8s-guestbook   Synced        Healthy         managed-by=applicationset

삭제

kubectl delete applicationset guestbook -n argocd

dev_suha

이전 포스트

CI/CD STUDY 5주차

다음 포스트

CI/CD STUDY 6주차 1

Cluster Management

kind mgmt k8s 배포 + ingress-nginx + Argo CD

kind k8s 배포

nginx ingress 배포

secret 생성

argocd-values.yaml

argocd helm

password 확인

argocd CLI 로그인

password 변경

kind dev/prd k8s 배포 & k8s 자격증명 수정

docker network 확인 : mgmt 컨테이너 IP 확인

kind k8s 배포

alias 설정

docker network 확인

dev/prd k8s 에 api server 주소 컨테이너 IP로 변경

도메인 통신 확인

Argo CD에 다른 K8S Cluster 등록

alias 설정

dev k8s 등록

k9s 확인

cluseter 확인

Argo CD로 3개의 K8S Cluster 에 각각 Nginx 배포

context 확인

IP 확인

argocd app 배포

확인

nginx cluster 통신 mgmt

nginx cluster 통신 dev

ApplicationSet

app of apps 패턴

applicationSet

핵심 구성요소

ApplicationSet List 제네레이터

ip check

argocd app 배포

확인

troubleshooting (token 만료 )

sync 진행

applicationset 삭제

ApplicationSet Cluster 제네레이터

applicationSet cluster 배포

확인

sync

application yaml 확인

삭제

dev k8s 만 배포 ( ApplicationSet Cluster 제네레이터)

dev만 배포된거 확인

삭제

CI/CD STUDY 5주차

CI/CD STUDY 6주차 2 KeyCloak SSO with CI/CD

0개의 댓글