3주차 - Jenkins CI/ArgoCD + K8S(2/2)

김성중·2024년 12월 21일

argocd gogs jenkins k8s kind

CI/CD

목록 보기

4/4

CloudNet@ 가시다님이 진행하는 단기 CI/CD 과정 Study 내용을 실습한 내용과 개인적인 경험을 정리하고자 합니다. 3주차 학습한 내용은 Jenkins CI/ArgoCD + K8S 입니다.

3. Jenkins CI/CD + K8S(Kind)

3.1 Jenkins 컨테이너 내부에 툴 설치 : kubectl, helm

Jenkins Privileged 권한으로 실행


# Privileged 권한으로 Jenkins 컨테이너 Shell 접속
❯ docker compose exec --privileged -u root jenkins bash

# Install kubectl, helm
root@05294177dcea:/# curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   138  100   138    0     0    657      0 --:--:-- --:--:-- --:--:--   660
100 53.2M  100 53.2M    0     0  10.4M      0  0:00:05  0:00:05 --:--:-- 11.0M

root@05294177dcea:/# install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
root@05294177dcea:/# kubectl version --client=true
Client Version: v1.32.0
Kustomize Version: v5.5.0

# Install Helm
root@05294177dcea:/# curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11903  100 11903    0     0  33859      0 --:--:-- --:--:-- --:--:-- 33911
Downloading https://get.helm.sh/helm-v3.16.4-linux-arm64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm

root@05294177dcea:/# helm version
version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"clean", GoVersion:"go1.22.7"}

root@05294177dcea:/# exit
exit
--------------------------------------------
❯ docker compose exec jenkins kubectl version --client=true
Client Version: v1.32.0
Kustomize Version: v5.5.0

❯ docker compose exec jenkins helm version
version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"clean", GoVersion:"go1.22.7"}

3.2 kubeconfig 파일을 Jenkins 컨테이너의 jenkins 홈디렉터리로 복사 : 보안상 비권장


# kubeconfig 파일을 Jenkins 컨테이너의 jenkins 홈디렉터리로 복사 : 보안상 비권장
❯ docker compose exec jenkins mkdir -p /var/jenkins_home/.kube
❯ docker compose cp kubeconfig jenkins:/var/jenkins_home/.kube/config
[+] Copying 1/0
 ✔ jenkins copy kubeconfig to jenkins:/var/jenkins_home/.kube/config Copied 
❯ docker compose exec --privileged -u root jenkins chown jenkins:jenkins /var/jenkins_home/.kube/config

# Jenkins 컨테이너에서 파드 정보 조회 확인
❯ docker compose exec jenkins kubectl get pod -A
NAMESPACE            NAME                                          READY   STATUS    RESTARTS   AGE
kube-system          coredns-55cb58b774-7tvrk                      1/1     Running   0          94s
kube-system          coredns-55cb58b774-9wnzt                      1/1     Running   0          94s
kube-system          etcd-myk8s-control-plane                      1/1     Running   0          111s
kube-system          kindnet-bxnlm                                 1/1     Running   0          95s
kube-system          kindnet-f7nxr                                 1/1     Running   0          91s
kube-system          kindnet-lgb99                                 1/1     Running   0          92s
kube-system          kube-apiserver-myk8s-control-plane            1/1     Running   0          111s
kube-system          kube-controller-manager-myk8s-control-plane   1/1     Running   0          112s
kube-system          kube-proxy-cp9c4                              1/1     Running   0          95s
kube-system          kube-proxy-czshn                              1/1     Running   0          91s
kube-system          kube-proxy-h27k7                              1/1     Running   0          92s
kube-system          kube-scheduler-myk8s-control-plane            1/1     Running   0          111s
local-path-storage   local-path-provisioner-7d4d9bdcc5-fr4j5       1/1     Running   0          94s

3.3 Jenkins Item 생성(Pipeline) : item name(k8s-cmd)

Jenkins Pipeline : k8s-cmd


pipeline {
    agent any
    environment {
        KUBECONFIG = credentials('k8s-crd')
    }
    stages {
        stage('List Pods') {
            steps {
                sh '''
                # Fetch and display Pods
                kubectl get pods -A --kubeconfig "$KUBECONFIG"
                '''
            }
        }
    }
}

Jenkins 파이프라인 실행결과

3.4 Jenkins 를 이용한 blue-green 배포 준비

디플로이먼트 / 서비스 yaml 파일 작성 - http-echo 및 코드 push


# dev-app으로 디렉토리 변경
❯ cd dev-app

# Deploy용 Directory 생성
❯ mkdir deploy

# Blue deployment 생성
❯ cat > deploy/echo-server-blue.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server-blue
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo-server
      version: blue
  template:
    metadata:
      labels:
        app: echo-server
        version: blue
    spec:
      containers:
      - name: echo-server
        image: hashicorp/http-echo
        args:
        - "-text=Hello from Blue"
        ports:
        - containerPort: 5678
EOF

# Echo Server 서비스 생성
❯ cat > deploy/echo-server-service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: echo-server-service
spec:
  selector:
    app: echo-server
    version: blue
  ports:
  - protocol: TCP
    port: 80
    targetPort: 5678
    nodePort: 30000
  type: NodePort
EOF

❯ cat > deploy/echo-server-green.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server-green
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo-server
      version: green
  template:
    metadata:
      labels:
        app: echo-server
        version: green
    spec:
      containers:
      - name: echo-server
        image: hashicorp/http-echo
        args:
        - "-text=Hello from Green"
        ports:
        - containerPort: 5678
EOF

# Git push
❯ git add . && git commit -m "Add echo server yaml" && git push -u origin main

[main bd86a11] Add echo server yaml
 3 files changed, 60 insertions(+)
 create mode 100644 deploy/echo-server-blue.yaml
 create mode 100644 deploy/echo-server-green.yaml
 create mode 100644 deploy/echo-server-service.yaml
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 789 bytes | 789.00 KiB/s, done.
Total 6 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
To http://mypc:3000/devops/dev-app.git
   fb350a4..bd86a11  main -> main
branch 'main' set up to track 'origin/main'.

3.5 직접 블루-그린 업데이트 실행


❯ cd deploy

❯ kubectl delete deploy,svc --all
service "kubernetes" deleted

❯ kubectl apply -f .
deployment.apps/echo-server-blue created
deployment.apps/echo-server-green created
service/echo-server-service created

❯ kubectl get deploy,svc,ep -owide
NAME                                READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS    IMAGES                SELECTOR
deployment.apps/echo-server-blue    2/2     2            2           15s   echo-server   hashicorp/http-echo   app=echo-server,version=blue
deployment.apps/echo-server-green   2/2     2            2           15s   echo-server   hashicorp/http-echo   app=echo-server,version=green

NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/echo-server-service   NodePort    10.96.144.56   <none>        80:30000/TCP   15s   app=echo-server,version=blue
service/kubernetes            ClusterIP   10.96.0.1      <none>        443/TCP        5s    <none>

NAME                            ENDPOINTS                         AGE
endpoints/echo-server-service   10.244.1.3:5678,10.244.2.3:5678   14s
endpoints/kubernetes            172.18.0.4:6443                   5s

❯ curl -s http://127.0.0.1:30000
Hello from Bluecurl -s http://127.0.0.1:30000

❯ kubectl patch svc echo-server-service -p '{"spec": {"selector": {"version": "green"}}}'
service/echo-server-service patched

❯ kubectl get deploy,svc,ep -owide

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS    IMAGES                SELECTOR
deployment.apps/echo-server-blue    2/2     2            2           3m    echo-server   hashicorp/http-echo   app=echo-server,version=blue
deployment.apps/echo-server-green   2/2     2            2           3m    echo-server   hashicorp/http-echo   app=echo-server,version=green

NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/echo-server-service   NodePort    10.96.144.56   <none>        80:30000/TCP   3m      app=echo-server,version=green
service/kubernetes            ClusterIP   10.96.0.1      <none>        443/TCP        2m50s   <none>

NAME                            ENDPOINTS                         AGE
endpoints/echo-server-service   10.244.1.2:5678,10.244.2.4:5678   2m59s
endpoints/kubernetes            172.18.0.4:6443                   2m50s

❯ curl -s http://127.0.0.1:30000
Hello from Green

❯ kubectl patch svc echo-server-service -p '{"spec": {"selector": {"version": "blue"}}}'
service/echo-server-service patched

❯ kubectl get deploy,svc,ep -owide

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS    IMAGES                SELECTOR
deployment.apps/echo-server-blue    2/2     2            2           5m36s   echo-server   hashicorp/http-echo   app=echo-server,version=blue
deployment.apps/echo-server-green   2/2     2            2           5m36s   echo-server   hashicorp/http-echo   app=echo-server,version=green

NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/echo-server-service   NodePort    10.96.144.56   <none>        80:30000/TCP   5m36s   app=echo-server,version=blue
service/kubernetes            ClusterIP   10.96.0.1      <none>        443/TCP        5m26s   <none>

NAME                            ENDPOINTS                         AGE
endpoints/echo-server-service   10.244.1.3:5678,10.244.2.3:5678   5m35s
endpoints/kubernetes            172.18.0.4:6443                   5m26s

❯ curl -s http://127.0.0.1:30000
Hello from Blue

# 삭제
kubectl delete -f .
cd ..

3.6 Jenkins 통한 k8s 기본 배포

Pipeline : item name(k8s-bluegreen)
이전 실습에 디플로이먼트, 서비스 삭제


kubectl delete deploy,svc timeserver

반복 접속 미리 실행


while true; do curl -s --connect-timeout 1 http://127.0.0.1:30000 ; echo ; sleep 1  ; kubectl get deploy -owide ; echo ; kubectl get svc,ep echo-server-service -owide ; echo "------------" ; done
혹은 
while true; do curl -s --connect-timeout 1 http://127.0.0.1:30000 ; date ; echo "------------" ; sleep 1 ; done

pipeline script


pipeline {
    agent any

    environment {
        KUBECONFIG = credentials('k8s-crd')
    }

    stages {
        stage('Checkout') {
            steps {
                 git branch: 'main',
                 url: 'http://192.168.254.124:3000/devops/dev-app.git',  // Git에서 코드 체크아웃
                 credentialsId: 'gogs-crd'  // Credentials ID
            }
        }

        stage('container image build') {
            steps {
                echo "container image build"
            }
        }

        stage('container image upload') {
            steps {
                echo "container image upload"
            }
        }

        stage('k8s deployment blue version') {
            steps {
                sh "kubectl apply -f ./deploy/echo-server-blue.yaml --kubeconfig $KUBECONFIG"
                sh "kubectl apply -f ./deploy/echo-server-service.yaml --kubeconfig $KUBECONFIG"
            }
        }

        stage('approve green version') {
            steps {
                input message: 'approve green version', ok: "Yes"
            }
        }

        stage('k8s deployment green version') {
            steps {
	        	sh "kubectl apply -f ./deploy/echo-server-green.yaml --kubeconfig $KUBECONFIG"
            }
        }

        stage('approve version switching') {
            steps {
                script {
                    returnValue = input message: 'Green switching?', ok: "Yes", parameters: [booleanParam(defaultValue: true, name: 'IS_SWITCHED')]
                    if (returnValue) {
                        sh "kubectl patch svc echo-server-service -p '{\"spec\": {\"selector\": {\"version\": \"green\"}}}' --kubeconfig $KUBECONFIG"
                    }
                }
            }
        }

        stage('Blue Rollback') {
            steps {
                script {
                    returnValue = input message: 'Blue Rollback?', parameters: [choice(choices: ['done', 'rollback'], name: 'IS_ROLLBACk')]
                    if (returnValue == "done") {
                        sh "kubectl delete -f ./deploy/echo-server-blue.yaml --kubeconfig $KUBECONFIG"
                    }
                    if (returnValue == "rollback") {
                        sh "kubectl patch svc echo-server-service -p '{\"spec\": {\"selector\": {\"version\": \"blue\"}}}' --kubeconfig $KUBECONFIG"
                    }
                }
            }
        }
    }
}

지금 배포 후 동작 확인 (version=blue)
Green 배포 승인
- approve green version에서 승인 클릭
- jenkins stage view에서 green deployment step 진행됨
- k8s green deployment가 배포 됨
Green으로 version으로 Switching
- jenkins stage 에서 Green switching 승인
- k8s에서 service selector가 version=green으로 변경되고 'Hello from Green' 출력 됨
Blue로 Rollback
- done 대신 rollback 선택
- k8s에서 service selector가 version=blue으로 변경되고 'Hello from Blue' 출력
- Jekins output log

실습 완료 후 삭제


kubectl delete deploy echo-server-blue echo-server-green
kubectl delete svc echo-server-service

4. Jenkins CI + Argo CD + K8S(Kind)

4.1 Argo 참고자료

Argo - 공홈 & CD Docs & Rollouts Docs & Blog
악분님 ArgoCD 정리 글 'ArgoCD' 태그의 글 목록

4.2 Argo CD 소개

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes
Application definitions, configurations, and environments should be declarative and version controlled.
Application deployment and lifecycle management should be automated, auditable, and easy to understand.
How it works : Argo CD follows the GitOps pattern of using Git repositories as the source of truth for defining the desired application state. Kubernetes manifests can be specified in several ways:
- kustomize applications
- helm charts
- jsonnet files
- Plain directory of YAML/json manifests
- Any custom config management tool configured as a config management plugin
Architecture - Docs
- API Server : Web UI 대시보드, k8s api 처럼 API 서버 역할
  - The API server is a gRPC/REST server which exposes the API consumed by the Web UI, CLI, and CI/CD systems. It has the following responsibilities:
  - application management and status reporting
  - invoking of application operations (e.g. sync, rollback, user-defined actions)
  - repository and cluster credential management (stored as K8s secrets)
  - authentication and auth delegation to external identity providers
  - RBAC enforcement
  - listener/forwarder for Git webhook events
- Repository Server : Git 연결 및 배포할 yaml 생성
  - The repository server is an internal service which maintains a local cache of the Git repository holding the application manifests. It is responsible for generating and returning the Kubernetes manifests when provided the following inputs:
  - repository URL
  - revision (commit, tag, branch)
  - application path
  - template specific settings: parameters, helm values.yaml
- Application Controller : k8s 리소스 모니터링, Git과 비교
  - The application controller is a Kubernetes controller which continuously monitors running applications and compares the current, live state against the desired target state (as specified in the repo). It detects OutOfSync application state and optionally takes corrective action. It is responsible for invoking any user-defined hooks for lifecycle events (PreSync, Sync, PostSync)
- Redis : k8s api와 git 요청을 줄이기 위한 캐싱
- Notification : 이벤트 알림, 트리거
- Dex : 외부 인증 관리
- ApplicationSet Controller : 멀티 클러스터를 위한 App 패키징 관리
기능
- Automated deployment of applications to specified target environments
- Support for multiple config management/templating tools (Kustomize, Helm, Jsonnet, plain-YAML)
- Ability to manage and deploy to multiple clusters
- SSO Integration (OIDC, OAuth2, LDAP, SAML 2.0, GitHub, GitLab, Microsoft, LinkedIn)
- Multi-tenancy and RBAC policies for authorization
- Rollback/Roll-anywhere to any application configuration committed in Git repository
- Health status analysis of application resources
- Automated configuration drift detection and visualization
- Automated or manual syncing of applications to its desired state
- Web UI which provides real-time view of application activity
- CLI for automation and CI integration
- Webhook integration (GitHub, BitBucket, GitLab)
- Access tokens for automation
- PreSync, Sync, PostSync hooks to support complex application rollouts (e.g.blue/green & canary upgrades)
- Audit trails for application events and API calls
- Prometheus metrics
- Parameter overrides for overriding helm parameters in Git
Core Concepts - Docs
- Application A group of Kubernetes resources as defined by a manifest. This is a Custom Resource Definition (CRD).
- Application source type Which Tool is used to build the application.
- Target state The desired state of an application, as represented by files in a Git repository.
- Live state The live state of that application. What pods etc are deployed.
- Sync status Whether or not the live state matches the target state. Is the deployed application the same as Git says it should be?
- Sync The process of making an application move to its target state. E.g. by applying changes to a Kubernetes cluster.
- Sync operation status Whether or not a sync succeeded.
- Refresh Compare the latest code in Git with the live state. Figure out what is different.
- Health The health of the application, is it running correctly? Can it serve requests?
- Tool A tool to create manifests from a directory of files. E.g. Kustomize. See Application Source Type.
- Configuration management tool See Tool.
- Configuration management plugin A custom tool.

4.3 Argo CD 설치 및 기본 설정 - helm_chart

Argo CD 설치


# 네임스페이스 생성 및 파라미터 파일 작성
❯ kubectl create ns argocd
namespace/argocd created

❯ cat <<EOF > argocd-values.yaml
dex:
  enabled: false

server:
  service:
    type: NodePort
    nodePortHttps: 30002
EOF

# 설치
❯ helm repo add argo https://argoproj.github.io/argo-helm
"argo" already exists with the same configuration, skipping

❯ helm install argocd argo/argo-cd --version 7.7.10 -f argocd-values.yaml --namespace argocd

NAME: argocd
LAST DEPLOYED: Sat Dec 21 21:42:38 2024
NAMESPACE: argocd
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
In order to access the server UI you have the following options:

1. kubectl port-forward service/argocd-server -n argocd 8080:443
    and then open the browser on http://localhost:8080 and accept the certificate

2. enable ingress in the values file `server.ingress.enabled` and either
      - Add the annotation for ssl passthrough: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-1-ssl-passthrough
      - Set the `configs.params."server.insecure"` in the values file and terminate SSL at your ingress: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-2-multiple-ingress-objects-and-hosts

After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by running:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

(You should delete the initial secret afterwards as suggested by the Getting Started Guide: https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli)

# 확인
❯ kubectl get pod,svc,ep -n argocd

NAME                                                    READY   STATUS      RESTARTS   AGE
pod/argocd-application-controller-0                     1/1     Running     0          44s
pod/argocd-applicationset-controller-856f6bd788-tb262   1/1     Running     0          44s
pod/argocd-notifications-controller-764b9d6597-ct2bf    1/1     Running     0          44s
pod/argocd-redis-5c67786686-dkb9w                       1/1     Running     0          44s
pod/argocd-redis-secret-init-2dxf2                      0/1     Completed   0          70s
pod/argocd-repo-server-c9f8b6dbf-qlk48                  1/1     Running     0          44s
pod/argocd-server-7bff46b6bd-wvdcm                      1/1     Running     0          44s

NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/argocd-applicationset-controller   ClusterIP   10.96.100.98    <none>        7000/TCP                     44s
service/argocd-redis                       ClusterIP   10.96.128.156   <none>        6379/TCP                     44s
service/argocd-repo-server                 ClusterIP   10.96.109.161   <none>        8081/TCP                     44s
service/argocd-server                      NodePort    10.96.163.254   <none>        80:30080/TCP,443:30002/TCP   44s

NAME                                         ENDPOINTS                         AGE
endpoints/argocd-applicationset-controller   10.244.2.8:7000                   44s
endpoints/argocd-redis                       10.244.1.7:6379                   44s
endpoints/argocd-repo-server                 10.244.1.8:8081                   44s
endpoints/argocd-server                      10.244.2.7:8080,10.244.2.7:8080   44s

❯ kubectl get crd | grep argo
applications.argoproj.io      2024-12-21T12:43:06Z
applicationsets.argoproj.io   2024-12-21T12:43:06Z
appprojects.argoproj.io       2024-12-21T12:43:06Z

# 최초 접속 암호 확인
❯ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ;echo
YKTa2PVifgqLpzWW

# Argo CD 웹 접속 주소 확인 : 초기 암호 입력 (admin 계정)
open "https://127.0.0.1:30002"

Argo CD 웹 접속 확인
- User info → UPDATE PASSWORD 로 admin 계정 암호 변경 (qwe12345)
기본 정보 확인 (Settings) : Clusters, Projects, Accounts
ops-deploy Repo 등록 : Settings → Repositories → CONNECT REPO 클릭
- connection method : VIA HTTPS
- Type : git
- Project : default
- Repo URL : http://192.168.0.49:3000/devops/ops-deploy
- Username : devops
- Password : <Gogs 토큰>
  ⇒ 입력 후 CONNECT 클릭
  - connect click 후

4.4 (기초) helm chart 를 통한 배포 실습

helm chart 생성

❯ mkdir nginx-chart
❯ cd nginx-chart
❯ mkdir templates

❯ cat > templates/configmap.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}
data:
  index.html: |
{{ .Values.indexHtml | indent 4 }}
EOF

❯ cat > templates/deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ .Release.Name }}
    spec:
      containers:
      - name: nginx
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        ports:
        - containerPort: 80
        volumeMounts:
        - name: index-html
          mountPath: /usr/share/nginx/html/index.html
          subPath: index.html
      volumes:
      - name: index-html
        configMap:
          name: {{ .Release.Name }}
EOF

❯ cat > templates/service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
spec:
  selector:
    app: {{ .Release.Name }}
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30000
  type: NodePort
EOF

❯ cat > values.yaml <<EOF
indexHtml: |
  <!DOCTYPE html>
  <html>
  <head>
    <title>Welcome to Nginx!</title>
  </head>
  <body>
    <h1>Hello, Kubernetes!</h1>
    <p>Nginx version 1.26.1</p>
  </body>
  </html>

image:
  repository: nginx
  tag: 1.26.1

replicaCount: 1
EOF

❯ cat > Chart.yaml <<EOF
apiVersion: v2
name: nginx-chart
description: A Helm chart for deploying Nginx with custom index.html
type: application
version: 1.0.0
appVersion: "1.26.1"
EOF

❯ tree .
.
├── Chart.yaml
├── templates
│   ├── configmap.yaml
│   ├── deployment.yaml
│   └── service.yaml
└── values.yaml

2 directories, 5 files

# 이전 timeserver/service(nodeport) 삭제
❯ kubectl delete deploy,svc --all
service "kubernetes" deleted

# 직접 배포 해보기
❯ helm install dev-nginx . -f values.yaml
NAME: dev-nginx
LAST DEPLOYED: Sat Dec 21 22:06:30 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

❯ helm list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
dev-nginx       default         1               2024-12-21 22:06:30.570676 +0900 KST    deployed        nginx-chart-1.0.0       1.26.1   

❯ kubectl get deploy,svc,ep,cm dev-nginx -owide

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES         SELECTOR
deployment.apps/dev-nginx   1/1     1            1           56s   nginx        nginx:1.26.1   app=dev-nginx

NAME                TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/dev-nginx   NodePort   10.96.77.136   <none>        80:30000/TCP   56s   app=dev-nginx

NAME                  ENDPOINTS       AGE
endpoints/dev-nginx   10.244.2.9:80   56s

NAME                  DATA   AGE
configmap/dev-nginx   1      56s

#
❯ curl http://127.0.0.1:30000

<!DOCTYPE html>
<html>
<head>
  <title>Welcome to Nginx!</title>
</head>
<body>
  <h1>Hello, Kubernetes!</h1>
  <p>Nginx version 1.26.1</p>
</body>
</html>

❯ curl -s http://127.0.0.1:30000 | grep version

  <p>Nginx version 1.26.1</p>
  
❯ open http://127.0.0.1:30000


# value 값 변경 후 적용 해보기 : version/tag, replicaCount
cat > values.yaml <<EOF
indexHtml: |
  <!DOCTYPE html>
  <html>
  <head>
    <title>Welcome to Nginx!</title>
  </head>
  <body>
    <h1>Hello, Kubernetes!</h1>
    <p>Nginx version 1.26.2</p>
  </body>
  </html>

image:
  repository: nginx
  tag: 1.26.2

replicaCount: 2
EOF

# helm chart 업그레이드 적용
helm upgrade dev-nginx . -f values.yaml

# 확인
helm list
kubectl get deploy,svc,ep,cm dev-nginx -owide
curl http://127.0.0.1:30000
curl -s http://127.0.0.1:30000 | grep version
open http://127.0.0.1:30000

# 확인 후 삭제
❯ helm uninstall dev-nginx
release "dev-nginx" uninstalled

4.5 Repo(ops-deploy) 에 nginx helm chart 를 Argo CD를 통한 배포 1

git 작업


❯ cd ~/cicd-labs
❯ git clone http://192.168.0.49:3000/devops/ops-deploy.git
Cloning into 'ops-deploy'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 241 bytes | 80.00 KiB/s, done.
cd ops-deploy

#
❯ git config user.name "devops"
❯ git config user.email "a@a.com"
❯ git config init.defaultBranch main
❯ git config credential.helper store

#
❯ VERSION=1.26.1
❯ mkdir nginx-chart
❯ mkdir nginx-chart/templates

❯ cat > nginx-chart/VERSION <<EOF
$VERSION
EOF

❯ cat > nginx-chart/templates/configmap.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}
data:
  index.html: |
{{ .Values.indexHtml | indent 4 }}
EOF

❯ cat > nginx-chart/templates/deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ .Release.Name }}
    spec:
      containers:
      - name: nginx
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        ports:
        - containerPort: 80
        volumeMounts:
        - name: index-html
          mountPath: /usr/share/nginx/html/index.html
          subPath: index.html
      volumes:
      - name: index-html
        configMap:
          name: {{ .Release.Name }}
EOF

❯ cat > nginx-chart/templates/service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
spec:
  selector:
    app: {{ .Release.Name }}
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30000
  type: NodePort
EOF

❯ cat > nginx-chart/values-dev.yaml <<EOF
indexHtml: |
  <!DOCTYPE html>
  <html>
  <head>
    <title>Welcome to Nginx!</title>
  </head>
  <body>
    <h1>Hello, Kubernetes!</h1>
    <p>DEV : Nginx version $VERSION</p>
  </body>
  </html>

image:
  repository: nginx
  tag: $VERSION

replicaCount: 1
EOF

❯ cat > nginx-chart/values-prd.yaml <<EOF
indexHtml: |
  <!DOCTYPE html>
  <html>
  <head>
    <title>Welcome to Nginx!</title>
  </head>
  <body>
    <h1>Hello, Kubernetes!</h1>
    <p>PRD : Nginx version $VERSION</p>
  </body>
  </html>

image:
  repository: nginx
  tag: $VERSION

replicaCount: 2
EOF

❯ cat > nginx-chart/Chart.yaml <<EOF
apiVersion: v2
name: nginx-chart
description: A Helm chart for deploying Nginx with custom index.html
type: application
version: 1.0.0
appVersion: "$VERSION"
EOF

❯ tree nginx-chart
nginx-chart
├── Chart.yaml
├── VERSION
├── templates
│   ├── configmap.yaml
│   ├── deployment.yaml
│   └── service.yaml
├── values-dev.yaml
└── values-prd.yaml

#
❯ git status && git add . && git commit -m "Add nginx helm chart" && git push -u origin main

On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        nginx-chart/

nothing added to commit but untracked files present (use "git add" to track)
[main cf91b8e] Add nginx helm chart
 7 files changed, 88 insertions(+)
 create mode 100644 nginx-chart/Chart.yaml
 create mode 100644 nginx-chart/VERSION
 create mode 100644 nginx-chart/templates/configmap.yaml
 create mode 100644 nginx-chart/templates/deployment.yaml
 create mode 100644 nginx-chart/templates/service.yaml
 create mode 100644 nginx-chart/values-dev.yaml
 create mode 100644 nginx-chart/values-prd.yaml
Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Delta compression using up to 8 threads
Compressing objects: 100% (10/10), done.
Writing objects: 100% (11/11), 1.44 KiB | 1.44 MiB/s, done.
Total 11 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
To http://192.168.0.49:3000/devops/ops-deploy.git
   c26ca29..cf91b8e  main -> main
branch 'main' set up to track 'origin/main'.

Argo CD에 App 등록 : Application → NEW APP
- GENERAL
  - App Name : dev-nginx
  - Project Name : default
  - SYNC POLICY : Manual
  - SYNC OPTIONS : AUTO-CREATE NAMESPACE(Check)
- Source
  - Repo URL : 설정되어 있는 것 선택
  - Revision : HEAD
  - PATH : nginx-chart
- DESTINATION
  - Cluster URL : <기본값>
  - NAMESPACE : dev-nginx
- HELM
  - Values files : values-dev.yaml
    ⇒ 작성 후 상단 CREATE 클릭
dev-nginx App 클릭 후 상세 정보 확인 → DIFF 클릭 확인


❯ kubectl get applications -n argocd
NAME        SYNC STATUS   HEALTH STATUS
dev-nginx   OutOfSync     Missing

❯ kubectl describe applications -n argocd dev-nginx

Name:         dev-nginx
Namespace:    argocd
Labels:       <none>
Annotations:  <none>
API Version:  argoproj.io/v1alpha1
Kind:         Application
Metadata:
  Creation Timestamp:  2024-12-21T13:25:06Z
  Generation:          3
  Resource Version:    20959
  UID:                 ba0f8d13-a0b5-4145-a720-bc060a8f974c
Spec:
  Destination:
    Namespace:  dev-nginx
    Server:     https://kubernetes.default.svc
  Project:      default
  Source:
    Helm:
      Value Files:
        values-dev.yaml
    Path:             nginx-chart
    Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
    Target Revision:  HEAD
  Sync Policy:
    Sync Options:
      CreateNamespace=true
Status:
  Controller Namespace:  argocd
  Health:
    Status:       Missing
  Reconciled At:  2024-12-21T13:28:09Z
  Resources:
    Health:
      Status:   Missing
    Kind:       ConfigMap
    Name:       dev-nginx
    Namespace:  dev-nginx
    Status:     OutOfSync
    Version:    v1
    Health:
      Status:   Missing
    Kind:       Service
    Name:       dev-nginx
    Namespace:  dev-nginx
    Status:     OutOfSync
    Version:    v1
    Group:      apps
    Health:
      Status:   Missing
    Kind:       Deployment
    Name:       dev-nginx
    Namespace:  dev-nginx
    Status:     OutOfSync
    Version:    v1
  Source Type:  Helm
  Summary:
  Sync:
    Compared To:
      Destination:
        Namespace:  dev-nginx
        Server:     https://kubernetes.default.svc
      Source:
        Helm:
          Value Files:
            values-dev.yaml
        Path:             nginx-chart
        Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
        Target Revision:  HEAD
    Revision:             cf91b8ed1ab26896069edeb271693fed89705536
    Status:               OutOfSync
Events:
  Type    Reason           Age    From                           Message
  ----    ------           ----   ----                           -------
  Normal  ResourceCreated  3m46s  argocd-server                  admin created application
  Normal  ResourceUpdated  3m44s  argocd-application-controller  Updated sync status:  -> OutOfSync
  Normal  ResourceUpdated  3m44s  argocd-application-controller  Updated health status:  -> Missing
  
# 반복 접속 시도
❯ while true; do curl -s --connect-timeout 1 http://127.0.0.1:30000 ; date ; echo "------------" ; sleep 1 ; done

Sat Dec 21 22:29:15 KST 2024
------------
Sat Dec 21 22:29:16 KST 2024
------------
Sat Dec 21 22:29:17 KST 2024
** Sync 후 **
<!DOCTYPE html>
<html>
<head>
  <title>Welcome to Nginx!</title>
</head>
<body>
  <h1>Hello, Kubernetes!</h1>
  <p>DEV : Nginx version 1.26.1</p>
</body>
</html>
Sat Dec 21 22:30:57 KST 2024

SYNC 클릭 으로 반영 확인


# 아래 처럼 yaml 로 APP 생성 가능
❯ kubectl get applications -n argocd
NAME        SYNC STATUS   HEALTH STATUS
dev-nginx   Synced        Healthy

❯ kubectl get applications -n argocd -o yaml | kubectl neat

apiVersion: v1
items:
- apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
    name: dev-nginx
    namespace: argocd
  spec:
    destination:
      namespace: dev-nginx
      server: https://kubernetes.default.svc
    project: default
    source:
      helm:
        valueFiles:
        - values-dev.yaml
      path: nginx-chart
      repoURL: http://192.168.0.49:3000/devops/ops-deploy
      targetRevision: HEAD
    syncPolicy:
      syncOptions:
      - CreateNamespace=true
kind: List
metadata: {}

# 배포 확인
❯ kubectl get all -n dev-nginx -o wide

NAME                             READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
pod/dev-nginx-77d44dfbf6-vddkp   1/1     Running   0          2m28s   10.244.2.10   myk8s-worker2   <none>           <none>

NAME                TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/dev-nginx   NodePort   10.96.195.61   <none>        80:30000/TCP   2m28s   app=dev-nginx

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES         SELECTOR
deployment.apps/dev-nginx   1/1     1            1           2m28s   nginx        nginx:1.26.1   app=dev-nginx

NAME                                   DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES         SELECTOR
replicaset.apps/dev-nginx-77d44dfbf6   1         1         1       2m28s   nginx        nginx:1.26.1   app=dev-nginx,pod-template-hash=77d44dfbf6

코드 수정 후 반영 확인


#
❯ VERSION=1.26.2

❯ cat > nginx-chart/VERSION <<EOF
$VERSION
EOF

❯ cat > nginx-chart/values-dev.yaml <<EOF
indexHtml: |
  <!DOCTYPE html>
  <html>
  <head>
    <title>Welcome to Nginx!</title>
  </head>
  <body>
    <h1>Hello, Kubernetes!</h1>
    <p>DEV : Nginx version $VERSION</p>
  </body>
  </html>

image:
  repository: nginx
  tag: $VERSION

replicaCount: 2
EOF

❯ cat > nginx-chart/values-dev.yaml <<EOF
indexHtml: |
  <!DOCTYPE html>
  <html>
  <head>
    <title>Welcome to Nginx!</title>
  </head>
  <body>
    <h1>Hello, Kubernetes!</h1>
    <p>DEV : Nginx version $VERSION</p>
  </body>
  </html>

image:
  repository: nginx
  tag: $VERSION

replicaCount: 2
EOF

#
❯ git status && git add . && git commit -m "Update nginx version $(cat nginx-chart/VERSION)" && git push -u origin main

On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   nginx-chart/VERSION
        modified:   nginx-chart/values-dev.yaml

no changes added to commit (use "git add" and/or "git commit -a")
[main 9f4fea9] Update nginx version 1.26.2
 2 files changed, 4 insertions(+), 4 deletions(-)
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 433 bytes | 433.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
To http://192.168.0.49:3000/devops/ops-deploy.git
   cf91b8e..9f4fea9  main -> main
branch 'main' set up to track 'origin/main'.

Argo CD 웹 확인 → REFRESH 클릭 - Interval
- REFRESH 클릭 후 OutOfSync 감지됨
- DIFF로 확인
  
  🤔 How often does Argo CD check for changes to my Git or Helm repository ?
  The default polling interval is 3 minutes (180 seconds). You can change the setting by updating the timeout.reconciliation value in the argocd-cm config map
SYNC 클릭 → SYNCHRONIZE 클릭


# 배포 확인
❯ kubectl get all -n dev-nginx -o wide

NAME                             READY   STATUS              RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
pod/dev-nginx-5db658bd4f-pj4lq   0/1     ContainerCreating   0          15s   <none>        myk8s-worker2   <none>           <none>
pod/dev-nginx-77d44dfbf6-l6sh6   0/1     ContainerCreating   0          15s   <none>        myk8s-worker    <none>           <none>
pod/dev-nginx-77d44dfbf6-vddkp   1/1     Running             0          11m   10.244.2.10   myk8s-worker2   <none>           <none>

NAME                TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/dev-nginx   NodePort   10.96.195.61   <none>        80:30000/TCP   11m   app=dev-nginx

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES         SELECTOR
deployment.apps/dev-nginx   1/2     1            1           11m   nginx        nginx:1.26.2   app=dev-nginx

NAME                                   DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES         SELECTOR
replicaset.apps/dev-nginx-5db658bd4f   1         1         0       15s   nginx        nginx:1.26.2   app=dev-nginx,pod-template-hash=5db658bd4f
replicaset.apps/dev-nginx-77d44dfbf6   2         2         1       11m   nginx        nginx:1.26.1   app=dev-nginx,pod-template-hash=77d44dfbf6

❯ kubectl get all -n dev-nginx -o wide

NAME                             READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
pod/dev-nginx-5db658bd4f-pj4lq   1/1     Running   0          45s   10.244.2.11   myk8s-worker2   <none>           <none>
pod/dev-nginx-5db658bd4f-wfq2q   1/1     Running   0          26s   10.244.1.12   myk8s-worker    <none>           <none>

NAME                TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/dev-nginx   NodePort   10.96.195.61   <none>        80:30000/TCP   11m   app=dev-nginx

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES         SELECTOR
deployment.apps/dev-nginx   2/2     2            2           11m   nginx        nginx:1.26.2   app=dev-nginx

NAME                                   DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES         SELECTOR
replicaset.apps/dev-nginx-5db658bd4f   2         2         2       45s   nginx        nginx:1.26.2   app=dev-nginx,pod-template-hash=5db658bd4f
replicaset.apps/dev-nginx-77d44dfbf6   0         0         0       11m   nginx        nginx:1.26.1   app=dev-nginx,pod-template-hash=77d44dfbf6

Argo CD 웹에서 App 삭제


watch -d kubectl get all -n dev-nginx -o wide

4.6 Repo(ops-deploy)에 nginx helm chart를 Argo CD를 통한 배포2

ArgoCD Declarative Setup - ArgoCD 애플리케이션 자체를 yaml로 생성
ArgoCD Declarative Setup - Project, applications, ArgoCD Settings - Docs
dev-nginx App 생성 및 Auto SYNC


#
❯ cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dev-nginx
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    helm:
      valueFiles:
      - values-dev.yaml
    path: nginx-chart
    repoURL: http://192.168.0.49:3000/devops/ops-deploy
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
    - CreateNamespace=true
  destination:
    namespace: dev-nginx
    server: https://kubernetes.default.svc
EOF

Warning: metadata.finalizers: "resources-finalizer.argocd.argoproj.io": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
application.argoproj.io/dev-nginx created

#
❯ kubectl get applications -n argocd dev-nginx

NAME        SYNC STATUS   HEALTH STATUS
dev-nginx   Synced        Healthy
❯ kubectl get applications -n argocd dev-nginx -o yaml | kubectl neat

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dev-nginx
  namespace: argocd
spec:
  destination:
    namespace: dev-nginx
    server: https://kubernetes.default.svc
  project: default
  source:
    helm:
      valueFiles:
      - values-dev.yaml
    path: nginx-chart
    repoURL: http://192.168.0.49:3000/devops/ops-deploy
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
    - CreateNamespace=true
❯ kubectl describe applications -n argocd dev-nginx

Name:         dev-nginx
Namespace:    argocd
Labels:       <none>
Annotations:  <none>
API Version:  argoproj.io/v1alpha1
Kind:         Application
Metadata:
  Creation Timestamp:  2024-12-21T13:47:06Z
  Finalizers:
    resources-finalizer.argocd.argoproj.io
  Generation:        11
  Resource Version:  22914
  UID:               5172a7cd-3698-4f88-8aec-40d4090395c5
Spec:
  Destination:
    Namespace:  dev-nginx
    Server:     https://kubernetes.default.svc
  Project:      default
  Source:
    Helm:
      Value Files:
        values-dev.yaml
    Path:             nginx-chart
    Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
    Target Revision:  HEAD
  Sync Policy:
    Automated:
      Prune:  true
    Sync Options:
      CreateNamespace=true
Status:
  Controller Namespace:  argocd
  Health:
    Status:  Healthy
  History:
    Deploy Started At:  2024-12-21T13:47:06Z
    Deployed At:        2024-12-21T13:47:07Z
    Id:                 0
    Initiated By:
      Automated:  true
    Revision:     9f4fea9e77326479477135b14471fbaea31aebbe
    Source:
      Helm:
        Value Files:
          values-dev.yaml
      Path:             nginx-chart
      Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
      Target Revision:  HEAD
  Operation State:
    Finished At:  2024-12-21T13:47:07Z
    Message:      successfully synced (all tasks run)
    Operation:
      Initiated By:
        Automated:  true
      Retry:
        Limit:  5
      Sync:
        Prune:     true
        Revision:  9f4fea9e77326479477135b14471fbaea31aebbe
        Sync Options:
          CreateNamespace=true
    Phase:       Succeeded
    Started At:  2024-12-21T13:47:06Z
    Sync Result:
      Resources:
        Group:       
        Hook Phase:  Running
        Kind:        ConfigMap
        Message:     configmap/dev-nginx created
        Name:        dev-nginx
        Namespace:   dev-nginx
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       
        Hook Phase:  Running
        Kind:        Service
        Message:     service/dev-nginx created
        Name:        dev-nginx
        Namespace:   dev-nginx
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/dev-nginx created
        Name:        dev-nginx
        Namespace:   dev-nginx
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
      Revision:      9f4fea9e77326479477135b14471fbaea31aebbe
      Source:
        Helm:
          Value Files:
            values-dev.yaml
        Path:             nginx-chart
        Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
        Target Revision:  HEAD
  Reconciled At:          2024-12-21T13:47:07Z
  Resources:
    Kind:       ConfigMap
    Name:       dev-nginx
    Namespace:  dev-nginx
    Status:     Synced
    Version:    v1
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       dev-nginx
    Namespace:  dev-nginx
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       dev-nginx
    Namespace:  dev-nginx
    Status:     Synced
    Version:    v1
  Source Type:  Helm
  Summary:
    Images:
      nginx:1.26.2
  Sync:
    Compared To:
      Destination:
        Namespace:  dev-nginx
        Server:     https://kubernetes.default.svc
      Source:
        Helm:
          Value Files:
            values-dev.yaml
        Path:             nginx-chart
        Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
        Target Revision:  HEAD
    Revision:             9f4fea9e77326479477135b14471fbaea31aebbe
    Status:               Synced
Events:
  Type    Reason              Age   From                           Message
  ----    ------              ----  ----                           -------
  Normal  OperationStarted    44s   argocd-application-controller  Initiated automated sync to '9f4fea9e77326479477135b14471fbaea31aebbe'
  Normal  ResourceUpdated     44s   argocd-application-controller  Updated sync status:  -> OutOfSync
  Normal  ResourceUpdated     44s   argocd-application-controller  Updated health status:  -> Missing
  Normal  ResourceUpdated     43s   argocd-application-controller  Updated sync status: OutOfSync -> Synced
  Normal  ResourceUpdated     43s   argocd-application-controller  Updated health status: Missing -> Progressing
  Normal  OperationCompleted  43s   argocd-application-controller  Sync operation to 9f4fea9e77326479477135b14471fbaea31aebbe succeeded
  Normal  ResourceUpdated     42s   argocd-application-controller  Updated health status: Progressing -> Healthy
❯ kubectl get pod,svc,ep,cm -n dev-nginx

NAME                             READY   STATUS    RESTARTS   AGE
pod/dev-nginx-5db658bd4f-gxfgh   1/1     Running   0          51s
pod/dev-nginx-5db658bd4f-kmgb4   1/1     Running   0          51s

NAME                TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/dev-nginx   NodePort   10.96.172.37   <none>        80:30000/TCP   51s

NAME                  ENDPOINTS                       AGE
endpoints/dev-nginx   10.244.1.13:80,10.244.2.12:80   51s

NAME                         DATA   AGE
configmap/dev-nginx          1      51s
configmap/kube-root-ca.crt   1      17m

#
❯ curl http://127.0.0.1:30000

<!DOCTYPE html>
<html>
<head>
  <title>Welcome to Nginx!</title>
</head>
<body>
  <h1>Hello, Kubernetes!</h1>
  <p>DEV : Nginx version 1.26.2</p>
</body>
</html>
❯ open http://127.0.0.1:30000

# Argo CD App 삭제
❯ kubectl delete applications -n argocd dev-nginx
application.argoproj.io "dev-nginx" deleted

prd-nginx App 생성 및 Auto SYNC


#
❯ cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prd-nginx
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: prd-nginx
    server: https://kubernetes.default.svc
  project: default
  source:
    helm:
      valueFiles:
      - values-prd.yaml
    path: nginx-chart
    repoURL: http://192.168.0.49:3000/devops/ops-deploy
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
    - CreateNamespace=true
EOF

Warning: metadata.finalizers: "resources-finalizer.argocd.argoproj.io": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
application.argoproj.io/prd-nginx created
❯ kubectl get applications -n argocd prd-nginx

NAME        SYNC STATUS   HEALTH STATUS
prd-nginx   Synced        Healthy
❯ kubectl describe applications -n argocd prd-nginx

Name:         prd-nginx
Namespace:    argocd
Labels:       <none>
Annotations:  <none>
API Version:  argoproj.io/v1alpha1
Kind:         Application
Metadata:
  Creation Timestamp:  2024-12-21T13:50:59Z
  Finalizers:
    resources-finalizer.argocd.argoproj.io
  Generation:        13
  Resource Version:  23369
  UID:               6feaa0a1-2daa-4b3f-9682-b655120af70a
Spec:
  Destination:
    Namespace:  prd-nginx
    Server:     https://kubernetes.default.svc
  Project:      default
  Source:
    Helm:
      Value Files:
        values-prd.yaml
    Path:             nginx-chart
    Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
    Target Revision:  HEAD
  Sync Policy:
    Automated:
      Prune:  true
    Sync Options:
      CreateNamespace=true
Status:
  Controller Namespace:  argocd
  Health:
    Status:  Healthy
  History:
    Deploy Started At:  2024-12-21T13:50:59Z
    Deployed At:        2024-12-21T13:51:02Z
    Id:                 0
    Initiated By:
      Automated:  true
    Revision:     9f4fea9e77326479477135b14471fbaea31aebbe
    Source:
      Helm:
        Value Files:
          values-prd.yaml
      Path:             nginx-chart
      Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
      Target Revision:  HEAD
  Operation State:
    Finished At:  2024-12-21T13:51:02Z
    Message:      successfully synced (all tasks run)
    Operation:
      Initiated By:
        Automated:  true
      Retry:
        Limit:  5
      Sync:
        Prune:     true
        Revision:  9f4fea9e77326479477135b14471fbaea31aebbe
        Sync Options:
          CreateNamespace=true
    Phase:       Succeeded
    Started At:  2024-12-21T13:50:59Z
    Sync Result:
      Resources:
        Group:       
        Hook Phase:  Running
        Kind:        Namespace
        Message:     namespace/prd-nginx created
        Name:        prd-nginx
        Namespace:   
        Status:      Synced
        Sync Phase:  PreSync
        Version:     v1
        Group:       
        Hook Phase:  Running
        Kind:        ConfigMap
        Message:     configmap/prd-nginx created
        Name:        prd-nginx
        Namespace:   prd-nginx
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       
        Hook Phase:  Running
        Kind:        Service
        Message:     service/prd-nginx created
        Name:        prd-nginx
        Namespace:   prd-nginx
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/prd-nginx created
        Name:        prd-nginx
        Namespace:   prd-nginx
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
      Revision:      9f4fea9e77326479477135b14471fbaea31aebbe
      Source:
        Helm:
          Value Files:
            values-prd.yaml
        Path:             nginx-chart
        Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
        Target Revision:  HEAD
  Reconciled At:          2024-12-21T13:51:02Z
  Resources:
    Kind:       ConfigMap
    Name:       prd-nginx
    Namespace:  prd-nginx
    Status:     Synced
    Version:    v1
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       prd-nginx
    Namespace:  prd-nginx
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       prd-nginx
    Namespace:  prd-nginx
    Status:     Synced
    Version:    v1
  Source Type:  Helm
  Summary:
    Images:
      nginx:1.26.1
  Sync:
    Compared To:
      Destination:
        Namespace:  prd-nginx
        Server:     https://kubernetes.default.svc
      Source:
        Helm:
          Value Files:
            values-prd.yaml
        Path:             nginx-chart
        Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
        Target Revision:  HEAD
    Revision:             9f4fea9e77326479477135b14471fbaea31aebbe
    Status:               Synced
Events:
  Type    Reason              Age   From                           Message
  ----    ------              ----  ----                           -------
  Normal  OperationStarted    26s   argocd-application-controller  Initiated automated sync to '9f4fea9e77326479477135b14471fbaea31aebbe'
  Normal  ResourceUpdated     26s   argocd-application-controller  Updated sync status:  -> OutOfSync
  Normal  ResourceUpdated     26s   argocd-application-controller  Updated health status:  -> Missing
  Normal  ResourceUpdated     23s   argocd-application-controller  Updated sync status: OutOfSync -> Synced
  Normal  ResourceUpdated     23s   argocd-application-controller  Updated health status: Missing -> Progressing
  Normal  OperationCompleted  23s   argocd-application-controller  Sync operation to 9f4fea9e77326479477135b14471fbaea31aebbe succeeded
  Normal  ResourceUpdated     22s   argocd-application-controller  Updated health status: Progressing -> Healthy
❯ kubectl get pod,svc,ep,cm -n prd-nginx

NAME                             READY   STATUS    RESTARTS   AGE
pod/prd-nginx-69cb647476-64r78   1/1     Running   0          30s
pod/prd-nginx-69cb647476-x5h77   1/1     Running   0          30s

NAME                TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/prd-nginx   NodePort   10.96.222.163   <none>        80:30000/TCP   30s

NAME                  ENDPOINTS                       AGE
endpoints/prd-nginx   10.244.1.14:80,10.244.2.13:80   30s

NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      33s
configmap/prd-nginx          1      30s

#
❯ curl http://127.0.0.1:30000

<!DOCTYPE html>
<html>
<head>
  <title>Welcome to Nginx!</title>
</head>
<body>
  <h1>Hello, Kubernetes!</h1>
  <p>PRD : Nginx version 1.26.1</p>
</body>
</html>
❯ open http://127.0.0.1:30000

# Argo CD App 삭제
❯ kubectl delete applications -n argocd prd-nginx

Gogs 에서 Webhook 를 통해 Argo CD 에 코드 변경 즉시 반영하여 k8s 배포 할 수 있게 설정 - Docs
Mastering Argo CD image updater with Helm: a complete configuration guide - Docs , Blog
Full CI/CD 구성도

4.7 ops-deploy Repo 코드 작업


#
❯ cd ops-deploy
❯ mkdir dev-app

# 도커 계정 정보
DHUSER=<도커 허브 계정>
❯ DHUSER=kimseongjung

# 버전 정보 
❯ VERSION=0.0.1

#
❯ cat > dev-app/VERSION <<EOF
$VERSION
EOF

❯ cat > dev-app/timeserver.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: timeserver
spec:
  replicas: 2
  selector:
    matchLabels:
      pod: timeserver-pod
  template:
    metadata:
      labels:
        pod: timeserver-pod
    spec:
      containers:
      - name: timeserver-container
        image: docker.io/$DHUSER/dev-app:$VERSION
      imagePullSecrets:
      - name: dockerhub-secret
EOF

❯ cat > dev-app/service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: timeserver
spec:
  selector:
    pod: timeserver-pod
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    nodePort: 30000
  type: NodePort
EOF

#
❯ git status && git add . && git commit -m "Add dev-app deployment yaml" && git push -u origin main
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        dev-app/

nothing added to commit but untracked files present (use "git add" to track)
[main 8ef6043] Add dev-app deployment yaml
 3 files changed, 33 insertions(+)
 create mode 100644 dev-app/VERSION
 create mode 100644 dev-app/service.yaml
 create mode 100644 dev-app/timeserver.yaml
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 777 bytes | 777.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
To http://192.168.0.49:3000/devops/ops-deploy.git
   9f4fea9..8ef6043  main -> main
branch 'main' set up to track 'origin/main'.

4.8 Argo CD app 생성


#
❯ cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: timeserver
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    path: dev-app
    repoURL: http://192.168.0.49:3000/devops/ops-deploy
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
    - CreateNamespace=true
  destination:
    namespace: default
    server: https://kubernetes.default.svc
EOF

Warning: metadata.finalizers: "resources-finalizer.argocd.argoproj.io": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
application.argoproj.io/timeserver created

#
❯ kubectl get applications -n argocd timeserver
NAME         SYNC STATUS   HEALTH STATUS
timeserver   Synced        Healthy

❯ kubectl get applications -n argocd timeserver -o yaml | kubectl neat
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: timeserver
  namespace: argocd
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: dev-app
    repoURL: http://192.168.0.49:3000/devops/ops-deploy
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
    - CreateNamespace=true

❯ kubectl describe applications -n argocd timeserver
Name:         timeserver
Namespace:    argocd
Labels:       <none>
Annotations:  <none>
API Version:  argoproj.io/v1alpha1
Kind:         Application
Metadata:
  Creation Timestamp:  2024-12-21T14:02:13Z
  Finalizers:
    resources-finalizer.argocd.argoproj.io
  Generation:        35
  Resource Version:  25787
  UID:               4d8c5d87-bc02-484c-9b01-2a5304cfc439
Spec:
  Destination:
    Namespace:  default
    Server:     https://kubernetes.default.svc
  Project:      default
  Source:
    Path:             dev-app
    Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
    Target Revision:  HEAD
  Sync Policy:
    Automated:
      Prune:  true
    Sync Options:
      CreateNamespace=true
Status:
  Controller Namespace:  argocd
  Health:
    Status:  Healthy
  History:
    Deploy Started At:  2024-12-21T14:16:17Z
    Deployed At:        2024-12-21T14:16:17Z
    Id:                 0
    Initiated By:
      Username:  admin
    Revision:    8ef60434972c0b95588a8b8b2c28e635d51e363a
    Source:
      Path:             dev-app
      Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
      Target Revision:  HEAD
  Operation State:
    Finished At:  2024-12-21T14:16:17Z
    Message:      successfully synced (all tasks run)
    Operation:
      Initiated By:
        Username:  admin
      Retry:
      Sync:
        Revision:  8ef60434972c0b95588a8b8b2c28e635d51e363a
        Sync Options:
          CreateNamespace=true
        Sync Strategy:
          Hook:
    Phase:       Succeeded
    Started At:  2024-12-21T14:16:17Z
    Sync Result:
      Resources:
        Group:       
        Hook Phase:  Running
        Kind:        Service
        Message:     service/timeserver created
        Name:        timeserver
        Namespace:   default
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/timeserver unchanged
        Name:        timeserver
        Namespace:   default
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
      Revision:      8ef60434972c0b95588a8b8b2c28e635d51e363a
      Source:
        Path:             dev-app
        Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
        Target Revision:  HEAD
  Reconciled At:          2024-12-21T14:16:17Z
  Resources:
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       timeserver
    Namespace:  default
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       timeserver
    Namespace:  default
    Status:     Synced
    Version:    v1
  Source Type:  Directory
  Summary:
    Images:
      docker.io/kimseongjung/dev-app:0.0.1
  Sync:
    Compared To:
      Destination:
        Namespace:  default
        Server:     https://kubernetes.default.svc
      Source:
        Path:             dev-app
        Repo URL:         http://192.168.0.49:3000/devops/ops-deploy
        Target Revision:  HEAD
    Revision:             8ef60434972c0b95588a8b8b2c28e635d51e363a
    Status:               Synced
Events:
  Type     Reason              Age    From                           Message
  ----     ------              ----   ----                           -------
  Normal   OperationStarted    14m    argocd-application-controller  Initiated automated sync to '8ef60434972c0b95588a8b8b2c28e635d51e363a'
  Normal   ResourceUpdated     14m    argocd-application-controller  Updated sync status:  -> OutOfSync
  Normal   OperationStarted    45s    argocd-server                  admin initiated sync to HEAD (8ef60434972c0b95588a8b8b2c28e635d51e363a)
  Normal   ResourceUpdated     45s    argocd-application-controller  Updated sync status: OutOfSync -> Synced
  Normal   ResourceUpdated     45s    argocd-application-controller  Updated health status: Missing -> Healthy
  Normal   OperationCompleted  45s    argocd-application-controller  Sync operation to 8ef60434972c0b95588a8b8b2c28e635d51e363a succeeded

❯ kubectl get deploy,rs,pod
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/timeserver   2/2     2            2           14m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/timeserver-5dcc8df79c   2         2         2       14m

NAME                              READY   STATUS    RESTARTS   AGE
pod/timeserver-5dcc8df79c-g6dsl   1/1     Running   0          14m
pod/timeserver-5dcc8df79c-tbdzb   1/1     Running   0          14m

❯ kubectl get svc,ep timeserver
NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/timeserver   NodePort   10.96.39.227   <none>        80:30000/TCP   57s

NAME                   ENDPOINTS                       AGE
endpoints/timeserver   10.244.1.15:80,10.244.2.14:80   57s

#
❯ curl http://127.0.0.1:30000

The time is 2:18:23 PM, VERSION 0.0.1
Server hostname: timeserver-5dcc8df79c-g6dsl

❯ open http://127.0.0.1:30000

4.9 dev-app Repo 코드 작업

dev-app Repo에 VERSION 업데이트 시 → ops-deploy Repo 에 dev-app 에 파일에 버전 정보 업데이트 작업 추가
1. 기존 버전 정보는 VERSION 파일 내에 정보를 가져와서 변수 지정 : OLDVER=$(cat dev-app/VERSION)
2. 신규 버전 정보는 environment 도커 태그 정보를 가져와서 변수 지정 : NEWVER=$(echo ${DOCKER_TAG})
3. 이후 sed 로 ops-deploy Repo 에 dev-app/VERSION, timeserver.yaml 2개 파일에 ‘기존 버전’ → ‘신규 버전’으로 값 변경
4. 이후 ops-deploy Repo 에 git push ⇒ Argo CD app 가 최대 3분 사이에 변경 확인 후 AutoSync 로 신규 버전 업데이트 진행
아래는 dev-app 에 위치한 Jenkinsfile 로 젠킨스에 SCM-Pipeline(SCM:git) 으로 사용되고 있는 파일을 수정해서 실습에 사용


pipeline {
    agent any
    environment {
        DOCKER_IMAGE = 'kimseongjung/dev-app' // Docker 이미지 이름
        GOGSCRD = credentials('gogs-crd')
    }
    stages {
        stage('dev-app Checkout') {
            steps {
                 git branch: 'main',
                 url: 'http://192.168.0.49:3000/devops/dev-app.git',  // Git에서 코드 체크아웃
                 credentialsId: 'gogs-crd'  // Credentials ID
            }
        }
        stage('Read VERSION') {
            steps {
                script {
                    // VERSION 파일 읽기
                    def version = readFile('VERSION').trim()
                    echo "Version found: ${version}"
                    // 환경 변수 설정
                    env.DOCKER_TAG = version
                }
            }
        }
        stage('Docker Build and Push') {
            steps {
                script {
                    docker.withRegistry('https://index.docker.io/v1/', 'dockerhub-crd') {
                        // DOCKER_TAG 사용
                        def appImage = docker.build("${DOCKER_IMAGE}:${DOCKER_TAG}")
                        appImage.push()
                        appImage.push("latest")
                    }
                }
            }
        }
        stage('ops-deploy Checkout') {
            steps {
                 git branch: 'main',
                 url: 'http://192.168.0.49:3000/devops/ops-deploy.git',  // Git에서 코드 체크아웃
                 credentialsId: 'gogs-crd'  // Credentials ID
            }
        }
        stage('ops-deploy version update push') {
            steps {
                sh '''
                OLDVER=$(cat dev-app/VERSION)
                NEWVER=$(echo ${DOCKER_TAG})
                sed -i -e "s/$OLDVER/$NEWVER/" dev-app/timeserver.yaml
                sed -i -e "s/$OLDVER/$NEWVER/" dev-app/VERSION
                git add ./dev-app
                git config user.name "devops"
                git config user.email "a@a.com"
                git commit -m "version update ${DOCKER_TAG}"
                git push http://${GOGSCRD_USR}:${GOGSCRD_PSW}@192.168.0.49:3000/devops/ops-deploy.git
                '''
            }
        }
    }
    post {
        success {
            echo "Docker image ${DOCKER_IMAGE}:${DOCKER_TAG} has been built and pushed successfully!"
        }
        failure {
            echo "Pipeline failed. Please check the logs."
        }
    }
}

아래는 dev-app (Repo) 에서 git push 수행


# VERSION 파일 수정 : 0.0.4
# server.py 파일 수정 : 0.0.4

# git push : VERSION, server.py, Jenkinsfile
git add . && git commit -m "VERSION $(cat VERSION) Changed" && git push -u origin main

4.10 동작 확인

Argo CD app 가 최대 3분 사이에 변경 확인 후 AutoSync 로 신규 버전 업데이트 진행 → Argo CD WebHook 설정 시 즉시 반영 가능
dev-app Repo 에서 한번 더 버전 업데이트 수행


# VERSION 파일 수정 : 0.0.5
# server.py 파일 수정 : 0.0.5

# git push : VERSION, server.py, Jenkinsfile
❯ git add . && git commit -m "VERSION $(cat VERSION) Changed" && git push -u origin main

[main dabb07f] VERSION 0.0.5 Changed
 2 files changed, 2 insertions(+), 2 deletions(-)
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 330 bytes | 330.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
To http://mypc:3000/devops/dev-app.git
   564d47e..dabb07f  main -> main
branch 'main' set up to track 'origin/main'.


# VERSION 파일 수정 : 0.0.6
# server.py 파일 수정 : 0.0.6

# git push : VERSION, server.py, Jenkinsfile
❯ git add . && git commit -m "VERSION $(cat VERSION) Changed" && git push -u origin main

[main dd5d6e7] VERSION 0.0.6 Changed
 2 files changed, 2 insertions(+), 2 deletions(-)
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 328 bytes | 328.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
To http://mypc:3000/devops/dev-app.git
   dabb07f..dd5d6e7  main -> main
branch 'main' set up to track 'origin/main'.

💘 즉, 개발팀 dev-app Repo 에서만 코드 업데이트 작업 시, jenkins pipeline 에서 ops-deploy Repo 에 버전 정보 업데이트를 하고,
이후 Argo CD가 자동으로 신규 버전 정보로 배포를 하게 된다.

5. Argo Rollout + K8S(Kind)

5.1 Argo Rollouts 소개

Kubernetes Progressive Delivery Controller - Docs
Argo Rollouts : Argo Rollouts is a Kubernetes controller and set of CRDs which provide advanced deployment capabilities such as blue-green, canary, canary analysis, experimentation, and progressive delivery features to Kubernetes.
- Argo Rollouts (optionally) integrates with ingress controllers and service meshes, leveraging their traffic shaping abilities to gradually shift traffic to the new version during an update. Additionally, Rollouts can query and interpret metrics from various providers to verify key KPIs and drive automated promotion or rollback during an update.
Why Argo Rollouts?
- The native Kubernetes Deployment Object supports the RollingUpdate strategy which provides a basic set of safety guarantees (readiness probes) during an update. However the rolling update strategy faces many limitations:
- Few controls over the speed of the rollout
- Inability to control traffic flow to the new version
- Readiness probes are unsuitable for deeper, stress, or one-time checks
- No ability to query external metrics to verify an update
- Can halt the progression, but unable to automatically abort and rollback the update
Controller Features
- Blue-Green update strategy
- Canary update strategy
- - Fine-grained, weighted traffic shifting
- Automated rollbacks and promotions
- Manual judgement
- Customizable metric queries and analysis of business KPIs
- Ingress controller integration: NGINX, ALB, Apache APISIX
- Service Mesh integration: Istio, Linkerd, SMI
- Simultaneous usage of multiple providers: SMI + NGINX, Istio + ALB, etc.
- Metric provider integration: Prometheus, Wavefront, Kayenta, Web, Kubernetes Jobs, Datadog, New Relic, Graphite, InfluxDB
아키텍처 - Docs
- Argo Rollouts controller :
- Rollout resource :
- Replica sets for old and new version :
- Ingress/Service :
- AnalysisTemplate and AnalysisRun :
- Metric providers :
- CLI and UI :

5.2 Argo Rollouts 설치 및 Sample 테스트 - Docs

Argo Rollouts 설치


# 네임스페이스 생성 및 파라미터 파일 작성
❯ cd $PWD

❯ kubectl create ns argo-rollouts
❯ cat <<EOT > argorollouts-values.yaml
dashboard:
  enabled: true
  service:
    type: NodePort
    nodePort: 30003
EOT

# 설치
❯ helm install argo-rollouts argo/argo-rollouts --version 2.35.1 -f argorollouts-values.yaml --namespace argo-rollouts

NAME: argo-rollouts
LAST DEPLOYED: Sat Dec 21 23:54:19 2024
NAMESPACE: argo-rollouts
STATUS: deployed
REVISION: 1
TEST SUITE: None

# 확인
❯ kubectl get all -n argo-rollouts

NAME                                           READY   STATUS    RESTARTS   AGE
pod/argo-rollouts-86469b5878-b88bp             1/1     Running   0          99s
pod/argo-rollouts-86469b5878-p8n4d             1/1     Running   0          99s
pod/argo-rollouts-dashboard-7c88d965fc-pl626   1/1     Running   0          99s

NAME                              TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/argo-rollouts-dashboard   NodePort   10.96.30.74   <none>        3100:30003/TCP   100s

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argo-rollouts             2/2     2            2           100s
deployment.apps/argo-rollouts-dashboard   1/1     1            1           100s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/argo-rollouts-86469b5878             2         2         2       99s
replicaset.apps/argo-rollouts-dashboard-7c88d965fc   1         1         1       99s

❯ kubectl get crds
NAME                                   CREATED AT
analysisruns.argoproj.io               2024-12-21T14:54:20Z
analysistemplates.argoproj.io          2024-12-21T14:54:20Z
applications.argoproj.io               2024-12-21T12:43:06Z
applicationsets.argoproj.io            2024-12-21T12:43:06Z
appprojects.argoproj.io                2024-12-21T12:43:06Z
clusteranalysistemplates.argoproj.io   2024-12-21T14:54:20Z
experiments.argoproj.io                2024-12-21T14:54:20Z
rollouts.argoproj.io                   2024-12-21T14:54:20Z

# Argo rollouts 대시보드 접속 주소 확인
❯ echo "http://127.0.0.1:30003"
http://127.0.0.1:30003

Deploying a Rollout


spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {}
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}


# Run the following command to deploy the initial Rollout and Service:
❯ kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/rollout.yaml
rollout.argoproj.io/rollouts-demo created

❯ kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/basic/service.yaml
service/rollouts-demo created

# 확인
❯ kubectl get rollout
NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
rollouts-demo   5         5         5            5           41s

❯ kubectl describe rollout

Name:         rollouts-demo
Namespace:    default
Labels:       <none>
Annotations:  rollout.argoproj.io/revision: 1
API Version:  argoproj.io/v1alpha1
Kind:         Rollout
Metadata:
  Creation Timestamp:  2024-12-21T15:01:48Z
  Generation:          1
  Resource Version:    30598
  UID:                 d5c9ffbf-0501-4cab-9605-f039c5dfbe8d
Spec:
  Replicas:                5
  Revision History Limit:  2
  Selector:
    Match Labels:
      App:  rollouts-demo
  Strategy:
    Canary:
      Steps:
        Set Weight:  20
        Pause:
        Set Weight:  40
        Pause:
          Duration:  10
        Set Weight:  60
        Pause:
          Duration:  10
        Set Weight:  80
        Pause:
          Duration:  10
  Template:
    Metadata:
      Labels:
        App:  rollouts-demo
    Spec:
      Containers:
        Image:  argoproj/rollouts-demo:blue
        Name:   rollouts-demo
        Ports:
          Container Port:  8080
          Name:            http
          Protocol:        TCP
        Resources:
          Requests:
            Cpu:     5m
            Memory:  32Mi
Status:
  HPA Replicas:        5
  Available Replicas:  5
  Blue Green:
  Canary:
  Conditions:
    Last Transition Time:  2024-12-21T15:01:48Z
    Last Update Time:      2024-12-21T15:01:48Z
    Message:               RolloutCompleted
    Reason:                RolloutCompleted
    Status:                True
    Type:                  Completed
    Last Transition Time:  2024-12-21T15:01:58Z
    Last Update Time:      2024-12-21T15:01:58Z
    Message:               Rollout is healthy
    Reason:                RolloutHealthy
    Status:                True
    Type:                  Healthy
    Last Transition Time:  2024-12-21T15:01:48Z
    Last Update Time:      2024-12-21T15:01:58Z
    Message:               ReplicaSet "rollouts-demo-687d76d795" has successfully progressed.
    Reason:                NewReplicaSetAvailable
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2024-12-21T15:01:58Z
    Last Update Time:      2024-12-21T15:01:58Z
    Message:               Rollout has minimum availability
    Reason:                AvailableReason
    Status:                True
    Type:                  Available
  Current Pod Hash:        687d76d795
  Current Step Hash:       f64cdc9d
  Current Step Index:      8
  Observed Generation:     1
  Phase:                   Healthy
  Ready Replicas:          5
  Replicas:                5
  Selector:                app=rollouts-demo
  Stable RS:               687d76d795
  Updated Replicas:        5
Events:
  Type    Reason                  Age   From                 Message
  ----    ------                  ----  ----                 -------
  Normal  RolloutAddedToInformer  59s   rollouts-controller  Rollout resource added to informer: default/rollouts-demo
  Normal  RolloutNotCompleted     59s   rollouts-controller  Rollout not completed, started update to revision 1 (687d76d795)
  Normal  RolloutUpdated          59s   rollouts-controller  Rollout updated to revision 1
  Normal  NewReplicaSetCreated    59s   rollouts-controller  Created ReplicaSet rollouts-demo-687d76d795 (revision 1)
  Normal  ScalingReplicaSet       59s   rollouts-controller  Scaled up ReplicaSet rollouts-demo-687d76d795 (revision 1) from 0 to 5
  Normal  RolloutCompleted        59s   rollouts-controller  Rollout completed update to revision 1 (687d76d795): Initial deploy
  
❯ kubectl get pod -l app=rollouts-demo

NAME                             READY   STATUS    RESTARTS   AGE
rollouts-demo-687d76d795-7wffh   1/1     Running   0          84s
rollouts-demo-687d76d795-8fx6c   1/1     Running   0          84s
rollouts-demo-687d76d795-dpfz5   1/1     Running   0          84s
rollouts-demo-687d76d795-m5wss   1/1     Running   0          84s
rollouts-demo-687d76d795-tk76f   1/1     Running   0          84s

❯ kubectl get svc,ep rollouts-demo

NAME                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/rollouts-demo   ClusterIP   10.96.128.7   <none>        80/TCP    85s

NAME                      ENDPOINTS                                                        AGE
endpoints/rollouts-demo   10.244.1.21:8080,10.244.1.22:8080,10.244.2.19:8080 + 2 more...   85s

❯ kubectl get rollouts rollouts-demo -o json | grep rollouts-demo

            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"Rollout\",\"metadata\":{\"annotations\":{},\"name\":\"rollouts-demo\",\"namespace\":\"default\"},\"spec\":{\"replicas\":5,\"revisionHistoryLimit\":2,\"selector\":{\"matchLabels\":{\"app\":\"rollouts-demo\"}},\"strategy\":{\"canary\":{\"steps\":[{\"setWeight\":20},{\"pause\":{}},{\"setWeight\":40},{\"pause\":{\"duration\":10}},{\"setWeight\":60},{\"pause\":{\"duration\":10}},{\"setWeight\":80},{\"pause\":{\"duration\":10}}]}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"rollouts-demo\"}},\"spec\":{\"containers\":[{\"image\":\"argoproj/rollouts-demo:blue\",\"name\":\"rollouts-demo\",\"ports\":[{\"containerPort\":8080,\"name\":\"http\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"5m\",\"memory\":\"32Mi\"}}}]}}}}\n",
        "name": "rollouts-demo",
                "app": "rollouts-demo"
                    "app": "rollouts-demo"
                        "image": "argoproj/rollouts-demo:blue",
                        "name": "rollouts-demo",
                "message": "ReplicaSet \"rollouts-demo-687d76d795\" has successfully progressed.",
        "selector": "app=rollouts-demo",

👉 default 네임스페이스 선택 → rollout-demo 클릭

Updating a Rollout


# Run the following command to update the rollouts-demo Rollout with the "yellow" version of the container:
❯ kubectl edit rollouts rollouts-demo
..
     - image: argoproj/rollouts-demo:yellow
...

# 파드 label 정보 확인
❯ watch -d kubectl get pod -l app=rollouts-demo -owide --show-labels
Every 2.0s: kubectl get pod -l app=rollouts-demo -owide --show-labels                                                                              sjkim-m1.local: Sun Dec 22 00:12:05 2024

NAME                             READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES   LABELS
rollouts-demo-687d76d795-7wffh   1/1     Running   0          10m     10.244.2.19   myk8s-worker2   <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=687d76d
795
rollouts-demo-687d76d795-8fx6c   1/1     Running   0          10m     10.244.2.20   myk8s-worker2   <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=687d76d
795
rollouts-demo-687d76d795-dpfz5   1/1     Running   0          10m     10.244.1.22   myk8s-worker    <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=687d76d
795
rollouts-demo-687d76d795-m5wss   1/1     Running   0          10m     10.244.1.21   myk8s-worker    <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=687d76d
795
rollouts-demo-6cf78c66c5-4hql8   1/1     Running   0          2m11s   10.244.2.22   myk8s-worker2   <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=6cf78c6

Promoting a Rollout

# 아래 입력 혹은 UI에서 Promote Yes 클릭

# 정보 확인
❯ kubectl get rollouts rollouts-demo -o json | grep rollouts-demo

            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"Rollout\",\"metadata\":{\"annotations\":{},\"name\":\"rollouts-demo\",\"namespace\":\"default\"},\"spec\":{\"replicas\":5,\"revisionHistoryLimit\":2,\"selector\":{\"matchLabels\":{\"app\":\"rollouts-demo\"}},\"strategy\":{\"canary\":{\"steps\":[{\"setWeight\":20},{\"pause\":{}},{\"setWeight\":40},{\"pause\":{\"duration\":10}},{\"setWeight\":60},{\"pause\":{\"duration\":10}},{\"setWeight\":80},{\"pause\":{\"duration\":10}}]}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"rollouts-demo\"}},\"spec\":{\"containers\":[{\"image\":\"argoproj/rollouts-demo:blue\",\"name\":\"rollouts-demo\",\"ports\":[{\"containerPort\":8080,\"name\":\"http\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"5m\",\"memory\":\"32Mi\"}}}]}}}}\n",
        "name": "rollouts-demo",
                "app": "rollouts-demo"
                    "app": "rollouts-demo"
                        "image": "argoproj/rollouts-demo:yellow",
                        "name": "rollouts-demo",
                "message": "ReplicaSet \"rollouts-demo-6cf78c66c5\" has successfully progressed.",
        "selector": "app=rollouts-demo",
        
Every 2.0s: kubectl get pod -l app=rollouts-demo -owide --show-labels                                                                              sjkim-m1.local: Sun Dec 22 00:17:46 2024

NAME                             READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES   LABELS
rollouts-demo-6cf78c66c5-4hql8   1/1     Running   0          7m52s   10.244.2.22   myk8s-worker2   <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=6cf78c6
6c5
rollouts-demo-6cf78c66c5-58bst   1/1     Running   0          80s     10.244.2.23   myk8s-worker2   <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=6cf78c6
6c5
rollouts-demo-6cf78c66c5-kvtsp   1/1     Running   0          69s     10.244.1.24   myk8s-worker    <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=6cf78c6
6c5
rollouts-demo-6cf78c66c5-mffd6   1/1     Running   0          58s     10.244.2.24   myk8s-worker2   <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=6cf78c6
6c5
rollouts-demo-6cf78c66c5-w8xsf   1/1     Running   0          95s     10.244.1.23   myk8s-worker    <none>           <none>            app=rollouts-demo,rollouts-pod-template-hash=6cf78c6
6c5

6.학습 후기

❤️❤️❤️❤️❤️ 평소 꼭 알고 싶었던 ci/cd를 활용한 k8s 배포와 argo cd/rollout을 이번 학습을 통해 충분한 실습을 할 수 있었다.
🎯🎯🎯🎯🎯 3주 과정이었지만 매우 알찬 학습시간이었다. 2024년 CloudNet@ 가시다님과의 학습을 통해 많은 성장을 하게 되어 매우 기쁘고 감사합니다.

김성중

I'm SJ

이전 포스트

3주차 - Jenkins CI/ArgoCD + K8S(1/2)

1개의 댓글

JUNKYU KIM

2024년 12월 22일

정리하시느라 수고 많으셨습니다. 좋은 주말 되세요~

답글 달기