3주차 - Jenkins CI/ArgoCD + K8S(1/2)

김성중·2024년 12월 15일
argocdgogsjenkinsk8skind

CI/CD

목록 보기
3/4
post-thumbnail

CloudNet@ 가시다님이 진행하는 단기 CI/CD 과정 Study 내용을 실습한 내용과 개인적인 경험을 정리하고자 합니다. 3주차 학습한 내용은 Jenkins CI/ArgoCD + K8S 입니다.

1. 실습환경 구성(/w macOS)

1.1 Docker에 컨테이너 2대(jenkins, gogs)

  • 호스트 OS 포트 노출(expose) 하여 접속 및 사용할 수 있도록 함
  • Jenkis, gogs 컨테이너 기동

# 작업 디렉토리 생성 후 이동
❯ mkdir cicd-labs
❯ cd cicd-labs

# docker-compose.yaml manifest 생성 
❯ cat <<EOT > docker-compose.yaml
services:

  jenkins:
    container_name: jenkins
    image: jenkins/jenkins
    restart: unless-stopped
    networks:
      - cicd-network
    ports:
      - "8080:8080"
      - "50000:50000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - jenkins_home:/var/jenkins_home

  gogs:
    container_name: gogs
    image: gogs/gogs
    restart: unless-stopped
    networks:
      - cicd-network
    ports:
      - "10022:22"
      - "3000:3000"
    volumes:
      - gogs-data:/data

volumes:
  jenkins_home:
  gogs-data:

networks:
  cicd-network:
    driver: bridge
EOT


# 배포
❯ docker compose up -d
[+] Running 5/5
 ✔ Network cicd-3w_cicd-network   Created                                                                                                                  0.0s 
 ✔ Volume "cicd-3w_gogs-data"     Created                                                                                                                  0.0s 
 ✔ Volume "cicd-3w_jenkins_home"  Created                                                                                                                  0.0s 
 ✔ Container jenkins              Started                                                                                                                  0.3s 
 ✔ Container gogs                 Started
 
❯ docker compose ps
NAME      IMAGE             COMMAND                  SERVICE   CREATED          STATUS                    PORTS
gogs      gogs/gogs         "/app/gogs/docker/st…"   gogs      54 seconds ago   Up 53 seconds (healthy)   0.0.0.0:3000->3000/tcp, 0.0.0.0:10022->22/tcp
jenkins   jenkins/jenkins   "/usr/bin/tini -- /u…"   jenkins   54 seconds ago   Up 53 seconds             0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp

# 기본 정보 확인
❯ for i in gogs jenkins ; do echo ">> container : $i <<"; docker compose exec $i sh -c "whoami && pwd"; echo; done

>> container : gogs <<
root
/app/gogs

>> container : jenkins <<
jenkins
/

# 도커를 이용하여 각 컨테이너로 접속
❯ docker compose exec jenkins bash
jenkins@05294177dcea:/$ exit
exit
❯ docker compose exec gogs bash
88be52d387a9:/app/gogs# exit
exit

1.2 Jenkins 설정

  • Jenkins 컨테이너 초기 설정
# Jenkins 초기 암호 확인
❯ docker compose exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
e2bd81faaa944998a0df7c90dd7fe95b

# Jenkins 웹 접속 주소 확인 : 계정 / 암호 입력 >> admin / qwe123
open "http://127.0.0.1:8080" # macOS
웹 브라우저에서 http://127.0.0.1:8080 접속 # Windows

# (참고) 로그 확인 : 플러그인 설치 과정 확인
❯ docker compose logs jenkins -f
jenkins  | Running from: /usr/share/jenkins/jenkins.war
jenkins  | webroot: /var/jenkins_home/war
jenkins  | 2024-12-21 02:05:01.310+0000 [id=1]  INFO    winstone.Logger#logInternal: Beginning extraction from war file
jenkins  | 2024-12-21 02:05:01.824+0000 [id=1]  WARNING o.e.j.ee9.nested.ContextHandler#setContextPath: Empty contextPath
jenkins  | 2024-12-21 02:05:01.852+0000 [id=1]  INFO    org.eclipse.jetty.server.Server#doStart: jetty-12.0.14; built: 2024-09-30T14:22:54.197Z; git: e77516598a07cca826d27fa8a4f7c70e953921a6; jvm 17.0.13+11

--중간 생략--

h.model.UpdateCenter$DownloadJob#run: Installation successful: dark-theme
jenkins  | 2024-12-21 02:34:53.171+0000 [id=91] INFO    h.m.UpdateCenter$CompleteBatchJob#run: Completing installing of plugin batch…
jenkins  | 2024-12-21 02:34:54.332+0000 [id=91] INFO    h.p.b.g.GlobalTimeOutConfiguration#load: global timeout not set
jenkins  | 2024-12-21 02:34:56.077+0000 [id=91] INFO    h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
jenkins  | 2024-12-21 02:34:57.794+0000 [id=91] INFO    h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.plugins.gradle.GradleInstaller
jenkins  | 2024-12-21 02:34:58.023+0000 [id=238]        INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
jenkins  | 2024-12-21 02:34:58.035+0000 [id=244]        INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
jenkins  | 2024-12-21 02:34:58.036+0000 [id=244]        INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
jenkins  | 2024-12-21 02:34:58.049+0000 [id=242]        INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
jenkins  | 2024-12-21 02:34:58.050+0000 [id=243]        INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
jenkins  | 2024-12-21 02:34:58.054+0000 [id=241]        INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded
jenkins  | 2024-12-21 02:34:58.054+0000 [id=241]        INFO    jenkins.InitReactorRunner$1#onAttained: System config adapted
jenkins  | 2024-12-21 02:34:58.105+0000 [id=244]        INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
jenkins  | 2024-12-21 02:34:58.106+0000 [id=238]        INFO    jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
jenkins  | 2024-12-21 02:34:58.319+0000 [id=245]        INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
jenkins  | 2024-12-21 02:34:58.321+0000 [id=91] INFO    h.m.UpdateCenter$CompleteBatchJob#run: Completed installation of 88 plugins in 1 min 42 sec

  • Jenkins URL 설정 : 각자 자신의 PC의 IP를 입력

    • 🍎 macOS에서 자신의 IP 확인 방법 :
      ❯ ipconfig getifaddr en0
      192.168.0.49

  • ⚠️ 실습 완료 후 해당 컨테이너 중지 상태로 둘 경우 → 재부팅 및 이후에 다시 실습을 위해 컨테이너 시작 시


# 실습 완료 후 해당 컨테이너 중지 상태로 둘 경우
docker compose stop
docker compose ps
docker compose ps -a

# mac 재부팅 및 이후에 다시 실습을 위해 컨테이너 시작 시
docker compose start
docker compose ps
  • ⚠️ 특정 컨테이너만 삭제 후 다시 초기화 상태로 기동 시

# gogs : 볼륨까지 삭제
docker compose down gogs -v
docker compose up gogs -d

# jenkins : 볼륨까지 삭제
docker compose down jenkins -v
docker compose up jenkins -d
  • ⛔️ Jenkins Container 내에서 docker 명령어 실행 퍼미션 오류 발생 시

# 소켓 파일에 그룹이 다시 원상태로 복귀...
docker compose exec --privileged -u root jenkins ls -l /var/run/docker.sock
srw-rw---- 1 root root 0 Dec 21 02:04 /var/run/docker.sock
# 소켓 파일에 docker 그룹을 다시 지정
docker compose exec --privileged -u root jenkins chgrp docker /var/run/docker.sock
# 확인
docker compose exec --privileged -u root jenkins ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Dec 21 02:04 /var/run/docker.sock
docker compose exec jenkins docker info

  • Jenkins 컨테이너에서 호스트에 도커 데몬 사용 설정 (Docker-out-of-Docker)

  • DooD로 Build 환경구성 위해 Jenkins 컨테이너 내부에 Docker Client 실행파일 설치

# Jenkins 컨테이너 내부에 도커 실행 파일 설치
❯ docker compose exec --privileged -u root jenkins bash

root@05294177dcea:/# id
uid=0(root) gid=0(root) groups=0(root)

root@05294177dcea:/# curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc

root@05294177dcea:/# chmod a+r /etc/apt/keyrings/docker.asc

root@05294177dcea:/# echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null

root@05294177dcea:/# apt-get update && apt install docker-ce-cli curl tree jq yq -y
Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 https://download.docker.com/linux/debian bookworm InRelease [43.3 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
--중간 생략--
Processing triggers for libc-bin (2.36-9+deb12u9) ...
root@05294177dcea:/# 

root@05294177dcea:/# docker info
Client: Docker Engine - Community
 Version:    27.4.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 2
  Paused: 0
  Stopped: 2
 Images: 26
 Server Version: 27.4.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.654GiB
 Name: docker-desktop
 ID: 3c04a1af-86e9-44fe-91ed-cc64e951af51
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/sjkim/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

root@05294177dcea:/# docker ps
CONTAINER ID   IMAGE             COMMAND                  CREATED             STATUS                       PORTS                                              NAMES
05294177dcea   jenkins/jenkins   "/usr/bin/tini -- /u…"   About an hour ago   Up About an hour             0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   jenkins
88be52d387a9   gogs/gogs         "/app/gogs/docker/st…"   About an hour ago   Up About an hour (healthy)   0.0.0.0:3000->3000/tcp, 0.0.0.0:10022->22/tcp      gogs

root@05294177dcea:/# which docker
/usr/bin/docker

# Jenkins 컨테이너 내부에서 root가 아닌 jenkins 유저도 docker를 실행할 수 있도록 권한을 부여
root@05294177dcea:/# groupadd -g 2000 -f docker
root@05294177dcea:/# chgrp docker /var/run/docker.sock

root@05294177dcea:/# ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Dec 21 02:04 /var/run/docker.sock

root@05294177dcea:/# grep docker /etc/group
docker:x:2000:

root@05294177dcea:/# usermod -aG docker jenkins
root@05294177dcea:/# grep docker /etc/group
docker:x:2000:jenkins
exit
--------------------------------------------

# jenkins 재기동
❯ docker compose restart jenkins
[+] Restarting 1/1
 ✔ Container jenkins  Started                                                           
 
# jenkins user로 docker 명령 실행 확인
❯ docker compose exec jenkins id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins),2000(docker)
                                                                   0.4s 
❯ docker compose exec jenkins docker info
Client: Docker Engine - Community
 Version:    27.4.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 2
  Paused: 0
  Stopped: 2
 Images: 26
 Server Version: 27.4.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.654GiB
 Name: docker-desktop
 ID: 3c04a1af-86e9-44fe-91ed-cc64e951af51
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/sjkim/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

❯ docker compose exec jenkins docker ps
CONTAINER ID   IMAGE             COMMAND                  CREATED             STATUS                       PORTS                                              NAMES
05294177dcea   jenkins/jenkins   "/usr/bin/tini -- /u…"   About an hour ago   Up 39 seconds                0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   jenkins
88be52d387a9   gogs/gogs         "/app/gogs/docker/st…"   About an hour ago   Up About an hour (healthy)   0.0.0.0:3000->3000/tcp, 0.0.0.0:10022->22/tcp      gogs

1.3 Gogs 설정

  • Gogs 컨테이너 초기 설정 : Repo(Private) - dev-app , ops-deploy

  • 초기 설정 웹 접속

  • 초기 설정

    • 데이터베이스 유형 : SQLite3
    • 애플리케이션 URL : http://<각자 자신의 PC IP>:3000/
    • 기본 브랜치 : main
    • 관리자 계정 설정 클릭 : 이름(계정명 - 닉네임 사용 devops), 비밀번호(계정암호 qwe123), 이메일 입력

  • Gogs 설치하기 클릭 ⇒ 관리자 계정으로 로그인 후 접속

    • 로그인 후 → Your Settings → Applications : Generate New Token 클릭 - Token Name(devops) ⇒ Generate Token 클릭 : 메모해두기! b5b3eb2f31abecae7c5a9c21bcf52cb909cc8ea9

  • New Repository 1 : 개발팀용

    • Repository Name : dev-app
    • Visibility : (Check) This repository is Private
    • .gitignore : Python
    • Readme : Default → (Check) initialize this repository with selected files and template
      ⇒ Create Repository 클릭 : Repo 주소 확인

  • New Repository 2 : 데브옵스팀용

    • Repository Name : ops-deploy
    • Visibility : (Check) This repository is Private
    • .gitignore : macOS
    • Readme : Default → (Check) initialize this repository with selected files and template
      ⇒ Create Repository 클릭 : Repo 주소 확인

  • 생성된 dev-app, ops-deploy Repositories

1.4 Gogs 실습을 위한 저장소 설정 : 호스트에서 직접 git 작업

  • git clone & source push

# (옵션) GIT 인증 정보 초기화
git credential-cache exit

# dev-app repo clone
❯  git clone http://mypc:3000/devops/dev-app.git
Cloning into 'dev-app'...
Username for 'http://192.168.254.124:3000': devops  # Gogs 계정명
Password for 'http://devops@192.168.254.124:3000': <토큰> # 혹은 계정암호
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (4/4), 713 bytes | 356.00 KiB/s, done.

# config 설정
❯ cd dev-app
❯ git config user.name "devops"
❯ git config user.email "a@a.com"
❯ git config init.defaultBranch main
❯ git config credential.helper store

# branch 확인
❯ git branch
* main

❯ git remote -v
origin  http://mypc:3000/devops/dev-app.git (fetch)
origin  http://mypc:3000/devops/dev-app.git (push)

# server.py 파일 작성
❯ cat > server.py <<EOF
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
from datetime import datetime
import socket

class RequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/plain')
        self.end_headers()

        now = datetime.now()
        hostname = socket.gethostname()
        response_string = now.strftime("The time is %-I:%M:%S %p, VERSION 0.0.1\n")
        response_string += f"Server hostname: {hostname}\n"
        self.wfile.write(bytes(response_string, "utf-8"))

def startServer():
    try:
        server = ThreadingHTTPServer(('', 80), RequestHandler)
        print("Listening on " + ":".join(map(str, server.server_address)))
        server.serve_forever()
    except KeyboardInterrupt:
        server.shutdown()

if __name__ == "__main__":
    startServer()
EOF

# Dockerfile 생성
❯ cat > Dockerfile <<EOF
FROM python:3.12
ENV PYTHONUNBUFFERED 1
COPY . /app
WORKDIR /app
CMD python3 server.py
EOF

# VERSION 파일 생성
❯ echo "0.0.1" > VERSION

# git 상태확인 및 소스 push
❯ git status

On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        Dockerfile
        VERSION
        server.py

nothing added to commit but untracked files present (use "git add" to track)
❯ git add .
❯ git commit -m "Add dev-app"
[main e6f4b47] Add dev-app
 3 files changed, 32 insertions(+)
 create mode 100644 Dockerfile
 create mode 100644 VERSION
 create mode 100644 server.py
❯ git push -u origin main
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 900 bytes | 900.00 KiB/s, done.
Total 5 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
To http://mypc:3000/devops/dev-app.git
   f7a525d..e6f4b47  main -> main
branch 'main' set up to track 'origin/main'.

  • Gogs Repos에서 확인

  • 자신의 도커 허브 계정에 dev-app (Private) repo 생성, (옵션) Token 발급

    • Dockerhub 로그인 후 token 발급
    • Generate new token : 만료일(편한대로), Access permissions(Read, Write, Delete)
    • 발급된 token 메모
    • Host에서 docker login 해보기
      ❯ docker login -u kimseongjung
      Password:
      Login Succeeded

2. Jenkins CI + K8S(Kind)

2.1 Kind는

  • ‘도커 IN 도커 docker in docker’로 쿠버네티스 클러스터 환경을 구성 - Link
    • kind or kubernetes in docker is a suite of tooling for local Kubernetes “clusters” where each “node” is a Docker container
    • kind is targeted at testing Kubernetes , kind supports multi-node (including HA) clusters
    • kind uses kubeadm to configure cluster nodes

2.2 Kind 설치

    1. Docker Desktop 설치 - Link
    • 실습 환경 : Kind 사용하는 도커 엔진 리소스에 최소 vCPU 4, Memory 8GB 할당을 권고 - 링크
    1. kind 및 툴 설치
    • 필수 툴 설치
    # Install Kind
brew install kind
kind --version
kind v0.26.0 go1.23.4 darwin/arm64

# Install kubectl
brew install kubernetes-cli
kubectl version --client=true
Client Version: v1.32.0
Kustomize Version: v5.5.0
Kubecolor Version: v0.4.0

## kubectl -> k 단축키 설정
echo "alias kubectl=kubecolor" >> ~/.zshrc

# Install Helm
brew install helm
helm version
version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"dirty", GoVersion:"go1.23.4"}
    • (권장) 유용한 툴 설치
    # 툴 설치
brew install krew
brew install kube-ps1
brew install kubectx

# kubectl 출력 시 하이라이트 처리
brew install kubecolor
echo "alias kubectl=kubecolor" >> ~/.zshrc
echo "compdef kubecolor=kubectl" >> ~/.zshrc

# krew 플러그인 설치
kubectl krew install neat stren
    1. kind 기본 사용 - 클러스터 배포 및 확인
      ⚠️ 회사에서 사용 중인 kubeconfig 가 있다면, 미리 kubeconfig 백업 후 kind 실습을 하시거나 혹은 아래 kubeconfig 변수 지정 사용 권장
    • (옵션) 별도 kubeconfig 지정 후 사용
    # 방안1 : 환경변수 지정
export KUBECONFIG=/Users/<Username>/Downloads/kind/config

# 방안2 : 혹은 --kubeconfig ./config 지정 가능

# 클러스터 생성
kind create cluster

# kubeconfig 파일 확인
ls -l /Users/<Username>/Downloads/kind/config
-rw-------  1 <Username>  staff  5608  4 24 09:05 /Users/<Username>/Downloads/kind/config

# 파드 정보 확인
ls 

# 클러스터 삭제
kind delete cluster
unset KUBECONFIG

2.3 Kind로 k8s 배포

  • 기본 정보 확인

# 클러스터 배포 전 확인
❯ docker ps
CONTAINER ID   IMAGE             COMMAND                  CREATED       STATUS                       PORTS                                              NAMES
05294177dcea   jenkins/jenkins   "/usr/bin/tini -- /u…"   3 hours ago   Up 2 hours                   0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   jenkins
88be52d387a9   gogs/gogs         "/app/gogs/docker/st…"   3 hours ago   Up About an hour (healthy)   0.0.0.0:3000->3000/tcp, 0.0.0.0:10022->22/tcp      gogs

# 방안1 : 환경변수 지정
cd ..
export KUBECONFIG=$PWD/kubeconfig

# Create a cluster with kind
MyIP=<각자 자신의 PC IP>
MyIP=192.168.0.49

❯ cat > kind-3node.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: "$MyIP"
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 30000
  - containerPort: 30001
    hostPort: 30001
  - containerPort: 30002
    hostPort: 30002
  - containerPort: 30003
    hostPort: 30003
- role: worker
- role: worker
EOF

❯ kind create cluster --config kind-3node.yaml --name myk8s --image kindest/node:v1.30.6

Creating cluster "myk8s" ...
 ✓ Ensuring node image (kindest/node:v1.30.6) 🖼 
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-myk8s"
You can now use your cluster with:

kubectl cluster-info --context kind-myk8s
Have a nice day! 👋

# 확인
❯ kind get nodes --name myk8s
myk8s-control-plane
myk8s-worker2
myk8s-worker

❯ kubens default
Context "kind-myk8s" modified.
Active namespace is "default".

# kind 는 별도 도커 네트워크 생성 후 사용 : 기본값 172.18.0.0/16
❯ docker network ls
NETWORK ID     NAME                   DRIVER    SCOPE
a9d45eee4624   bridge                 bridge    local
5d5870cbc369   cicd-3w_cicd-network   bridge    local
88153fefe41f   host                   host      local
69ae0c31875a   kind                   bridge    local
6e2fe0aabee3   none                   null      local

❯ docker inspect kind | jq

[
  {
    "Name": "kind",
    "Id": "69ae0c31875a22cdc2fc428961d44314d099d13a213b8b2faf2d2fe93dc15656",
    "Created": "2024-09-22T13:55:28.516763084Z",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": true,
    "IPAM": {
      "Driver": "default",
      "Options": {},
      "Config": [
        {
          "Subnet": "172.18.0.0/16",
          "Gateway": "172.18.0.1"
        },
        {
          "Subnet": "fc00:f853:ccd:e793::/64",
          "Gateway": "fc00:f853:ccd:e793::1"
        }
      ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
      "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {
      "16c5bcd4652e0861c055654ddc0442b5c0702de17116596a0effe3b6299adfbc": {
        "Name": "myk8s-control-plane",
        "EndpointID": "74183efa33ab764bba39708d315fe4e4a88eb5a34209232279b09b7e8574223f",
        "MacAddress": "02:42:ac:12:00:04",
        "IPv4Address": "172.18.0.4/16",
        "IPv6Address": "fc00:f853:ccd:e793::4/64"
      },
      "24be15dec4eaec72389d2fe346b19d49b9a9213d96fca82ebf022e8674ad5cdb": {
        "Name": "myk8s-worker",
        "EndpointID": "4252350dd1a37e654c93b7ff1f19795a995fb92149c50bcccd0b7a489f93df83",
        "MacAddress": "02:42:ac:12:00:02",
        "IPv4Address": "172.18.0.2/16",
        "IPv6Address": "fc00:f853:ccd:e793::2/64"
      },
      "d7d161664ddafafe43663e17c92eca3d10b7b7b3be8db6e7b150ca3621107a14": {
        "Name": "myk8s-worker2",
        "EndpointID": "3739bf49d48dfbea08cafb7554ab4ee124a8d9a0d190b0639f694d5675d1849e",
        "MacAddress": "02:42:ac:12:00:03",
        "IPv4Address": "172.18.0.3/16",
        "IPv6Address": "fc00:f853:ccd:e793::3/64"
      }
    },
    "Options": {
      "com.docker.network.bridge.enable_ip_masquerade": "true",
      "com.docker.network.driver.mtu": "65535"
    },
    "Labels": {}
  }
]

# k8s api 주소 확인 : 어떻게 로컬에서 접속이 되는 걸까?
❯ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:53846
CoreDNS is running at https://127.0.0.1:53846/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

# 노드 정보 확인 : CRI 는 containerd 사용
❯ kubectl get node -o wide

NAME                  STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
myk8s-control-plane   Ready    control-plane   10m   v1.30.6   172.18.0.4    <none>        Debian GNU/Linux 12 (bookworm)   6.10.14-linuxkit   containerd://1.7.18
myk8s-worker          Ready    <none>          10m   v1.30.6   172.18.0.2    <none>        Debian GNU/Linux 12 (bookworm)   6.10.14-linuxkit   containerd://1.7.18
myk8s-worker2         Ready    <none>          10m   v1.30.6   172.18.0.3    <none>        Debian GNU/Linux 12 (bookworm)   6.10.14-linuxkit   containerd://1.7.18

# 파드 정보 확인 : CNI 는 kindnet 사용
❯ kubectl get pod -A -o wide

NAMESPACE            NAME                                          READY   STATUS    RESTARTS   AGE   IP           NODE                  NOMINATED NODE   READINESS GATES
kube-system          coredns-55cb58b774-tmgzs                      1/1     Running   0          11m   10.244.0.3   myk8s-control-plane   <none>           <none>
kube-system          coredns-55cb58b774-zxks4                      1/1     Running   0          11m   10.244.0.2   myk8s-control-plane   <none>           <none>
kube-system          etcd-myk8s-control-plane                      1/1     Running   0          11m   172.18.0.4   myk8s-control-plane   <none>           <none>
kube-system          kindnet-nrw56                                 1/1     Running   0          11m   172.18.0.3   myk8s-worker2         <none>           <none>
kube-system          kindnet-qgwsf                                 1/1     Running   0          11m   172.18.0.4   myk8s-control-plane   <none>           <none>
kube-system          kindnet-vmsnt                                 1/1     Running   0          11m   172.18.0.2   myk8s-worker          <none>           <none>
kube-system          kube-apiserver-myk8s-control-plane            1/1     Running   0          11m   172.18.0.4   myk8s-control-plane   <none>           <none>
kube-system          kube-controller-manager-myk8s-control-plane   1/1     Running   0          11m   172.18.0.4   myk8s-control-plane   <none>           <none>
kube-system          kube-proxy-6gm4v                              1/1     Running   0          11m   172.18.0.2   myk8s-worker          <none>           <none>
kube-system          kube-proxy-6sfdf                              1/1     Running   0          11m   172.18.0.3   myk8s-worker2         <none>           <none>
kube-system          kube-proxy-pcb7s                              1/1     Running   0          11m   172.18.0.4   myk8s-control-plane   <none>           <none>
kube-system          kube-scheduler-myk8s-control-plane            1/1     Running   0          11m   172.18.0.4   myk8s-control-plane   <none>           <none>
local-path-storage   local-path-provisioner-7d4d9bdcc5-b588h       1/1     Running   0          11m   10.244.0.4   myk8s-control-plane   <none>           <none>

# 네임스페이스 확인 >> 도커 컨테이너에서 배운 네임스페이스와 다릅니다!
❯ kubectl get namespaces

NAME                 STATUS   AGE
default              Active   11m
kube-node-lease      Active   11m
kube-public          Active   11m
kube-system          Active   11m
local-path-storage   Active   11m

# 컨트롤플레인/워커 노드(컨테이너) 확인 : 도커 컨테이너 이름은 myk8s-control-plane , myk8s-worker/worker-2 임을 확인
❯ docker ps

CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS                       PORTS                                                             NAMES
16c5bcd4652e   kindest/node:v1.30.6   "/usr/local/bin/entr…"   12 minutes ago   Up 12 minutes                0.0.0.0:30000-30003->30000-30003/tcp, 127.0.0.1:53846->6443/tcp   myk8s-control-plane
d7d161664dda   kindest/node:v1.30.6   "/usr/local/bin/entr…"   12 minutes ago   Up 12 minutes                                                                                  myk8s-worker2
24be15dec4ea   kindest/node:v1.30.6   "/usr/local/bin/entr…"   12 minutes ago   Up 12 minutes                                                                                  myk8s-worker
05294177dcea   jenkins/jenkins        "/usr/bin/tini -- /u…"   4 hours ago      Up 3 hours                   0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp                  jenkins
88be52d387a9   gogs/gogs              "/app/gogs/docker/st…"   4 hours ago      Up About an hour (healthy)   0.0.0.0:3000->3000/tcp, 0.0.0.0:10022->22/tcp                     gogs

❯ docker images
REPOSITORY                                                                     TAG             IMAGE ID       CREATED         SIZE
jenkins/jenkins                                                                latest          5ac04a752146   2 weeks ago     813MB
gogs/gogs                                                                      latest          081dab22debc   2 weeks ago     160MB
kindest/node                                                                   v1.30.6         b6d08db72079   6 weeks ago     1.43GB

# 디버그용 내용 출력에 ~/.kube/config 권한 인증 로드
❯ kubectl get pod -v6

I1221 14:56:39.516708   77568 loader.go:402] Config loaded from file:  /Users/sjkim/Labs/CloudNeta/cicd/cicd-3w/kubeconfig
I1221 14:56:39.517567   77568 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1221 14:56:39.517579   77568 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I1221 14:56:39.517582   77568 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I1221 14:56:39.517589   77568 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1221 14:56:39.533830   77568 round_trippers.go:560] GET https://127.0.0.1:53846/api/v1/namespaces/default/pods?limit=500 200 OK in 11 milliseconds
No resources found in default namespace.

# kube config 파일 확인
❯ cat $KUBECONFIG

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJUEZDaHdFSS8xUmN3RFFZSkt...
    server: https://127.0.0.1:53846
  name: kind-myk8s
contexts:
- context:
    cluster: kind-myk8s
    namespace: default
    user: kind-myk8s
  name: kind-myk8s
current-context: kind-myk8s
kind: Config
preferences: {}
users:
- name: kind-myk8s
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLVENDQWhHZ0F3SUJBZ0lJRmhHRGVCQ1dFdWt3RFFZSkt...
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBcnVRS2hYTjdUTWNmQ1pISHF...
    
❯ ls -l $KUBECONFIG
-rw-------  1 sjkim  staff  5631 Dec 21 14:42 /Users/sjkim/Labs/CloudNeta/cicd/cicd-3w/kubeconfig

2.4 kube-ops-view

  • kube-ops-view 설치

# kube-ops-view add
❯ helm repo add geek-cookbook https://geek-cookbook.github.io/charts/

❯ helm show values geek-cookbook/kube-ops-view
#
# IMPORTANT NOTE
#
# This chart inherits from our common library chart. You can check the default values/options here:
# https://github.com/k8s-at-home/library-charts/tree/main/charts/stable/common/values.yaml
#

image:
  # -- image repository
  repository: hjacobs/kube-ops-view
  # -- image tag
  # @default -- chart.appVersion
  tag:
  # -- image pull policy
  pullPolicy: IfNotPresent

# -- environment variables. See more environment variables in the [kube-ops-view documentation](https://codeberg.org/hjacobs/kube-ops-view/#configuration).
# @default -- See below
env:
  # -- Set the container timezone
  TZ: UTC

# -- Configures service settings for the chart.
# @default -- See values.yaml
service:
  main:
    ports:
      http:
        port: 8080

securityContext:
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

serviceAccount:
  # -- create needed service account
  create: true

ingress:
  # -- Enable and configure ingress settings for the chart under this key.
  # @default -- See values.yaml
  main:
    enabled: false

❯ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "geek-cookbook" chart repository
Update Complete. ⎈Happy Helming!⎈

❯ helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=NodePort,service.main.ports.http.nodePort=30001 --set env.TZ="Asia/Seoul" --namespace kube-system
NAME: kube-ops-view
LAST DEPLOYED: Sat Dec 21 15:02:12 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace kube-system -o jsonpath="{.spec.ports[0].nodePort}" services kube-ops-view)
  export NODE_IP=$(kubectl get nodes --namespace kube-system -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
  
# 설치 확인
❯ kubectl get deploy,pod,svc,ep -n kube-system -l app.kubernetes.io/instance=kube-ops-view
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-ops-view   1/1     1            1           4m18s

NAME                                 READY   STATUS    RESTARTS   AGE
pod/kube-ops-view-796947d6dc-n2rhp   1/1     Running   0          4m18s

NAME                    TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/kube-ops-view   NodePort   10.96.73.96   <none>        8080:30001/TCP   4m18s

NAME                      ENDPOINTS         AGE
endpoints/kube-ops-view   10.244.2.2:8080   4m18s

# kube-ops-view 접속 URL 확인 (1.5 , 2 배율)
open "http://127.0.0.1:30001/#scale=1.5"
open "http://127.0.0.1:30001/#scale=2"

![](https://velog.velcdn.com/images/sejkim/post/e303a5de-dbac-4c42-8f9c-a89e846a2432/image.png)

2.5 k8s cluster 삭제


# 클러스터 삭제
kind delete cluster --name myk8s
docker ps
cat $KUBECONFIG
unset KUBECONFIG

2.6 Jenkins 설정: Plugin 설치, 자격증명 설정

  • Jenkins plugin 설치
  • Jenkins Plugin 설치
    • Pipeline Stage View - Docs
    • Docker Pipeline : building, testing, and using Docker images from Jenkins Pipeline - Docs
    • Gogs : Webhook Plugin - Docs
      • 예시 : http(s)://<< jenkins-server >>/gogs-webhook/?job=<<jobname>>
  • 자격증명 설정 : Jenkins 관리 → Credentials → Globals → Add Credentials
    1. Gogs Repo 자격증명 설정 : gogs-crd
    • Kind : Username with password
    • Username : devops
    • Password : <Gogs 토큰>
    • ID : gogs-crd
    1. 도커 허브 자격증명 설정 : dockerhub-crd
    • Kind : Username with password
    • Username : <도커 계정명>
    • Password : <도커 계정 암호 혹은 토큰>
    • ID : dockerhub-crd
    1. k8s(kind) 자격증명 설정 : k8s-crd
    • Kind : Secret file
    • File : <kubeconfig 파일 업로드>
    • ID : k8s-crd
  • Jenkins Credentials 설정된 내용

2.7 Jenkins Item 생성(Pipeline) : item name(pipeline-ci)

  • Pipeline script

pipeline {
    agent any
    environment {
        DOCKER_IMAGE = '<자신의 도커 허브 계정>/dev-app' // Docker 이미지 이름
    }
    stages {
        stage('Checkout') {
            steps {
                 git branch: 'main',
                 url: 'http://192.168.254.124:3000/devops/dev-app.git',  // Git에서 코드 체크아웃
                 credentialsId: 'gogs-crd'  // Credentials ID
            }
        }
        stage('Read VERSION') {
            steps {
                script {
                    // VERSION 파일 읽기
                    def version = readFile('VERSION').trim()
                    echo "Version found: ${version}"
                    // 환경 변수 설정
                    env.DOCKER_TAG = version
                }
            }
        }
        stage('Docker Build and Push') {
            steps {
                script {
                    docker.withRegistry('https://index.docker.io/v1/', 'dockerhub-crd') {
                        // DOCKER_TAG 사용
                        def appImage = docker.build("${DOCKER_IMAGE}:${DOCKER_TAG}")
                        appImage.push()
                        appImage.push("latest")
                    }
                }
            }
        }
    }
    post {
        success {
            echo "Docker image ${DOCKER_IMAGE}:${DOCKER_TAG} has been built and pushed successfully!"
        }
        failure {
            echo "Pipeline failed. Please check the logs."
        }
    }
}

  • 지금 빌드 → 콘솔 Output 확인

  • Dockerhub 확인

2.8 k8s Deploying an application

  • Deploying to Kubernetes
  • 원하는 상태 설정 시 k8s 는 충족을 위해 노력함 : Kubernetes uses declarative configuration, where you declare the state you want (like “I want 3 copies of my container running in the cluster”) in a configuration file. Then, submit that config to the cluster, and Kubernetes will strive to meet the requirements you specified.
  • Kubernetes manifest
  • Deployment 배포
# 디플로이먼트 오브젝트 배포 : 리플리카(파드 2개), 컨테이너 이미지 >> 아래 도커 계정 부분만 변경해서 배포해보자
DHUSER=<도커 허브 계정명>
❯ DHUSER=kimseongjung

❯ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: timeserver
spec:
  replicas: 2
  selector:
    matchLabels:
      pod: timeserver-pod
  template:
    metadata:
      labels:
        pod: timeserver-pod
    spec:
      containers:
      - name: timeserver-container
        image: docker.io/$DHUSER/dev-app:0.0.1
        imagePullPolicy: Always
        
EOF

deployment.apps/timeserver created

watch -d kubectl get deploy,pod -o wide

# 배포 상태 확인 : kube-ops-view 웹 확인
❯ kubectl get deploy,pod -o wide
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS             IMAGES                                 SELECTOR
deployment.apps/timeserver   0/2     2            0           34s   timeserver-container   docker.io/kimseongjung/dev-app:0.0.1   pod=timeserver-pod

NAME                              READY   STATUS         RESTARTS   AGE   IP           NODE            NOMINATED NODE   READINESS GATES
pod/timeserver-5dc66ff4d5-fpgvc   0/1     ErrImagePull   0          34s   10.244.1.6   myk8s-worker2   <none>           <none>
pod/timeserver-5dc66ff4d5-nbrvg   0/1     ErrImagePull   0          34s   10.244.2.7   myk8s-worker    <none>           <none>

❯ kubectl describe pod timeserver-5569965ccf-4hzg6
...
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  80s                default-scheduler  Successfully assigned default/timeserver-5dc66ff4d5-2nzxl to myk8s-worker2
  Normal   Pulling    41s (x3 over 79s)  kubelet            Pulling image "docker.io/kimseongjung/dev-app:0.0.1"
  Warning  Failed     40s (x3 over 78s)  kubelet            Failed to pull image "docker.io/kimseongjung/dev-app:0.0.1": failed to pull and unpack image "docker.io/kimseongjung/dev-app:0.0.1": failed to resolve reference "docker.io/kimseongjung/dev-app:0.0.1": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     40s (x3 over 78s)  kubelet            Error: ErrImagePull
  Normal   BackOff    1s (x5 over 77s)   kubelet            Back-off pulling image "docker.io/kimseongjung/dev-app:0.0.1"
  Warning  Failed     1s (x5 over 77s)   kubelet            Error: ImagePullBackOff```

- # k8s secret : 도커 자격증명 설정 
```bash

❯ kubectl get secret -A  # 생성 시 타입 지정

DHUSER=<도커 허브 계정>
DHPASS=<도커 허브 암호 혹은 토큰>
echo $DHUSER $DHPASS

❯ DHUSER=kimseongjung
❯ DHPASS=dckr_pat_vk2fd8*******ImPOucQab8
❯ echo $DHUSER $DHPASS

❯ kubectl create secret docker-registry dockerhub-secret \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=$DHUSER \
  --docker-password=$DHPASS

# 확인
❯ kubectl get secret
❯ kubectl describe secret

❯ kubectl get secrets -o yaml | kubectl neat # base64 인코딩 확인
❯ SECRET=eyJhdXRocyI6eyJodH****9fX0=
❯ echo "$SECRET" | base64 -d ; echo

# 디플로이먼트 오브젝트 업데이트 : 시크릿 적용 >> 아래 도커 계정 부분만 변경해서 배포해보자
❯ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: timeserver
spec:
  replicas: 2
  selector:
    matchLabels:
      pod: timeserver-pod
  template:
    metadata:
      labels:
        pod: timeserver-pod
    spec:
      containers:
      - name: timeserver-container
        image: docker.io/$DHUSER/dev-app:0.0.1
        imagePullPolicy: Always
      imagePullSecrets:
      - name: dockerhub-secret
EOF

# 확인
❯ kubectl get deploy,pod -o wide
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS             IMAGES                                 SELECTOR
deployment.apps/timeserver   2/2     2            2           40s   timeserver-container   docker.io/kimseongjung/dev-app:0.0.1   pod=timeserver-pod
NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE            NOMINATED NODE   READINESS GATES
pod/timeserver-b75764679-6bzml   1/1     Running   0          40s   10.244.1.7   myk8s-worker2   <none>           <none>
pod/timeserver-b75764679-jlfmq   1/1     Running   0          40s   10.244.2.8   myk8s-worker    <none>           <none>

# 접속을 위한 curl 파드 생성
❯ kubectl run curl-pod --image=curlimages/curl:latest --command -- sh -c "while true; do sleep 3600; done"

❯ kubectl get pod -owide
NAME                         READY   STATUS    RESTARTS   AGE     IP           NODE            NOMINATED NODE   READINESS GATES
curl-pod                     1/1     Running   0          22s     10.244.1.8   myk8s-worker2   <none>           <none>
timeserver-b75764679-6bzml   1/1     Running   0          2m34s   10.244.1.7   myk8s-worker2   <none>           <none>
timeserver-b75764679-jlfmq   1/1     Running   0          2m34s   10.244.2.8   myk8s-worker    <none>           <none>

# timeserver 파드 IP 1개 확인 후 접속 확인
PODIP1=<timeserver-Y 파드 IP>
❯ PODIP1=10.244.1.7

❯ kubectl exec -it curl-pod -- curl $PODIP1
The time is 7:48:03 AM, VERSION 0.0.1
Server hostname: timeserver-b75764679-6bzml

❯ kubectl exec -it curl-pod -- curl $PODIP1
The time is 7:48:08 AM, VERSION 0.0.1
Server hostname: timeserver-b75764679-6bzml

# 로그 확인
❯ kubectl logs deploy/timeserver

Found 2 pods, using pod/timeserver-b75764679-6bzml
Listening on 0.0.0.0:80
10.244.1.8 - - [21/Dec/2024 07:48:03] "GET / HTTP/1.1" 200 -
10.244.1.8 - - [21/Dec/2024 07:48:08] "GET / HTTP/1.1" 200 -

❯ kubectl logs deploy/timeserver -f
❯ kubectl stern deploy/timeserver
+ timeserver-b75764679-6bzml › timeserver-container
+ timeserver-b75764679-jlfmq › timeserver-container
timeserver-b75764679-6bzml timeserver-container Listening on 0.0.0.0:80
timeserver-b75764679-6bzml timeserver-container 10.244.1.8 - - [21/Dec/2024 07:48:03] "GET / HTTP/1.1" 200 -
timeserver-b75764679-6bzml timeserver-container 10.244.1.8 - - [21/Dec/2024 07:48:08] "GET / HTTP/1.1" 200 -
timeserver-b75764679-jlfmq timeserver-container Listening on 0.0.0.0:80

❯ kubectl stern -l pod=timeserver-pod
+ timeserver-b75764679-6bzml › timeserver-container
+ timeserver-b75764679-jlfmq › timeserver-container
timeserver-b75764679-jlfmq timeserver-container Listening on 0.0.0.0:80
timeserver-b75764679-6bzml timeserver-container Listening on 0.0.0.0:80
timeserver-b75764679-6bzml timeserver-container 10.244.1.8 - - [21/Dec/2024 07:48:03] "GET / HTTP/1.1" 200 -
timeserver-b75764679-6bzml timeserver-container 10.244.1.8 - - [21/Dec/2024 07:48:08] "GET / HTTP/1.1" 200 -
  • 파드 1개 삭제 후 동작 확인 → 접속 확인

POD1NAME=<파드 1개 이름>
POD1NAME=timeserver-b75764679-6bzml

kubectl get pod -owide
kubectl delete pod $POD1NAME && kubectl get pod -w

# 셀프 힐링 , 파드 IP 변경 -> 고정 진입점(고정 IP/도메인네임) 필요 => Service
kubectl get deploy,rs,pod -owide
  • Relationship between the Service and the Pods it targets (selects)
  • Service 생성

# 서비스 생성
❯ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: timeserver
spec:
  selector:
    pod: timeserver-pod
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    nodePort: 30000
  type: NodePort
EOF

# Service, Endpoints 조회
❯ kubectl get service,ep timeserver -owide
NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/timeserver   NodePort   10.96.31.251   <none>        80:30000/TCP   41s   pod=timeserver-pod

NAME                   ENDPOINTS                     AGE
endpoints/timeserver   10.244.1.9:80,10.244.2.8:80   41s

# Service(ClusterIP)로 접속 확인 : 도메인네임, ClusterIP
❯ kubectl exec -it curl-pod -- curl timeserver
The time is 7:57:45 AM, VERSION 0.0.1
Server hostname: timeserver-b75764679-llmps

❯ kubectl exec -it curl-pod -- curl $(kubectl get svc timeserver -o jsonpath={.spec.clusterIP})
The time is 7:58:13 AM, VERSION 0.0.1
Server hostname: timeserver-b75764679-llmps

# Service(NodePort)로 접속 확인 "노드IP:NodePort"
❯ curl http://127.0.0.1:30000
The time is 7:58:32 AM, VERSION 0.0.1
Server hostname: timeserver-b75764679-llmps

# 반복 접속 해두기 : 부하분산 확인
❯ while true; do curl -s --connect-timeout 1 http://127.0.0.1:30000 | grep name ; sleep 1 ; done

Server hostname: timeserver-b75764679-llmps
Server hostname: timeserver-b75764679-jlfmq
Server hostname: timeserver-b75764679-llmps
Server hostname: timeserver-b75764679-llmps
Server hostname: timeserver-b75764679-llmps
Server hostname: timeserver-b75764679-jlfmq
Server hostname: timeserver-b75764679-llmps
...

# 파드 복제복 증가 : service endpoint 대상에 자동 추가
❯ kubectl scale deployment timeserver --replicas 4
deployment.apps/timeserver scaled

❯ kubectl get service,ep timeserver -owide
NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/timeserver   NodePort   10.96.31.251   <none>        80:30000/TCP   4m53s   pod=timeserver-pod

NAME                   ENDPOINTS                                                AGE
endpoints/timeserver   10.244.1.10:80,10.244.1.9:80,10.244.2.8:80 + 1 more...   4m53s

# 반복 접속 해두기 : 부하분산 확인
❯ while true; do curl -s --connect-timeout 1 http://127.0.0.1:30000 | grep name ; sleep 1 ; done

Server hostname: timeserver-b75764679-fv8zp
Server hostname: timeserver-b75764679-rjcz9
Server hostname: timeserver-b75764679-llmps
Server hostname: timeserver-b75764679-fv8zp
Server hostname: timeserver-b75764679-llmps
^C%                                                                                                                                                       
❯ for i in {1..100};  do curl -s http://127.0.0.1:30000 | grep name; done | sort | uniq -c | sort -nr
  31 Server hostname: timeserver-b75764679-fv8zp
  28 Server hostname: timeserver-b75764679-rjcz9
  27 Server hostname: timeserver-b75764679-jlfmq
  14 Server hostname: timeserver-b75764679-llmps
  • 샘플 앱 server.py 코드 변경 → 새 0.0.2 버전 태그로 컨테이너 이미지 빌드 → 컨테이너 저장소 Push ⇒ k8s deployment 업데이트 배포

# VERSION 변경 : 0.0.2
# server.py 변경 : 0.0.2
❯ git add . && git commit -m "VERSION $(cat VERSION) Changed" && git push -u origin main
[main 95c2f58] VERSION 0.0.2 Changed
 2 files changed, 2 insertions(+), 2 deletions(-)
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 342 bytes | 342.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
To http://mypc:3000/devops/dev-app.git
   e6f4b47..95c2f58  main -> main
branch 'main' set up to track 'origin/main'.
  • 태그는 버전 정보 사용을 권장 : You can make this tag anything you like, but it’s a good convention to use version numbers.

# 파드 복제복 증가
❯ kubectl scale deployment timeserver --replicas 4
deployment.apps/timeserver scaled

❯ kubectl get service,ep timeserver -owide
NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/timeserver   NodePort   10.96.31.251   <none>        80:30000/TCP   11m   pod=timeserver-pod

NAME                   ENDPOINTS                                                AGE
endpoints/timeserver   10.244.1.10:80,10.244.1.9:80,10.244.2.8:80 + 1 more...   11m

# 반복 접속 해두기 : 부하분산 확인
❯ while true; do curl -s --connect-timeout 1 http://127.0.0.1:30000 | grep name ; sleep 1 ; done

Server hostname: timeserver-b75764679-fv8zp
Server hostname: timeserver-b75764679-rjcz9
Server hostname: timeserver-b75764679-fv8zp
Server hostname: timeserver-b75764679-jlfmq
Server hostname: timeserver-b75764679-jlfmq
Server hostname: timeserver-b75764679-rjcz9
Server hostname: timeserver-b75764679-jlfmq
Server hostname: timeserver-b75764679-llmps
Server hostname: timeserver-b75764679-jlfmq
Server hostname: timeserver-b75764679-jlfmq
Server hostname: timeserver-b75764679-llmps
Server hostname: timeserver-b75764679-rjcz9
Server hostname: timeserver-b75764679-rjcz9
^C%                                                                                                                                                       
❯ for i in {1..100};  do curl -s http://127.0.0.1:30000 | grep name; done | sort | uniq -c | sort -nr
  27 Server hostname: timeserver-b75764679-jlfmq
  26 Server hostname: timeserver-b75764679-rjcz9
  24 Server hostname: timeserver-b75764679-llmps
  23 Server hostname: timeserver-b75764679-fv8zp
  
#
❯ kubectl set image deployment timeserver timeserver-container=$DHUSER/dev-app:0.0.2 && watch -d "kubectl get deploy,ep timeserver; echo; kubectl get rs,pod"

# 롤링 업데이트 확인
❯ kubectl get deploy,rs,pod,svc,ep -owide
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS             IMAGES                       SELECTOR
deployment.apps/timeserver   4/4     4            4           25m   timeserver-container   kimseongjung/dev-app:0.0.2   pod=timeserver-pod

NAME                                    DESIRED   CURRENT   READY   AGE   CONTAINERS             IMAGES                                 SELECTOR
replicaset.apps/timeserver-5449b46ff7   4         4         4       57s   timeserver-container   kimseongjung/dev-app:0.0.2             pod=timeserver-pod,pod-template-hash=5449b46ff7
replicaset.apps/timeserver-b75764679    0         0         0       25m   timeserver-container   docker.io/kimseongjung/dev-app:0.0.1   pod=timeserver-pod,pod-template-hash=b75764679

NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
pod/curl-pod                      1/1     Running   0          23m   10.244.1.8    myk8s-worker2   <none>           <none>
pod/timeserver-5449b46ff7-27kfw   1/1     Running   0          52s   10.244.2.11   myk8s-worker    <none>           <none>
pod/timeserver-5449b46ff7-nk8gx   1/1     Running   0          57s   10.244.2.10   myk8s-worker    <none>           <none>
pod/timeserver-5449b46ff7-nsf7d   1/1     Running   0          57s   10.244.1.11   myk8s-worker2   <none>           <none>
pod/timeserver-5449b46ff7-nvz2r   1/1     Running   0          52s   10.244.1.12   myk8s-worker2   <none>           <none>

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE    SELECTOR
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        149m   <none>
service/timeserver   NodePort    10.96.31.251   <none>        80:30000/TCP   13m    pod=timeserver-pod

NAME                   ENDPOINTS                                                  AGE
endpoints/kubernetes   172.18.0.4:6443                                            149m
endpoints/timeserver   10.244.1.11:80,10.244.1.12:80,10.244.2.10:80 + 1 more...   13m

# kubectl get deploy $DEPLOYMENT_NAME
❯ kubectl get deploy timeserver

NAME         READY   UP-TO-DATE   AVAILABLE   AGE
timeserver   4/4     4            4           26m
❯ kubectl get pods -l pod=timeserver-pod

NAME                          READY   STATUS    RESTARTS   AGE
timeserver-5449b46ff7-27kfw   1/1     Running   0          108s
timeserver-5449b46ff7-nk8gx   1/1     Running   0          113s
timeserver-5449b46ff7-nsf7d   1/1     Running   0          113s
timeserver-5449b46ff7-nvz2r   1/1     Running   0          108s

#
❯ curl http://127.0.0.1:30000
The time is 8:11:18 AM, VERSION 0.0.2
Server hostname: timeserver-5449b46ff7-nvz2r

2.9 Gogs Webhooks 설정 : Jenkins Job Trigger

  • gogs에 app.ini 파일 수정 후 컨테이너 재기동 - issue

    • app.ini 내용 추가
    
[security]
INSTALL_LOCK = true
SECRET_KEY   = j2xaUPQcbAEwpIu
LOCAL_NETWORK_ALLOWLIST = 192.168.254.124 # 각자 자신의 PC IP
    • 수정하는 화면
    • docker compose restart gogs

  • Webhooks are much like basic HTTP POST event triggers. Whenever something occurs in Gogs,

  • we will handle the notification to the target host you specify. %!(EXTRA string=https://gogs.io/docs/features/webhook.html)

  • Webhooks 설정

    • Payload URL : http://mypc:8080/gogs-webhook/?job=SCM-Pipeline/
    • Content Type : application/json
    • Secret : jenkins123
    • When should this webhook be triggered? : Just the push event
    • Active : Check
      ⇒ Add webhook

2.10 Jenkins Item 생성(Pipeline) : item name(SCM-Pipeline)

  • GitHub Project, 소스관리-Git
  • Jenkins SCM-Pipeline 설정
    • GitHub project : http://***<mac IP>***:3000/***<Gogs 계정명>***/dev-app ← .git 은 제거
    • Use Gogs secret : qwe123
    • Build Triggers : Build when a change is pushed to Gogs 체크
    • Pipeline script from SCM
      • SCM : Git
        • Repo URL(http://***<mac IP>***:3000/***<Gogs 계정명>***/dev-app)
        • Credentials(devops/***)
        • Branch(*/main)
      • Script Path : Jenkinsfile

2.11 Jenkinsfile 작성 후 Git push

  • 호스트에서 직접 git 작업

# Jenkinsfile 빈 파일 작성
touch Jenkinsfile

# VERSION 파일 : 0.0.3 수정
# server.py 파일 : 0.0.3 수정
  • VSCODE 로 jenkins 컨테이너 내부 파일 Open 후 작성

pipeline {
    agent any
    environment {
        DOCKER_IMAGE = '<자신의 도커 허브 계정>/dev-app' // Docker 이미지 이름
    }
    stages {
        stage('Checkout') {
            steps {
                 git branch: 'main',
                 url: 'http://mypc:3000/devops/dev-app.git',  // Git에서 코드 체크아웃
                 credentialsId: 'gogs-crd'  // Credentials ID
            }
        }
        stage('Read VERSION') {
            steps {
                script {
                    // VERSION 파일 읽기
                    def version = readFile('VERSION').trim()
                    echo "Version found: ${version}"
                    // 환경 변수 설정
                    env.DOCKER_TAG = version
                }
            }
        }
        stage('Docker Build and Push') {
            steps {
                script {
                    docker.withRegistry('https://index.docker.io/v1/', 'dockerhub-crd') {
                        // DOCKER_TAG 사용
                        def appImage = docker.build("${DOCKER_IMAGE}:${DOCKER_TAG}")
                        appImage.push()
                        appImage.push("latest")
                    }
                }
            }
        }
    }
    post {
        success {
            echo "Docker image ${DOCKER_IMAGE}:${DOCKER_TAG} has been built and pushed successfully!"
        }
        failure {
            echo "Pipeline failed. Please check the logs."
        }
    }
}
  • 작성된 파일 push

[main fb350a4] VERSION 0.0.3 Changed
 3 files changed, 48 insertions(+), 2 deletions(-)
 create mode 100644 Jenkinsfile
Enumerating objects: 8, done.
Counting objects: 100% (8/8), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 1.05 KiB | 1.05 MiB/s, done.
Total 5 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
To http://mypc:3000/devops/dev-app.git
   95c2f58..fb350a4  main -> main
branch 'main' set up to track 'origin/main'.

  • Jenkins 트리거 빌드 확인

  • 도커 저장소 확인

  • Gogs WebHook 기록 확인

  • k8s 에 신규 버전 적용

kubectl set image deployment timeserver timeserver-container=$DHUSER/dev-app:0.0.3 && watch -d "kubectl get deploy,ep timeserver; echo; kubectl get rs,pod"

profile
김성중
I'm SJ
이전 포스트

2주차 - GitHub Actions CI/CD

다음 포스트

3주차 - Jenkins CI/ArgoCD + K8S(2/2)

0개의 댓글