CICD PipeLine 구축
- GithubAction을 사용하여 CI구축을 완료했었다
- 아직 메인 서비스 연결이 완성되지 않아 manifest를 사용하여 CD를 적용시키지 못했는데 replica1개로 하여 테스팅이 가능해져 CD를 구축하게 되었다
문제상황
- 서비스 코드를 push를 할때마다 tag 버전이 자동 증가하고 ECR에 저장이 되도록 했다. 이제 문제는 manifest에 있는 이미지 태그를 어떻게 동기화 시켜 적용할 것인가...!!
- 주로 Jenkins를 많이 사용하다보니 GithubAction & ArgoCD의 CICD조합 정보를 찾기 어려웠다
- kustomize를 사용한 방식으로 적용결정
- 참고자료 : https://viassh.github.io/network/github-action-ecr-cicd/
- Ops코드인 Kustomeize.yaml이 있는 Manifest github Repository를 생성해두기
- Dev코드는 GithubAction으로 커밋을 하면 자동으로 ECR에 Push가 된다
- 2번이 수행되면서 1번으로 이미지 태그가 동기화 된다
- 이를 ArgoCD에 등록하여 자동화시킨다
~/Downloads/project/eyestalk-manifest/ [main] tree
.
├── overlays
│ └── prd
│ └── kustomization.yaml
└── resources
├── back-config.yaml
├── back-deploy.yaml
├── front-deploy-svc-ingress.yaml
├── kustomization.yaml
├── secret.yaml
└── sig-deploy.yaml
3 directories, 7 files
ressources 디렉토리에 manifest파일을 전부 집어 넣기
여기서 kustomization.yaml에는 이미지 태그 변경이 필요한 파일을 작성한다
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: back
name: back-deploy
namespace: default ## 추후 ArgoCD동기화때 필요함
spec:
replicas: 1
selector:
matchLabels:
app: back
template:
metadata:
labels:
app: back
spec:
containers:
- env:
- name: SERVER_PORT
valueFrom:
configMapKeyRef:
name: eyestalk-cm
key: SERVER_PORT
- name: SPRING_DATASOURCE_URL
valueFrom:
configMapKeyRef:
name: eyestalk-cm
key: SPRING_DATASOURCE_URL
- name: expose
valueFrom:
configMapKeyRef:
name: eyestalk-cm
key: expose
- name: SPRING_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
name: username-pwd-secret
key: username
- name: SPRING_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
name: username-pwd-secret
key: password
image: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-spring:v0.1.6
name: back
ports:
- containerPort: 8080
name: back-container
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
labels:
app: back
name: back-svc
namespace: default ## 추후 ArgoCD동기화때 필요함
spec:
type: ClusterIP
ports:
- port: 8080
selector:
app: back
resources:
- back-deploy.yaml
- front-deploy-svc-ingress.yaml
- sig-deploy.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
/overlays/prd
경로 kustomization.yamlapiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
- name: /
newName: /
- name: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-python
newName: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-python
newTag: v1.4.7
- name: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-react
newName: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-react
newTag: v0.2.44
- name: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-spring
newName: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-spring
newTag: v0.1.20 ## 버전이 업데이트 될때마다 바뀜
resources:
- ../../resources
Secret Key추가하기
- 기존에는 자동생성된 GithubSecret을 사용하였으나 manifest repository를 빌드하면서 자동으로 workflow를 연결시켜 줘야 하기 때문에 권한을 줘야한다. github token넣기
manifest Repository에도 githubToken넣어주기!!
name: eyestalk-beta-test
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v2
- name: Bump version and push tag
id: tag_version
uses: mathieudutour/github-tag-action@v5.5 # 가져다 쓸 auto tagging 프로그램
with:
github_token: ${{ secrets.GITHUB_TOKEN }} # secrets.GITHUB_TOKEN는 자동생성됨
- name: Create a GitHub release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.PAT }}
with:
tag_name: ${{ steps.tag_version.outputs.new_tag }}
release_name: Release ${{ steps.tag_version.outputs.new_tag }}
body: ${{ steps.tag_version.outputs.changelog }}
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ap-northeast-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Build, tag, and push image to Amazon ECR
id: image-info
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: eyestalk-react
IMAGE_TAG: ${{ steps.tag_version.outputs.new_tag }}
run: |
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
- name: Checkout for Kustomize repository
uses: actions/checkout@v2
with:
# kubernetes yaml 파일 저장
repository: muji-StudyRoom/eyestalk-manifest # k8s yaml 파일이 있는 repo
ref: main # branch 이름
# 내 repository에 push 하기 위한 Personal Access Token이 필요
token: ${{ secrets.PAT }} # Github Action token을 발급받아서 repo secrect에 등록해줘야한다
path: eyestalk-manifest # 최상위 경로로 repository와 동일하게 설정
# 새 이미지 버전으로 파일의 태그값 수정
# cd path 수정
# kustomize로 image tag 값 변경
- name: Update Kubernetes resources
run: |
pwd
cd eyestalk-manifest/overlays/prd/
kustomize edit set image ${{ steps.login-ecr.outputs.registry }}/eyestalk-react=${{ steps.login-ecr.outputs.registry }}/eyestalk-react:${{ steps.tag_version.outputs.new_tag }}
cat kustomization.yaml
# 수정된 kustomization.yaml 파일 commit push
- name: Commit manifest files
env:
GITHUB_TOKEN: ${{ secrets.PAT }}
run: |
cd eyestalk-manifest
git config --global user.email "namju2912@gmail.com"
git config --global user.name "Jupiter-J"
git config --global github.token ${{ secrets.PAT }}
git commit -am "Update image tag ${{ steps.tag_version.outputs.new_tag }}"
git push -u origin main
name: eyestalk-beta-test
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v2
- name: Bump version and push tag
id: tag_version
uses: mathieudutour/github-tag-action@v5.5 # 가져다 쓸 auto tagging 프로그램
with:
github_token: ${{ secrets.GITHUB_TOKEN }} # secrets.GITHUB_TOKEN는 자동생성됨
- name: Create a GitHub release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.PAT }}
with:
tag_name: ${{ steps.tag_version.outputs.new_tag }}
release_name: Release ${{ steps.tag_version.outputs.new_tag }}
body: ${{ steps.tag_version.outputs.changelog }}
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ap-northeast-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Build, tag, and push image to Amazon ECR
id: image-info
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: eyestalk-spring
IMAGE_TAG: ${{ steps.tag_version.outputs.new_tag }}
run: |
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
- name: Checkout for Kustomize repository
uses: actions/checkout@v2
with:
# kubernetes yaml 파일 저장
repository: muji-StudyRoom/eyestalk-manifest # k8s yaml 파일이 있는 repo
ref: main # branch 이름
# 내 repository에 push 하기 위한 Personal Access Token이 필요
token: ${{ secrets.PAT }} # Github Action token을 발급받아서 repo secrect에 등록해줘야한다
path: eyestalk-manifest # 최상위 경로로 repository와 동일하게 설정
# 새 이미지 버전으로 파일의 태그값 수정
# cd path 수정
# kustomize로 image tag 값 변경
- name: Update Kubernetes resources
run: |
pwd
cd eyestalk-manifest/overlays/prd/
kustomize edit set image ${{ steps.login-ecr.outputs.registry }}/eyestalk-spring=${{ steps.login-ecr.outputs.registry }}/eyestalk-spring:${{ steps.tag_version.outputs.new_tag }}
cat kustomization.yaml
# 수정된 kustomization.yaml 파일 commit push
- name: Commit manifest files
env:
GITHUB_TOKEN: ${{ secrets.PAT }}
run: |
cd eyestalk-manifest
git config --global user.email "namju2912@gmail.com"
git config --global user.name "Jupiter-J"
git config --global github.token ${{ secrets.PAT }}
git commit -am "Update image tag ${{ steps.tag_version.outputs.new_tag }}"
git push -u origin main
name: eyestalk-beta-test
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v2
- name: Bump version and push tag
id: tag_version
uses: mathieudutour/github-tag-action@v5.5 # 가져다 쓸 auto tagging 프로그램
with:
github_token: ${{ secrets.GITHUB_TOKEN }} # secrets.GITHUB_TOKEN는 자동생성됨
- name: Create a GitHub release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.PAT }}
with:
tag_name: ${{ steps.tag_version.outputs.new_tag }}
release_name: Release ${{ steps.tag_version.outputs.new_tag }}
body: ${{ steps.tag_version.outputs.changelog }}
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ap-northeast-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Build, tag, and push image to Amazon ECR
id: image-info
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: eyestalk-python
IMAGE_TAG: ${{ steps.tag_version.outputs.new_tag }}
run: |
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
- name: Checkout for Kustomize repository
uses: actions/checkout@v2
with:
# kubernetes yaml 파일 저장
repository: muji-StudyRoom/eyestalk-manifest # k8s yaml 파일이 있는 repo
ref: main # branch 이름
# 내 repository에 push 하기 위한 Personal Access Token이 필요
token: ${{ secrets.PAT }} # Github Action token을 발급받아서 repo secrect에 등록해줘야한다
path: eyestalk-manifest # 최상위 경로로 repository와 동일하게 설정
# 새 이미지 버전으로 파일의 태그값 수정
# cd path 수정
# kustomize로 image tag 값 변경
- name: Update Kubernetes resources
run: |
pwd
cd eyestalk-manifest/overlays/prd/
kustomize edit set image ${{ steps.login-ecr.outputs.registry }}/eyestalk-python=${{ steps.login-ecr.outputs.registry }}/eyestalk-python:${{ steps.tag_version.outputs.new_tag }}
cat kustomization.yaml
# 수정된 kustomization.yaml 파일 commit push
- name: Commit manifest files
env:
GITHUB_TOKEN: ${{ secrets.PAT }}
run: |
cd eyestalk-manifest
git config --global user.email "namju2912@gmail.com"
git config --global user.name "Jupiter-J"
git config --global github.token ${{ secrets.PAT }}
git commit -am "Update image tag ${{ steps.tag_version.outputs.new_tag }}"
git push -u origin main
dev repository가 정상적으로 Action되면 manifest repository를 확인한다
new Tag부분이 버전에 따라 동기화되어 바뀐다
rapa0005:~/environment $ kubectl create namespace argocd
namespace/argocd created
rapa0005:~/environment $ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
rapa0005:~/environment $ VERSION=$(curl --silent "https://api.github.com/repos/argoproj/argo-cd/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
rapa0005:~/environment $ sudo curl --silent --location -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64
rapa0005:~/environment $ sudo chmod +x /usr/local/bin/argocd
rapa0005:~/environment $ kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
service/argocd-server patched
rapa0005:~/environment $ export ARGOCD_SERVER=`kubectl get svc argocd-server -n argocd -o json | jq --raw-output .status.loadBalancer.ingress[0].hostname`
rapa0005:~/environment $ echo $ARGOCD_SERVER
a432101fe900e479985c2b870590ad6a-1182218392.ap-northeast-2.elb.amazonaws.com
rapa0005:~/environment $ ARGO_PWD=`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`
rapa0005:~/environment $ echo $ARGO_PWD
a23EQ3vVMmV3CRSI
manifest repository에서 적었던 yaml부분에
namespace=default
를 모두 지정해줘야한다
자동 동기화
APP DETAILS 버튼 클릭 > ENABLE AUTO-SYNC 클릭
CI/CD의 내용과 별개로 서비스의 Manifest에 대해 정리하고자함
~/Downloads/project/eyestalk-manifest/ [main] tree
.
├── overlays
│ └── prd
│ └── kustomization.yaml
└── resources
├── back-config.yaml
├── back-deploy.yaml
├── front-deploy-svc-ingress.yaml
├── kustomization.yaml
├── secret.yaml
└── sig-deploy.yaml
3 directories, 7 files
개인 RDS를 사용하기 때문에 보안을 위하여 Secret을 사용했다.
데이터베이스 사용되는 username과 password를 적었다.
namespace: default는 ArgoCD 연결을 위해 사용된다
apiVersion: v1
kind: Secret
metadata:
name: username-pwd-secret
namespace: default
stringData:
username: "root"
password: "imsohappy"
데이터베이스, 스프링부트 환경변수를 Config.yaml에 사용했다.
스키마명, 포트번호, RDS 엔드포인트와 노출할 포트 및 연결할 서비스의 레이블을 넣는다
apiVersion: v1
kind: ConfigMap
metadata:
name: eyestalk-cm
namespace: default
data:
MARIADB_DATABASE: "eyestalk"
SERVER_PORT: "8080"
SPRING_DATASOURCE_URL: "jdbc:mysql://jupiterdb.cbll7n774vcb.ap-northeast-2.rds.amazonaws.com:3306/test2db?useSSL=false&characterEncoding=UTF-8&serverTimezone=Asia/Seoul&allowPublicKeyRetrieval=true"
expose: "8080"
API_IP: "myapp:8080"
apiVersion: apps/v1 ## 쿠버네티스의 apps/v1 API를 사용
kind: Deployment ## Deployment의 작업으로 명시
metadata:
labels:
app: back ## Deployment의 레이블 설정
name: back-deploy ## Deployment의 이름을 설정
namespace: default ## ArgoCD 연결을 위해 사용
spec:
replicas: 1 ## 테스트이기 때문에 1개로 설정
selector:
matchLabels:
app: back
template:
metadata:
labels:
app: back
spec:
containers:
- env: ## config에 사용한 변수 설정
- name: SERVER_PORT
valueFrom:
configMapKeyRef:
name: eyestalk-cm
key: SERVER_PORT
- name: SPRING_DATASOURCE_URL
valueFrom:
configMapKeyRef:
name: eyestalk-cm
key: SPRING_DATASOURCE_URL
- name: expose
valueFrom:
configMapKeyRef:
name: eyestalk-cm
key: expose
- name: SPRING_DATASOURCE_USERNAME
valueFrom:
secretKeyRef: ## secret에 사용한 변수 설정
name: username-pwd-secret
key: username
- name: SPRING_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
name: username-pwd-secret
key: password
image: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-spring:v0.1.6
name: back
ports:
- containerPort: 8080
name: back-container
volumeMounts:
- name: tz-seoul
mountPath: /etc/localtime
volumes:
- name: tz-seoul
hostPath:
path: /usr/share/zoneinfo/Asia/Seoul
restartPolicy: Always
---
apiVersion: v1 ## 네트워크 연결을 하기위해 노출해야함으로 service사용
kind: Service
metadata:
labels:
app: back
name: back-svc
namespace: default
spec:
type: ClusterIP
ports:
- port: 8080
selector:
app: back
StickySession추가 및 로드밸런서, Redis 추가
apiVersion: v1
kind: Service
metadata:
labels:
app: sig
name: sig-svc
namespace: default
spec:
type: ClusterIP
sessionAffinity: ClientIP
ports:
- port: 5000
selector:
app: sig
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-sig
namespace: default
annotations: ## 외부노출을 위해 alb 사용
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/subnets: subnet-0c1b964b7812dd512, subnet-021a53c08b0b9865d, subnet-05b587954a63c6d82, subnet-049c715b9991e8390
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:675694928506:certificate/580ad154-219c-4bcd-9560-535d06351f43
alb.ingress.kubernetes.io/target-group-atrtributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600 ## sticky session을 적용
alb.ingress.kubernetes.io/target-type: ip # using IP routing policy of ALB
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: sig-svc # refer to the service defined in deploy.yaml
port:
number: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sig-deploy
namespace: default
labels:
app: sig
spec:
replicas: 1
selector:
matchLabels:
app: sig
template:
metadata:
labels:
app: sig
spec:
containers:
- name: sig-container
image: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-python:v1.4.3
ports:
- containerPort: 5000
env:
- name: ES_IP
value: "abb6ec4d0ace44a07b0aa781fb01e587-123484602.ap-northeast-2.elb.amazonaws.com"
- name: ES_PORT
value: '9200'
- name: SPRING_IP
value: "back-svc"
- name: SPRING_PORT
value: '8080'
- name: REDIS_IP
value: "redis://redis-svc" ## redis연결
- name: REDIS_PORT
value: '6379'
apiVersion: v1
kind: Service
metadata:
name: front-svc
namespace: default
spec:
type: ClusterIP
selector:
app: front
ports:
- port: 3000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-front-end
namespace: default
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip # using IP routing policy of ALB
alb.ingress.kubernetes.io/subnets: subnet-0c1b964b7812dd512, subnet-021a53c08b0b9865d, subnet-05b587954a63c6d82, subnet-049c715b9991e8390
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-2:675694928506:certificate/86da4222-4ae8-4d42-82f5-68a7f9bbb0f4
alb.ingress.kubernetes.io/ssl-redirect: '443'
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: front-svc # refer to the service defined in deploy.yaml
port:
number: 3000
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: front
name: front-deploy
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: front
template:
metadata:
labels:
app: front
spec:
containers:
- name: front-container
image: 675694928506.dkr.ecr.ap-northeast-2.amazonaws.com/eyestalk-react:v0.2.19
ports:
- containerPort: 3000
restartPolicy: Always
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-config
data:
redis-config: |
maxmemory 20mb
maxmemory-policy allkeys-lru
---
apiVersion: v1
kind: Service
metadata:
name: redis-svc
labels:
app: redis
spec:
selector:
app: redis
ports:
- name: redis
protocol: TCP
port: 6379
targetPort: 6379
---
apiVersion: v1
kind: Pod
metadata:
name: redis
labels:
app: redis
spec:
containers:
- name: redis
image: redis:latest
command:
- redis-server
- "/redis-master/redis.conf"
env:
- name: MASTER
value: "true"
ports:
- containerPort: 6379
name: redis
volumeMounts:
- mountPath: /redis-master-data
name: data
- mountPath: /redis-master
name: config
volumes:
- name: data
emptyDir: {}
- name: config
configMap:
name: redis-config
items:
- key: redis-config
path: redis.conf