[파드 디자인 패턴과 잡(Job) 실행] Job을 활용한 파드 생성 개념과 실습

IMKUNYOUNG·2023년 8월 19일
k8s
0

쿠버네티스

목록 보기
53/64

Job과 CronJob의 필요성

  • 쿠버네티스 클러스터를 운영할 때 일정 주기마다 돌아가야 하는 작업들이 존재
  • 넷플릭스가 2013년에 공개한 아키텍처

개인에게 맞춤 추천 시스템을 사용하기에는 너무 많은 리소스가 필요하기 때문에 넷플릭스의 추천 시스템은 실시간 (OnLine) 분석, 준 실시간 (Nearline) 분석, 오프라인 (Offline) 분석으로 나누어서 사용

so 예를 들어, 리소스가 상대적으로 많이 남게 되는 새벽 시간에 리소스가 많이 들어가는 작업들 (모델링이라든지..) 등을 CronJob 형태로 실행할 수 있음.



Job이란?

  • 하나 이상의 파드를 만들고 지정된 수의 파드가 성공적으로 종료될 때까지 Pods 실행을 계속 재시도
  • 작업 (Job)을 삭제하면 생성한 파드가 정리
  • 작업을 일시 중단하면 작업이 다시 재개될 때까지 활성 파드가 삭제
  • 작업을 사용하여 여러 파드를 병렬 실행 가능
# job-pi.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4 # 재시도 횟수 4번으로 제한


잡에 대한 병렬 실행 방법

  • 비병렬 잡: 기본값, 일반적으로 파드가 하나만 실행되고 파드 종료되면 Job이 완료됨
  • 정해진 횟수를 반복하는 잡: .spec.comletions에 0이 아닌 양수를 지정하면 정해진 횟수까지 파드가 반복적 실행
  • 병렬 실행 가능 수를 지정: .spec.parallelism에 0이 아닌 양수를 지정하면 정해진 개수만큼 파드가 동시 실행 가능
# job-pi-parallelism.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: pi-parallelism
spec:
  completions: 5 # 목표 완료 파드 개수
  parallelism: 2 # 동시 실행 가능 파드 개수
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4 # 재시도 횟수 4번으로 제한


실습

Job 실행

# job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4 # 재시도 횟수 4번으로 제한
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl create -f job.yaml
job.batch/pi created
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get job,pod
NAME           COMPLETIONS   DURATION   AGE
job.batch/pi   0/1           22s        22s

NAME                                                READY   STATUS              RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running             0          23h
pod/pi-v2f9k                                        0/1     ContainerCreating   0          22s
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get job,pod
NAME           COMPLETIONS   DURATION   AGE
job.batch/pi   1/1           33s        113s

NAME                                                READY   STATUS      RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running     0          23h
pod/pi-v2f9k                                        0/1     Completed   0          113s

Job 삭제

imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get job
NAME   COMPLETIONS   DURATION   AGE
pi     1/1           33s        4m6s
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl delete job pi
job.batch "pi" deleted
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pods
NAME                                            READY   STATUS    RESTARTS   AGE
nginx-sidecar                                   3/3     Running   0          23h

Job 병렬 실행

# job-pi-parallelism.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: pi-parallelism
spec:
  completions: 5 # 목표 완료 파드 개수
  parallelism: 2 # 동시 실행 가능 파드 개수
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4 # 재시도 횟수 4번으로 제한
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl create -f job-pi-parallelism.yaml
job.batch/pi-parallelism created
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS    RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running   0          23h
pod/pi-parallelism-5v5f6                            1/1     Running   0          5s
pod/pi-parallelism-xjphq                            1/1     Running   0          5s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   0/5           6s         6s



imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS      RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running     0          23h
pod/pi-parallelism-5v5f6                            0/1     Completed   0          15s
pod/pi-parallelism-xjphq                            0/1     Completed   0          15s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   0/5           15s        15s



imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS              RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running             0          23h
pod/pi-parallelism-5v5f6                            0/1     Completed           0          17s
pod/pi-parallelism-vjvdz                            0/1     ContainerCreating   0          1s
pod/pi-parallelism-vt24c                            0/1     ContainerCreating   0          1s
pod/pi-parallelism-xjphq                            0/1     Completed           0          17s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   2/5           17s        17s
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS      RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running     0          23h
pod/pi-parallelism-5v5f6                            0/1     Completed   0          19s
pod/pi-parallelism-vjvdz                            1/1     Running     0          3s
pod/pi-parallelism-vt24c                            1/1     Running     0          3s
pod/pi-parallelism-xjphq                            0/1     Completed   0          19s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   2/5           20s        20s



imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS      RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running     0          23h
pod/pi-parallelism-5v5f6                            0/1     Completed   0          30s
pod/pi-parallelism-vjvdz                            0/1     Completed   0          14s
pod/pi-parallelism-vt24c                            0/1     Completed   0          14s
pod/pi-parallelism-xjphq                            0/1     Completed   0          30s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   2/5           30s        30s



imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS              RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running             0          23h
pod/pi-parallelism-5v5f6                            0/1     Completed           0          32s
pod/pi-parallelism-tn4mc                            0/1     ContainerCreating   0          1s
pod/pi-parallelism-vjvdz                            0/1     Completed           0          16s
pod/pi-parallelism-vt24c                            0/1     Completed           0          16s
pod/pi-parallelism-xjphq                            0/1     Completed           0          32s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   4/5           32s        32s
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS      RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running     0          23h
pod/pi-parallelism-5v5f6                            0/1     Completed   0          34s
pod/pi-parallelism-tn4mc                            1/1     Running     0          3s
pod/pi-parallelism-vjvdz                            0/1     Completed   0          18s
pod/pi-parallelism-vt24c                            0/1     Completed   0          18s
pod/pi-parallelism-xjphq                            0/1     Completed   0          34s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   4/5           34s        34s



imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS      RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running     0          23h
pod/pi-parallelism-5v5f6                            0/1     Completed   0          38s
pod/pi-parallelism-tn4mc                            0/1     Completed   0          7s
pod/pi-parallelism-vjvdz                            0/1     Completed   0          22s
pod/pi-parallelism-vt24c                            0/1     Completed   0          22s
pod/pi-parallelism-xjphq                            0/1     Completed   0          38s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   4/5           39s        39s



imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pod,job
NAME                                                READY   STATUS      RESTARTS   AGE
pod/nginx-sidecar                                   3/3     Running     0          23h
pod/pi-parallelism-5v5f6                            0/1     Completed   0          41s
pod/pi-parallelism-tn4mc                            0/1     Completed   0          10s
pod/pi-parallelism-vjvdz                            0/1     Completed   0          25s
pod/pi-parallelism-vt24c                            0/1     Completed   0          25s
pod/pi-parallelism-xjphq                            0/1     Completed   0          41s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/pi-parallelism   5/5           40s        41s

Job 삭제

imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get job
NAME             COMPLETIONS   DURATION   AGE
pi-parallelism   5/5           40s        9m35s
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl delete job pi-parallelism
job.batch "pi-parallelism" deleted
imkunyoung@cloudshell:~/job (k8s-inflearn)$ kubectl get pods
NAME                                            READY   STATUS    RESTARTS   AGE
nginx-sidecar                                   3/3     Running   0          24h
profile
IMKUNYOUNG
다음 포스트

[파드 디자인 패턴과 잡(Job) 실행] CronJob을 활용한 예약 파드 생성 개념과 실습

0개의 댓글