[Gitlab] kubernetes 기반 gitlab-runner 서비스 어카운트 관련 에러

helm chart를 통해 kubernetes 기반 gitlab-runner를 설치할 수 있다.

크게 아래에 있는 3가지를 수정하면,

gitlabUrl: ""

runnerToken: ""

rbac:
  create: true
  rules: 
    - resources: ["configmaps", "events", "pods", "pods/attach", "pods/exec", "secrets", "services"]
      verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create", "patch", "delete"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get"]

다음 명령어를 통해 설치할 수 있다.

helm install --namespace <NAMESPACE> <GITLAB RUNNER NAME> -f values.yaml gitlab/gitlab-runner

이렇게 생성한 gitlab-runner는 pod, role, rolebinding, sa가 모두 생성이 된다.

 ~/dev/ml-ops/h/charts │ master !1 kubectl get pod -n gitlab-runner        
NAME                            READY   STATUS    RESTARTS   AGE
gitlab-runner-f55f6ffc5-n7ds2   0/1     Running   0          7s

 ~/dev/ml-ops/h/charts │ master !1 kubectl get role -n gitlab-runner         
NAME            CREATED AT
gitlab-runner   2024-04-03T02:08:22Z

 ~/dev/ml-ops/h/charts │ master !1 kubectl get sa -n gitlab-runner       
NAME            SECRETS   AGE
default         0         2m52s
gitlab-runner   0         19s

 ~/dev/ml-ops/h/charts │ master !1 kubectl get rolebinding -n gitlab-runner 
NAME            ROLE                 AGE
gitlab-runner   Role/gitlab-runner   24s

문제 상황

이렇게만 보면 설치가 다 잘 된 것 같으나, 한 가지 문제가 있다.

바로 kubectl apply ~ script를 실행하는 pipeline에서 다음과 같은 에러가 뜨는 것.

kubectl apply -f nginx-service.yaml
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=pods", GroupVersionKind: "/v1, Kind=Pod"
Name: "nginx-test", Namespace: "gitlab-runner"
from server for: "nginx-service.yaml": pods "nginx-test" is forbidden: User "system:serviceaccount:gitlab-runner:default" cannot get resource "pods" in API group "" in the namespace "gitlab-runner"

왜 인지는 구체적으로 모르겠으나, runner가 gitlab-runner namespace에 생성된 sa를 읽어오지 못하고 default namespace에 존재하는 sa를 읽어온다.

해결 방법

See https://stackoverflow.com/questions/72490488/user-systemserviceaccountgitlabdefault-cannot-get-resource-deployments-in

똑같은 문제를 겪은 사람이 있었다.

아래처럼 runner configuration 부분에 service_account에 대한 정보를 articulate한다.

runners:
  # runner configuration, where the multi line string is evaluated as a
  # template so you can specify helm values inside of it.
  #
  # tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
  # runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "alpine"
        privileged = true 
        service_account = "<GITLAB RUNNER NAME>"

이에 따라서, 정상적으로 gitlab-runner를 통해 pipeline을 실행하려면

values.yaml 파일에서 다음 항목들을 override해야한다.

gitlabUrl: ""

runnerToken: ""

rbac:
  create: true
  rules: 
    - resources: ["configmaps", "events", "pods", "pods/attach", "pods/exec", "secrets", "services"]
      verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create", "patch", "delete"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get"]

runners:
  # runner configuration, where the multi line string is evaluated as a
  # template so you can specify helm values inside of it.
  #
  # tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
  # runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "alpine"
        privileged = true 
        service_account = "<GITLAB RUNNER NAME>"

TEST 결과

다음과 같이 정상적으로 pipeline이 실행되고, test pod도 정상적으로 생성된다.

Running with gitlab-runner 16.9.1 (782c6ecb)
  on gitlab-runner-2-5587d9b79c-j5bv9 HivRSMpwN, system ID: r_SaQDLzeKneIT
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gltest
Using Kubernetes executor with image bitnami/kubectl:latest ...
Using attach strategy to execute scripts...
Preparing environment
00:13
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod gltest/runner-hivrsmpwn-project-373-concurrent-0-j9n06s78 to be running, status is Pending
Waiting for pod gltest/runner-hivrsmpwn-project-373-concurrent-0-j9n06s78 to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gltest/runner-hivrsmpwn-project-373-concurrent-0-j9n06s78 to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gltest/runner-hivrsmpwn-project-373-concurrent-0-j9n06s78 to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-hivrsmpwn-project-373-concurrent-0-j9n06s78 via gitlab-runner-2-5587d9b79c-j5bv9...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/on-premise/simple-nginx/.git/
Created fresh repository.
Checking out 5363df71 as detached HEAD (ref is main)...
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for default-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:01
$ kubectl apply -f nginx-service.yaml
pod/nginx-test created