[kubebuilder] 프로젝트 설정 과정

zzery·2022년 4월 23일
zzerym오퍼레이터

일지(2022~2024)

목록 보기
5/25

작업 위치 지정

# 프로젝트 디렉토리에서 작업
❯ cd demo-operator

# golang 버전은 1.16이 권장 (1.17은 안될 수 있음)
❯ gvm use go1.16
Now using version go1.16
❯ go version
go version go1.16 linux/amd64

init project

❯ kubebuilder init demo-operator --repo demo-operator
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.10.0
Update dependencies:
$ go mod tidy
Next: define a resource with:
$ kubebuilder create api

create api

❯ kubebuilder create api --group demoapp --version v1 --kind Demo
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/demo_types.go
controllers/demo_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.7.0
go get: added sigs.k8s.io/controller-tools v0.7.0
/mnt/c/Users/user/Desktop/demo-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

확인 및 CRD 정의

api 생성 후에 만들어진 디렉토리를 살펴보면

❯ ls api/v1
demo_types.go  groupversion_info.go  zz_generated.deepcopy.go

[kind명]_types.go 에서 CRD에 대한 내용을 작성할 수 있음.

이제 spec을 정의해본다. 기본 생성되는 템플릿은 아래와 같음.

// DemoSpec defines the desired state of Demo
type DemoSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Foo is an example field of Demo. Edit demo_types.go to remove/update
	Foo string `json:"foo,omitempty"`
}

Foo를 지우고 Size 라는 필드를 넣었다.

// DemoSpec defines the desired state of Demo
type DemoSpec struct {
	// Size of Demo
	Size int32 `json:"size"`
}

컨트롤러 개발

controller를 설정한다. 프로젝트의 디렉토리에서 controllers를 살펴보면

❯ ls controllers
demo_controller.go  suite_test.go

[kind명]_types.go 에서 CR에 대한 컨트롤러 로직을 작성할 수 있음.

기본 템플릿은 아래와 같음.

package controllers

import (
	"context"

	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	demoappv1 "demo-operator/api/v1"
)

// DemoReconciler reconciles a Demo object
type DemoReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=demoapp.my.domain,resources=demoes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=demoapp.my.domain,resources=demoes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=demoapp.my.domain,resources=demoes/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Demo object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.10.0/pkg/reconcile
func (r *DemoReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here

	return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *DemoReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&demoappv1.Demo{}).
		Complete(r)
}

여기에서 주로 Reconcile() 함수를 주로 사용한다.

  • 개발 내용은 생각보다 길어서 이 글에서는 생략함.

커스텀 컨트롤러 동작 확인

CRD 정의와 Controller 로직 작성이 끝나면 실제 적용해본다.

❯ make generate
/mnt/c/Users/user/Desktop/demo-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."

bin에 결과물이 생성된다.

❯ make manifests
/mnt/c/Users/user/Desktop/demo-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
  • demoapp.my.domain_demoes.yaml - config/crd/bases에 생성
  • role.yaml - config/rbac에 생성

커스텀 컨트롤러를 current-context에 적용한다.

❯ make install
/mnt/c/Users/user/Desktop/demo-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
go: downloading sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go: downloading k8s.io/client-go v0.18.10
# 필요한 패키지가 없는 경우 다운받음.
# ...
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
/mnt/c/Users/user/Desktop/demo-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/demoes.demoapp.my.domain created

❯ k get crds
NAME                       CREATED AT
demoes.demoapp.my.domain   2022-04-23T07:54:12Z

❯ k get demo
No resources found in default namespace.

❯ k get po
No resources found in default namespace.

에러가 없다면 커스텀 컨트롤러가 클러스터에 적용된다. 이 상태에선 cr이나 파드는 안올라온다.

원래는 없던 demo 라는 리소스가 보이는 것으로 보아 정상 적용됨.

❯ k explain demo
KIND:     Demo
VERSION:  demoapp.my.domain/v1

DESCRIPTION:
     Demo is the Schema for the demoes API

FIELDS:
   apiVersion   <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata     <Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

   spec <Object>
     DemoSpec defines the desired state of Demo

   status       <map[string]>
     DemoStatus defines the observed state of Demo

커스텀 컨트롤러 실행 2

커스텀 컨트롤러 - 오퍼레이터를 실행해서 적용해본다.
지금까지 만든 프로젝트를 run하면 오퍼레이터가 실행된다.
실행 방법은 여러가지가 있는데

# 템플릿에서 제공하는 Makefile의 run 사용
❯ make run

# go run 커맨드
❯ go run main.go

make파일 내용은 이렇다.

.PHONY: run
run: manifests generate fmt vet ## Run a controller from your host.
	go run ./main.go

이번엔 go run으로 실행해본다.

❯ go run main.go
2022-04-23T17:02:29.736+0900    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2022-04-23T17:02:29.736+0900    INFO    setup   starting manager
2022-04-23T17:02:29.737+0900    INFO    starting metrics server {"path": "/metrics"}
2022-04-23T17:02:29.737+0900    INFO    controller.demo Starting EventSource    {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo", "source": "kind source: /, Kind="}
2022-04-23T17:02:29.737+0900    INFO    controller.demo Starting EventSource    {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo", "source": "kind source: /, Kind="}
2022-04-23T17:02:29.737+0900    INFO    controller.demo Starting EventSource    {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo", "source": "kind source: /, Kind="}
2022-04-23T17:02:29.737+0900    INFO    controller.demo Starting Controller     {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}
2022-04-23T17:02:29.837+0900    INFO    controller.demo Starting workers        {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo", "worker count": 1}
# ctrl+C로 종료
2022-04-23T17:07:09.258+0900  INFO    controller.demo Shutdown signal received, waiting for all workers to finish    {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}
2022-04-23T17:07:09.259+0900    INFO    controller.demo All workers finished    {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}

실행은 했는데 이전 결과와 동일하다.
클러스터에 cr부터 적용해야 실제로 리소스가 만들어진다.

config/samples에 CR manifest 템플릿이 있어서 수정해서 적용해주면 된다.

apiVersion: demoapp.my.domain/v1
kind: Demo
metadata:
  name: demo-sample
spec:
  # TODO(user): Add fields here

아래 내용을 배포해본다.

apiVersion: demoapp.my.domain/v1
kind: Demo
metadata:
  name: demo-sample
spec:
  size: 1

배포 결과

❯ k apply -f demo-sample.yaml
demo.demoapp.my.domain/demo-sample created

# 적용된 내용 확인
❯ k get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/demo-sample-6765b8c7fd-stl6j   1/1     Running   0          4m18s

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/demo-sample   ClusterIP   10.111.39.79   <none>        8080/TCP   4m18s
service/kubernetes    ClusterIP   10.96.0.1      <none>        443/TCP    48d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-sample   1/1     1            1           4m18s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-sample-6765b8c7fd   1         1         1       4m18s

# docker-desktop 이라 docker ps로 CR을 통해 nginx가 생성된 것을 확인할 수 있다.
❯ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS     NAMES
886581f8015f   nginx     "/docker-entrypoint.…"   3 minutes ago   Up 3 minutes             k8s_nginx-app_demo-sample-6765b8c7fd-stl6j_default_c7962e2d-2264-4637-8479-b5d62833ebe9_0

Service 객체와 Nginx Deployment를 추상화하여 배포하는 컨트롤러인데,
실수로 Service 객체를 8080 포트로 지정했다. 이건 나중에 수정.

CR의 스펙 값을 변경한다.

apiVersion: demoapp.my.domain/v1
kind: Demo
metadata:
  name: demo-sample
spec:
  size: 2
❯ k apply -f demo-sample.yaml
demo.demoapp.my.domain/demo-sample configured

❯ k get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/demo-sample-6765b8c7fd-fp9z8   1/1     Running   0          15s
pod/demo-sample-6765b8c7fd-stl6j   1/1     Running   0          10m

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/demo-sample   ClusterIP   10.111.39.79   <none>        8080/TCP   10m
service/kubernetes    ClusterIP   10.96.0.1      <none>        443/TCP    48d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-sample   2/2     2            2           10m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-sample-6765b8c7fd   2         2         2       10m

❯ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS     NAMES
39fe5331594d   nginx     "/docker-entrypoint.…"   39 seconds ago   Up 38 seconds             k8s_nginx-app_demo-sample-6765b8c7fd-fp9z8_default_b0312647-7c7c-4142-9c39-f0708b1e921f_0
886581f8015f   nginx     "/docker-entrypoint.…"   10 minutes ago   Up 10 minutes             k8s_nginx-app_demo-sample-6765b8c7fd-stl6j_default_c7962e2d-2264-4637-8479-b5d62833ebe9_0

상태 변화를 확인한다.
Deployment를 삭제.

❯ k get deploy
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
demo-sample   2/2     2            2           12m

❯ k delete deploy/demo-sample
deployment.apps "demo-sample" deleted

# 삭제후에도 다시 생성됨
❯ k get deploy
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
demo-sample   0/2     2            0           5s

# 서비스 삭제
❯ k get svc
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
demo-sample   ClusterIP   10.111.39.79   <none>        8080/TCP   14m
kubernetes    ClusterIP   10.96.0.1      <none>        443/TCP    48d

❯ k delete svc/demo-sample
service "demo-sample" deleted

❯ k get svc
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
demo-sample   ClusterIP   10.100.135.202   <none>        8080/TCP   2s
kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP    48d

# 전체 확인
❯ k get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/demo-sample-5c976ccc47-4kdm4   1/1     Running   0          3m31s
pod/demo-sample-5c976ccc47-rwdsx   1/1     Running   0          3m31s

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/demo-sample   ClusterIP   10.100.135.202   <none>        8080/TCP   102s
service/kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP    48d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-sample   2/2     2            2           3m31s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-sample-5c976ccc47   2         2         2       3m31s

CR 자체를 삭제해보면...

❯ k get demo
NAME          AGE
demo-sample   18m

❯ k delete demo/demo-sample
demo.demoapp.my.domain "demo-sample" deleted

❯ k get demo
No resources found in default namespace.

# 연관된거 다 지워진다.
❯ k get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   48d

❯ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

오퍼레이터 자체를 끈다.

❯ k get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/demo-sample-6765b8c7fd-9jz9t   1/1     Running   0          56s
pod/demo-sample-6765b8c7fd-gzlx8   1/1     Running   0          56s

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/demo-sample   ClusterIP   10.96.158.125   <none>        8080/TCP   56s
service/kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP    48d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-sample   2/2     2            2           56s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-sample-6765b8c7fd   2         2         2       56s

# run 종료
2022-04-23T17:30:43.818+0900  INFO    controller.demo Shutdown signal received, waiting for all workers to finish    {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}
2022-04-23T17:30:43.818+0900    INFO    controller.demo All workers finished    {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}

# 생성했던 cr은 남아있음.
❯ k get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/demo-sample-6765b8c7fd-9jz9t   1/1     Running   0          2m25s
pod/demo-sample-6765b8c7fd-gzlx8   1/1     Running   0          2m25s

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/demo-sample   ClusterIP   10.96.158.125   <none>        8080/TCP   2m25s
service/kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP    48d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-sample   2/2     2            2           2m25s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-sample-6765b8c7fd   2         2         2       2m25s

❯ k delete demo/demo-sample
demo.demoapp.my.domain "demo-sample" deleted

❯ k get demo
No resources found in default namespace.

❯ k get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   48d

여기서 svc만 지우면

❯ k delete svc/demo-sample
service "demo-sample" deleted

❯ k get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/demo-sample-6765b8c7fd-sq6wm   1/1     Running   0          25s
pod/demo-sample-6765b8c7fd-tzprv   1/1     Running   0          25s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   48d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-sample   2/2     2            2           25s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-sample-6765b8c7fd   2         2         2       25s

즉 cr의 구성요소 (여기선 deploy, service) 를 지우면 다시 반영되지 않음.
만약 cr 자체를 지우면 남아있는 것들은 다 같이 삭제되는 듯.

참고 문서

profile
zzery
이 블로그의 모든 글은 수제로 짜여져 있습니다...
이전 포스트

[kubebuilder] install

다음 포스트

[kubebuilder] 사용 과정

0개의 댓글