1. 오퍼레이터 개발 툴
golang으로 개발할 경우, 두 툴의 사용법은 똑같음. (CLI 이름만 다르다)
Operator SDK
- SDK는 Go로 개발할때 kubebuilder를 기반으로 사용함. (helm, ansible도 가능)
- SDK를 사용하면서 kubebuilder 문서를 참고하는 것이 가능.
- https://sdk.operatorframework.io/docs/faqs/
kubebuilder
2. kubebuilder 설치 과정
- 사용 golang 버전과 호환성 확인 필요 (https://book.kubebuilder.io/quick-start.html)
- 버전 리스트: https://github.com/kubernetes-sigs/kubebuilder/releases
설치 커맨드 (Ubuntu)
- go: v1.16
- kubebuilder: v3.2.0
❯ wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v3.2.0/kubebuilder_linux_amd64
❯ chmod +x kubebuilder && mv kubebuilder /usr/local/bin/
❯ kubebuilder version
Version: main.version{KubeBuilderVersion:"3.2.0", KubernetesVendor:"1.22.1", GitCommit:"b7a730c84495122a14a0faff95e9e9615fffbfc5", BuildDate:"2021-10-29T18:32:16Z", GoOs:"linux", GoArch:"amd64"}gvm - go 버전 관리
- go 버전 여러 개를 사용할 경우, 템플릿 생성 이전에 버전 확인 권장.
- 사용법 - https://goax.tistory.com/7
# golang 버전은 1.16이 권장 (Operator-SDK 경우)
❯ gvm use go1.16
Now using version go1.16
❯ go version
go version go1.16 linux/amd643. 프로젝트 템플릿 생성
# 프로젝트 디렉토리에서 작업
❯ mkdir demo-operator && cd demo-operator
❯ gvm use go1.16
Now using version go1.16
# 템플릿 생성
❯ kubebuilder init demo-operator --repo demo-operator
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.10.0
Update dependencies:
$ go mod tidy
Next: define a resource with:
$ kubebuilder create api
# api 템플릿 생성 (CRD, controller 포함)
❯ kubebuilder create api --group demoapp --version v1 --kind Demo
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/demo_types.go
controllers/demo_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.7.0
go get: added sigs.k8s.io/controller-tools v0.7.0
Desktop/demo-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests4. CRD 템플릿 내용
❯ ls api/v1
demo_types.go groupversion_info.go zz_generated.deepcopy.go
❯ ls controllers
demo_controller.go suite_test.goapis/[버전]/[kind명]_types.go 에서 CRD에 대한 내용을 작성할 수 있음.
// DemoSpec defines the desired state of Demo
type DemoSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
// Foo is an example field of Demo. Edit demo_types.go to remove/update
Foo string `json:"foo,omitempty"`
}[kind명]_controller.go 에서 CR에 대한 컨트롤러 로직을 작성할 수 있음.
package controllers
import (
"context"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/log"
demoappv1 "demo-operator/api/v1"
)
// DemoReconciler reconciles a Demo object
type DemoReconciler struct {
client.Client
Scheme *runtime.Scheme
}
//+kubebuilder:rbac:groups=demoapp.my.domain,resources=demoes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=demoapp.my.domain,resources=demoes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=demoapp.my.domain,resources=demoes/finalizers,verbs=update
// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Demo object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.10.0/pkg/reconcile
func (r *DemoReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// TODO(user): your logic here
return ctrl.Result{}, nil
}
// SetupWithManager sets up the controller with the Manager.
func (r *DemoReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&demoappv1.Demo{}).
Complete(r)
}5. demo 개발 내용
- nginx deployment와 service를 생성
- demo.spec.size 는 deployment의 replicas 값으로 들어감.
- https://github.com/solidcellaMoon/demo-operator
6. k8s 클러스터에 CRD 적용
manifest 생성
demoapp.my.domain_demoes.yaml- config/crd/bases에 생성role.yaml- config/rbac에 생성
# 이건 잘 모르겠음.
❯ make generate
Desktop/demo-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
# CRD, role manifest 생성
❯ make manifests
Desktop/demo-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/basesCRD를 current-context에 적용
❯ make install
Desktop/demo-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
go: downloading sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go: downloading k8s.io/client-go v0.18.10
# 필요한 패키지가 없는 경우 다운받음.
# ...
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
Desktop/demo-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/demoes.demoapp.my.domain created
# demo CRD 생성 확인
❯ k get crds
NAME CREATED AT
demoes.demoapp.my.domain 2022-04-23T07:54:12Z
# 이제 demo 라는 리소스를 조회할 수 있음.
❯ k get demo
No resources found in default namespace.
❯ k explain demo
KIND: Demo
VERSION: demoapp.my.domain/v1
DESCRIPTION:
Demo is the Schema for the demoes API
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string>
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <Object>
Standard object's metadata. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec <Object>
DemoSpec defines the desired state of Demo
status <map[string]>
DemoStatus defines the observed state of Demo7. k8s 클러스터에 Custom Controller 적용
- 테스트 용도: 실행 커맨드로 적용
- 실제 배포: 적용에 필요한 manifest 파일들 구성 필요
- Custom Controller는 Deployment로 적용됨
# 템플릿에서 제공하는 Makefile의 run 사용
❯ make run
# go run 커맨드
❯ go run main.go실행 예시
❯ go run main.go
2022-04-23T17:02:29.736+0900 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2022-04-23T17:02:29.736+0900 INFO setup starting manager
2022-04-23T17:02:29.737+0900 INFO starting metrics server {"path": "/metrics"}
2022-04-23T17:02:29.737+0900 INFO controller.demo Starting EventSource {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo", "source": "kind source: /, Kind="}
2022-04-23T17:02:29.737+0900 INFO controller.demo Starting EventSource {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo", "source": "kind source: /, Kind="}
2022-04-23T17:02:29.737+0900 INFO controller.demo Starting EventSource {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo", "source": "kind source: /, Kind="}
2022-04-23T17:02:29.737+0900 INFO controller.demo Starting Controller {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}
2022-04-23T17:02:29.837+0900 INFO controller.demo Starting workers {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo", "worker count": 1}
# ctrl+C로 종료
2022-04-23T17:07:09.258+0900 INFO controller.demo Shutdown signal received, waiting for all workers to finish {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}
2022-04-23T17:07:09.259+0900 INFO controller.demo All workers finished {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}8. k8s 클러스터에 CR 적용 및 Reconcile 확인
- 프로젝트의
config/samples에서 CR 템플릿을 제공함.
apiVersion: demoapp.my.domain/v1
kind: Demo
metadata:
name: demo-sample
spec:
# TODO(user): Add fields heredemo-sample.yaml
apiVersion: demoapp.my.domain/v1
kind: Demo
metadata:
name: demo-sample
spec:
size: 1CR 적용 확인
❯ k apply -f demo-sample.yaml
demo.demoapp.my.domain/demo-sample created
# 적용된 내용 확인
❯ k get all
NAME READY STATUS RESTARTS AGE
pod/demo-sample-6765b8c7fd-stl6j 1/1 Running 0 4m18s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/demo-sample ClusterIP 10.111.39.79 <none> 80/TCP 4m18s
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 48d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/demo-sample 1/1 1 1 4m18s
NAME DESIRED CURRENT READY AGE
replicaset.apps/demo-sample-6765b8c7fd 1 1 1 4m18s
# docker-desktop 이라 docker ps로도 확인
❯ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
886581f8015f nginx "/docker-entrypoint.…" 3 minutes ago Up 3 minutes k8s_nginx-app_demo-sample-6765b8c7fd-stl6j_default_c7962e2d-2264-4637-8479-b5d62833ebe9_0demo.spec.size=2 로 바꿔서 적용
# demo.spec.size 1 -> 2 로 바꿔서 적용
❯ k apply -f demo-sample.yaml
demo.demoapp.my.domain/demo-sample configured
❯ k get all
NAME READY STATUS RESTARTS AGE
pod/demo-sample-6765b8c7fd-fp9z8 1/1 Running 0 15s
pod/demo-sample-6765b8c7fd-stl6j 1/1 Running 0 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/demo-sample ClusterIP 10.111.39.79 <none> 80/TCP 10m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 48d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/demo-sample 2/2 2 2 10m
NAME DESIRED CURRENT READY AGE
replicaset.apps/demo-sample-6765b8c7fd 2 2 2 10m
❯ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39fe5331594d nginx "/docker-entrypoint.…" 39 seconds ago Up 38 seconds k8s_nginx-app_demo-sample-6765b8c7fd-fp9z8_default_b0312647-7c7c-4142-9c39-f0708b1e921f_0
886581f8015f nginx "/docker-entrypoint.…" 10 minutes ago Up 10 minutes k8s_nginx-app_demo-sample-6765b8c7fd-stl6j_default_c7962e2d-2264-4637-8479-b5d62833ebe9_0리소스(deploy, service) 삭제 테스트
# 1. deploy 삭제
❯ k get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
demo-sample 2/2 2 2 12m
❯ k delete deploy/demo-sample
deployment.apps "demo-sample" deleted
# 삭제후에도 다시 생성됨
❯ k get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
demo-sample 0/2 2 0 5s
# 2. service 삭제
❯ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-sample ClusterIP 10.111.39.79 <none> 80/TCP 14m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 48d
❯ k delete svc/demo-sample
service "demo-sample" deleted
# 삭제후에도 다시 생성됨
❯ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-sample ClusterIP 10.100.135.202 <none> 80/TCP 2s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 48dCR 삭제
- 코드상에 CR 삭제시 리소스들도 다 삭제되도록 작성했었음.
❯ k get demo
NAME AGE
demo-sample 18m
❯ k delete demo/demo-sample
demo.demoapp.my.domain "demo-sample" deleted
❯ k get demo
No resources found in default namespace.
❯ k get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 48d
❯ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESCustom Cotroller 종료 테스트
# run 종료
2022-04-23T17:30:43.818+0900 INFO controller.demo Shutdown signal received, waiting for all workers to finish {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}
2022-04-23T17:30:43.818+0900 INFO controller.demo All workers finished {"reconciler group": "demoapp.my.domain", "reconciler kind": "Demo"}
# 생성했던 cr 내용은 남아있음.
❯ k get all
NAME READY STATUS RESTARTS AGE
pod/demo-sample-6765b8c7fd-9jz9t 1/1 Running 0 2m25s
pod/demo-sample-6765b8c7fd-gzlx8 1/1 Running 0 2m25s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/demo-sample ClusterIP 10.96.158.125 <none> 80/TCP 2m25s
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 48d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/demo-sample 2/2 2 2 2m25s
NAME DESIRED CURRENT READY AGE
replicaset.apps/demo-sample-6765b8c7fd 2 2 2 2m25s
# service 삭제 테스트
❯ k delete svc/demo-sample
service "demo-sample" deleted
# Reconcile X
❯ k get all
NAME READY STATUS RESTARTS AGE
pod/demo-sample-6765b8c7fd-sq6wm 1/1 Running 0 25s
pod/demo-sample-6765b8c7fd-tzprv 1/1 Running 0 25s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 48d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/demo-sample 2/2 2 2 25s
NAME DESIRED CURRENT READY AGE
replicaset.apps/demo-sample-6765b8c7fd 2 2 2 25s9. 기타
cluster-api도 kubebuilder 템플릿 기반으로 개발되었음.
이 블로그의 모든 글은 수제로 짜여져 있습니다...