Kubebuilder
- https://book.kubebuilder.io/
- kubebuilder 아키텍처
- controller-runtime 은 무엇
- kubebuilder 로 Custom Controller 만들어서 올려보기
다음글 : https://velog.io/@sawa1989/Custom-Controller-2
- kubebuilder 로 Custom Controller 만들어서 올려보기
예시 kubeflow의Katib 보면서 실사례 익히기
이제 우리는 무엇을 할수있나?
참고
CRD(Custom Resource Definition)
기존에 있는 Resource에 더해, 사용자가 독자적인 Resource를 정의하고, 클러스터상에서 이용할 수 있음.
Custom Resource Definition(CRD)을 생성하면 Kubernetes에 새로운 사용자 정의 리소스 유형을 등록할 수 있음.
CRD만으로는 해당 리소스에 대해 자동으로 아무런 동작을 수행하지 않음!
CRD를 통해 새로운 리소스 유형을 정의하는 것은 그 리소스를 저장하고 조회할 수 있는 데이터 구조를 만드는 것일 뿐임…
crd를 만들었을때 controller가 없다면 아무런 효능 없는 Object일 뿐이다…
그래서 관리하는 것을 만들어야겠지? ⇒ controller!!!
yusa@YUSAui-MacBookAir Everything % kubectl get crd
NAME CREATED AT
alertmanagerconfigs.monitoring.coreos.com 2024-06-15T08:23:32Z
certificaterequests.cert-manager.io 2024-06-15T08:23:25Z
certificates.cert-manager.io 2024-06-15T08:23:25Z
challenges.acme.cert-manager.io 2024-06-15T08:23:25Z
clusterissuers.cert-manager.io 2024-06-15T08:23:25Z
cronjobs.batch.tutorial.kubebuilder.io 2024-06-15T08:18:33Z
issuers.cert-manager.io 2024-06-15T08:23:25Z
orders.acme.cert-manager.io 2024-06-15T08:23:25Z
podmonitors.monitoring.coreos.com 2024-06-15T08:23:32Z
probes.monitoring.coreos.com 2024-06-15T08:23:32Z
prometheusrules.monitoring.coreos.com 2024-06-15T08:23:32Z
scrapeconfigs.monitoring.coreos.com 2024-06-15T08:23:32Z
servicemonitors.monitoring.coreos.com 2024-06-15T08:23:32Z- CRD는 데이터 구조 정의
-
CRD는 새로운 리소스 유형을 정의하여 Kubernetes API에 등록
→ 해당 리소스를 저장하고 조회할 수 있게됨.
-
- 컨트롤러는 동작 정의:
-
컨트롤러는 CRD로 정의된 리소스에 대한 상태 관리 및 자동화를 수행
-
리소스 상태 관리 ( desired state와 actual state 간의 차이를 조정해, 일관성있는 상태 유지 )
-
자동화 작업 수행
-
로직 처리 ( 리소스 의존성 관리, 고가용성 보장, 자동 복구 )
-
k8s 기능 확장
→ 리소스의 생성, 업데이트, 삭제 등의 이벤트에 반응하고 필요한 동작을 자동으로 수행
-
Kubebuilder란?
- Kubebuilder는 Kubernetes에서 컨트롤러와 오퍼레이터를 만들기 위한 도구로, 프로젝트의 초기 설정 및 API 정의 작성을 쉽게하며 개발자가 구현에 집중할수 있게 해줌
- Kubebuilder는 프로젝트와 API 정의를 생성하기 위한 스캐폴딩 기능을 제공하여 개발자가 오퍼레이터 및 컨트롤러를 빠르게 시작할 수 있도록 도와주는 역할을 함.
- 스캐폴딩 이란 ?
-
소프트웨어 개발에서 일반적으로 사용되는 용어로, 기본적인 구조를 제공하고 초기 설정을 자동으로 생성하는 도구나 프로세스를 말함.
-
Kubebuilder의 스캐폴딩 기능은 새로운 프로젝트를 시작할 때 초기 디렉터리 구조와 필요한 파일을 자동으로 생성함.
⇒ 개발자가 프로젝트 설정 및 초기화에 시간을 소비하지 않고 빠르게 작업을 시작할 수 있도록 도와준다는 것!
-
Controller-runtime
- Kubernetes의 컨트롤러를 개발하기 위한 라이브러리
- k8s-sig-api-machinery ( Kubernetes의 API Machinery 관련 작업을 담당하는 Kubernetes 공식 커뮤니티 그룹 담당 )
- Used by Kubebuilder and Operator SDK
- k8s CRD와 Kubernetes API를 모두 조작
- Kubernetes 스타일의 컨트롤러 구축을 위한 툴 제공
- Kubernetes API를 구현하고 있으며 Operator, Workload API, Configuration API, Autoscalers 등을 구축하기 위한 기반으로 생각하면 이해가 쉬울 것이라고 생각함!
- 각각 블록이 추상화되어 들어가있음
- 종속성을 제공, 일반적인 controller pattern이 encapsulate되어있음
⇒ controller-runtime을 이용함으로써, k8s 특유의 component를 의식하여 구현할 필요가 없게됨!
예시 소스 분석
https://pkg.go.dev/sigs.k8s.io/controller-runtime#example-package
- Manager:
- 애플리케이션의 라이프 사이클를 관리하고, 여러 컨트롤러를 실행하는 데 사용됨
Manager는 여러 컨트롤러와 웹훅 서버를 실행할 수 있는 실행 환경을 제공함.- 컨트롤러와 웹훅을 실행하고, 공통 의존성(공유 캐시와 클라이언트 등)을 설정하며 리더 선출을 관리
- SetupSignalHandler 예시
if err := manager.Start(ctrl.SetupSignalHandler()); err != nil { log.Error(err, "could not start manager") os.Exit(1) }
- Controller:
- 특정 Kubernetes 리소스를 감시하고 이벤트를 처리
- 이벤트를 사용하여 reconcile 요청을 트리거
- 컨트롤러는 특정 리소스의 생성, 업데이트, 삭제 이벤트를 감지하여, 해당 리소스의 상태를 원하는 상태로 맞춰줌.
- NewControllerManagedBy 예시
err = ctrl. NewControllerManagedBy(manager). // Create the Controller For(&appsv1.ReplicaSet{}). // ReplicaSet is the Application API Owns(&corev1.Pod{}). // ReplicaSet owns Pods created by it Complete(&ReplicaSetReconciler{Client: manager.GetClient()}) if err != nil { log.Error(err, "could not create controller") os.Exit(1) }
- Reconciler:
- 리소스의 현재 상태를 원하는 상태로 맞추기 위해 실행되는 핵심 로직을 포함
Reconciler는 리소스가 변경될 때마다 호출되어, 필요한 수정 작업을 수행- 리소스를 reconcile하는 로직을 구현
- func (a *ReplicaSetReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)
- Reconcile 함수 예시
func (a *ReplicaSetReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { rs := &appsv1.ReplicaSet{} err := a.Get(ctx, req.NamespacedName, rs) if err != nil { return ctrl.Result{}, err } pods := &corev1.PodList{} err = a.List(ctx, pods, client.InNamespace(req.Namespace), client.MatchingLabels(rs.Spec.Template.Labels)) if err != nil { return ctrl.Result{}, err } rs.Labels["pod-count"] = fmt.Sprintf("%v", len(pods.Items)) err = a.Update(ctx, rs) if err != nil { return ctrl.Result{}, err } return ctrl.Result{}, nil }
- Client / Cache
- Kubernetes API 서버와 상호 작용하여 리소스를 생성, 업데이트, 삭제하는 데 사용
- 캐시는 감시된 객체와 요청된 객체를 자동으로 설정해, 객체를 빠르게 조회 가능
Client는Get,List,Create,Update,Delete등의 메서드를 제공- Get 예시
rs := &appsv1.ReplicaSet{} err := a.Get(ctx, req.NamespacedName, rs) if err != nil { return ctrl.Result{}, err }
- Scheme:
- Go 타입과 Kubernetes API Kind (Group-Version-Kind)에 매핑하는 데 사용
Scheme은 컨트롤러가 여러 종류의 리소스를 이해하고 처리할 수 있도록 도와줌
kubebuilder
아키텍처
https://book.kubebuilder.io/architecture
- Process : 클러스터당 하나씩 존재하거나, HA를 사용하는 경우 여러개를 만들수있음
- Manager : 프로레스당 하나씩 만들수있음.
- HA(리더선출)처리, 메트릭 내보내기, 웹훅 인증서처리, 캐시 이벤트, 컨트롤러들에게 이벤트를 브로드캐스트, 시그널 핸들링, 종료 처리
- client : api 서버와 통신하여, 인증 및 프로트콜을 핸들링한다.
- cache : 최근에 watch, get한 오브젝트를 보관하며, 컨트롤러와 웹훅에서 사용됨, client들을 사용함
- controller : reconscile되는 kind(Kubernetes 리소스의 유형)당 하나씩 존재함 ( crd당 하나씩 있다고 생각하면 됨 )
- 생성한 리소스를 소유함
- filter를 통해 이벤트를 가져오고, 캐시와 클라이언트를 사용함
- 이벤트를 받을때마다 reconciler를 호출함
- back-off 처리 및 이벤트의 queue와 re-queue를 처리함
- Predicate : 이벤트 스트림을 필터링하여, reconciler에 처리가 필요한 이벤트만 넘겨준다.
- Reconciler : 사용자가 제공한 로직이 reconciler.Reconcile 함수에 추가됨
- Webhook : 웹훅이 0 개 또는 1개라고 보면 됨, reconcile되는 kind( Kubernetes 리소스의 유형)당 1개임
- Defaulter : spec에서 설정되지 않는 필드는 여기서 설정됨
- Validator : 잘못 설정된 오브젝트를 거부함.
kubebuilder 로 Custom Controller 만들어서 올려보기
Kubebuilder 설치
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/
kubebuilder version
Version: main.version{KubeBuilderVersion:"4.0.0", KubernetesVendor:"1.27.1", GitCommit:"6c08ed1db5804042509a360edd971ebdc4ae04d8", BuildDate:"2024-05-24T08:36:23Z", GoOs:"darwin", GoArch:"arm64"}minikube 실행
minikube start --driver=docker- 실행 명령어 결과
# 실행 minikube start --driver=docker 😄 minikube v1.33.1 on Darwin 14.5 (arm64) ✨ Using the docker driver based on user configuration 📌 Using Docker Desktop driver with root privileges 👍 Starting "minikube" primary control-plane node in "minikube" cluster 🚜 Pulling base image v0.0.44 ... 🔥 Creating docker container (CPUs=2, Memory=4000MB) ... 🐳 Preparing Kubernetes v1.30.0 on Docker 26.1.1 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default # 확인 kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-7db6d8ff4d-g8z8c 1/1 Running 0 4s kube-system etcd-minikube 1/1 Running 0 18s kube-system kube-apiserver-minikube 1/1 Running 0 19s kube-system kube-controller-manager-minikube 1/1 Running 0 18s kube-system kube-proxy-7n95n 1/1 Running 0 4s kube-system kube-scheduler-minikube 1/1 Running 0 18s kube-system storage-provisioner 1/1 Running 0 17s
설치 확인을 위한 Quick start ( 클러스터 내부에서 테스트 )
# 디렉토리 생성 ( 최대한 quick start와 동일하게 작성하였음. )
mkdir -p ~/projects/guestbook
cd ~/projects/guestbook
# 프로젝트 만들기
kubebuilder init --domain my.domain --repo my.domain/guestbook
# API 만들기 API(webapp/v1), CRD(Guestbook)
kubebuilder create api --group webapp --version v1 --kind Guestbook
# TEST 하기
make install
# Install Instances of Custom Resources
kubectl apply -k config/samples/
# build, push 하기
make docker-build docker-push IMG=goyo9815/server1:guestbook
# deploy 하기
make deploy IMG=goyo9815/server1:guestbook
# crd 확인
kubectl get guestbook
NAME AGE
guestbook-sample 9m11s
kubectl get pods -n guestbook-system
NAME READY STATUS RESTARTS AGE
guestbook-controller-manager-6fb8dbd9b5-r5f6g 1/1 Running 0 78s
# uninstall, undeploy
make uninstall
make undeploy- 실행 명령어 결과
mkdir -p ~/projects/guestbook cd ~/projects/guestbook # 프로젝트 만들기 kubebuilder init --domain my.domain --repo my.domain/guestbook INFO Writing kustomize manifests for you to edit... INFO Writing scaffold for you to edit... INFO Get controller runtime: $ go get sigs.k8s.io/controller-runtime@v0.18.2 go: downloading sigs.k8s.io/controller-runtime v0.18.2 go: downloading k8s.io/apimachinery v0.30.0 go: downloading k8s.io/client-go v0.30.0 go: downloading k8s.io/api v0.30.0 go: downloading github.com/evanphx/json-patch/v5 v5.9.0 go: downloading k8s.io/apiextensions-apiserver v0.30.0 go: downloading gomodules.xyz/jsonpatch/v2 v2.4.0 go: downloading golang.org/x/term v0.18.0 go: downloading golang.org/x/net v0.23.0 go: downloading github.com/evanphx/json-patch v4.12.0+incompatible go: downloading golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e go: downloading github.com/prometheus/client_golang v1.16.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading golang.org/x/sys v0.18.0 go: downloading golang.org/x/oauth2 v0.12.0 go: downloading github.com/google/uuid v1.3.0 go: downloading github.com/fsnotify/fsnotify v1.7.0 go: downloading github.com/google/go-cmp v0.6.0 go: downloading sigs.k8s.io/yaml v1.3.0 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading github.com/prometheus/client_model v0.4.0 go: downloading github.com/prometheus/common v0.44.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.2.0 go: downloading github.com/prometheus/procfs v0.12.0 go: downloading golang.org/x/text v0.14.0 go: downloading google.golang.org/appengine v1.6.7 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4 go: upgraded go 1.22 => 1.22.0 go: added toolchain go1.22.3 INFO Update dependencies: $ go mod tidy go: downloading github.com/onsi/ginkgo/v2 v2.17.1 go: downloading github.com/onsi/gomega v1.32.0 go: downloading github.com/stretchr/testify v1.8.4 go: downloading github.com/go-logr/zapr v1.3.0 go: downloading go.uber.org/zap v1.26.0 go: downloading github.com/pmezard/go-difflib v1.0.0 go: downloading go.uber.org/goleak v1.3.0 go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c go: downloading github.com/kr/pretty v0.3.1 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.10.0 go: downloading github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 go: downloading golang.org/x/tools v0.18.0 go: downloading github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 Next: define a resource with: $ kubebuilder create api # API 만들기 API(webapp/v1), CRD(Guestbook) kubebuilder create api --group webapp --version v1 --kind Guestbook INFO Create Resource [y/n] y INFO Create Controller [y/n] y INFO Writing kustomize manifests for you to edit... INFO Writing scaffold for you to edit... INFO api/v1/guestbook_types.go INFO api/v1/groupversion_info.go INFO internal/controller/suite_test.go INFO internal/controller/guestbook_controller.go INFO internal/controller/guestbook_controller_test.go INFO Update dependencies: $ go mod tidy INFO Running make: $ make generate mkdir -p /Users/yusa/projects/guestbook/bin Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.15.0 go: downloading sigs.k8s.io/controller-tools v0.15.0 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/gobuffalo/flect v1.0.2 go: downloading golang.org/x/net v0.24.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading golang.org/x/sys v0.19.0 go: downloading golang.org/x/sync v0.7.0 go: downloading golang.org/x/mod v0.17.0 /Users/yusa/projects/guestbook/bin/controller-gen-v0.15.0 object:headerFile="hack/boilerplate.go.txt" paths="./..." Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with: $ make manifests # TEST 하기 make install /Users/yusa/projects/guestbook/bin/controller-gen-v0.15.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases Downloading sigs.k8s.io/kustomize/kustomize/v5@v5.4.1 go: downloading sigs.k8s.io/kustomize/kustomize/v5 v5.4.1 go: downloading sigs.k8s.io/kustomize/cmd/config v0.14.0 go: downloading sigs.k8s.io/kustomize/api v0.17.1 go: downloading sigs.k8s.io/kustomize/kyaml v0.17.0 go: downloading github.com/go-errors/errors v1.4.2 go: downloading k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading github.com/xlab/treeprint v1.2.0 go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 go: downloading gopkg.in/evanphx/json-patch.v4 v4.12.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5 go: downloading github.com/golang/protobuf v1.5.3 /Users/yusa/projects/guestbook/bin/kustomize-v5.4.1 build config/crd | kubectl apply -f - customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.my.domain created # Install Instances of Custom Resources kubectl apply -k config/samples/ guestbook.webapp.my.domain/guestbook-sample created # make docker-build docker-push IMG=goyo9815/server1:guestbook make docker-build docker-push IMG=goyo9815/server1:guestbook docker build -t goyo9815/server1:guestbook . [+] Building 60.6s (18/18) FINISHED docker:desktop-linux => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.32kB 0.0s => [internal] load metadata for gcr.io/distroless/static:nonroot 2.5s => [internal] load metadata for docker.io/library/golang:1.22 3.4s => [auth] library/golang:pull token for registry-1.docker.io 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 160B 0.0s => [builder 1/9] FROM docker.io/library/golang:1.22@sha256:969349b8121a56d51c74f4c273ab974c15b3a8ae246a5cffc1df7d28b66cf978 31.9s => => resolve docker.io/library/golang:1.22@sha256:969349b8121a56d51c74f4c273ab974c15b3a8ae246a5cffc1df7d28b66cf978 0.0s => => sha256:969349b8121a56d51c74f4c273ab974c15b3a8ae246a5cffc1df7d28b66cf978 9.74kB / 9.74kB 0.0s => => sha256:30084394d0cf649a54efbfe4973e32888d59a4a9d7357ec0fd61c651ab1e22b6 2.32kB / 2.32kB 0.0s => => sha256:bcbc8ae385b43681b3fe25456ff7c6bd6a2dc999c3c9aece2d91463e6412a361 2.88kB / 2.88kB 0.0s => => sha256:91e301773f03e9e0fabc5c177fe6bfe8daf14e992ab816f151692b814ddc462c 49.61MB / 49.61MB 15.2s => => sha256:15856ca26414127b85cee6d10acbc4cee6eba9070f3f5a04b9cc72ce95abfa7f 23.59MB / 23.59MB 9.7s => => sha256:30ed4c12791345d3f20f66024e1f22275ce507868c508509b83dcf231b1c9adc 63.99MB / 63.99MB 14.7s => => sha256:5059c70b90bd7b288bae321bd0a56c1b79429fdb9fffe0b60e1976ec8002ee17 86.25MB / 86.25MB 27.4s => => sha256:45c0d35bf3e901b1024e7abc65a8b1332e085c592b5bdd1d9422364771935e1e 66.27MB / 66.27MB 26.8s => => extracting sha256:91e301773f03e9e0fabc5c177fe6bfe8daf14e992ab816f151692b814ddc462c 1.4s => => sha256:8b44748e16fdad7adca70f49e9b2aecbeb891450c0d849b42ce3757d7b374873 126B / 126B 15.5s => => sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B 15.7s => => extracting sha256:15856ca26414127b85cee6d10acbc4cee6eba9070f3f5a04b9cc72ce95abfa7f 0.4s => => extracting sha256:30ed4c12791345d3f20f66024e1f22275ce507868c508509b83dcf231b1c9adc 1.6s => => extracting sha256:5059c70b90bd7b288bae321bd0a56c1b79429fdb9fffe0b60e1976ec8002ee17 1.4s => => extracting sha256:45c0d35bf3e901b1024e7abc65a8b1332e085c592b5bdd1d9422364771935e1e 2.8s => => extracting sha256:8b44748e16fdad7adca70f49e9b2aecbeb891450c0d849b42ce3757d7b374873 0.0s => => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.0s => [internal] load build context 0.0s => => transferring context: 41.01kB 0.0s => [stage-1 1/3] FROM gcr.io/distroless/static:nonroot@sha256:e9ac71e2b8e279a8372741b7a0293afda17650d926900233ec3a7b2b7c22a246 1.7s => => resolve gcr.io/distroless/static:nonroot@sha256:e9ac71e2b8e279a8372741b7a0293afda17650d926900233ec3a7b2b7c22a246 0.0s => => sha256:47721f497d85ba1fd88e1d4e401aba95939de67d9e5c1ff19c044ffce442639f 1.95kB / 1.95kB 0.0s => => sha256:d12dd1747fc756df543a17efb415b43b1aaac8074fe01b91c3d32c76946b6a49 104.18kB / 104.18kB 0.4s => => sha256:8b7cf39414e22f1d988a99bc61cb3a0f0cd1d0fbb7129c3c9002b090313f0a6b 1.52kB / 1.52kB 0.0s => => sha256:058cf3d8c2ba04ad7c064698c08c5e886a8623c0ad6171b8d72684253534417d 537.71kB / 537.71kB 0.7s => => sha256:e9ac71e2b8e279a8372741b7a0293afda17650d926900233ec3a7b2b7c22a246 1.51kB / 1.51kB 0.0s => => sha256:e8d9a567199d7a318c875f2558a679ba8a924f817afacbb428afc3ffe6be6828 13.37kB / 13.37kB 0.5s => => extracting sha256:d12dd1747fc756df543a17efb415b43b1aaac8074fe01b91c3d32c76946b6a49 0.0s => => sha256:b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71 67B / 67B 0.9s => => extracting sha256:e8d9a567199d7a318c875f2558a679ba8a924f817afacbb428afc3ffe6be6828 0.0s => => sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212 188B / 188B 0.9s => => extracting sha256:058cf3d8c2ba04ad7c064698c08c5e886a8623c0ad6171b8d72684253534417d 0.1s => => sha256:33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8 122B / 122B 1.0s => => extracting sha256:b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71 0.0s => => extracting sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212 0.0s => => sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd 168B / 168B 1.2s => => sha256:27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3 93B / 93B 1.2s => => extracting sha256:33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8 0.0s => => sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f 385B / 385B 1.4s => => extracting sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd 0.0s => => extracting sha256:27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3 0.0s => => sha256:9aee425378d2c16cd44177dc54a274b312897f5860a8e78fdfda555a0d79dd71 130.50kB / 130.50kB 1.6s => => sha256:da7816fa955ea24533c388143c78804c28682eef99b4ee3723b548c70148bba6 321B / 321B 1.6s => => extracting sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f 0.0s => => extracting sha256:da7816fa955ea24533c388143c78804c28682eef99b4ee3723b548c70148bba6 0.0s => => extracting sha256:9aee425378d2c16cd44177dc54a274b312897f5860a8e78fdfda555a0d79dd71 0.0s => [builder 2/9] WORKDIR /workspace 0.4s => [builder 3/9] COPY go.mod go.mod 0.0s => [builder 4/9] COPY go.sum go.sum 0.0s => [builder 5/9] RUN go mod download 8.0s => [builder 6/9] COPY cmd/main.go cmd/main.go 0.0s => [builder 7/9] COPY api/ api/ 0.0s => [builder 8/9] COPY internal/controller/ internal/controller/ 0.0s => [builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -o manager cmd/main.go 16.6s => [stage-1 2/3] COPY --from=builder /workspace/manager . 0.1s => exporting to image 0.1s => => exporting layers 0.1s => => writing image sha256:ab00cc2e599a5c592a63d3ef87c5d0ff9e3967453fa0fae5414ac1e8b37aad1a 0.0s => => naming to docker.io/goyo9815/server1:guestbook 0.0s View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/84cocer8yl4ml22bhgeo0o5ob docker push goyo9815/server1:guestbook The push refers to repository [docker.io/goyo9815/server1] 43f9e78247a9: Pushed b336e209998f: Pushed f4aee9e53c42: Pushed 1a73b54f556b: Pushed 2a92d6ac9e4f: Pushed bbb6cacb8c82: Pushed ac805962e479: Pushed af5aa97ebe6c: Pushed 4d049f83d9cf: Pushed 945d17be9a3e: Pushed 49626df344c9: Pushed 5f935a4f5ae5: Pushed guestbook: digest: sha256:92f22c1adf57a89ccd80201484a4b30dd2fe00580175ab76d44ef74383867fa9 size: 2814 # deploy하기 make deploy IMG=goyo9815/server1:guestbook /Users/yusa/projects/guestbook/bin/controller-gen-v0.15.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases cd config/manager && /Users/yusa/projects/guestbook/bin/kustomize-v5.4.1 edit set image controller=goyo9815/server1:guestbook /Users/yusa/projects/guestbook/bin/kustomize-v5.4.1 build config/default | kubectl apply -f - namespace/guestbook-system created customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.my.domain unchanged serviceaccount/guestbook-controller-manager created role.rbac.authorization.k8s.io/guestbook-leader-election-role created clusterrole.rbac.authorization.k8s.io/guestbook-guestbook-editor-role created clusterrole.rbac.authorization.k8s.io/guestbook-guestbook-viewer-role created clusterrole.rbac.authorization.k8s.io/guestbook-manager-role created rolebinding.rbac.authorization.k8s.io/guestbook-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/guestbook-manager-rolebinding created deployment.apps/guestbook-controller-manager created # crd 확인 kubectl get guestbook NAME AGE guestbook-sample 9m11s # uninstall make uninstall /Users/yusa/projects/guestbook/bin/controller-gen-v0.15.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases /Users/yusa/projects/guestbook/bin/kustomize-v5.4.1 build config/crd | kubectl delete --ignore-not-found=false -f - customresourcedefinition.apiextensions.k8s.io "guestbooks.webapp.my.domain" deleted # undeploy make undeploy /Users/yusa/projects/guestbook/bin/kustomize-v5.4.1 build config/default | kubectl delete --ignore-not-found=false -f - namespace "guestbook-system" deleted serviceaccount "guestbook-controller-manager" deleted role.rbac.authorization.k8s.io "guestbook-leader-election-role" deleted clusterrole.rbac.authorization.k8s.io "guestbook-guestbook-editor-role" deleted clusterrole.rbac.authorization.k8s.io "guestbook-guestbook-viewer-role" deleted clusterrole.rbac.authorization.k8s.io "guestbook-manager-role" deleted rolebinding.rbac.authorization.k8s.io "guestbook-leader-election-rolebinding" deleted clusterrolebinding.rbac.authorization.k8s.io "guestbook-manager-rolebinding" deleted deployment.apps "guestbook-controller-manager" deleted
참고 : 밖에서 테스트할때는 make run
Project 분석
- tree
% tree . ├── Dockerfile ├── Makefile ├── PROJECT ├── README.md ├── api │ └── v1 │ ├── groupversion_info.go │ ├── guestbook_types.go │ └── zz_generated.deepcopy.go ├── bin │ ├── controller-gen-v0.15.0 │ └── kustomize-v5.4.1 ├── cmd │ └── main.go ├── config │ ├── crd │ │ ├── bases │ │ │ └── webapp.my.domain_guestbooks.yaml │ │ ├── kustomization.yaml │ │ └── kustomizeconfig.yaml │ ├── default │ │ ├── kustomization.yaml │ │ ├── manager_metrics_patch.yaml │ │ └── metrics_service.yaml │ ├── manager │ │ ├── kustomization.yaml │ │ └── manager.yaml │ ├── prometheus │ │ ├── kustomization.yaml │ │ └── monitor.yaml │ ├── rbac │ │ ├── guestbook_editor_role.yaml │ │ ├── guestbook_viewer_role.yaml │ │ ├── kustomization.yaml │ │ ├── leader_election_role.yaml │ │ ├── leader_election_role_binding.yaml │ │ ├── role.yaml │ │ ├── role_binding.yaml │ │ └── service_account.yaml │ └── samples │ ├── kustomization.yaml │ └── webapp_v1_guestbook.yaml ├── go.mod ├── go.sum ├── hack │ └── boilerplate.go.txt ├── internal │ └── controller │ ├── guestbook_controller.go │ ├── guestbook_controller_test.go │ └── suite_test.go └── test ├── e2e │ ├── e2e_suite_test.go │ └── e2e_test.go └── utils └── utils.go 19 directories, 39 files api/: CRD에 해당 하는 type, crd 메니페스트가 해당 폴더내용으로 생성됨, 생성되는 템플릿을 바탕으로 정의하고 싶은 spec, status를 넣음-
struct를 보면 우리가 항상 yaml에서 보던 spec, status를 볼수있음.
/* Copyright 2024. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1 import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) // EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN! // NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized. // GuestbookSpec defines the desired state of Guestbook type GuestbookSpec struct { // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster // Important: Run "make" to regenerate code after modifying this file // Foo is an example field of Guestbook. Edit guestbook_types.go to remove/update Foo string `json:"foo,omitempty"` } // GuestbookStatus defines the observed state of Guestbook type GuestbookStatus struct { // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster // Important: Run "make" to regenerate code after modifying this file } // +kubebuilder:object:root=true // +kubebuilder:subresource:status // Guestbook is the Schema for the guestbooks API type Guestbook struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec GuestbookSpec `json:"spec,omitempty"` Status GuestbookStatus `json:"status,omitempty"` } // +kubebuilder:object:root=true // GuestbookList contains a list of Guestbook type GuestbookList struct { metav1.TypeMeta `json:",inline"` metav1.ListMeta `json:"metadata,omitempty"` Items []Guestbook `json:"items"` } func init() { SchemeBuilder.Register(&Guestbook{}, &GuestbookList{}) }
-
config/samples: Kubernetes Custom Resource(CR)의 샘플 YAML 파일, 사용자 정의 리소스파일의 예시를 제공함.``` apiVersion: webapp.my.domain/v1 kind: Guestbook metadata: labels: app.kubernetes.io/name: guestbook app.kubernetes.io/managed-by: kustomize name: guestbook-sample spec: # TODO(user): Add fields here ```internal/controller/: Reconcile 함수에 로직이 들어감, 생성되는 템플릿을 이용해서 controller-runtime이 구현됨.- 참고 : Kubernetes 컨트롤러를 개발할 때 리소스에 대한 RBAC(Role-Based Access Control) 권한을 설정하기 위한 어노테이션도 같이 들어가있는데, 만약 다른 rbac권한이 필요하다면 여기에 추가하거나 제거해줘야함.
- Reconcile 예시로 하나 설명해하자면…
func (r *GuestbookReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// TODO(user): your logic here
return ctrl.Result{}, nil
}
context와 reconcile.request를 가지는데, reconcile을 자세히보게되면
이러한 리소스에 어떤 이벤트가 있을때 어떻게 할지 클러스터와 연결해준다고 보면됨.
type Request struct {
types.NamespacedName
}
type NamespacedName struct {
Namespace string
Name string
}
그래서 결과적으로 자기가 하고싶은 일을 reconcile에 넣게된다는 것
func (r *GuestbookReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// 예시입니당..
// 빈 pod 객체 만들기
pod := &corev1.Pod{}
// 네임스페이스와 이름을 사용해서 pod객체를 가져옴
if err := r.Get(ctx, req.NamespacedName, pod); err != nil{
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// Pod 객체 출력
fmt.Printf("Found Pod: %s\n", pod.Name)
// 종료
return ctrl.Result{}, nil
}
여기서 r.Get(ctx, req.NamespacedName, pod) 는 인터페이스로 client에 있는 get을 사용하는 것
client.client 인터페이스로 정의되는데, Kubernetes 리소스를 생성, 읽기, 업데이트, 삭제하는 메서드를 제공
Get(ctx context.Context, key client.ObjectKey, obj client.Object) error
List(ctx context.Context, list client.ObjectList, opts ...client.ListOption) error
Create(ctx context.Context, obj client.Object, opts ...client.CreateOption) error
Update(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error
Delete(ctx context.Context, obj client.Object, opts ...client.DeleteOption) error
go.mod: 기본 종속성이 있는 프로젝트와 일치하는 새로운 Go 모듈Makefile: 컨트롤러 빌드 및 배포를 위한 대상 만들기PROJECT: 새 구성 요소를 스캐폴딩하기 위한 Kubebuilder 메타데이터- api와 컨트롤러 확인 가능
domain: my.domain layout: - go.kubebuilder.io/v4 projectName: guestbook repo: my.domain/guestbook resources: - api: crdVersion: v1 namespaced: true controller: true domain: my.domain group: webapp kind: Guestbook <- 새로운 리소스 유형을 볼수있음. path: my.domain/guestbook/api/v1 version: v1 version: "3" config/default: 표준 구성에서 컨트롤러를 시작하기 위한 Kustomize 기반이 포함.- kustomize란? ( 블로그 링크로 대체 ) "Differenet varitions Of YAML" https://velog.io/@sawa1989/kustomize
- kustomize란? ( 블로그 링크로 대체 ) "Differenet varitions Of YAML" https://velog.io/@sawa1989/kustomize
config/manager: 컨트롤러를 클러스터의 포드로 시작 ( Makefile에 있는 image와 동일함 )config/rbac: 자체 서비스 계정으로 컨트롤러를 실행하는 데 필요한 권한cmd/main.go: 컨트롤러를 실행하는 메인 프로세스
controller-runtime에서 확인한 모든 요소들을 확인할수있음
client, scheme, cache 밑에 코드를 확인하면 볼수있다.
var (
scheme = runtime.NewScheme()
setupLog = ctrl.Log.WithName("setup")
)
func init() {
utilruntime.Must(clientgoscheme.AddToScheme(scheme))
utilruntime.Must(webappv1.AddToScheme(scheme))
// +kubebuilder:scaffold:scheme
}
func main() {
var metricsAddr string
var enableLeaderElection bool
var probeAddr string
var secureMetrics bool
var enableHTTP2 bool
flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metric endpoint binds to. "+
"Use the port :8080. If not set, it will be 0 in order to disable the metrics server")
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
flag.BoolVar(&enableLeaderElection, "leader-elect", false,
"Enable leader election for controller manager. "+
"Enabling this will ensure there is only one active controller manager.")
flag.BoolVar(&secureMetrics, "metrics-secure", false,
"If set the metrics endpoint is served securely")
flag.BoolVar(&enableHTTP2, "enable-http2", false,
"If set, HTTP/2 will be enabled for the metrics and webhook servers")
opts := zap.Options{
Development: true,
}
opts.BindFlags(flag.CommandLine)
flag.Parse()
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
// if the enable-http2 flag is false (the default), http/2 should be disabled
// due to its vulnerabilities. More specifically, disabling http/2 will
// prevent from being vulnerable to the HTTP/2 Stream Cancellation and
// Rapid Reset CVEs. For more information see:
// - https://github.com/advisories/GHSA-qppj-fm5r-hxr3
// - https://github.com/advisories/GHSA-4374-p667-p6c8
disableHTTP2 := func(c *tls.Config) {
setupLog.Info("disabling http/2")
c.NextProtos = []string{"http/1.1"}
}
tlsOpts := []func(*tls.Config){}
if !enableHTTP2 {
tlsOpts = append(tlsOpts, disableHTTP2)
}
webhookServer := webhook.NewServer(webhook.Options{
TLSOpts: tlsOpts,
})
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
Metrics: metricsserver.Options{
BindAddress: metricsAddr,
SecureServing: secureMetrics,
TLSOpts: tlsOpts,
},
WebhookServer: webhookServer,
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "ecaf1259.my.domain",
// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
// when the Manager ends. This requires the binary to immediately end when the
// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
// speeds up voluntary leader transitions as the new leader don't have to wait
// LeaseDuration time first.
//
// In the default scaffold provided, the program ends immediately after
// the manager stops, so would be fine to enable this option. However,
// if you are doing or is intended to do any operation such as perform cleanups
// after the manager stops then its usage might be unsafe.
// LeaderElectionReleaseOnCancel: true,
})
if err != nil {
setupLog.Error(err, "unable to start manager")
os.Exit(1)
}
if err = (&controller.GuestbookReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "Guestbook")
os.Exit(1)
}
// +kubebuilder:scaffold:builder
if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
setupLog.Error(err, "unable to set up health check")
os.Exit(1)
}
if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
setupLog.Error(err, "unable to set up ready check")
os.Exit(1)
}
setupLog.Info("starting manager")
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
setupLog.Error(err, "problem running manager")
os.Exit(1)
}
}
참고
https://book.kubebuilder.io/
https://qiita.com/shiei_kawa/items/e745f20040ad5911fcc2
https://youtu.be/QYwTRZjEyK4?si=SMM9r6Tg6FD8o_vK
https://nakamasato.medium.com/kubernetes-operator-series-2-overview-of-controller-runtime-f8454522a539
https://youtu.be/_XUJ1HoinWA?si=_B2qMumEqNL0pNmR
https://youtu.be/CD33-TRYwJc?si=VwapPDQMukyOtqEZ
https://devocean.sk.com/blog/techBoardDetail.do?ID=164260
https://devocean.sk.com/blog/techBoardDetail.do?ID=165032&boardType=techBlog#none
https://nakamasato.medium.com/kubernetes-operator-series-1-controller-runtime-aa50d1d93c5c
Are you nervous? Don't be