64 / 120 Kubernetes 고가용성 Kubespray

RungNyeok·2022년 5월 16일
0

클라우드부트캠프

목록 보기
1/2
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 192.168.150.100:6443 --token 2uz27p.pbmn88113lvdwalb \
        --discovery-token-ca-cert-hash sha256:1a0c54c588f6a7ad362732f79b80810afbc07899c2aa1f10e21647db7ebd6110 \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.150.100:6443 --token 2uz27p.pbmn88113lvdwalb \
        --discovery-token-ca-cert-hash sha256:1a0c54c588f6a7ad362732f79b80810afbc07899c2aa1f10e21647db7ebd6110



Worker Node 추가

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
Joining your nodes

추가 순서

  1. Vagranfile 추가 VM 추가
    • cpu:2, mem:2G
  2. Docker 설치
  3. kubeadm, kubectl, kubelet 설치 = 1.22.8
  4. kubeadm join xxx(worker에서 실행)
  5. kubeadm get nodes(master에서 실행)


1. VM 추가

새로운 VM 설정 추가
code .\Vagrantfile

config.vm.define "worknd" do |centos|
		centos.vm.box = "ubuntu/focal64"
		centos.vm.hostname = "worknd"
		centos.vm.network "private_network", ip: "192.168.150.101"
		centos.vm.provider "virtualbox" do |vb|
			vb.name = "worknd"
			vb.cpus = 2
			vb.memory = 2048
		end
	end

간편하게 ssh 접속을 위한 설정
C:\Users\ParkNeunglyeok\.ssh\config

Host worknd
    HostName 192.168.150.101
    User vagrant
    IdentityFile C:\Users\ParkNeunglyeok\vagrant\container\.vagrant\machines\worknd\virtualbox\private_key
# VM 시작
vagrant up worknd

# 접속
ssh worknd


2. Docker 설치

https://docs.docker.com/engine/install/ubuntu/

$ sudo apt update
    
$ sudo apt install ca-certificates curl gnupg lsb-release

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

$ echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  
$ sudo apt update
  
$ sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

$ sudo usermod -aG docker vagrant

재접속



3. kubeadm, kubectl, kubelet 설치 = 1.22.8

$ sudo apt-get update

$ sudo apt-get install -y apt-transport-https ca-certificates curl

$ sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

$ echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

$ sudo apt-get update

$ sudo apt-get install kubeadm=1.22.8-00 kubelet=1.22.8-00 kubectl=1.22.8-00 -y

$ sudo apt-mark hold kubelet kubeadm kubectl

$ sudo vi /etc/docker/daemon.json
~~~~~~~~~~~
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
~~~~~~~~~~~

$ sudo systemctl restart docker

$ sudo systemctl daemon-reload && sudo systemctl restart kubelet


k8s 클러스터 조인


docker VM 에서 진행

토큰 생성
24시간만 유지 된다.

$ kubeadm token create

$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
k8gw7m.iq35pg2jcom10eyx   23h         2022-05-17T00:44:00Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

해쉬값 확인

$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'
1a0c54c588f6a7ad362732f79b80810afbc07899c2aa1f10e21647db7ebd6110

worknd VM 에서 진행

$ kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>

$ sudo kubeadm join --token k8gw7m.iq35pg2jcom10eyx 192.168.150.100:6443 --discovery-token-ca-cert-hash sha256:1a0c54c588f6a7ad362732f79b80810afbc07899c2aa1f10e21647db7ebd6110

docker VM 에서 진행

node가 연결 된 것을 확인 할 수 있다.

$ kubectl get nodes
NAME     STATUS     ROLES                  AGE     VERSION
docker   Ready      control-plane,master   2d18h   v1.22.8
worknd   NotReady   <none>                 21s     v1.22.8




쿠버네티스 고가용성 구성

Options for Highly Available Topology

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/

Stacked etcd topology

고 가용성을 위해
여러개의 control node 와
VM을 하나더 생성하여 LB를 구성하여 연결해줘야 한다.

External etcd topology

ethd 를 별도의 VM 으로 구성

Creating Highly Available Clusters with kubeadm

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
위 아키텍쳐 등을 구성하는 명령어 설명




k8s 클러스터 업그레이드

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

버전 차이(skew) 정책

https://kubernetes.io/releases/version-skew-policy/

  • 고가용성(HA) 클러스터를 위해서 최신 및 가장 오래된 kube-apiserver 인스턴스가 각각 한 단계 마이너 버전 이내에 있어야 한다.
  • kube-controller-manager, kube-scheduler 그리고 cloud-controller-manager는 1.24 과 1.23 을 지원한다.
  • kubelet은 kube-apiserver 2단계의 낮은 마이너 버전까지 지원한다.

업그레이드 순서

  1. kube-apiserver
  2. kube-controller-manager, kube-scheduler, cloud-controller-manager
  3. kubelet ( Control Plane -> Worker Node )
  4. kube-proxy ( Control Plane -> Worker Node )

정리

  1. 기본 컨트롤 플레인 노드를 업그레이드한다.
  2. 추가 컨트롤 플레인 노드를 업그레이드한다.
  3. 워커(worker) 노드를 업그레이드한다.

Control Plane ( api -> cm, ccm, sched -> let, proxy)
=> Worker Node (let, proxy)




EX] kubeadm 1.22.8 -> 1.22.9

  1. Control Plane의 kubeadm 업그레이드
  2. Control Plane의 kubeadm 으로 api, cm, sched 업그레이드
  3. Control plane의 kubelet, kubectl 업그레이드
  4. Worknd 의 kubeadm 업그레이드
  5. Worknd 의 kubeadm 으로 api, cm, sched 업그레이드
  6. Worknd 의 kubelet, kubectl 업그레이드

버전확인 (VERSION : 1.22.8)

$ kubectl get nodes
NAME     STATUS   ROLES                  AGE     VERSION
docker   Ready    control-plane,master   2d19h   v1.22.8
worknd   Ready    <none>                 8m55s   v1.22.8

Control node

1. kubeadm 업그레이드

$ sudo apt-mark unhold kubeadm

$ sudo apt upgrade kubeadm=1.22.9-00 -y

$  kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-04-13T19:56:28Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

$ sudo apt-mark hold kubeadm

2. Control Plane의 kubeadm 으로 api, cm, sched 업그레이드

$ sudo kubeadm upgrade plan

$ sudo kubeadm upgrade apply v1.22.9
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.22.9". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

3. Control plane의 kubelet, kubectl 업그레이드

$ sudo apt-mark unhold kubectl kubelet

$ sudo apt upgrade kubectl=1.22.9-00 kubelet=1.22.9-00

$kubelet --version
Kubernetes v1.22.9
$kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-04-13T19:57:43Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

$ sudo apt-mark hold kubelet kubectl

$ sudo systemctl daemon-reload && sudo systemctl restart kubelet

확인

$ kubectl get nodes
NAME     STATUS   ROLES                  AGE     VERSION
docker   Ready    control-plane,master   2d20h   v1.22.9

work node

1. kubeadm 업그레이드

$ sudo apt-mark unhold kubeadm

$ sudo apt upgrade kubeadm=1.22.9-00 -y

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-04-13T19:56:28Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

$ sudo apt-mark hold kubeadm

2. Work node 의 kubeadm 으로 api, cm, sched 업그레이드

$ sudo kubeadm upgrade node

3. Work node의 kubelet, kubectl 업그레이드

$ sudo apt-mark unhold kubectl kubelet

$ sudo apt upgrade kubectl=1.22.9-00 kubelet=1.22.9-00

$kubelet --version
Kubernetes v1.22.9
$kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-04-13T19:57:43Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

$ sudo apt-mark hold kubelet kubectl

$ sudo systemctl daemon-reload && sudo systemctl restart kubelet

확인

$ kubectl get nodes
NAME     STATUS   ROLES                  AGE     VERSION
docker   Ready    control-plane,master   2d20h   v1.22.9
worknd   Ready    <none>                 91m     v1.22.9

만약 설치 과정중 인터넷이 느리면 명령어 실행
$ sudo sed -i 's/archive.ubuntu.com/mirror.kakao.com/g' /etc/apt/sources.list
$ sudo sed -i 's/security.ubuntu.com/mirror.kakao.com/g' /etc/apt/sources.list





Kubespray

https://kubernetes.io/ko/docs/setup/production-environment/tools/kubespray/

https://kubespray.io/#/
https://github.com/kubernetes-sigs/kubespray



1. 새로운 VM 생성 node1, node2, node3 생성

구성
Control Plane 1
Work Node 2

code .\Vagrantfile

Vagrant.configure("2") do |config|
	# Define VM
	config.vm.define "node1" do |centos|
		centos.vm.box = "ubuntu/focal64"
		centos.vm.hostname = "node1"
		centos.vm.network "private_network", ip: "192.168.100.101"
		centos.vm.provider "virtualbox" do |vb|
			vb.name = "node1"
			vb.cpus = 2
			vb.memory = 2048
		end
	end

ansible을 사용하기 위해 아래 명령어를 통해 password 할수 있게 설정

      config.vm.provision "shell", inline: <<-SHELL
	  sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config
	  sed -i 's/archive.ubuntu.com/mirror.kakao.com/g' /etc/apt/sources.list
	  sed -i 's/security.ubuntu.com/mirror.kakao.com/g' /etc/apt/sources.list
	  systemctl restart ssh
	SHELL
end


2. SSH 키 생성 및 복사

node1에서 진행

# 키 생성
$ ssh-keygen

# 키 복사
$ ssh-copy-id vagrant@192.168.100.101
$ ssh-copy-id vagrant@192.168.100.102
$ ssh-copy-id vagrant@192.168.100.103


3. kubespray 소스 다운로드

$ git clone -b v2.18.1 https://github.com/kubernetes-sigs/kubespray.gitㅁ


4. ansible, netaddr, jinja 패키지 설치

$ cd kubespray/

$ sudo apt update

$ sudo apt install python3-pip -y

$ cat requirements.txt
ansible==3.4.0
ansible-base==2.10.15
cryptography==2.8
jinja2==2.11.3
netaddr==0.7.19
pbr==5.4.4
jmespath==0.9.5
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.6
MarkupSafe==1.1.1

$ sudo pip3 install -r requirements.txt


5. 인벤토리 구성

$ cd inventory/

$ cp -rpf sample/ mycluster

$ cd mycluster/

$ vi inventory.ini

[all]
node1 ansible_host=192.168.100.100 ip=192.168.100.101
node2 ansible_host=192.168.100.101 ip=192.168.100.102
node3 ansible_host=192.168.100.102 ip=192.168.100.103

[kube_control_plane]
node1

[etcd]
node1

[kube_node]
node1
node2
node3


6. 변수 설정

inventory/mycluster/group_vars



7. 플레이북 실행

$ ansible all -m ping -i inventory/mycluster/inventory.ini

$ ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml -b


8. 검증

$ mkdir ~/.kube
$ sudo cp /etc/kubernetes/admin.conf ~/.kube/config
$ sudo chown vagrant:vagrant ~/.kube/config

$ kubectl get nodes
NAME    STATUS   ROLES                  AGE   VERSION
node1   Ready    control-plane,master   15m   v1.22.8
node2   Ready    <none>                 14m   v1.22.8
node3   Ready    <none>                 14m   v1.22.8

$ kubectl get pods -A




Kubernetes Objects

  • Label/LabelSelector

  • Workload
    - Pod : 컨테이너가 담기게 된다.
    - Controller : Pod를 제어
    - ReplicationController
    - ReplicaSets
    - DaemonSets
    - Jobs
    - CronJobs
    - Deployments
    - StatefulSets
    - HorizontalPodAutoscaler

  • Network
    - Service : L4 LB
    - Endpoints : LB의 backend
    - Ingress : L7 LB 애드온으로 추가해서 사용

  • Storage
    - PersistentVolume
    - PersistentVolumeClaim
    - ConfigMap
    - Secret

  • Authentication
    - ServiceAccount
    - RBAC
    - Role
    - ClusterRole
    - RoleBinding
    - ClusterRoleBinding

  • Resource Isolation
    - Namespaces

  • Resource Limits
    - Limits
    - Requests
    - ResourceQuota
    - LimitRange

  • Scheduling
    - NodeName
    - NodeSelector
    - Affinity
    - Node Affinity
    - Pod Affinity
    - Pod Anti Affinity
    - Taints/Tolerations
    - Drain/Cordon

profile
RungNyeok
TIL(Today I Learned)을 실천하기 위한 블로그

0개의 댓글