Install Kubespray on CentOS 7

Haje Kim·2021년 8월 17일
CentOS 7CentOS7centosclusterk8skuberneteskubespraylinux개발개발환경쿠버네티스클러스터환경
0

Kubernetes

목록 보기
4/4

Kubespray 간단한 소개


Kubernetes를 장인 정신으로 한땀한땀 소중히 구성하는 것도 좋은 접근입니다....만 여러 노드에 구성하고 잘 운영한다는건 매우매우매우 많은 노력을 요합니다.
그래서 Kubespray는 Ansible Playbook, Inventory, Provisioning 도구와 일반적인 운영체제, Kubernetes Cluster 설정 관리 작업을 위해 개발되었습니다.
AWS, GCE, Azure, OpenStack, vSphere, Equinix Metal (bare metal), Oracle Cloud Infrastructure (Experimental), or 베어메탈을 지원하며, 고가용성 클러스터를 구성할 수 있습니다.
자 Kubespray로 쉽게 가시죠!

Preface

다른 글에 비해 서론이 유난히 무척 긴 듯한 기분은 착각입니다.
본 문서에서는 Kubernetes Cluster 구성을 위하여 kubespray를 이용합니다.
Kubernetes Cluster 구성에는 kubeadm, kubespray, kops, minikube 등 여러 방법이 있습니다.
차이점이 자세하게 궁금하다면 이 문서를 읽어봅시다.
가장 큰 차이는 Kubespray는 Ansible을 이용하여 프로비저닝하고 관리한다.

  • Ansible은 환경 구성 자동화 도구이며, Chef, Puppet 등과 같은 구성 관리 도구입니다.
  • 2012년에 개발되었고, 2015년에 RedHat에 인수되었습니다.
  • Terraform과 많이 비교하고 있지만 목적이 다릅니다.
    • Terraform: Orchestration Tool
    • Ansible: Configuration Management Tool
  • Terraform은 선언적(Declarative) 도구이며, Ansible은 절차적(Procedure) 도구입니다. (Ansible이 많이 업그레이드 되서 하이브리드스러워지긴 했지만)
  • 기능적으로 겹쳐진 영역이 존재하지만, 서로 보안하며 사용하는 것이 좋습니다.
  • 이런 아이들을 Infrastructure as Code (IaC)라고도 부릅니다.

Pre-Installation Requirements

  • Ansible
  • Python3
  • 여러 대의 Virtual Machine (본 문서에서는 Vagrant와 CentOS 7을 이용합니다)

Kubespray의 특징으로 Ansible과 Python3가 필요합니다.
그리고 설치 환경은 제 최애 CentOS입니다.
VM은 AWS, Azure, GCP, Oracle Cloud 등 클라우드에서 프로비저닝하셔도 좋지만, 저는 가난한 관계로 로컬 환경에서 Vagrant를 이용하여 VM을 구성합니다. 만약 Vagrant로 구성하고자 한다면 Prepare CentOS 7 with Vagrant & VirtualBox 참고하여 주세요.

Architecture

Kubernetes Cluster를 구성하는 컴포넌트는 다음과 같습니다.

출처: Kubernetes Components

Control Plane Componenets은 etcd, kube-apiserver, kube-scheduler, kube-controller-manager 등 클러스터를 관리하는 Master Node입니다.
그리고 kubelet 등 실제 서비스를 배포하는
Node Components는 모든 노드에서 실행되어 실행 중인 포드를 유지 관리하고 Kubernetes 런타임 환경을 제공하는 Worker Node입니다.

본 문서에서 VM 2개를 구성하며 아래와 같은 아키텍처로 구성합니다.

  • Master Node : cent01
  • Worker Node : cent02
    로컬 환경이 작아서 마스터 노드는 하나, 워커 노드 하나로 구성합니다.


나중에 여유가 되면 가용성 테스트를 위해 마스터 3, 워커 3으로 구성해보겠습니다.

설정 및 설치하기

마스터 노드와 워커 노드에 설정이 차이가 있습니다.
우선 모든 서버에 공통적으로 적용하는 설정부터 시작하겠습니다.

세팅하기 [모든 서버]

01. Swap 메모리 비활성화

Swap은 메모리가 부족하면 Disk Storage를 메모리처럼 사용하는 방식이지만, 메모리에 비해 I/O 속도가 떨어지기 때문에 오히려 장애가 발생할 수 있습니다.
끄도록 합니다.

sudo swapoff -a

02. IP Forward 설정

# 설정 확인
cat /proc/sys/net/ipv4/ip_forward
0

# 설정 변경
sudo sysctl -w net.ipv4.ip_forward=1
cat /proc/sys/net/ipv4/ip_forward
1

03. /etc/hosts 설정

각 VM의 IP Address와 호스트네임을 설정합니다.

sudo vi /etc/hosts
192.168.1.11	cent01 master01
192.168.1.12	cent02 node01

세팅하기 [마스터 노드]

04. Ansible & Python 설치

Yum 패키지 매니저를 이용하여 Python3, Ansible을 설치합니다.

sudo yum install -y epel-release
sudo yum update -y
sudo yum install -y ansible
sudo yum install -y python3

05. SSH Key 생성 및 설정

먼저 SSH Key를 생성합니다.

ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/vagrant/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/vagrant/.ssh/id_rsa.
Your public key has been saved in /home/vagrant/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:0JRjph0/KfCal/Ywj/kSyJ0CsXysP8cvEzCNkqF8B2M vagrant@cent01
The key's randomart image is:
+---[RSA 2048]----+
|        ..       |
|    E .o*        |
| . + O.X.+ .     |
|  o B O.= +      |
|   . B BS+ .     |
|    . * @        |
|     . = X       |
|      o O o      |
|       o *o      |
+----[SHA256]-----+

그 다음에 다른 노드에도 접속이 가능하도록 cent01의 SSH Public Key를 전달합니다.

ssh-copy-id cent01
ssh-copy-id cent02

정상적으로 각 노드에 Public Key가 설정이 되었는지 확인하기 위해서 아래의 파일을 확인하여 cent01의 SSH Public Key와 비교해봅니다.

~/.ssh/known_hosts

06. Kubespray 준비하기

세팅은 얼추 끝났으니, Kubespray를 본격적으로 시작해봅시다.
시작하기 위해 GitHub 레포지토리를 클론을 뜹니다.

git clone https://github.com/kubernetes-sigs/kubespray.git

Cloning into 'kubespray'...
remote: Enumerating objects: 55117, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 55117 (delta 14), reused 37 (delta 12), pack-reused 55072
Receiving objects: 100% (55117/55117), 15.92 MiB | 10.34 MiB/s, done.
Resolving deltas: 100% (31108/31108), done.

07. Requirement 패키지 설치하기

kubespray에 필요한 패키지가 명시된 requirements.txt를 이용하여 설치합니다.

cd kubespray

sudo pip3 install -r requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting ansible==3.4.0 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/a9/f3/7e7e3647d58b266768a000b5830c7fca7c02eac4e724e9b23309b735f9b2/ansible-3.4.0.tar.gz (31.9MB)
    100% |████████████████████████████████| 31.9MB 44kB/s
Collecting ansible-base==2.10.11 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/a0/93/9d8c26b2a4e1d2b48802a577ca6ada084f58e576f67917c9e28858bf747b/ansible-base-2.10.11.tar.gz (6.0MB)
    100% |████████████████████████████████| 6.0MB 241kB/s
Collecting cryptography==2.8 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/45/73/d18a8884de8bffdcda475728008b5b13be7fbef40a2acc81a0d5d524175d/cryptography-2.8-cp34-abi3-manylinux1_x86_64.whl (2.3MB)
    100% |████████████████████████████████| 2.3MB 627kB/s
Collecting jinja2==2.11.3 (from -r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/7e/c2/1eece8c95ddbc9b1aeb64f5783a9e07a286de42191b7204d67b7496ddf35/Jinja2-2.11.3-py2.py3-none-any.whl (125kB)
    100% |████████████████████████████████| 133kB 6.7MB/s
Collecting netaddr==0.7.19 (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/ba/97/ce14451a9fd7bdb5a397abf99b24a1a6bb7a1a440b019bebd2e9a0dbec74/netaddr-0.7.19-py2.py3-none-any.whl (1.6MB)
    100% |████████████████████████████████| 1.6MB 877kB/s
Collecting pbr==5.4.4 (from -r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/7a/db/a968fd7beb9fe06901c1841cb25c9ccb666ca1b9a19b114d1bbedf1126fc/pbr-5.4.4-py2.py3-none-any.whl (110kB)
    100% |████████████████████████████████| 112kB 7.4MB/s
Collecting jmespath==0.9.5 (from -r requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/a3/43/1e939e1fcd87b827fe192d0c9fc25b48c5b3368902bfb913de7754b0dc03/jmespath-0.9.5-py2.py3-none-any.whl
Collecting ruamel.yaml==0.16.10 (from -r requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/a6/92/59af3e38227b9cc14520bf1e59516d99ceca53e3b8448094248171e9432b/ruamel.yaml-0.16.10-py2.py3-none-any.whl (111kB)
    100% |████████████████████████████████| 112kB 8.5MB/s
Collecting ruamel.yaml.clib==0.2.2 (from -r requirements.txt (line 9))
  Downloading https://files.pythonhosted.org/packages/88/ff/ec25dc01ef04232a9e68ff18492e37dfa01f1f58172e702ad4f38536d41b/ruamel.yaml.clib-0.2.2-cp36-cp36m-manylinux1_x86_64.whl (549kB)
    100% |████████████████████████████████| 552kB 2.3MB/s
Collecting MarkupSafe==1.1.1 (from -r requirements.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting PyYAML (from ansible-base==2.10.11->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/7a/5b/bc0b5ab38247bba158504a410112b6c03f153c652734ece1849749e5f518/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640kB)
    100% |████████████████████████████████| 645kB 2.1MB/s
Collecting packaging (from ansible-base==2.10.11->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/3c/77/e2362b676dc5008d81be423070dd9577fa03be5da2ba1105811900fda546/packaging-21.0-py3-none-any.whl (40kB)
    100% |████████████████████████████████| 40kB 10.2MB/s
Collecting cffi!=1.11.3,>=1.8 (from cryptography==2.8->-r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/f2/cd/3f5f059fed635d71047fa9ce507635088f982ab280fc24cde91d9afb9c1c/cffi-1.14.6-cp36-cp36m-manylinux1_x86_64.whl (401kB)
    100% |████████████████████████████████| 409kB 2.9MB/s
Collecting six>=1.4.1 (from cryptography==2.8->-r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
Collecting pyparsing>=2.0.2 (from packaging->ansible-base==2.10.11->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
    100% |████████████████████████████████| 71kB 8.5MB/s
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography==2.8->-r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
    100% |████████████████████████████████| 112kB 9.3MB/s
Installing collected packages: MarkupSafe, jinja2, PyYAML, pycparser, cffi, six, cryptography, pyparsing, packaging, ansible-base, ansible, netaddr, pbr, jmespath, ruamel.yaml.clib, ruamel.yaml
  Running setup.py install for ansible-base ... done
  Running setup.py install for ansible ... done
Successfully installed MarkupSafe-1.1.1 PyYAML-5.4.1 ansible-3.4.0 ansible-base-2.10.11 cffi-1.14.6 cryptography-2.8 jinja2-2.11.3 jmespath-0.9.5 netaddr-0.7.19 packaging-21.0 pbr-5.4.4 pycparser-2.20 pyparsing-2.4.7 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.2 six-1.16.0

08. Inventory 복사하기

Oracle Database나 WebLogic Server를 사용하면 만나는 Inventory라는 디렉토리를 Kubespray에서도 볼 줄 몰랐네요. (물론 두 개는 일절 관련 없습니다ㅋㅋㅋ)
inventory 디렉토리에서 Kubernetes Cluster를 관리합니다.
샘플을 복사하여 사용합니다.

cp -rfp inventory/sample inventory/firstCluster

09. inventory.ini 설정

클러스터를 구성할 노드 리스트를 입력합니다.
서버 리스트와 마스터 노드, 워커 노드, etcd 등을 지정할 수 있습니다.
저는 cent01(master01)과 cent02(node01) 두 개만 있기 때문에 아래와 같이 설정하였습니다. 

./inventory/first_cluster/inventory.ini

[all]
master01        ansible_host=192.168.1.11       ip=192.168.1.11
node01          ansible_host=192.168.1.12       ip=192.168.1.12

[kube_control_plane]
master01

[etcd]
master01

[kube_node]
node01

[k8s_cluster:children]
kube_control_plane
kube_node
calico_rr

10. 서버에게 Ping 테스트

Master Node(cent01)에서 inventory에 등록한 서버에 접근이 가능한지 테스트를 합니다.

ansible all -i inventory/firstCluster/inventory.ini -m ping

아래와 같이 출력되면 설정이 제대로 되었다는 의미입니다.

node01 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
master01 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}

11. Ansible Playbook으로 클러스터 설정

위와 같이 Ping이 정상적으로 가는 것을 확인하였다면, Ansible Playbook으로 K8s 클러스터를 구성합니다.
약 설정까지 18분이 소요되었습니다.

ansible-playbook -i inventory/firstcluster/inventory.ini -become --become-user=root cluster.yml

PLAY [localhost] *************************************************************************************************************************************************************************************************************************************************************************************************************
Tuesday 17 August 2021  07:23:48 +0000 (0:00:00.042)       0:00:00.042 ********

TASK [Check 2.9.0 <= Ansible version < 2.11.0] *******************************************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}
Tuesday 17 August 2021  07:23:48 +0000 (0:00:00.041)       0:00:00.084 ********

TASK [Check Ansible version > 2.10.11 when using ansible 2.10] ***************************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}
Tuesday 17 August 2021  07:23:48 +0000 (0:00:00.040)       0:00:00.125 ********

TASK [Check that python netaddr is installed] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}
Tuesday 17 August 2021  07:23:48 +0000 (0:00:00.059)       0:00:00.184 ********

TASK [Check that jinja is not too old (install via pip)] *********************************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY [Add kube-master nodes to kube_control_plane] ***************************************************************************************************************************************************************************************************************************************************************************
Tuesday 17 August 2021  07:23:49 +0000 (0:00:00.057)       0:00:00.241 ********

TASK [add nodes to kube_control_plane group] *********************************************************************************************************************************************************************************************************************************************************************************
changed: [master01]
[WARNING]: Could not match supplied host pattern, ignoring: kube-node

PLAY [Add kube-node nodes to kube_node] **************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: k8s-cluster

PLAY [Add k8s-cluster nodes to k8s_cluster] **********************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: calico-rr

PLAY [Add calico-rr nodes to calico_rr] **************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: no-floating

PLAY [Add no-floating nodes to no_floating] **********************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: bastion

PLAY [bastion[0]] ************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched

...생략...

Tuesday 17 August 2021  07:42:05 +0000 (0:00:00.222)       0:18:16.720 ********
Tuesday 17 August 2021  07:42:05 +0000 (0:00:00.165)       0:18:16.885 ********
Tuesday 17 August 2021  07:42:05 +0000 (0:00:00.117)       0:18:17.002 ********

PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
master01                   : ok=571  changed=121  unreachable=0    failed=0    skipped=1145 rescued=0    ignored=1
node01                     : ok=367  changed=73   unreachable=0    failed=0    skipped=640  rescued=0    ignored=0

Tuesday 17 August 2021  07:42:05 +0000 (0:00:00.141)       0:18:17.144 ********
===============================================================================
container-engine/docker : ensure docker packages are installed ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 128.05s
download : download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 81.49s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 46.37s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 45.22s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 43.95s
download : download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 34.61s
download : download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 32.58s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 32.30s
kubernetes/kubeadm : Join to cluster --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 30.82s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 30.57s
kubernetes/control-plane : kubeadm | Initialize first master --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 29.75s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 27.84s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 27.47s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 23.20s
download : download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 20.70s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 19.90s
download : download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 18.01s
download : download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 17.19s
download : download_container | Download image if required ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 15.28s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.28s
[vagrant@cent01 kubespray]$
[vagrant@cent01 kubespray]$ Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.
profile
Haje Kim
Solutions Architect
이전 포스트

Docker 101

0개의 댓글