현재 k8s 자동 프로비저닝 툴을 개발 중 입니다.

external etcd에 member를 새로 추가하려면 2가지 방법이 있지만
하나는 아예 데이터를 지우고 재구성하는 방법이기 때문에 추천하지 않는다.
왠만하면 member add를 사용하기를 추천한다.

Cluster 정보

NAME             STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
temp-cp-node-1   Ready    control-plane   4d16h   v1.29.6   10.50.1.175   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28
temp-cp-node-2   Ready    control-plane   4d16h   v1.29.6   10.50.1.42    <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28
temp-cp-node-3   Ready    control-plane   4d16h   v1.29.6   10.50.1.48    <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28
temp-cp-node-4   Ready    control-plane   4d16h   v1.29.6   10.50.1.203   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28
temp-cp-node-5   Ready    control-plane   4d16h   v1.29.6   10.50.1.45    <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28
temp-wk-node-1   Ready    node            4d16h   v1.29.6   10.50.1.184   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28
temp-wk-node-2   Ready    node            4d16h   v1.29.6   10.50.1.208   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28
temp-wk-node-3   Ready    node            4d16h   v1.29.6   10.50.1.159   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28

Etcd 정보

test@temp-cp-node-1:~$ etcdctl member list --write-out=table
+------------------+---------+----------------+--------------------------+--------------------------+------------+
|        ID        | STATUS  |      NAME      |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+----------------+--------------------------+--------------------------+------------+
| 12c777858143401d | started | temp-cp-node-1 | https://10.50.1.175:2380 | https://10.50.1.175:2379 |      false |
| ac04d47586d8688a | started | temp-cp-node-2 |  https://10.50.1.42:2380 |  https://10.50.1.42:2379 |      false |
| d6eeedc86fad8737 | started | temp-cp-node-3 |  https://10.50.1.48:2380 |  https://10.50.1.48:2379 |      false |
+------------------+---------+----------------+--------------------------+--------------------------+------------+

Etcd cluster 추가(추천)

해당 방법은 순서가 매우 중요함

아래의 명령어를 실행(etcd cluster가 구성된 곳이면 상관없음)

test@temp-cp-node-1:~$ etcdctl --endpoints=https://10.50.1.175:2379 member add temp-cp-node-4 --peer-urls=https://10.50.1.203:2380

• 추가할 노드의 etcd.conf
  • temp-cp-node-4
  • 10.50.1.203

ETCD_INITIAL_CLUSTER와 ETCD_INITIAL_CLUSTER_STATE의 설정 값에 유의한다.

#[member]
ETCD_NAME=temp-cp-node-4

ETCD_DATA_DIR=/data/etcd
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.50.1.203:2380
######### 일단 temp-cp-node-4만 추가할 것이기 때문에 해당 설정으로 구성 #########
ETCD_INITIAL_CLUSTER=temp-cp-node-1=https://10.50.1.175:2380,temp-cp-node-2=https://10.50.1.42:2380,temp-cp-node-3=https://10.50.1.48:2380,temp-cp-node-4=https://10.50.1.203:2380
######### 추가할 때는 existing을 사용 #########
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_INITIAL_CLUSTER_TOKEN=etcd-k8-cluster
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
ETCD_ADVERTISE_CLIENT_URLS=https://10.50.1.203:2379
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"

#[proxy]
ETCD_PROXY="off"

#[security]
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE=/etc/kubernetes/pki/etcd/ca.crt
ETCD_CERT_FILE=/etc/kubernetes/pki/etcd/server.crt
ETCD_KEY_FILE=/etc/kubernetes/pki/etcd/server.key
ETCD_PEER_TRUSTED_CA_FILE=/etc/kubernetes/pki/etcd/ca.crt
ETCD_PEER_CERT_FILE=/etc/kubernetes/pki/etcd/peer.crt
ETCD_PEER_KEY_FILE=/etc/kubernetes/pki/etcd/peer.key

명령어 실행 후 정상작동 확인

test@temp-cp-node-4:~$ sudo systemctl stop etcd
test@temp-cp-node-4:~$ sudo rm -rf /data/etcd/*
test@temp-cp-node-4:~$ sudo systemctl start etcd

temp-cp-node-5도 추가하려면 위와 똑같이 실행하는데
아래와 같이 기존 etcd cluster의 정보를 다 넣어줘야 정상작동한다.

ETCD_INITIAL_CLUSTER=temp-cp-node-1=https://10.50.1.175:2380,temp-cp-node-2=https://10.50.1.42:2380,temp-cp-node-3=https://10.50.1.48:2380,temp-cp-node-4=https://10.50.1.203:2380,temp-cp-node-5=https://10.50.1.45:2380

Etcd 정보 재구성(비추천)

모든 노드의 etcd.conf의 설정을 해당 파일처럼 변경한다.
각 설정 세부사항들은 맞춰서 변경해야한다.

#[member]
################## 각 노드 hostname ##################
ETCD_NAME=temp-cp-node-5

ETCD_DATA_DIR=/data/etcd
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""

#[cluster]
######### 해당 etcd 노드가 다른 etcd 노드들과 통신할 때 사용할 URL #########
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.50.1.45:2380
######### etcd cluster 정보 #########
ETCD_INITIAL_CLUSTER=temp-cp-node-1=https://10.50.1.175:2380,temp-cp-node-2=https://10.50.1.42:2380,temp-cp-node-3=https://10.50.1.48:2380,temp-cp-node-4=https://10.50.1.203:2380,temp-cp-node-5=https://10.50.1.45:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_INITIAL_CLUSTER_TOKEN=etcd-k8-cluster
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
######### 클라이언트가 해당 etcd 노드와 통신할 때 사용할 URL #########
ETCD_ADVERTISE_CLIENT_URLS=https://10.50.1.45:2379
ETCD_LISTEN_CLIENT_URLS="https://0.0.0.0:2379"

#[proxy]
ETCD_PROXY="off"

#[security]
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE=/etc/kubernetes/pki/etcd/ca.crt
ETCD_CERT_FILE=/etc/kubernetes/pki/etcd/server.crt
ETCD_KEY_FILE=/etc/kubernetes/pki/etcd/server.key
ETCD_PEER_TRUSTED_CA_FILE=/etc/kubernetes/pki/etcd/ca.crt
ETCD_PEER_CERT_FILE=/etc/kubernetes/pki/etcd/peer.crt
ETCD_PEER_KEY_FILE=/etc/kubernetes/pki/etcd/peer.key

아래의 명령어를 모든 노드에서 실행하면 etcd cluster가 재설정된다.

sudo systemctl stop etcd
sudo rm -rf /data/etcd/*
sudo systemctl start etcd

+------------------+---------+----------------+--------------------------+--------------------------+------------+
|        ID        | STATUS  |      NAME      |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+----------------+--------------------------+--------------------------+------------+
| 12c777858143401d | started | temp-cp-node-1 | https://10.50.1.175:2380 | https://10.50.1.175:2379 |      false |
| 1ba5156e6155cbfa | started | temp-cp-node-5 |  https://10.50.1.45:2380 |  https://10.50.1.45:2379 |      false |
| 9eb84f487ddb73fd | started | temp-cp-node-4 | https://10.50.1.203:2380 | https://10.50.1.203:2379 |      false |
| ac04d47586d8688a | started | temp-cp-node-2 |  https://10.50.1.42:2380 |  https://10.50.1.42:2379 |      false |
| d6eeedc86fad8737 | started | temp-cp-node-3 |  https://10.50.1.48:2380 |  https://10.50.1.48:2379 |      false |
+------------------+---------+----------------+--------------------------+--------------------------+------------+