Istio multicluster 설치(상세)

이민규·2023년 7월 3일

1. 사전준비

1.1 cluster-1, cluster-2 준비

cluster-1번의 ./kube/config 파일을 아래와 같이 수정하여
ctx1 --> cluster-1
ctx2 --> cluster-2
로 지정되도록 설정한다.

apiVersion: v1

clusters:
- cluster:
    api-version: v1
    certificate-authority-data: XXXXXXXXXXXXXXXXXXXXX
    server: "https://10.100.1.x:6443"
  name: "cluster-1"
- cluster:
    api-version: v1
    certificate-authority-data: XXXXXXXXXXXXXXXXXXXXX
    server: "https://10.100.1.x:6443"
  name: "cluster-2"

kind: Config
preferences: {}
users:
- name: "user-1"
  user:
    client-certificate-data: XXXXXXXXXXXXXXXXXXXXX
    client-key-data: XXXXXXXXXXXXXXXXXXXXX
- name: "user-2"
  user:
    client-certificate-data: XXXXXXXXXXXXXXXXXXXXX
    client-key-data: XXXXXXXXXXXXXXXXXXXXX

contexts:
- context:
    cluster: cluster-1
    user: user-1
  name: ctx-1
- context:
    cluster: cluster-2
    user: user-2
  name: ctx-2

current-context: ctx-1

1.2 Cluster 확인

아래명령어로 두개의 클러스터에 접속이 가능한지 테스트한다.

$ kubectl get nodes --context=ctx-1
NAME             STATUS     ROLES           AGE     VERSION
cp-cluster-a-1   Ready      control-plane   4h45m   v1.25.6
cp-cluster-a-2   NotReady   <none>          4h44m   v1.25.6
$ kubectl get nodes --context=ctx-2
NAME             STATUS     ROLES           AGE     VERSION
cp-cluster-b-1   Ready      control-plane   4h41m   v1.25.6
cp-cluster-b-2   NotReady   <none>          4h39m   v1.25.6

2.인증서생성

step라는 오픈소스를 사용한다.

## step 다운로드 및 /usr/bin 배치
$ wget https://github.com/smallstep/cli/releases/download/v0.24.4/step_linux_0.24.4_amd64.tar.gz -O step.tar.gz
$ tar -xvzf step.tar.gz
$ sudo mv step_0.24.4/bin/step /usr/bin/
$ sudo chmod +x /usr/bin/step

## 폴더 생성
$ mkdir certs
$ mkdir -p certs/cluster-1
$ mkdir -p certs/cluster-2

## rootCA 생성
$ step certificate create root.istio.io certs/root-cert.pem certs/root-ca.key \
  --profile root-ca --no-password --insecure --san root.istio.io \
  --not-after 87600h --kty RSA

## 1번 Cluster 인증서 생성
$ step certificate create cluster-1.intermediate.istio.io certs/cluster-1/ca-cert.pem certs/cluster-1/ca-key.pem --ca certs/root-cert.pem --ca-key certs/root-ca.key --profile intermediate-ca --not-after 87600h --no-password --insecure --san cluster-1.intermediate.istio.io --kty RSA

$ cat certs/cluster-1/ca-cert.pem certs/root-cert.pem > certs/cluster-1/cert-chain.pem


## 2번 Cluster 인증서 생성
# step certificate create cluster-2.intermediate.istio.io certs/cluster-2/ca-cert.pem certs/cluster-2/ca-key.pem --ca certs/root-cert.pem --ca-key certs/root-ca.key --profile intermediate-ca --not-after 87600h --no-password --insecure --san cluster-2.intermediate.istio.io --kty RSA

$ cat certs/cluster-1/ca-cert.pem certs/root-cert.pem > certs/cluster-1/cert-chain.pem

아래와 같이 생성된다.

$ tree certs
certs
├── cluster-1
│   ├── ca-cert.pem
│   ├── ca-key.pem
│   └── cert-chain.pem
├── cluster-2
│   ├── ca-cert.pem
│   ├── ca-key.pem
│   └── cert-chain.pem
├── root-ca.key
└── root-cert.pem

3.Istio 설치

$ wget -c "https://github.com/istio/istio/releases/download/1.15.7/istio-1.15.7-linux-amd64.tar.gz" -O istio.tar.gz
$ tar -xvzf istio.tar.gz
$ sudo mv istio-1.15.7/bin/istioctl /usr/bin/

4.istio 컴포넌트를 위한 Resouce Manifest 생성

Cluster1용 manifest

namespace-1.yaml

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: istio-system
    topology.istio.io/network: network-1
  name: istio-system

controlplain-1.yaml

apiVersion: install.istio.io/v1alpha1
metadata:
  name: istio-controlplane
  namespace: istio-system
kind: IstioOperator
spec:
  profile: demo
  meshConfig:
    enablePrometheusMerge: false
    defaultConfig:
      proxyMetadata:
        ISTIO_META_DNS_CAPTURE: "true"
        ISTIO_META_DNS_AUTO_ALLOCATE: "true"
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: false
    ingressGateways:
    - name: istio-ingressgateway
      enabled: false
  values:
    sidecarInjectorWebhook:
      rewriteAppHTTPProbe: false
    global:
      meshID: usmesh
      multiCluster:
        clusterName: cluster-1
      network: network-1

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-eastwestgateway
  namespace: istio-system
spec:
  profile: empty
  components:
    ingressGateways:
    - name: istio-eastwestgateway
      label:
        istio: eastwestgateway
        app: istio-eastwestgateway
        topology.istio.io/network: network-1
      enabled: true
      k8s:
        env:
        - name: ISTIO_META_ROUTER_MODE
          value: "sni-dnat"
        # The network to which traffic is routed
        - name: ISTIO_META_REQUESTED_NETWORK_VIEW
          value: network-1
        service:
          ports:
          - name: status-port
            port: 15021
            targetPort: 15021
          - name: mtls
            port: 15443
            targetPort: 15443
          - name: tcp-istiod
            port: 15012
            targetPort: 15012
          - name: tcp-webhook
            port: 15017
            targetPort: 15017
  values:
    global:
      meshID: usmesh
      multiCluster:
        clusterName: cluster-1
      network: network-1

expose-services.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cross-network-gateway
  namespace: istio-system
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        number: 15443
        name: tls
        protocol: TLS
      tls:
        mode: AUTO_PASSTHROUGH
      hosts:
        - "*.local"

Cluster2용 manifest

namespace-2.yaml
controlplane-2.yaml
eastwest-gateway-2.yaml
위 3개 파일을 신규로 생성하되
내용의 network, cluster name을 2번 cluster에 맞춰 변경해 놓는다.

5. 리소스 배포

5.1 cluster1로 실행

$ kubectl --context="ctx-1" apply -f tmp/namespace-1.yaml

$ kubectl create secret generic cacerts -n istio-system \
        --from-file=./certs/cluster-1/ca-cert.pem \
        --from-file=./certs/cluster-1/ca-key.pem \
        --from-file=./certs/root-cert.pem \
        --from-file=./certs/cluster-1/cert-chain.pem --dry-run -o yaml > tmp/certs.yaml
$ kubectl --context="ctx-1" -n istio-system apply -f tmp/certs.yaml

$ istioctl --context="ctx-1" install -y -f tmp/controlplane-1.yaml

$ istioctl --context="ctx-${CLUSTER_INDEX}" install -y -f tmp/eastwest-gateway-1.yaml

$ kubectl --context="ctx-1" apply -n istio-system -f ./expose-services.yaml
$ kubectl --context="ctx-1" get nodes -o json | jq '.items[].status.addresses[0].address' > tmp/nodes.txt
$ export NODES=`sed -z 's/\n/,/g;s/,$/\n/' tmp/nodes.txt`
$ kubectl --context="ctx-1" patch service istio-eastwestgateway --patch "{\"spec\": {\"externalIPs\": [${NODES}]}}" -n istio-system
    
$ istioctl --context="ctx-1" x create-remote-secret --name="cluster-1" > tmp/cluster-secret-1.yaml
$ kubectl --context="ctx-1" apply -f tmp/cluster-secret-1.yaml

5.2 Cluster2로 실행

$ kubectl --context="ctx-2" apply -f tmp/namespace-2.yaml

$ kubectl create secret generic cacerts -n istio-system \
        --from-file=./certs/cluster-2/ca-cert.pem \
        --from-file=./certs/cluster-2/ca-key.pem \
        --from-file=./certs/root-cert.pem \
        --from-file=./certs/cluster-2/cert-chain.pem --dry-run -o yaml > tmp/certs.yaml
$ kubectl --context="ctx-2" -n istio-system apply -f tmp/certs.yaml

$ istioctl --context="ctx-2" install -y -f tmp/controlplane-2.yaml

$ istioctl --context="ctx-2" install -y -f tmp/eastwest-gateway-2.yaml

$ kubectl --context="ctx-2" apply -n istio-system -f ./expose-services.yaml
$ kubectl --context="ctx-2" get nodes -o json | jq '.items[].status.addresses[0].address' > tmp/nodes.txt
$ export NODES=`sed -z 's/\n/,/g;s/,$/\n/' tmp/nodes.txt`
$ kubectl --context="ctx-2" patch service istio-eastwestgateway --patch "{\"spec\": {\"externalIPs\": [${NODES}]}}" -n istio-system
$ istioctl --context="ctx-2" x create-remote-secret --name="cluster-2" > tmp/cluster-secret-1.yaml
$ kubectl --context="ctx-2" apply -f tmp/cluster-secret-2.yaml

6.테스트

$ kubectl --context=ctx-1 create ns sample
$ kubectl --context=ctx-2 create ns sample
$ 
$ kubectl --context=ctx-1 label namespace sample istio-injecti
on=enabled --overwrite
$ kubectl --context=ctx-2 label namespace sample istio-injecti
on=enabled --overwrite

테스트용 Yaml 생성

아래 두개 Yaml 파일은 istio를 다운받으면 동일하게 존재하는 Yaml이다.

helloworld.yaml

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v1
  labels:
    app: helloworld
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v1
  template:
    metadata:
      labels:
        app: helloworld
        version: v1
    spec:
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v1
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent #Always
        ports:
        - containerPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v2
  labels:
    app: helloworld
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v2
  template:
    metadata:
      labels:
        app: helloworld
        version: v2
    spec:
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v2
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent #Always
        ports:
        - containerPort: 5000

sleep.yaml

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
        sidecar.istio.io/inject: "true"
    spec:
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent

ctx-1에 배포

$ kubectl --context=ctx-1 -n sample apply -f test/helloworld.yaml  

$ kubectl --context=ctx-1 -n sample get pods
NAME                             READY   STATUS    RESTARTS   AGE
helloworld-v1-78b9f5c87f-x7tb9   2/2     Running   0          67m
helloworld-v2-54dddc5567-rg2tr   2/2     Running   0          67m

$ kubectl --context=ctx-1 -n sample get service
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
helloworld   ClusterIP   10.233.18.239   <none>        5000/TCP   67m

ctx-2에 배포

$ kubectl --context=ctx-2 -n sample apply -f test/sleep.yaml

$ kubectl --context=ctx-2 -n sample get pods
NAME                     READY   STATUS    RESTARTS   AGE
sleep-5cc8999566-g77ws   2/2     Running   0          97m

$ kubectl --context=ctx-2 -n sample get service
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
helloworld   ClusterIP   10.233.10.184   <none>        5000/TCP   3h49m
sleep        ClusterIP   10.233.29.143   <none>        80/TCP     3h49m

curl로 확인

ctx-2번에서 실행중인 sleep pod에 접속하여 helloworld.test:5000으로 curl을 실행하면 정상적으로 리턴되는 것을 볼 수 있다.

ubuntu@cp-cluster-a-1:~/istio-multi-cluster-initializer/tmp$ kubectl --context=ctx-2 -n sample exec -it sleep-5cc8999566-g77ws sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
~ $ curl helloworld.test:5000/hello
Hello version: v2, instance: helloworld-v2-54dddc5567-rg2tr
~ $ curl helloworld.test:5000/hello
Hello version: v1, instance: helloworld-v1-78b9f5c87f-x7tb9

이민규

이전 포스트

multi istio

다음 포스트