[K8s] 인그레스(Ingress)

Aiden·2021년 7월 18일

DKOS k8s kubernetes

K8S Study

목록 보기

9/9

1. Ingress 란?

클러스터 외부에서 클러스터 내부 서비스로 HTTP와 HTTPS 경로를 노출
인그레스 기능 : HTTP(서비스) 부하분산 , 카나리 업그레이드
인그레스 컨트롤러 : 인그레스의 실제 동작 구현은 인그레스 컨트롤러(Nginx, Kong 등)가 담당
인그레스 + 인그레스 컨트롤러(Nginx) 기능 : HTTP(서비스) 부하분산 , 카나리 업그레이드 , HTTPS 처리(TLS 종료)

2. Ingress 구성

마스터 노드에 인그레스 컨트롤러(Nginx) 파드를 생성, NodePort 로 외부에 노출
인그레스 정책 설정 : Path routing

2-1. 디플로이먼트 및 서비스 생성

YAML 템플릿

svc1-pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy1-websrv
spec:
  replicas: 1
  selector:
    matchLabels:
      app: websrv
  template:
    metadata:
      labels:
        app: websrv
    spec:
      containers:
      - name: pod-web
        image: nginx
---
apiVersion: v1
kind: Service
metadata:
  name: svc1-web
spec:
  ports:
    - name: web-port
      port: 9001
      targetPort: 80
  selector:
    app: websrv
  type: **ClusterIP**

svc2-pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy2-guestsrv
spec:
  replicas: 2
  selector:
    matchLabels:
      app: guestsrv
  template:
    metadata:
      labels:
        app: guestsrv
    spec:
      containers:
      - name: pod-guest
        image: gcr.io/google-samples/kubernetes-bootcamp:v1
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: svc2-guest
spec:
  ports:
    - name: guest-port
      port: 9002
      targetPort: 8080
  selector:
    app: guestsrv
  type: **NodePort**

svc3-pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy3-adminsrv
spec:
  replicas: 3
  selector:
    matchLabels:
      app: adminsrv
  template:
    metadata:
      labels:
        app: adminsrv
    spec:
      containers:
      - name: pod-admin
        image: k8s.gcr.io/echoserver:1.5
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: svc3-admin
spec:
  ports:
    - name: admin-port
      port: 9003
      targetPort: 8080
  selector:
    app: adminsrv

생성 및 확인

생성

# kubectl apply -f svc1-pod.yaml
deployment.apps/deploy1-websrv created
service/svc1-web created
# kubectl apply -f svc2-pod.yaml
deployment.apps/deploy2-guestsrv created
service/svc2-guest created
# kubectl apply -f svc3-pod.yaml
deployment.apps/deploy3-adminsrv created
service/svc3-admin created

확인

watch -d 'kubectl get pods,svc,ep -o wide'
Every 2.0s: kubectl get pods,svc,ep -o wide                                                                                                          master: Sun Jul 18 06:24:24 2021

NAME                                    READY   STATUS    RESTARTS   AGE   IP               NODE      NOMINATED NODE   READINESS GATES
pod/deploy1-websrv-574449b8df-mzxmg     1/1     Running   0          46s   172.16.235.140   worker1   <none>           <none>
pod/deploy2-guestsrv-6f7f766988-9hxjw   1/1     Running   0          41s   172.16.189.105   worker2   <none>           <none>
pod/deploy2-guestsrv-6f7f766988-q9r4j   1/1     Running   0          41s   172.16.235.141   worker1   <none>           <none>
pod/deploy3-adminsrv-7bd5d589b9-drllf   1/1     Running   0          39s   172.16.189.106   worker2   <none>           <none>
pod/deploy3-adminsrv-7bd5d589b9-snjvg   1/1     Running   0          39s   172.16.182.43    worker3   <none>           <none>
pod/deploy3-adminsrv-7bd5d589b9-vl9lh   1/1     Running   0          39s   172.16.235.142   worker1   <none>           <none>

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE     SELECTOR
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          4m51s   <none>
service/svc1-web     ClusterIP   10.99.16.110    <none>        9001/TCP         46s     app=websrv
service/svc2-guest   NodePort    10.111.55.249   <none>        9002:30371/TCP   41s     app=guestsrv
service/svc3-admin   ClusterIP   10.99.94.161    <none>        9003/TCP         39s     app=adminsrv

NAME                   ENDPOINTS                                                    AGE
endpoints/kubernetes   192.168.1.211:6443                                           4m51s
endpoints/svc1-web     172.16.235.140:80                                            46s
endpoints/svc2-guest   172.16.189.105:8080,172.16.235.141:8080                      41s
endpoints/svc3-admin   172.16.182.43:8080,172.16.189.106:8080,172.16.235.142:8080   39s

2-2. 인그레스 생성

YAML 템플릿

ingress1.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-1
  annotations:
    kubernetes.io/ingress.class: "nginx"
    #nginx.ingress.kubernetes.io/upstream-hash-by: "true"
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: svc1-web
          servicePort: 80
      - path: /guest
        backend:
          serviceName: svc2-guest
          servicePort: 8080
      - path: /admin
        backend:
          serviceName: svc3-admin
          servicePort: 8080

생성 및 확인

생성

# kubectl apply -f ingress1.yaml
ingress.networking.k8s.io/ingress-1 created

확인

# k get ing
NAME        CLASS    HOSTS   ADDRESS   PORTS   AGE
ingress-1   <none>   *                 80      14s

watch -d 'kubectl get ing,svc,ep -o wide'
Every 2.0s: kubectl get ing,svc,ep -o wide                                                                                                           master: Sun Jul 18 06:30:06 2021

NAME                                  CLASS    HOSTS   ADDRESS         PORTS   AGE
**ingress.networking.k8s.io/ingress-1   <none>   *       192.168.1.211   80      67s**

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE     SELECTOR
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          10m     <none>
service/svc1-web     ClusterIP   10.99.16.110    <none>        9001/TCP         6m28s   app=websrv
service/svc2-guest   NodePort    10.111.55.249   <none>        9002:30371/TCP   6m23s   app=guestsrv
service/svc3-admin   ClusterIP   10.99.94.161    <none>        9003/TCP         6m21s   app=adminsrv

NAME                   ENDPOINTS                                                    AGE
endpoints/kubernetes   192.168.1.211:6443                                           10m
endpoints/svc1-web     172.16.235.140:80                                            6m28s
endpoints/svc2-guest   172.16.189.105:8080,172.16.235.141:8080                      6m23s
endpoints/svc3-admin   172.16.182.43:8080,172.16.189.106:8080,172.16.235.142:8080   6m21s

2-3. 인그레스 컨트롤러 생성

간편한 테스트를 위해 NodePort 타입(externalTrafficPolicy: Local) 로 마스터 노드에 생성한다.

YAML 템플릿

양이 길어 다운로드 받아 활용

# curl -s -o nginx-ingress.yaml -O -L curl -s -O https://raw.githubusercontent.com/gasida/DKOS/main/6/nginx-ingress.yaml
# sed -i 's/**k8s-m**/master/g' nginx-ingress.yaml

생성 및 확인

생성

# kubectl apply -f nginx-ingress.yaml
namespace/ingress-nginx unchanged
serviceaccount/ingress-nginx unchanged
configmap/ingress-nginx-controller configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
role.rbac.authorization.k8s.io/ingress-nginx unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
service/ingress-nginx-controller-admission unchanged
service/ingress-nginx-controller unchanged
deployment.apps/ingress-nginx-controller configured
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
serviceaccount/ingress-nginx-admission unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
job.batch/ingress-nginx-admission-create unchanged
job.batch/ingress-nginx-admission-patch unchanged

확인

# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.102.103.48   <none>        80:32182/TCP,443:31768/TCP   6d18h
ingress-nginx-controller-admission   ClusterIP   10.110.11.197   <none>        443/TCP

# kubectl describe svc -n ingress-nginx ingress-nginx-controller | grep Local
External Traffic Policy:  Local

ingress-nginx-controller NodePort(HTTP/HTTPS) 변수 지정

# export IngHttp=$(kubectl get service -n ingress-nginx ingress-nginx-controller -o jsonpath='{.spec.ports[0].nodePort}')
# echo $IngHttp
32182

# export IngHttps=$(kubectl get service -n ingress-nginx ingress-nginx-controller -o jsonpath='{.spec.ports[1].nodePort}')
# echo $IngHttps
31768

2-4. 인그레스를 통한 내부 접속 확인

YAML 템플릿

HTTP / HTTPS 포트 확인

# watch -d 'kubectl get pod -n ingress-nginx -o wide -l app.kubernetes.io/component=controller; echo; kubectl get svc -n ingress-nginx; echo; kubectl get ingress,ep;'
Every 2.0s: kubectl get pod -n ingress-nginx -o wide -l app.kubernetes.io/component=controller; echo; kubectl get svc -n ingress-nginx; echo; ku...  master: Sun Jul 18 06:46:21 2021

NAME                                        READY   STATUS    RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES
ingress-nginx-controller-566455f794-w7bnc   1/1     Running   2          6d18h   172.16.219.69   master   <none>           <none>

NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
**ingress-nginx-controller             NodePort    10.102.103.48   <none>        80:32182/TCP,443:31768/TCP   6d18h**
ingress-nginx-controller-admission   ClusterIP   10.110.11.197   <none>        443/TCP                      6d18h

NAME                                  CLASS    HOSTS   ADDRESS         PORTS   AGE
ingress.networking.k8s.io/ingress-1   <none>   *       192.168.1.211   80      17m

NAME                   ENDPOINTS                                                    AGE
endpoints/kubernetes   192.168.1.211:6443                                           26m
endpoints/svc1-web     172.16.235.140:80                                            22m
endpoints/svc2-guest   172.16.189.105:8080,172.16.235.141:8080                      22m
endpoints/svc3-admin   172.16.182.43:8080,172.16.189.106:8080,172.16.235.142:8080   22m

svc1("/") 접속

# MASTER=192.168.1.211
# curl $MASTER:$IngHttp
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
... 생략 ...

svc2("/guest") 접속

# for i in {1..100}; do curl -s $MASTER:$IngHttp/guest ; done | sort | uniq -c | sort -nr
     50 Hello Kubernetes bootcamp! | Running on: deploy2-guestsrv-6f7f766988-q9r4j | v=1
     50 Hello Kubernetes bootcamp! | Running on: deploy2-guestsrv-6f7f766988-9hxjw | v=1

svc3("/admin") 접속

# for i in {1..100}; do curl -s $MASTER:$IngHttp/admin | grep Hostname ; done | sort | uniq -c | sort -nr
     34 Hostname: deploy3-adminsrv-7bd5d589b9-snjvg
     33 Hostname: deploy3-adminsrv-7bd5d589b9-vl9lh
     33 Hostname: deploy3-adminsrv-7bd5d589b9-drllf

삭제

# kubectl delete deployments,svc,ingress --all
deployment.apps "deploy1-websrv" deleted
deployment.apps "deploy2-guestsrv" deleted
deployment.apps "deploy3-adminsrv" deleted
service "kubernetes" deleted
service "svc1-web" deleted
service "svc2-guest" deleted
service "svc3-admin" deleted
ingress.networking.k8s.io "ingress-1" deleted

3. Ingress 활용

3-1. Host 기반 라우팅

YAML 템플릿

ingress2.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-2
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /dkos
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: **aiden.com**
    http:
      paths:
      - path: /
        backend:
          serviceName: svc3-admin
          servicePort: 8080
  - host: "***.aiden.com**"
    http:
      paths:
      - path: /echo
        backend:
          serviceName: svc3-admin
          servicePort: 8080

생성 및 확인

생성(파드는 이전 실습때 사용한 svc3-pod.yaml 활용)

# kubectl apply -f ingress2.yaml
ingress.networking.k8s.io/ingress-2 created
# kubectl apply -f svc3-pod.yaml
deployment.apps/deploy3-adminsrv created
service/svc3-admin created

확인

# kubectl get ingress
NAME        CLASS    HOSTS           ADDRESS         PORTS   AGE
ingress-2   <none>   aiden,*.aiden   192.168.1.211   80      71s

# kubectl describe ingress ingress-2
Name:             ingress-2
Namespace:        default
Address:          192.168.1.211
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  aiden       
              /   svc3-admin:8080 ()
  *.aiden     
              /echo   svc3-admin:8080 ()
Annotations:  kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/rewrite-target: /dkos
Events:
  Type    Reason  Age                 From                      Message
  ----    ------  ----                ----                      -------
  Normal  Sync    74s (x2 over 105s)  nginx-ingress-controller  Scheduled for sync

인그레스를 통한 접속 확인

변수 할당

MASTER=192.168.1.211
MYDOMAIN1=aiden.com
MYDOMAIN2=test.aiden.com

/etc/hosts 수정 (도메인 이름으로 접속하기 위해)

cat << EOF >> /etc/hosts
$MASTER $MYDOMAIN1
$MASTER $MYDOMAIN2
EOF

접속 확인

# curl $MYDOMAIN1:$IngHttp

Hostname: deploy3-adminsrv-7bd5d589b9-ppwbc

Pod Information:
        -no pod information available-

Server values:
        server_version=nginx: 1.13.0 - lua: 10008

Request Information:
        client_address=172.16.219.69
        method=GET
... 생략 ...

삭제

# kubectl delete deployments,svc,ingress --all
deployment.apps "deploy3-adminsrv" deleted
service "kubernetes" deleted
service "svc3-admin" deleted
ingress.networking.k8s.io "ingress-2" deleted

3-2. 카나리 업그레이드

YAML 템플릿(디플로이먼트, 서비스)

canary-svc1-pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dp-v1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: svc-v1
  template:
    metadata:
      labels:
        app: svc-v1
    spec:
      containers:
      - name: pod-v1
        image: k8s.gcr.io/echoserver:1.5
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: svc-v1
spec:
  ports:
    - name: web-port
      port: 9001
      targetPort: 8080
  selector:
    app: svc-v1

canary-svc2-pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dp-v2
spec:
  replicas: 3
  selector:
    matchLabels:
      app: svc-v2
  template:
    metadata:
      labels:
        app: svc-v2
    spec:
      containers:
      - name: pod-v2
        image: k8s.gcr.io/echoserver:1.6
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: svc-v2
spec:
  ports:
    - name: web-port
      port: 9001
      targetPort: 8080
  selector:
    app: svc-v2

생성 및 확인(디플로이먼트, 서비스)

생성

# kubectl apply -f canary-svc1-pod.yaml
deployment.apps/dp-v1 created
service/svc-v1 created
# kubectl apply -f canary-svc2-pod.yaml
deployment.apps/dp-v2 created
service/svc-v2 created

확인

# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP    3m57s
svc-v1       ClusterIP   10.105.182.84   <none>        9001/TCP   44s
svc-v2       ClusterIP   10.97.254.145   <none>        9001/TCP   41s

# for pod in $(kubectl get pod -o wide -l app=svc-v1 |awk 'NR>1 {print $6}'); do curl -s $pod:8080 | egrep '(Hostname|nginx)'; done
Hostname: dp-v1-cdd8dc687-2cbk4
        server_version=nginx: 1.13.0 - lua: 10008
Hostname: dp-v1-cdd8dc687-6mnzh
        server_version=nginx: 1.13.0 - lua: 10008
Hostname: dp-v1-cdd8dc687-jsmnh
        server_version=nginx: 1.13.0 - lua: 10008

YAML 템플릿(디플로이먼트, 서비스)

canary-ingress1.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-canary-v1
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: aiden.com
    http:
      paths:
      - backend:
          serviceName: svc-v1
          servicePort: 8080

canary-ingress2.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-canary-v2
  annotations:
    kubernetes.io/ingress.class: "nginx"
    **nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"**
spec:
  rules:
  - host: aiden.com
    http:
      paths:
      - backend:
          serviceName: svc-v2
          servicePort: 8080

생성 및 확인

생성

확인

# for i in {1..100};  do curl -s $MYDOMAIN1:$IngHttp | grep nginx ; done | sort | uniq -c | sort -nr
     90         server_version=nginx: 1.13.0 - lua: 10008
     10         server_version=nginx: 1.13.1 - lua: 10008
# for i in {1..100};  do curl -s $MYDOMAIN1:$IngHttp | grep nginx ; done | sort | uniq -c | sort -nr
     84         server_version=nginx: 1.13.0 - lua: 10008
     16         server_version=nginx: 1.13.1 - lua: 10008

약 90:10 비율로 로드밸런싱 된다.

비율 조정 및 확인

# kubectl get ingress ingress-canary-v2 -o yaml | sed -e "s/weight: \"10\"/weight: \"50\"/" | kubectl apply -f -
ingress.networking.k8s.io/ingress-canary-v2 configured

# for i in {1..100};  do curl -s $MYDOMAIN1:$IngHttp | grep nginx ; done | sort | uniq -c | sort -nr
     50         server_version=nginx: 1.13.1 - lua: 10008
     50         server_version=nginx: 1.13.0 - lua: 10008
# for i in {1..100};  do curl -s $MYDOMAIN1:$IngHttp | grep nginx ; done | sort | uniq -c | sort -nr
     55         server_version=nginx: 1.13.1 - lua: 10008
     45         server_version=nginx: 1.13.0 - lua: 10008

weight를 10→50으로 변경 후 약 50:50 비율로 로드밸런싱 되었다.

삭제

# kubectl delete deployments,svc,ingress --all
deployment.apps "dp-v1" deleted
deployment.apps "dp-v2" deleted
service "kubernetes" deleted
service "svc-v1" deleted
service "svc-v2" deleted
ingress.networking.k8s.io "ingress-canary-v1" deleted
ingress.networking.k8s.io "ingress-canary-v2" deleted

3-3. HTTPS 처리 (TLS 종료)

YAML 템플릿

svc-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-https
  labels:
    app: https
spec:
  containers:
  - name: container
    image: k8s.gcr.io/echoserver:1.6
---
apiVersion: v1
kind: Service
metadata:
  name: svc-https
spec:
  selector:
    app: https
  ports:
  - port: 8080

ssl-termination-ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: https
spec:
  tls:
  - hosts:
    - aiden.com
    secretName: secret-https
  rules:
    - host: aiden.com
      http:
        paths:
        - backend:
            serviceName: svc-https
            servicePort: 8080

생성 및 확인

생성

# kubectl apply -f svc-pod.yaml
pod/pod-https created
service/svc-https created
# kubectl apply -f ssl-termination-ingress.yaml
ingress.networking.k8s.io/https created

인증서 생성

# mkdir key && cd key
# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=aiden.com/O=aiden.com"
Generating a RSA private key
.......................................+++++
...............................+++++
writing new private key to 'tls.key'
-----

Secret 생성

# kubectl create secret tls secret-https --key tls.key --cert tls.crt
secret/secret-https created
# kubectl get secrets secret-https
NAME           TYPE                DATA   AGE
secret-https   kubernetes.io/tls   2      9s
# kubectl get secrets secret-https -o yaml
\apiVersion: v1
data:
... 생략 ...

변수 지정(테스트용)

# export IngHttps=$(kubectl get service -n ingress-nginx ingress-nginx-controller -o jsonpath='{.spec.ports[1].nodePort}')
# echo $IngHttps
31768

확인

# curl -k https://$MYDOMAIN1:$IngHttps

Hostname: pod-https

Pod Information:
        -no pod information available-

Server values:
        server_version=nginx: 1.13.1 - lua: 10008

Request Information:
        client_address=172.16.219.69
... 생략 ...

삭제(Nginx 인그레스 컨트롤러 )

# kubectl delete -f nginx-ingress.yaml

삭제(오브젝트)

# kubectl delete pod,svc,ingress --all
pod "pod-https" deleted
service "kubernetes" deleted
service "svc-https" deleted
ingress.networking.k8s.io "https" deleted

Aiden

기억이 안되면 기록이라도🐳

이전 포스트

[K8s] 인그레스(Ingress)

K8S Study

1. Ingress 란?

2. Ingress 구성

2-1. 디플로이먼트 및 서비스 생성

YAML 템플릿

생성 및 확인

2-2. 인그레스 생성

YAML 템플릿

생성 및 확인

2-3. 인그레스 컨트롤러 생성

YAML 템플릿

생성 및 확인

2-4. 인그레스를 통한 내부 접속 확인

YAML 템플릿

3. Ingress 활용

3-1. Host 기반 라우팅

YAML 템플릿

생성 및 확인

인그레스를 통한 접속 확인

3-2. 카나리 업그레이드

YAML 템플릿(디플로이먼트, 서비스)

생성 및 확인(디플로이먼트, 서비스)

YAML 템플릿(디플로이먼트, 서비스)

생성 및 확인

3-3. HTTPS 처리 (TLS 종료)

YAML 템플릿

생성 및 확인

[K8s] 서비스(Service)

0개의 댓글