관리 클러스터 - Ingress Nginx
권한 획득
# IP를 예약하려는 지역으로 사용자 주체 권한을 설정
vi ~/.oci/oci_cli_rc
[OCI_CLI_SETTINGS]
default_profile=SEOUL
oci os ns get # 확인
IP 예약
- nlb-ip: IP 예약 수신 컨트롤러에 할당된 외부 IP는 모든 수신 트래픽이 라우팅되어야 하는 IP
oci network public-ip create --lifetime RESERVED --compartment-id $OCI_COMPARTMENT_ID --display-name nlb-ip
"ip-address": "158.179.160.98",DNS 레코드 등록
루트 도메인(code-lab.kr)에 대한 A 레코드가 존재해야 인증서 발행자 상태가 true 가 될 수 있음
제어하는 DNS 영역(예: xxx.code-lab.kr)에 대한 A 레코드로 예약한 IP를 등록
헬름 설치
brew install kubernetes-helm
--
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 && \
chmod 700 get_helm.sh && ./get_helm.sh
인그레스
NGINX 인그레스 컨트롤러 배포
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
# helm show values ingress-nginx/ingress-ngin > value.yaml
# network load balancer (L4)
# helm delete ingress-nginx -n ingress-nginx
# helm uninstall ingress-nginx --namespace ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx --create-namespace \
--set controller.service.annotations."oci\.oraclecloud\.com/load-balancer-type"="nlb" \
--set controller.service.annotations."oci-network-load-balancer\.oraclecloud\.com/security-list-management-mode"="All" \
--set controller.service.externalTrafficPolicy=Local \
--set controller.service.loadBalancerIP="158.179.160.98"
➜ kubectl get all --namespace ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-controller-6858749594-b7bhr 1/1 Running 0 103s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.96.70.153 10.0.20.141,158.179.160.98 80:31635/TCP,443:32017/TCP 103s
service/ingress-nginx-controller-admission ClusterIP 10.96.188.123 <none> 443/TCP 103s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 103s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-6858749594 1 1 1 104s
# 인그레스 컨트롤러 설치 확인
kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watchcert-manager(인증서 관리자) 설치
helm repo add jetstack https://charts.jetstack.io
helm repo update
# helm delete cert-manager -n cert-manager
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager --create-namespace \
--version v1.15.3 \
--set crds.enabled=true
# cert-manager 설치 확인
kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-74654c4948-7zqwp 1/1 Running 0 5h7m
cert-manager-cainjector-77644bff8-fwwg4 1/1 Running 0 5h7m
cert-manager-webhook-54d7657dbb-vms6g 1/1 Running 0 5h7m
cluster-issuer(인증서 발행자) 생성
# cluster-issuer.yaml
cat <<EOT > cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: master@code-lab.kr
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging-tls
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: master@code-lab.kr
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod-tls
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
EOT
# kubectl delete -f cluster-issuer.yaml
k apply -f cluster-issuer.yaml
clusterissuer.cert-manager.io/letsencrypt-staging created
clusterissuer.cert-manager.io/letsencrypt-prod created
# clusterissuer 확인
kubectl get clusterissuer
NAME READY AGE
letsencrypt-prod True 4m19s
letsencrypt-staging True 4m19s
kubectl describe clusterissuer letsencrypt-staging
kubectl describe clusterissuer letsencrypt-prod
테스트용 서비스 생성
➜ kubectl create ns test
➜ kubectl create deploy nginx --image=nginx --replicas=2 --namespace test
deployment.apps/nginx created
➜ kubectl expose deploy nginx --port=80 --target-port=80 --namespace test
ingress 생성 - prod
# ingress-nginx-prod.yaml
cat <<EOT> ingress-test-prod.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-prod
namespace: test
annotations:
acme.cert-manager.io/http01-ingress-class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-body-size: 999m # 동영상 파일 업로드시 413 에러가 발생하여 추가
spec:
ingressClassName: nginx
rules:
- host: www.code-lab.kr
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: Prefix
- host: code-lab.kr
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- www.code-lab.kr
- code-lab.kr
secretName: test-prod-tls
EOT
kubectl apply -f ingress-test-prod.yaml
# 시간지나도 true로 안 바뀌면 도메인과 ip가 도메인 영역에 등록되었는지 확인
➜ kubectl get certificate --namespace test
NAME READY SECRET AGE
test-prod-tls True test-prod-tls 3m38s
➜ kubectl describe certificate test-prod-tls
➜ kubectl describe secret test-prod-tls
➜ kubectl get ingress --namespace test
NAME CLASS HOSTS ADDRESS PORTS AGE
test-prod nginx www.code-lab.kr,code-lab.kr 10.0.20.203,146.56.188.188 80, 443 4m49sNLB 건전성 검사 수정
- oci 콘솔에서 nlb 확인해서 건정성 검사 프로토콜 변경(http -> tcp)하면 건전성 정상으로 변경됨
참조
주요 참조
공식문서-cert-manager-nginx-ingress-튜토리얼
블로그-한글-전체 컨셉을 잡게 해줌-그대로 참고할 수는 없음
블로그-영어-기본 개념을 쉽게 설명해줌
보조 참조
https://github.com/kubernetes/ingress-nginx
https://github.com/oracle/oci-cloud-controller-manager/blob/master/docs/tutorial-ssl.md
https://github.com/oracle/oci-cloud-controller-manager/blob/master/docs/load-balancer-annotations.md
Think about a better architecture