📌 쿠버네티스 버전 확인
[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready master 8d v1.19.16
worker1 Ready <none> 8d v1.19.16
worker2 Ready <none> 8d v1.19.16
프로메테우스가 서버에 접근해서 정보를 가져오는 방식 (pull방식)
프로메테우스느 데이터를 수집 그라파나는 그 데이터를 가져다 시각화 시켜주는 것.
쿠버네티스 클러스터 내에 프로메테우스 설치.(pod-kub-state-metrics, Node-exporter;에이전트)
📌 master 1에서 진행하기
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.1/components.yaml
metric server 관련 설치 내용은 kube-system namespace에 설치됨.
[root@master ~]# kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/coredns-f9fd979d6-45rxr 1/1 Running 1 8d
pod/coredns-f9fd979d6-rvnf6 1/1 Running 1 8d
pod/etcd-master 1/1 Running 1 8d
pod/kube-apiserver-master 1/1 Running 1 8d
pod/kube-controller-manager-master 1/1 Running 1 8d
pod/kube-proxy-tmw8f 1/1 Running 1 8d
pod/kube-proxy-vc9wp 1/1 Running 1 8d
pod/kube-proxy-wp8k4 1/1 Running 1 8d
pod/kube-scheduler-master 1/1 Running 1 8d
pod/metrics-server-6594d67d48-zdpb5 0/1 Running 0 3m40s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 8d
service/metrics-server ClusterIP 10.98.109.235 <none> 443/TCP 3m40s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-proxy 3 3 3 3 3 kubernetes.io/os=linux 8d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 2/2 2 2 8d
deployment.apps/metrics-server 0/1 1 0 3m40s
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-f9fd979d6 2 2 2 8d
replicaset.apps/metrics-server-6594d67d48 1 1 0 3m40s
->여기서 확인해보니 정상적으로 실행되지 않고 있음. (metric server)
-> 수정 필요
[root@master ~]# kubectl edit deployment.apps -n kube-system metrics-server
spec 부분에 라인 맞춰서 아래 문구 추가.
--kubelet-insecure-tls
[root@master ~]# kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/coredns-f9fd979d6-45rxr 1/1 Running 1 8d
pod/coredns-f9fd979d6-rvnf6 1/1 Running 1 8d
pod/etcd-master 1/1 Running 1 8d
pod/kube-apiserver-master 1/1 Running 1 8d
pod/kube-controller-manager-master 1/1 Running 1 8d
pod/kube-proxy-tmw8f 1/1 Running 1 8d
pod/kube-proxy-vc9wp 1/1 Running 1 8d
pod/kube-proxy-wp8k4 1/1 Running 1 8d
pod/kube-scheduler-master 1/1 Running 1 8d
pod/metrics-server-6888856cbc-8npjf 1/1 Running 0 89s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 8d
service/metrics-server ClusterIP 10.98.109.235 <none> 443/TCP 14m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-proxy 3 3 3 3 3 kubernetes.io/os=linux 8d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 2/2 2 2 8d
deployment.apps/metrics-server 1/1 1 1 14m
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-f9fd979d6 2 2 2 8d
replicaset.apps/metrics-server-6594d67d48 0 0 0 14m
replicaset.apps/metrics-server-6888856cbc 1 1 1 89s
정상적으로 1/1 running 된 것 확인
[root@master ~]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master 168m 8% 999Mi 27%
worker1 46m 4% 375Mi 42%
worker2 41m 4% 358Mi 40%
[root@master ~]# kubectl top pod ##default에는 아무것도 만들지 않음.
No resources found in default namespace.
[root@master ~]# kubectl top pod -n kube-system
NAME CPU(cores) MEMORY(bytes)
coredns-f9fd979d6-45rxr 4m 13Mi
coredns-f9fd979d6-rvnf6 4m 14Mi
etcd-master 19m 53Mi
kube-apiserver-master 82m 311Mi
kube-controller-manager-master 18m 49Mi
kube-proxy-tmw8f 1m 19Mi
kube-proxy-vc9wp 1m 17Mi
kube-proxy-wp8k4 1m 19Mi
kube-scheduler-master 3m 21Mi
metrics-server-6888856cbc-8npjf 3m 17Mi
워커노드 테인트 없는 것확인
[root@master ~]# kubectl describe node worker1 | grep Taints
Taints: <none>프로메테우스 설치 전 네임스페이스 생성
[root@master ~]# kubectl create ns monitoring
namespace/monitoring created
[root@master ~]# kubectl get ns
NAME STATUS AGE
default Active 8d
kube-flannel Active 8d
kube-node-lease Active 8d
kube-public Active 8d
kube-system Active 8d
monitoring Active 4s
프로메테우스 설치
[root@master ~]# yum install -y git
[root@master ~]# git clone https://github.com/hali-linux/my-prometheus-grafana.git
[root@master ~]# ls
anaconda-ks.cfg my-prometheus-grafana
[root@master ~]# cd my-prometheus-grafana/
[root@master my-prometheus-grafana]# ll
total 44
-rw-r--r-- 1 root root 1006 Aug 9 15:12 grafana.yaml
<!-- 역할주기
-rw-r--r-- 1 root root 1811 Aug 9 15:12 kube-state-cluster-role.yaml
-rw-r--r-- 1 root root 932 Aug 9 15:12 kube-state-deployment.yaml
-rw-r--r-- 1 root root 98 Aug 9 15:12 kube-state-svcaccount.yaml
-rw-r--r-- 1 root root 330 Aug 9 15:12 kube-state-svc.yaml
-->
-rw-r--r-- 1 root root 652 Aug 9 15:12 prometheus-cluster-role.yaml ## 쿠버네티스 api접근해서 셋팅하기 위한 역할 부여해주는 yaml (첫번재 설치)
-rw-r--r-- 1 root root 5344 Aug 9 15:12 prometheus-config-map.yaml
-rw-r--r-- 1 root root 930 Aug 9 15:12 prometheus-deployment.yaml
-rw-r--r-- 1 root root 710 Aug 9 15:12 prometheus-node-exporter.yaml # 워커 노드들에 대한 자원 수집. (하드웨어 모니터링 - daemon set으로 구성.)
-rw-r--r-- 1 root root 308 Aug 9 15:12 prometheus-svc.yaml # 프로메테우스는 GUI. 진입하기 위한 서비스.
->파일 설치 순서는 따로 없지만 프로메테우스 먼저, 그다음에 kube-state 야믈 설치.
yaml 파일 자체에 네임스페이스 설정되어 있어서 따로 명령어 옵션 추가해주지 않아도 됨.
[root@master my-prometheus-grafana]# kubectl apply -f prometheus-cluster-role.yaml
[root@master my-prometheus-grafana]# kubectl apply -f prometheus-config-map.yaml
[root@master my-prometheus-grafana]# kubectl apply -f prometheus-deployment.yaml
[root@master my-prometheus-grafana]# kubectl apply -f prometheus-node-exporter.yaml
[root@master my-prometheus-grafana]# kubectl apply -f prometheus-svc.yaml
설치 확인
[root@master my-prometheus-grafana]# kubectl get pod -n monitoring
NAME READY STATUS RESTARTS AGE
node-exporter-78pxz 1/1 Running 0 3m2s
node-exporter-db7wm 1/1 Running 0 3m2s
prometheus-deployment-6fc48d76-gswqb 1/1 Running 0
[root@master my-prometheus-grafana]# kubectl get pod -n monitoring -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-exporter-78pxz 1/1 Running 0 4m17s 10.244.1.3 worker1 <none> <none>
node-exporter-db7wm 1/1 Running 0 4m17s 10.244.2.4 worker2 <none> <none>
prometheus-deployment-6fc48d76-gswqb 1/1 Running 0 6m5s 10.244.2.3 worker2 <none> <none>
4m50s
kube-state 설치
[root@master my-prometheus-grafana]# kubectl apply -f kube-state-cluster-role.yaml
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
[root@master my-prometheus-grafana]# kubectl apply -f kube-state-deployment.yaml
deployment.apps/kube-state-metrics created
[root@master my-prometheus-grafana]# kubectl apply -f kube-state-svcaccount.yaml
serviceaccount/kube-state-metrics created
[root@master my-prometheus-grafana]# kubectl apply -f kube-state-svc.yaml
service/kube-state-metrics created
->kube-system이라는 ns 에 들어가있음.
잘 러닝되고 있는지 확인
프로메테우스 진입하기 위한 포트번호 확인 및 전체적으로 확인.
[root@master my-prometheus-grafana]# kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/node-exporter-78pxz 1/1 Running 0 12m
pod/node-exporter-db7wm 1/1 Running 0 12m
pod/prometheus-deployment-6fc48d76-gswqb 1/1 Running 0 14m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/prometheus-service NodePort 10.101.238.230 <none> 8080:30003/TCP 10m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-exporter 2 2 2 2 2 <none> 12m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/prometheus-deployment 1/1 1 1 14m
NAME DESIRED CURRENT READY AGE
replicaset.apps/prometheus-deployment-6fc48d76 1 1 1 14m
->node port 30003 포트
-> 보기 쉽지 않음. grafana 설치해보자.
[root@master my-prometheus-grafana]# kubectl apply -f grafana.yaml
deployment.apps/grafana created
service/grafana created
[root@master my-prometheus-grafana]# kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/grafana-5d479cc7dc-hmd98 1/1 Running 0 45s
pod/node-exporter-78pxz 1/1 Running 0 24m
pod/node-exporter-db7wm 1/1 Running 0 24m
pod/prometheus-deployment-6fc48d76-gswqb 1/1 Running 0 26m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grafana NodePort 10.111.205.242 <none> 3000:30004/TCP 44s
service/prometheus-service NodePort 10.101.238.230 <none> 8080:30003/TCP 22m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-exporter 2 2 2 2 2 <none> 24m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 45s
deployment.apps/prometheus-deployment 1/1 1 1 26m
NAME DESIRED CURRENT READY AGE
replicaset.apps/grafana-5d479cc7dc 1 1 1 45s
replicaset.apps/prometheus-deployment-6fc48d76 1 1 1 26m
-> 포트번호 30004
[root@master my-prometheus-grafana]# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana NodePort 10.111.205.242 <none> 3000:30004/TCP 22m
prometheus-service NodePort 10.101.238.230 <none> 8080:30003/TCP 44m
http://10.101.238.230:8080 -> 프로메테우스의 클러스터 IP : 8080
그라파나 홈페이지에서 필요한 대시보드 설치하기
대시보드 창에서 검색창에 kubernetes 검색 - cluster monitering for kubernetes 
다른거 import 해보기
📌 오토스케일링 실습(HPA : Horizontal Pod Autoscaler)
[root@master my-prometheus-grafana]# cd ~
[root@master ~]# mkdir hpa && cd $_
[root@master hpa]#
[root@master hpa]# vi php-apache.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
spec:
selector:
matchLabels:
run: php-apache
replicas: 2 ## desired capacity 시작 갯수
template:
metadata:
labels:
run: php-apache
spec:
containers:
- name: php-apache
image: k8s.gcr.io/hpa-example ##부함 올리기 좋은 이미지. 검토할ㄸ내느 이미지만 바꿔주면 됨.
ports:
- containerPort: 80 ## 정보성
resources:
limits:
cpu: 500m ## 한도
requests:
cpu: 200m ## 최소사항
---
apiVersion: v1
kind: Service
metadata:
name: php-apache
labels:
run: php-apache
spec:
ports:
- port: 80 ##실제 포트 설정
selector:
run: php-apache
[root@master hpa]# kubectl apply -f php-apache.yaml
[root@master hpa]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/php-apache-d4cf67d68-ppkbl 1/1 Running 0 54s
pod/php-apache-d4cf67d68-rdjj5 1/1 Running 0 54s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
service/php-apache ClusterIP 10.98.204.31 <none> 80/TCP 54s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/php-apache 2/2 2 2 54s
NAME DESIRED CURRENT READY AGE
replicaset.apps/php-apache-d4cf67d68 2 2 2 54s
[root@master hpa]# vi hpa.yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
spec:
maxReplicas: 4
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
targetCPUUtilizationPercentage: 50
status:
currentCPUUtilizationPercentage: 0
currentReplicas: 2
desiredReplicas: 2
[root@master hpa]# kubectl apply -f hpa.yaml
[root@master hpa]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/php-apache-d4cf67d68-ppkbl 1/1 Running 0 25m
pod/php-apache-d4cf67d68-rdjj5 1/1 Running 0 25m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8d
service/php-apache ClusterIP 10.98.204.31 <none> 80/TCP 25m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/php-apache 2/2 2 2 25m
NAME DESIRED CURRENT READY AGE
replicaset.apps/php-apache-d4cf67d68 2 2 2 25m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/php-apache Deployment/php-apache <unknown>/50% 1 4 0 14s
[root@master hpa]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 4 2 64s
[root@master ~]# kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
->부하 발생시키기 ; http://php-apache 쿠버네티스의 자체 dns ; coredns .
똑같은 탭 하나 더 만들어서 확인하기
kubectl get hpa
첫번째 탭에서 control + c 부하중단
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 4 1 23m
📙 ✔️✏️📢⭐️📌
꾸준히, 차근차근