Kubernetes Monitoring & Logging

---

Kubernetes설치

---

HELM 설치

---

Prometheus Monitoring

https://soundcloud.com/
프로메테우스가 원래 사용되었던 서비스가 soundcloud였는데 오픈소스로 공개했다고 한다.
쿠버네티스만을 위한 구성은 아니고 다양한 환경에서 사용될 수 있다.

CPU, Memoty, Network IO, Disk IO

• Heapster + InfluxDB: X
  - metrics-server: DB 없음, 실시간
  - CPU, Memory
  - Prometheus

https://prometheus.io/docs/introduction/overview/

TSDB는 Time Series DB
시계열 DB
관계형 데이터베이스가 아니라 시간축에 대한 값으로 정리해놓은 것이 시계열 DB라고 한다.
Grafana는 데이터 시각화 도구

https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

prom-values.yaml

grafana:
  service:
    type: LoadBalancer

kubectl create ns monitor

helm install prom prometheus-community/kube-prometheus-stack -f prom-values.yaml -n monitor

### -n옵션은 monitor라는 namespace에 만든다는 뜻

크롬에 접속
http://192.168.100.24X
ID: admin
PWD: prom-operator

---

EFK Logging

helm uninstall prom -n monitor
이런식으로 이전에 설치했던것들은 제거를해서 cpu와 memory를 좀 확보하고 설치하기를 권장함

ELK Stack: Elasticsearch + Logstash + Kibana
EFK Stack: Elasticsearch + Fluentd + Kibana <<이번에 사용할 구성
Elasticsearch + Fluent Bit + Kibana
Elastic Stack: Elasticsearch + Beat + Kibana

Elasticsearch

helm repo add elastic https://helm.elastic.co
helm repo update

helm show values elastic/elasticsearch > es-value.yaml

es-value.yaml

 18 replicas: 1
 19 minimumMasterNodes: 1
 
 80 resources:
 81   requests:
 82     cpu: "500m"
 83     memory: "1Gi"
 84   limits:
 85     cpu: "500m"
 86     memory: "1Gi"

kubectl create ns logging

helm install elastic elastic/elasticsearch -f es-value.yaml -n logging

Fluent Bit

https://github.com/fluent/fluent-bit-kubernetes-logging

git clone https://github.com/fluent/fluent-bit-kubernetes-logging.git

cd  fluent-bit-kubernetes-logging

kubectl create -f fluent-bit-service-account.yaml
kubectl create -f fluent-bit-role-1.22.yaml
kubectl create -f fluent-bit-role-binding-1.22.yaml

kubectl create -f output/elasticsearch/fluent-bit-configmap.yaml

output/elasticsearch/fluent-bit-ds.yaml

 32         - name: FLUENT_ELASTICSEARCH_HOST
 33           value: "elasticsearch-master"

kubectl create -f output/elasticsearch/fluent-bit-ds.yaml

Kibana

helm show values elastic/kibana > kibana-value.yaml

kibana-value.yaml

 49 resources:
 50   requests:
 51     cpu: "500m"
 52     memory: "1Gi"
 53   limits:
 54     cpu: "500m"
 55     memory: "1Gi"
 
119 service:
120   type: LoadBalancer

helm install kibana elastic/kibana -f kibana-value.yaml -n logging

http://192.168.100.X:5601

• 햄버거 -> Management -> Stack Management
  - Kibana -> Index Pattern
  - Create Index Pattern 우상
  - Name: logstash-*
  - Timestamp: @timestamp
• 햄버거 -> Analystics -> Discover

---

로깅 실습

kubectl apply -f https://kubernetes.io/examples/contorllers/job.yaml
kubectl get po
kubens default
kubectl get po -n logging

kubectl get po -n -o wide
ssh 192.168.100.101				#node2로 접속

sudo -i
cd /var/log/
ls
cd containers/
ls

### 보면 pi값을 계산한 log가 있다.

cd logging_pi--1-1[tab]
ls
cd pi/
ls

cat 0.log

Pod가 제거되더라도 로그는 계속 이친구가 저장하게된다.