Cloudwatch Pod Logging 생성 - FluentD

https://medium.com/attest-product-and-technology/kubernetes-logs-to-aws-cloudwatch-with-fluentd-ede8d88a1b4e

1. Fluentd 란?

• 로그 수집하고 저장소에 저장하는 로그 적재기
• 서로 다른 애플리케이션에서 로그를 수집하고 트래픽을 조정해 로그저장소에 로그를 수집힌다.
• 경량화된 버전으로 http, tcp 등 다양한 데이터를 수집가능하다,.

참고 - https://ta-starter.tistory.com/41

[사전 조건]

• EKS Cluster에 접속 할 환경에 eksctl, kubectl, git 설치
• EKS Node Role에 Cloudwatch Logs 전체 권한(CloudwatchAgentServerPolicy) 할당

※중요※ 일반 Worker Nodegroup에 해당사항, daemonSet의 경우 fargate는 적용 방법이 다르다
fargate - https://aws.amazon.com/ko/blogs/containers/how-to-capture-application-logs-when-using-amazon-eks-on-aws-fargate/

실습

1. Namespace 생성합니다.

mkdir -p cloudwatch/flutenD && cd cloudwatch/flutenD

cat << EOF > cloudwatch.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: amazon-cloudwatch
  labels:
    name: amazon-cloudwatch
kubectl apply -f cloudwatch.yaml

2. iamserviceaccount 생성

eksctl create iamserviceaccount --name cwagent-prometheus --namespace amazon-cloudwatch --cluster <cluster-name> --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy --approve --override-existing-serviceaccounts

3. ConfigMap 생성

kubectl create configmap cluster-info --from-literal=cluster.name=<cluster name> --from-literal=logs.region=<region-code> -n amazon-cloudwatch

4. Fluentd 배포(수정 본)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: amazon-cloudwatch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd-role
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - pods/logs
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd-role
subjects:
  - kind: ServiceAccount
    name: fluentd
    namespace: amazon-cloudwatch
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: amazon-cloudwatch
  labels:
    k8s-app: fluentd-cloudwatch
data:
  fluent.conf: |
    @include containers.conf #아래에서 container.conf 설정 부분을 include한다는 의미

    <match fluent.**>
      @type null #수정 가능 출력 방식임(stdout)
    </match>
  containers.conf: |
    <source>
      @type tail
      @id in_tail_container_logs
      @label @containers
      path /var/log/containers/wscf-deployment-*.log #Pod 로그만 지정
 #또는path /var/log/containers/*default*.log #Default Namespace에 대한 모든 Pod에 log를 축출
      tag skills #뭔지 모르겠음 그런데 * -> 다른 문자열로 변경하면 Log가 적어짐
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>
    <label @containers>
      <filter **>
        @type kubernetes_metadata
        @id filter_kube_metadata
      </filter>

      <filter **>
        @type record_transformer
        @id filter_containers_stream_transformer
        <record>
          stream_name deploymentpodlog #여기서 Stream 이름 지정
        </record>
      </filter>
      <filter **>
        @type concat
        key log
        multiline_start_regexp /^\S/
        separator ""
        flush_interval 5
        timeout_label @NORMAL
      </filter>

      <match **>
        @type relabel
        @label @NORMAL
      </match>
    </label>

    <label @NORMAL>
      <match **>
        @type cloudwatch_logs
        @id out_cloudwatch_logs_containers
        region "#{ENV.fetch('REGION')}"
        log_group_name "wsi/aws/eks" #LogGroupName
        log_stream_name_key stream_name
        remove_log_stream_name_key true
        auto_create_stream true
        <buffer>
          flush_interval 5
          chunk_limit_size 2m
          queued_chunks_limit_size 32
          retry_forever true
        </buffer>
      </match>
    </label>
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-cloudwatch
  namespace: amazon-cloudwatch
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-cloudwatch
  template:
    metadata:
      labels:
        k8s-app: fluentd-cloudwatch
      annotations:
        configHash: 8915de4cf9c3551a8dc74c0137a3e83569d28c71044b0359c2578d2e0461825
    spec:
      serviceAccountName: fluentd
      terminationGracePeriodSeconds: 30
      initContainers:
        - name: copy-fluentd-config
          image: busybox
          command: ['sh', '-c', 'cp /config-volume/..data/* /fluentd/etc']
          volumeMounts:
            - name: config-volume
              mountPath: /config-volume
            - name: fluentdconf
              mountPath: /fluentd/etc
        - name: update-log-driver
          image: busybox
          command: ['sh','-c','']
      containers:
        - name: fluentd-cloudwatch
          image: fluent/fluentd-kubernetes-daemonset:v1.7.3-debian-cloudwatch-1.0
          env:
            - name: REGION
              valueFrom:
                configMapKeyRef:
                  name: cluster-info
                  key: logs.region
            - name: CLUSTER_NAME
              valueFrom:
                configMapKeyRef:
                  name: cluster-info
                  key: cluster.name
            - name: CI_VERSION
              value: "k8s/1.0.1"
          resources:
            limits:
              memory: 400Mi
            requests:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - name: config-volume
              mountPath: /config-volume
            - name: fluentdconf
              mountPath: /fluentd/etc
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: runlogjournal
              mountPath: /run/log/journal
              readOnly: true
            - name: dmesg
              mountPath: /var/log/dmesg
              readOnly: true
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-config
        - name: fluentdconf
          emptyDir: {}
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: runlogjournal
          hostPath:
            path: /run/log/journal
        - name: dmesg
          hostPath:
            path: /var/log/dmesg

출력

• 최후에 방법은 이제

  kubectl logs -f [PodName] > pods.log #이렇게 한다음 awslogs 사용해서 Cloudwatch Log Group으로 Log 생성