공식문서 컴포넌트 참고

구조

공식문서_로키 아키텍처
참고블로그- 로키 아키
Single Store
Loki stores all data in a single object storage backend. This mode of operation became generally available with Loki 2.0 and is fast, cost-effective, and simple, not to mention where all current and future development lies. This mode uses an adapter called boltdb_shipper to store the index in object storage (the same way we store chunks).

• Read Path

1. The querier receives an HTTP/1 request for data.
2. The querier passes the query to all ingesters for in-memory data.
3. The ingesters receive the read request and return data matching the query, if any.
4. The querier lazily loads data from the backing store and runs the query against it if no ingesters returned data.
5. The querier iterates over all received data and deduplicates, returning a final set of data over the HTTP/1 connection.

• Write Path

1. The distributor receives an HTTP/1 request to store data for streams.
2. Each stream is hashed using the hash ring.
3. The distributor sends each stream to the appropriate ingesters and their replicas (based on the configured replication factor).
4. Each ingester will create a chunk or append to an existing chunk for the stream’s data. A chunk is unique per tenant and per labelset.
5. The distributor responds with a success code over the HTTP/1 connection.

해쉬링 참고글

설치

IAM 롤 생성

loki-s3-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LokiStorage",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>",
                "arn:aws:s3:::<bucket_name>/*"
            ]
        }
    ]
}

trust-relationship.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${계정번호}:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/${oidc}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/${oidc}:sub": "system:serviceaccount:loki-stack:loki-sa",
                    "oidc.eks.us-east-1.amazonaws.com/id/${oidc}:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

로키스택 배포

프로메테우스 스택은 미리 배포 된 것을 사용

prometheus-values.yaml
아래에 로키스택을 추가해준다.

grafana: 
  additionalDataSources:        
    - name: Loki
      type: loki
      access: proxy
      isDefault: false
      url: http://loki-stack.loki-stack:3100/
      jsonData:
        timeInterval: 30s

그라파나 로키 스택 배포

helm install loki-stack grafana/loki-stack \
  --values values.yaml \
  --namespace loki-stack \
  --create-namespace 
  
kubectl -n loki-stack get sa

values.yaml

loki:
  auth_enabled: false
  commonConfig:
    path_prefix: /var/loki
    replication_factor: 1
  compactor:
    apply_retention_interval: 1h
    compaction_interval: 5m
    retention_delete_worker_count: 500
    retention_enabled: true
    shared_store: s3
    working_directory: /data/compactor
  config:  
      schema_config:
        configs:
        - from: 2020-05-15
          store: boltdb-shipper # https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/
          object_store: s3
          schema: v11
          index:
            period: 24h
            prefix: loki_index_
        
      storage_config:
        aws:
          region: ${자신의 s3 리전}
          bucketnames: ${자신의 s3 이름}
          s3forcepathstyle: false
          #s3forcepathstyle: true  <-- This is the main culprit; comment it out ? -? https://github.com/grafana/loki/issues/7024
        boltdb_shipper:
          shared_store: s3
          cache_ttl: 24h
  serviceAccount:
    create: true
    name: loki-sa
    annotations:
       eks.amazonaws.com/role-arn: "${iam role arn}"
  write:
     replicas: 2
  read:
    replicas: 1

grafana:
    enabled: false

s3 확인

grafana 확인

16.PLG % kubectl logs nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/09/26 06:14:55 [notice] 1#1: using the "epoll" event method
2023/09/26 06:14:55 [notice] 1#1: nginx/1.25.2
2023/09/26 06:14:55 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14) 
2023/09/26 06:14:55 [notice] 1#1: OS: Linux 5.10.186-179.751.amzn2.x86_64
2023/09/26 06:14:55 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/09/26 06:14:55 [notice] 1#1: start worker processes
2023/09/26 06:14:55 [notice] 1#1: start worker process 29
2023/09/26 06:14:55 [notice] 1#1: start worker process 30

로그를 저렇게 보는것과 똑같이 나옴!

LogQL

쿼리 공식문서
Cheat Sheet

---

참고

로키 - 쿠버네티스 로깅