poc1013a

Young-Kyoo Kim·2025년 10월 12일

🎓 모범 사례 (Best Practices)

1. Scrape 간격 최적화

# 빠른 감지가 필요한 메트릭 - 짧은 간격
scrape_configs:
  - job_name: 'minio-cluster-health'
    scrape_interval: 30s        # 클러스터 건강 상태
    scrape_timeout: 10s
    
  - job_name: 'minio-api-requests'
    scrape_interval: 30s        # API 성능 메트릭
    
  - job_name: 'minio-system-drives'
    scrape_interval: 1m         # 드라이브 상태
    
# 변화가 느린 메트릭 - 긴 간격
  - job_name: 'minio-cluster-usage'
    scrape_interval: 5m         # 용량 사용량
    
  - job_name: 'minio-cluster-iam'
    scrape_interval: 2m         # IAM 상태

권장 간격:

  • Critical metrics (health, erasure-set): 30초
  • Performance metrics (API, requests): 30초
  • Drive metrics: 1분
  • Usage metrics: 5분
  • IAM/Config metrics: 2-5분

2. 레이블 활용

# ❌ 나쁜 예: 모든 데이터 조회
minio_cluster_health_drives_online_count

# ✅ 좋은 예: 특정 서버만 조회
minio_cluster_health_drives_online_count{server="node-1"}

# ✅ 특정 pool/set 조회
minio_cluster_erasure_set_health{pool_id="0", set_id="0"}

# ✅ 특정 버킷만 조회
minio_bucket_api_total{bucket="important-data"}

# ✅ 정규표현식 활용
minio_api_requests_total{name=~"s3\\.Get.*"}  # 모든 GET 요청

# ✅ 여러 조건 조합
minio_bucket_api_5xx_errors_total{
  bucket=~"prod-.*",
  server="node-1"
}

레이블 활용 팁:

  • 불필요한 전체 조회 피하기
  • 필요한 차원만 선택하여 카디널리티 감소
  • 정규표현식으로 유사 메트릭 그룹화
  • 레이블 기반 alert 작성

3. Alert 피로도 방지

groups:
  - name: minio_alerts
    rules:
      # ❌ 나쁜 예: 즉시 알림
      - alert: MinIODriveOffline
        expr: minio_cluster_health_drives_offline_count > 0
        labels:
          severity: critical

      # ✅ 좋은 예: 일시적 스파이크 무시
      - alert: MinIODriveOffline
        expr: minio_cluster_health_drives_offline_count > 0
        for: 5m  # 5분간 지속되어야 알림
        labels:
          severity: critical

      # ✅ 더 좋은 예: 중요도에 따른 시간 차등
      - alert: MinIODrivesMultipleOffline
        expr: minio_cluster_health_drives_offline_count >= 2
        for: 2m  # 여러 드라이브 오프라인은 빠르게 알림
        labels:
          severity: critical

      - alert: MinIODriveSingleOffline
        expr: minio_cluster_health_drives_offline_count == 1
        for: 10m  # 단일 드라이브는 여유 있게
        labels:
          severity: warning

Alert 피로도 방지 전략:
1. 적절한 for 절 사용: 일시적 장애 무시
2. Severity 계층화: critical / warning / info
3. Alert grouping: 관련 알림 묶기
4. Threshold 조정: 너무 민감하지 않게
5. Silence 활용: 유지보수 시 일시 중지

# AlertManager에서 grouping 설정
route:
  group_by: ['alertname', 'cluster', 'component']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  
  routes:
    # Critical은 즉시
    - match:
        severity: critical
      receiver: 'pagerduty'
      group_wait: 10s
      repeat_interval: 1h
    
    # Warning은 그룹으로
    - match:
        severity: warning
      receiver: 'slack'
      group_interval: 10m
    
    # Info는 배치로
    - match:
        severity: info
      receiver: 'email'
      group_interval: 1h
      repeat_interval: 24h

4. High Cardinality 주의

# ❌ 문제: 모든 버킷 메트릭 수집
scrape_configs:
  - job_name: 'minio-bucket-api'
    metrics_path: /minio/metrics/v3/bucket/api
    # buckets 파라미터 없음 = 모든 버킷!

# ✅ 해결: 중요한 버킷만 선택
scrape_configs:
  - job_name: 'minio-bucket-api'
    metrics_path: /minio/metrics/v3/bucket/api
    params:
      buckets: ['prod-data', 'critical-backups', 'user-uploads']
    # 3개 버킷만 모니터링

High Cardinality 관리:

# 1. 불필요한 메트릭 제외
scrape_configs:
  - job_name: 'minio-system-drives'
    metrics_path: /minio/metrics/v3/system/drive
    static_configs:
      - targets: ['minio:9000']
    
    # 캐시 메트릭 제외
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'minio_system_drive_cache_.*'
        action: drop

# 2. 특정 레이블만 유지
    metric_relabel_configs:
      - source_labels: [__name__, drive]
        regex: 'minio_system_drive_health;/data.*'
        action: keep  # /data로 시작하는 드라이브만

# 3. 레이블 값 단순화
    metric_relabel_configs:
      - source_labels: [drive]
        regex: '/mnt/data(\d+)/.*'
        target_label: drive_id
        replacement: 'drive-$1'

카디널리티 모니터링:

# 시계열 수 확인
count(minio_system_drive_health)

# 레이블별 카디널리티
count(minio_system_drive_health) by (__name__)
count(minio_system_drive_health) by (server)
count(minio_system_drive_health) by (drive)

5. 보안 고려사항

# ✅ TLS 사용
scrape_configs:
  - job_name: 'minio-metrics-secure'
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/certs/ca.crt
      cert_file: /etc/prometheus/certs/client.crt
      key_file: /etc/prometheus/certs/client.key
      insecure_skip_verify: false  # 인증서 검증 활성화
    
    # Bearer token을 파일로 관리 (환경변수보다 안전)
    bearer_token_file: /etc/prometheus/secrets/minio_bearer_token

# ❌ 나쁜 예: 토큰 하드코딩
    bearer_token: 'eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9...'  # 절대 금지!

Bearer 토큰 관리:

# 1. 토큰 생성
mc admin prometheus generate minio-prod > /tmp/minio_token.yml

# 2. 토큰만 추출
grep 'bearer_token:' /tmp/minio_token.yml | \
  awk '{print $2}' > /etc/prometheus/secrets/minio_bearer_token

# 3. 권한 설정
chmod 600 /etc/prometheus/secrets/minio_bearer_token
chown prometheus:prometheus /etc/prometheus/secrets/minio_bearer_token

# 4. Kubernetes Secret (K8s 환경)
kubectl create secret generic minio-bearer-token \
  --from-file=token=/etc/prometheus/secrets/minio_bearer_token \
  -n monitoring

Kubernetes에서 Secret 사용:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    scrape_configs:
      - job_name: 'minio'
        bearer_token_file: /etc/prometheus/secrets/minio-token
        
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  template:
    spec:
      containers:
      - name: prometheus
        volumeMounts:
        - name: minio-token
          mountPath: /etc/prometheus/secrets
          readOnly: true
      volumes:
      - name: minio-token
        secret:
          secretName: minio-bearer-token

토큰 교체 정책:

  • 정기적으로 토큰 재생성 (예: 90일마다)
  • 침해 의심 시 즉시 재생성
  • 토큰 버전 관리

6. 다중 클러스터 모니터링

# 방법 1: Global external labels
global:
  scrape_interval: 30s
  external_labels:
    cluster: 'production-minio-us-east'
    region: 'us-east-1'
    environment: 'production'
    datacenter: 'dc1'

scrape_configs:
  - job_name: 'minio-cluster-health'
    static_configs:
      - targets: ['minio-us-east:9000']
    # 자동으로 위 레이블이 추가됨

---
# 방법 2: Job별 레이블
scrape_configs:
  - job_name: 'minio-prod-us-east'
    static_configs:
      - targets: ['prod-minio-us-east:9000']
        labels:
          environment: 'production'
          cluster: 'us-east'
          region: 'us-east-1'
  
  - job_name: 'minio-prod-eu-west'
    static_configs:
      - targets: ['prod-minio-eu-west:9000']
        labels:
          environment: 'production'
          cluster: 'eu-west'
          region: 'eu-west-1'
  
  - job_name: 'minio-staging'
    static_configs:
      - targets: ['staging-minio:9000']
        labels:
          environment: 'staging'
          cluster: 'staging'

---
# 방법 3: Relabeling으로 동적 레이블
scrape_configs:
  - job_name: 'minio-multi-cluster'
    static_configs:
      - targets: 
          - 'prod-us-east-minio:9000'
          - 'prod-eu-west-minio:9000'
          - 'staging-minio:9000'
    
    relabel_configs:
      # target에서 환경 추출
      - source_labels: [__address__]
        regex: '(prod|staging)-(.*)-minio:.*'
        target_label: environment
        replacement: '$1'
      
      # target에서 리전 추출
      - source_labels: [__address__]
        regex: '.*-(us-east|eu-west)-.*'
        target_label: region
        replacement: '$1'

다중 클러스터 쿼리:

# 모든 클러스터 합계
sum(minio_cluster_health_drives_online_count)

# 클러스터별 합계
sum(minio_cluster_health_drives_online_count) by (cluster)

# 특정 환경만
minio_cluster_health_drives_online_count{environment="production"}

# 리전 간 비교
sum(minio_api_requests_total) by (region)

# 환경별 에러율 비교
sum(rate(minio_api_requests_5xx_errors_total[5m])) by (environment) /
sum(rate(minio_api_requests_total[5m])) by (environment)

Federation 구성 (중앙 집중식):

# 중앙 Prometheus
scrape_configs:
  - job_name: 'federate-us-east'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="minio-cluster-health"}'
        - '{job="minio-erasure-sets"}'
    static_configs:
      - targets:
          - 'prometheus-us-east:9090'
        labels:
          region: 'us-east-1'
  
  - job_name: 'federate-eu-west'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="minio-cluster-health"}'
        - '{job="minio-erasure-sets"}'
    static_configs:
      - targets:
          - 'prometheus-eu-west:9090'
        labels:
          region: 'eu-west-1'

7. Recording Rules로 성능 최적화

# recording_rules.yml
groups:
  - name: minio_recording_rules
    interval: 30s
    rules:
      # 자주 사용하는 계산 미리 저장
      - record: minio:cluster:capacity_used_percent
        expr: |
          (minio_cluster_health_capacity_usable_total_bytes - 
           minio_cluster_health_capacity_usable_free_bytes) / 
          minio_cluster_health_capacity_usable_total_bytes * 100
      
      - record: minio:api:error_rate:5m
        expr: |
          sum(rate(minio_api_requests_errors_total[5m])) /
          sum(rate(minio_api_requests_total[5m])) * 100
      
      - record: minio:api:5xx_rate:5m
        expr: |
          sum(rate(minio_api_requests_5xx_errors_total[5m])) /
          sum(rate(minio_api_requests_total[5m])) * 100
      
      - record: minio:drive:online_percent
        expr: |
          minio_cluster_health_drives_online_count /
          minio_cluster_health_drives_count * 100
      
      # 클러스터별 집계
      - record: minio:cluster:qps:5m
        expr: sum(rate(minio_api_requests_total[5m])) by (cluster)
      
      # API별 처리량
      - record: minio:api:requests_by_name:5m
        expr: sum(rate(minio_api_requests_total[5m])) by (name)
      
      # P95 TTFB
      - record: minio:api:ttfb_p95:5m
        expr: |
          histogram_quantile(0.95,
            rate(minio_api_requests_ttfb_seconds_distribution_bucket[5m])
          )
      
      # 드라이브별 사용률
      - record: minio:drive:used_percent
        expr: |
          (minio_system_drive_used_bytes / 
           minio_system_drive_total_bytes) * 100

사용 예시:

# ❌ 원본 쿼리 (느림, 복잡)
(minio_cluster_health_capacity_usable_total_bytes - 
 minio_cluster_health_capacity_usable_free_bytes) / 
minio_cluster_health_capacity_usable_total_bytes * 100

# ✅ Recording rule 사용 (빠름, 간단)
minio:cluster:capacity_used_percent

# Alert에서도 사용
- alert: MinIOCapacityHigh
  expr: minio:cluster:capacity_used_percent > 90

8. Retention 및 Storage 관리

# prometheus.yml
global:
  # 데이터 보관 기간
  retention_time: 15d        # 15일
  
  # 또는 용량 기반
  retention_size: 50GB       # 50GB 초과 시 자동 삭제

storage:
  tsdb:
    # 압축 활성화
    compaction:
      enabled: true
    
    # Out of order 샘플 허용
    out_of_order_time_window: 30m

계층적 스토리지 전략:

# 1. 단기 데이터 (Prometheus)
  - 고해상도 메트릭
  - 15-30일 보관
  - 빠른 쿼리

# 2. 중기 데이터 (Thanos/Cortex)
  - 다운샘플링된 데이터
  - 90일 보관
  - 장기 트렌드 분석

# 3. 장기 데이터 (Object Storage)
  - 대폭 다운샘플링
  - 1년+ 보관
  - 컴플라이언스, 감사

9. Service Discovery 활용

# Kubernetes Service Discovery
scrape_configs:
  - job_name: 'minio-kubernetes'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - minio
    
    relabel_configs:
      # MinIO pods만 선택
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: minio
        action: keep
      
      # 포트 9000만
      - source_labels: [__meta_kubernetes_pod_container_port_number]
        regex: "9000"
        action: keep
      
      # Pod 이름을 instance 레이블로
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: instance
      
      # Namespace 추가
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace

---
# Consul Service Discovery
scrape_configs:
  - job_name: 'minio-consul'
    consul_sd_configs:
      - server: 'consul.service.consul:8500'
        services:
          - minio
    
    relabel_configs:
      - source_labels: [__meta_consul_service]
        target_label: service
      - source_labels: [__meta_consul_node]
        target_label: node

10. 성능 튜닝

# prometheus.yml - 성능 최적화
global:
  scrape_interval: 30s
  scrape_timeout: 10s
  
  # 쿼리 성능
  query_log_file: /var/log/prometheus/queries.log
  
# 동시 scrape 제한
scrape_configs:
  - job_name: 'minio'
    scrape_interval: 30s
    scrape_timeout: 10s
    
    # 배치 크기 조정
    sample_limit: 10000  # scrape당 최대 샘플
    
    # Target 수가 많을 경우
    target_limit: 100    # job당 최대 target

# 명령줄 옵션
# --storage.tsdb.retention.time=15d
# --storage.tsdb.retention.size=50GB
# --query.max-concurrency=20
# --query.timeout=2m
# --web.max-connections=512

쿼리 최적화:

# ❌ 비효율적
sum(minio_api_requests_total)

# ✅ 효율적 (시간 범위 명시)
sum(rate(minio_api_requests_total[5m]))

# ❌ 전체 history 스캔
minio_cluster_health_drives_online_count[30d]

# ✅ Recording rule 사용
minio:drive:online_percent

# ❌ 과도한 aggregation
sum(minio_system_drive_used_bytes) by (server, drive, pool_index, set_index)

# ✅ 필요한 차원만
sum(minio_system_drive_used_bytes) by (server)

📊 유용한 PromQL 쿼리 모음

Capacity 관련

# 사용 가능한 용량 비율
(minio_cluster_health_capacity_usable_free_bytes / 
 minio_cluster_health_capacity_usable_total_bytes) * 100

# 사용 중인 용량
minio_cluster_health_capacity_usable_total_bytes - 
minio_cluster_health_capacity_usable_free_bytes

# 용량 증가율 (시간당 GB)
rate(minio_cluster_usage_objects_total_bytes[1h]) * 3600 / 1024 / 1024 / 1024

# 예상 용량 고갈 시점 (일 단위)
(minio_cluster_health_capacity_usable_free_bytes) / 
(rate(minio_cluster_usage_objects_total_bytes[24h]) * 86400)

# 버킷별 사용량 Top 10
topk(10, minio_cluster_usage_buckets_total_bytes)

# 버킷 증가율
rate(minio_cluster_usage_buckets_total_bytes[1h]) * 3600

Performance 관련

# 전체 요청 처리량 (QPS)
sum(rate(minio_api_requests_total[5m]))

# API별 요청 분포 Top 10
topk(10, sum(rate(minio_api_requests_total[5m])) by (name))

# 전체 에러율
sum(rate(minio_api_requests_errors_total[5m])) / 
sum(rate(minio_api_requests_total[5m])) * 100

# 5xx 에러율
sum(rate(minio_api_requests_5xx_errors_total[# MinIO Prometheus 모니터링 구성 가이드

## 📋 개요

MinIO AIStor는 Prometheus Data Model을 사용하여 클러스터 및 노드 메트릭을 제공합니다.
이 문서는 프로덕션 환경에서 필수적으로 설정해야 할 주요 메트릭과 Prometheus 구성을 다룹니다.

---

## 🔑 메트릭 엔드포인트

### 기본 엔드포인트

http://HOSTNAME:PORT/minio/metrics/v3


### 인증 설정

**Bearer 토큰 생성**:
```bash
mc admin prometheus generate ALIAS

인증 비활성화 (테스트 환경만):

export MINIO_PROMETHEUS_AUTH_TYPE=public

🎯 핵심 모니터링 메트릭 카테고리

1. Cluster Health (최우선 필수) ⭐⭐⭐

엔드포인트: /minio/metrics/v3/cluster/health

# prometheus.yml
scrape_configs:
  - job_name: 'minio-cluster-health'
    metrics_path: /minio/metrics/v3/cluster/health
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

주요 메트릭:

# 오프라인 드라이브 수
minio_cluster_health_drives_offline_count

# 온라인 드라이브 수
minio_cluster_health_drives_online_count

# 전체 드라이브 수
minio_cluster_health_drives_count

# 오프라인 노드 수
minio_cluster_health_nodes_offline_count

# 온라인 노드 수
minio_cluster_health_nodes_online_count

# 전체 용량 (raw)
minio_cluster_health_capacity_raw_total_bytes

# 사용 가능한 용량 (raw)
minio_cluster_health_capacity_raw_free_bytes

# 실제 사용 가능한 용량 (usable - erasure coding 고려)
minio_cluster_health_capacity_usable_total_bytes
minio_cluster_health_capacity_usable_free_bytes

권장 Alert 규칙:

groups:
  - name: minio_cluster_health
    rules:
      # 드라이브 오프라인 감지
      - alert: MinIODriveOffline
        expr: minio_cluster_health_drives_offline_count > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "MinIO drive(s) are offline"
          description: "{{ $value }} drive(s) are offline in the cluster"

      # 노드 오프라인 감지
      - alert: MinIONodeOffline
        expr: minio_cluster_health_nodes_offline_count > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "MinIO node(s) are offline"
          description: "{{ $value }} node(s) are offline"

      # 용량 부족 경고 (usable 기준 90%)
      - alert: MinIOLowStorage
        expr: |
          (minio_cluster_health_capacity_usable_total_bytes - 
           minio_cluster_health_capacity_usable_free_bytes) / 
          minio_cluster_health_capacity_usable_total_bytes > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "MinIO cluster storage is running low"
          description: "Usable storage is over 90% full"

      # 용량 위험 (95%)
      - alert: MinIOCriticalStorage
        expr: |
          (minio_cluster_health_capacity_usable_total_bytes - 
           minio_cluster_health_capacity_usable_free_bytes) / 
          minio_cluster_health_capacity_usable_total_bytes > 0.95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "MinIO cluster storage is critically low"
          description: "Usable storage is over 95% full"

2. Erasure Set Health (필수) ⭐⭐⭐

엔드포인트: /minio/metrics/v3/cluster/erasure-set

scrape_configs:
  - job_name: 'minio-erasure-sets'
    metrics_path: /minio/metrics/v3/cluster/erasure-set
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 1m

주요 메트릭:

# 전체 클러스터 쓰기 쿼럼 상태
minio_cluster_erasure_set_overall_write_quorum

# 전체 클러스터 건강 상태 (1=healthy, 0=unhealthy)
minio_cluster_erasure_set_overall_health

# Pool/Set별 읽기 쿼럼
minio_cluster_erasure_set_read_quorum{pool_id="0", set_id="0"}

# Pool/Set별 쓰기 쿼럼
minio_cluster_erasure_set_write_quorum{pool_id="0", set_id="0"}

# 온라인 드라이브 수
minio_cluster_erasure_set_online_drives_count{pool_id="0", set_id="0"}

# 치유 중인 드라이브 수
minio_cluster_erasure_set_healing_drives_count{pool_id="0", set_id="0"}

# Erasure set 건강 상태
minio_cluster_erasure_set_health{pool_id="0", set_id="0"}

# 읽기 장애 허용 범위
minio_cluster_erasure_set_read_tolerance{pool_id="0", set_id="0"}

# 쓰기 장애 허용 범위
minio_cluster_erasure_set_write_tolerance{pool_id="0", set_id="0"}

권장 Alert 규칙:

groups:
  - name: minio_erasure_set_health
    rules:
      # Erasure set 전체 상태 불량
      - alert: MinIOErasureSetUnhealthy
        expr: minio_cluster_erasure_set_overall_health == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "MinIO erasure set is unhealthy"
          description: "Overall erasure set health is degraded"

      # 특정 Set의 건강 상태 불량
      - alert: MinIOErasureSetDegraded
        expr: minio_cluster_erasure_set_health == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Erasure set {{ $labels.pool_id }}/{{ $labels.set_id }} is unhealthy"
          description: "Pool {{ $labels.pool_id }} Set {{ $labels.set_id }} is degraded"

      # 드라이브 치유 중
      - alert: MinIODrivesHealing
        expr: minio_cluster_erasure_set_healing_drives_count > 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "MinIO drives are healing"
          description: "{{ $value }} drive(s) in pool {{ $labels.pool_id }} set {{ $labels.set_id }} are healing"

      # 읽기 허용 범위 낮음
      - alert: MinIOLowReadTolerance
        expr: minio_cluster_erasure_set_read_tolerance < 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Low read tolerance in erasure set"
          description: "Pool {{ $labels.pool_id }} Set {{ $labels.set_id }} has read tolerance of {{ $value }}"

      # 쓰기 허용 범위 위험
      - alert: MinIOCriticalWriteTolerance
        expr: minio_cluster_erasure_set_write_tolerance < 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Critical write tolerance in erasure set"
          description: "Pool {{ $labels.pool_id }} Set {{ $labels.set_id }} has write tolerance of {{ $value }}"

3. System Drive Metrics (필수) ⭐⭐⭐

엔드포인트: /minio/metrics/v3/system/drive

scrape_configs:
  - job_name: 'minio-system-drives'
    metrics_path: /minio/metrics/v3/system/drive
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 1m

주요 메트릭:

# 드라이브 사용량
minio_system_drive_used_bytes{drive="/data1", server="node-1"}

# 드라이브 여유 공간
minio_system_drive_free_bytes{drive="/data1", server="node-1"}

# 드라이브 전체 용량
minio_system_drive_total_bytes{drive="/data1", server="node-1"}

# 드라이브 건강 상태 (0=offline, 1=healthy, 2=healing)
minio_system_drive_health{drive="/data1", server="node-1"}

# I/O 에러 수
minio_system_drive_io_errors_total{drive="/data1", server="node-1"}

# 타임아웃 에러 수
minio_system_drive_timeout_errors_total{drive="/data1", server="node-1"}

# 가용성 에러 (I/O + 타임아웃)
minio_system_drive_availability_errors_total{drive="/data1", server="node-1"}

# API 레이턴시 (마이크로초)
minio_system_drive_api_latency_micros{drive="/data1", api="storage.StatVol"}

# 드라이브 읽기 성능
minio_system_drive_reads_per_sec{drive="/data1", server="node-1"}
minio_system_drive_reads_kb_per_sec{drive="/data1", server="node-1"}
minio_system_drive_reads_await{drive="/data1", server="node-1"}

# 드라이브 쓰기 성능
minio_system_drive_writes_per_sec{drive="/data1", server="node-1"}
minio_system_drive_writes_kb_per_sec{drive="/data1", server="node-1"}
minio_system_drive_writes_await{drive="/data1", server="node-1"}

# 드라이브 사용률
minio_system_drive_perc_util{drive="/data1", server="node-1"}

권장 Alert 규칙:

groups:
  - name: minio_drive_health
    rules:
      # 드라이브 오프라인
      - alert: MinIODriveOffline
        expr: minio_system_drive_health == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "MinIO drive is offline"
          description: "Drive {{ $labels.drive }} on {{ $labels.server }} is offline"

      # 드라이브 치유 중
      - alert: MinIODriveHealing
        expr: minio_system_drive_health == 2
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "MinIO drive is healing"
          description: "Drive {{ $labels.drive }} on {{ $labels.server }} is healing"

      # I/O 에러 증가
      - alert: MinIODriveIOErrors
        expr: rate(minio_system_drive_io_errors_total[5m]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "MinIO drive has I/O errors"
          description: "Drive {{ $labels.drive }} on {{ $labels.server }} is experiencing I/O errors"

      # 높은 레이턴시
      - alert: MinIODriveHighLatency
        expr: minio_system_drive_api_latency_micros > 1000000  # 1초
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "MinIO drive has high latency"
          description: "Drive {{ $labels.drive }} API {{ $labels.api }} latency is {{ $value }}µs"

      # 드라이브 포화 상태
      - alert: MinIODriveSaturated
        expr: minio_system_drive_perc_util > 90
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "MinIO drive is saturated"
          description: "Drive {{ $labels.drive }} on {{ $labels.server }} utilization is {{ $value }}%"

4. API Request Metrics (중요) ⭐⭐

엔드포인트: /minio/metrics/v3/api/requests

scrape_configs:
  - job_name: 'minio-api-requests'
    metrics_path: /minio/metrics/v3/api/requests
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

주요 메트릭:

# 총 요청 수
minio_api_requests_total{name="s3.PutObject", type="s3"}

# 4xx 에러
minio_api_requests_4xx_errors_total{name="s3.GetObject"}

# 5xx 에러
minio_api_requests_5xx_errors_total{name="s3.PutObject"}

# 진행 중인 요청
minio_api_requests_inflight_total{name="s3.GetObject"}

# 대기 중인 요청
minio_api_requests_waiting_total

# TTFB (Time To First Byte) 분포
minio_api_requests_ttfb_seconds_distribution{le="0.5"}

# 트래픽 (송신)
minio_api_requests_traffic_sent_bytes

# 트래픽 (수신)
minio_api_requests_traffic_received_bytes

# 인증 실패
minio_api_requests_rejected_auth_total

# 취소된 요청
minio_api_requests_canceled_total

권장 Alert 규칙:

groups:
  - name: minio_api_performance
    rules:
      # 높은 에러율 (5xx)
      - alert: MinIOHighServerErrorRate
        expr: |
          rate(minio_api_requests_5xx_errors_total[5m]) / 
          rate(minio_api_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High 5xx error rate in MinIO"
          description: "5xx error rate is {{ $value | humanizePercentage }}"

      # 높은 에러율 (4xx)
      - alert: MinIOHighClientErrorRate
        expr: |
          rate(minio_api_requests_4xx_errors_total[5m]) / 
          rate(minio_api_requests_total[5m]) > 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High 4xx error rate in MinIO"
          description: "4xx error rate is {{ $value | humanizePercentage }}"

      # 많은 대기 요청
      - alert: MinIOHighWaitingRequests
        expr: minio_api_requests_waiting_total > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Many requests waiting in MinIO"
          description: "{{ $value }} requests are waiting in queue"

      # 높은 인증 실패율
      - alert: MinIOHighAuthFailures
        expr: rate(minio_api_requests_rejected_auth_total[5m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High authentication failure rate"
          description: "Auth failures: {{ $value }}/sec"

5. Bucket-level API Metrics (버킷별 모니터링) ⭐⭐

엔드포인트: /minio/metrics/v3/bucket/api?buckets=bucket1,bucket2

scrape_configs:
  - job_name: 'minio-bucket-api'
    metrics_path: /minio/metrics/v3/bucket/api
    params:
      buckets: ['important-bucket', 'critical-data']
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

주요 메트릭:

# 버킷별 총 요청
minio_bucket_api_total{bucket="my-bucket", name="s3.GetObject"}

# 버킷별 에러
minio_bucket_api_4xx_errors_total{bucket="my-bucket"}
minio_bucket_api_5xx_errors_total{bucket="my-bucket"}

# 버킷별 트래픽
minio_bucket_api_traffic_sent_bytes{bucket="my-bucket"}
minio_bucket_api_traffic_received_bytes{bucket="my-bucket"}

# 버킷별 TTFB
minio_bucket_api_ttfb_seconds_distribution{bucket="my-bucket"}

6. Cluster Usage (용량 관리) ⭐⭐

엔드포인트: /minio/metrics/v3/cluster/usage/buckets

scrape_configs:
  - job_name: 'minio-cluster-usage'
    metrics_path: /minio/metrics/v3/cluster/usage/buckets
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 5m

주요 메트릭:

# 버킷별 사용량
minio_cluster_usage_buckets_total_bytes{bucket="my-bucket"}

# 버킷별 객체 수
minio_cluster_usage_buckets_objects_count{bucket="my-bucket"}

# 버킷별 버전 수
minio_cluster_usage_buckets_versions_count{bucket="my-bucket"}

# 버킷별 삭제 마커 수
minio_cluster_usage_buckets_delete_markers_count{bucket="my-bucket"}

# 버킷 쿼터
minio_cluster_usage_buckets_quota_total_bytes{bucket="my-bucket"}

# 전체 클러스터 사용량
minio_cluster_usage_objects_total_bytes

# 전체 객체 수
minio_cluster_usage_objects_count

# 전체 버킷 수
minio_cluster_usage_objects_buckets_count

권장 Alert 규칙:

groups:
  - name: minio_usage
    rules:
      # 버킷 쿼터 근접
      - alert: MinIOBucketQuotaNearLimit
        expr: |
          minio_cluster_usage_buckets_total_bytes / 
          minio_cluster_usage_buckets_quota_total_bytes > 0.9
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Bucket {{ $labels.bucket }} near quota limit"
          description: "Bucket usage is {{ $value | humanizePercentage }} of quota"

      # 버킷 급격한 증가
      - alert: MinIOBucketRapidGrowth
        expr: |
          rate(minio_cluster_usage_buckets_total_bytes[1h]) > 
          1073741824  # 1GB/hour
        for: 6h
        labels:
          severity: info
        annotations:
          summary: "Bucket {{ $labels.bucket }} growing rapidly"
          description: "Bucket growing at {{ $value | humanize }}B/hour"

7. Replication Metrics (복제 사용 시) ⭐⭐

엔드포인트: /minio/metrics/v3/replication

scrape_configs:
  - job_name: 'minio-replication'
    metrics_path: /minio/metrics/v3/replication
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 1m

주요 메트릭:

# 활성 복제 워커
minio_replication_current_active_workers

# 복제 대기 중인 객체 수
minio_replication_average_queued_count

# 복제 대기 중인 바이트
minio_replication_average_queued_bytes

# 복제 전송 속도
minio_replication_current_data_transfer_rate

# 최근 백로그
minio_replication_recent_backlog_count

# 버킷별 복제 실패
minio_bucket_replication_last_minute_failed_count{bucket="my-bucket"}
minio_bucket_replication_total_failed_count{bucket="my-bucket"}

# 버킷별 복제 성공
minio_bucket_replication_sent_count{bucket="my-bucket"}
minio_bucket_replication_sent_bytes{bucket="my-bucket"}

# 복제 레이턴시
minio_bucket_replication_latency_ms{bucket="my-bucket", operation="replication"}

권장 Alert 규칙:

groups:
  - name: minio_replication
    rules:
      # 복제 백로그 증가
      - alert: MinIOReplicationBacklog
        expr: minio_replication_recent_backlog_count > 1000
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "High replication backlog"
          description: "{{ $value }} objects in replication backlog"

      # 복제 실패 증가
      - alert: MinIOReplicationFailures
        expr: rate(minio_bucket_replication_total_failed_count[5m]) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Replication failures for bucket {{ $labels.bucket }}"
          description: "Replication failing at {{ $value }}/sec"

      # 높은 복제 레이턴시
      - alert: MinIOHighReplicationLatency
        expr: minio_bucket_replication_latency_ms > 60000  # 60초
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "High replication latency for {{ $labels.bucket }}"
          description: "Replication latency is {{ $value }}ms"

8. System Resources (시스템 모니터링) ⭐

엔드포인트:

  • /minio/metrics/v3/system/cpu
  • /minio/metrics/v3/system/memory
  • /minio/metrics/v3/system/network/internode
scrape_configs:
  - job_name: 'minio-system-cpu'
    metrics_path: /minio/metrics/v3/system/cpu
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

  - job_name: 'minio-system-memory'
    metrics_path: /minio/metrics/v3/system/memory
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

  - job_name: 'minio-system-network'
    metrics_path: /minio/metrics/v3/system/network/internode
    static_configs:
      - targets: ['minio.example.com:9000']
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

주요 메트릭:

# CPU
minio_system_cpu_load_perc
minio_system_cpu_avg_idle
minio_system_cpu_avg_iowait

# Memory
minio_system_memory_used_perc
minio_system_memory_available
minio_system_memory_total

# Network (Internode)
minio_system_network_internode_errors_total
minio_system_network_internode_dial_errors_total
minio_system_network_internode_sent_bytes_total
minio_system_network_internode_recv_bytes_total

# Process
minio_system_process_cpu_total_seconds
minio_system_process_file_descriptor_open_total
minio_system_process_file_descriptor_limit_total
minio_system_process_resident_memory_bytes

권장 Alert 규칙:

groups:
  - name: minio_system_resources
    rules:
      # 높은 CPU 사용률
      - alert: MinIOHighCPUUsage
        expr: minio_system_cpu_load_perc > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.server }}"
          description: "CPU load is {{ $value }}%"

      # 높은 메모리 사용률
      - alert: MinIOHighMemoryUsage
        expr: minio_system_memory_used_perc > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.server }}"
          description: "Memory usage is {{ $value }}%"

      # Internode 통신 에러
      - alert: MinIOInternodeErrors
        expr: rate(minio_system_network_internode_errors_total[5m]) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Internode communication errors"
          description: "{{ $value }} errors/sec on {{ $labels.server }}"

      # File descriptor 고갈
      - alert: MinIOFileDescriptorLimit
        expr: |
          minio_system_process_file_descriptor_open_total / 
          minio_system_process_file_descriptor_limit_total > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "File descriptor usage high on {{ $labels.server }}"
          description: "Using {{ $value | humanizePercentage }} of FD limit"

📊 전체 Prometheus 구성 예시

# prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s
  external_labels:
    cluster: 'production-minio'
    region: 'us-east-1'

# Alert manager 구성
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'alertmanager:9093'

# Alert 규칙 파일
rule_files:
  - 'minio_alerts.yml'

scrape_configs:
  # 1. Cluster Health (최우선)
  - job_name: 'minio-cluster-health'
    metrics_path: /minio/metrics/v3/cluster/health
    static_configs:
      - targets: 
          - 'minio-node-1:9000'
          - 'minio-node-2:9000'
          - 'minio-node-3:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s
    scrape_timeout: 10s

  # 2. Erasure Sets (필수)
  - job_name: 'minio-erasure-sets'
    metrics_path: /minio/metrics/v3/cluster/erasure-set
    static_configs:
      - targets:
          - 'minio-node-1:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 1m

  # 3. System Drives (필수)
  - job_name: 'minio-system-drives'
    metrics_path: /minio/metrics/v3/system/drive
    static_configs:
      - targets:
          - 'minio-node-1:9000'
          - 'minio-node-2:9000'
          - 'minio-node-3:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 1m

  # 4. API Requests
  - job_name: 'minio-api-requests'
    metrics_path: /minio/metrics/v3/api/requests
    static_configs:
      - targets:
          - 'minio-node-1:9000'
          - 'minio-node-2:9000'
          - 'minio-node-3:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

  # 5. Bucket API (중요 버킷만)
  - job_name: 'minio-bucket-api'
    metrics_path: /minio/metrics/v3/bucket/api
    params:
      buckets: ['production-data', 'critical-backups', 'user-uploads']
    static_configs:
      - targets:
          - 'minio-node-1:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 1m

  # 6. Cluster Usage
  - job_name: 'minio-cluster-usage'
    metrics_path: /minio/metrics/v3/cluster/usage/buckets
    static_configs:
      - targets:
          - 'minio-node-1:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 5m

  # 7. Replication (복제 사용 시)
  - job_name: 'minio-replication'
    metrics_path: /minio/metrics/v3/replication
    static_configs:
      - targets:
          - 'minio-node-1:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 1m

  # 8. System Resources
  - job_name: 'minio-system-cpu'
    metrics_path: /minio/metrics/v3/system/cpu
    static_configs:
      - targets:
          - 'minio-node-1:9000'
          - 'minio-node-2:9000'
          - 'minio-node-3:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

  - job_name: 'minio-system-memory'
    metrics_path: /minio/metrics/v3/system/memory
    static_configs:
      - targets:
          - 'minio-node-1:9000'
          - 'minio-node-2:9000'
          - 'minio-node-3:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

  - job_name: 'minio-system-network'
    metrics_path: /minio/metrics/v3/system/network/internode
    static_configs:
      - targets:
          - 'minio-node-1:9000'
          - 'minio-node-2:9000'
          - 'minio-node-3:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

  - job_name: 'minio-system-process'
    metrics_path: /minio/metrics/v3/system/process
    static_configs:
      - targets:
          - 'minio-node-1:9000'
          - 'minio-node-2:9000'
          - 'minio-node-3:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 30s

  # 9. IAM (인증 사용 시)
  - job_name: 'minio-cluster-iam'
    metrics_path: /minio/metrics/v3/cluster/iam
    static_configs:
      - targets:
          - 'minio-node-1:9000'
    bearer_token: 'YOUR_BEARER_TOKEN'
    scrape_interval: 2m

🎯 우선순위별 구성 전략

Tier 1: 최소 필수 구성 (Production 필수)

scrape_configs:
  - job_name: 'minio-cluster-health'
    metrics_path: /minio/metrics/v3/cluster/health
    scrape_interval: 30s
    
  - job_name: 'minio-erasure-sets'
    metrics_path: /minio/metrics/v3/cluster/erasure-set
    scrape_interval: 1m
    
  - job_name: 'minio-system-drives'
    metrics_path: /minio/metrics/v3/system/drive
    scrape_interval: 1m

Tier 2: 권장 구성 (Performance 모니터링)

# Tier 1 + 추가:
  - job_name: 'minio-api-requests'
    metrics_path: /minio/metrics/v3/api/requests
    scrape_interval: 30s
    
  - job_name: 'minio-cluster-usage'
    metrics_path: /minio/metrics/v3/cluster/usage/buckets
    scrape_interval: 5m

Tier 3: 전체 구성 (완전한 관찰성)

# Tier 1 + Tier 2 + 추가:
  - job_name: 'minio-replication'
  - job_name: 'minio-bucket-api'
  - job_name: 'minio-system-resources'
  - job_name: 'minio-cluster-iam'

📈 Grafana 대시보드 권장 패널

Dashboard 1: Cluster Health Overview

{
  "panels": [
    {
      "title": "Cluster Health Status",
      "targets": [
        {
          "expr": "minio_cluster_erasure_set_overall_health",
          "legendFormat": "Overall Health"
        }
      ],
      "alert": {
        "conditions": [
          {
            "evaluator": { "params": [1], "type": "lt" },
            "query": { "params": ["A", "5m", "now"] }
          }
        ]
      }
    },
    {
      "title": "Online/Offline Drives",
      "targets": [
        {
          "expr": "minio_cluster_health_drives_online_count",
          "legendFormat": "Online Drives"
        },
        {
          "expr": "minio_cluster_health_drives_offline_count",
          "legendFormat": "Offline Drives"
        }
      ]
    },
    {
      "title": "Storage Capacity",
      "targets": [
        {
          "expr": "minio_cluster_health_capacity_usable_total_bytes",
          "legendFormat": "Total Usable"
        },
        {
          "expr": "minio_cluster_health_capacity_usable_free_bytes",
          "legendFormat": "Free Usable"
        }
      ]
    },
    {
      "title": "Erasure Sets Health by Pool/Set",
      "targets": [
        {
          "expr": "minio_cluster_erasure_set_health",
          "legendFormat": "Pool {{pool_id}} Set {{set_id}}"
        }
      ]
    }
  ]
}

Dashboard 2: Performance Metrics

{
  "panels": [
    {
      "title": "Request Rate",
      "targets": [
        {
          "expr": "sum(rate(minio_api_requests_total[5m])) by (name)",
          "legendFormat": "{{name}}"
        }
      ]
    },
    {
      "title": "Error Rates",
      "targets": [
        {
          "expr": "sum(rate(minio_api_requests_4xx_errors_total[5m]))",
          "legendFormat": "4xx Errors"
        },
        {
          "expr": "sum(rate(minio_api_requests_5xx_errors_total[5m]))",
          "legendFormat": "5xx Errors"
        }
      ]
    },
    {
      "title": "TTFB (Time To First Byte) P95",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(minio_api_requests_ttfb_seconds_distribution_bucket[5m]))",
          "legendFormat": "P95 TTFB"
        }
      ]
    },
    {
      "title": "Throughput",
      "targets": [
        {
          "expr": "sum(rate(minio_api_requests_traffic_sent_bytes[5m]))",
          "legendFormat": "Sent"
        },
        {
          "expr": "sum(rate(minio_api_requests_traffic_received_bytes[5m]))",
          "legendFormat": "Received"
        }
      ]
    }
  ]
}

Dashboard 3: Drive Health

{
  "panels": [
    {
      "title": "Drive Health Status",
      "targets": [
        {
          "expr": "minio_system_drive_health",
          "legendFormat": "{{server}}-{{drive}}"
        }
      ]
    },
    {
      "title": "Drive I/O Errors",
      "targets": [
        {
          "expr": "rate(minio_system_drive_io_errors_total[5m])",
          "legendFormat": "{{server}}-{{drive}}"
        }
      ]
    },
    {
      "title": "Drive Latency",
      "targets": [
        {
          "expr": "minio_system_drive_api_latency_micros / 1000",
          "legendFormat": "{{server}}-{{drive}}-{{api}}"
        }
      ]
    },
    {
      "title": "Drive Utilization %",
      "targets": [
        {
          "expr": "minio_system_drive_perc_util",
          "legendFormat": "{{server}}-{{drive}}"
        }
      ]
    },
    {
      "title": "Drive IOPS",
      "targets": [
        {
          "expr": "minio_system_drive_reads_per_sec",
          "legendFormat": "{{server}}-{{drive}} reads"
        },
        {
          "expr": "minio_system_drive_writes_per_sec",
          "legendFormat": "{{server}}-{{drive}} writes"
        }
      ]
    }
  ]
}

🚨 전체 Alert 규칙 파일 (minio_alerts.yml)

groups:
  # ========================================
  # 1. Critical Alerts (즉시 대응 필요)
  # ========================================
  - name: minio_critical
    rules:
      # Cluster 전체 건강 상태
      - alert: MinIOClusterUnhealthy
        expr: minio_cluster_erasure_set_overall_health == 0
        for: 2m
        labels:
          severity: critical
          component: cluster
        annotations:
          summary: "MinIO cluster is unhealthy"
          description: "Overall erasure set health is degraded"
          runbook: "https://docs.min.io/troubleshooting/erasure-sets"

      # 노드 다운
      - alert: MinIONodeDown
        expr: minio_cluster_health_nodes_offline_count > 0
        for: 1m
        labels:
          severity: critical
          component: node
        annotations:
          summary: "MinIO node(s) are offline"
          description: "{{ $value }} node(s) are offline"
          action: "Check node status immediately"

      # 드라이브 다운
      - alert: MinIODriveOffline
        expr: minio_cluster_health_drives_offline_count > 0
        for: 5m
        labels:
          severity: critical
          component: drive
        annotations:
          summary: "MinIO drive(s) are offline"
          description: "{{ $value }} drive(s) are offline"
          action: "Check drive health and replace if necessary"

      # 심각한 용량 부족
      - alert: MinIOStorageCritical
        expr: |
          (minio_cluster_health_capacity_usable_total_bytes - 
           minio_cluster_health_capacity_usable_free_bytes) / 
          minio_cluster_health_capacity_usable_total_bytes > 0.95
        for: 5m
        labels:
          severity: critical
          component: capacity
        annotations:
          summary: "MinIO storage critically low"
          description: "Usable storage is {{ $value | humanizePercentage }} full"
          action: "Add capacity immediately or delete data"

      # 쓰기 불가능
      - alert: MinIOWriteQuorumLost
        expr: minio_cluster_erasure_set_write_tolerance < 1
        for: 2m
        labels:
          severity: critical
          component: erasure-set
        annotations:
          summary: "Write quorum lost in erasure set"
          description: "Pool {{ $labels.pool_id }} Set {{ $labels.set_id }} cannot accept writes"
          action: "Restore failed drives immediately"

      # 높은 5xx 에러율
      - alert: MinIOHighServerErrors
        expr: |
          rate(minio_api_requests_5xx_errors_total[5m]) / 
          rate(minio_api_requests_total[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
          component: api
        annotations:
          summary: "High server error rate"
          description: "5xx error rate is {{ $value | humanizePercentage }}"

  # ========================================
  # 2. Warning Alerts (주의 필요)
  # ========================================
  - name: minio_warning
    rules:
      # 용량 경고
      - alert: MinIOStorageWarning
        expr: |
          (minio_cluster_health_capacity_usable_total_bytes - 
           minio_cluster_health_capacity_usable_free_bytes) / 
          minio_cluster_health_capacity_usable_total_bytes > 0.85
        for: 15m
        labels:
          severity: warning
          component: capacity
        annotations:
          summary: "MinIO storage getting full"
          description: "Usable storage is {{ $value | humanizePercentage }} full"
          action: "Plan capacity expansion"

      # 드라이브 치유 중
      - alert: MinIODrivesHealing
        expr: sum(minio_cluster_erasure_set_healing_drives_count) > 0
        for: 1h
        labels:
          severity: warning
          component: drive
        annotations:
          summary: "Drives are healing"
          description: "{{ $value }} drive(s) are in healing state"
          action: "Monitor healing progress"

      # 읽기 허용 범위 낮음
      - alert: MinIOLowReadTolerance
        expr: minio_cluster_erasure_set_read_tolerance < 2
        for: 10m
        labels:
          severity: warning
          component: erasure-set
        annotations:
          summary: "Low read tolerance"
          description: "Pool {{ $labels.pool_id }} Set {{ $labels.set_id }} has read tolerance {{ $value }}"

      # 드라이브 I/O 에러
      - alert: MinIODriveIOErrors
        expr: rate(minio_system_drive_io_errors_total[5m]) > 0
        for: 5m
        labels:
          severity: warning
          component: drive
        annotations:
          summary: "Drive I/O errors detected"
          description: "Drive {{ $labels.drive }} on {{ $labels.server }} has I/O errors"
          action: "Check drive health"

      # 높은 드라이브 레이턴시
      - alert: MinIODriveHighLatency
        expr: minio_system_drive_api_latency_micros > 500000  # 500ms
        for: 10m
        labels:
          severity: warning
          component: drive
        annotations:
          summary: "High drive latency"
          description: "Drive {{ $labels.drive }} latency is {{ $value }}µs"

      # 높은 API 대기 큐
      - alert: MinIOHighWaitingRequests
        expr: minio_api_requests_waiting_total > 50
        for: 5m
        labels:
          severity: warning
          component: api
        annotations:
          summary: "Many requests waiting"
          description: "{{ $value }} requests in waiting queue"
          action: "Check cluster load and capacity"

      # 높은 CPU 사용률
      - alert: MinIOHighCPU
        expr: minio_system_cpu_load_perc > 85
        for: 15m
        labels:
          severity: warning
          component: system
        annotations:
          summary: "High CPU usage"
          description: "CPU load on {{ $labels.server }} is {{ $value }}%"

      # 높은 메모리 사용률
      - alert: MinIOHighMemory
        expr: minio_system_memory_used_perc > 90
        for: 10m
        labels:
          severity: warning
          component: system
        annotations:
          summary: "High memory usage"
          description: "Memory usage on {{ $labels.server }} is {{ $value }}%"

      # 복제 백로그
      - alert: MinIOReplicationBacklog
        expr: minio_replication_recent_backlog_count > 1000
        for: 30m
        labels:
          severity: warning
          component: replication
        annotations:
          summary: "High replication backlog"
          description: "{{ $value }} objects in replication backlog"

      # 인증 실패 증가
      - alert: MinIOHighAuthFailures
        expr: rate(minio_api_requests_rejected_auth_total[5m]) > 5
        for: 10m
        labels:
          severity: warning
          component: security
        annotations:
          summary: "High authentication failure rate"
          description: "Auth failures: {{ $value }}/sec"
          action: "Check for potential security issues"

  # ========================================
  # 3. Info Alerts (정보성)
  # ========================================
  - name: minio_info
    rules:
      # 버킷 급격한 증가
      - alert: MinIOBucketRapidGrowth
        expr: |
          rate(minio_cluster_usage_buckets_total_bytes[1h]) > 
          10737418240  # 10GB/hour
        for: 6h
        labels:
          severity: info
          component: capacity
        annotations:
          summary: "Bucket {{ $labels.bucket }} growing rapidly"
          description: "Growing at {{ $value | humanize }}B/hour"

      # 복제 실패 발생
      - alert: MinIOReplicationFailures
        expr: rate(minio_bucket_replication_total_failed_count[10m]) > 0
        for: 15m
        labels:
          severity: info
          component: replication
        annotations:
          summary: "Replication failures for {{ $labels.bucket }}"
          description: "Some objects failed to replicate"

      # TLS 인증서 만료 임박
      - alert: MinIOCertificateExpiringSoon
        expr: minio_system_network_certificate_expires_in < 2592000  # 30 days
        labels:
          severity: info
          component: security
        annotations:
          summary: "TLS certificate expiring soon"
          description: "Certificate expires in {{ $value | humanizeDuration }}"
          action: "Renew certificate"

  # ========================================
  # 4. SLO/SLA Alerts (서비스 수준 목표)
  # ========================================
  - name: minio_slo
    rules:
      # Availability SLO (99.9%)
      - alert: MinIOAvailabilitySLOBreach
        expr: |
          (
            sum(rate(minio_api_requests_total[5m])) - 
            sum(rate(minio_api_requests_5xx_errors_total[5m]))
          ) / 
          sum(rate(minio_api_requests_total[5m])) < 0.999
        for: 10m
        labels:
          severity: warning
          component: slo
        annotations:
          summary: "Availability SLO breached"
          description: "Availability is {{ $value | humanizePercentage }}, below 99.9% SLO"

      # Latency SLO (P95 < 100ms)
      - alert: MinIOLatencySLOBreach
        expr: |
          histogram_quantile(0.95, 
            rate(minio_api_requests_ttfb_seconds_distribution_bucket[5m])
          ) > 0.1
        for: 10m
        labels:
          severity: warning
          component: slo
        annotations:
          summary: "Latency SLO breached"
          description: "P95 latency is {{ $value }}s, above 100ms SLO"

🔧 베어러 토큰 생성 및 관리

# 베어러 토큰 생성
mc admin prometheus generate ALIAS

# 출력 예시:
# scrape_configs:
# - job_name: minio-job
#   bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9...
#   metrics_path: /minio/v2/metrics/cluster
#   scheme: http
#   static_configs:
#   - targets: ['localhost:9000']

# 토큰을 파일로 저장
mc admin prometheus generate ALIAS > minio_bearer_token.yml

# Prometheus 구성에서 파일 참조
# bearer_token_file: /etc/prometheus/minio_bearer_token.txt

📝 모니터링 체크리스트

초기 설정

  • Bearer 토큰 생성 및 저장
  • Prometheus scrape 구성 추가
  • Alert 규칙 파일 생성
  • AlertManager 구성
  • Grafana 대시보드 import

필수 모니터링 (Tier 1)

  • Cluster health
  • Erasure set health
  • Drive health
  • Capacity usage

권장 모니터링 (Tier 2)

  • API request metrics
  • Error rates
  • TTFB / Latency
  • Throughput

선택적 모니터링 (Tier 3)

  • Replication metrics
  • Bucket-level metrics
  • System resources
  • IAM metrics

Alert 설정

  • Critical alerts (PagerDuty/OnCall)
  • Warning alerts (Slack/Email)
  • Info alerts (Logging/Dashboard)

🎓 모범 사례 (Best Practices)

1. Scrape 간격 최적화

  • Health metrics: 30초 (빠른 감지 필요)
  • Drive metrics: 1분 (균형)
  • Usage metrics: 5분 (변화가 느림)
  • System metrics: 30초-1분

2. 레이블 활용

# 특정 서버만 조회
minio_cluster_health_drives_online_count{server="node-1"}

# 특정 pool/set 조회
minio_cluster_erasure_set_health{pool_id="0", set_id="0"}

# 특정 버킷만 조회
minio_bucket_api_total{bucket="important-data"}

3. Alert 피로도 방지

  • for 절을 적절히 설정 (일시적 스파이크 무시)
  • 중요도에 따라 severity 차별화
  • Alert grouping 활용
  • Silence 기능 활용 (유지보수 시)
profile
Young-Kyoo Kim
engineer
이전 포스트

poc1013

다음 포스트

poc1015

0개의 댓글