Instructions: Setting up Prometheus scrape_config for a MinIO Cluster

Follow these steps to configure Prometheus to scrape metrics from all nodes in your MinIO cluster:

1. Gather MinIO Node Information

• Identify the hostname or IP address and the metrics port (commonly 9000 or 9001) for each MinIO node in your cluster.
• Note that MinIO v3 metrics are available at the /minio/metrics/v3 endpoint on port 9000.
• MinIO v3 provides metrics organized into several major categories:
  • /minio/metrics/v3/api - API request metrics
  • /minio/metrics/v3/audit - Audit functionality metrics
  • /minio/metrics/v3/cluster - Cluster-wide metrics including config, health, IAM, and usage
  • /minio/metrics/v3/debug - Go runtime metrics
  • /minio/metrics/v3/ilm - ILM (Lifecycle Management) metrics
  • /minio/metrics/v3/logger - Logger webhook metrics
  • /minio/metrics/v3/notification - Notification functionality metrics
  • /minio/metrics/v3/replication - Site and bucket replication metrics
  • /minio/metrics/v3/scanner - Scanner metrics
  • /minio/metrics/v3/system - System metrics including CPU, memory, drives, and network

2. Configure Authentication

• MinIO v3 metrics require bearer token authentication

• Generate a bearer token using the MinIO admin credentials:

  # First, configure your MinIO server alias if not already done
  mc alias set myminio http://your-minio-server:9000 admin your-password
  
  # Generate the bearer token for v3 metrics
  mc admin prometheus generate myminio --api-version v3

  The command will output a prometheus config that looks like this:

  scrape_configs:
  - job_name: minio-job
    bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiYWRtaW4iLCJleHAiOjQ5MDA2ODQ0MjF9._irfBI8ODRTyhf4tWYCUEl2aAihPkSD0VCbGR12cmUhLnywkziOIP29tFQ8Zan_HCTDdnoXfhryX4a-8GXPkxQ
    metrics_path: /minio/metrics/v3
    scheme: http
    static_configs:
    - targets: ['minio-1-1:9000']  

  You can also verify the token works by testing the metrics endpoint:

  # Test the bearer token with curl
  curl -s -H "Authorization: Bearer "your-token" http://minio-node1.example.com:9000/minio/metrics/v3
  
  # If successful, you should see MinIO metrics output
  # If the token is invalid, you'll get an authentication error

3. Open Prometheus Configuration

• Locate and open your prometheus.yml file (usually found in /etc/prometheus/ or your Prometheus deployment directory).

4. Add a MinIO scrape_config Section

• Under the scrape_configs: section, add a new job for MinIO.
• Use the static_configs block to list all MinIO nodes under the targets parameter.
• Each node should be listed in the format: host:port.
• For MinIO v3, make sure to use port 9000 and the /minio/metrics/v3 endpoint.
• Add the bearer_token field with your generated token.
• For bucket-specific metrics, you must specify the buckets you want to monitor using the buckets query parameter.

Example:

scrape_configs:
  # ... other scrape configs ...

  - job_name: 'minio'
    metrics_path: '/minio/metrics/v3'
    bearer_token: 'your-generated-bearer-token'
    static_configs:
      - targets:
          - 'minio-node1.example.com:9000'
          - 'minio-node2.example.com:9000'
          - 'minio-node3.example.com:9000'

  # Optional: Configure separate jobs for specific metric types
  - job_name: 'minio-cluster'
    metrics_path: '/minio/metrics/v3/cluster'
    bearer_token: 'your-generated-bearer-token'
    static_configs:
      - targets:
          - 'minio-node1.example.com:9000'

  # Per-bucket API metrics configuration
  - job_name: 'minio-bucket-api'
    bearer_token: 'your-generated-bearer-token'
    file_sd_configs:
      - files:
          - /opt/prometheus/minio_buckets.json
    relabel_configs:
      - source_labels: [__metrics_path__]
        target_label: __metrics_path__

Create the file `/opt/prometheus/minio_buckets.json` with the following content:
```json
[
  {
    "targets": ["minio-node1.example.com:9000"],
    "labels": {
      "__metrics_path__": "/minio/metrics/v3"
    }
  },
  {
    "targets": ["minio-node1.example.com:9000"],
    "labels": {
      "__metrics_path__": "/minio/metrics/v3/bucket/api/warp-bucket"
    }
  }
]

This configuration allows you to:

• Use file-based service discovery for dynamic bucket configuration
• Specify different metrics paths for different buckets
• Add or remove buckets by updating the JSON file without modifying Prometheus configuration
• Use relabeling to ensure the correct metrics path is used for each target

5. Save and Reload Prometheus

• Save your changes to prometheus.yml.
• Reload or restart Prometheus to apply the new configuration.

6. Verify

• Visit your Prometheus targets page (usually at http://<prometheus-server>:9090/targets) to ensure all MinIO nodes are listed and being scraped.
• You can verify the metrics are being collected by checking for MinIO v3 specific metrics in the Prometheus expression browser.

---

Note:

• If your MinIO nodes use a different port, update the port number accordingly.
• If you add or remove nodes from your MinIO cluster, update the targets list to match.
• The /minio/metrics/v3 endpoint is specific to MinIO v3. If you're using a different version, adjust the metrics path accordingly.
• Using specific metric paths (like /minio/metrics/v3/cluster) can help reduce the amount of data collected and improve scraping performance if you only need certain types of metrics.
• For bucket-specific metrics (/minio/metrics/v3/bucket/api), you must explicitly list all buckets you want to monitor in the buckets parameter. Metrics will only be collected for the specified buckets.
• If you add new buckets to your MinIO cluster, remember to update the buckets parameter in your Prometheus configuration to include them.
• Keep your bearer token secure and rotate it periodically for better security.
• The bearer token is required for all MinIO v3 metrics endpoints.