here are a number of metrics and traces you can look at for live debugging. Three surfaces, three granularities:

surface granularity answers
metrics (v3) bucket + API operation, scraped what is hot and where
mc support top api, mc admin trace --stats API operation, rolling 1 min, live what is hot right now
mc admin trace, audit log individual request who - access key, bucket, prefix
Metrics and top api give you the operation and the bucket. Only trace and audit give you the
client. Most incidents resolve at the first two.

1. Metrics - the signal that moves first
   Goroutine count is a lagging symptom. The metric that leads it is the API wait queue.

Cluster-wide, from /minio/metrics/v3/api/requests:

minio_api_requests_waiting_total # queued behind maxClients — leading indicator
minio_api_requests_incoming_total
minio_api_requests_inflight_total # labels: name, type
minio_api_requests_total # labels: name, type
minio_api_requests_canceled_total # labels: name, type — clients giving up
minio_api_requests_4xx_errors_total # labels: name, type
minio_api_requests_5xx_errors_total # labels: name, type
minio_api_requests_ttfb_seconds_distribution # labels: name, type, le
The name label is the S3 operation, so this collector alone breaks load down per API.

Goroutines, from /minio/metrics/v3/system/process:

minio_system_process_go_routine_total
There is also a /minio/metrics/v3/debug/go endpoint exposing the standard Go runtime collector,
which adds GC and scheduler detail if you need to distinguish "more work queued" from "runtime
under pressure."

The one to set up if you haven't, from /minio/metrics/v3/bucket/api - these carry a
bucket label in addition to the API name, which is what answers your question directly:

minio_bucket_api_total # labels: bucket, name, type
minio_bucket_api_inflight_total # labels: bucket, name, type
minio_bucket_api_canceled_total # labels: bucket, name, type
minio_bucket_api_4xx_errors_total # labels: bucket, name, type
minio_bucket_api_5xx_errors_total # labels: bucket, name, type
minio_bucket_api_ttfb_seconds_distribution # labels: bucket, name, le, type
minio_bucket_api_traffic_sent_bytes # labels: bucket, type
minio_bucket_api_traffic_received_bytes # labels: bucket, type
Queries worth building in advance:

the leading indicator - alert on this

sum(minio_api_requests_waiting_total)

which bucket + operation is driving request volume

topk(10, sum by (bucket, name) (rate(minio_bucket_api_total[1m])))

which bucket + operation is holding requests open - closest proxy for the goroutine spike

topk(10, sum by (bucket, name) (minio_bucket_api_inflight_total))

latency shift by bucket and operation

histogram_quantile(0.99,
sum by (bucket, name, le) (rate(minio_bucket_api_ttfb_seconds_distribution[5m])))
inflight_total broken down by bucket and operation is the most useful single panel for this,
because each in-flight request holds handler goroutines. Graph it beside
minio_system_process_go_routine_total and the correlation usually names the culprit on its own.

2. Live aggregation - during the incident
   Both are read-only and safe on a busy production cluster.

rolling last-1-minute view of all S3 API calls, refreshing live

mc support top api ALIAS

filter by substring of the API name

mc support top api ALIAS --name List
mc support top api ALIAS --name s3.PutObject

only calls returning errors

mc support top api ALIAS --errors

scope to a node, or split per host

mc support top api ALIAS --node minio-3
mc support top api ALIAS --nodes minio-3,minio-4
mc admin trace --stats gives an aggregated table rather than a firehose — Call, Count, RPM,
Avg/Min/Max Time, Avg/Max TTFB, Avg Size, Rate per minute, and Errors, with live RX/TX rates:

mc admin trace ALIAS --stats --stats-n 20
This is usually the most useful single command during an incident, because it ranks by volume
while showing latency and errors in the same view. The loudest operation and the slowest one are
frequently not the same, and that distinction is often the whole diagnosis.

Worth ruling out causes that aren't request volume, since these produce goroutine build-up too:

mc support top locks ALIAS # lock contention parks goroutines
mc support top rpc ALIAS # internode grid health
mc support top drive ALIAS # slow drives back-pressuring the API layer
mc support top net ALIAS
If top api looks unremarkable but goroutines are climbing, the answer is almost always in
top locks or top drive.

3. Attribution — identifying the client

server-side latency filters — the server filters, so these are safe to leave running

mc admin trace ALIAS --response-duration 500ms
mc admin trace ALIAS --response-ttfb-duration 200ms

scope to one operation, with full request detail including headers

mc admin trace ALIAS --funcname 's3.ListObjectsV2' -v

errors only, or one status code

mc admin trace ALIAS --errors
mc admin trace ALIAS --status-code 503

one node, or filter by request size

mc admin trace ALIAS --node minio-3
mc admin trace ALIAS --filter-size 100MiB
--response-duration and --response-ttfb-duration are evaluated server-side, which is what
makes tracing practical under load — you aren't shipping every event to your terminal.

One caveat before relying on it: --path silently excludes bucket-level listing calls. Use
--funcname, or run unfiltered with --response-duration, when that's what you're hunting.

Audit log - the better attribution layer
Trace is a live sample; audit is complete and retained. Each entry already carries what you need
to answer "who," in structured form:

accessKey — the client credential
parentUser — for STS or assumed-role, the underlying identity
api.name, api.bucket, api.object
api.statusCode
api.timeToResponseInNS — per-request latency
api.inputBytes, api.outputBytes
So one query against your audit sink answers "which access key issued the most of operation X in
the last five minutes, against which bucket and prefix" - usually the answer you actually want,
and available without having caught the incident live.

If audit logging isn't currently reaching a durable, queryable target, that is the single
highest-value change for this class of incident. It turns "we have to catch it happening" into
"we can query it afterwards," and it's the only surface that ties load to a specific credential
at scale.

One sizing note: audit targets are all-or-nothing - there is no status-code or per-API filtering
in the pipeline - so the sink has to absorb full request volume.

Suggested triage order
minio_api_requests_waiting_total rising — the alert that should page you
mc support top api or mc admin trace --stats - which operation is hot
topk on minio_bucket_api_total / minio_bucket_api_inflight_total - which bucket
mc support top locks / top drive - rule out contention or slow drives as the real driver
mc admin trace --funcname -v, or an audit query - which access key and prefix
mc support profile for the post-mortem, once stable
Steps 1–3 usually settle what you're dealing with and take under a minute once the dashboards
exist. Worth rehearsing outside an incident so the queries are already written.