DCGM monitoring at scale: the alerts that actually matter

If you operate NVIDIA GPUs at scale, DCGM (Data Center GPU Manager) is the standard monitoring agent. Getting metrics flowing takes a couple of hours. Getting useful alerts from those metrics takes a couple of years.

This is operator notes from running DCGM in production AI infrastructure. The focus: which alerts actually catch real problems, which become noise, and how to set thresholds that match your workload.

What is DCGM? NVIDIA’s GPU monitoring agent. Exposes 80+ metrics about GPU health, performance, and utilization. The standard dcgm-exporter integrates with Prometheus. This article assumes you have it running and want to know which metrics matter.

The 3 alerts that catch real problems

If you only set up 3 alerts, these are the ones. They have caught real hardware failures in production and almost never fire false positives.

Alert	Threshold	Why
Uncorrectable ECC error	Any non-zero value	Memory corruption occurred; GPU should be replaced
GPU offline	Exporter unreachable >2 min	Hardware, host, or monitoring pipeline failure
Temperature critical	>87°C sustained 10 min	Cooling failure; throttling imminent at 88-90°C

The full alert set we run in production has ~12 rules total, but these 3 are the foundation. Everything else is workload-specific tuning.

The rest of this article documents the full set and the methodology behind threshold selection. If you just want the minimum, the table above gets you 80% of the value.

Quick reference: full alert set

Severity	Metric	Threshold	Sustained for
🔴 Critical	Uncorrectable ECC (DCGM_FI_DEV_ECC_DBE_VOL_TOTAL)	> 0	Immediate
🔴 Critical	GPU offline (up{job="dcgm-exporter"})	= 0	2 minutes
🔴 Critical	Temperature (DCGM_FI_DEV_GPU_TEMP)	> 87°C	10 minutes
🟡 Warning	Correctable ECC rate	> 100/hr	4 hours
🟡 Warning	Temperature elevated	> 82°C	30 minutes
🟡 Warning	Memory utilization (DCGM_FI_DEV_FB_USED)	> 95% of capacity	6 hours
🟡 Warning	Power draw anomaly	stddev > 50W from 7-day baseline	1 hour
🟡 Warning	XID errors (codes 13, 31, 43, 45, 48, 63, 64, 79)	Any occurrence	Immediate
🟡 Warning	vGPU license (DCGM_FI_DEV_VGPU_LICENSE_STATUS)	!= “Licensed”	30 minutes

Note: thresholds shown are starting points. The next section explains how to tune them for your workload.

Why GPU monitoring is harder than CPU monitoring

Before getting into threshold tuning, understanding why this is hard helps. The naive approach — graph everything, alert on anything that looks important — generates alert fatigue within two weeks. We tried it.

The reasons:

Metrics are highly correlated. GPU utilization, memory, power, temperature all move together during workloads. “GPU at 100% utilization” tells you nothing useful because that’s what GPUs are supposed to do.

Normal looks like distress. Healthy GPU under training load looks identical to one about to fail. 80°C+ temperature, 300W+ power draw, 100% utilization are normal for training, would be emergency signals on any other hardware.

Failure modes are subtle. GPUs rarely fail catastrophically. They degrade — ECC errors creep up, performance drops, thermal throttling kicks in occasionally. These need baselines to detect.

Multi-tenant adds complexity. Sharing GPUs via vGPU or MIG (see vGPU on Dell VxRail for context) requires attributing metrics back to tenants.

The implication: thresholds depend on your workload. The next section explains how.

Thresholds depend on workload type

This is where most setup guides fail. They give universal thresholds. In practice, what’s “high” for inference is “normal” for training. Setting thresholds by workload type catches real problems while suppressing normal variation.

We operate three workload classes with distinct profiles:

Workload 1: AI training (batch jobs)

Characteristics: Sustained high utilization (4-72 hours), high memory consumption, high power draw, datasets sequentially read.

Normal baseline (A100, sustained training):

• Temperature: 72-79°C (P50 to P95)
• Power: 280-380W
• Memory utilization: 85-95% during job
• GPU utilization: 95-100%

Tuned alert thresholds:

• Temperature critical: 87°C (P99 + 1σ)
• Memory: alert only if >95% sustained 6 hours (training jobs can run at 95% normally)
• Power anomaly: stddev >60W from baseline (higher tolerance for bursts)

Workload 2: AI inference (live services)

Characteristics: Bursty utilization, lower sustained memory, latency-sensitive, customer-facing.

Normal baseline (A100, production inference):

• Temperature: 60-72°C
• Power: 150-280W (highly variable)
• Memory utilization: 40-70% steady state
• GPU utilization: 20-80% (depends on traffic)

Tuned alert thresholds:

• Temperature critical: 80°C (lower than training; sustained high temp means something wrong)
• Memory: alert if >85% sustained 2 hours (memory leak or model bloat)
• Power: stddev >30W from baseline (tighter range, inference should be predictable)

Workload 3: Mixed banking + AI (shared infrastructure)

Characteristics: Multi-tenant via MIG (see vSAN policy design for isolation context), variable patterns, strict compliance requirements.

Tuned alert thresholds:

• Use training thresholds for training partitions
• Use inference thresholds for inference partitions
• Additional alert: cross-partition contention (vGPU/MIG specific metrics)

Key insight: A single threshold across these three workloads either misses real issues (too high) or generates noise (too low). Per-workload thresholds work; universal thresholds don’t.

How to set thresholds for your workload

Instead of copying our numbers, derive yours from production data. The methodology:

Step 1: Establish baseline (1-2 weeks)

For each metric, measure during normal operation:

• P50 (median normal value)
• P95 (high but normal)
• P99 (very high but still observed in normal operation)
• Standard deviation

Run DCGM with metrics flowing into Prometheus for 7-14 days under normal workload. Calculate percentiles using PromQL quantile_over_time().

Step 2: Set alert at P99 + safety margin

Critical alerts trigger at: P99 + 1 standard deviation, sustained for time appropriate to the metric.

Warning alerts trigger at: P95 + 1 standard deviation, sustained longer.

Example calculation (training workload, A100 temperature):

• P50: 72°C, P95: 79°C, P99: 84°C, σ: 3°C
• Critical alert: 84 + 3 = 87°C sustained 10 min ✓ (matches our setting)
• Warning alert: 79 + 3 = 82°C sustained 30 min ✓

Step 3: Sustained, not instantaneous

Almost every alert should have a “sustained for X” condition. Instantaneous alerts catch transient spikes that resolve themselves. Sustained alerts catch real problems.

In Prometheus: use for: 10m syntax in alerting rules. In Grafana: use “for” duration in alert conditions.

Step 4: Multi-condition for noise reduction

Single-metric alerts often false positive. Combinations are more reliable:

Thermal throttling (combined):

DCGM_FI_DEV_GPU_TEMP > 85
AND
DCGM_FI_DEV_SM_CLOCK < (base_clock - 100)
sustained for 5 minutes

This fires only when temperature is high AND clock is throttling. Either alone might be benign.

Workload distress (combined):

GPU memory utilization > 95%
AND
power draw within 10% of TDP
AND
application heartbeat missed
sustained 2 minutes

Saturation + unresponsive app indicates real issue beyond either signal alone.

What we ignore

After tuning, we stopped alerting on metrics that produce noise without signal:

• “GPU utilization too low” — bursty workloads have legitimate idle periods
• “GPU utilization too high” — that’s what GPUs are for
• Memory utilization current (without sustained period) — AI workloads allocate normally
• Generic XID warnings — most XID codes are informational (only 8 codes warrant alerting; see below)
• Clock speed variations — GPUs throttle and boost normally
• Fan speeds in isolation — only matter when correlated with temperature

These metrics still appear on dashboards for human review, but don’t trigger automated alerts.

XID codes — the ones that actually matter

NVIDIA XID errors are GPU driver-level error codes. There are many. Most don’t matter. After multi-year observation, these are the ones correlated with real action needed:

XID	Description	Action
13	Graphics SM warp error	Workload restart usually fixes
31	GPU memory page fault	Investigate workload, possible memory corruption
43	Reset failure	Hardware issue likely
45	Preemptive cleanup	Driver state issue, monitor
48	Double-bit ECC	Critical, replace GPU
63	ECC error in retired pages	Memory degrading, plan replacement
64	Excessive ECC errors	Memory failing, plan replacement
79	GPU fallen off the bus	Hardware or PCIe issue

Mute alerts on other XID codes. They generate noise without predicting real problems.

Dashboards that earn their place

Three dashboards we actually use. Pretty dashboards that don’t answer questions get ignored.

1. Real-time operational (kept open during work hours)

Answers: “is anything wrong right now?”

Top: GPU online count, active alerts (24h), ECC rate (1h), avg temperature
Middle: GPU utilization heatmap, memory heatmap, power total, active workloads
Bottom: 24h trends — temperature, utilization, ECC, throttling events

2. Capacity planning (weekly ops review)

Answers: “are we running out?” / “are workloads balanced?”

• vGPU license utilization (current vs purchased)
• GPU memory over time per workload class
• Compute capacity vs forecast
• Power draw averages

3. Per-workload investigation (incident response)

Answers: “what changed for this specific workload?”

Filters by namespace/pod/VM. Shows that workload’s metrics with ECC events overlaid.

Operational lessons

A few patterns that aren’t in any documentation:

Driver version compatibility. DCGM exporter metrics change between driver versions. Pin DCGM exporter to driver version. Update both together during planned maintenance.

Multi-tenant attribution. DCGM tags metrics with GPU UUID and (for MIG) instance ID. Pod-to-GPU mapping needs Nvidia device plugin. We built an enrichment layer joining DCGM metrics with workload metadata.

Alert routing prevents fatigue. Same metric: critical for tier-zero, informational for dev/test. Route appropriately via Alertmanager.

Dashboard hygiene. Review quarterly. Remove panels nobody looks at. Promote investigation queries to dashboards if used 3+ times.

💡 Translate uptime targets to monitoring SLAs: Use our error budget calculator to convert SLO targets into concrete downtime budgets. If your error budget is 43 minutes/month, your monitoring needs to detect issues in minutes, not hours — this drives alert “sustained for” thresholds.

Cost of running DCGM monitoring

For context on what this costs:

• Compute overhead: DCGM exporter ~200m CPU + 256Mi memory per GPU node (negligible)
• Network: ~5KB/sec per GPU (minimal on modern networks)
• Storage: Prometheus ~10GB/month per 64 GPUs; long-term Thanos ~120GB/year
• Operational time: Initial setup 1-2 weeks; ongoing ~2hr/month per cluster

Total cost is minimal. The value: catching problems before they become incidents.

FAQ

Do I need DCGM if I’m using cloud GPUs?

Cloud providers expose some GPU metrics natively (CloudWatch, Stackdriver). DCGM provides more detail. For on-premise or hybrid, DCGM is standard. For cloud-only with simple monitoring needs, cloud-native may suffice.

Should I alert on GPU utilization?

Generally no. High utilization is what GPUs are for. Low utilization is an optimization opportunity, not an alert condition. The exception: if you’re paying for cloud GPU instances, low sustained utilization indicates cost waste — worth a dashboard panel but not a page.

How often should I scrape DCGM metrics?

Production: 30 seconds (low overhead, captures most anomalies). During incident investigation: 5 seconds temporarily. For long-term retention via Thanos: 1-minute downsampling.

Does DCGM work with vGPU and MIG?

Yes. DCGM exporter tags metrics with GPU UUID and MIG instance ID where applicable. Some metrics require recent driver and DCGM versions. For vGPU operational context, see our vGPU on Dell VxRail article.

What’s the difference between SBE and DBE ECC errors?

SBE = single-bit, automatically corrected by GPU memory. Some SBEs are normal (cosmic rays, etc.). High SBE rate is a warning. DBE = double-bit, uncorrectable. Any DBE means data corruption occurred — replace the GPU.

How do I monitor GPUs in Kubernetes?

Deploy DCGM exporter as DaemonSet on GPU nodes via NVIDIA’s Helm chart. ServiceMonitor (Prometheus Operator) picks up metrics. Standard scrape interval 30s. Pair with the NVIDIA device plugin for pod-to-GPU attribution.

Can DCGM monitor consumer GPUs?

DCGM officially supports data center GPUs (A-series, H-series, Tesla). Some metrics work on consumer GPUs (RTX, GeForce) but feature set is limited. For consumer cards in dev environments, nvidia-smi plus custom scripting is often sufficient.

What alerts do I really need to start?

The 3 alerts in the table at the top of this article. Uncorrectable ECC, GPU offline, temperature critical. Add other alerts only after you observe real issues your alerts missed.

Closing notes

DCGM provides comprehensive metrics. Useful monitoring isn’t about having more metrics — it’s identifying which ones signal real problems and tuning thresholds to your environment.

The 3 alerts at the top of this article catch most real issues. The rest is per-workload refinement. Resist the temptation to alert on everything: alert fatigue compounds, and the on-call engineer eventually stops responding seriously.

Future articles will cover the Prometheus integration patterns (Kubernetes-native), the enrichment layer for multi-tenant attribution, and runbooks tied to each alert. Subscribe to follow along.

---

Operator notes from running NVIDIA DCGM monitoring in production AI infrastructure. Specific thresholds reflect our environment; your workloads will require different baselines. Metric names align with DCGM exporter as of recent versions; check NVIDIA documentation for current naming. I am an operator, not an NVIDIA engineer — this is operational perspective, not authoritative configuration guidance.