GPU Health Monitoring and Remediation - EMEP

GPU nodes are the most expensive and failure-prone resources in EMEP. A single faulty GPU can corrupt a merge, waste evaluation time, or silently degrade results. This page defines the health metrics, the check flow, and the auto-remediation actions that keep the GPU pool reliable.

Health Metrics

Metric	Source	Threshold	Action on Breach
ECC errors (corrected)	nvidia-smi	> 100 per hour	Log warning; schedule maintenance
ECC errors (uncorrected)	nvidia-smi	Any	Immediate quarantine
Thermal (GPU temp)	nvidia-smi	> 85 C	Throttle; alert; quarantine if sustained
XID errors	dmesg / NVML	Any non-zero	Immediate quarantine
Memory failures	nvidia-smi	Any row remapping event	Immediate quarantine
PCIe errors	lspci / dmesg	> 10 per minute	Alert; inspect link quality
GPU utilization	NVML	< 5% for 10 min while assigned	Alert; check for hung process

Health Check Flow

Quarantine Procedure

A quarantined GPU is immediately removed from the scheduler pool. Active tasks on that GPU are marked for retry on another node. The GPU health agent records:

Timestamp of quarantine
Metric that triggered it
Value at trigger
Last assigned experiment ID

The GPU remains quarantined until a manual review clears it or auto-remediation succeeds.

Auto-Remediation

Condition	Action	Cooldown
Corrected ECC spike	Reset GPU; clear caches	5 minutes
Thermal threshold	Reduce clock; increase fan	10 minutes
Hung process (low utilization)	Kill process; release GPU	2 minutes
PCIe errors	Reseat notification (alert only)	N/A

Auto-remediation is never attempted for uncorrected ECC errors, XID errors, or memory failures. These require physical inspection or replacement.

Manual Recovery

After remediation, an operator runs a full health check and a short burn-in test. The GPU re-enters the pool only if all metrics pass for the duration of the burn-in.

Cross-Links