> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU Health Monitoring and Remediation

> GPU health metrics, automated health checks, quarantine procedures, and auto-remediation for ECC errors, thermal events, XID errors, and memory failures.

GPU nodes are the most expensive and failure-prone resources in EMEP. A single faulty GPU can corrupt a merge, waste evaluation time, or silently degrade results. This page defines the health metrics, the check flow, and the auto-remediation actions that keep the GPU pool reliable.

## Health Metrics

| Metric                   | Source        | Threshold                       | Action on Breach                         |
| ------------------------ | ------------- | ------------------------------- | ---------------------------------------- |
| ECC errors (corrected)   | nvidia-smi    | > 100 per hour                  | Log warning; schedule maintenance        |
| ECC errors (uncorrected) | nvidia-smi    | Any                             | Immediate quarantine                     |
| Thermal (GPU temp)       | nvidia-smi    | > 85 C                          | Throttle; alert; quarantine if sustained |
| XID errors               | dmesg / NVML  | Any non-zero                    | Immediate quarantine                     |
| Memory failures          | nvidia-smi    | Any row remapping event         | Immediate quarantine                     |
| PCIe errors              | lspci / dmesg | > 10 per minute                 | Alert; inspect link quality              |
| GPU utilization          | NVML          | \< 5% for 10 min while assigned | Alert; check for hung process            |

## Health Check Flow

```mermaid theme={null}
flowchart TD
    START([Poll GPU]) --> READ[Read Counters]
    READ --> THRESHOLD[Threshold Check]
    THRESHOLD --> HEALTHY{Healthy?}
    HEALTHY -->|YES| MARK[Mark Available]
    HEALTHY -->|NO| QUARANTINE[Quarantine GPU]
    QUARANTINE --> ALERT[Alert: Severity by Metric]
    ALERT --> AUTO[Auto-Remediation?]
    AUTO -->|YES| REMEDIATE[Attempt Remediation]
    AUTO -->|NO| MANUAL[Wait for Manual Action]
    REMEDIATE --> RETEST[Re-test after Cooldown]
    RETEST --> HEALTHY
    MARK --> END([End])
```

## Quarantine Procedure

A quarantined GPU is immediately removed from the scheduler pool. Active tasks on that GPU are marked for retry on another node. The GPU health agent records:

* Timestamp of quarantine
* Metric that triggered it
* Value at trigger
* Last assigned experiment ID

The GPU remains quarantined until a manual review clears it or auto-remediation succeeds.

## Auto-Remediation

| Condition                      | Action                           | Cooldown   |
| ------------------------------ | -------------------------------- | ---------- |
| Corrected ECC spike            | Reset GPU; clear caches          | 5 minutes  |
| Thermal threshold              | Reduce clock; increase fan       | 10 minutes |
| Hung process (low utilization) | Kill process; release GPU        | 2 minutes  |
| PCIe errors                    | Reseat notification (alert only) | N/A        |

Auto-remediation is never attempted for uncorrected ECC errors, XID errors, or memory failures. These require physical inspection or replacement.

## Manual Recovery

After remediation, an operator runs a full health check and a short burn-in test. The GPU re-enters the pool only if all metrics pass for the duration of the burn-in.

## Cross-Links

* [Monitoring](/operations/monitoring)
* [Alerting](/operations/alerting)
* [Disaster Recovery](/operations/disaster-recovery)