> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Alert Catalog and Escalation Procedures

> Alert definitions by severity, routing rules, acknowledgment requirements, and the escalation flow for unacknowledged alerts in EMEP operations.

Alerts notify operators when EMEP metrics cross thresholds or when components fail health checks. This page defines the alert catalog, severity levels, routing rules, and the escalation flow for alerts that are not acknowledged in time.

## Severity Levels

| Level | Name     | Response Time | Example                                                    |
| ----- | -------- | ------------- | ---------------------------------------------------------- |
| P1    | Critical | 5 minutes     | All GPUs offline; registry database unreachable            |
| P2    | High     | 15 minutes    | MergeEngine crash loop; safety evaluation FAIL             |
| P3    | Medium   | 1 hour        | Storage capacity over 80%; experiment queue depth over 100 |
| P4    | Low      | 4 hours       | Model deprecation approaching; key rotation due in 7 days  |

## Alert Catalog

| Alert Name               | Severity | Condition                                           | Routing             | Runbook Link                                             |
| ------------------------ | -------- | --------------------------------------------------- | ------------------- | -------------------------------------------------------- |
| gpu.all\_unavailable     | P1       | Zero GPUs report healthy                            | On-call SRE         | [GPU Health](/operations/gpu-health)                     |
| registry.db\_unavailable | P1       | Registry connection failure for 60 seconds          | On-call SRE         | [Runbook](/operations/runbook)                           |
| merge.crash\_loop        | P2       | MergeEngine restarts more than 3 times in 5 minutes | On-call SRE         | [Runbook](/operations/runbook)                           |
| safety.eval\_fail        | P2       | Any candidate receives safety evaluation FAIL       | Security team       | [Security Requirements](/security/security-requirements) |
| storage.capacity\_high   | P3       | Used capacity over 80% for 10 minutes               | Storage ops         | [Storage Operations](/operations/storage-operations)     |
| experiment.queue\_deep   | P3       | Queue depth over 100 for 30 minutes                 | Experiment lead     | [Runbook](/operations/runbook)                           |
| key.rotation\_due        | P4       | Signing key expires in 7 days                       | Security automation | [Artifact Signing](/security/artifact-signing)           |
| model.deprecation\_due   | P4       | Model reaches deprecation date in 14 days           | Registry curator    | [Model Registry](/tracking/model-registry)               |

## Escalation Flow

```mermaid theme={null}
flowchart TD
    START([Alert Fires]) --> SEVERITY{Severity?}
    SEVERITY -->|P1| ROUTE1[Route: On-call SRE]
    SEVERITY -->|P2| ROUTE2[Route: On-call SRE]
    SEVERITY -->|P3| ROUTE3[Route: Component Owner]
    SEVERITY -->|P4| ROUTE4[Route: Automation / Ticket]
    ROUTE1 --> ACK{Acknowledged?}
    ROUTE2 --> ACK
    ROUTE3 --> ACK
    ROUTE4 --> ACK
    ACK -->|YES within SLA| HANDLE[Handle Alert]
    ACK -->|NO| ESCALATE[Escalate to Next Tier]
    ESCALATE --> MANAGER[Page Manager / Director]
    HANDLE --> RESOLVE[Resolve + Document]
    RESOLVE --> END([End])
```

P1 and P2 alerts page the on-call SRE immediately. If the SRE does not acknowledge within the SLA, the alert escalates to the next tier. P3 and P4 alerts open tickets and notify the component owner. Automation handles P4 alerts where possible.

## Routing Rules

* Business hours: route to regional on-call
* Outside hours: page global on-call for P1 and P2; queue P3 and P4 for next business day
* Maintenance windows: suppress capacity and queue alerts; keep safety and GPU alerts active

## Cross-Links

* [Monitoring](/operations/monitoring)
* [GPU Health](/operations/gpu-health)
* [Runbook](/operations/runbook)