Alert Catalog and Escalation Procedures - EMEP

Alerts notify operators when EMEP metrics cross thresholds or when components fail health checks. This page defines the alert catalog, severity levels, routing rules, and the escalation flow for alerts that are not acknowledged in time.

Severity Levels

Level	Name	Response Time	Example
P1	Critical	5 minutes	All GPUs offline; registry database unreachable
P2	High	15 minutes	MergeEngine crash loop; safety evaluation FAIL
P3	Medium	1 hour	Storage capacity over 80%; experiment queue depth over 100
P4	Low	4 hours	Model deprecation approaching; key rotation due in 7 days

Alert Catalog

Alert Name	Severity	Condition	Routing	Runbook Link
gpu.all_unavailable	P1	Zero GPUs report healthy	On-call SRE	GPU Health
registry.db_unavailable	P1	Registry connection failure for 60 seconds	On-call SRE	Runbook
merge.crash_loop	P2	MergeEngine restarts more than 3 times in 5 minutes	On-call SRE	Runbook
safety.eval_fail	P2	Any candidate receives safety evaluation FAIL	Security team	Security Requirements
storage.capacity_high	P3	Used capacity over 80% for 10 minutes	Storage ops	Storage Operations
experiment.queue_deep	P3	Queue depth over 100 for 30 minutes	Experiment lead	Runbook
key.rotation_due	P4	Signing key expires in 7 days	Security automation	Artifact Signing
model.deprecation_due	P4	Model reaches deprecation date in 14 days	Registry curator	Model Registry

Escalation Flow

P1 and P2 alerts page the on-call SRE immediately. If the SRE does not acknowledge within the SLA, the alert escalates to the next tier. P3 and P4 alerts open tickets and notify the component owner. Automation handles P4 alerts where possible.

Routing Rules

Business hours: route to regional on-call
Outside hours: page global on-call for P1 and P2; queue P3 and P4 for next business day
Maintenance windows: suppress capacity and queue alerts; keep safety and GPU alerts active

Cross-Links