Disaster Recovery Plan and Procedures

Failure classification, recovery procedures, RPO/RTO targets, and quarantine protocols for total site loss, storage loss, registry loss, and GPU cluster loss.

EMEP experiments run for hours or days on expensive GPU resources. Losing state mid-experiment is costly. This page defines the disaster recovery plan, failure classifications, and the decision flow for recovery versus abort.

Recovery Objectives

Objective	Target	Tracking ID
Recovery Point Objective (RPO)	TBD	OQ-009
Recovery Time Objective (RTO)	TBD	OQ-009

These values are set per deployment tier during initial provisioning.

Failure Classes

Class	Description	Typical Cause
Total site loss	Complete unavailability of primary deployment	Datacenter failure, regional outage
Storage loss	Object store or database corruption	Disk failure, operator error, ransomware
Registry loss	ModelRegistry metadata unavailable	Database corruption, schema migration failure
Network partition	Split between control plane and GPU workers	Switch failure, misconfigured firewall
GPU cluster loss	All GPU nodes unavailable	Power loss, thermal event, firmware bug

Recovery Flow

A failure is recoverable if the experiment checkpoint, model weights, and registry metadata are intact. Validation checks hash integrity, registry consistency, and experiment state machine validity.

Per-Class Procedures

Total Site Loss

Fail over to the standby deployment. Restore from the latest backup. Replay experiments from the last checkpoint. All in-flight experiments are marked FAILED and must be resubmitted.

Storage Loss

Restore the object store or database from backup. Run a consistency scan to identify artifacts with missing chunks. Re-download or re-merge artifacts that fail the scan. Experiments referencing intact artifacts may resume.

Registry Loss

Restore the registry database from backup. Rebuild indexes from artifact manifests stored in the object store. Verify that every registered model has a corresponding artifact. Flag orphaned artifacts for review.

Network Partition

If the partition heals within the task timeout, workers reconnect and resume. If the timeout expires, tasks are marked INCOMPLETE and rescheduled. GPU nodes that miss three heartbeats are marked unavailable.

GPU Cluster Loss

Reschedule all active tasks to remaining GPU nodes. If no GPUs are available, queue tasks with a priority boost. When GPUs return, run a health check before marking them available.

Quarantine Protocol

Affected resources are moved to a quarantine prefix with restricted access. The incident record includes failure class, timestamp, affected artifacts or nodes, recovery action taken, and operator identity.

Cross-Links

• Backup Strategy
• Runbook
• GPU Health