Model Rollback: Regression Recovery Protocol

Rollback flowchart for production regression. Detect, freeze, restore, verify, resume, and post-mortem. Automated and manual triggers.

ModelRollback defines the protocol for restoring a previous production version when a regression is detected. This page specifies the rollback flowchart, automated and manual triggers, verification steps, and the post-mortem requirement.

Rollback Flowchart

The protocol below shows the sequence from regression detection to full recovery.

Detection Triggers

Trigger	Source	Response Time
Automated latency alert	Monitoring	Immediate
Automated error rate alert	Monitoring	Immediate
Benchmark regression	EvaluationEngine	Within evaluation window
Manual trigger	On-call engineer or Release Manager	Immediate

Automated triggers initiate the freeze step without human approval. Manual triggers require explicit confirmation from the on-call engineer or Release Manager.

Freeze Traffic

Freeze Traffic stops all new requests from reaching the current version. In-flight requests are allowed to complete or timed out based on the backend configuration. DeploymentManager updates the routing layer to send traffic to a maintenance endpoint or to the previous version if already staged.

Restore vN-1

The previous version is restored from ArtifactStore. The restoration process:

1. Locate the artifact for version N-1 using ModelRegistry
2. Verify the artifact hash against the registry record
3. Load the artifact into InferenceBackend
4. Run a smoke test with reference prompts

Verify Hash

Hash verification compares the restored artifact SHA-256 against the canonical hash stored at release time. A mismatch aborts the automated rollback and escalates to manual recovery.

Resume Traffic

After successful verification, traffic is gradually resumed:

1. Internal health checks pass for 60 seconds
2. 1% traffic shift for 5 minutes
3. 10% traffic shift for 5 minutes
4. 50% traffic shift for 5 minutes
5. Full traffic restoration

Any alert during the ramp re-triggers the freeze.

RPO and RTO

Metric	Target	Status
RPO (Recovery Point Objective)	Zero unprocessed requests	Placeholder pending operational calibration
RTO (Recovery Time Objective)	Under 5 minutes	Placeholder pending operational calibration

RPO and RTO values are engineering assumptions. Final targets will be set after load testing and operational rehearsal.

Post-Mortem

Every rollback requires a post-mortem within 48 hours. The post-mortem is stored in ExperimentTracker and includes:

• Regression root cause
• Detection latency
• Rollback execution time
• Verification results
• Recommended preventive action

Integration

ModelRollback is triggered by DeploymentManager and Monitoring. It reads artifacts from ArtifactStore and updates ModelRegistry status. Post-mortems are linked to Model Lineage.