> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Rollback: Regression Recovery Protocol

> Rollback flowchart for production regression. Detect, freeze, restore, verify, resume, and post-mortem. Automated and manual triggers.

ModelRollback defines the protocol for restoring a previous production version when a regression is detected. This page specifies the rollback flowchart, automated and manual triggers, verification steps, and the post-mortem requirement.

## Rollback Flowchart

The protocol below shows the sequence from regression detection to full recovery.

```mermaid theme={null}
flowchart TD
    START([START]) --> DETECT[Detect Regression]
    DETECT --> FREEZE[Freeze Traffic]
    FREEZE --> RESTORE[Restore vN-1]
    RESTORE --> VERIFY[Verify Hash + Smoke Test]
    VERIFY --> PASS{Pass?}
    PASS -->|YES| RESUME[Resume Traffic]
    PASS -->|NO| ESCALATE[Escalate to On-Call + Manual Recovery]
    RESUME --> POST[Post-Mortem]
    POST --> END_PASS([END: RESTORED])
    ESCALATE --> END_FAIL([END: MANUAL RECOVERY])
```

## Detection Triggers

| Trigger                    | Source                                    | Response Time            |
| -------------------------- | ----------------------------------------- | ------------------------ |
| Automated latency alert    | [Monitoring](/operations/monitoring)      | Immediate                |
| Automated error rate alert | [Monitoring](/operations/monitoring)      | Immediate                |
| Benchmark regression       | [EvaluationEngine](/evaluation/framework) | Within evaluation window |
| Manual trigger             | On-call engineer or Release Manager       | Immediate                |

Automated triggers initiate the freeze step without human approval. Manual triggers require explicit confirmation from the on-call engineer or Release Manager.

## Freeze Traffic

Freeze Traffic stops all new requests from reaching the current version. In-flight requests are allowed to complete or timed out based on the backend configuration. [DeploymentManager](/deployment/deployment-specification) updates the routing layer to send traffic to a maintenance endpoint or to the previous version if already staged.

## Restore vN-1

The previous version is restored from [ArtifactStore](/tracking/artifact-registry). The restoration process:

1. Locate the artifact for version N-1 using [ModelRegistry](/tracking/model-registry)
2. Verify the artifact hash against the registry record
3. Load the artifact into [InferenceBackend](/deployment/inference)
4. Run a smoke test with reference prompts

## Verify Hash

Hash verification compares the restored artifact SHA-256 against the canonical hash stored at release time. A mismatch aborts the automated rollback and escalates to manual recovery.

## Resume Traffic

After successful verification, traffic is gradually resumed:

1. Internal health checks pass for 60 seconds
2. 1% traffic shift for 5 minutes
3. 10% traffic shift for 5 minutes
4. 50% traffic shift for 5 minutes
5. Full traffic restoration

Any alert during the ramp re-triggers the freeze.

## RPO and RTO

| Metric                         | Target                    | Status                                      |
| ------------------------------ | ------------------------- | ------------------------------------------- |
| RPO (Recovery Point Objective) | Zero unprocessed requests | Placeholder pending operational calibration |
| RTO (Recovery Time Objective)  | Under 5 minutes           | Placeholder pending operational calibration |

<Note>
  RPO and RTO values are engineering assumptions. Final targets will be set after load testing and operational rehearsal.
</Note>

## Post-Mortem

Every rollback requires a post-mortem within 48 hours. The post-mortem is stored in [ExperimentTracker](/tracking/experiment-tracking) and includes:

* Regression root cause
* Detection latency
* Rollback execution time
* Verification results
* Recommended preventive action

## Integration

ModelRollback is triggered by [DeploymentManager](/deployment/deployment-specification) and [Monitoring](/operations/monitoring). It reads artifacts from [ArtifactStore](/tracking/artifact-registry) and updates [ModelRegistry](/tracking/model-registry) status. Post-mortems are linked to [Model Lineage](/tracking/model-lineage).