EMEP Failure Injection and Recovery Testing

Failure injection cases for EMEP including GPU crash, storage 500, corrupted weights, invalid tokenizer, and network partition with recovery expectations.

Failure injection tests verify that EMEP components recover gracefully from faults. Each injection case defines the fault, the expected detection, containment, recovery, and final status.

Injection Cases

GPU Crash

• Fault: Kill the GPU process or trigger a CUDA error during merge or benchmark.
• Detection: GPU Orchestration detects the process exit code or CUDA exception.
• Containment: The job is marked FAILED. Other jobs on the same node are rescheduled.
• Recovery: The job is retried on a different GPU node up to 3 times. If all retries fail, the experiment state is FAILED.
• Final Status: FAILED after retries exhausted. Logs and partial artifacts are preserved in ArtifactStore.

Storage 500

• Fault: Return HTTP 500 from ArtifactStore or ModelRegistry backend.
• Detection: HTTP client catches the 500 status.
• Containment: The operation aborts. No partial writes are committed.
• Recovery: Exponential backoff retry (1s, 2s, 4s, 8s). If persistent, the operation fails and alerts are sent.
• Final Status: FAILED if retries exhausted. The user receives a structured error.

Corrupted Weights

• Fault: Inject a bit flip or truncate a weights file before loading.
• Detection: ModelLoader checksum validation fails or TensorEngine detects a shape/dtype mismatch.
• Containment: The model is not loaded. Memory is not allocated.
• Recovery: The user is prompted to re-register the model from a clean source. No automatic retry.
• Final Status: INVALID. The model record is updated with an error annotation.

Invalid Tokenizer

• Fault: Provide a tokenizer with mismatched vocabulary size or malformed JSON.
• Detection: ModelCompatibilityAnalyzer rejects the tokenizer during VALIDATED transition.
• Containment: The model cannot transition past IMPORTED. It is blocked from merge and evaluation.
• Recovery: The user uploads a corrected tokenizer. The validation is re-run.
• Final Status: VALIDATED after correction, or INVALID if correction fails.

Network Partition

• Fault: Drop network between MergeEngine and ModelRegistry or between EvaluationEngine and BenchmarkEngine.
• Detection: RPC timeout or connection refused.
• Containment: The operation pauses. In-flight jobs are marked RUNNING but stalled.
• Recovery: Retry with exponential backoff. If partition lasts longer than the job timeout, the job is marked FAILED.
• Final Status: FAILED if timeout exceeded. CANCELLED if the user aborts manually.

Recovery Expectations Summary

Fault	Detection	Retry	Max Retries	Final Status
GPU crash	Process exit / CUDA exception	Yes, different node	3	FAILED
Storage 500	HTTP 500	Yes, exponential backoff	4	FAILED
Corrupted weights	Checksum / shape mismatch	No	0	INVALID
Invalid tokenizer	Validation failure	No	0	INVALID (until fixed)
Network partition	RPC timeout	Yes, exponential backoff	Until timeout	FAILED or CANCELLED

Test Execution

Failure injection tests run in a dedicated test environment with chaos tooling. They are not run in production. The test environment mirrors production topology but uses synthetic faults.

Integration Points

• Runbook: recovery procedures are documented in Runbook.
• Disaster Recovery: persistent failures trigger disaster recovery protocols. See Disaster Recovery.
• Alerting: detection events trigger alerts per Alerting.
• Monitoring: fault injection metrics are exported for analysis. See Monitoring.