Skip to main content
The failure modes catalog documents every known failure mode in EMEP. Each entry specifies how the failure is detected, contained, recovered, retried, rolled back, logged, alerted, and what final status it produces.

​
FM-001: MergeEngine GPU OOM

  • Detection: CUDA out-of-memory exception during tensor operation.
  • Containment: The merge job aborts. GPU memory is freed. Other jobs are unaffected.
  • Recovery: Retry with reduced batch size or reduced memory fraction. If still failing, retry on a GPU with more memory.
  • Retry Policy: Up to 3 retries with halving memory fraction each time.
  • Rollback Policy: No state change to source models. The candidate model is not created.
  • Logging: Full stack trace, GPU memory snapshot, and model IDs are logged.
  • Alerting: Alert if 3+ OOM failures occur in 10 minutes on the same node.
  • Final Status: FAIL.

​
FM-002: ModelCompatibilityAnalyzer Incompatible Models

  • Detection: Tensor shape mismatch, tokenizer mismatch, or architecture mismatch.
  • Containment: The merge request is rejected before any GPU allocation.
  • Recovery: User selects different models or adjusts compatibility parameters.
  • Retry Policy: No automatic retry. User-initiated only.
  • Rollback Policy: No state change.
  • Logging: Detailed mismatch report with layer names and expected vs actual shapes.
  • Alerting: No alert. This is expected behavior.
  • Final Status: INCOMPATIBLE.

​
FM-003: ArtifactStore Corruption

  • Detection: SHA-256 checksum mismatch on download or signature verification failure.
  • Containment: The artifact is quarantined. No downstream component loads it.
  • Recovery: Restore from backup replica. If no replica, mark artifact as lost.
  • Retry Policy: Retry download from a different replica up to 3 times.
  • Rollback Policy: If the artifact was a merged model, revert the model record to INVALID.
  • Logging: Artifact ID, expected checksum, actual checksum, and replica IDs.
  • Alerting: Critical alert on any corruption detection.
  • Final Status: INVALID or FAILED.

​
FM-004: BenchmarkEngine Backend Crash

  • Detection: InferenceBackend process exits or returns an error during benchmark.
  • Containment: The benchmark job aborts. Partial results are discarded.
  • Recovery: Retry on a different backend instance. If persistent, mark backend as unhealthy.
  • Retry Policy: Up to 2 retries on different nodes.
  • Rollback Policy: No state change to the candidate model.
  • Logging: Backend logs, model ID, benchmark suite, and error code.
  • Alerting: Alert if backend crash rate exceeds 5% in 1 hour.
  • Final Status: FAIL.

​
FM-005: EvolutionEngine Stagnation

  • Detection: No improvement in fitness for N generations (configurable, default 20).
  • Containment: The evolution run continues but is flagged for review.
  • Recovery: Increase mutation rate, inject new random individuals, or restart with different seed population.
  • Retry Policy: No automatic retry. Operator intervention required.
  • Rollback Policy: Archive the run. Do not promote any candidate.
  • Logging: Fitness trajectory, generation count, and population diversity metrics.
  • Alerting: Warning alert after 20 generations of stagnation.
  • Final Status: INCOMPLETE (operator may mark COMPLETED or FAILED).

​
FM-006: ExperimentTracker Database Unavailable

  • Detection: Connection timeout or query failure to the database backend.
  • Containment: In-flight experiments continue but cannot update state. New experiments are queued.
  • Recovery: Retry with exponential backoff. If persistent, switch to a read-only replica or local cache.
  • Retry Policy: Up to 5 retries with backoff (1s, 2s, 4s, 8s, 16s).
  • Rollback Policy: If the database recovers, replay queued state updates. If not, experiments may have stale state.
  • Logging: Connection errors, retry count, and fallback actions.
  • Alerting: Critical alert after 3 consecutive failures.
  • Final Status: Experiments may remain in RUNNING until manually reconciled.

​
FM-007: Invalid Tokenizer Upload

  • Detection: Tokenizer JSON parse failure or vocabulary size mismatch.
  • Containment: The model cannot transition to VALIDATED. It remains in IMPORTED.
  • Recovery: User uploads a corrected tokenizer.
  • Retry Policy: No automatic retry. User-initiated only.
  • Rollback Policy: No state change.
  • Logging: Parse error details and expected vs actual vocabulary size.
  • Alerting: No alert.
  • Final Status: INVALID (until corrected).

​
FM-008: Network Partition Between Services

  • Detection: RPC timeout or connection refused between MergeEngine and ModelRegistry, or EvaluationEngine and BenchmarkEngine.
  • Containment: The affected operation pauses. No partial state is committed.
  • Recovery: Retry with exponential backoff. If partition lasts longer than job timeout, fail the job.
  • Retry Policy: Retry until timeout (default 3600s for merges, 7200s for benchmarks).
  • Rollback Policy: No state change if the partition is resolved before timeout. If timed out, mark job FAILED.
  • Logging: Partition duration, affected services, and retry count.
  • Alerting: Critical alert if partition lasts longer than 60 seconds.
  • Final Status: FAILED or CANCELLED.

​
FM-009: Catastrophic Forgetting in Fine-Tuned Model

  • Detection: Regression evaluation shows a score drop on prior tasks.
  • Containment: The model is blocked from promotion. Status is REGRESSION.
  • Recovery: Apply forgetting controls (rehearsal, EWC, distillation) and retrain.
  • Retry Policy: User-initiated retraining.
  • Rollback Policy: Revert to the pre-fine-tuning checkpoint.
  • Logging: Pre and post scores, task names, and control methods attempted.
  • Alerting: No alert. Expected during experimentation.
  • Final Status: REGRESSION (may become PASS after retraining).

​
FM-010: Dependency CVE Introduction

  • Detection: Security scan finds a new CVE in a dependency.
  • Containment: The CI gate blocks the merge. The dependency is not introduced.
  • Recovery: Upgrade the dependency, apply a patch, or add an exception with justification.
  • Retry Policy: Retry after fix.
  • Rollback Policy: Revert the dependency change.
  • Logging: CVE ID, severity, affected package, and fix version.
  • Alerting: Alert on HIGH or CRITICAL CVEs.
  • Final Status: FAIL (CI gate) or PASS (after fix).

​
FM-011: Quantization Engine Numerical Overflow

  • Detection: NaN or Inf in quantized output tensors.
  • Containment: The quantization job aborts. No quantized artifact is produced.
  • Recovery: Reduce dynamic range, switch to a higher bit width, or apply scaling.
  • Retry Policy: Up to 2 retries with adjusted parameters.
  • Rollback Policy: No state change to the source model.
  • Logging: Layer name, tensor statistics, and quantization config.
  • Alerting: Alert if 3+ overflow failures in 1 hour.
  • Final Status: FAIL.

​
FM-012: Unauthorized Model Access

  • Detection: API request for a model without sufficient permissions.
  • Containment: The request is rejected with HTTP 403.
  • Recovery: User requests elevated permissions or accesses a public model.
  • Retry Policy: No automatic retry.
  • Rollback Policy: No state change.
  • Logging: Requester ID, model ID, and required permission level.
  • Alerting: Alert on repeated unauthorized access attempts from the same source.
  • Final Status: Rejected.

​
Integration Points

  • Risk Register: each failure mode maps to a risk ID. See Risk Register.
  • Runbook: recovery procedures are detailed in Runbook.
  • Disaster Recovery: persistent failures trigger DR protocols. See Disaster Recovery.
  • Monitoring and Alerting: detection and alerting integrate with Monitoring and Alerting.