EMEP Integration Test Plan and Scenarios

End-to-end integration test scenarios for EMEP covering register, merge, evaluate, and promote workflows.

Integration tests verify that EMEP components work together across realistic workflows. Tests use real backends where possible and fall back to high-fidelity mocks only for unavailable GPU or storage resources.

Scenario: Register to Promote

This scenario exercises ModelRegistry, ModelCompatibilityAnalyzer, MergeEngine, MergeValidation, EvaluationEngine, BenchmarkEngine, and DeploymentManager in sequence.

Scenario: Evolution Run

1. Seed population with 4 registered models.
2. Start EvolutionEngine with NSGA-II (Deb et al. 2002).
3. CandidateGenerator produces offspring via crossover and mutation.
4. MergeEngine executes merges for each offspring.
5. EvaluationEngine runs benchmarks on each candidate.
6. FitnessEngine computes multi-objective fitness.
7. Selection reduces population to configured size.
8. Loop until max generations or convergence.
9. Archive final population in ExperimentTracker.

Scenario: Offline Pack and Deploy

1. Register a model with emep model register.
2. Run emep offline pack to produce a self-contained package.
3. Transfer package to air-gapped environment.
4. Load package via DeploymentManager.
5. Validate inference with InferenceBackend.
6. Promote to DEPLOYED.

Test Data

Integration tests use a curated set of small models and datasets stored in ArtifactStore. Model sizes are kept small to fit CI runners. Datasets use the Optimization Set split only; the Hidden Test Set is never used in integration tests.

Backend Matrix

Backend	Used In	Mock Fallback
vLLM	BenchmarkEngine, InferenceBackend	Mock with fixed latencies
Kubernetes GPU	GPU Orchestration	Kind + CPU simulation
S3	ArtifactStore	Local MinIO
Postgres	ModelRegistry, ExperimentTracker	SQLite

Failure Injection in Integration Tests

Selected integration tests inject controlled failures:

• Kill MergeEngine mid-merge and verify ArtifactStore cleanup.
• Drop network to ModelRegistry and verify retry with exponential backoff.
• Corrupt a checkpoint file and verify ModelLoader error handling.

See Failure Injection for the full catalog.

Integration Points

• CI: integration tests run on every commit and nightly.
• ExperimentTracker: each scenario is logged as an experiment.
• ArtifactStore: stores test outputs, logs, and failure artifacts.
• Performance Testing: selected scenarios also collect latency and memory metrics. See Performance Testing.