EMEP Testing Strategy and Pyramid

Testing pyramid for EMEP covering unit, integration, numerical, merge regression, benchmark regression, performance, reproducibility, failure recovery, and security tests.

EMEP testing is organized as a pyramid. Unit tests form the base. Integration and numerical tests sit in the middle. Merge regression, benchmark regression, performance, reproducibility, failure recovery, and security tests form the top. Every test category has a defined scope, frequency, and owner.

Testing Pyramid

Unit Tests

Unit tests cover individual components in isolation: ModelRegistry, ModelLoader, ModelCompatibilityAnalyzer, TensorEngine, MergeEngine, MergeStrategy, CandidateGenerator, EvolutionEngine, EvaluationEngine, BenchmarkEngine, FitnessEngine, ExperimentTracker, ArtifactStore, DatasetRegistry, QuantizationEngine, InferenceBackend, DeploymentManager. Each component has a target coverage threshold. See Unit Test Plan.

Integration Tests

Integration tests verify end-to-end workflows: register, merge, evaluate, promote. They exercise multiple components together with real or mocked backends. See Integration Test Plan.

Numerical Tests

Numerical tests enforce tolerance thresholds per dtype (fp32, bf16, fp16, fp8, int8). They validate NaN/Inf policies and reference math specifications. See Numerical Testing and Tensor Math.

Merge Regression Tests

Merge regression tests compare merged outputs against golden references for each MergeStrategy. Tolerance policies are strategy-specific. See Model Merge Tests.

Benchmark Regression Tests

Benchmark regression tests run the full benchmark harness on known models and compare scores against historical baselines. Any deviation triggers investigation. See Evaluation Testing.

Performance Tests

Performance tests measure latency, throughput, memory, tokens per second, and time to first token (TTFT). Results are compared against Performance Validation thresholds.

Reproducibility Tests

Reproducibility tests rerun experiments with fixed seeds and compare outputs bit-for-bit or within tolerance. They validate the reproducibility protocol. See Reproducibility Protocol.

Failure Recovery Tests

Failure recovery tests inject faults (GPU crash, storage failure, corrupted weights) and verify recovery behavior. See Failure Injection.

Security Tests

Security tests verify signature verification, tampered manifest detection, malicious model detection, and dependency CVE scanning. See Security Testing.

Test Frequency

Category	Frequency	Trigger
Unit	Every commit	CI
Integration	Every commit	CI
Numerical	Every commit	CI
Merge regression	Every merge strategy change	CI + nightly
Benchmark regression	Nightly	Scheduled
Performance	Weekly + on release	Scheduled + manual
Reproducibility	Weekly	Scheduled
Failure recovery	Weekly	Scheduled
Security	Weekly + on dependency update	Scheduled + CI

Integration Points

• CI/CD: all test categories run in CI with artifact collection.
• ExperimentTracker: logs test runs as experiments with state COMPLETED or FAILED.
• ArtifactStore: stores golden references, baselines, and test outputs.
• Alerting: test failures trigger alerts per Alerting.