> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# EMEP Testing Strategy and Pyramid

> Testing pyramid for EMEP covering unit, integration, numerical, merge regression, benchmark regression, performance, reproducibility, failure recovery, and security tests.

EMEP testing is organized as a pyramid. Unit tests form the base. Integration and numerical tests sit in the middle. Merge regression, benchmark regression, performance, reproducibility, failure recovery, and security tests form the top. Every test category has a defined scope, frequency, and owner.

## Testing Pyramid

```mermaid theme={null}
flowchart TD
    subgraph TOP["Top: System & Acceptance"]
        A["Security Testing"]
        B["Failure Recovery Testing"]
        C["Reproducibility Testing"]
        D["Performance Testing"]
        E["Benchmark Regression Testing"]
        F["Merge Regression Testing"]
    end
    subgraph MID["Middle: Integration & Numerical"]
        G["Numerical Testing"]
        H["Integration Test Plan"]
    end
    subgraph BASE["Base: Unit"]
        I["Unit Test Plan"]
    end
    I --> H
    H --> G
    G --> F
    F --> E
    E --> D
    D --> C
    C --> B
    B --> A
```

## Unit Tests

Unit tests cover individual components in isolation: ModelRegistry, ModelLoader, ModelCompatibilityAnalyzer, TensorEngine, MergeEngine, MergeStrategy, CandidateGenerator, EvolutionEngine, EvaluationEngine, BenchmarkEngine, FitnessEngine, ExperimentTracker, ArtifactStore, DatasetRegistry, QuantizationEngine, InferenceBackend, DeploymentManager. Each component has a target coverage threshold. See [Unit Test Plan](/testing/unit-test-plan).

## Integration Tests

Integration tests verify end-to-end workflows: register, merge, evaluate, promote. They exercise multiple components together with real or mocked backends. See [Integration Test Plan](/testing/integration-test-plan).

## Numerical Tests

Numerical tests enforce tolerance thresholds per dtype (fp32, bf16, fp16, fp8, int8). They validate NaN/Inf policies and reference math specifications. See [Numerical Testing](/testing/numerical-testing) and [Tensor Math](/math/tensor-math).

## Merge Regression Tests

Merge regression tests compare merged outputs against golden references for each MergeStrategy. Tolerance policies are strategy-specific. See [Model Merge Tests](/testing/model-merge-tests).

## Benchmark Regression Tests

Benchmark regression tests run the full benchmark harness on known models and compare scores against historical baselines. Any deviation triggers investigation. See [Evaluation Testing](/testing/evaluation-testing).

## Performance Tests

Performance tests measure latency, throughput, memory, tokens per second, and time to first token (TTFT). Results are compared against [Performance Validation](/deployment/performance-validation) thresholds.

## Reproducibility Tests

Reproducibility tests rerun experiments with fixed seeds and compare outputs bit-for-bit or within tolerance. They validate the reproducibility protocol. See [Reproducibility Protocol](/tracking/reproducibility-protocol).

## Failure Recovery Tests

Failure recovery tests inject faults (GPU crash, storage failure, corrupted weights) and verify recovery behavior. See [Failure Injection](/testing/failure-injection).

## Security Tests

Security tests verify signature verification, tampered manifest detection, malicious model detection, and dependency CVE scanning. See [Security Testing](/testing/security-testing).

## Test Frequency

| Category             | Frequency                     | Trigger            |
| -------------------- | ----------------------------- | ------------------ |
| Unit                 | Every commit                  | CI                 |
| Integration          | Every commit                  | CI                 |
| Numerical            | Every commit                  | CI                 |
| Merge regression     | Every merge strategy change   | CI + nightly       |
| Benchmark regression | Nightly                       | Scheduled          |
| Performance          | Weekly + on release           | Scheduled + manual |
| Reproducibility      | Weekly                        | Scheduled          |
| Failure recovery     | Weekly                        | Scheduled          |
| Security             | Weekly + on dependency update | Scheduled + CI     |

## Integration Points

* **CI/CD**: all test categories run in CI with artifact collection.
* **ExperimentTracker**: logs test runs as experiments with state COMPLETED or FAILED.
* **ArtifactStore**: stores golden references, baselines, and test outputs.
* **Alerting**: test failures trigger alerts per [Alerting](/operations/alerting).