> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# EvaluationEngine and BenchmarkEngine Framework

> Specification for the EMEP evaluation framework, covering the full candidate evaluation pipeline and explicit split isolation between Optimization, Validation, and Hidden Test sets.

The EvaluationEngine and BenchmarkEngine execute candidate assessment in EMEP. They run the full pipeline from model integrity checks through metric aggregation to candidate status assignment. This page specifies the complete evaluation flowchart, split isolation rules, and integration with the EvolutionEngine.

## Candidate Evaluation Flowchart

```mermaid theme={null}
flowchart TD
    START([START]) --> CANDIDATE[Candidate Model]
    CANDIDATE --> INTEGRITY[Integrity Check]
    INTEGRITY -->|Fail| INVALID[Status: INVALID]
    INTEGRITY -->|Pass| LOAD[Load Model]
    LOAD --> PROFILE[Select Benchmark Profile]
    PROFILE --> RUN[Run Benchmark Suite]
    RUN --> COLLECT[Collect Metrics]
    COLLECT --> CALC[Calculate Scores]
    CALC --> AGG[Aggregate Results]
    AGG --> BASELINE[Compare Baseline]
    BASELINE --> REGRESS[Regression Check]
    REGRESS --> FITNESS[Fitness Calculation]
    FITNESS --> PERSIST[Persist Results]
    PERSIST --> STATUS[Candidate Status]
    STATUS --> END([END])
    INVALID --> END
```

## Phase Descriptions

**Candidate Model**

The input is a materialized candidate from the MergeEngine. It includes the model weights, genome, and parent references. The candidate must be in CANDIDATE lifecycle state.

**Integrity Check**

The EvaluationEngine verifies:

* Model file checksums match the ArtifactStore record
* Required metadata files are present
* Tensor shapes are consistent with the genome specification
* No NaN or Inf values in weight tensors (sampled check)

Failures produce INVALID status and terminate evaluation.

**Load Model**

The ModelLoader loads the candidate into the InferenceBackend. Loading includes device placement, dtype configuration, and optional quantization application. Memory usage is recorded at this stage.

**Select Benchmark Profile**

The BenchmarkEngine selects which benchmarks to run based on:

* Experiment configuration
* Candidate genome (for example code benchmarks for code-merge candidates)
* Available compute budget

Profiles are pre-defined sets of benchmarks with specified splits and repetition counts.

**Run Benchmark Suite**

Benchmarks execute against the Optimization Set. Each benchmark runs with fixed seeds and decoding parameters. Results are collected per-sample and per-benchmark.

<Warning>
  The Hidden Test Set is never accessed during this phase. The Validation Set is also excluded during evolution. Only the Optimization Set drives evaluation results that feed back into the EvolutionEngine.
</Warning>

**Collect Metrics**

Raw outputs are converted to metrics: accuracy, exact match, ROUGE, BLEU, calibrated log-likelihood, latency, throughput, and memory usage. Metrics are stored with full provenance: benchmark ID, sample ID, seed, and timestamp.

**Calculate Scores**

Per-benchmark scores are computed from metrics. Scoring methods include:

* Mean accuracy across samples
* Macro-average across categories
* Weighted combination for composite benchmarks

**Aggregate Results**

Scores are aggregated into a single result dictionary per candidate. Aggregation respects benchmark weights from the experiment configuration.

**Compare Baseline**

The candidate's aggregate scores are compared against the baseline model specified in the experiment. The baseline is typically the highest-performing parent or a previous release candidate.

**Regression Check**

Statistical tests determine if the candidate is significantly worse than baseline on any critical metric. The RegressionEngine uses bootstrap confidence intervals. See [Regression Evaluation](/evaluation/regression-evaluation) for the full protocol.

**Fitness Calculation**

The FitnessEngine computes scalar or vector fitness from the aggregated Optimization Set results. This value is passed back to the EvolutionEngine. See [Fitness](/evolution/fitness) for formulas.

**Persist Results**

All raw metrics, scores, and status are written to the ExperimentTracker and ArtifactStore. The candidate record is updated with evaluation timestamps and resource usage.

**Candidate Status**

Final status is assigned:

* **PASS**: All checks passed, fitness above threshold
* **FAIL**: Below minimum thresholds
* **REGRESSION**: Statistically worse than baseline on critical metrics
* **INVALID**: Integrity or loading failure
* **INCOMPLETE**: Evaluation interrupted

## Split Isolation Diagram

```mermaid theme={null}
flowchart LR
    subgraph Splits
        OPT[Optimization Set]
        VAL[Validation Set]
        HID[Hidden Test Set]
    end
    subgraph Consumers
        EVOL[EvolutionEngine]
        REPORT[Validation Report]
        FINAL[Final Assessment]
    end
    OPT --> EVOL
    OPT --> REPORT
    VAL --> REPORT
    HID --> FINAL
    HID -.->|Blocked| EVOL
    VAL -.->|Blocked| EVOL
```

## Integration Points

| Component         | Role                                              |
| ----------------- | ------------------------------------------------- |
| ModelLoader       | Loads candidate weights                           |
| InferenceBackend  | Executes inference during benchmarks              |
| BenchmarkEngine   | Manages benchmark execution and metric collection |
| FitnessEngine     | Computes fitness from results                     |
| RegressionEngine  | Detects performance regression                    |
| ExperimentTracker | Logs all evaluation events                        |
| ArtifactStore     | Stores model artifacts and results                |

<Info>
  Split isolation is enforced at the data layer, not just by convention. The DatasetRegistry tags every sample with its split, and the BenchmarkEngine validates that only Optimization Set samples are loaded during evolution-phase evaluation.
</Info>