> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# EMEP Data Flow and Class Flow Map

> Data-flow diagram showing Weights, Tokenizer, Config, Task Vector, Metric, Manifest, and Signature flows, with explicit benchmark split boundaries.

EMEP processes seven primary data classes through its pipeline. This page maps each class to its origin, transformations, and destinations, with explicit boundaries between benchmark splits.

## Data Classes

| Class       | Origin                                                 | Transformations                                                                 | Destination                                                     |
| ----------- | ------------------------------------------------------ | ------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| Weights     | ModelLoader loads from ArtifactStore                   | MergeEngine combines via TensorEngine; QuantizationEngine reduces precision     | CandidateGenerator, InferenceBackend, DeploymentManager         |
| Tokenizer   | ModelLoader loads from ArtifactStore                   | ModelCompatibilityAnalyzer validates vocabulary alignment                       | EvaluationEngine (for benchmark tokenization), InferenceBackend |
| Config      | ModelLoader loads from ArtifactStore                   | MergeEngine produces merged config; QuantizationEngine adds quantization config | CandidateGenerator, DeploymentManager                           |
| Task Vector | TensorEngine computes (fine-tuned minus pre-trained)   | MergeEngine scales, trims, drops, or interpolates                               | MergeEngine output weights                                      |
| Metric      | EvaluationEngine computes from InferenceBackend output | FitnessEngine normalizes and aggregates                                         | ExperimentTracker logs, EvolutionEngine uses as fitness         |
| Manifest    | CandidateGenerator assembles from merge metadata       | ExperimentTracker appends experiment ID and timestamps                          | ArtifactStore, ModelRegistry                                    |
| Signature   | ArtifactStore computes ed25519 hash                    | DeploymentManager verifies on release                                           | ArtifactStore, DeploymentManager                                |

## Data Flow Diagram

```mermaid theme={null}
flowchart TB
    subgraph Input["Input"]
        W["Weights"]
        T["Tokenizer"]
        C["Config"]
    end

    subgraph Processing["Processing"]
        TV["Task Vector"]
        MW["Merged Weights"]
        QW["Quantized Weights"]
    end

    subgraph Evaluation["Evaluation"]
        M["Metric"]
        F["Fitness"]
    end

    subgraph Output["Output"]
        MF["Manifest"]
        S["Signature"]
        D["Deployed Model"]
    end

    subgraph Splits["Benchmark Splits"]
        OS["Optimization Set"]
        VS["Validation Set"]
        HT["Hidden Test Set"]
    end

    W --> TV
    W --> MW
    TV --> MW
    T --> MW
    C --> MW
    MW --> QW
    MW --> M
    QW --> M
    T --> M

    M --> F
    M --> OS
    M --> VS
    M --> HT

    MW --> MF
    QW --> MF
    MF --> S
    S --> D

    style OS fill:#fff3e0
    style VS fill:#fff3e0
    style HT fill:#ffebee
```

## Benchmark Split Boundaries

The diagram shows three distinct evaluation data flows:

* **Optimization Set (orange)**: Metrics feed directly into FitnessEngine and EvolutionEngine. This split is used for hyperparameter tuning and merge strategy selection.
* **Validation Set (orange)**: Metrics are logged to ExperimentTracker for monitoring and early stopping. Not used for fitness computation.
* **Hidden Test Set (red)**: Metrics are computed once per experiment, after evolution converges, and are never exposed to EvolutionEngine or FitnessEngine. This enforces CONST-05 and ADR-006/007.

## Flow Constraints

1. **Weights flow is unidirectional**: Weights move from ArtifactStore → ModelLoader → TensorEngine/MergeEngine → CandidateGenerator → QuantizationEngine → InferenceBackend. There is no reverse flow.
2. **Task Vectors are ephemeral**: Task Vectors exist only in memory during merge operations. They are not persisted to ArtifactStore.
3. **Metrics are append-only**: Once computed, metrics are written to ExperimentTracker and cannot be modified. Invalid metrics are marked INVALID, not overwritten.
4. **Signatures are computed post-merge**: The ed25519 signature covers the merged weights, config, and manifest. It is computed after CandidateGenerator produces the manifest and before DeploymentManager releases the artifact.
5. **Hidden Test Set metrics are isolated**: The red boundary indicates that Hidden Test Set metrics never enter the EvolutionEngine loop. Access is logged and audited.

## Next Steps

See [Experiment Lifecycle](/maps/experiment-lifecycle) for the state machine governing these data flows, or [System Dependency Map](/maps/system-dependency-map) for the executive pipeline view.