> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Reproducibility Requirements for EMEP Experiments

> Specification for what is required to reproduce an EMEP experiment, covering code, model, dataset, benchmark, environment, seed, and configuration versions.

Reproducibility is a core requirement for EMEP. Every experiment must be fully reproducible from its recorded metadata and artifacts. This page specifies the complete set of requirements and how they are captured.

## Reproducibility Checklist

An experiment is reproducible if and only if all of the following are available:

1. **Code version**: exact git commit of the EMEP codebase
2. **Model version**: version hashes of all parent models
3. **Dataset version**: exact dataset versions used for training and evaluation
4. **Benchmark version**: exact benchmark versions and their specifications
5. **Environment version**: dependency versions including CUDA, PyTorch, and framework libraries
6. **Random seed**: master seed and per-benchmark seeds
7. **Configuration**: full experiment configuration including genome parameters
8. **Hardware**: GPU type and count (for performance reproduction, not correctness)

## Captured Fields

| Requirement         | Captured In                                 | Verification                    |
| ------------------- | ------------------------------------------- | ------------------------------- |
| Code version        | experiment.git\_commit                      | Git tag or hash lookup          |
| Model version       | experiment.model\_revisions                 | ArtifactStore hash verification |
| Dataset version     | experiment.dataset\_version                 | DatasetRegistry version lookup  |
| Benchmark version   | experiment.benchmark\_version               | BenchmarkCatalog version lookup |
| Environment version | experiment.framework\_versions              | Dependency manifest             |
| Random seed         | experiment.random\_seed, experiment.seeds   | Direct value                    |
| Configuration       | experiment.merge\_parameters                | Genome record                   |
| Hardware            | experiment.gpu\_type, experiment.gpu\_count | Resource log                    |

## Reproducibility Gaps

The following are known gaps that may affect exact reproduction:

* **Non-deterministic GPU operations**: some CUDA kernels have non-deterministic behavior. Setting `torch.use_deterministic_algorithms(True)` mitigates but does not eliminate this.
* **Floating point order**: merge operations may execute in different orders on different GPU counts, producing slightly different results.
* **System-level noise**: background processes on shared systems may affect timing but not correctness.

## Reproducibility Score

EMEP computes a reproducibility score for each experiment:

```text theme={null}
score = sum(field_present * weight) / sum(weights)
```

| Field               | Weight |
| ------------------- | ------ |
| git\_commit         | 1.0    |
| model\_revisions    | 1.0    |
| dataset\_version    | 1.0    |
| benchmark\_version  | 1.0    |
| framework\_versions | 0.8    |
| random\_seed        | 1.0    |
| merge\_parameters   | 1.0    |
| gpu\_type           | 0.5    |

A score of 1.0 means all critical fields are present. Scores below 0.9 trigger a warning. Scores below 0.7 block experiment completion.

## Verification

Reproducibility is verified by:

1. Checking that all required fields exist in the experiment record
2. Verifying that referenced artifacts exist in the ArtifactStore
3. Confirming that dataset and benchmark versions are available in their registries
4. Validating that the git commit exists in the repository

## Integration

The reproducibility check is performed automatically at experiment completion. Results are stored in the experiment record and displayed in the experiment summary. See [Reproducibility Protocol](/tracking/reproducibility-protocol) for the step-by-step reproduction procedure.

<Info>
  Reproducibility verification does not re-run the experiment. It only confirms that all required inputs and metadata are available and consistent. Actual reproduction requires following the protocol.
</Info>