Reproducibility Requirements for EMEP Experiments

Specification for what is required to reproduce an EMEP experiment, covering code, model, dataset, benchmark, environment, seed, and configuration versions.

Reproducibility is a core requirement for EMEP. Every experiment must be fully reproducible from its recorded metadata and artifacts. This page specifies the complete set of requirements and how they are captured.

Reproducibility Checklist

An experiment is reproducible if and only if all of the following are available:

1. Code version: exact git commit of the EMEP codebase
2. Model version: version hashes of all parent models
3. Dataset version: exact dataset versions used for training and evaluation
4. Benchmark version: exact benchmark versions and their specifications
5. Environment version: dependency versions including CUDA, PyTorch, and framework libraries
6. Random seed: master seed and per-benchmark seeds
7. Configuration: full experiment configuration including genome parameters
8. Hardware: GPU type and count (for performance reproduction, not correctness)

Captured Fields

Requirement	Captured In	Verification
Code version	experiment.git_commit	Git tag or hash lookup
Model version	experiment.model_revisions	ArtifactStore hash verification
Dataset version	experiment.dataset_version	DatasetRegistry version lookup
Benchmark version	experiment.benchmark_version	BenchmarkCatalog version lookup
Environment version	experiment.framework_versions	Dependency manifest
Random seed	experiment.random_seed, experiment.seeds	Direct value
Configuration	experiment.merge_parameters	Genome record
Hardware	experiment.gpu_type, experiment.gpu_count	Resource log

Reproducibility Gaps

The following are known gaps that may affect exact reproduction:

• Non-deterministic GPU operations: some CUDA kernels have non-deterministic behavior. Setting torch.use_deterministic_algorithms(True) mitigates but does not eliminate this.
• Floating point order: merge operations may execute in different orders on different GPU counts, producing slightly different results.
• System-level noise: background processes on shared systems may affect timing but not correctness.

Reproducibility Score

EMEP computes a reproducibility score for each experiment:

score = sum(field_present * weight) / sum(weights)

Field	Weight
git_commit	1.0
model_revisions	1.0
dataset_version	1.0
benchmark_version	1.0
framework_versions	0.8
random_seed	1.0
merge_parameters	1.0
gpu_type	0.5

A score of 1.0 means all critical fields are present. Scores below 0.9 trigger a warning. Scores below 0.7 block experiment completion.

Verification

Reproducibility is verified by:

1. Checking that all required fields exist in the experiment record
2. Verifying that referenced artifacts exist in the ArtifactStore
3. Confirming that dataset and benchmark versions are available in their registries
4. Validating that the git commit exists in the repository

Integration

The reproducibility check is performed automatically at experiment completion. Results are stored in the experiment record and displayed in the experiment summary. See Reproducibility Protocol for the step-by-step reproduction procedure.

Reproducibility verification does not re-run the experiment. It only confirms that all required inputs and metadata are available and consistent. Actual reproduction requires following the protocol.