Skip to main content
The reproducibility protocol defines the exact steps to re-run an EMEP experiment and verify that results match the original. This page specifies the procedure, failure conditions, and success criteria.

​
Reproduction Flowchart

​
Step Descriptions

Load Experiment Record Retrieve the full experiment record from the ExperimentTracker by experiment_id. Verify that the record is complete and not corrupted. Verify Reproducibility Score Check the pre-computed reproducibility score. If below 0.7, reproduction is blocked. If between 0.7 and 0.9, reproduction proceeds with warnings. Prepare Environment Provision hardware matching the original experiment: same GPU type and count if possible. GPU count differences are acceptable for correctness reproduction but will affect timing. Checkout Code Version Clone the EMEP repository and checkout the exact git commit from the experiment record. Verify the commit hash matches. Install Dependencies Install the exact dependency versions from framework_versions. Use the same CUDA and PyTorch versions. Container images are recommended for isolation. Download Parent Models Download parent model weights from the ArtifactStore using the hashes in model_revisions. Verify SHA-256 checksums after download. Download Datasets Download dataset versions from the DatasetRegistry. Verify split manifests and checksums. Load Configuration Load the experiment configuration including merge parameters, benchmark selections, and population settings from the experiment record. Set Random Seeds Set the master random seed and all per-benchmark seeds. Configure PyTorch deterministic mode if available. Execute Experiment Run the experiment using the EMEP CLI or API. Do not modify any parameters. Log all outputs. Compare Results Compare reproduction results to original:
  • Aggregate scores within 0.1%: match
  • Per-benchmark scores within 0.5%: match
  • Latency and throughput: informational only, not required to match
  • Memory usage within 10%: match

​
Failure Conditions

Reproduction fails if any of the following occur:
  1. Reproducibility score below 0.7
  2. Required artifacts missing from ArtifactStore
  3. Required dataset versions missing from DatasetRegistry
  4. Git commit not found in repository
  5. Dependency versions no longer available
  6. Hardware requirements cannot be met and experiment requires specific hardware features
  7. Results differ beyond tolerance and difference is not explainable by known non-determinism

​
Tolerable Differences

The following differences are expected and do not constitute failure:
  • Timing variations due to system load
  • Minor floating point differences from GPU operation ordering
  • Memory usage differences within 10% due to allocator behavior

​
Success Criteria

Reproduction is successful when:
  • All steps complete without error
  • Aggregate and per-benchmark scores match within tolerance
  • Differences, if any, are explainable by known non-determinism
  • Full reproduction log is stored in the ExperimentTracker

​
Integration

The reproduction protocol is implemented as a CLI command:
Results are stored as a new experiment record with reproduced_from referencing the original.
Reproduction is not automatic. It requires manual initiation and resource allocation. The protocol provides the exact procedure; execution is the responsibility of the user or automation system.