DatasetRegistry for Versioned Data Management

Specification for the DatasetRegistry in EMEP, covering versioned datasets, split manifests, checksums, and per-dataset license tracking.

The DatasetRegistry manages all datasets used for training, evaluation, and benchmarking in EMEP. It enforces versioning, split integrity, and provenance tracking. This page specifies the registry structure, dataset records, and operational invariants.

Dataset Record Schema

Field	Type	Required	Description
dataset_id	string	Yes	Canonical identifier
version	string	Yes	Semantic version
name	string	Yes	Human-readable name
description	string	Yes	Purpose and contents
source_url	string	No	Origin URL or path
license	string	Yes	License identifier
license_url	string	No	Full license text URL
format	string	Yes	jsonl, parquet, csv, etc.
sample_count	int	Yes	Total samples
split_manifest	object	Yes	Split names and sample counts
checksums	Dict[string, string]	Yes	File-level SHA-256 hashes
tags	List[string]	No	Category labels
created_at	ISO8601	Yes	Registration timestamp
registered_by	string	Yes	User or system component

Split Manifest

Every dataset specifies its splits at registration time. The split manifest is immutable.

{
  "optimization_set": {
    "count": 5000,
    "file": "optimization.jsonl",
    "checksum": "sha256:abc..."
  },
  "validation_set": {
    "count": 1000,
    "file": "validation.jsonl",
    "checksum": "sha256:def..."
  },
  "hidden_test_set": {
    "count": 1000,
    "file": "hidden_test.jsonl",
    "checksum": "sha256:ghi..."
  }
}

Versioning

Dataset versions follow semantic versioning:

• Major: new samples, removed samples, changed splits
• Minor: metadata updates, documentation improvements
• Patch: bug fixes in preprocessing

When a dataset is updated, existing experiments remain bound to the version they used. The DatasetRegistry maintains all historical versions.

Checksums and Integrity

Every file in a dataset is hashed with SHA-256 at registration. The BenchmarkEngine verifies checksums before loading. Mismatches trigger an error and halt evaluation.

License Tracking

Each dataset record includes a license field. The DatasetRegistry validates that the license is one of the known identifiers:

• apache-2.0
• mit
• cc-by-4.0
• cc-by-nc-4.0
• cc0-1.0
• proprietary
• unknown

Datasets with unknown or proprietary licenses require manual approval before use in experiments. The license field is used by the LicenseCompliance component for audit reports.

API Operations

DatasetRegistry.register(dataset_metadata, files) -> dataset_id
DatasetRegistry.get(dataset_id, version) -> DatasetRecord
DatasetRegistry.list(filters) -> List[DatasetRecord]
DatasetRegistry.get_splits(dataset_id, version) -> SplitManifest
DatasetRegistry.verify_checksums(dataset_id, version) -> bool
DatasetRegistry.get_samples(dataset_id, version, split, indices) -> List[Sample]

Split Access Control

The DatasetRegistry enforces split access rules:

• Any component can request Optimization Set samples
• Validation Set samples require experiment phase >= EVALUATING
• Hidden Test Set samples require experiment phase == FINAL_ASSESSMENT

These rules are enforced at the data layer, not by convention.

Integration

Component	Usage
BenchmarkEngine	Loads samples by split for evaluation
EvaluationEngine	Requests split-filtered data
ExperimentTracker	Logs which dataset versions were used
LicenseCompliance	Audits license compatibility

Dataset records are immutable after registration. Any change, even to metadata, creates a new version. This ensures that experiments always reference the exact data they were run against.