> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# DatasetRegistry for Versioned Data Management

> Specification for the DatasetRegistry in EMEP, covering versioned datasets, split manifests, checksums, and per-dataset license tracking.

The DatasetRegistry manages all datasets used for training, evaluation, and benchmarking in EMEP. It enforces versioning, split integrity, and provenance tracking. This page specifies the registry structure, dataset records, and operational invariants.

## Dataset Record Schema

| Field           | Type                  | Required | Description                   |
| --------------- | --------------------- | -------- | ----------------------------- |
| dataset\_id     | string                | Yes      | Canonical identifier          |
| version         | string                | Yes      | Semantic version              |
| name            | string                | Yes      | Human-readable name           |
| description     | string                | Yes      | Purpose and contents          |
| source\_url     | string                | No       | Origin URL or path            |
| license         | string                | Yes      | License identifier            |
| license\_url    | string                | No       | Full license text URL         |
| format          | string                | Yes      | jsonl, parquet, csv, etc.     |
| sample\_count   | int                   | Yes      | Total samples                 |
| split\_manifest | object                | Yes      | Split names and sample counts |
| checksums       | Dict\[string, string] | Yes      | File-level SHA-256 hashes     |
| tags            | List\[string]         | No       | Category labels               |
| created\_at     | ISO8601               | Yes      | Registration timestamp        |
| registered\_by  | string                | Yes      | User or system component      |

## Split Manifest

Every dataset specifies its splits at registration time. The split manifest is immutable.

```json theme={null}
{
  "optimization_set": {
    "count": 5000,
    "file": "optimization.jsonl",
    "checksum": "sha256:abc..."
  },
  "validation_set": {
    "count": 1000,
    "file": "validation.jsonl",
    "checksum": "sha256:def..."
  },
  "hidden_test_set": {
    "count": 1000,
    "file": "hidden_test.jsonl",
    "checksum": "sha256:ghi..."
  }
}
```

## Versioning

Dataset versions follow semantic versioning:

* **Major**: new samples, removed samples, changed splits
* **Minor**: metadata updates, documentation improvements
* **Patch**: bug fixes in preprocessing

When a dataset is updated, existing experiments remain bound to the version they used. The DatasetRegistry maintains all historical versions.

## Checksums and Integrity

Every file in a dataset is hashed with SHA-256 at registration. The BenchmarkEngine verifies checksums before loading. Mismatches trigger an error and halt evaluation.

## License Tracking

Each dataset record includes a license field. The DatasetRegistry validates that the license is one of the known identifiers:

* apache-2.0
* mit
* cc-by-4.0
* cc-by-nc-4.0
* cc0-1.0
* proprietary
* unknown

Datasets with `unknown` or `proprietary` licenses require manual approval before use in experiments. The license field is used by the LicenseCompliance component for audit reports.

## API Operations

```text theme={null}
DatasetRegistry.register(dataset_metadata, files) -> dataset_id
DatasetRegistry.get(dataset_id, version) -> DatasetRecord
DatasetRegistry.list(filters) -> List[DatasetRecord]
DatasetRegistry.get_splits(dataset_id, version) -> SplitManifest
DatasetRegistry.verify_checksums(dataset_id, version) -> bool
DatasetRegistry.get_samples(dataset_id, version, split, indices) -> List[Sample]
```

## Split Access Control

The DatasetRegistry enforces split access rules:

* Any component can request Optimization Set samples
* Validation Set samples require experiment phase >= EVALUATING
* Hidden Test Set samples require experiment phase == FINAL\_ASSESSMENT

These rules are enforced at the data layer, not by convention.

## Integration

| Component         | Usage                                 |
| ----------------- | ------------------------------------- |
| BenchmarkEngine   | Loads samples by split for evaluation |
| EvaluationEngine  | Requests split-filtered data          |
| ExperimentTracker | Logs which dataset versions were used |
| LicenseCompliance | Audits license compatibility          |

<Info>
  Dataset records are immutable after registration. Any change, even to metadata, creates a new version. This ensures that experiments always reference the exact data they were run against.
</Info>