> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark Integrity and Data Contamination Prevention

> Specification for data contamination detection, leakage classification, and Hidden Test Set access control enforcement in EMEP benchmarks.

Benchmark integrity ensures that evaluation results reflect true model capability, not memorization or data leakage. EMEP enforces contamination detection at dataset registration and access control at the data layer. This page specifies the detection methods, leakage classes, and enforcement architecture.

## Contamination Detection Pipeline

```mermaid theme={null}
flowchart TD
    START([START]) --> REG[Register Dataset]
    REG --> NGRAM[n-gram Overlap Analysis]
    NGRAM --> HASH[Hash Set Deduplication]
    HASH --> CLASS[Classify Leakage]
    CLASS --> SCORE[Contamination Score]
    SCORE --> THRESH{Score > Threshold?}
    THRESH -->|Yes| REJECT[Reject Dataset]
    THRESH -->|No| ACCEPT[Accept Dataset]
    ACCEPT --> SPLIT[Assign Splits]
    SPLIT --> ACCESS[Enforce Access Control]
    ACCESS --> END([END])
    REJECT --> END
```

## n-gram Overlap Analysis

The DatasetRegistry computes n-gram overlap between new datasets and existing benchmarks. Overlap is measured at multiple granularities:

* **Character 5-grams**: catches near-exact matches with minor edits
* **Word 3-grams**: catches paraphrased or lightly edited content
* **Sentence-level exact match**: catches direct copies

Overlap percentages are computed relative to the smaller dataset. High overlap triggers manual review.

## Hash Set Deduplication

Every benchmark sample is hashed using SHA-256 of its normalized text. Hashes are stored in a global index. New datasets are checked against this index. Duplicate samples are flagged and either removed or assigned to the same split as the original.

## Leakage Classes

| Class      | Description                                | Action                 |
| ---------- | ------------------------------------------ | ---------------------- |
| None       | No significant overlap detected            | Accept dataset         |
| Minor      | \< 1% overlap, likely incidental           | Accept with annotation |
| Moderate   | 1-5% overlap, possible contamination       | Require manual review  |
| Severe     | > 5% overlap, likely direct inclusion      | Reject dataset         |
| Known Leak | Dataset explicitly contains benchmark data | Reject or quarantine   |

## Hidden Test Set Access Control

The Hidden Test Set is protected by enforcement at the data layer. This is not a policy or convention: it is a technical control.

```mermaid theme={null}
flowchart LR
    subgraph Access Control
        REQ[Access Request]
        ROLE{Role Check}
        PHASE{Experiment Phase}
        ALLOW[Allow Access]
        DENY[Deny Access]
    end
    REQ --> ROLE
    ROLE -->|EvolutionEngine| DENY
    ROLE -->|EvaluationEngine| PHASE
    PHASE -->|Evolution Phase| DENY
    PHASE -->|Final Assessment| ALLOW
    ROLE -->|Admin Override| AUDIT[Audit Log]
    AUDIT --> ALLOW
```

**Enforcement Rules**

* The EvolutionEngine cannot request Hidden Test Set samples under any configuration
* The EvaluationEngine can access Hidden Test Set only when the experiment is in the final assessment phase
* Admin override is possible but requires audit logging
* All access attempts are logged to the ExperimentTracker

## Split Isolation Guarantees

The DatasetRegistry maintains split assignments at the sample level. Samples are tagged with their split at ingestion time. The BenchmarkEngine loads only samples matching the requested split. There is no runtime split reassignment.

## Contamination Monitoring

After model release, EMEP monitors for public reports of benchmark contamination. If a model is found to have been trained on Hidden Test Set data, the evaluation is invalidated and the model is flagged in the ModelRegistry.

## Integration

| Component         | Responsibility                                 |
| ----------------- | ---------------------------------------------- |
| DatasetRegistry   | Registration, overlap detection, hash indexing |
| BenchmarkEngine   | Split-filtered loading, access validation      |
| EvaluationEngine  | Phase-aware split requests                     |
| ExperimentTracker | Audit logging of all access attempts           |

<Warning>
  Hidden Test Set access control is a critical security control. Bypassing it invalidates all evaluation results from the affected experiment. Any detected bypass triggers an automatic experiment failure and incident report.
</Warning>

<Info>
  Benchmark integrity checks run at dataset registration time, not at evaluation time. This prevents contaminated datasets from entering the system before any experiment can use them.
</Info>