Benchmark Integrity and Data Contamination Prevention

Specification for data contamination detection, leakage classification, and Hidden Test Set access control enforcement in EMEP benchmarks.

Benchmark integrity ensures that evaluation results reflect true model capability, not memorization or data leakage. EMEP enforces contamination detection at dataset registration and access control at the data layer. This page specifies the detection methods, leakage classes, and enforcement architecture.

Contamination Detection Pipeline

n-gram Overlap Analysis

The DatasetRegistry computes n-gram overlap between new datasets and existing benchmarks. Overlap is measured at multiple granularities:

• Character 5-grams: catches near-exact matches with minor edits
• Word 3-grams: catches paraphrased or lightly edited content
• Sentence-level exact match: catches direct copies

Overlap percentages are computed relative to the smaller dataset. High overlap triggers manual review.

Hash Set Deduplication

Every benchmark sample is hashed using SHA-256 of its normalized text. Hashes are stored in a global index. New datasets are checked against this index. Duplicate samples are flagged and either removed or assigned to the same split as the original.

Leakage Classes

Class	Description	Action
None	No significant overlap detected	Accept dataset
Minor	< 1% overlap, likely incidental	Accept with annotation
Moderate	1-5% overlap, possible contamination	Require manual review
Severe	> 5% overlap, likely direct inclusion	Reject dataset
Known Leak	Dataset explicitly contains benchmark data	Reject or quarantine

Hidden Test Set Access Control

The Hidden Test Set is protected by enforcement at the data layer. This is not a policy or convention: it is a technical control.

Enforcement Rules

• The EvolutionEngine cannot request Hidden Test Set samples under any configuration
• The EvaluationEngine can access Hidden Test Set only when the experiment is in the final assessment phase
• Admin override is possible but requires audit logging
• All access attempts are logged to the ExperimentTracker

Split Isolation Guarantees

The DatasetRegistry maintains split assignments at the sample level. Samples are tagged with their split at ingestion time. The BenchmarkEngine loads only samples matching the requested split. There is no runtime split reassignment.

Contamination Monitoring

After model release, EMEP monitors for public reports of benchmark contamination. If a model is found to have been trained on Hidden Test Set data, the evaluation is invalidated and the model is flagged in the ModelRegistry.

Integration

Component	Responsibility
DatasetRegistry	Registration, overlap detection, hash indexing
BenchmarkEngine	Split-filtered loading, access validation
EvaluationEngine	Phase-aware split requests
ExperimentTracker	Audit logging of all access attempts

Hidden Test Set access control is a critical security control. Bypassing it invalidates all evaluation results from the affected experiment. Any detected bypass triggers an automatic experiment failure and incident report.

Benchmark integrity checks run at dataset registration time, not at evaluation time. This prevents contaminated datasets from entering the system before any experiment can use them.