Benchmark Record Schema and Configuration

Specification for benchmark records in EMEP, covering schema fields, versioning, seeds, prompt templates, decoding parameters, and metric definitions.

Benchmarks in EMEP are versioned, reproducible evaluation configurations. The BenchmarkEngine executes benchmarks according to their specification records. This page defines the benchmark schema, metric semantics, and configuration invariants.

Benchmark Record Schema

Field	Type	Required	Description
benchmark_id	string	Yes	Canonical identifier, format: category_name_vN
version	string	Yes	Semantic version of the benchmark
category	string	Yes	Persian, English, Reasoning, Code, Safety, etc.
description	string	Yes	Human-readable purpose
dataset_refs	List[string]	Yes	DatasetRegistry IDs for source data
split_policy	string	Yes	Optimization / Validation / Hidden Test
prompt_template	string	Yes	Jinja2 template for formatting inputs
decoding_params	object	Yes	Temperature, top_p, max_tokens, etc.
metric_definitions	List[object]	Yes	Metrics to compute and their formulas
seeds	List[int]	Yes	Random seeds for reproducible sampling
sample_count	int	Yes	Number of samples to evaluate
baseline_model	string	No	Model ID for relative scoring
tags	List[string]	No	Arbitrary labels for filtering

Versioning

Benchmark versions follow semantic versioning:

• Major: changes to prompt templates, metric definitions, or dataset composition
• Minor: additional samples, seed additions, documentation updates
• Patch: bug fixes in scoring code

When a benchmark is updated, existing experiment records remain bound to the version they used. The BenchmarkCatalog maintains all historical versions.

Seeds and Reproducibility

Every benchmark specifies a list of integer seeds. The BenchmarkEngine uses these seeds for:

• Dataset shuffling and sampling
• Decoding randomness (when temperature > 0)
• Any stochastic elements in metric computation

Results are reported per-seed and aggregated across seeds. The number of seeds determines the statistical power of the evaluation.

Prompt Templates

Prompts are Jinja2 templates with access to the following variables:

• input: the raw input text or structured data
• instruction: task-specific instruction
• context: optional background context
• examples: few-shot examples if configured

Templates are validated at benchmark registration time. Invalid templates produce an error before any experiment can use the benchmark.

Decoding Parameters

Parameter	Type	Default	Description
temperature	float	0.0	Sampling temperature; 0.0 for greedy
top_p	float	1.0	Nucleus sampling threshold
max_tokens	int	256	Maximum generation length
stop_sequences	List[string]	[]	Strings that terminate generation
repetition_penalty	float	1.0	Penalty for repeated tokens

Decoding parameters are part of the benchmark specification. Changing them creates a new benchmark version.

Metric Definitions

Each metric definition includes:

Field	Description
metric_id	Canonical name
type	accuracy / exact_match / rouge / bleu / log_likelihood / custom
aggregation	mean / median / macro / weighted
higher_is_better	Boolean direction
calibration	Optional normalization or scaling

Supported Metrics

Accuracy

Proportion of correct predictions. For classification, exact label match. For multiple choice, selected option match.

Exact Match

String-level equality between generated and reference text, after normalization (whitespace stripping, lowercasing if specified).

ROUGE

ROUGE-L and ROUGE-1 scores for summarization and generation tasks. Computed using standard tokenization.

BLEU

BLEU-4 for translation and generation. Uses benchmark-specific tokenization.

Calibrated Log-Likelihood

Per-token log-likelihood normalized by sequence length. Used for perplexity and confidence estimation.

Split Policy

Every benchmark assigns its samples to one of three splits:

• Optimization Set: used by EvolutionEngine and FitnessEngine
• Validation Set: used for post-evolution validation reports
• Hidden Test Set: reserved for final, unbiased assessment

The split is fixed at dataset registration time. It cannot be changed without creating a new dataset version.

Benchmark specifications are immutable after registration. Any change, even to metadata, requires a new version. This guarantees that experiments referencing benchmark_v3 always execute the same evaluation protocol.