> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark Record Schema and Configuration

> Specification for benchmark records in EMEP, covering schema fields, versioning, seeds, prompt templates, decoding parameters, and metric definitions.

Benchmarks in EMEP are versioned, reproducible evaluation configurations. The BenchmarkEngine executes benchmarks according to their specification records. This page defines the benchmark schema, metric semantics, and configuration invariants.

## Benchmark Record Schema

| Field               | Type          | Required | Description                                      |
| ------------------- | ------------- | -------- | ------------------------------------------------ |
| benchmark\_id       | string        | Yes      | Canonical identifier, format: category\_name\_vN |
| version             | string        | Yes      | Semantic version of the benchmark                |
| category            | string        | Yes      | Persian, English, Reasoning, Code, Safety, etc.  |
| description         | string        | Yes      | Human-readable purpose                           |
| dataset\_refs       | List\[string] | Yes      | DatasetRegistry IDs for source data              |
| split\_policy       | string        | Yes      | Optimization / Validation / Hidden Test          |
| prompt\_template    | string        | Yes      | Jinja2 template for formatting inputs            |
| decoding\_params    | object        | Yes      | Temperature, top\_p, max\_tokens, etc.           |
| metric\_definitions | List\[object] | Yes      | Metrics to compute and their formulas            |
| seeds               | List\[int]    | Yes      | Random seeds for reproducible sampling           |
| sample\_count       | int           | Yes      | Number of samples to evaluate                    |
| baseline\_model     | string        | No       | Model ID for relative scoring                    |
| tags                | List\[string] | No       | Arbitrary labels for filtering                   |

## Versioning

Benchmark versions follow semantic versioning:

* **Major**: changes to prompt templates, metric definitions, or dataset composition
* **Minor**: additional samples, seed additions, documentation updates
* **Patch**: bug fixes in scoring code

When a benchmark is updated, existing experiment records remain bound to the version they used. The BenchmarkCatalog maintains all historical versions.

## Seeds and Reproducibility

Every benchmark specifies a list of integer seeds. The BenchmarkEngine uses these seeds for:

* Dataset shuffling and sampling
* Decoding randomness (when temperature > 0)
* Any stochastic elements in metric computation

Results are reported per-seed and aggregated across seeds. The number of seeds determines the statistical power of the evaluation.

## Prompt Templates

Prompts are Jinja2 templates with access to the following variables:

* `input`: the raw input text or structured data
* `instruction`: task-specific instruction
* `context`: optional background context
* `examples`: few-shot examples if configured

Templates are validated at benchmark registration time. Invalid templates produce an error before any experiment can use the benchmark.

## Decoding Parameters

| Parameter           | Type          | Default | Description                          |
| ------------------- | ------------- | ------- | ------------------------------------ |
| temperature         | float         | 0.0     | Sampling temperature; 0.0 for greedy |
| top\_p              | float         | 1.0     | Nucleus sampling threshold           |
| max\_tokens         | int           | 256     | Maximum generation length            |
| stop\_sequences     | List\[string] | \[]     | Strings that terminate generation    |
| repetition\_penalty | float         | 1.0     | Penalty for repeated tokens          |

Decoding parameters are part of the benchmark specification. Changing them creates a new benchmark version.

## Metric Definitions

Each metric definition includes:

| Field              | Description                                                       |
| ------------------ | ----------------------------------------------------------------- |
| metric\_id         | Canonical name                                                    |
| type               | accuracy / exact\_match / rouge / bleu / log\_likelihood / custom |
| aggregation        | mean / median / macro / weighted                                  |
| higher\_is\_better | Boolean direction                                                 |
| calibration        | Optional normalization or scaling                                 |

### Supported Metrics

**Accuracy**

Proportion of correct predictions. For classification, exact label match. For multiple choice, selected option match.

**Exact Match**

String-level equality between generated and reference text, after normalization (whitespace stripping, lowercasing if specified).

**ROUGE**

ROUGE-L and ROUGE-1 scores for summarization and generation tasks. Computed using standard tokenization.

**BLEU**

BLEU-4 for translation and generation. Uses benchmark-specific tokenization.

**Calibrated Log-Likelihood**

Per-token log-likelihood normalized by sequence length. Used for perplexity and confidence estimation.

## Split Policy

Every benchmark assigns its samples to one of three splits:

* **Optimization Set**: used by EvolutionEngine and FitnessEngine
* **Validation Set**: used for post-evolution validation reports
* **Hidden Test Set**: reserved for final, unbiased assessment

The split is fixed at dataset registration time. It cannot be changed without creating a new dataset version.

<Info>
  Benchmark specifications are immutable after registration. Any change, even to metadata, requires a new version. This guarantees that experiments referencing benchmark\_v3 always execute the same evaluation protocol.
</Info>