> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark Catalog by Category and Purpose

> Catalog of EMEP benchmark categories covering Persian, English, reasoning, code, safety, and enterprise tasks with split policies and primary metrics.

The BenchmarkCatalog organizes all evaluation tasks by category. Each entry specifies the purpose, data sources, split policy, and primary metric. This page provides the full category listing for EMEP experiments.

## Persian

**Purpose**: Evaluate Persian language understanding, generation, and cultural knowledge.

**Sources**: Placeholder (Persian Wikipedia, Persian News Corpus, Persian QA datasets).

**Split Policy**: Optimization Set 60%, Validation Set 20%, Hidden Test Set 20%.

**Primary Metric**: Accuracy on reading comprehension and cloze tasks.

See [Persian Evaluation](/evaluation/persian-evaluation) for script coverage and normalization concerns.

## English

**Purpose**: Baseline general English proficiency across reading comprehension, grammar, and commonsense reasoning.

**Sources**: Placeholder (standard academic benchmarks).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Exact match and accuracy.

## Instruction Following

**Purpose**: Measure adherence to explicit instructions, format compliance, and task completion.

**Sources**: Placeholder (instruction tuning evaluation sets).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Instruction-level accuracy and format correctness.

## Reasoning

**Purpose**: Assess logical, mathematical, and multi-step reasoning capabilities.

**Sources**: Placeholder (mathematical word problems, logical deduction sets).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Exact match on reasoning chains.

## Mathematics

**Purpose**: Numerical computation, symbolic math, and proof understanding.

**Sources**: Placeholder (grade-school math, competition math).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Exact match on final answer.

## Programming

**Purpose**: Code generation, code completion, and bug repair.

**Sources**: Placeholder (programming competition problems, open-source function datasets).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Pass-at-k (functional correctness).

## General Knowledge

**Purpose**: Factual recall across science, history, geography, and culture.

**Sources**: Placeholder (trivia and knowledge QA datasets).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Accuracy.

## Summarization

**Purpose**: Condense long documents while preserving key information.

**Sources**: Placeholder (news articles, academic abstracts).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: ROUGE-L.

## Translation

**Purpose**: Bidirectional translation quality between Persian and English.

**Sources**: Placeholder (parallel corpora, translation benchmarks).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: BLEU-4.

## Safety

**Purpose**: Detect harmful outputs, jailbreak susceptibility, and inappropriate refusals.

**Sources**: Placeholder (harmful prompts, adversarial test sets).

**Split Policy**: Optimization Set 40%, Validation Set 30%, Hidden Test Set 30%.

**Primary Metric**: Safety pass rate and refusal appropriateness.

See [Safety Evaluation](/evaluation/safety-evaluation) for category definitions.

## Hallucination

**Purpose**: Measure fabrication of facts, contradictions with source text, and ungrounded generation.

**Sources**: Placeholder (fact-checking datasets, grounded QA).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Factual accuracy against verified references.

## Retrieval

**Purpose**: Evaluate information retrieval augmented generation and citation accuracy.

**Sources**: Placeholder (retrieval-augmented QA benchmarks).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Citation accuracy and answer correctness.

## Enterprise Tasks

**Purpose**: Domain-specific tasks for enterprise deployment (for example legal, medical, financial text analysis).

**Sources**: Placeholder (enterprise domain datasets).

**Split Policy**: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%.

**Primary Metric**: Task-specific accuracy or F1 score.

## Catalog Usage

Experiments reference benchmarks by ID. The BenchmarkEngine resolves IDs to specification records at runtime. Benchmarks can be combined into profiles for multi-domain evaluation.

<Info>
  The Hidden Test Set for every category is never used during evolution. It is reserved for final candidate assessment after the EvolutionEngine has terminated.
</Info>