Persian
Purpose: Evaluate Persian language understanding, generation, and cultural knowledge. Sources: Placeholder (Persian Wikipedia, Persian News Corpus, Persian QA datasets). Split Policy: Optimization Set 60%, Validation Set 20%, Hidden Test Set 20%. Primary Metric: Accuracy on reading comprehension and cloze tasks. See Persian Evaluation for script coverage and normalization concerns.English
Purpose: Baseline general English proficiency across reading comprehension, grammar, and commonsense reasoning. Sources: Placeholder (standard academic benchmarks). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Exact match and accuracy.Instruction Following
Purpose: Measure adherence to explicit instructions, format compliance, and task completion. Sources: Placeholder (instruction tuning evaluation sets). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Instruction-level accuracy and format correctness.Reasoning
Purpose: Assess logical, mathematical, and multi-step reasoning capabilities. Sources: Placeholder (mathematical word problems, logical deduction sets). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Exact match on reasoning chains.Mathematics
Purpose: Numerical computation, symbolic math, and proof understanding. Sources: Placeholder (grade-school math, competition math). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Exact match on final answer.Programming
Purpose: Code generation, code completion, and bug repair. Sources: Placeholder (programming competition problems, open-source function datasets). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Pass-at-k (functional correctness).General Knowledge
Purpose: Factual recall across science, history, geography, and culture. Sources: Placeholder (trivia and knowledge QA datasets). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Accuracy.Summarization
Purpose: Condense long documents while preserving key information. Sources: Placeholder (news articles, academic abstracts). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: ROUGE-L.Translation
Purpose: Bidirectional translation quality between Persian and English. Sources: Placeholder (parallel corpora, translation benchmarks). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: BLEU-4.Safety
Purpose: Detect harmful outputs, jailbreak susceptibility, and inappropriate refusals. Sources: Placeholder (harmful prompts, adversarial test sets). Split Policy: Optimization Set 40%, Validation Set 30%, Hidden Test Set 30%. Primary Metric: Safety pass rate and refusal appropriateness. See Safety Evaluation for category definitions.Hallucination
Purpose: Measure fabrication of facts, contradictions with source text, and ungrounded generation. Sources: Placeholder (fact-checking datasets, grounded QA). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Factual accuracy against verified references.Retrieval
Purpose: Evaluate information retrieval augmented generation and citation accuracy. Sources: Placeholder (retrieval-augmented QA benchmarks). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Citation accuracy and answer correctness.Enterprise Tasks
Purpose: Domain-specific tasks for enterprise deployment (for example legal, medical, financial text analysis). Sources: Placeholder (enterprise domain datasets). Split Policy: Optimization Set 50%, Validation Set 25%, Hidden Test Set 25%. Primary Metric: Task-specific accuracy or F1 score.Catalog Usage
Experiments reference benchmarks by ID. The BenchmarkEngine resolves IDs to specification records at runtime. Benchmarks can be combined into profiles for multi-domain evaluation.The Hidden Test Set for every category is never used during evolution. It is reserved for final candidate assessment after the EvolutionEngine has terminated.