Persian Language Evaluation Concerns and Benchmarks

Specification for Persian-specific evaluation in EMEP, covering script coverage, diacritics, ZWNJ, transliteration, register variation, and cultural knowledge.

Persian evaluation in EMEP requires handling script complexity, normalization ambiguity, and cultural specificity that generic multilingual benchmarks do not address. This page specifies the concerns, preprocessing rules, and placeholder benchmark IDs for Persian language assessment.

Script Coverage

Persian text uses Arabic script with additional letters. Models must correctly handle:

• Standard Arabic letters present in Persian (for example alef, beh, tehh)
• Persian-specific letters: peh (پ), cheh (چ), zheh (ژ), gaf (گ)
• Contextual forms: isolated, initial, medial, final

Evaluation datasets include samples that test discrimination between Arabic and Persian homoglyphs. A model that conflates Arabic sad (ص) with Persian-specific variants receives lower accuracy.

Diacritics

Persian text is typically written without short vowel diacritics (harakat). However, some datasets include fully vocalized text for reading comprehension tasks. The BenchmarkEngine normalizes diacritics according to the benchmark specification:

• Strip mode: remove all harakat before evaluation
• Preserve mode: keep diacritics and require exact match
• Optional mode: accept answers with or without diacritics

Default is strip mode for consistency with real-world Persian text.

Zero Width Non-Joiner (ZWNJ)

ZWNJ (U+200C) is essential for correct Persian orthography. It appears between morpheme boundaries (for example می‌نویسم). The BenchmarkEngine treats ZWNJ as significant:

• Missing ZWNJ in generated text counts as an error in exact match mode
• ROUGE and BLEU computations preserve ZWNJ
• Tokenizers must not strip ZWNJ during pre-tokenization

Transliteration

Some benchmarks include transliterated Persian (for example Latin script representations of Persian words). The BenchmarkEngine does not normalize transliteration. Models are evaluated on their ability to handle mixed-script inputs if present in the dataset.

Register Variation

Persian evaluation covers formal and conversational registers:

• Formal: news, academic text, legal documents
• Conversational: social media, dialogue, informal messaging

Benchmarks are tagged by register. Models are expected to perform competently across both. Register-specific scores are reported separately.

Cultural Knowledge

Persian cultural knowledge includes history, literature, geography, and customs. Benchmarks include questions requiring culturally specific answers (for example identifying Hafez, recognizing Nowruz). Generic multilingual models often underperform on these items.

Placeholder Benchmark IDs

Benchmark ID	Category	Description
persian_reading_comp_v1	Persian	Reading comprehension on Persian Wikipedia articles
persian_cloze_v1	Persian	Cloze deletion on Persian news text
persian_grammar_v1	Persian	Morphological and syntactic correctness
persian_culture_v1	Persian	Cultural and historical knowledge
persian_dialogue_v1	Persian	Conversational response quality
persian_translation_v1	Translation	Persian to English and English to Persian

Preprocessing Pipeline

Integration

Persian benchmarks are part of the standard BenchmarkCatalog. They follow the same split policies and metric definitions as other categories. The BenchmarkEngine applies Persian-specific preprocessing only when the benchmark category is Persian.

Tokenizer compatibility for Persian is validated during model import. See Tokenizer Compatibility for the validation rules.