> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Persian Language Evaluation Concerns and Benchmarks

> Specification for Persian-specific evaluation in EMEP, covering script coverage, diacritics, ZWNJ, transliteration, register variation, and cultural knowledge.

Persian evaluation in EMEP requires handling script complexity, normalization ambiguity, and cultural specificity that generic multilingual benchmarks do not address. This page specifies the concerns, preprocessing rules, and placeholder benchmark IDs for Persian language assessment.

## Script Coverage

Persian text uses Arabic script with additional letters. Models must correctly handle:

* Standard Arabic letters present in Persian (for example alef, beh, tehh)
* Persian-specific letters: peh (پ), cheh (چ), zheh (ژ), gaf (گ)
* Contextual forms: isolated, initial, medial, final

Evaluation datasets include samples that test discrimination between Arabic and Persian homoglyphs. A model that conflates Arabic sad (ص) with Persian-specific variants receives lower accuracy.

## Diacritics

Persian text is typically written without short vowel diacritics (harakat). However, some datasets include fully vocalized text for reading comprehension tasks. The BenchmarkEngine normalizes diacritics according to the benchmark specification:

* **Strip mode**: remove all harakat before evaluation
* **Preserve mode**: keep diacritics and require exact match
* **Optional mode**: accept answers with or without diacritics

Default is strip mode for consistency with real-world Persian text.

## Zero Width Non-Joiner (ZWNJ)

ZWNJ (U+200C) is essential for correct Persian orthography. It appears between morpheme boundaries (for example می‌نویسم). The BenchmarkEngine treats ZWNJ as significant:

* Missing ZWNJ in generated text counts as an error in exact match mode
* ROUGE and BLEU computations preserve ZWNJ
* Tokenizers must not strip ZWNJ during pre-tokenization

## Transliteration

Some benchmarks include transliterated Persian (for example Latin script representations of Persian words). The BenchmarkEngine does not normalize transliteration. Models are evaluated on their ability to handle mixed-script inputs if present in the dataset.

## Register Variation

Persian evaluation covers formal and conversational registers:

* **Formal**: news, academic text, legal documents
* **Conversational**: social media, dialogue, informal messaging

Benchmarks are tagged by register. Models are expected to perform competently across both. Register-specific scores are reported separately.

## Cultural Knowledge

Persian cultural knowledge includes history, literature, geography, and customs. Benchmarks include questions requiring culturally specific answers (for example identifying Hafez, recognizing Nowruz). Generic multilingual models often underperform on these items.

## Placeholder Benchmark IDs

| Benchmark ID               | Category    | Description                                         |
| -------------------------- | ----------- | --------------------------------------------------- |
| persian\_reading\_comp\_v1 | Persian     | Reading comprehension on Persian Wikipedia articles |
| persian\_cloze\_v1         | Persian     | Cloze deletion on Persian news text                 |
| persian\_grammar\_v1       | Persian     | Morphological and syntactic correctness             |
| persian\_culture\_v1       | Persian     | Cultural and historical knowledge                   |
| persian\_dialogue\_v1      | Persian     | Conversational response quality                     |
| persian\_translation\_v1   | Translation | Persian to English and English to Persian           |

## Preprocessing Pipeline

```mermaid theme={null}
flowchart LR
    INPUT[Raw Persian Text] --> NORM1[Normalize Unicode]
    NORM1 --> NORM2[Handle ZWNJ]
    NORM2 --> NORM3[Strip or Preserve Diacritics]
    NORM3 --> TOKENIZE[Tokenizer]
    TOKENIZE --> MODEL[Model Inference]
    MODEL --> POST[Post-process Output]
    POST --> SCORE[Scoring]
```

## Integration

Persian benchmarks are part of the standard BenchmarkCatalog. They follow the same split policies and metric definitions as other categories. The BenchmarkEngine applies Persian-specific preprocessing only when the benchmark category is Persian.

<Info>
  Tokenizer compatibility for Persian is validated during model import. See [Tokenizer Compatibility](/compatibility/tokenizer-compatibility) for the validation rules.
</Info>