> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenizer Compatibility Algorithm and Checks

> Tokenizer compatibility specification for EMEP. Covers vocab overlap, special token matching, BPE and SentencePiece family checks, and the decision flow to COMPATIBLE, CONDITIONALLY_COMPATIBLE, or INCOMPATIBLE.

Tokenizer compatibility determines whether two models can share a vocabulary or require remapping. This page specifies the algorithm, thresholds, and exact state transitions.

## Check Dimensions

The ModelCompatibilityAnalyzer evaluates tokenizers across these dimensions:

| Dimension             | Check                                         |
| --------------------- | --------------------------------------------- |
| Vocab overlap         | Intersection size / union size, as percentage |
| Special token match   | BOS, EOS, PAD, UNK IDs and strings            |
| Tokenizer family      | BPE, SentencePiece, WordPiece, Tiktoken       |
| Added tokens          | Extra tokens beyond base vocab                |
| Unicode normalization | NFC, NFD, NFKC, NFKD, or none                 |
| Pre-tokenizer         | Whitespace, ByteLevel, or custom              |
| Decoder               | Mapping from tokens to text                   |

## Compatibility Decision Flow

```mermaid theme={null}
flowchart TD
    START([START]) --> INPUT[Input: Tokenizer A + Tokenizer B]
    INPUT --> FAMILY{Same tokenizer family?}
    FAMILY -->|no| NORM{Same Unicode normalization?}
    NORM -->|no| INCOMPATIBLE[INCOMPATIBLE: family and norm mismatch]
    NORM -->|yes| OVERLAP{Vocab overlap >= 95%?}
    OVERLAP -->|no| INCOMPATIBLE
    OVERLAP -->|yes| SPECIAL{Special tokens match?}
    SPECIAL -->|no| CONDITIONALLY_COMPATIBLE_1[CONDITIONALLY_COMPATIBLE: special token mismatch]
    SPECIAL -->|yes| ADDED{Added tokens identical?}
    ADDED -->|no| CONDITIONALLY_COMPATIBLE_2[CONDITIONALLY_COMPATIBLE: added token mismatch]
    ADDED -->|yes| COMPATIBLE[COMPATIBLE]
    FAMILY -->|yes| OVERLAP_2{Vocab overlap >= 98%?}
    OVERLAP_2 -->|no| CONDITIONALLY_COMPATIBLE_3[CONDITIONALLY_COMPATIBLE: low overlap within family]
    OVERLAP_2 -->|yes| SPECIAL_2{Special tokens match?}
    SPECIAL_2 -->|no| CONDITIONALLY_COMPATIBLE_4[CONDITIONALLY_COMPATIBLE: special token mismatch]
    SPECIAL_2 -->|yes| ADDED_2{Added tokens identical?}
    ADDED_2 -->|no| CONDITIONALLY_COMPATIBLE_5[CONDITIONALLY_COMPATIBLE: added token mismatch]
    ADDED_2 -->|yes| COMPATIBLE_2[COMPATIBLE]
    COMPATIBLE --> END([END])
    COMPATIBLE_2 --> END
    CONDITIONALLY_COMPATIBLE_1 --> END
    CONDITIONALLY_COMPATIBLE_2 --> END
    CONDITIONALLY_COMPATIBLE_3 --> END
    CONDITIONALLY_COMPATIBLE_4 --> END
    CONDITIONALLY_COMPATIBLE_5 --> END
    INCOMPATIBLE --> END
```

## Threshold Rationale

| Threshold                      | Rationale                                                             |
| ------------------------------ | --------------------------------------------------------------------- |
| 98% overlap (same family)      | Engineering assumption: minor vocab drift is acceptable for merge     |
| 95% overlap (different family) | Engineering assumption: cross-family requires substantial overlap     |
| Special token exact match      | Required for generation correctness; mismatch triggers remapping need |

## CONDITIONALLY\_COMPATIBLE Cases

| Case                      | Required Action                               |
| ------------------------- | --------------------------------------------- |
| Special token mismatch    | Remap special token IDs or retrain embeddings |
| Added token mismatch      | Intersect added tokens or retrain             |
| Low overlap within family | Vocab remapping or fallback tokenizer         |

## INCOMPATIBLE Cases

| Case                     | Reason                                  |
| ------------------------ | --------------------------------------- |
| Family and norm mismatch | Tokenization diverges at the byte level |
| Overlap below 95%        | Too many tokens have no correspondence  |

## Cross-Links

* [Model Compatibility](/compatibility/model-compatibility) for the full decision tree.
* [Architecture Compatibility](/compatibility/architecture-compatibility) for the architecture-only branch.
* [Tensor Shape Validation](/compatibility/tensor-shape-validation) for the tensor branch.