Tokenizer Compatibility Algorithm and Checks

Tokenizer compatibility specification for EMEP. Covers vocab overlap, special token matching, BPE and SentencePiece family checks, and the decision flow to COMPATIBLE, CONDITIONALLY_COMPATIBLE, or INCOMPATIBLE.

Tokenizer compatibility determines whether two models can share a vocabulary or require remapping. This page specifies the algorithm, thresholds, and exact state transitions.

Check Dimensions

The ModelCompatibilityAnalyzer evaluates tokenizers across these dimensions:

Dimension	Check
Vocab overlap	Intersection size / union size, as percentage
Special token match	BOS, EOS, PAD, UNK IDs and strings
Tokenizer family	BPE, SentencePiece, WordPiece, Tiktoken
Added tokens	Extra tokens beyond base vocab
Unicode normalization	NFC, NFD, NFKC, NFKD, or none
Pre-tokenizer	Whitespace, ByteLevel, or custom
Decoder	Mapping from tokens to text

Compatibility Decision Flow

Threshold Rationale

Threshold	Rationale
98% overlap (same family)	Engineering assumption: minor vocab drift is acceptable for merge
95% overlap (different family)	Engineering assumption: cross-family requires substantial overlap
Special token exact match	Required for generation correctness; mismatch triggers remapping need

CONDITIONALLY_COMPATIBLE Cases

Case	Required Action
Special token mismatch	Remap special token IDs or retrain embeddings
Added token mismatch	Intersect added tokens or retrain
Low overlap within family	Vocab remapping or fallback tokenizer

INCOMPATIBLE Cases

Case	Reason
Family and norm mismatch	Tokenization diverges at the byte level
Overlap below 95%	Too many tokens have no correspondence

Cross-Links

• Model Compatibility for the full decision tree.
• Architecture Compatibility for the architecture-only branch.
• Tensor Shape Validation for the tensor branch.