Which languages are in-vocabulary?
Same character. Three tokenizers. Three completely different fates.
ϩ
Native token
Gemma-2
1 dedicated token. The model saw this exact character directly in training —
efficient, and a real language-level signal.
ϩ
Byte-fallback (BPE)
GPT-4o
2 tokens — split into raw UTF-8 bytes 207 169.
Round-trips fine, but the model never sees the character as a unit, only bytes
it saw rarely in training.
ϩ
Not supported
mBERT
0 tokens. No byte-fallback path exists, so the character becomes
[UNK] or is silently dropped before the model ever
sees it.
Same character throughout: Coptic ϩ, U+03E9 ("Shima"). Real tokenizer output — GPT-4o's byte split computed directly from its published o200k_base vocabulary; Gemma-2 and mBERT verdicts from data/coverage/*.json.
Gemma-3 natively tokenizes two-thirds of Akkadian cuneiform. Every other model tested is at 0–2%.
Native = character has its own dedicated token (Tier 0), not byte-fallback. 44 of 53 models were scored on historical/dead-script languages — the 9 most recently added weren't run through that part of the pipeline yet.
Best vs. worst tokenizer, by mean coverage across 310 languages
Gemma-2 leads at 0.961; Falcon-7B trails at 0.749 — a real 0.21 gap on a scale where 0.95+ is "Excellent" and below 0.50 is "Poor."
Model size doesn't move language coverage. The tokenizer does.
Mistral-7B and Mixtral-8x22B (141B) score identically — a 20× size range, zero difference — because they share a tokenizer. Same story for Qwen2.5 (0.5B→72B) and Gemma-2 (2B→27B).
Try it yourself
Check any character or language against every model at once, or explore a model's full coverage report, tier by tier.
https://apjanco-loove.hf.space/
Bold
By Andrew Janco
