Liquid AI details in-place tokenizer upgrade recipe

Liquid AI has published the method behind the 128K-token tokenizer in LFM2.5-8B-A1B, the edge model we covered at launch. The method takes a model that has already been pre-trained and upgrades its tokenizer, which avoids running pre-training again. A technical report with ablations and per-device numbers accompanies the post.

The original 65K byte-level BPE tokenizer was built for English, code and a fixed set of languages, and it split Hindi, Vietnamese and Thai into many small pieces. To fix this, Liquid seeded a new merge table with the original merges, froze them, and continued BPE training on multilingual data. Most old tokens carry over unchanged. Every new token breaks down exactly into old tokens, so its embedding starts as the mean of its sub-tokens' embeddings, with no random initialization.

Adaptation ran in two stages. First, Liquid trained only the new embedding rows on 600B tokens. Then it ran full continued pre-training on 400B multilingual tokens. Swapping in the new tokenizer cost 5.8 aggregate benchmark points. Stage 1 recovered 4.8 of them, and stage 2 ended above the source model, a gain Liquid attributes mostly to the extra pre-training.

Token counts dropped 4.0× for Thai, 3.4× for Bengali, 2.6× for Vietnamese and 2.4× for Hindi. Liquid estimates this makes on-device decoding 2.2–3.7× faster per character for these languages. Languages the old tokenizer already handled well decode up to about 9% slower. Liquid stopped at 128K because a 256K vocabulary cut throughput on a Snapdragon phone by up to 37%.

The recipe only works when you own the original tokenizer's merge rules. It does not cover moving to a third-party tokenizer. Liquid recently open-sourced a tokenizer trainer, but the post does not say whether that tool was used here.