Liquid AI open-sources agent-built tokenizer trainer
Liquid AI has open-sourced toktoktok, a BPE tokenizer trainer that coding agents built on their own, and reports that the toy versions passed every test but the trainers didn't scale to production data without verification loops.
Liquid AI described an experiment from late 2025 that asked whether coding agents could solve a production-grade problem from scratch with no human help. The result is toktoktok, a byte-pair encoding (BPE) tokenizer trainer that is now open source on GitHub.
The company needed a trainer that could process trillions of tokens on a single machine for its research on vocabulary size in edge LLMs. It said existing tools fell short. Sentencepiece is slow for BPE, Hugging Face tokenizers ran out of memory on its corpora, and tiktoken cannot train tokenizers at all. Liquid also wanted two features it says existing libraries lack: warm-starting from an existing tokenizer and setting a vocabulary budget per language.
Two agents worked in planning mode: Claude Opus 4.5, and Codex running GPT-5.2. Each got a single AGENTS.md/CLAUDE.md spec that listed goals and constraints, with memory as the main constraint. Each also got sandboxed access to Liquid's production dataset on a machine with an AMD EPYC 9755, 128 cores and 2 TB of RAM. Neither agent could change the verification harness. It checked that the output loaded in both tiktoken and Hugging Face tokenizers, that text survived encode and decode round trips, and that the two libraries gave the same token IDs across languages, numbers, CRLF line endings and source code. The operator read only the harness results and never looked at the code.
Both agents produced a working trainer within 30 minutes that passed its own unit tests on a few megabytes of data. Neither trainer worked on the full production dataset without further loops, because small, clean inputs never triggered the failures. The excerpt provided ends before Liquid explains those failures or how the loops fixed them.