OpenAI builds Pando to scale Git
OpenAI has replaced its monorepo's Git backend with Pando, a home-built system that cut median push latency nearly fivefold as Codex-driven parallel development pushed source control past its limits.
OpenAI described Pando in an engineering post. It is a new backend for Git repositories, and it now powers all development in the company's monorepo. OpenAI says it moved over without interrupting engineering work or changing how its team uses Git.
The company says Codex caused the problem. Engineers now run many changes in parallel, which means more branches, pushes, merges, and fetches from development environments, research jobs, and CI. One engineer may have several changes ready for CI at the same time, and each push can start several jobs that fetch the same commit. From around the end of last year, OpenAI says, its existing systems struggled: fetches and pushes were slow, merges failed, and reliability at times fell below 90%.
Pando was built for two goals: writes that scale horizontally across many branches, and reads fast enough to saturate a client's network bandwidth. To scale writes, it coordinates updates to each branch independently, so pushes to unrelated branches can run in parallel. The tradeoff is that a reader looking at several branches may see them at different points in time. To scale reads, Pando reuses packfiles across fetches instead of building a custom one for each client. Clients sometimes download objects they already have, but the server can send the same bytes to many readers with little computation.
In production, OpenAI says, Pando cut median push latency by nearly 5x and now serves nearly a petabyte of Git data per day. It works with the standard Git protocol and client, so engineers and agents did not have to change their workflows. OpenAI also says it plans to contribute client improvements from its fork back to upstream Git.