OpenAI details how it scaled Cosmos DB

OpenAI has published the second part of its engineering series on Habitat, the internal storage platform behind its products. Habitat serves more than 500 petabytes of data to over 1 billion users across 40 geographic regions, with millisecond response times. As we reported earlier, Part I covered Habitat's move from a Python client library to a service. Part II covers the storage layer underneath it, Azure Cosmos DB.

OpenAI describes going in a circle. It started with one multi-tenant Cosmos DB container per account, moved to single-tenant backing containers, and then went back to a multi-tenant container by default. The current setup, which the company calls "co-managed containers," was worked out with Microsoft's Azure team.

The first design had three problems. Clients picked logical partition keys themselves, so common keys such as user_id and workspace_id collided across products and shared the same throughput limits. The documented 10,000 RU/s limit per logical partition key is actually enforced at the physical partition, so noisy neighbors could hit unrelated keys. OpenAI says workloads for niche features sometimes disrupted core login and signup flows. Finally, as data grew, physical partitions split while provisioned throughput stayed the same. That cut the throughput available to each partition, so storage growth alone could squeeze active workloads even when traffic was flat.

The post's summary materials also cite more than 20 million queries per second on a single Cosmos DB account, a 12x cut in p50 read latency, and an in-house exabyte-scale storage system in development. The excerpt reviewed here does not explain how OpenAI got those results or give details of the new system.