Running ChatGPT for more than a billion people is a storage problem as much as an AI one: a billion users’ data needs serving, access control, data residency, and headroom for launch-day spikes — all from one shared platform. That platform is Habitat, the OpenAI storage platform behind every OpenAI product, and its engineering team has just published a detailed two-part engineering account of what it takes to run it.
Habitat is not a database product you can download. It is an internal online storage platform – request/response plus change data capture – that serves ChatGPT, Codex, and OpenAI’s internal services. The headline numbers are hard to internalize: over one billion users every week, across roughly 40 geographic regions, holding more than 500 petabytes of data, answering more than 70 million requests per second. The platform wraps its backing stores in the features a company of that size needs to survive: caching, access control, data residency, encryption, multi-tenancy, rate limiting, and routing. Under the hood, as the post details, Habitat is backed by Azure Cosmos DB, Nanobase, Valkey caches, and blob storage.
What makes the two-part engineering post interesting is not the scale itself but the journey. Over three phases, the team looped all the way around the multi-tenancy question: start shared, go isolated, then return to shared – this time engineered. It is one of the clearest public case studies of how AI workloads break conventional capacity planning, and what the fix actually looks like.
Habitat: the OpenAI storage platform behind a billion ChatGPT users
To understand the three-phase journey, it helps to know what Habitat had to become. In Part I of the series, the team traces the platform’s origin to a Python client-side library launched at DevDay 2023 for GPTs. It was a pragmatic choice – the company had, in the engineers’ own words, “grown more than 10x year-over-year for the last three years,” and they needed to ship fast.
For Habitat, the client-side model broke down. Every change took days of client-side coordination, and one rollback caused an outage. The mid-2025 migration turned Habitat into a standalone service, and the team made a deliberate decision to ship it in Python despite the GIL and tail-latency costs. Their reasoning was disarmingly honest: “We bet that by the time a full migration off Python was required, Codex and GPT would make that migration achievable. That bet eventually proved correct.” When your company builds the coding models, you can plan to use them.
Why multi-tenancy is the hard problem for AI workloads
At the center of the story is a tension every platform team recognizes. Run each workload in its own isolated tenant and you get a small blast radius and clear ownership – but every product team ends up doing its own capacity planning, and everyone over-provisions to be safe. Share one big multi-tenant platform and you get centralized expertise and better provisioning – but noisy neighbors and shared limits can ripple a workload’s bad day into the other workloads sharing its capacity.
AI workloads make both extremes painful at the same time. Growth is explosive and lumpy: a new feature can go from zero to a million queries per second on launch day. Traffic concentrates in hot keys and hot organizations. And a single platform serves wildly different workload shapes – interactive user requests with tight latency budgets next to bulk offline jobs that mostly need throughput. The Habitat team tried the shared extreme first.
Phase I: one shared Cosmos DB container and what broke
Habitat started on Azure Cosmos DB with one default container per account, relying on the documented 10,000 RU/s per logical partition key as a back-pressure mechanism. The team’s assumption at the time was that the documented per-logical-key cap would hold individual keys in check. But a documented maximum is not a guaranteed budget: logical partitions share the capacity of their physical partition, so a hot key could still degrade everything colocated with it. The cap offered less protection than the team assumed.
In practice, three failure modes emerged. First, logical partition key collisions. Early API design choices exposed partition-key selection to clients, and different use cases independently picked the same keys – common ones like user_id and workspace_id collided across unrelated features. Second, noisy neighbors. Throughput limits in Cosmos DB are enforced at the physical partition, so logical partitions that happen to share a physical partition share a ceiling with unrelated workloads. Niche-feature workloads ended up disrupting core flows like login and signup. Third, weak partitions. As data grows, Cosmos DB splits physical partitions automatically – and each child receives half the throughput. So storage growth alone halved per-partition throughput, from 10K down to 5K RU/s, even when request rates stayed flat. Your throughput could shrink simply because your data got bigger.
As the engineers put it: “When companies grow to OpenAI’s scale, managed services become less ‘managed.'”
Phase II: dedicated containers per workload and the capacity-planning tax
The obvious fix was isolation: give each critical use case its own dedicated container with its own independent physical partitions. A control-plane script synced provisioned throughput to physical partition counts. The isolation problems went away.
But isolation charged its own tax, and it was steep. Every container needed individualized capacity planning. Product teams forgot to pre-scale, and launch days turned into incident rooms. Dedicated containers also lost the one great benefit of multi-tenancy – workload mixing. Consider the arithmetic the team lays out: a launch that ramps from zero to a million QPS consumes roughly 5 percent of a large shared container’s aggregate headroom, but 10,000 percent of a dedicated container provisioned at 10K RU/s. The same launch that is a rounding error on shared infrastructure is a catastrophic overload in isolation.
0 → 1,000,000 QPS launch
Shared container: 1M QPS ≈ 5% of aggregate headroom
Dedicated container: 1M QPS = 10,000% of a 10K RU/s containerThe failure mode became organizational. Capacity planning “essentially” involved every product team at the company. And the legacy critical use cases stuck in the original default container had no data migration path to better homes at all. Isolation had solved the technical problem and created a human one.
Phase III: co-managed containers and the multi-tenant return
So the team went back to multi-tenant by default – but this time engineered, in what the post calls co-managed containers, built through a close working partnership with the Azure Cosmos DB team. “Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.”
The return-to-multi-tenant design targets the contention and capacity-planning failure modes with concrete mechanisms. Hot key throttling throttles the hottest keys first, instead of letting them saturate a shared physical partition. QoS request tiering gives interactive user requests priority over offline bulk workloads. And a new improved serverless mode scales every physical partition to 10K RU/s with splits driven by actual RU usage – which, the team says, ended the manual monitoring and scaling work entirely. These map onto the noisy-neighbor problem and the capacity-planning tax; the post does not detail a mechanism for preventing cross-use-case partition-key collisions, which remains an unestablished mechanism in the supplied evidence.
The “co-managed” part is organizational, and it reads like the hard-won lesson of the whole series. The teams share tests, shared dashboards, and direct engineering channels; they page each other directly without support tiers; OpenAI tests Azure changes against a production-sized non-production Cosmos DB account and jointly tests SDK changes. “Any system large or critical enough necessitates this kind of teamwork: breaking down ownership walls and operating across organizational boundaries.”
At this scale, even failover is a physics problem. The post recounts a late-2024 regional outage where a power failure in the primary region revealed that OpenAI’s largest Cosmos DB accounts would take several hours to fail over. Worse, failing back triggered a connection stampede that overwhelmed the original region’s gateway – traced to a quirk in the gateway-mode client’s failover handling. Their largest account stores tens of petabytes and serves over 20 million QPS; to the team’s knowledge, they are the only company deploying Cosmos DB at this scale.
Then came the read-performance program. Reads are over 90 percent of Habitat requests, and the team layered four strategies: storing data near users where capacity allows, broadly replicating only the smallest latency-sensitive datasets (some accounts replicate to all 40 regions, others to at least one region per continent), an integrated Valkey lookaside cache with change-data-capture invalidation propagating globally within seconds (following Meta’s TAO pattern), and a Rust rewrite of the Habitat proxy overhead. Global read p50 fell from around 120 milliseconds to about 10 – a 12x improvement, verified against the post’s numbers before being carried into this article.
On the results, the claim is striking: “We have not had any provisioning or noisy neighbor incidents in the months since, despite accelerating product development.” The remaining limitation the team names is where the story ends: they are building a custom in-house exabyte-scale online storage system for the next billion users – a RocksDB Cloud core from Rockset with Raft-based replication, the same document model as Cosmos DB, disaggregated storage to cut flash capacity, at a lower peak QPS. It is announced, not shipped.
What teams running AI workloads should take away
Habitat’s journey is a case study in the central tradeoff of AI infrastructure: share versus isolate. The lessons transfer even at far smaller scales.
First, contain by workload need, not by default. Isolation buys a small blast radius and clear ownership, but it trades away aggregate headroom – the shared capacity pool that absorbs sudden launches. The question is not “should we share” but “what does this workload need containment for,” and the default should lean toward sharing unless a specific failure domain demands isolation.
Second, exploit the workload-mixing benefit deliberately. Multi-tenancy’s superpower is statistical multiplexing: the launch spike that would sink a dedicated 10K RU/s container is five percent of a shared container’s headroom. In Habitat’s design, provisioning each workload separately meant paying the capacity-planning tax while losing the pooling benefit across separately provisioned workloads. Tier your requests – interactive above bulk – as one of several controls, alongside hot-key throttling, capacity management, and operational cooperation, that reduce contention when sharing.
Third, design partition keys for cross-use-case collisions from day one, and treat data residency, routing, and rate shaping as first-class platform features rather than bolt-ons. Habitat’s Phase I failures were not exotic: exposing partition-key choice to clients produced cross-use-case key collisions, while physical-partition contention and storage-driven splits were distinct mechanisms with their own dynamics. And at serious scale, the operational answer to “the managed service can’t manage this” is to co-manage it – shared tests, shared dashboards, direct channels – rather than pretending the abstraction boundary still holds.
