Why a cache on every server wastes work
When requests that share a prompt opening are spread across servers, a separate cache on every server recomputes that opening on each server that sees it, so its hit rate falls as the fleet grows.
Picture a popular system prompt. Requests that carry it are spread over many servers. With a separate cache on each one, every server computes the opening for itself the first time it sees it. The more servers there are, the more times the same work is done, and the smaller the share of requests that find a saved copy.
The lab’s one fixed test instance, which has no limit on cache size, shows the effect. It has 512 requests over 32 different prompt openings:
- With one pooled cache, the hit rate stays at 0.9375 at every fleet size the lab ran.
- With a separate cache on each server, it falls as servers are added, from 0.875 for the smallest fleet to 0.3398 for the largest.
- one pooled cache
- a separate cache on each server
Hit rate on the lab’s one fixed test instance as the fleet grows, as a fraction of all requests. Solid bars are one pooled cache. Hollow bars are a separate cache on each server. The values come from the certificate for this instance.
Pooling is not the sole route to a higher per-server hit rate. A router can send requests that share an opening to the server that already holds it. The Mooncake paper pairs a pool with a scheduler that is aware of what the cache holds. Routing and pooling answer the same waste from two sides.
What pooled placement changes
Pooled placement keeps one shared pool of cached work for the whole fleet, so an opening computed on one server can be fetched by any other.
The idea is published design. The Mooncake paper builds a production serving system around a pool of cached work spread over a GPU cluster, and its title states the trade: more storage for less computation. The first post in this series sets out the background.
What pooling costs is movement. A block fetched from the pool crosses a network, and that time counts against the time saved. A count of misses leaves that time out, so it cannot settle on its own whether a pool is faster in practice.
What the one test instance shows
On that one fixed test instance, pooled placement reached 32 misses, and an outside solver confirmed that no placement can do better in the lab’s model of that instance.
The instance is fixed and uncapacitated. Each of its 16 hosts keeps whatever it fetches, with no limit on how much it can hold. The lab counted the misses, which are the times a request found nothing to reuse. Pooled placement had 32. The lab then gave the same instance to CP-SAT, the constraint solver in Google’s open-source OR-Tools suite. The solver returned a proven optimum equal to that count, with a matching bound of 32.
That optimum is also the number of distinct openings in the instance. With no limit on what a host holds, a pool misses only the first time each opening appears. No demand cache that starts empty can avoid those first misses. They are the compulsory misses of the textbook. So the solver confirms a floor that simple arithmetic already implied.
What an outside solver adds
An outside solver adds an independent check on the lab’s own claim that the pooled count is the best possible.
The lab’s code says that no placement can do better than the pooled count. A solver the lab did not write, OR-Tools CP-SAT, returns both a solution and a proven bound, and the two agree. Anyone with the lab’s code and OR-Tools can rerun the check and compare. The certificate also records a case where a greedy placement falls short of the best, so it does not claim that greedy is optimal. The solver is there to test the claim, not to flatter the pool.
That habit is worth copying. A claim of optimality is easy to make and hard to check, and the cheapest way to check it is to hand the same instance to a tool that you did not write and that has no stake in the answer.
Where the hard part begins
The hard part begins when memory is limited, because then a fleet must choose what to keep and what to forget.
This instance has no such limit. Real fleets do. With a limit on each host or on the pool, the questions are which blocks to hold, where, and for how long. A pooled design also needs a rule for which server handles a request and how it fetches what the pool holds. The lab’s result does not cover that regime, and it says so. The optimum is for the uncapacitated instance only.
The ceiling in the hit-rate post still applies. A pool of demand caches cannot beat the share of lookups that are repeats. Pooling helps a fleet get close to that share. It cannot lift it.
What this means for you
Pooling is a design choice with a measurable payoff, and each kind of reader can ask a concrete question about it.
- Inference providers. Measure the hit rate each server reaches on its own, and compare it with what one pool would reach on your routing. The gap is work you repeat.
- GPU clouds. A pool is shared state across machines, and across tenants if you let it be. Pair any pool with the isolation checks in the post on leaks.
- Enterprise AI platforms. Ask whether a provider’s cache is per server or pooled, and how its hit rate changes as the fleet grows.
- Auditors. Ask for the instance, the count of misses and the floor the count is checked against. A claim of optimality needs an independent check.
What this does not show
This is one fixed test instance, not a fleet benchmark, and it says nothing about caches that must evict.
- The optimum is for an instance with no limit on host memory. It does not cover caches that must forget.
- It counts misses. It does not count the time spent moving cached work between machines.
- The hit rates in the chart come from the lab’s saved runs of this one instance. They are not a forecast for real traffic.
- It does not claim that pooling is optimal at any scale.
The lab’s own limits for this result, word for word
ONE committed instance, not a fleet benchmark — the CP-SAT certification is of THAT instance and the Lean half is the general statement; they are separate objects and must not be merged into "provably optimal at any scale".