Cache offload headroom depends on the traffic

A KV cache keeps work a model server has already done on a prompt, so a repeated prompt opening is not computed again. GPU memory is small, so engines can spill cache blocks to host memory and fetch them back. The lab asked how much such a host tier could add to the hit rate at most, and whether that ceiling is a property of the traffic. It ran one calculation, at a fixed GPU cache of 65,536 tokens, on two independently sourced streams of chat requests.

What it shows

In this model: one calculation, run unchanged on two streams of cache-block lookups, with the GPU cache held at the same number of tokens for both. The host tier has no size limit and no transfer cost.

On a slice of the public Mooncake trace, the most a host tier could add over GPU-only LRU is 13.004 percentage points of hit rate. On a stream derived from WildChat conversations, at the same block size, it is 0.642 percentage points.

Why the two traces differ

The ceiling is set before any cache policy runs; the headroom also depends on how much of that ceiling GPU-only LRU already reaches at this cache size, and that is what separates the two traces. On the Mooncake slice, most lookups are the first for their block: the share of lookups that must miss is 0.8313, against 0.3267 on the WildChat stream. A first lookup misses in every cache, so no tier can recover it.

The share of reachable hits shows the rest. GPU-only LRU already reaches 22.9% of the hits any cache could reach on the Mooncake slice, and 99.0% on the WildChat stream. On the second, there is almost nothing left for a host tier to win.

A control for block size

The two streams cut prompts into blocks of different sizes, and smaller blocks raise a hit rate on their own. So the lab also regrouped the WildChat stream into blocks of the Mooncake size, from 16 to 64 tokens. Its headroom moved from 0.5732 to 0.6421 percentage points. Block size explains 0.5% of the gap between the two traces.

Why it matters, and to whom

For teams that build or buy KV cache offloading, the result says to measure the traffic before the connector. The ceiling comes from two counts of a trace: its lookups and its distinct blocks. On one trace it leaves real room for a host tier; on the other, almost none. A benchmark that reports offload gains on one trace has measured that trace's reuse pattern, how often and how far apart it repeats against the GPU cache, as much as the connector.

It also says what to report. Two offload results on different traces can only be compared when each reports, beside its gain, its trace's share of first lookups and what GPU-only caching already reaches on it.

Why now

Engines and cache layers now ship offloading of KV cache from GPU memory to host memory and storage, as in the open-source LMCache project cited below. Whether a deployment gains from it depends on how often its own traffic repeats. This ceiling can be computed from a buyer's own request log, before any offload hardware or software is bought.

How it was checked

Both figures come from one oracle, unchanged, run on the two streams: LRU in the GPU alone, against a host tier with no size limit. The results file states the method, the counts behind each figure, and the block-size control. It records the code revision it was generated at, and that the working copy had uncommitted changes at the time. The lab's later audit re-ran its command and recorded the same numbers; the extract linked below carries that audit's entry as written.

How to reproduce

The published results file names the lab's command, the code revision, and every count behind the two figures. Anyone with access to the lab's code can rerun it. The lab's code is not public yet. The results file is published here, and its hash is listed in the site's receipt manifest, so a downloaded copy can be checked byte for byte.

In words: a trace τ makes N lookups of d distinct cache blocks. A cache that starts empty and fetches a block only when asked must miss each block once, so no such cache, of any size, can serve more than one minus d over N of the lookups. The headroom is that ceiling minus what least-recently-used (LRU) caching reaches with only the GPU's cache of size C. It bounds what an offload tier that starts empty and fetches only on request could add over GPU-only LRU on that trace, and measures no such tier; a tier that prefetches, or starts warm, is not bounded by it.