The problem
AI clusters spread each transfer over many network paths, so the pieces arrive out of order. The receiving network card must track what is still missing, in on-chip memory that is small and fixed.
AI networks and caches have hard limits. We find them, and design for them.
Designs to license for AI network-card makers: a receiver that tracked out-of-order data in about half the memory of our STrack model, in simulation, now that AI Ethernet spreads traffic over many paths. Also: proved bounds for shared-cache models.
Why this matters now
Why now
Transports for AI clusters, such as Ultra Ethernet and STrack, spread each transfer over many paths, so data under them arrives out of order. Windows grow with link speed, and the memory on a network card is small.
Who it is for
Makers of network cards and switch chips for AI Ethernet; inference providers and GPU clouds weighing a cache vendor’s hit-rate promise; and the diligence teams who check both.
The results
Each result has its own page, with its conditions.
Networking · simulation
Receiver memory for out-of-order data
In simulation, our receiver tracked out-of-order data in about half the memory our STrack model needed, by geometric mean over the settings tried.
Read the resultTracking memory each design needed, relative to our receiver (ten blocks), over the simulated settings; the STrack ratio is a geometric mean
Our receiver
Our simulator’s model of STrack, a published design: 1.965x
A second fixed-size design, modelled in our simulator: 1.443x
Caching · proof
A hit-rate ceiling on a public trace
Lookups on the test slice of our excerpt of the public Mooncake trace, one block per hundredth
- The ceiling for a demand cache that starts empty on this trace slice, from a proved bound: 19.645%
- Least-recently-used caching, simulated at 1,693 blocks: 5.497%
On a test slice of our Mooncake trace excerpt, no cache that starts empty and fetches only on request beats a proved ceiling.
Read the resultCaching · outside solver
Pooled cache placement reaches the optimum
Misses on one fixed test instance, one block per miss
The pooled cache: 32
The fewest possible, proved by an outside solver (Google OR-Tools): 32
On one fixed test instance, a pooled KV cache reaches the fewest misses possible, and an outside solver certifies it.
0.9375Hit rate with one pooled cache, vs. 0.3398 with a cache per host, at 16 servers, on one test instance with no cache-size limit
Read the resultWho is behind this
Nick Harris, founder. CTO of VivaMed BioPharma; co-founder of MedSim.ai, FastRead.io and Formulai.
A proof library in Lean
The inequality behind the cache ceiling is machine-checked in Lean; the ceiling’s value comes from our trace counts. Read the proof.
Checked by outside programs
Google’s OR-Tools solver confirms the placement optimum and the Lean proof checker the inequality behind the cache ceiling. The network result is a simulation, which neither checks. The solver’s certificate.
Open-source tools
16 code repositories on GitHub, including a server that lets AI agents check a proof step by step. See each one.
AI agents do our research and engineering. Where a result is proved, a program decides whether the proof holds; whether it fits a real system is our judgment, stated on each result page. Write to nick@latticegraph.com.
Check one of our files yourself
Drop one of our result files here and your browser confirms it is exactly the file we published. Nothing is sent anywhere.
Theorems & Bounds
Two results, each with its conditions stated beside it. A program checked each one: the Lean proof checker, or an outside solver. The plain-English statement comes first; the mathematics sits beside it. The hit-rate ceiling is set out the same way on the theorems page.
The bookkeeping floor for a shared cache
Take T customers (tenants) sharing a cache of n stored blocks. If any customer may be granted any block independently, a cache whose state must answer exactly, for every customer and block, whether that customer may use it (so it never serves a block to a customer without the right, and never refuses one with it) must keep at least T×n bits of bookkeeping about who may use what: one bit for each customer and block.
Pooled placement reaches the optimum on one instance
On one fixed, synthetic serving instance with no limit on cache size, a pooled KV cache reaches the proven minimum number of misses, and separate caches on each server miss more, with requests spread as this instance spreads them; Google’s OR-Tools solver certifies the optimum.
Intellectual property
The drafted claims cover checking what each request may do before an AI serving system runs it: rights to cached data, safe reuse across servers, wiping memory before reuse, and admission checks.
727 claims drafted
Patent pending (as stated by AxiomLimit; filing details on request). What the claims cover.
Where the numbers come from
Every number on this page links to the file it comes from. All published files.
- Loadingshown only once its file has loaded in your browser
- Not checkeda question we have not checked would say so, with no number
- Checked, nothing founda search that found nothing would say so, with no number
- No valuea file that holds no value for the question would say so
- File missinga number whose file is missing or altered would be hidden
- Unclear subjecttwo files that disagree about what a number describes would both be shown
- Small samplea number from a small sample would carry its sample size
- Conflicting filestwo files giving different values would both be shown
- Not publishablea file we may not publish would be named by its fingerprint only
- Out of datea measurement older than a week would carry its age
- Does not applya question that does not apply to this page would say so
- Run faileda measurement whose program failed would say so, with no number