# D2 / Sprint 43 — **positions per expensive pass: the oracle ceiling on structured output**

**Certificate:** `results/d2_positions_per_pass.json` · generator
`modal_app/positions_per_pass.py` · CPU only, **~$0.05** (the plan budgeted $3 of GPU; the
oracle needs a tokenizer and a grammar, not a model)

## Hypothesis

A decoding regime can commit > 2 tokens per expensive forward pass on structured output at
unchanged token identity. It is the only lever past the 1.648× Amdahl ceiling (§19.3),
because it changes passes per token rather than bytes per pass. **Gate, as Sprint 43 wrote
it:** oracle commit rate ≥ 3 on structured output → worth a real sprint; < 1.5 → dead.

The oracle is given the true continuation. A position is free if the JSON grammar admits
exactly one next token (jump-forward) or the next tokens are copyable from the prompt
(prompt-lookup, n-gram 3). A perfect committer pays one pass per non-free token. Four
domains: prose (control), JSON, tool calls, code patches; 200 public samples each, sha-pinned.

## Result — **between the gates: 1.54 with a declared schema**

| domain | n | tokens | copy only | + builtin JSON grammar | + declared (hindsight) schema | grammar-forced share (schema) | prompt-copyable share |
|---|---:|---:|---:|---:|---:|---:|---:|
| prose (control) | 200 | 66,902 | **1.040** | 1.040 | 1.040 | 0 | 3.8% |
| JSON | 100 | 6,714 | 1.255 | 1.255 | **1.542** | 15.0% | 20.3% |
| tool calls | 200 | 5,878 | 1.220 | 1.220 | **1.537** | 17.2% | 18.0% |
| code patches | 200 | 41,054 | **4.831** | 4.831 | 4.831 | 0 | 79.3% |

Three things the number says, in order of how much they were expected:

1. **Schema-less JSON mode has nothing to jump.** The builtin JSON grammar admits whitespace
   almost everywhere, so the jump-forward string is empty at every position: grammar-forced
   share **0.0** on 12,592 structured tokens. Every gain on the JSON and tool-call rows comes
   from a *declared* schema (key strings, structure) — which is the realistic serving case
   (`response_format: json_schema`, tool definitions) and the only one worth building for.
2. **With a declared schema, a perfect committer gets 1.54 positions per pass** on both JSON
   and tool calls. Above the 1.5 kill line, below the 3.0 sprint line. A learned committer
   would have to capture most of a 35% ceiling to matter; jump-forward alone (grammar-forced
   15–17%) is worth roughly 1.2×.
3. **Code patches: 4.83** from prompt-copy alone (79% of tokens) — SpecDecode-Bench's 4.9× on
   code editing, reproduced with a different tokenizer and corpus. Known, and not ours.

The run also took three attempts to get the oracle right: a list-typed prompt in
`json-mode-eval` (crash), an "exactly one admitted token" criterion that scored 0 because a
dozen whitespace-variant tokens share every forced character (0.0 on 6,714 tokens), and
glaive's single-quoted inner arguments that left the JSON grammar on 195 of 200 calls until
normalised to the wire format (now 0 of 200 invalid). Each is in the certificate.

**Disposition.** The decoding wedge is neither dead nor a sprint. It is a **1.5× ceiling on
declared-schema structured output** for a mechanism vLLM does not ship and SGLang does. That
is a one-line argument for jump-forward in vLLM and not a research program.

## Prior art

*Searched 2026-09-02.*

- SGLang compressed-FSM jump-forward — https://www.lmsys.org/blog/2024-02-05-compressed-fsm/ —
  (Feb 2024) commits grammar-forced runs without a pass. XGrammar 2411.15100 supports it;
  XGrammar-2 2601.04426. **vLLM does not ship jump-forward** (logit-processor architecture,
  vLLM blog 2025-01-14).
- Prompt lookup decoding (Nov 2023) — https://github.com/apoorvumang/prompt-lookup-decoding —
  the copyable half; vLLM's n-gram speculative decoding.
- SpecDecode-Bench, 2601.11580 — https://arxiv.org/abs/2601.11580 — oracle simulator,
  "Oracle Combine" 4.9× on code editing, n-gram spans 1.1–15 tokens; **no JSON, no tool-call
  workload, no grammar**.
- Constrained decoding with speculative lookaheads, 2412.10418; draft-conditioned
  constrained decoding, 2603.03305; Φ-estimation, 2605.07698.
- **What is left:** the ceiling on positions per pass for JSON and tool calls with grammar
  and prompt-copy combined, on a tokenizer vLLM users run. Not a mechanism; a number the
  literature lacks, and a motivation (or an obituary) for jump-forward in vLLM.

## Does not prove

- A ceiling. Every real committer captures a fraction.
- Grammar-forced positions assume the caller declared structured output.
- One tokenizer; 200 public samples per domain from the head of each stream.