{
  "artifact": "d2_positions_per_pass",
  "sprint": "D2 / 43",
  "cost_usd": 0.0033,
  "claim": "The oracle ceiling on positions committed per expensive forward pass on prose, JSON, tool calls and code patches, from grammar-forced and prompt-copyable positions of the TRUE continuation.",
  "method": "XGrammar builtin JSON grammar: a position is free when the mask admits exactly one token; prompt-lookup with min_ngram=3: free when the true next token follows a matching n-gram in the prompt. Positions per pass = tokens / non-free tokens. No model, no GPU.",
  "gate": {
    "worth_a_sprint": ">= 3 on JSON or tool calls",
    "dead": "< 1.5"
  },
  "tokenizer": "Qwen/Qwen2.5-1.5B-Instruct",
  "vocab_size": 151665,
  "min_ngram": 3,
  "xgrammar": "0.1.19",
  "transformers": "4.51.3",
  "domains": {
    "prose": {
      "dataset": "HuggingFaceH4/ultrachat_200k",
      "n_samples": 200,
      "sample_sha256": "91e9d60d099a5188e6375096d8a90dc7652ef3b11f3633c684bf02e55493e0f1",
      "tokens": 66902,
      "positions_per_pass_oracle": {
        "copy_only": 1.0396,
        "copy_plus_builtin_grammar": 1.0396,
        "copy_plus_schema_grammar": 1.0396
      },
      "positions_per_pass_median_sample_schema": 1.0212,
      "frac_grammar_forced_builtin": 0.0,
      "frac_grammar_forced_schema": 0.0,
      "frac_prompt_copyable": 0.038101,
      "n_left_builtin_grammar": 0,
      "n_left_schema_grammar": 0,
      "n_schema_compile_failures": 0,
      "n_not_valid_json": 0,
      "frac_not_valid_json": 0.0,
      "elapsed_s": 2.1
    },
    "json": {
      "dataset": "NousResearch/json-mode-eval",
      "n_samples": 100,
      "sample_sha256": "bfde93a551ff4e9705790b0d6cc92035d47c9636c810e413ab768235a75fefe7",
      "tokens": 6714,
      "positions_per_pass_oracle": {
        "copy_only": 1.2552,
        "copy_plus_builtin_grammar": 1.2552,
        "copy_plus_schema_grammar": 1.542
      },
      "positions_per_pass_median_sample_schema": 1.4936,
      "frac_grammar_forced_builtin": 0.0,
      "frac_grammar_forced_schema": 0.149538,
      "frac_prompt_copyable": 0.203307,
      "n_left_builtin_grammar": 0,
      "n_left_schema_grammar": 0,
      "n_schema_compile_failures": 0,
      "n_not_valid_json": 0,
      "frac_not_valid_json": 0.0,
      "elapsed_s": 9.2
    },
    "tool_calls": {
      "dataset": "glaiveai/glaive-function-calling-v2",
      "n_samples": 200,
      "sample_sha256": "5b6102c7eb16942a6410c556e74955da8f974965652d241ee9919fa866d91ef1",
      "tokens": 5878,
      "positions_per_pass_oracle": {
        "copy_only": 1.2195,
        "copy_plus_builtin_grammar": 1.2195,
        "copy_plus_schema_grammar": 1.5371
      },
      "positions_per_pass_median_sample_schema": 1.5,
      "frac_grammar_forced_builtin": 0.0,
      "frac_grammar_forced_schema": 0.171657,
      "frac_prompt_copyable": 0.179993,
      "n_left_builtin_grammar": 0,
      "n_left_schema_grammar": 0,
      "n_schema_compile_failures": 0,
      "n_not_valid_json": 0,
      "frac_not_valid_json": 0.0,
      "elapsed_s": 12.3
    },
    "code_patches": {
      "dataset": "nuprl/EditPackFT",
      "n_samples": 200,
      "sample_sha256": "91f886658b636976ba003ccc0bce0bc4beeafc681960d36cea59a1668b246e3e",
      "tokens": 41054,
      "positions_per_pass_oracle": {
        "copy_only": 4.831,
        "copy_plus_builtin_grammar": 4.831,
        "copy_plus_schema_grammar": 4.831
      },
      "positions_per_pass_median_sample_schema": 6.4451,
      "frac_grammar_forced_builtin": 0.0,
      "frac_grammar_forced_schema": 0.0,
      "frac_prompt_copyable": 0.793004,
      "n_left_builtin_grammar": 0,
      "n_left_schema_grammar": 0,
      "n_schema_compile_failures": 0,
      "n_not_valid_json": 0,
      "frac_not_valid_json": 0.0,
      "elapsed_s": 4.0
    }
  },
  "verdict": "BETWEEN THE GATES: 1.54 positions per pass on structured output with a declared schema. Not dead, not a sprint; a learned committer would need to capture most of this to matter. Schema-less JSON mode reaches 1.26 -- the builtin grammar admits whitespace almost everywhere and forces almost nothing. Code patches, the prompt-lookup regime, reach 4.83 (SpecDecode-Bench's 4.9x on code editing, reproduced).",
  "wall_s": 39.1,
  "does_not_prove": [
    "A ceiling: the oracle sees the true continuation. Every real committer -- draft model, n-gram, grammar jump-forward -- captures a fraction of it.",
    "Grammar-forced positions assume a JSON grammar is available at serve time, i.e. the caller declared structured output. Unconstrained JSON generation gets only the prompt-copyable share.",
    "One tokenizer (Qwen2.5). Positions per pass depend on how the tokenizer splits structural characters; a tokenizer that merges '\":' differently moves the number.",
    "Public datasets, 200 samples per domain, deterministic head of each stream. Not a production distribution of structured outputs."
  ],
  "generated_utc": "2026-09-02T15:07:18.936498+00:00",
  "git_rev": "c16fd85",
  "git_dirty": true,
  "run_id": "99d32c349616421f",
  "host_kind": "Darwin-arm64",
  "provenance_schema": "gpu-cert-provenance/1"
}