# D3 -- prefix-cache-state nondeterminism at temperature 0: the controlled 2x2. ~$1 on an L4.
#
# WHAT S53 FOUND AND DID NOT TEST
# ===============================
# results/s53_demo.json: six identical temperature-0 requests to a freshly booted engine --
# the FIRST returned different bytes, the next five were byte-identical to each other. The
# mechanism was ASSERTED (cache-hit vs cache-miss prefill reduce in a different order) and
# not tested: there was no prefix-caching-off arm and no batch-invariant arm.
#
# Prior art (HYPOTHESES/037): vLLM issue #40896 (Apr 2026, open) reports the identical symptom
# with no root cause; Thinking Machines (Sep 2025) names chunked-vs-one-shot prefill reduction
# order; vLLM ships VLLM_BATCH_INVARIANT=1 (beta) whose docs do not state cache-hit/miss
# equivalence. Nobody has run the 2x2. This does, and pre-registers what each cell means.
#
# DESIGN
# ======
# Four cells: {enable_prefix_caching: on, off} x {VLLM_BATCH_INVARIANT: 0, 1}. Per cell, SIX
# COLD BOOTS (a fresh `vllm serve` each); per boot, SIX identical temperature-0 requests sent
# one at a time, for TWO prompts (short factual, longer open-ended). Recorded per boot: the
# six completions, whether the first differs from the rest, whether the rest agree.
#
# DECISION RULE (pre-registered)
# ==============================
#   apc_off cells show NO first-request divergence and apc_on/invariant=0 does
#       -> the divergence is prefix-cache-state dependent (S53's mechanism confirmed).
#   apc_on/invariant=1 shows none while apc_on/invariant=0 does
#       -> the existing flag closes it; post the table on #40896 and stop (D3 is KNOWN).
#   apc_on/invariant=1 still diverges
#       -> an open engine gap with the first controlled characterisation; W1 gains the
#          warm-state clause with a cause.
#   divergence appears in apc_off too
#       -> S53's mechanism was wrong; it is boot-state, not cache-state. Report it.
#
# Cost: 24 boots x ~75 s + 24 x 12 requests x ~1 s on an L4 at $0.80/h ~ $0.55; cap 3.0.

artifact: d3_determinism_2x2
sprint: "D3"
claim: >-
  Whether the first-request-after-boot output divergence at temperature 0 depends on prefix
  caching, on batch-invariant mode, on both, or on neither -- measured as a 2x2 with six cold
  boots per cell.

model: Qwen/Qwen2.5-1.5B-Instruct
provider: modal
gpu: L4
cost_cap_usd: 4.0        # worst case = 4 cells x 1 h function timeout x $0.80; expected ~$0.55
boots_per_cell: 6
requests_per_boot: 6
max_tokens: 48

server_args:
  max_model_len: 4096
  max_num_seqs: 64
  gpu_memory_utilization: 0.85

cells:
  - {name: apc_on_inv0,  enable_prefix_caching: true,  batch_invariant: 0}
  - {name: apc_off_inv0, enable_prefix_caching: false, batch_invariant: 0}
  - {name: apc_on_inv1,  enable_prefix_caching: true,  batch_invariant: 1}
  - {name: apc_off_inv1, enable_prefix_caching: false, batch_invariant: 1}

prompts:
  - {id: colours,  text: "Name three primary colours."}
  - {id: bridge,   text: "Explain in four sentences why a suspension bridge can carry more load than a beam bridge of the same span."}

does_not_prove:
  - "One model, one GPU class, one engine version (the pinned vLLM). Determinism behaviour is
     kernel- and version-specific."
  - "Sequential requests only. Concurrent batch composition is a SEPARATE source of
     nondeterminism (the one batch-invariant mode is documented to address) and is not varied
     here; the cold-start effect is isolated by sending one request at a time."
  - "Six boots per cell resolves a divergence that happens on most boots, not a rare one. A
     cell reporting 0/6 bounds the per-boot probability at roughly < 0.4 at 95%, no tighter."