{ "artifact": "order_effect", "sprint": "1 / E3 / C14", "claim": "Whether the ORDER in which paired arms are run biases the result -- Sprint 1's unrun E3, answered from committed replicates.", "method": "A/A runs ONLY, where both arms are the identical configuration so any systematic first-vs-second difference is an order effect by construction. Real A/Bs are excluded rather than pooled, because there order confounds with treatment. Paired log-ratios of first to second arm, tested with bench.stats.paired_t.", "n_replicates": 30, "replicates": [ { "file": "s01_aa_null_sharded.raw.json", "rep": 0, "order": [ "a1", "a2" ] }, { "file": "s01_aa_null_sharded.raw.json", "rep": 1, "order": [ "a2", "a1" ] }, { "file": "s01_aa_null_sharded.raw.json", "rep": 2, "order": [ "a2", "a1" ] }, { "file": "s01_aa_null_sharded.raw.json", "rep": 3, "order": [ "a1", "a2" ] }, { "file": "s01_aa_null_sharded.raw.json", "rep": 4, "order": [ "a2", "a1" ] }, { "file": "s01_aa_null_tiny.raw.json", "rep": 0, "order": [ "a1", "a2" ] }, { "file": "s01_aa_null_tiny.raw.json", "rep": 1, "order": [ "a2", "a1" ] }, { "file": "s01_aa_null_tiny.raw.json", "rep": 2, "order": [ "a2", "a1" ] }, { "file": "s01_aa_null_tiny.raw.json", "rep": 3, "order": [ "a1", "a2" ] }, { "file": "s01_aa_null_tiny.raw.json", "rep": 4, "order": [ "a2", "a1" ] }, { "file": "s01_variance_levels.raw.json", "rep": 0, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels.raw.json", "rep": 1, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels.raw.json", "rep": 2, "order": [ "a2", "a1" ] }, { "file": "s01_variance_levels.raw.json", "rep": 3, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels.raw.json", "rep": 4, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L1.raw.json", "rep": 0, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L1.raw.json", "rep": 1, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L1.raw.json", "rep": 2, "order": [ "a2", "a1" ] }, { "file": "s01_variance_levels_L1.raw.json", "rep": 3, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L1.raw.json", "rep": 4, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L2.raw.json", "rep": 0, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L2.raw.json", "rep": 1, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L2.raw.json", "rep": 2, "order": [ "a2", "a1" ] }, { "file": "s01_variance_levels_L2.raw.json", "rep": 3, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L2.raw.json", "rep": 4, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L3.raw.json", "rep": 0, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L3.raw.json", "rep": 1, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L3.raw.json", "rep": 2, "order": [ "a2", "a1" ] }, { "file": "s01_variance_levels_L3.raw.json", "rep": 3, "order": [ "a1", "a2" ] }, { "file": "s01_variance_levels_L3.raw.json", "rep": 4, "order": [ "a1", "a2" ] } ], "excluded_as_ab": [ "s01_positive_control.raw.json", "s01_positive_control_eager.raw.json" ], "first_position_counts": { "a1": 20, "a2": 10 }, "metrics": [ "ttft_p95_ms", "tpot_p95_ms", "output_tok_s", "gpu_seconds", "e2e_p95_ms" ], "rows": [ { "metric": "ttft_p95_ms", "n": 30, "median_ratio_first_over_second": 0.994327, "p_value": 0.754792, "ci95_ratio": [ 0.972553, 1.020605 ], "significant_at_05": false }, { "metric": "tpot_p95_ms", "n": 30, "median_ratio_first_over_second": 0.999955, "p_value": 0.581361, "ci95_ratio": [ 0.998313, 1.000965 ], "significant_at_05": false }, { "metric": "output_tok_s", "n": 30, "median_ratio_first_over_second": 1.000005, "p_value": 0.263884, "ci95_ratio": [ 0.999958, 1.000149 ], "significant_at_05": false }, { "metric": "gpu_seconds", "n": 0, "insufficient": true }, { "metric": "e2e_p95_ms", "n": 30, "median_ratio_first_over_second": 0.998964, "p_value": 0.125423, "ci95_ratio": [ 0.998224, 1.000229 ], "significant_at_05": false } ], "n_metrics_tested": 4, "n_metrics_significant": 0, "excluded_effect_size_pct_ttft_p95": 2.745, "verdict": "NO DETECTABLE ORDER EFFECT. Across 30 A/A replicates, none of the 4 metrics shows a first-versus-second difference significant at alpha=0.05. In an A/A the arms are identical by construction, so any systematic difference WOULD be an order effect -- there is nothing else it could be -- and none is found. Sprint 1's E3 is answered from data already on disk: the randomised arm order in `paired_sweep` is doing its job, and vLLM's prefix cache being stateful across arms does not measurably leak through a fresh server per replicate. The bound is worth stating precisely rather than as 'no large effect': on ttft_p95 the 95% interval on the first/second ratio is [0.9726, 1.0206] at n=30, so an order effect bigger than about 2.7% is excluded on this metric. Smaller ones are not, and a null is not proof of absence.", "does_not_prove": [ "A null is not proof of absence. This rules out a LARGE order effect at the available n; a small systematic leak would not be detected and this does not bound it. Sprint 1's E3 as specified (a deliberate ABBA/BAAB design) would have more power than pooling incidental randomisation after the fact.", "Fixed-load replays only -- these artifacts carry a per-arm `summary`, not a sweep. The knee is a different statistic and S44p found its spread is dominated by the derivation rather than the measurement, so an order effect could in principle show up in a knee and not here.", "Pools heterogeneous runs (different models, GPUs and nesting levels) to get n. That is legitimate for detecting a SHARED order effect and would dilute one that exists in only a single configuration.", "The randomisation was incidental, not balanced: first-position counts are uneven, so this is an observational test on a randomised process rather than a designed crossover." ], "cost_usd": 0.0, "generated_utc": "2026-08-28T13:15:36.769038+00:00", "git_rev": "17767d3", "git_dirty": true, "run_id": "55a1ef7a5fca4781", "host_kind": "Darwin-arm64", "provenance_schema": "gpu-cert-provenance/1" }