{"id":"7f1b9fa9-9fff-4102-b61f-ebcefc2609a3","arxiv_id":"2607.21535","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Restricting the built-in draft head's attention to a 4K sink-plus-recent window cuts long-context speculative-decoding step cost by 28-44% without changing the target model's output distribution.","lead":"A draft model that predicts several tokens at once normally reads the full million-token history at every step, making long-context speculative decoding slow. This paper shows that restricting only that draft's attention to a small recent window—while the main model still checks every token—cuts per-step cost by 28-44% at 1M tokens and reclaims 7.7-11% of the KV cache.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 7 itself contradicts the t_verify-identical assumption: Nemotron's speculative-bookkeeping term drops 1.2 ms across arms, so part of the headline cost win may be misattributed to windowing.","rationale":"The reader's weakest_assumption already identifies the decomposition and the t_verify-identical requirement as the shaky point; my reading confirms this and sharpens it with a concrete internal inconsistency: Table 7's own numbers show Nemotron's tovh_verify varying by 1.2 ms between arms, which the text glosses over as 'essentially identical.' This matters because the causal story is that windowing removes the draft phase's O(S) read, and the headline margin is presented as the cost-side win. If part of that margin is actually an acceptance-dependent bookkeeping effect, the attribution is not clean. The paper otherwise has strong support: the mechanism is lossless by construction, the end-to-end TPOT speedups are direct measurements across many cells, the backend ablation controls for FlashInfer-vs-Triton, and the reproduction package is a real asset. The concern is addressable by an AL-matched profiler comparison and does not overturn the overall contribution; it only asks for a tighter attribution in the cost model. Hence the reader's CONDITIONAL verdict remains appropriate, and no further verdict change is needed.","tokens_in":23707,"tokens_out":9861,"duration_ms":103504,"concrete_test":"Instrument SGLang's nsys trace to bin decode iterations by accepted length AL for both native and windowed Nemotron at d=7, and compare tovh_verify (tree build + index/sampling) within AL-matched bins. If tovh_verify is flat (±0.2 ms) at matched AL, the 5.8→4.6 drop is just acceptance composition and the current attribution is fine; if it still differs by >0.5 ms at matched AL, the decomposition assumption fails and Table 1's 'entire saving is in t_draft' statement must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's cost-side attribution (Eq. 1; App. B) rests on the decomposition t_step = t_verify + t_ctx_draft + γ·t_fwd_draft, with t_verify identical across native and windowed, so the entire saving can be assigned to the draft phase. Table 7 (nsys phase decomposition) violates this: for Nemotron, tovh_verify (tree build/index/sampling) is 5.8 ms native vs 4.6 ms windowed, a 1.2 ms drop outside t_draft, in the same run where windowed acceptance is lower (AL 3.75→3.61). The paper calls tovh_verify 'fixed speculative bookkeeping' and asserts all terms are acceptance-independent, but the observed drop tracks the acceptance change, not the window. If part of the measured win is an acceptance/bookkeeping artifact rather than a reduction in the draft's full-context KV read, the causal claim — that windowing specifically removes the draft-attention tax — is not fully supported. The magnitude matters: 1.2 ms is ~16% of Nemotron's 7.3 ms per-step saving and ~4% of the step time, so the headline +28% could be misattributed by several points, even though the end-to-end TPOT speedup (Table 13) is a direct measurement and stands.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a long-context 'draft-attention tax' in built-in Multi-Token-Prediction (MTP/NEXTN) heads: at million-token context the draft's full-context attention read dominates the decode step, eroding or inverting speculative-decoding speedups. It proposes Windowed-MTP, which restricts only the draft's attention to a StreamingLLM-style sink plus a sliding window while leaving the target's full-attention verification intact. The paper argues this is training-free, drop-in, and lossless by construction, and reports per-decode-step cost reductions of +28% to +44% over the native MTP draft at 1M context across three architecture families (Qwen3.6-35B GDN-MoE, Qwen3.5-122B GDN-MoE, and a Mamba2-hybrid NoPE 120B), with acceptance preserved within confidence intervals, end-to-end speedups over native MTP of +4% to +53%, and reclamation of 7.7-11% of total KV as dead draft KV. The evidence includes an analytic latency model (Eq. 1-3), phase-decomposed kernel timings (App. B), a backend ablation, a best-depth comparison, batch/TP Pareto sweeps, and a reproducibility package.","tokens_in":23949,"tokens_out":5059,"duration_ms":56080,"significance":"If the central claims hold, this is a practically important systems contribution. It directly attacks a real bottleneck for long-context serving with built-in speculative heads, and the intervention is simple enough to be adopted in production. The losslessness logic is a standard speculative-decoding theorem and is correctly stated. The paper's strengths are the direct phase-decomposed measurements (App. B), the backend ablation showing Triton is a handicap rather than the source of the win (Table 5), the explicit separation of durable per-forward slope reduction from implementation-dependent index-build costs (Table 6 and §7), and the release of a reproduction package. The claimed 'input-invariant' cost-side margin, if confirmed, would be a durable hardware-level effect. The main weakness is that the exact decomposition used to attribute the entire saving to the draft phase is undermined by a violation of the 't_verify identical' assumption in one model, and the headline cells rest on single runs.","major_comments":[{"comment":"The claim that t_verify is identical across native and windowed is contradicted by the Nemotron row of Table 7: tovh_verify ('fixed speculative bookkeeping') is 5.8 ms native vs 4.6 ms windowed, a 1.2 ms drop inside t_verify, in the same cell where acceptance length drops from 3.75 to 3.61. The paper asserts these terms are acceptance-independent, but the observed drop tracks the acceptance change, not the window. Because the intercept drop Δc in Table 6 is attributed entirely to removal of the draft's O(S) index build (Δc = Δt_ctx_draft), and the headline saving Δt_draft in Table 1 includes that intercept drop, part of the reported +28.3% Nemotron margin may be an acceptance/bookkeeping artifact rather than a reduction in the draft's full-context KV read. Please trace the specific kernels inside tovh_verify that change, report them as a function of accepted length, and recompute the cos","section":"App. B, Table 7; Eq. (1)"},{"comment":"The headline percentages — +28% to +44% per-step cost reduction, +4% to +53% end-to-end win, and the 'never regresses' claim — are supported by a single seeded run per cell. The paper does report within-run aggregation over many decode steps, but the central cross-cell claims (e.g., q35 NIAH-mv win/nat of 1.04×, q122 at 2K in Table 10 net 0.97×) are small enough that single runs cannot distinguish a real effect from run-to-run noise. Given the strength of the 'input-invariant' and 'never regresses' language, please provide repeat runs (at least 3) or bootstrap confidence intervals for Tables 1, 2, and 13, and for the best-depth Table 3 where margins are as small as 1.22×. This is a load-bearing point for the quantitative claims, not just a presentation issue.","section":"§6, Tables 1/2/13; App. A 'Statistical protocol'"},{"comment":"The paper claims an 'all-cell greedy output diff showing no divergence beyond the verifier's pre-existing bf16 non-determinism' as an empirical confirmation of losslessness, but no diff results are reported anywhere in the paper or appendices: there is no table of token-level agreement counts, no false-positive rate, and no comparison of native-vs-dense divergence against windowed-vs-native divergence. The distributional losslessness is a theorem and does not need this support, but the empirical claim of 'no added divergence' is unsupported as written. Please add a table with per-cell divergence counts (number of differing tokens, number of cells) and, if possible, a direct comparison of native-vs-dense versus windowed-vs-native divergence to substantiate the 'same verify-noise envelope' statement.","section":"§4, §1 contribution 2"}],"minor_comments":[{"comment":"Eq. (3) defines win/native = (step_native/step_window)·(AL_window/AL_native), but Table 1 labels its last column 'win/nat' as the matched-acceptance ratio. Please clarify in the caption that Table 1 reports the acceptance-cancelled cost-side ratio and Table 2 reports the full end-to-end ratio including acceptance changes.","section":"Eq. (3) and Table 1"},{"comment":"The fitted slopes and intercepts in Table 6 are reported with R² but without standard errors. Given the attribution hinges on Δc ≈ Δt_ctx_draft, reporting standard errors or confidence intervals would strengthen the decomposition and would also make the 'input-invariant' claim more falsifiable.","section":"App. B, Table 6"},{"comment":"Table 13 uses abbreviations (NIAH-s/mv/mq, VT, CWE, FWE, Code-QA) without expansion in the caption. The reader can infer from the text, but expanding them in the caption would improve readability.","section":"§6, Table 13"},{"comment":"The pseudocode for the windowing change is helpful, but the ring-buffer slot mapping is only described in a comment ('slot = ring_base(r) + logical_pos'). A one-sentence explanation of how the ring wraps and how the +d slack is used would make the implementation claim easier to verify.","section":"App. A, pseudocode"}],"recommendation":"major_revision","confidential_remarks":"The reader's skeptical concern about Table 7 is valid and lands. The paper is a strong systems contribution with a sound central idea and excellent measurement detail, but the attribution of the entire saving to the draft phase needs to be corrected or explicitly hedged. The single-run nature of the headline numbers also needs addressing. I do not see a reason for rejection: the end-to-end TPOT speedups and the per-forward slope reductions are direct measurements, and the issues can be fixed within the manuscript's scope with additional analysis and modest additional runs. The citation and novelty landscape appears appropriately handled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is worth reading if you care about long-context speculative decoding. It makes a simple move — apply a StreamingLLM-style window to the draft attention only, keep the verifier full-attention — and reports 28–44% per-step cost reduction at 1M context across three model families, with a reproduction package. The contribution is real: prior long-context SD work used separate or trained drafts, while this targets the already-shipped MTP/NEXTN head, is drop-in and training-free, and adds a ring buffer to reclaim draft KV (7.7–11% of total memory).\n\nThe experimental work is genuinely careful: phase decomposition, a backend ablation showing Triton is a handicap rather than the source of the win, H100 replication, batch/Pareto sweeps, and a decision-invariance probe. Losslessness is correctly argued as a standard property of speculative decoding. The per-position acceptance analysis is a nice addition — it shows the window preserves the shape of acceptance rather than truncating it.\n\nThe main soft spot is the cost attribution. The paper assumes t_verify is identical across native and windowed, but Table 7 shows Nemotron's tovh_verify drops 1.2 ms (5.8→4.6) in the same run where acceptance drops (AL 3.75→3.61). The paper labels this \"fixed speculative bookkeeping,\" but it looks acceptance-dependent. That means part of the headline 7.3 ms saving for Nemotron may be misattributed to windowing; the durable draft-phase reduction could be a few points smaller. This is not fatal — the end-to-end TPOT speedups are direct measurements, and the per-forward slope reduction in Table 6 is a hardware-level effect — but the decomposition should be tightened, ideally with a matched-acceptance control or an explicit model of bookkeeping cost.\n\nOther, minor concerns: headline percentages are single-run without run-to-run CIs, and the §5 claim that acceptance tracks native \"within Wilson intervals at essentially every position\" is strained by the Qwen-122B α2 drop in Table 9. The paper itself flags the index-build term as implementation-dependent, which is honest and bounds the headline claim.\n\nWho is this for: systems people doing serving or speculative decoding with MTP/NEXTN heads. It deserves a serious referee. I would send it out, with major-but-addressable revisions: fix the t_verify decomposition, add multi-run CIs, and soften the acceptance-invariance wording.\n\nRecommendation: engage with it, and tell the authors to look at the Nemotron bookkeeping term.","headline":"A solid, well-measured systems paper on windowing the built-in MTP draft; the central result holds, but the cost decomposition has a small leak.","tokens_in":24537,"tokens_out":2455,"would_cite":true,"duration_ms":25180,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Windowing the draft's attention cuts long-context speculation cost up to 44%","keywords":["speculative decoding","multi-token prediction","draft attention","attention sink","sliding window","KV cache","long context","lossless acceleration"],"falsifier":"Take a needle-in-haystack task at 1M context with the needle placed just outside the 4K recent window (and not in the sink), and compare acceptance length and end-to-end latency of Windowed-MTP versus the native full-context draft. If the acceptance drop exceeds the cost-side saving, the central claim is refuted.","tokens_in":23505,"feed_emoji":"⚡","tokens_out":6927,"duration_ms":59723,"temperature":0.7,"pith_summary":"At million-token context, the built-in draft head that makes speculative decoding cheap is itself the bottleneck: it runs full attention over the entire KV cache at every draft step, so its cost grows linearly with context and can make speculation slower than no speculation. The paper's claim is that this 'draft-KV tax' can be removed by windowing the draft's attention only—a fixed attention sink plus a sliding window of recent tokens—while leaving the full-attention target verification untouched. This is lossless by construction: the target decides every accepted token, so windowing changes which tokens are proposed, never which are accepted. Across three architecture families at 1M context the per-decode-step cost falls by 28–44% with acceptance preserved, and the unread draft KV is reclaimed as a compact ring buffer.","feed_headline":"Windowing the draft's attention cuts long-context speculation cost up to 44%","feed_subtitle":"The full-context target still verifies every token, so the speedup is lossless—only proposals change, never acceptances.","key_machinery":"The central mechanism is the *draft-attention window*: a StreamingLLM-style attention sink plus sliding window applied only to the draft head's KV read, not to the target. It is implemented by truncating the draft's per-request KV index to the first `n_sink` tokens plus the last `W` tokens; because RoPE positions are baked into cached keys, this does not disturb scores. The argument is carried by a per-decode-step cost identity, t_step = t_verify + t_ctx_draft + γ·t_fwd_draft, where the two O(S) draft-phase terms shrink to O(W) under windowing while the verify term is untouched. The losslessness argument is the speculative-decoding acceptance rule: the full-context target distribution decide","core_discovery":"The central discovery is that a built-in Multi-Token-Prediction (MTP) draft head—a small extra head that predicts several future tokens in one pass—pays an O(context-length) attention read per draft step, so at million-token context the draft phase adds roughly 92–138% on top of the bare verification cost, nearly doubling the decode step and, on hard low-acceptance tasks, pushing deep native drafts below the no-speculation baseline. The paper shows that restricting the draft's attention to a fixed sink plus the most recent W tokens (4K in practice) eliminates this tax while preserving the target's verified output distribution exactly in exact arithmetic. Windowing changes only the proposal d","pith_inferences":["Inference beyond the paper: the same windowing should transfer to trained draft heads (e.g., EAGLE-style) and diffusion drafters, which pay the same O(S) draft-attention tax; the paper itself flags this as natural future work.","Inference beyond the paper: because the windowed draft never reads intermediate draft KV, the draft's prefill pass can be pruned to O(W+sink), which would lower time-to-first-token on long prompts—an extension the paper leaves to future work.","Inference beyond the paper: an adaptive draft-depth controller could tune depth per input cheaply, since the windowed draft's cost no longer rises steeply with depth; the paper's best-depth sweep already hints that shallower depths often suffice.","Inference beyond the paper: the measured margin is framework-relative; on a leaner serving runtime that lacks a per-step O(S) index build, the headline 28–44% would shrink toward the durable per-forward attention-slope reduction (22–40% in the paper's own fit)."],"forward_implications":["At 1M context the draft's KV working set is bounded by a constant window (~4K), so per-step speculation cost no longer scales with context length—the saving widens as the prompt grows.","The target's verified output distribution is unchanged (greedy-exact in exact arithmetic; bf16 verifier noise affects native and windowed drafts identically).","End-to-end latency improves by the same cost-side margin at matched acceptance, and by more where windowing also lifts acceptance; hard tasks where native MTP dropped below dense are restored to net-positive speedup.","The draft KV pool, measured at 7.7–11% of total KV, can be compacted into a ring buffer, freeing memory for extra concurrent requests or deeper drafts at the same budget.","Windowing composes with tree-shaped speculation and with KV quantization (fp8), and is training-free and drop-in, so it can be gated on a context-length threshold."],"fun_headline_variants":["Windowed MTP draft cuts 1M-context speculation cost 44%","Slash draft KV tax up to 44% at million-token context","MTP draft: bound the KV window, keep verified output","Windowing draft attention: 99% fewer KV entries, 44% faster","Remove full-context draft-KV tax with a sliding window"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the windowed draft's proposals are accepted by the full-context target often enough that the per-step cost saving is never outweighed by a change in acceptance length.","fun_headline_variants_meta":{"raw":{"variants":["Windowed MTP draft cuts 1M-context speculation cost 44%","Slash draft KV tax up to 44% at million-token context","MTP draft: bound the KV window, keep verified output","Windowing draft attention: 99% fewer KV entries, 44% faster","Remove full-context draft-KV tax with a sliding window"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00069,"raw_usage":{"total_tokens":3054,"prompt_tokens":926,"completion_tokens":2128,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":2031}},"tokens_in":670,"tokens_out":2128,"duration_ms":14098,"temperature":1.0,"reasoning_tokens":2031,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T07:08:15.721930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a needle-in-haystack task at 1M context with the needle placed just outside the 4K recent window (and not in the sink), and compare acceptance length and end-to-end latency of Windowed-MTP versus the native full-context draft. If the acceptance drop exceeds the cost-side saving, the central claim is refuted.","supporting_citations":[],"review_version":1}