{"id":"327ed6d8-6c17-4b39-ae93-832a5c27e0f2","arxiv_id":"2509.04474","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"N-gram based speculative decoding methods, especially SAM and hybrid SAM[EAGLE-3], achieve strong speedups in LLM test-time scaling by exploiting repetitive reasoning patterns.","lead":"A benchmark that tests which speculative decoding methods best speed up LLM test-time scaling reasoning frameworks like Best-of-N and multi-round thinking. The findings suggest that simple n-gram based methods, which reuse repeated token patterns, are surprisingly effective and that hybrid combinations achieve the best speedups.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'SAM rivals EAGLE-3' claim rests on one DSL-8B T=0 cell where the EAGLE-3 checkpoint is anomalously weak; in every other reported setting EAGLE-3 beats SAM, so the central generalization is not supported.","rationale":"I read the paper in good faith: it is a useful first benchmark, and the hybrid SAM[EAGLE-3] finding is consistent across almost all tables. However, the paper's headline generalization about n-gram methods 'rivaling' training-based methods is not supported by the full set of results. The reader's weakest assumption was about experimental configuration representativeness (30 problems, M=2, N=4, one GPU, no repeats). My concern is different and more internal: the key comparative evidence for SAM over EAGLE-3 comes from a single cell, and the authors themselves provide an alternative explanation for that cell (draft checkpoint quality). This means the central claim is conditional on the EAGLE-3 checkpoint being representative of training-based methods, which the paper does not establish. The reader's CONDITIONAL verdict is therefore appropriate, but the condition should be sharpened: rerun the DSL-8B comparison with a stronger EAGLE-3 drafter before claiming that n-gram methods rival trained drafters. My recommended verdict is unchanged from the reader's CONDITIONAL, because the paper can still be valuable if the comparison is repaired or the claims are correspondingly narrowed.","tokens_in":21608,"tokens_out":10291,"duration_ms":121955,"concrete_test":"Retrain or obtain an EAGLE-3 draft model for DeepSeek-R1-Distill-Llama-8B trained with a comparable 600k samples (matching the QW3-8B checkpoint) and rerun the exact Table 3 DSL-8B T=0 multi-round condition. If EAGLE-3's speedup rises to ~2.6× or above, SAM no longer 'rivals' EAGLE-3 and the abstract's generalization should be revised. As a lightweight proxy before retraining, measure the existing DSL-8B EAGLE-3 checkpoint's speedup on short (~200 token) vs long (>2000 token) outputs; a much higher short-task speedup would confirm the long-generation training deficiency the authors hypothesize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2.1's central claim that SAM 'even rival[s] the training-based method EAGLE-3' rests almost entirely on the DSL-8B, T=0, multi-round cell (Table 3): SAM 2.66× vs EAGLE-3 1.93×. In every other reported comparison EAGLE-3 wins: QW3-8B T=0 (2.91× vs 2.28×), QW3-8B T=0.6 (2.73× vs 1.78×), DSL-8B T=0.6 (1.91× vs 1.69×), BoN DSL/QW3 (1.74×/2.70× vs 1.70×/1.95×), and QW3-4B/14B (Table 5). Moreover, the one favorable cell is the cell the authors themselves flag as anomalous: they attribute DSL-8B EAGLE-3's low speedup and MAT to the third-party draft checkpoint possibly being undertrained for long generations and note the QW3-8B drafter used 600k training samples (Section 4.2). Thus the abstract's 'unique potential' of n-gram methods and the reader's strongest claim are not robustly established; they depend on a single weak drafter rather than a systematic n-gram advantage. The hybrid SAM[EAGLE-3] result is more consistent, but that does not rescue the broader n-gram-vs-trained-drafter claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a benchmark for evaluating speculative decoding methods in the context of test-time scaling (Best-of-N and multi-round thinking). It compares three families of methods — model-based (SpS), training-based (EAGLE-3), and n-gram-based (PLD, REST, Lookahead, PIA, SAM, Recycling), plus a hybrid (SAM[EAGLE-3]) — on 120 problems drawn from AIME24, AIME25, MATH500, and GPQA, using DSL-8B and several Qwen3 models, with MAT and walltime speedup as metrics. The central claims are that n-gram methods, especially SAM, capture repetitive reasoning patterns effectively and can rival or complement training-based draft models; that temperature degrades token n-gram methods but not probabilistic n-gram or trained methods; and that the hybrid SAM[EAGLE-3] achieves the highest overall speedups.","tokens_in":22030,"tokens_out":4358,"duration_ms":50452,"significance":"If the empirical claims were robust, the benchmark would be a useful resource: it spans three speculative-decoding families, includes a hybrid, reports turn-level and module-level time breakdowns, and considers multiple model scales. The paper also has the virtue of testing a concrete prior hypothesis rather than fitting parameters to produce findings. However, the contribution is currently limited by the thinness and specificity of the evidence: 30 problems per dataset (the first 30 for MATH500/GPQA), M=2 and N=4 for the main comparisons, a single GPU, no repeated runs or error bars, no released code or data, and an unspecified Best-of-N verifier. The headline n-gram-versus-trained-drafter claim rests on one cell that the authors themselves flag as an artifact. These issues affect the load-bearing conclusions, not merely the presentation.","major_comments":[{"comment":"The central claim that SAM 'even rival[s] the training-based method EAGLE-3' is supported by only one overall cell: DSL-8B, T=0, multi-round thinking (Table 3: SAM 2.66× vs EAGLE-3 1.93×). In every other reported comparison EAGLE-3 wins: DSL-8B T=0.6 (1.69× vs 1.91×), QW3-8B T=0 (2.28× vs 2.91×), QW3-8B T=0.6 (1.78× vs 2.73×), BoN on DSL-8B (1.70× vs 1.74×), BoN on QW3-8B (1.95× vs 2.70×), and QW3-4B/14B (1.89×/1.70× vs 2.27×/2.23×). Moreover, the one favorable cell is exactly the cell the authors explain in §4.2 as likely reflecting an undertrained or long-generation-degraded EAGLE-3 checkpoint. The abstract's 'unique potential' of n-gram-based methods and the reader-facing generalization are therefore not robustly established; they rest on a single anomalous checkpoint rather than a systematic advantage.","section":"§4.2.1, Tables 3–5"},{"comment":"The benchmark is configured as 120 problems (first 30 of MATH500 and GPQA, all 30 AIME problems), M=2 thinking rounds, N=4 BoN candidates, temperatures {0, 0.6}, and one A6000 GPU with no repeated runs. No variance, confidence intervals, or significance tests are reported. Walltime speedups are noisy measurements; differences such as 1.93× vs 2.66× in Table 3 can plausibly change with hardware state, batch scheduling, or sampling seeds. Since the paper's general claims about 'unique potential' and 'consistent SOTA' are rankings over these measurements, the absence of repeated trials and error bars is a load-bearing gap. Please add multiple seeds/runs and CI, or explicitly restrict the conclusions to the configuration evaluated.","section":"§3.2, §4.1"},{"comment":"The Best-of-N verifier is never specified. Section 3.1.1 lists possible verifiers (heuristic checker, reward model, process verifier) but the experiments do not state which verifier was used, how it was invoked, or whether verifier latency is included in the reported speedup. This matters because BoN speedup is the ratio of total walltime, and the selection stage can dominate for small N. In addition, no code, data subsets, hyperparameter files, or timing scripts are released, which is a serious limitation for a paper whose stated contribution is a benchmark. Please specify the verifier and timing methodology precisely and release the benchmark artifacts.","section":"§3.1.1, §4.1"},{"comment":"The 'progressive acceleration across turns' claim is based on very limited evidence. In multi-round thinking the paper evaluates only two rounds, so the 33–45% increases in Figure 3a are a single turn-to-turn comparison. In BoN, the four trajectories are independent samples, not iterative refinements, so calling the across-candidate speedup 'progressive acceleration' conflates cache reuse in independent samples with multi-turn refinement. The hybrid SAM[EAGLE-3] result is more consistent across settings, but its advantage over EAGLE-3 in Table 5 is small (e.g., 2.25× vs 2.23× for QW3-14B multi-round), which weakens the 'unlocking distinct potential' wording. Please re-analyze or temper these claims.","section":"§4.2.3, Figure 3"}],"minor_comments":[{"comment":"The title page includes 'Work in progress.' This should be removed before submission; it is inconsistent with a benchmark paper claiming comprehensive results.","section":"Title page"},{"comment":"Figure 2 appears garbled in the submitted PDF, with obfuscated/corrupted text in the illustration. The figure needs to be regenerated so that the framework diagram is legible.","section":"Figure 2"},{"comment":"The reference list contains two entries titled 'Lookahead' (Fu et al. 2024 and Zhao et al. 2024). Zhao et al. is cited for PIA, but the listed title says 'Lookahead', which is confusing. Please correct the bibliographic entry for PIA and ensure citations match the intended methods.","section":"§3.3 and References"},{"comment":"The terms 'multi-round thinking' and 'multi-turn thinking' are used interchangeably (e.g., §4.1 vs Figure 3). Please standardize the terminology.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The empirical basis is too thin for the strength of the claims, and the central 'SAM rivals EAGLE-3' assertion depends on a single cell that the authors themselves diagnose as a checkpoint artifact. This is fixable within the manuscript's scope by adding more configurations, repeated trials, error bars, and a clearer statement of what the measurements do and do not support. The lack of code/data and the unspecified BoN verifier are also serious for a benchmark paper. I would be willing to look at a revised version if these load-bearing reproducibility and statistical issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is genuinely the first systematic comparison of speculative decoding methods under test-time scaling, and it produces some plausible qualitative findings: n-gram methods exploit repetition, they show progressive acceleration across turns, they are temperature-sensitive, and the hybrid SAM[EAGLE-3] is consistently strong. Second, the headline claim that SAM 'rivals' EAGLE-3 is not supported by the data. It rests almost entirely on the DSL-8B, T=0 multi-round cell, where EAGLE-3's third-party draft checkpoint is anomalously weak (the authors themselves say it may be undertrained for long generations). In every other reported comparison, EAGLE-3 beats SAM. So the abstract's 'unique potential' of n-gram methods is an overstatement.\n\nWhat the paper does well: it sets up a reasonable protocol across two test-time scaling paradigms, uses four reasoning datasets, and compares nine methods with default settings. The hybrid result is consistent and worth taking seriously. The observation that retrieval-based methods accelerate across turns is new and interesting.\n\nThe soft spots are real. The evidence base is thin: 120 problems (the first 30 per dataset, not random), M=2, N=4, a single GPU, no variance or significance testing. The Best-of-N verifier is never described, which makes it hard to interpret the BoN results. No code or data is released. The paper itself says 'Work in progress,' which is honest, but it doesn't fix the mismatch between the claims and the evidence.\n\nThis paper is for people working on efficient inference for reasoning models. It gives a useful landscape and testable hypotheses, but I would not treat its quantitative comparisons as definitive. It deserves a serious referee, but the authors need to expand the experiments, add uncertainty quantification, specify the verifier, and tone down the claims. Send it to peer review with major revision expectations.","headline":"Useful first benchmark with a plausible qualitative story, but the 'n-gram rivals trained drafters' claim is propped up by one weak EAGLE-3 checkpoint and the evidence base is too thin to support it.","tokens_in":22473,"tokens_out":2134,"would_cite":false,"duration_ms":24768,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark of speculative decoding for LLM test-time scaling comes to a sharp conclusion: training-free n-gram drafting, especially SAM, rivals or beats the trained EAGLE-3 draft model, and a hybrid of the two is fastest.","keywords":["speculative decoding","test-time scaling","Best-of-N sampling","multi-round thinking","n-gram decoding","suffix automaton","EAGLE-3","reasoning models"],"falsifier":"A randomized re-run of the benchmark with several hundred MATH-500 and GPQA problems, six thinking rounds, and eight Best-of-N candidates would settle the claim: if SAM's speedup over EAGLE-3 disappears or reverses in that configuration, the paper's central claim fails.","tokens_in":21565,"feed_emoji":"⚡","tokens_out":5451,"duration_ms":60566,"temperature":0.7,"pith_summary":"This paper tries to establish that the repetitive reasoning traces produced by test-time scaling—Best-of-N sampling and multi-round thinking—can be accelerated just as effectively by lightweight n-gram caches as by expensively trained draft models. If true, practitioners can speed up reasoning-heavy LLM inference without training auxiliary drafters, and can get even more by combining both strategies. The benchmark covers nine speculative-decoding methods on two popular 8B reasoning models and four math/science datasets. Lossless acceleration is maintained throughout, so the speedups do not trade away output distribution quality.","feed_headline":"N-gram drafting rivals trained draft models for LLM test-time scaling","feed_subtitle":"First benchmark of nine drafter methods finds cache-based repetition matches EAGLE-3; hybrid wins overall.","key_machinery":"The central object is SAM's suffix automaton: a data structure built from tokens the model has already produced that finds, in amortized constant time per step, the longest suffix of the current context that appeared earlier, and returns the tokens that followed it as a speculative draft. EAGLE-3 contributes a trained draft head, and the hybrid SAM[EAGLE-3] switches between them, using EAGLE-3 when SAM's matched suffix is too short to be useful. The paper's empirical core is the accept-length-versus-suffix-length curve: matched suffixes beyond about ten tokens give SAM accept lengths nearly double EAGLE-3's, which is what makes repetition caching the load-bearing acceleration mechanism.","core_discovery":"The paper reports the first controlled comparison of nine speculative-decoding methods embedded inside two test-time scaling loops: Best-of-N sampling and multi-round thinking. Its central quantitative finding is that cache-based n-gram drafting—especially SAM, which reuses previously generated token suffixes—produces end-to-end speedups comparable to or better than the trained EAGLE-3 drafter in the tested configurations. For example, on DeepSeek-R1-Distill-Llama-8B under greedy multi-round thinking, SAM reaches 2.66x versus EAGLE-3's 1.93x, while the hybrid SAM[EAGLE-3] reaches 3.97x. The paper also documents that token n-gram gains degrade when sampling temperature rises, whereas the prob","pith_inferences":["Extrapolating the turn-level trend, increasing the number of thinking rounds beyond two should widen SAM's advantage over EAGLE-3; this is testable and goes beyond what the paper measured.","The temperature sensitivity points toward an inexpensive adaptive policy: monitor generation entropy or repetition rate and switch between SAM and EAGLE-3, or between SAM and Recycling, to preserve speedup under stochastic sampling.","Because the benchmark fixes Best-of-N at only four candidates, real deployments using dozens of candidates would produce far more cross-trajectory repetition, likely amplifying n-gram gains beyond the paper's measurements.","The paper measures speed, not final task accuracy; a direct extension is to verify that the lossless-distribution guarantee also preserves answer selection when Best-of-N voting or verifiers amplify small distribution shifts."],"forward_implications":["SAM matches or beats EAGLE-3 on a reasoning model in greedy multi-round thinking, so training-free n-gram drafting is a viable drop-in accelerator for repetition-heavy reasoning.","SAM[EAGLE-3] is the fastest method in almost every tested setting, so combining semantic alignment with repetition caching is more useful than either approach alone.","SAM and PIA speed up on later turns by reusing prior computations, meaning retrieval-based acceleration compounds as test-time scaling loops get longer.","N-gram speedups fall when sampling temperature rises, so sampling-heavy test-time scaling needs temperature-robust methods such as Recycling or hybrids to keep the gains.","SpS's high token acceptance does not translate into large wall-clock speedup when the draft model is large relative to the target, showing that draft overhead matters as much as acceptance rate."],"supporting_citations":[{"why":"Defines Best-of-N sampling, one of the two test-time scaling frameworks that the benchmark accelerates.","marker":"Snell et al., 2025"},{"why":"Defines multi-round thinking, the other test-time scaling framework that the benchmark accelerates.","marker":"Tian et al., 2025"},{"why":"Introduces speculative sampling, which serves as the model-based baseline SpS.","marker":"Chen et al., 2023"},{"why":"Introduces EAGLE-3, the training-based draft method that n-gram methods are compared against.","marker":"Li et al., 2025"},{"why":"Introduces SAM and its suffix-automaton drafting mechanism, the central n-gram method.","marker":"Hu et al., 2025"},{"why":"Introduces REST, the retrieval-datastore n-gram baseline evaluated in the benchmark.","marker":"He et al., 2024"},{"why":"Introduces Recycling, the probabilistic n-gram baseline that caches top-k next-token logits.","marker":"Luo et al., 2025"},{"why":"Introduces PIA, the trie-based n-gram baseline evaluated in the benchmark.","marker":"Zhao et al., 2024"},{"why":"Introduces Lookahead, the parallel n-gram candidate baseline evaluated in the benchmark.","marker":"Fu et al., 2024"},{"why":"Supplies the evaluation pipeline used to compare speculative decoding methods.","marker":"Xia et al., 2024"}],"fun_headline_variants":["N-gram drafter outruns EAGLE-3 in test-time scaling","Hybrid n-gram + model drafting gives 3.97x speedup","New benchmark: n-gram drafting rivals trained drafters","Cache-based drafting matches EAGLE-3 for LLM scaling","SAM n-gram beats EAGLE-3 on repetitive reasoning tasks"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The general conclusions assume the fixed benchmark configuration—the first 30 problems of each dataset, two thinking rounds, four Best-of-N candidates, temperatures 0 and 0.6, and a single GPU run per method—stands in for real test-time scaling workloads.","fun_headline_variants_meta":{"raw":{"variants":["N-gram drafter outruns EAGLE-3 in test-time scaling","Hybrid n-gram + model drafting gives 3.97x speedup","New benchmark: n-gram drafting rivals trained drafters","Cache-based drafting matches EAGLE-3 for LLM scaling","SAM n-gram beats EAGLE-3 on repetitive reasoning tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1346,"prompt_tokens":783,"completion_tokens":563,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":466}},"tokens_in":527,"tokens_out":563,"duration_ms":6433,"temperature":1.0,"reasoning_tokens":466,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:43:31.500074+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A randomized re-run of the benchmark with several hundred MATH-500 and GPQA problems, six thinking rounds, and eight Best-of-N candidates would settle the claim: if SAM's speedup over EAGLE-3 disappears or reverses in that configuration, the paper's central claim fails.","supporting_citations":[],"review_version":1}