{"id":"94111916-6495-4f16-ace4-2d1829013be2","arxiv_id":"2505.07858","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Speculative decoding acceptance rate and throughput are described by empirical log-linear scaling laws in pretraining tokens, draft depth, and batch size, yielding the Scylla recipe for faster LLM inference.","lead":"The paper fits log-linear scaling laws for how a speculative decoding draft model's acceptance rate grows with pretraining tokens, decoder count, and batch size, and builds a system called Scylla that uses these trends to speed up LLM decoding. It reports 1.5-2.2 percentage-point higher acceptance rates than EAGLE2 and up to 2x higher throughput, but the laws are empirical curve fits with weak validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stated constants in Theorems 1.1-1.3 do not match the paper's own Figure 1 and Table 5 data, so the quantitative scaling laws are not established.","rationale":"The reader's REJECT verdict is supported. The reader identified the roofline/optimal-topk validation as the weakest assumption; that is a genuine external-validation concern. My stress-test found a more direct, internal problem: the constants in Theorems 1.1-1.3 are not consistent with the very points the paper plots as validation. Before asking whether Eq. 7's I=Icrit criterion holds on real hardware, the equations should reproduce the displayed evidence, and they do not. This makes the central claim (three quantitative scaling laws with specific coefficients) unreliable; the qualitative directions (more pretraining, more decoders, larger batches help) remain plausible and are not what I contest. I am not treating the absence of released code or the EAGLE3 comparison as the primary issue, although those also weaken the paper. The proposed test is a simple arithmetic re-fit, so it is cheap and decisive. Because the verdict was already REJECT and this concern only adds support, no verdict change is needed; hence UNCHANGED.","tokens_in":17257,"tokens_out":10031,"duration_ms":97350,"concrete_test":"Re-run the three log-linear regressions from the exact (x,y) coordinates reported in Figure 1 and Table 5 (seven pretrain-token points, four decoder-count points, and the throughput-vs-batch points), reporting the estimated slope/intercept and 95% confidence intervals. If the intervals exclude the stated Theorem 1.1-1.3 constants, or if the point estimates differ by more than 20%, the laws as written do not describe the paper's own data and the supporting figures need correction or the theorems need re-derivation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims are not recoverable from the paper's own validation plots. Theorem 1.1 states acceptance = 0.08 log10(T_pretrain) + 5.05. Using the seven pretrain-token points in Figure 1 (left) and the MT-Bench values in the caption/text (x = 1,2,5,10,20,50,100 B; y = 5.13,5.15,5.29,5.33,5.30,5.43,5.40), an ordinary least squares fit gives slope ~0.15 and intercept ~5.00, not (0.08, 5.05). Theorem 1.2 states acceptance = 0.74 log10(D) + 4.61, but the four points in Figure 1 (middle) (D = 1,2,5,10; y = 5.03,5.24,5.65,5.83) fit slope ~0.83 and intercept ~5.02. These discrepancies are not unit artifacts: log10 of a rescaled x changes only the intercept, not the slope, and the slope discrepancy is about 2x. Theorem 1.3 states throughput = 286.79 log2(b) + 7.54; this predicts ~7.5 tokens/s at b=1 while Table 5 reports Scylla+Opt.topk at 250 tokens/s, and it predicts ~1728 tokens/s at b=64, which is below Table 5's 2150 and far above the theoretical scale in Figure 1 (right). If the displayed points are the evidence, the fitted constants differ substantially from the theorem statements, and the paper provides no alternative raw data or fitting procedure that would recover (0.08, 5.05), (0.74, 4.61), or (286.79, 7.54). Since Scylla's design choices (10B pretraining, decoder scaling, batch-adaptive topk) are selected from these equations, the quantitative part of the central claim is unsupported even before considering external hardware validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes three log-linear scaling laws for speculative decoding: acceptance rate as a function of pretraining-token volume (Theorem 1.1), acceptance rate as a function of draft-model decoder count (Theorem 1.2), and decoding throughput/optimal top-k path as functions of batch size (Theorem 1.3). On the basis of these laws, the authors build Scylla, a draft-model training and inference configuration that scales pretraining data, decoder depth, and batch-adaptive top-k selection. The paper reports that Scylla achieves higher acceptance rates than EAGLE2/EAGLE3 on several benchmarks and about 2x throughput over EAGLE2 on an industrial inference engine.","tokens_in":17738,"tokens_out":5433,"duration_ms":50388,"significance":"If the quantitative scaling laws were established, the paper would provide actionable guidance for draft-model pretraining budgets, decoder-capacity choices, and batch-dependent tree sizes in speculative decoding, and the Scylla system would be a useful demonstration. The empirical study covers multiple backbones and benchmarks, and the industrial throughput comparison is a useful data point. However, the central quantitative claims are not recoverable from the paper's own figures and tables: the stated constants in Theorems 1.1 and 1.2 do not match the displayed data points, and Theorem 1.3's throughput equation contradicts the measured throughput in Table 5. The validation of the optimal-top-k formula is circular because it is derived from a roofline model and then 'validated' against that same model's output. Since the scaling laws are the load-bearing contribution, these problems substantially weaken the paper's main claim.","major_comments":[{"comment":"The stated constants α≈0.08 and β≈5.05 are not supported by the paper's own data. Using the seven points visible in Figure 1 (left), which are also given in §4.1 (x = 1, 2, 5, 10, 20, 50, 100 B; y = 5.13, 5.15, 5.29, 5.33, 5.30, 5.43, 5.40), an ordinary least-squares fit yields a slope of roughly 0.15 and an intercept of roughly 5.00, not (0.08, 5.05). The discrepancy is not a unit artifact: rescaling the x-axis by a constant changes only the intercept, while the slope is off by nearly a factor of two. The paper provides no fitting procedure, raw data, or repeated-runs summary that would recover the stated coefficients, so Theorem 1.1 as written is not established by the displayed evidence.","section":"§3.1, Theorem 1.1, Figure 1 (left)"},{"comment":"The four points in Figure 1 (middle) (D = 1, 2, 5, 10; y = 5.03, 5.24, 5.65, 5.83) fit a log-linear line with slope approximately 0.83 and intercept approximately 5.02, whereas Theorem 1.2 states α≈0.74 and β≈4.61. With only four points and no error bars, an R² of 0.99 is not strong evidence of a precise law. If these are the data behind the theorem, the stated constants are not recoverable; if other data were used, that data and the fitting procedure must be reported.","section":"§3.1, Theorem 1.2, Figure 1 (middle)"},{"comment":"Equation (3) predicts roughly 7.5 tokens/s at b = 1 and 1728 tokens/s at b = 64, while Table 5 reports Scylla + Opt.topk at 250 and 2150 tokens/s, respectively. The equation therefore cannot be a fit to the measured system throughput; it must be a fit to the theoretical roofline curves of Figure 5. The paper should state this distinction explicitly and provide a direct measured-versus-predicted comparison. As written, the claim that Theorem 1.3 is 'validated' by the regression is misleading, because the displayed throughput values in Table 5 do not match the theorem.","section":"§3.2, §4.3, Theorem 1.3, Table 5"},{"comment":"The optimal-top-k formula in Eq. (4) is derived by imposing I(b, topk,opt) = I_crit, and the claimed validation of Eq. (4) is performed by fitting the same roofline model's theoretical curves (Figure 1 right and Figure 5). This is circular: the model is used to generate the data that then confirms the model. There is no independent hardware measurement of the optimal top-k value for Qwen2.5-72B across batch sizes. Without such a measurement, Theorem 1.3's top-k law is not empirically validated.","section":"§3.2, Eq. (7), Figure 1 (right)"},{"comment":"The three statements are called 'Theorems' but are empirical curve fits with no derivation, no standard errors, and no model-selection analysis. The pretraining effect itself is small: the acceptance rate changes from 5.13% to 5.43% over two orders of magnitude of tokens, and the paper reports no repeated-run variability or error bars. Since the design decisions behind Scylla (10B pretraining, decoder scaling, batch-adaptive top-k) are justified by these quantitative laws, the lack of a reproducible fitting protocol is a load-bearing gap.","section":"§3, Theorems 1.1–1.3"}],"minor_comments":[{"comment":"'1.5-2.2 higher acceptance rate' is ambiguous: the reported numbers are 1.47–2.22 percentage-point improvements over EAGLE2, not a 1.5–2.2x multiplicative increase. Please state which quantity is meant.","section":"Abstract and §4.4"},{"comment":"The right panel uses a linear batch-size axis while the caption says 'log scale'; clarify the axis transformation. Also specify whether the points in each panel are single-benchmark values, means over benchmarks, or different from the values quoted in the text.","section":"Figure 1"},{"comment":"Equation (8) introduces an extra parameter k described only as 'ranging from 0.9 to 1.2'. It should be stated how k is determined, how many configurations are used, and whether the fitted relationship is identified rather than an arbitrary parametric form.","section":"Appendix E, Eq. (8)"},{"comment":"The text says observed throughput discrepancies arise from communication overhead and kernel utilization but still uses Table 5 to support the scaling law. Provide a side-by-side comparison of theoretical and measured throughput for the same batch sizes so the reader can judge the size of the discrepancy.","section":"§4.4, Table 5"},{"comment":"The quantitative claims depend entirely on fits, but no raw data, fitting scripts, or code are provided. The statement in §6 that the scaling laws are 'verified only in Transformer-based Scylla' should also be reflected in the abstract and conclusion, where the laws are presented as general.","section":"§6 and 'Code will be released later'"}],"recommendation":"reject","confidential_remarks":"The paper's core quantitative claims are not supported by its own presented evidence: the fitted constants in Theorems 1.1 and 1.2 do not match the displayed points, and Theorem 1.3's throughput equation contradicts the measured numbers in Table 5 while its top-k law is validated only against the model from which it was derived. These are not presentation issues; the scaling-law framing is the paper's main contribution. The Scylla system itself shows consistent gains over EAGLE2 across benchmarks, so a revised submission focused on the system with corrected accuracy claims and an honest assessment of the empirical trends might be viable, but the current manuscript does not meet the bar for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll be direct: the paper's real contribution is the qualitative finding that pretraining tokens, draft depth, and batch size all matter for speculative decoding, and the Scylla recipe built on that intuition looks genuinely effective. The problem is that the three 'scaling laws' stated as theorems are not supported by the evidence in the paper.\n\nThe stress-test note is right. The constants in Theorems 1.1 and 1.2 don't match the points in Figure 1. If you take the seven pretrain-token points shown, an ordinary least squares fit gives a slope around 0.15, not 0.08. The decoder-capacity points give a slope around 0.83, not 0.74. These are not unit artifacts. And Theorem 1.3 is worse: the claimed log-linear throughput fit predicts roughly 7.5 tokens/s at batch 1, while Table 5 reports 250 for exactly the Scylla+Opt.topk configuration the theorem is supposed to describe. That's not a minor calibration issue; the batch-size law is disconnected from the paper's own measurements.\n\nWhat's good: the study is broad — Vicuna, Llama 2/3, Qwen, six benchmarks — and the acceptance-rate improvements over EAGLE2 are consistent and large. The roofline intuition that optimal tree width should shrink with batch size is plausible and worth testing. But the regressions are circular: the parameters are fit to the same data that are then called \"validation,\" there are no error bars, and no code or models are released. The EAGLE3 comparison is also muddier than the abstract suggests: the claimed 0.3-point gain comes from Scylla+8SFT, not Scylla, and EAGLE3 results are only shown for Vicuna-13B, so the comparison is incomplete.\n\nThe honest takeaway is that the qualitative directions are likely right, but the specific formula constants should not be used by anyone. This paper deserves a serious referee because the problem is relevant and the empirical data, if cleaned up and supplemented with release code, could be a useful contribution. But it needs major revision: report fits with uncertainty, validate on held-out data or real hardware, fix or retract the overclaimed theorems, and clarify the EAGLE3 numbers. I would not cite the laws in their current form, but I would read a revised version.","headline":"Useful qualitative recipe buried under quantitative scaling laws that the paper's own figures don't support.","tokens_in":752,"tokens_out":741,"would_cite":false,"duration_ms":32320,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims speculative decoding efficiency follows three log-linear scaling laws—over pretraining tokens, draft decoder count, and batch size—and that applying them yields up to 2x throughput.","keywords":["speculative decoding","scaling laws","acceptance rate","draft model","roofline model","TopK tree","batch size","throughput"],"falsifier":"Run Qwen2.5-72B on the same H800 setup at batch sizes 1, 4, 16, 32, and 64 and measure throughput for each TopK value from 10 to 80; if the argmax TopK deviates systematically from 27904√(1+0.034/b)−27897, for instance if the optimal width stays flat or falls with batch size, Theorem 1.3 is falsified. Independently, re-run the pretrain-token sweep on a single fixed corpus; an acceptance-rate slope far from 0.08 per decade, or a downturn past 50B tokens, would falsify Theorem 1.1.","tokens_in":17005,"feed_emoji":"⚡","tokens_out":8894,"duration_ms":77861,"temperature":0.7,"pith_summary":"The paper tries to establish that speculative decoding—generating draft tokens with a small model and verifying them in parallel with the large model—is governed by simple log-linear scaling laws, just as LLM pretraining loss is. Specifically, the draft's acceptance rate grows logarithmically with pretraining tokens (α≈0.08, β≈5.05) and with draft decoder count (α≈0.74, β≈4.61), while throughput grows logarithmically with batch size only when the number of parallel TopK verification paths is chosen optimally. The paper derives an optimal-TopK formula topk(b)=27904√(1+0.034/b)−27897 from a roofline model that balances compute and memory traffic. If these laws are right, draft-model training budgets, decoder depth, and batch-adaptive tree sizing can all be set by formula instead of by search. The authors combine the three laws into a system called Scylla and report acceptance rates 1.5–2.2 points higher than a strong baseline and roughly 2x decoding throughput in deployment.","feed_headline":"Speculative decoding obeys log-linear scaling laws","feed_subtitle":"Pretrain tokens, draft depth, and batch size each follow a simple formula; applying them roughly doubles throughput.","key_machinery":"The mechanism that carries the argument is a pair of empirical laws plus a roofline model. The pretrain and capacity laws are log-linear regressions of measured acceptance rate against log10 of pretraining tokens and decoder count, fitted over seven token budgets (1B–100B) and decoder depths (1, 2, 5, 10). The batch-size law comes from a roofline analysis: the paper hand-counts FLOPs and memory reads/writes for every component of a Scylla forward pass (draft FC layer, QKV projections, self-attention, MLP, LM head), forms the arithmetic intensity I(b,topk)=W_comp/W_mem, and asserts that maximum throughput occurs exactly where I equals the hardware critical intensity I_crit=P_peak/B_mem. Solving that equality for Qwen2.5-72B with 10K prefill yields the optimal-TopK formula. The roofline model is what turns the batch-size law from empirical curve-fitting into a closed-form, hardware-aware prediction.","core_discovery":"The central claim is that acceptance rate and throughput in speculative decoding are predictable, log-linear functions of three controllable resources. Theorems 1.1 and 1.2 state that acceptance rate equals α·log10(T_pretrain)+β with (α,β)≈(0.08,5.05), and α·log10(D)+β with (α,β)≈(0.74,4.61), where T_pretrain is pretraining tokens and D is draft decoder count. Theorem 1.3 states that throughput equals α·log2(b)+β with (α,β)≈(286.79,7.54) along the optimal TopK path topk(b)=27904√(1+0.034/b)−27897. The paper argues that pretraining data, decoder depth, and batch size can be scaled independently, that pretrain-token scaling beats SFT-data scaling at about half the training iterations, and that the combined system Scylla achieves a 1.5–2.2-point acceptance-rate gain at temperature T=0 and up to 2× throughput versus a fixed-policy baseline, with throughput gains concentrated at large batch sizes where a fixed tree width collapses.","pith_inferences":["If the log-linear form is general, the same pretrain-token and decoder-count curves could be used to predict when a draft model is good enough before running expensive verification experiments.","The functional form topk(b)∝√(1+C/b) likely transfers across dense models and GPUs, but the constant C should be recalibrated per model and accelerator; treating C=0.034 as universal would be an overreach the paper does not make.","The laws were measured on Transformer-style autoregressive draft models; testing them on Medusa-style heads or early-exit drafts would reveal whether they are laws of speculative decoding or of this architecture family.","Since the paper leaves RLHF out, a natural extension is to prepend a preference-optimization stage to the draft model; if the pretrain+SFT curves hold, RLHF should shift the intercept β upward without changing the log slope."],"forward_implications":["Pretraining a draft model on roughly 10–50B tokens is a cheap, inference-free way to raise acceptance rate, with gains continuing though saturating past 50B tokens.","Each additional log-unit of draft decoder depth buys a predictable acceptance-rate increase, so draft capacity can be budgeted against added inference cost.","Batch-adaptive TopK selection prevents the throughput collapse that fixed tree widths cause at large batches, keeping arithmetic intensity near the critical point.","At batch 64, a system following the laws keeps a 1.21x speedup over no speculative decoding while a fixed-policy baseline falls below it.","Pretrain scaling plus the same SFT data reaches a given acceptance rate with 4.5x training iterations versus 8x for SFT-only scaling."],"supporting_citations":[{"why":"Establishes the blockwise draft-and-verify decoding loop whose speed these laws predict.","marker":"[10]"},{"why":"Provides the autoregressive Transformer draft-model architecture that is scaled in Theorems 1.1 and 1.2.","marker":"[5]"},{"why":"Supplies the dynamic draft-tree baseline against which the acceptance-rate and throughput gains are measured.","marker":"[6]"},{"why":"Supplies the 8x SFT-data baseline that pretrain-token scaling is compared with.","marker":"[7]"},{"why":"Provides the data-composition recipe and backbone model family used in the pretraining sweeps.","marker":"[11]"},{"why":"Provides the Vicuna backbones and benchmark protocol used in acceptance-rate experiments.","marker":"[12]"},{"why":"Provides the Qwen2.5-72B model whose FLOP and memory counts ground the roofline derivation.","marker":"[13]"},{"why":"Extends the scaling-law validation to the Llama-3 backbone family.","marker":"[14]"},{"why":"Provides the hardware-aware tree configuration method that the closed-form TopK law extends.","marker":"[29]"},{"why":"Supplies the log-linear scaling-law methodology that the paper adapts from loss prediction to decoding efficiency.","marker":"[1]"}],"fun_headline_variants":["Log-linear laws predict speculative decoding speed from three inputs","Scylla coordinates pretrain tokens, draft depth, batch size to speed decoding","Acceptance rate scales log-linearly with pretrain tokens, draft depth, and batch size","Speculative decoding speedups come from scaling three resources log-linearly","Acceptance rate, throughput follow log-linear scaling laws in decoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire batch-size law rests on the assumption that hand-counted arithmetic intensity correctly predicts real performance and that the best tree width is exactly where it equals the hardware's critical compute-to-memory ratio; the resulting constants were never verified against measured optimal tree widths on actual hardware.","fun_headline_variants_meta":{"raw":{"variants":["Log-linear laws predict speculative decoding speed from three inputs","Scylla coordinates pretrain tokens, draft depth, batch size to speed decoding","Acceptance rate scales log-linearly with pretrain tokens, draft depth, and batch size","Speculative decoding speedups come from scaling three resources log-linearly","Acceptance rate, throughput follow log-linear scaling laws in decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":3154,"prompt_tokens":1051,"completion_tokens":2103,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":2008}},"tokens_in":667,"tokens_out":2103,"duration_ms":14439,"temperature":1.0,"reasoning_tokens":2008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:11:58.671863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Qwen2.5-72B on the same H800 setup at batch sizes 1, 4, 16, 32, and 64 and measure throughput for each TopK value from 10 to 80; if the argmax TopK deviates systematically from 27904√(1+0.034/b)−27897, for instance if the optimal width stays flat or falls with batch size, Theorem 1.3 is falsified. Independently, re-run the pretrain-token sweep on a single fixed corpus; an acceptance-rate slope far from 0.08 per decade, or a downturn past 50B tokens, would falsify Theorem 1.1.","supporting_citations":[{"cited_title":"Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.See https://vicuna","cited_arxiv_id":null,"evidence_quote":"Provides the Vicuna backbones and benchmark protocol used in acceptance-rate experiments."}],"review_version":1}