{"id":"255a86e0-e17b-4445-a6d5-6a6f6e2d12a9","arxiv_id":"2506.02391","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Consultant Decoding speeds up LLM generation by accepting draft tokens whose negative log-likelihood under the target model falls below a fixed threshold, reaching 2-3x speedups with comparable quality.","lead":"This paper proposes a faster way for small and large AI models to work together when generating text: the large model checks each draft token by looking at how surprised it is, rather than comparing probabilities between models. The authors report 2-3x speedups with roughly unchanged output quality and fewer calls to the big model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fixed ε=2.0 threshold is the linchpin of CD's quality-preservation claim, yet the paper's own Table 15 shows optimal thresholds of 4.0–7.5 and the Section 3.4 Top-P analogy is approximate, leaving the generic setting without a principled basis.","rationale":"The reader's weakest_assumption correctly identifies the fixed ε=2.0 threshold as load-bearing. My stress-test agrees but sharpens it with two concrete internal problems: (1) the Section 3.4 Top-P equivalence is invalid under the actual EMA smoothing and full-distribution resampling, so the theoretical quality guarantee is not established; (2) the paper's own Table 15 and Appendix A.1 show thresholds far from 2.0 and a substantial quality drop in one speed-optimal configuration, contradicting the broad 'comparable quality' claim. These issues do not refute the empirical speedup results on the tested benchmarks—the generic-setting tables (Tables 2 and 14) show quality within a few percent—so the verdict remains CONDITIONAL rather than REJECT. The concern mainly affects the generality and theoretical grounding of the method, not its demonstrated performance on the reported tasks. A concrete transfer test and ROC analysis would settle whether the threshold concern actually lands.","tokens_in":18269,"tokens_out":14733,"duration_ms":140021,"concrete_test":"Run CD with ε=2.0 and β=0.2 under the exact Generic protocol of Section 4 on a held-out model family (e.g., Mistral-7B/70B) and on the business dataset with draft length 20, measuring normalized quality and LLM-calls ratio. If any benchmark's normalized quality falls below 95% or the LLM-calls ratio exceeds 15%, the generic threshold does not deliver the claimed quality-speedup trade-off and the central claim fails to generalize. Independently, compute the ROC-optimal threshold for the NLL-based correctness rule on draft-token data from GSM8K and MT-Bench; if the optimal threshold deviates from 2.0 by more than 1.0 nats on either task, the fixed-threshold rationale is empirically unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—CD preserves target-model quality with a generic, tuning-free threshold—rests on the assumption that ε≈2.0 reliably separates correct draft tokens from incorrect ones across models, tasks, and contexts. The paper justifies this via the Chinchilla convergence loss (Eq. 9), but that is only an average training loss, not a per-token correctness boundary; many correct tokens will have NLL greater than 2.0, and many incorrect tokens can have NLL below it. This is not a formal derivation, and the Section 3.4 Top-P analogy is doubly flawed: it only holds in the limit β→0, whereas the actual algorithm uses EMA β=0.2, and it ignores the fact that after rejection CD resamples from the full target distribution p_{n+1}(·), not from the Top-P nucleus. Thus the accepted-token set is not equivalent to Top-P sampling, and the resampled token can fall outside the nucleus. The paper's own Table 15 reveals that the speed-optimal thresholds are 4.0–7.5 across tasks, far from 2.0. Although the generic setting empirically works on the tested Qwen2.5 and Llama-3.1 benchmarks, the supporting rationale is not internally consistent, and the Appendix A.1 business-dataset result (Table 8) shows a 16.5% quality drop in the speed-optimal setting, undermining the abstract's 'around 100%' quality claim. If ε=2.0 does not transfer to new model families or task distributions, the central drop-in verification claim collapses even though the speedup mechanism itself would survive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Consultant Decoding (CD), a modification of the verification step in speculative decoding. Instead of accepting a draft token with probability min(1, p/q) as in speculative sampling, CD accepts token x_i when the EMA-smoothed negative log-likelihood -log p_i(x_i) of the target model is below a fixed threshold ε, set to ε=2.0 in the 'Generic' configuration using the Chinchilla scaling-law estimate of the convergence loss. On rejection, CD resamples from the target distribution at the rejection position. The authors claim speedups up to 2.5x in the generic setting (and >3x in a speed-optimal configuration) with quality around 100% of the target model and large-model call ratios below 10%, and they present experiments on GSM8K, HumanEval, MT-Bench, AlpacaEval, a business dataset, on Qwen2.5 0.5B/1.5B/3B/7B/72B and Llama-3.1-8B/70B pairs, and on EAGLE-2. Section 3.4 argues that CD is approximately equivalent to Top-P sampling with nucleus size 1-e^{-ε}.","tokens_in":18639,"tokens_out":7501,"duration_ms":65620,"significance":"If the empirical results hold, CD is a simple, training-free, drop-in alternative to speculative sampling: no distribution matching is required, and the verification rule uses only target-model likelihoods. The paper's strengths are the breadth of the benchmark coverage (four public tasks plus a business benchmark; two model families; independent and self-drafting settings), the inclusion of sampling-decoding results, and a clear algorithm. The method is easy to reproduce from Algorithm 1. The principal caveat is that the paper's theoretical rationale for the fixed threshold is approximate, and the headline quality claim is stronger than some of the reported numbers (especially Table 8). The empirical generic-setting results are nevertheless substantial and would be useful to the speculative-decoding community if the claims are appropriately qualified.","major_comments":[{"comment":"The claimed equivalence to Top-P sampling is not mathematically established. The derivation requires β→0, but Algorithm 1 and the Generic setting use β=0.2, so the EMA smoothing is not negligible. More importantly, the acceptance criterion p(x_i) > e^{-ε} is a per-token probability threshold, while a Top-P nucleus is defined by cumulative probability mass; these sets do not coincide. Finally, upon rejection CD samples x'_{n+1} from the full target distribution p_{n+1}(·), so the final output distribution includes tokens outside any fixed nucleus. Consequently the sentence 'CD shares similar property of Top-P sampling' and the Conclusion's 'theoretical basis' overstate the support. Please either provide a formal statement (e.g., a divergence bound or a distributional identity) or explicitly label Section 3.4 as intuition.","section":"Section 3.4"},{"comment":"The fixed threshold ε=2.0 is load-bearing for the generic-setting quality claim. The derivation from the Chinchilla convergence loss gives only a training-set average; it does not imply that per-token NLL below 2.0 separates correct from incorrect draft tokens. Table 15 reports speed-optimal thresholds of 4.0 to 7.5 for the very same model pairs and tasks, and the Limitations section concedes that the optimal threshold depends on the question and model combination. The paper should add an analysis of the NLL distributions of accepted/rejected tokens under ε=2.0 (or an ablation across model families and tasks) to substantiate the generic threshold; without it, the generic-setting quality result remains an empirical finding limited to Qwen2.5 and Llama-3.1.","section":"Section 3.3 and Table 15"},{"comment":"The headline claim of 'comparable generation quality (around 100% of the target model's performance)' is not consistent with Appendix A.1. In the Speed-Optimal setting on the business dataset, CD achieves 83.5% of the target score (Table 8), a 16.5% drop, while Section 4.1.1 says the speed-optimal setting allows a performance decrease 'within 5%' of the target model. The abstract and Section 4.1.1 need to be qualified (e.g., restrict the 'around 100%' claim to the Generic setting), and the speed-optimal parameter search should either enforce the stated 5% bound or report the exception transparently.","section":"Abstract, Section 4.1.1, and Table 8"}],"minor_comments":[{"comment":"In the paragraph after Eq. (6), 'modls' should be 'models'; the sentence 'otherwise we sample a new token from the distribution given by the target model' would read more clearly as 'otherwise we sample a new token from the distribution given by the target model at the rejection position.'","section":"Section 3.2"},{"comment":"The figure contains the placeholder text 'Re-Sampling from ??' in the diagram; this needs to be cleaned up, and the label should state explicitly that the resampling is from the full target distribution p_{n+1}(·), since that is precisely the point where CD differs from Top-P sampling.","section":"Figure 2"},{"comment":"Line 17 computes r_{n+1} and returns it, but no subsequent iteration uses this value as shown; if it is intended as the EMA state for the next drafting block, say so explicitly, otherwise remove it.","section":"Algorithm 1"},{"comment":"The normalization description says metrics are normalized to the target model's greedy decoding unless otherwise specified, but Table 9 reports sampling-decoding results; it should be stated explicitly whether those normalized percentages are relative to greedy target output or to a sampling baseline, as this affects interpretation.","section":"Section 4"},{"comment":"The Conclusion says the paper provides a 'theoretical analysis' and a 'theoretical basis' for the threshold; given the approximations acknowledged in Section 3.4, it would be more accurate to describe the support as an empirical study plus an approximate analogy.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is reasonably thorough and the method is simple and reproducible, which makes the paper a plausible fit for the venue. The main editorial concern is overclaiming: the abstract and conclusion present the Top-P analogy as a theory and the quality preservation as universal, while the paper's own Limitations and Table 8 show important exceptions. I would support publication after a revision that corrects the theoretical framing, qualifies the headline claims, and adds threshold-transferability evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CD replaces the likelihood-ratio verification in speculative decoding with a fixed threshold on the target model's own token NLL, and the paper shows empirically that this simple rule works in a generic setting across several model pairs and tasks. That is the real contribution. The theoretical packaging around it is weaker than the experiments, and the 'around 100% quality' headline hides at least one bad case in the business dataset.\n\nWhat is actually new: the acceptance criterion itself is a target-confidence threshold, closely related to BiLD's verification, and the paper admits that similarity but does not compare against BiLD. The new details are the Chinchilla-derived default epsilon=2.0, the EMA smoothing, and the breadth of the experiments. Credit where due: the generic-setting results in Table 14 are honest and useful—about 2.4-2.5x speedup on Qwen2.5-0.5B/72B and 2.5x on Qwen2.5-7B/72B with accuracy within a few points of the target, and similar on Llama-3.1. The walltime, LLM-call ratios, and acceptance-length scalings are all reported, which makes the empirical claim reproducible enough to test independently.\n\nSoft spots. The Top-P equivalence in Section 3.4 is approximate in two ways the paper does not fully acknowledge: it only holds as beta -> 0, while the algorithm uses beta=0.2, and after rejection CD resamples from the full target distribution, not from the nucleus. So the theoretical 'consistency' argument does not actually establish that CD preserves target quality. The fixed epsilon=2.0 is a reasonable guess, but the paper's own Table 15 shows optimal thresholds of 4-7.5 across tasks, and the Limitations section says the fixed threshold is not optimal. That is a fair admission, but it means the generic setting is empirically validated rather than principled. The business dataset result (Table 8) is more concerning: in the speed-optimal setting CD scores 83.5 versus SD's 98.9, which is a 15-point drop and undermines the abstract's 'around 100%' phrasing. The quality claim should be scoped to the reported public benchmarks. No code is released, which is a real obstacle for a method whose main contribution is a drop-in verification rule.\n\nWho this is for: people working on speculative decoding or LLM serving who want a simple, training-free verification change. The method is easy to implement and the generic-setting numbers give a fair baseline. It deserves a serious referee: the experiments are strong enough to justify a revised version that adds a BiLD comparison, reports the business-dataset caveat honestly in the abstract, and tones down the Top-P equivalence. I would cite it if I worked in this area.","headline":"Simple, well-tested threshold-based verification rule for speculative decoding; the experiments are stronger than the theory, and the 'around 100% quality' headline needs caveats.","tokens_in":19165,"tokens_out":2705,"would_cite":true,"duration_ms":23259,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Consultant Decoding replaces speculative decoding's likelihood-ratio check with a fixed threshold of about 2.0 on the target model's token-level negative log-likelihood, claiming up to 2.5x speedup at near-full quality.","keywords":["speculative decoding","inference acceleration","token verification","negative log-likelihood","Chinchilla scaling law","nucleus sampling","large language models","draft-verify decoding"],"falsifier":"Run CD with the generic $\\varepsilon=2.0$ on a broad held-out suite using a model whose Chinchilla-estimated convergence loss is far from 2.0, then compare generation quality and acceptance rate against the target model's greedy decoding; if quality falls noticeably below the target baseline while acceptance stays high, the generic-threshold claim is refuted. The paper's own speed-optimal tables already show best thresholds from 4.0 to 7.5, so an even sharper test is to check whether CD with $\\varepsilon=2.0$ underperforms on exactly those tasks or on a new task with a different optimal threshold.","tokens_in":18050,"feed_emoji":"⚡","tokens_out":8283,"duration_ms":66696,"temperature":0.7,"pith_summary":"Consultant Decoding asks whether a draft token is acceptable by checking the large target model's own negative log-likelihood for that token, and accepts it when that value falls below a fixed threshold of about 2.0. The paper argues this small change does what speculative decoding's likelihood-ratio verification fails to do: it admits far more draft tokens without admitting low-quality ones, because a token the target model finds as predictable as a typical training token is likely a good continuation. Across math, code, chat, and instruction-following benchmarks, the paper reports up to a 2.5-fold wall-clock speedup over greedy target decoding, output quality at roughly 100 percent of the target model, and large-model call ratios below 10 percent. The claim matters because it would turn speculative decoding into a training-free, parameter-free, drop-in acceleration where the draft model can be orders of magnitude smaller than the target.","feed_headline":"Single threshold speeds LLM inference up to 2.5x","feed_subtitle":"Accept tokens a big model finds as predictable as its training data; quality stays near 100%, calls fall below 10%.","key_machinery":"The load-bearing object is the acceptance rule $V_{\\mathrm{CD}}(x_i)=\\varepsilon - (-\\log p_i(x_i))$ together with its EMA-smoothed version $V(x_i)=\\varepsilon-\\mathrm{EMA}_\\beta(r_i)$, where $r_i=-\\log p_i(x_i)$ and $\\beta=0.2$ in the generic setting. This is a single-token perplexity filter: the draft token is treated as pseudo ground truth, and it is accepted only if the target model assigns it a negative log-likelihood at or below the training-time convergence loss. The paper links the rule to nucleus sampling by observing that when $\\beta\\to 0$, acceptance is equivalent to $p_i(x_i) > e^{-\\varepsilon}$, so an accepted draft token lies inside the target model's high-probability nucleus of mass $1-e^{-\\varepsilon}$; this analogy is what is supposed to explain why quality is preserved with a fixed threshold.","core_discovery":"The central claim is that token correctness for speculative decoding can be verified by a target-model-only likelihood test rather than by a ratio between draft and target distributions. Formally, a draft token $x_i$ is accepted when $V_{\\mathrm{CD}}(x_i)=\\varepsilon - (-\\log p_i(x_i)) > 0$, with $\\varepsilon\\approx 2.0$ estimated from the Chinchilla convergence loss and smoothed by an exponential moving average over recent tokens. The paper argues that this aligns the acceptance criterion with the target model's own training objective, preserves the target model's output distribution approximately in the sense of Top-P sampling, and on some tasks even lets the draft-guided path outperform the target model's greedy decoding. The empirical claim is that this yields speedups of roughly 2x to 3x with quality near 100 percent of the target model and large-model calls below 10 percent, including on a 0.5B/72B draft-target pair.","pith_inferences":["This suggests that acceptance in speculative decoding may be better predicted by the target model's calibration than by the agreement between draft and target distributions; if so, per-model thresholds could be learned from a few held-out loss statistics instead of searched per task.","The reported cases where CD outperforms the greedy target model point to a voting or ensemble effect between the draft and target models, which could be tested directly by using several small draft models simultaneously and counting votes.","Since the rule is just a threshold on the target's own loss, it should compose naturally with block-wise, tree-based, and multi-candidate drafts, potentially pushing large-model call ratios even lower without extra training.","A practical extension would be to expose $\\varepsilon$ as a latency-quality dial: the paper's own tables show optimal thresholds between 4.0 and 7.5 for speed-optimal settings, and a learned per-request epsilon could let a server trade accuracy for throughput continuously."],"forward_implications":["Speculative decoding can be accelerated without retraining, without per-task tuning, and without a separate acceptance threshold per model pair, because the generic setting uses one value $\\varepsilon=2.0$.","Draft models much smaller than the target become practical: a 0.5B draft paired with a 72B target still yields speedups of roughly 2x to 3x, so the method scales to model pairs with two orders of magnitude in parameter count.","Longer draft windows become efficient: the speedup lost when draft length grows from 6 to 20 is much smaller for CD than for speculative decoding or Mentored Decoding, reducing the need to tune draft length.","Large-model call frequency drops below 10 percent on reasoning and coding tasks, which directly increases the throughput available for parallel serving.","The verification module can be swapped into self-drafting frameworks such as EAGLE-2 and still add speed, so the mechanism is not tied to a separate small draft model."],"supporting_citations":[{"why":"Defines the speculative decoding verification scheme that CD replaces.","marker":"Leviathan et al., 2023"},{"why":"Provides the importance-sampling based speculative sampling baseline and its distribution-matching guarantee.","marker":"Chen et al., 2023"},{"why":"Supplies the Chinchilla scaling law whose convergence-loss estimate yields the generic threshold epsilon=2.0.","marker":"Hoffmann et al., 2022"},{"why":"Defines Mentored Decoding, the relaxed distribution-ratio baseline CD is compared against.","marker":"Tran-Thien, 2023"},{"why":"Provides nucleus (Top-P) sampling, used as the theoretical analogy for why a fixed likelihood threshold preserves quality.","marker":"Holtzman et al., 2019"},{"why":"Provides the GSM8K math benchmark used in the main speedup and quality evaluations.","marker":"Cobbe et al., 2021"},{"why":"Provides the EAGLE-2 self-drafting framework used to show CD generalizes beyond small draft models.","marker":"Li et al., 2024a"}],"fun_headline_variants":["Consultant Decoding: 2.5x faster LLMs, quality intact","One likelihood check speeds up LLM inference 2.5x","LLM acceleration: likelihood-based accept, 2.5x speed","Draft model runs wild, big model checks in: 2.5x speed","Verify with likelihood, not ratios: 2.5x faster LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on a single number: that a fixed threshold of about 2.0 on the target model's negative log-likelihood marks the boundary between acceptable and unacceptable draft tokens across models, tasks, and contexts.","fun_headline_variants_meta":{"raw":{"variants":["Consultant Decoding: 2.5x faster LLMs, quality intact","One likelihood check speeds up LLM inference 2.5x","LLM acceleration: likelihood-based accept, 2.5x speed","Draft model runs wild, big model checks in: 2.5x speed","Verify with likelihood, not ratios: 2.5x faster LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3159,"prompt_tokens":931,"completion_tokens":2228,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2127}},"tokens_in":547,"tokens_out":2228,"duration_ms":17443,"temperature":1.0,"reasoning_tokens":2127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:24:46.403712+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CD with the generic $\\varepsilon=2.0$ on a broad held-out suite using a model whose Chinchilla-estimated convergence loss is far from 2.0, then compare generation quality and acceptance rate against the target model's greedy decoding; if quality falls noticeably below the target baseline while acceptance stays high, the generic-threshold claim is refuted. The paper's own speed-optimal tables already show best thresholds from 4.0 to 7.5, so an even sharper test is to check whether CD with $\\varepsilon=2.0$ underperforms on exactly those tasks or on a new task with a different optimal threshold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Mentored Decoding, the relaxed distribution-ratio baseline CD is compared against."}],"review_version":1}