{"id":"3a48045e-c600-4c57-9a3c-327d05c283cf","arxiv_id":"2412.01447","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PLD+ accelerates LLM inference on input-guided tasks by ranking prompt-derived draft spans with hidden states or attention heads, beating tuning-free baselines and often surpassing the tuned EAGLE method.","lead":"PLD+ speeds up LLM output by reusing chunks of the prompt as draft text and using the model's own internal attention and hidden states to pick which chunk to propose. The paper reports 1.5 to 5 times faster generation on summarization, code editing, and text editing, matching or beating tuned speculative decoding methods without any fine-tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main tables do not control for draft-length budget: PLD+ uses K=70 tuned on the evaluation task while PLD and other baselines use defaults, so the claimed ranking advantage is not isolated.","rationale":"The paper is a credible empirical extension of PLD; the verification protocol guarantees exact decoding, and the ablations in Figures 3-5 and Tables 6-8 show the components respond sensibly to K, layer, and head count. I am not alleging cherry-picking; Section 5.5 discloses the validation procedure. The concern is narrower and testable: the headline comparison conflates the ranking mechanism with the draft-length hyperparameter. Since speculative-decoding speedup is strongly affected by accepted draft length and verification batch size, a method allowed K=70 can beat a baseline run at its default K even if the ranking rule is no better. The reader's weakest_assumption correctly worries about transfer of the ranking rule across tasks; my concern is a prior step: the tables do not yet isolate the ranking rule from the K budget. If the proposed controlled comparison restores PLD+'s margin, the central claim stands and the manuscript only needs a revised protocol description; if PLD at equal K matches PLD+, the ranking claim would need to be downgraded, while the method may still be a useful K-tuning recipe. The absence of released code strengthens the need for this check, but I do not treat that absence as itself a correctness flaw.","tokens_in":826,"tokens_out":819,"duration_ms":94527,"concrete_test":"Run the Table 1 greedy benchmark on all five tasks with the draft budget fixed: use the same candidate positions from Eq. 1 and the same K=70 for both PLD and PLD+(h), changing only the ranking criterion (PLD's default heuristic vs Eq. 3), with three seeds. Separately report PLD+(h) with K=10 to bound the budget effect. If PLD at K=70 reaches PLD+, or PLD+ at K=10 still beats default-PLD, the artifact-ranking claim is not established; only if PLD+ at K=70 beats PLD at K=70 does the central claim survive. Repeat with hyperparameters selected on a held-out task set that excludes summarization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PLD+'s central empirical claim, that artifact-based ranking beats PLD's n-gram heuristic, rests on Tables 1 and 2. But Section 5.4 says \"We follow the default parameters for all of our speculative decoding baselines,\" while Section 5.5 tunes PLD+ to K=70, layer l=9, and top-50 attention heads using the summarization task, and those same tables include summarization. The comparison therefore varies two things at once: the ranking rule (Eq. 3 vs PLD's longest-prefix heuristic) and the draft/verification budget (K=70 vs the baselines' defaults). Figure 4 shows K has a large effect on acceptance length, so the reported margin over PLD may be partly, or wholly, a longer-K effect rather than evidence for the hidden-state/attention ranking. Moreover, because K/l/heads were chosen on the summarization set and then reported on that same set, the summarization row in Table 1 is validation performance rather than test performance. This does not make the method internally inconsistent, but it means the headline \"outperforms all tuning-free approaches\" is not yet supported by an apples-to-apples comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PLD+, a plug-and-play speculative decoding method for input-guided tasks such as code/text editing, summarization, and multi-turn conversation. Unlike PLD, which ranks candidate input spans by longest prefix match, PLD+ ranks the occurrences of the last generated token using model artifacts: cosine similarity between hidden states (Eq. 3) or aggregated attention scores from induction heads (Eq. 2). The best-ranked span is then used as the draft and verified against the target LLM. Experiments on Vicuna-7B/13B/33B over five tasks (greedy and sampling) report consistent speedups over PLD and other tuning-free baselines, and in greedy decoding, PLD+ sometimes exceeds EAGLE, a tuning-dependent method. A plug-and-play experiment on Mistral-7B and Llama-2-7B/13B is also included.","tokens_in":18967,"tokens_out":6196,"duration_ms":50518,"significance":"If the results are reproducible, PLD+ offers a practically attractive acceleration for a common class of tasks: it requires no draft-model training, no additional GPU memory for a drafter, and can be dropped into any transformer-based LLM. The use of hidden-state and attention-based ranking is an elegant and well-motivated extension of PLD, and the paper's connection to induction heads is a nice touch. The paper reports error bars and ablations over K, layer, and head counts. However, the main comparison is currently confounded by draft-length budget differences between PLD+ and the baselines, and the validation protocol is not clean because the tuning task (summarization) also appears in the main results. These issues make the central quantitative claims not yet fully supported.","major_comments":[{"comment":"The comparison between PLD+ and the baselines is not apples-to-apples because PLD+ uses K=70 tuned on summarization while the baselines are run with their default parameters. Section 5.4 says 'We follow the default parameters for all of our speculative decoding baselines,' and Section 5.5 states that the best K for PLD+ is 70 on the summarization validation set. Figure 4 shows that K strongly affects acceptance length. Thus Tables 1 and 2 are consistent with the possibility that much or all of PLD+’s advantage over PLD stems from the larger draft budget rather than from the hidden-state/attention ranking. Moreover, since hyperparameters were selected on the summarization task and summarization is included in the same tables, the summarization row is validation performance, not held-out test performance. Please provide a controlled comparison: either sweep K for PLD and PLD+ and report the full curves, or fix K for all methods to the same value (e.g., 70), and report the performance on a summarization split that was not used for tuning.","section":"Section 5.4/5.5, Tables 1-2, Figure 4"},{"comment":"The claim that PLD+ 'outperforms all tuning-free approaches' is contradicted by the paper's own tables. In Table 2 (T=1, Vicuna-7B), REST achieves 1.61x on multi-turn conversation while PLD+ (a) achieves 1.56x and PLD+ (h) 1.54x. In Table 1 (T=0, Vicuna-33B), PLD achieves 2.13x on summarization while PLD+ (a) achieves 2.07x and PLD+ (h) 2.09x. Section 6.1 also states that 'in the greedy decoding scenario, PLD+ either exceeds or matches performance of tuning-free baselines in all five tasks,' which is not true for the Vicuna-33B summarization row. Please qualify these claims to the specific tasks and model sizes where the ordering actually holds.","section":"Abstract and Section 6.1, Tables 1-2"},{"comment":"The descriptors 'tuning-free' and 'does not require any additional compute' are misleading as stated. Section 5.5 describes selecting K, layer l, and top-50 attention heads on a validation set, which is hyperparameter tuning even if no model fine-tuning occurs. Furthermore, Eq. 3 requires computing cosine similarities between the current hidden state and hidden states of all candidate positions in every decoding step, and Eq. 2 requires scanning attention maps; these are extra operations not present in PLD’s string matching. Please clarify that 'tuning-free' means 'no draft-model training' and report the measured overhead of the ranking step (e.g., fraction of wall-clock time) to substantiate the 'no additional compute' claim.","section":"Abstract, Section 1, Section 4.2.1, Section 5.5"},{"comment":"The plug-and-play experiment on Mistral-7B and Llama-2 models does not state how hyperparameters were transferred. The paper tunes l=9 and K=70 on Vicuna-7B, but Mistral-7B and Llama-2 have different layer counts and architectures, and the manuscript does not say whether l was re-tuned, mapped by layer index, or kept at 9. This matters because Figure 5 shows that acceptance length depends strongly on l. Please state the hyperparameter setting used for each model in Table 3, and ideally provide a small layer-sensitivity analysis for at least one non-Vicuna model. Additionally, EAGLE and REST results are absent for Mistral-7B, so the claim that PLD+ 'consistently outperforms all tuning-free baselines' across models is based only on a comparison with PLD in that column.","section":"Section 6.2, Table 3"}],"minor_comments":[{"comment":"There is a typo: 'autoretrogressive' should be 'autoregressive'.","section":"Abstract"},{"comment":"The caption lists 'L2C-7B' twice for the Llama-2-7B and Llama-2-13B columns; the second should be 'L2C-13B'.","section":"Table 3 caption"},{"comment":"In Section 2, the PLD/LLMA citation is given as '(Saxena, 2023; Yang et al., 2023b)', but the reference list only contains Yang et al. (2023a) as 'Inference with reference'; the 'Predictive pipelined decoding' paper is 2023b. Please align the in-text citation with the correct bibliography entry.","section":"References"},{"comment":"The sentence 'PLD+ even surpasses the best-performing fine-tuned approach, EAGLE, by margins ranging from 0.24x to 2.85x' is ambiguous: 2.85x is EAGLE's speedup on long text editing in Table 1, not the margin over EAGLE. Please state the margin explicitly (e.g., 5.16x vs. 2.85x for Vicuna-7B on long text editing).","section":"Section 6.1"},{"comment":"No code or public evaluation scripts are released. Providing the implementation would substantially improve reproducibility, especially given the need for the controlled K comparison requested above.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising and the paper is clearly written, but the evaluation currently conflates the ranking rule with a much larger draft budget (K=70 vs. baselines' defaults) and uses the tuning task as one of the main test tasks without a clean held-out split. These issues are fixable: a K-controlled comparison, a genuine validation split, and a tempered set of claims would make the paper suitable. I would also encourage the editor to ask for the code, since the method is empirical and the overhead of the ranking step needs to be measured rather than asserted to be zero."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PLD+ is a genuinely useful extension of PLD, but the paper's main comparison is confounded. The authors tune PLD+ to K=70 on the summarization task while all baselines run with their default parameters, so Tables 1-2 vary both the ranking rule and the draft/verification budget at once. I don't think the core idea is wrong, but the paper as written does not isolate it.\n\nWhat is actually new: PLD/LLMA picks a copy span by longest-prefix matching. PLD+ instead ranks candidate spans by cosine similarity of hidden states (Eq. 3) or by selected induction-head attention (Eq. 2). That is a reasonable, well-motivated change, and the ablations support it: Figures 3-4 show the ranking matters and selected heads beat pooling all heads. The gains over PLD are consistent across Vicuna-7B/13B/33B in both greedy and sampling, and the transfer to Llama-2 and Mistral on summarization is a nice touch. The attribution appendix is extra evidence that the ranking is pointing at something semantically real.\n\nSoft spots, in order of severity. First, the K confound. Section 5.4 says the authors follow default parameters for all baselines; Section 5.5 tunes K=70 for PLD+. Figure 4 shows acceptance length depends strongly on K. So the margin over PLD in Tables 1-2 may be partly or wholly a longer-K effect. A controlled comparison—same K for PLD and PLD+, or a K sweep—is necessary to support the claim that artifact-based ranking beats n-gram matching. Second, the tuning set is one of the reported tasks. K, layer l, and top-50 heads are chosen on the summarization set, and summarization appears in Tables 1 and 2. That row is validation performance, not test performance, so the claim to outperform all tuning-free approaches on that task is weaker than stated. Third, no code is released, which is a real obstacle for a paper whose selling point is plug-and-play usability. The threshold and averaged-hidden-state ablations are minor and don't change the main picture.\n\nWho is this for: someone building inference acceleration for code/text editing or summarization who wants a tuning-free option and is willing to re-run the comparison with matched budgets. If I were refereeing, I would ask for code and a K-matched evaluation, and I would want the summarization row flagged as validation or removed from the headline tables. The central idea—model artifacts can rank copy sources better than string matching—is plausible and probably right, but the current experiments don't yet nail it.\n\nI'd send it to peer review rather than desk reject. It's a legitimate empirical contribution, not a breakthrough. The confound is fixable with modest extra experiments, and the paper is worth a serious referee's time.","headline":"A useful PLD extension whose main comparison is undercut by a draft-budget confound; the ranking idea is plausible but not yet isolated.","tokens_in":19461,"tokens_out":2995,"would_cite":true,"duration_ms":25935,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tuning-free speculative decoder ranks copy candidates with the model's own hidden states and attention, beating tuned EAGLE on four of five input-guided tasks in greedy decoding.","keywords":["speculative decoding","LLM inference acceleration","prompt lookup decoding","input-guided tasks","hidden states","attention heads","code editing","summarization"],"falsifier":"Use a benchmark like code editing where the input contains repeated identical prefixes followed by different continuations, record for each accepted draft span the true copy source, and check whether PLD+(h)'s cosine-similarity ranking picks that source more often than a longest-prefix or random rule; if acceptance length does not rise accordingly, the ranking rule is not the cause of the reported speedup.","tokens_in":18516,"feed_emoji":"⚡","tokens_out":7659,"duration_ms":62746,"temperature":0.7,"pith_summary":"This paper tries to establish that large-language-model inference can be accelerated on tasks where the output overlaps with the input—code editing, text editing, summarization, and multi-turn conversation—without any fine-tuning or extra draft model. The proposed method, PLD+, finds every input position where the last generated token appears and picks the source to copy from by ranking those positions with model artifacts the forward pass already computes: cosine similarity between hidden states, or attention from selected induction heads. It then drafts the text following that source and verifies the draft in one parallel pass, so the generation is lossless with respect to standard autoregressive decoding. On five input-guided tasks the authors report that PLD+ outperforms every tuning-free baseline, and in greedy decoding it outperforms the tuned EAGLE on four of them, with up to 5.16x speedup on long text editing with Vicuna-7B. If the claim holds, any LLM can get speculative-decoding speedups out of the box using only its own intermediate states.","feed_headline":"Copy-from-input decoding hits 5.16x speedup with no tuning","feed_subtitle":"PLD+ ranks input spans by the model's own hidden states, beating every tuning-free baseline across five tasks.","key_machinery":"The load-bearing object is the occurrence-ranking rule in Eq. (3). For each candidate copy position j where the current token x_t appears in the input, PLD+ compares the hidden state just before that occurrence, H^(l)_{j-1}, with the hidden state just before the current token, H^(l)_{t-1}, using cosine similarity; the occurrence with the highest similarity becomes the copy source. The attention variant aggregates, over a set of induction heads identified by a prefix-matching and copying probe, the attention paid by token x_{t-1} to each candidate position and picks the max. Both variants produce a draft span of K tokens that the target model verifies in parallel, making the procedure lossless and free of added parameters.","core_discovery":"PLD+'s central claim is that the right span to copy from can be identified semantically rather than by surface heuristics. At decoding step t, PLD+ collects the positions j < t where x_j = x_t and selects j* by maximizing cos_sim(H^(l)_{j-1}, H^(l)_{t-1}) in the hidden-state variant, or by maximizing attention from pre-identified induction heads in the attention variant. The draft is x_{j*+1}, ..., x_{j*+K}, and a single forward pass of the target model verifies the draft tokens, so the output distribution is unchanged. The paper reports that this mechanism outperforms all tuning-free speculative decoding baselines on all five tested input-guided tasks and, in greedy decoding, outperforms the tuning-dependent EAGLE on four tasks (e.g., 5.16x vs 2.85x speedup on long text editing with Vicuna-7B, and 5.37x vs 3.32x on code editing with Vicuna-13B).","pith_inferences":["A natural extension is to apply the artifact-based ranking to retrieval-augmented generation, where the copied spans come from retrieved passages rather than the prompt; the paper lists RAG as input-guided but does not benchmark it.","The hidden-state cosine rule is a testable hypothesis about transformer internals, and measuring its agreement with human-annotated copy sources across models could serve as a cheap interpretability probe.","Because PLD+ points to the source of every accepted copied span, it could be developed into a token-level provenance tool without additional inference cost, beyond the preliminary attribution results in the appendix.","The same ranking rule could be combined with a learned or retrieval-based candidate generator to push acceptance length further, since the authors show longer drafts benefit from better ranking."],"forward_implications":["PLD+ can be applied to any LLM without training or a separate draft model, since it reads only attention and hidden states that inference already computes.","In greedy decoding, PLD+ reports speedups above the tuned EAGLE on code editing, short text editing, and long text editing across Vicuna-7B/13B/33B, with margins up to 2.31x in average speedup.","Under sampling (T=1), PLD+ still beats all tuning-free baselines on four of five tasks and trails REST on multi-turn conversation by at most 0.18x.","The selected copy position doubles as an attribution signal, pointing to the input span the model copied, with preliminary QuoteSum F1 matching a dedicated attribution method.","Because the hyperparameters (K, layer, and top-50 heads) are chosen on a single validation task, the method is straightforward to re-tune for new models or tasks."],"supporting_citations":[{"why":"Introduces the inference-with-reference idea that PLD+ builds on: draft by reusing text spans from the input, the tuning-free baseline PLD+ must beat.","marker":"Yang et al., 2023a"},{"why":"Provides the Prompt Lookup Decoding recipe of n-gram lookups whose longest-prefix ranking PLD+ replaces with artifact-based ranking.","marker":"Saxena, 2023"},{"why":"EAGLE is the tuning-dependent speculative decoder that PLD+ claims to outperform on four greedy-decoding tasks.","marker":"Li et al., 2024"},{"why":"REST is the retrieval-based tuning-free baseline that PLD+ improves on by avoiding an external datastore.","marker":"He et al., 2023"},{"why":"Supplies the induction-head mechanism that motivates selecting attention heads for copy-source ranking.","marker":"Olsson et al., 2022"},{"why":"Spec-Bench supplies the benchmark, datasets, baseline code, and the greedy and sampling evaluation protocol used in the main experiments.","marker":"Xia et al., 2024"},{"why":"Provides the hidden-state attribution method and QuoteSum evaluation used in Appendix D for preliminary attribution results.","marker":"Phukan et al., 2024"}],"fun_headline_variants":["PLD+ reuses hidden states for 5.16x speedup, no tuning needed","Copy-from-input decoding beats tuned EAGLE on four tasks","No-tuning PLD+ accelerates code editing 5.37x with Vicuna-13B","Input-overlap decoding: PLD+ exploits attention for faster LLMs","Tuning-free speculative decoding outruns all baselines on five tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the input position whose preceding hidden state is most cosine-similar to the current hidden state (or is most attended by selected induction heads) is the right source to copy from; the paper supports this with experiments, not a proof.","fun_headline_variants_meta":{"raw":{"variants":["PLD+ reuses hidden states for 5.16x speedup, no tuning needed","Copy-from-input decoding beats tuned EAGLE on four tasks","No-tuning PLD+ accelerates code editing 5.37x with Vicuna-13B","Input-overlap decoding: PLD+ exploits attention for faster LLMs","Tuning-free speculative decoding outruns all baselines on five tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1699,"prompt_tokens":984,"completion_tokens":715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":611}},"tokens_in":600,"tokens_out":715,"duration_ms":6771,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:22:09.993785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a benchmark like code editing where the input contains repeated identical prefixes followed by different continuations, record for each accepted draft span the true copy source, and check whether PLD+(h)'s cosine-similarity ranking picks that source more often than a longest-prefix or random rule; if acceptance length does not rise accordingly, the ranking rule is not the cause of the reported speedup.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Prompt Lookup Decoding recipe of n-gram lookups whose longest-prefix ranking PLD+ replaces with artifact-based ranking."}],"review_version":1}