{"id":"06c790d4-7aba-4b48-b962-cde7d5e0ca9d","arxiv_id":"2502.08923","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CopySpec accelerates LLM inference by copying repeated token sequences from context and verifying them with the target model, achieving up to 3.08x speedups without altering outputs.","lead":"CopySpec speeds up LLM generation by copying repeated text blocks from the conversation or input instead of regenerating them. It reports up to 3.08x faster second-turn responses and a 49% extra gain over speculative decoding, with unchanged outputs at greedy settings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 3.08x / '49% over SD' speedups come from MT-Redundant, whose second-turn prompts are constructed to request variations of the first answer; on unmodified MT-Bench standalone speedups are only ~1.15-1.3x, so the central speed claim may not generalize to organic workloads.","rationale":"I read the central claim charitably: CopySpec is an exact-verification speculative method, and its quality-neutrality under greedy decoding is supported by the algorithm. The paper also evaluates on several datasets and reports small positive gains even on MT-Bench, so I would not reject it. However, the magnitude of the advertised gains is driven by a self-created benchmark that mechanically maximizes the exact phenomenon the method exploits. The reader's weakest_assumption already pointed at the empirical repetition premise; my pass sharpens it by showing that the paper's own numbers on unmodified MT-Bench are modest, which is the right yardstick for generality. A single natural-corpus evaluation would settle whether the 3.08x and +49% headline reflect a general property or an artifact of benchmark construction. I also noted that the Appendix B pseudocode is sketchy about the replacement token after a rejected copy, but Section 3.2 and the reported experiments indicate the intended behavior, so I do not treat that as the load-bearing issue. The appropriate verdict remains CONDITIONAL, so no change from the reader's verdict is needed.","tokens_in":33806,"tokens_out":20439,"duration_ms":224802,"concrete_test":"Run the released CopySpec code on a naturally occurring multi-turn corpus (e.g., LMSYS-Chat-1M or ShareGPT) with the same settings (gamma=3, block=10, temperature=0), without filtering for turns that request revisions of a prior answer. Compute overall and per-turn tokens/sec speedup over the base model and over vanilla speculative decoding, plus the fraction of copied tokens accepted. If the median speedup is below about 1.2x standalone or below 1.1x over speculative decoding, the abstract's 3.08x and 49% figures should be explicitly re-scoped as benchmark-specific rather than general LLM speedups.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CopySpec's quality-preservation argument is sound for greedy decoding: copying merely proposes tokens that are verified against the target model, so rejected copies cannot change the output. What the central claim needs, therefore, is that repeated gamma-suffixes with faithful continuations occur often enough in real traffic to make copying profitable. The paper's largest speedups are measured on MT-Redundant (Section 4.4), whose second turn is deliberately rewritten to request variations of the first answer, guaranteeing high lexical overlap and thus high copy acceptance. On unmodified MT-Bench, the standalone speedup is only 1.15-1.3x (Tables 14-18) and the gain over speculative decoding is about 10-30% (Tables 19-20), not 49%. CNN/DM reaches 2.35x only because the prompt explicitly requests extractive summaries (Appendix H.2). The Limitations section itself concedes the method 'assumes that the repeated content appears in close proximity and with high lexical overlap.' Thus the load-bearing empirical premise—that organic second turns contain copyable blocks—is not established by the reported headline numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CopySpec, a speculative decoding add-on that detects repeated γ-token suffixes in the model's context and proposes the following block of tokens from that earlier occurrence as a draft. The draft is then verified by the target model in the same way as in speculative decoding, so at temperature 0 the greedy output is unchanged. The method requires no draft model when used standalone and no additional GPU memory. Experiments are reported on five instruction-tuned LLMs across MT-Bench, CNN/DM, GSM8K, HumanEval, and a new MT-Redundant dataset, with secondary experiments on Vicuna with EAGLE, PLD, and SAM-D. The headline results are up to 3.08× speed-up on second-turn MT-Redundant, 2.35× on CNN/DM, 2.66× on GSM8K self-correction, and a 49% additional speed-up over speculative decoding on MT-Redundant, all while preserving greedy output quality. The paper also includes an analysis of the γ hyperparameter and an appendix with detailed per-category tables, prompts, and examples.","tokens_in":34039,"tokens_out":4621,"duration_ms":50717,"significance":"If the empirical speedups hold on real workloads, CopySpec would be a valuable, lightweight addition to LLM inference because it is algorithmically simple, requires no changes to the target model, and its exactness argument for greedy decoding is sound: copied tokens are only proposals that are verified against the target model, so rejected copies cannot change the output distribution. The paper ships public code and a public dataset, reports throughput across several model families, and is unusually transparent in its Limitations section, which concedes that the method assumes repeated content with high lexical overlap. The main risk is not correctness of the algorithm but generality of the empirical speed claim: the largest speedups come from MT-Redundant, a dataset constructed by rewriting second turns to request variations of first-turn answers, and from CNN/DM with an explicitly extractive summarization prompt. On unmodified MT-Bench the speedups are modest. Because the central contribution is an empirical claim about when copying is profitable, the benchmark-specificity of the headline numbers is the load-bearing issue for this manuscript.","major_comments":[{"comment":"The central speedup claims are dominated by MT-Redundant, a dataset constructed by rewriting the second turn of MT-Bench to ask for variations of the first-turn answer (Section 4.4). On unmodified MT-Bench, the average second-turn speedups in Tables 14–18 are only 1.27–1.41×, and the gains over speculative decoding in Tables 19–20 are about 1–11%, not the 49% reported in the abstract. The 2.04× average and 3.08× category-level speedups in Table 2 therefore do not establish that organic multi-turn workloads contain enough lexical repetition for CopySpec to be profitable. This is a load-bearing point because the paper's conclusion generalizes the result to 'results across five LLMs and datasets' rather than restricting it to modified, redundancy-heavy benchmarks.","section":"Section 4.4, Tables 2 and 14–20"},{"comment":"The CNN/DM result is likewise driven by a prompt that explicitly demands extractive summarization: the prompt in Appendix H.2 instructs the model to 'produce an extractive summary' and to 'return only sentences from the original text.' This guarantees near-verbatim reuse and makes the 2.35× speedup unsurprising, but it does not reflect the abstractive summarization setting that the paper's introduction invokes. The same concern applies to the GSM8K self-correction experiments, where the third turn is a direct rewrite of the prior code. The paper should either present these as controlled demonstrations of what CopySpec can do under maximal redundancy or provide evidence on workloads where redundancy is not engineered into the task.","section":"Section 4.2 and Appendix H.2"},{"comment":"The EAGLE comparison is not clearly apples-to-apples. Appendix C states that key–value caching in the EAGLE heads was omitted due to implementation challenges, that EAGLE was set to generate 20-token batches, and that the full method 'EAGLE + CopySpec(γ=5)*' increases the speculation window from 10 to 50 tokens. It is not reported whether the EAGLE baseline was run with the same batching and KV-cache configuration; if not, the throughput advantage attributed to CopySpec may partly reflect these implementation differences. This is especially important because Section 6.2 uses the EAGLE+CopySpec numbers to argue that CopySpec 'complements' strong drafters such as EAGLE.","section":"Appendix C, Section 6.2, Tables 6–8"},{"comment":"The per-category MT-Redundant results in Tables 2 and 9–12 are based on at most 10 conversations per category (MT-Bench has 10 questions per category), yet the tables report standard-error-sized values such as 4.62 ± 0.01 and use these to claim speedups as high as 3.08×. No significance tests or confidence intervals are provided, and the reported uncertainty is implausibly small for categorical subsets of this size. The empirical claim would be strengthened by reporting per-sample variability, confidence intervals, or at least a statement of how many generations were pooled to compute each mean and standard deviation.","section":"Tables 2, 9–12, and 3"}],"minor_comments":[{"comment":"Sections 6.1 and 6.2 contain overlapping text and inconsistent numbers: the former says '2.4× throughput on Vicuna-v1.3-7B and 3.1× on Vicuna-v1.3-13B,' while the latter says '2.9× on Vicuna-v1.3-7B and 3.6× on Vicuna-v1.3-13B.' Please reconcile these statements and remove the duplicated prose.","section":"Sections 6.1 and 6.2"},{"comment":"The column header 'Both Score' in Table 1 is unexplained; from the caption it appears to be a GPT-4 score for MT-Redundant and MT-Bench, ROUGE-L for CNN/DM, and accuracy for GSM8K and HumanEval, but the header should make this explicit.","section":"Table 1"},{"comment":"The caption of Figure 17 says 'self-correction in code generation on CNN/DM,' but the example is an extractive summarization task, not code generation or self-correction; the caption should be corrected.","section":"Figure 17 caption"},{"comment":"The notation '|Scopyspec = 10|' in Table 4 and Section 3.2 should be written as '|Scopyspec| = 10' to distinguish the block-size hyperparameter from the copied token block itself.","section":"Section 3.2 and Table 4"},{"comment":"The mathematical typesetting in Appendix A is garbled: the product over 'Q (w,C)∈D' is not defined, and the left-γ skip-gram objective should be written with explicit probability notation. Please revise this passage for clarity.","section":"Appendix A"},{"comment":"The abstract says the method was evaluated on 'seven LLMs,' while the main experiments in Table 1 cover five models and the Vicuna experiments appear only in Appendix C; please clarify the main-table versus appendix model counts in the text.","section":"Abstract and Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The exactness argument at temperature 0 is sound, and the appendix is unusually thorough. The main risk is that the headline speedups are measured on a dataset constructed to maximize redundancy, and the paper's conclusions currently generalize beyond that evidence. I believe this is fixable with a recalibrated abstract/conclusion and either additional organic-workload experiments or an explicit framing of MT-Redundant as a stress test rather than a representative benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CopySpec is a straightforward, well-implemented system for exploiting repetitive text during LLM inference. The core idea—hash the last gamma tokens, copy the following block from context if matched, verify with the target model—is simple, and the exactness argument under greedy decoding is correct: copying only proposes tokens, and verification means the output distribution is unchanged. That part holds up. The paper also gives credit where due: it cites PLD, PLD+, and SAM-D, ships code and data, and compares against those baselines plus EAGLE. The integration with vanilla speculative decoding is clean, and the look at overhead (Table 4) is a nice touch. The authors are honest in the Limitations section that the method assumes repeated content appears in close proximity with high lexical overlap.\n\nThe soft spot is the headline numbers. The 3.08x second-turn speedup and the 49% boost over speculative decoding come from MT-Redundant, a benchmark the authors constructed by rewriting MT-Bench second turns to ask for variations of the first answer. That guarantees high lexical overlap. On unmodified MT-Bench, standalone speedup is only about 1.15-1.3x, and the gain over speculative decoding is 10-30%, not 49%. CNN/DM's 2.35x is inflated by an extractive-summarization prompt. So the central empirical premise—that organic multi-turn and RAG workloads contain enough copyable blocks to make this worthwhile—is not established by the headline numbers. The stress-test note is correct about this, and the paper itself concedes the assumption.\n\nThere are also some smaller issues. The hyperparameters gamma, block size, and draft length are tuned via sweeps on the evaluation sets, and the paper does not show held-out tuning. Sample sizes for CNN/DM and GSM8K are 100, which is acceptable for a systems paper but worth noting. Quality metrics are reported without error bars.\n\nOverall, this is a legitimate contribution for the LLM inference audience. It is simple, memory-free, and composes with existing methods. The main claim is conditional: it works when the workload contains repetition, and the paper has not yet shown that organic workloads do. I would send this to peer review, with the reviewer asked to push for validation on naturally occurring repetition workloads and held-out hyperparameter selection.","headline":"A clean, honest systems paper whose headline speedups are measured on a self-created redundancy-heavy benchmark; the method works as advertised, but generalization to organic workloads is unproven.","tokens_in":34581,"tokens_out":2344,"would_cite":true,"duration_ms":23292,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM inference speeds up when repeated text in context is copied as a draft and verified, with no quality loss.","keywords":["speculative decoding","LLM inference acceleration","copy mechanism","multi-turn conversation","token repetition","rolling hash","MT-Redundant","self-correction"],"falsifier":"Run CopySpec on a corpus of open-ended chit-chat or long-form fiction where responses rarely repeat the prompt verbatim, and measure the fraction of copied tokens and tokens-per-second; if the copy acceptance rate drops to near zero and throughput falls to or below baseline, the repetition assumption fails and the speed-up disappears.","tokens_in":33594,"feed_emoji":"⚡","tokens_out":3448,"duration_ms":34123,"temperature":0.7,"pith_summary":"CopySpec is a decoding-time accelerator that exploits a simple regularity: when the last few tokens an LLM has just generated already occurred earlier in its context, the text that followed that earlier occurrence is likely to follow again. The method keeps a rolling hash of all $\\gamma$-token substrings in the context, and when the current suffix matches, it copies the next block from context as a draft, which the target LLM then verifies exactly as in speculative decoding. Its central claim is that this copy-and-paste draft is nearly free, requiring no draft model and no additional GPU memory, and that it becomes more useful as context grows, exactly where ordinary decoding slows down. On their MT-Redundant benchmark of second-turn revision requests, the paper reports up to 3.08x faster second-turn inference, and adding CopySpec on top of speculative decoding yields an average 49% additional speed-up without altering output quality.","feed_headline":"Copy-paste speculation speeds LLM inference up to 3.08x","feed_subtitle":"Reusing repeated text from context as draft tokens adds a 49% boost over speculative decoding with no quality loss.","key_machinery":"The central object is the $\\gamma$-token suffix hash map: a rolling hash of every $\\gamma$-long token subsequence in the context. The mechanism works by searching this map for the last $\\gamma$ generated tokens, extracting the following block of up to $|\\text{Scopyspec}|$ tokens from the matched location, and submitting that block to the target LLM for verification. The hash map makes both match lookup and dictionary updates cost $O(\\gamma)$, independent of context length, so the overhead stays small and the copy opportunity grows with context size.","core_discovery":"The paper claims that verbatim repetition in an LLM's context can be turned into a draft source for speculative decoding. CopySpec maintains a hash map of all $\\gamma$-token substrings in the accepted context; once the last $\\gamma$ generated tokens match an earlier position, it copies the following block, called Scopyspec, treats those tokens as drafts with probability one, and verifies them with the target model. Accepted tokens are kept, rejected tokens are discarded, and key-value states are truncated to the accepted prefix. This copying mechanism is orthogonal to existing speculative decoders, so it can be layered on top of them; the paper's headline results are up to 3.08x speed-up on second-turn MT-Redundant inference, a 49% average boost over vanilla speculative decoding, and consistent gains across five LLMs and five datasets, with quality scores unchanged.","pith_inferences":["The method's value depends on how often real workloads contain exact $\\gamma$-token repeats followed by reused text; open-ended dialogue and abstractive writing may show far smaller gains than the benchmark suite suggests.","A natural extension is to select among multiple matches using semantic similarity or recency rather than the first match, which the paper lists as future work and which could improve copy quality on noisy contexts.","Since the hash map already tracks all $\\gamma$-substrings, a testable extension is to report the distribution of match lengths and rejection rates on production chat logs, giving a direct estimate of CopySpec's ceiling on general traffic.","The measured 49% boost over speculative decoding is specific to MT-Redundant second turns; a fair deployment question is whether serving systems can detect redundancy on the fly and enable copying only when it pays."],"forward_implications":["CopySpec reverses the usual trend of slower inference with growing context: later turns, which have richer histories, benefit more than first turns.","The method can be combined with existing speculative decoding frameworks such as vanilla speculative decoding and EAGLE, adding speed-ups beyond what the drafter alone achieves.","Standalone CopySpec requires no draft model and no additional GPU memory, so it can be deployed as a lightweight wrapper around an existing LLM.","Tasks with high lexical overlap, such as extraction, code revision, and self-correction, see the largest gains, while low-redundancy tasks like MT-Bench show modest but still positive speed-ups.","Copying verified text from context does not change the final output distribution compared with greedy decoding, apart from avoiding rejected tokens."],"supporting_citations":[{"why":"Supplies the speculative decoding verification procedure that CopySpec borrows for validating copied tokens.","marker":"Leviathan et al., 2023"},{"why":"Provides MT-Bench prompts and the GPT-4 judging protocol used to build and evaluate MT-Redundant.","marker":"Zheng et al., 2023"},{"why":"Introduces prompt lookup decoding, a baseline restricted to prompt-boundary n-grams that CopySpec generalizes.","marker":"Saxena, 2023"},{"why":"PLD+ extends prompt lookup decoding and is used as a comparison baseline in the head-to-head experiments.","marker":"Somasundaram et al., 2024"},{"why":"SAM-D is the closest suffix-alignment competitor and is compared in appendix experiments.","marker":"Hu et al., 2024"},{"why":"EAGLE is the external drafter used to demonstrate CopySpec's synergy with strong speculative decoding frameworks.","marker":"Li et al., 2025"},{"why":"Self-refine provides the iterative refinement framework for the GSM8K self-correction experiments.","marker":"Madaan et al., 2023"},{"why":"HumanEval is the code-generation benchmark used to measure CopySpec on code tasks.","marker":"Chen et al., 2021"}],"fun_headline_variants":["CopySpec reuses repeated text for 3x faster LLM inference","Speculate by copying: up to 3x speedup without quality loss","LLM speed boost: copy-paste from context nets 3.08x","Turn context repeats into speed: CopySpec adds 49% to speculative decoding","Reuse repetitions for LLM speedups: up to 3.08x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole speed-up rests on the assumption that if the last $\\gamma$ tokens of the LLM's output already appear in the context, the tokens that followed them there are likely to follow again in the output.","fun_headline_variants_meta":{"raw":{"variants":["CopySpec reuses repeated text for 3x faster LLM inference","Speculate by copying: up to 3x speedup without quality loss","LLM speed boost: copy-paste from context nets 3.08x","Turn context repeats into speed: CopySpec adds 49% to speculative decoding","Reuse repetitions for LLM speedups: up to 3.08x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3459,"prompt_tokens":1011,"completion_tokens":2448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":2346}},"tokens_in":627,"tokens_out":2448,"duration_ms":17919,"temperature":1.0,"reasoning_tokens":2346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:11:35.486002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CopySpec on a corpus of open-ended chit-chat or long-form fiction where responses rarely repeat the prompt verbatim, and measure the fraction of copied tokens and tokens-per-second; if the copy acceptance rate drops to near zero and throughput falls to or below baseline, the repetition assumption fails and the speed-up disappears.","supporting_citations":[],"review_version":1}