Beyond the Target: From Imitation to Collaboration in Speculative Decoding

Dong Li; Edith C.H. Ngai; Emad Barsoum; Guanchen Li; Jinfeng Xu; Jinze Li; Shuo Yang; Xuanwu Yin; Yang Zhang; Yixing Xu

arxiv: 2605.24793 · v1 · pith:APLPRCPCnew · submitted 2026-05-24 · 💻 cs.CL

Beyond the Target: From Imitation to Collaboration in Speculative Decoding

Jinze Li , Yixing Xu , Guanchen Li , Jinfeng Xu , Shuo Yang , Yang Zhang , Xuanwu Yin , Dong Li

show 2 more authors

Edith C.H. Ngai Emad Barsoum

This is my paper

Pith reviewed 2026-06-30 12:35 UTC · model grok-4.3

classification 💻 cs.CL

keywords speculative decodingcollaborative decodingreinforcement learningarbitration policydraft modeltarget modelLLM inferencetoken acceptance

0 comments

The pith

Reinforcement learning arbitration lets draft tokens override the target at mismatches to raise accuracy while keeping speculative speedups.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard speculative decoding always follows the target model when draft and target disagree, but the paper shows this rule is not optimal because the smaller draft model is sometimes the one that leads to the correct final answer. CoSpec replaces the strict match rule with a learned arbitration policy trained by reinforcement learning that chooses at each disagreement whether the draft or target token is better for the end result. The policy is applied only at mismatch points during the parallel verification step, so the core speedup mechanism stays intact. Experiments indicate the method delivers both the expected acceleration and higher task performance than target-only decoding. A reader would care because the work questions the default assumption that the larger model is always the better choice at the token level and shows a practical way to let two models collaborate instead of one imitating the other.

Core claim

CoSpec generalizes speculative decoding by training an arbitration policy via reinforcement learning to decide whether to accept tokens from the draft or target model, selectively accepting draft tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance.

What carries the argument

The reinforcement learning arbitration policy that chooses between draft and target tokens at disagreement points during verification.

If this is right

Speculative decoding no longer requires treating the target model as the sole token-level authority.
Draft tokens can be accepted even when they differ from the target prediction if they improve the final output.
Inference speed gains from parallel verification remain available while task performance exceeds target-only baselines.
The shift from imitation to selective collaboration changes how draft and target models interact at every step.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The arbitration approach could be tested on other parallel or tree-based decoding methods that also generate candidate tokens.
The policy might be made more general by training it across multiple model size pairs rather than one fixed draft-target pair.
Extending the same decision process to three or more models could allow finer selection among competing token choices.

Load-bearing premise

A meaningful fraction of draft-target disagreements have the draft token leading to the correct final answer, and an RL policy can learn to pick those cases without overhead that cancels the speedup.

What would settle it

Running the trained arbitration policy on held-out data and measuring whether accuracy stays at or below target-only levels or whether total wall-clock time per token increases compared with standard speculative decoding.

Figures

Figures reproduced from arXiv: 2605.24793 by Dong Li, Edith C.H. Ngai, Emad Barsoum, Guanchen Li, Jinfeng Xu, Jinze Li, Shuo Yang, Xuanwu Yin, Yang Zhang, Yixing Xu.

**Figure 1.** Figure 1: Illustration of the paradigm shift from imitation to collaboration. While existing SPD methods are strictly confined to the target model’s distribution, CoSpec leverages the complementary strengths of the draft model (highlighted in red) to surpass the performance ceiling of the target model alone. Large language models deliver strong performance across reasoning, code, dialogue, and knowledge-intensive… view at source ↗

**Figure 2.** Figure 2: Overview of CoSpec. The draft proposes a block and the target verifies it in one teacher [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

read the original abstract

Speculative decoding (SPD) accelerates large language model (LLM) inference by letting a smaller draft model propose multiple future tokens that are verified in parallel by a larger target model. The dominant SPD paradigm treats the target model as the sole reliable teacher, accepting a draft token only when it exactly matches the target prediction. This design implicitly assumes that the target is always the better choice at every position. In practice, this assumption does not hold. Although the draft is the weaker model overall, it is not uniformly inferior at the token level. In a meaningful fraction of cases where draft and target disagree, the draft's choice is the one that leads to the correct final answer. Inspired by this, we introduce \textbf{Collaborative Speculative Decoding (CoSpec)}, a generalization of SPD that no longer treats the target model as the sole token-level authority. CoSpec trains an arbitration policy via reinforcement learning to decide whether to accept tokens from the draft or target model, selectively accepting draft tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance. By shifting the emphasis from imitation to collaboration, CoSpec suggests a new perspective on speculative decoding.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

CoSpec trains an RL policy to pick draft tokens at mismatches when they lead to correct answers, but the abstract gives no metrics or training details and the label requirement looks like a real limit.

read the letter

The main new thing is moving SPD away from always deferring to the target at disagreements. They train a policy with RL to accept the draft token instead when that choice produces the right final answer. That is a clear step past the standard target-only imitation setup.

The observation that the draft is sometimes locally better is fair and worth exploring. Framing it as collaboration rather than pure imitation is a reasonable reframing.

The soft spots are the missing pieces. The abstract claims maintained speedups plus better performance, yet supplies no numbers, no baselines, no reward formulation, and no account of what tasks were used for the RL training. The stress-test concern holds up on the given text: the reward needs verifiable ground truth, which restricts the method to labeled tasks like QA or math and leaves open-ended generation unaddressed. Without those details it is impossible to judge whether the policy adds overhead that eats the speedup or whether it generalizes.

This is for people already working on speculative decoding and inference optimization. A reader who wants to see the next variant tried would find it worth a look if the experiments are solid.

It deserves peer review so the experimental setup and any generalization claims can be checked directly.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes Collaborative Speculative Decoding (CoSpec) as a generalization of standard speculative decoding. Rather than always accepting a draft token only on exact match with the target model, CoSpec trains an RL arbitration policy to decide at mismatches whether to accept the draft token when doing so yields a correct final answer. The central claim is that this collaborative approach maintains the inference speedups of speculative decoding while surpassing the performance of target-only decoding.

Significance. If the empirical claims are substantiated with quantitative results, the work would be significant for shifting speculative decoding from an imitation paradigm (target as sole authority) to a collaborative one. It directly challenges the uniform-superiority assumption at the token level and opens a path for smaller models to contribute positively even in disagreement cases, provided the setting admits verifiable ground-truth rewards.

major comments (2)

[Abstract] Abstract: the claim that 'Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance' is presented without any metrics, speedup ratios, accuracy deltas, baselines, or task list. This absence is load-bearing because the performance-superiority assertion is the primary empirical contribution.
[Method (RL arbitration)] RL arbitration description: the reward signal is defined via ground-truth correctness of the final answer, which presupposes labeled data (exact-match QA/math/coding benchmarks). The manuscript provides no details on the reward formulation, the tasks used to train the policy, or any mechanism for applying or evaluating the policy in unlabeled open-ended generation; this directly limits the scope of the 'surpassing target-only performance' claim.

minor comments (1)

[Introduction] The abstract and introduction would benefit from a short related-work paragraph situating CoSpec against prior speculative-decoding variants that relax exact-match acceptance (e.g., those using confidence thresholds or tree-based verification).

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We respond point by point to the major comments and indicate planned revisions.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that 'Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance' is presented without any metrics, speedup ratios, accuracy deltas, baselines, or task list. This absence is load-bearing because the performance-superiority assertion is the primary empirical contribution.

Authors: We agree that the abstract would be strengthened by including concrete metrics. The revised abstract will report specific speedup ratios, accuracy deltas relative to target-only decoding, the baselines used, and the task list. revision: yes
Referee: [Method (RL arbitration)] RL arbitration description: the reward signal is defined via ground-truth correctness of the final answer, which presupposes labeled data (exact-match QA/math/coding benchmarks). The manuscript provides no details on the reward formulation, the tasks used to train the policy, or any mechanism for applying or evaluating the policy in unlabeled open-ended generation; this directly limits the scope of the 'surpassing target-only performance' claim.

Authors: The reward is computed from exact-match correctness of the final answer on labeled benchmarks. The revised method section will include the precise reward formulation, the full list of training and evaluation tasks (QA, math, and coding benchmarks), and training details. We acknowledge that the framework requires verifiable ground-truth rewards and does not provide a mechanism for unlabeled open-ended generation; the revised manuscript will explicitly limit the performance-superiority claim to labeled settings with ground-truth rewards. revision: partial

Circularity Check

0 steps flagged

No circularity detected in claimed derivation

full rationale

The paper describes CoSpec as training an RL arbitration policy on external ground-truth correctness signals from labeled benchmarks to decide draft vs. target acceptance at mismatches. No equations, self-definitional reductions, fitted-input predictions, or load-bearing self-citations are present in the provided text that would make the performance claims equivalent to the method's own inputs by construction. The central mechanism relies on independent training data and verifiable external rewards, rendering the approach self-contained against external benchmarks rather than tautological.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review yields minimal ledger entries; the central domain assumption is the non-uniform token-level superiority of the target model.

axioms (1)

domain assumption In a meaningful fraction of draft-target disagreements, the draft token leads to the correct final answer.
Explicitly stated as the inspiration for moving beyond target-only acceptance.

pith-pipeline@v0.9.1-grok · 5780 in / 1137 out tokens · 32426 ms · 2026-06-30T12:35:18.807556+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages · 2 internal anchors

[1]

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D

URLhttps://arxiv.org/abs/2501.19309. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774, 2024. URLhttps://arxiv.org/abs/2401.10774. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespia...

work page arXiv 2024
[2]

Evaluating Large Language Models Trained on Code

URLhttps://arxiv.org/abs/2107.03374. Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chen-Chuan Chang, and Jie Huang. Cascade speculative drafting for even faster llm inference. InAdvances in Neural Information Processing Systems, 2024b. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/hash/9cb5b083ba4f5ca6bd05dd307a2fb354-Abstract-Co...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[3]

Training Verifiers to Solve Math Word Problems

URLhttps://arxiv.org/abs/2110.14168. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in large language models via rein- forcement learning.arXiv preprint arXiv:2501.12948, 2025. URL https://arxiv.org/abs/ 2501.12948. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V . S. Lakshmanan, and Ahmed Hassan ...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv 2025

[1] [1]

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D

URLhttps://arxiv.org/abs/2501.19309. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774, 2024. URLhttps://arxiv.org/abs/2401.10774. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespia...

work page arXiv 2024

[2] [2]

Evaluating Large Language Models Trained on Code

URLhttps://arxiv.org/abs/2107.03374. Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chen-Chuan Chang, and Jie Huang. Cascade speculative drafting for even faster llm inference. InAdvances in Neural Information Processing Systems, 2024b. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/hash/9cb5b083ba4f5ca6bd05dd307a2fb354-Abstract-Co...

work page internal anchor Pith review Pith/arXiv arXiv 2024

[3] [3]

Training Verifiers to Solve Math Word Problems

URLhttps://arxiv.org/abs/2110.14168. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in large language models via rein- forcement learning.arXiv preprint arXiv:2501.12948, 2025. URL https://arxiv.org/abs/ 2501.12948. Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V . S. Lakshmanan, and Ahmed Hassan ...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv 2025