Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

POSS: Position Specialist Generates Better Draft for Speculative Decoding

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Position specialists boost LLM draft acceptance and speed

desk verdict Useful pos-acc diagnostic and a plausible specialization idea, but the missing matched-capacity baseline keeps the mechanism claim unproven and the empirical reporting is a bit cherry-picked. read the letter →

arxiv 2506.03566 v1 pith:GUSABTS5 submitted 2025-06-04 cs.CL cs.AIcs.LG

Langlin Huang , Chengsong Huang , Jixuan Leng , Di Huang , Jiaxin Huang This is my paper
classification cs.CLcs.AIcs.LG
keywords speculativedecodingpositionspecialistsdraftmodelposition-wiseacceptanceratefeaturedeviationLLMinferenceaccelerationlength
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding speeds up LLM generation by having a small draft model propose several tokens that a large target model then verifies in parallel. The paper claims that draft-token quality collapses at later positions because the draft model's own features drift further from the target model's features, and that the fix is to assign each draft position its own 'position specialist' layer trained only for that position. On Llama-3-8B-Instruct and Llama-2-13B-chat across six datasets, the proposed POSS method raises average acceptance length by up to about 4.5% and speed-up ratio by up to about 5.7% over the HASS baseline. The idea matters because higher acceptance at later positions means fewer verification rounds and faster decoding without changing the target model.

What carries the argument

The central object is the position-specialized draft layer together with the position-wise acceptance rate (pos-acc) chain rule. In POSS-$n$, each specialist layer handles $n$ consecutive draft positions and is trained on the feature output of the preceding specialist, so it sees only the noise level typical of its own position; training combines token-level cross-entropy, Smooth L1 feature matching, and Top-K distillation losses. The chain rule $\mathrm{pos\text{-}acc}_i = P(A_i)/P(A_{i-1})$ shows that the probability of accepting $k$ draft tokens equals $P(A_1)$ times the product of the conditional acceptance rates, which explains why a drop at any one position is multiplied across the whole draft and why keeping late-position accuracy high is the lever for longer acceptance length.

What would settle it

Train a single HASS-style draft model with total parameter count and training epochs matched to those of the full POSS setup, then compare average acceptance length and pos-acc on the same six datasets. If the single model matches or beats POSS, the claim that position specialization itself is what helps would be falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the degradation of draft-token quality at later positions in speculative decoding is caused by accumulated feature deviation, and that this can be addressed by assigning each draft position (or small group of positions) to its own 'position specialist' layer rather than asking one layer to cover all positions. Each specialist is trained autoregressively on features produced by the preceding specialist, so it learns to handle the narrow, expected amount of noise at its own position. The introduced metric, position-wise acceptance rate (pos-acc), is the conditional probability of accepting the i-th draft token given the previous token was accepted; because overall acceptance length factors into a product of these conditional probabilities, a drop at any position multiplies into the whole draft. The experiments show POSS variants keep pos-acc relatively flat at deep positions where EAGLE-2 and HASS fall off, and this translates into higher average acceptance length and speed-up ratio.

Load-bearing premise

The paper's central claim rests on the assumption that the measured gains come from position specialization rather than from simply having more draft-model parameters or more training, a control that the experiments do not include.

Editorial extensions

If this is right

  • Higher position-wise acceptance rates at later positions directly raise average acceptance length per drafting round.
  • Fewer drafting rounds are needed to finish a generation, which is the mechanism behind the measured speed-up ratio.
  • The improvements hold across six datasets and two target model sizes, Llama-3-8B-Instruct and Llama-2-13B-chat.
  • POSS-3, which adds the fewest specialist layers, still outperforms HASS, showing that even a small number of specialists shifts the trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested alternative is that a single draft model with the same total parameter count and training budget would match POSS; the paper's experiments do not include this control, so the specialization mechanism is not yet isolated from added capacity.
  • Equal-width position blocks (POSS-n) may not be optimal; allocating more specialists to the deepest positions, where feature deviation accumulates fastest, could give larger acceptance gains.
  • The pos-acc chain-rule view suggests that other late-position fixes, such as feature alignment or position-conditioned adapters, could be combined with specialists to push acceptance length further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes POSS, a speculative decoding draft model composed of position-specialized Transformer layers, where each specialist is trained to predict tokens at one or several fixed draft positions. It introduces position-wise acceptance rate (pos-acc) as a diagnostic and argues that a single draft layer's accuracy decays at later positions because of accumulated feature deviation, while specialists can focus on a narrower band of deviation. Experiments on Llama-3-8B-Instruct and Llama-2-13B-Chat across six datasets report higher average acceptance length and speed-up ratio over EAGLE-2 and HASS, supported by analyses of pos-acc, computational overhead, and draft depth. The code is released.

Significance. The pos-acc metric is a clean and useful diagnostic for speculative decoding, and the idea of decomposing drafting by position is intuitive and could be practically valuable. The paper reports consistent, though modest, acceptance-length gains over strong baselines, and the efficiency analysis in Figure 5 attempts to quantify the overhead of additional specialists. Releasing the code is a strength. However, the experiments do not isolate the specialization mechanism from added model capacity and additional training, so the central claim that position specialization itself drives the gains is not yet established.

major comments (4)
  1. [§3.2, §3.3, §4.1, Table 1] The main claim that position specialization improves later-position acceptance is not isolated experimentally. In the 8B target setting the draft depth is 6 (Section 4.1), so POSS-1, POSS-2, and POSS-3 deploy 6, 3, and 2 Transformer layers respectively (Section 3.3), whereas EAGLE-2 and HASS use a single layer. The higher pos-acc of POSS in Figures 2 and 4 and the higher acceptance lengths in Table 1 could therefore result simply from a larger, more expressive draft model. A matched-capacity control is missing: for example, a single HASS-style draft model with 2, 3, or 6 layers, trained with the same initialization and the same total number of update steps, should be compared against POSS-n. Without such a control, the mechanism story illustrated in Figure 3 is unsupported.
  2. [§4.1, Table 2] The speed-up ratio, one of the two headline metrics, is computed after selecting the fastest of three runs per method: the paper states 'We repeat all experiments 3 times and select the fastest one among each method for speed-up ratio calculation.' This selection rule can systematically favor methods with higher run-to-run variance, and no standard deviation or significance test is reported. The reported speed-up advantages over HASS are often small (0.01x to 0.09x on average in Table 2), so this reporting choice is load-bearing for the claim that POSS 'consistently outperforms' baselines. The authors should report mean and variance over the three runs, or provide a clear justification for the fastest-run selection.
  3. [Appendix B] The training budgets are not matched. HASS is trained for 40 epochs from scratch, while POSS is initialized from the reproduced EAGLE-2 checkpoint and trained for an additional 20 epochs. This gives POSS a warm start from EAGLE-2 training in addition to its own 20 epochs, whereas HASS must learn the same mapping from scratch. The statement in Appendix B that this 'fairly compares' with HASS is therefore difficult to maintain. The authors should either train all methods from the same initialization for the same number of update steps or show that the comparison is insensitive to the initialization and training schedule.
  4. [Abstract, Table 1, Table 2] The abstract reports improvements of 'up to 4.5%' for acceptance length and 'up to 5.7%' for speed-up ratio, but these are per-dataset maxima rather than average gains. The average improvements over HASS are smaller, and in some settings negligible (e.g., Llama-2-13B at temperature 0: HASS 3.33x vs POSS-3 3.34x in Table 2). The authors should report average relative improvements and indicate where differences are within run-to-run noise.
minor comments (4)
  1. [Figure 4] The legend uses 'PosS' while the main text and other figures use 'POSS'; please use a single consistent notation.
  2. [Tables 1 and 2] The shorthand 'L3 8B' and 'L2 13B' is not defined in the captions; please spell out the model names at least once in each caption.
  3. [Appendix A] The limitations section mentions only memory and KV-cache overhead; it does not acknowledge the capacity and training-budget confound discussed above, which should be listed as a limitation or addressed experimentally.
  4. [Appendix D] The phrase 'EAGLE-2 and HASS... are de facto POSS-∞' is imprecise, since EAGLE-2 and HASS share one layer across all positions rather than using infinitely many specialists; please rephrase.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: pos-acc is a chain-rule decomposition and the POSS gains are externally measured, not fitted or self-cited into existence.

full rationale

The paper's derivation chain is self-contained. The pos-acc decomposition (Eqs. 3-4) is a direct application of conditional probability with the nesting A_i ⊂ A_{i-1}, not an assumption that encodes the conclusion. The POSS training losses (Eqs. 5-10) are standard token/feature cross-entropy and Smooth L1 objectives whose targets are target-model features and tokens, independently of the reported acceptance-length and speed-up metrics. The empirical pos-acc curves in Figures 2 and 4 are measurements, not quantities fitted to the headline Tables 1-2. No parameter is fitted to the evaluation metric, no uniqueness theorem is imported, and the few self-citations (Refs. 35-36) appear only in related work and are not load-bearing. The absence of a matched-capacity single-draft-model baseline is a possible confound for the mechanism claim but is not circularity, since the comparison is externally measured rather than true by construction. Score 0.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

No new physical entities are postulated. The 'position specialist' is a concrete trainable Transformer layer. The main free parameters are hyperparameters and training choices, while the key unverified assumption is monotonic growth of feature deviation with position.

free parameters (7)
  • number of positions per specialist n = 1, 2, 3
    Chosen by hand; controls the number of specialist layers. No principled selection criterion is given.
  • draft tree depth = 6 for Llama-3-8B, 7 for Llama-2-13B
    Selected per model as the depth with best performance in the hyperparameter sweep, which is a test-set-dependent choice.
  • draft tree width = 10
    Fixed at 10 for all experiments following the tree-drafting strategy.
  • total draft tokens per round = 60
    Fixed at 60; the appendix shows that 80 tokens sharply reduces speedup.
  • token-level loss weight w = 0.1
    Taken from prior work and not tuned here.
  • Top-K distillation K = 10
    Taken from HASS and not tuned.
  • training epochs for POSS = 20 additional epochs from the EAGLE-2 checkpoint
    Training budget choice; HASS was trained for 40 epochs, so the total optimization history differs.
assumptions (6)
  • standard math Standard chain rule of probability applies to acceptance events because accepted tokens are nested (A_i subset A_{i-1}).
    Used in Equations 3 and 4 to decompose overall acceptance into a product of position-wise acceptance rates.
  • domain assumption Target model hidden states are extractable and usable as features for draft prediction at inference time.
    All hidden-state-assisted methods rely on this; no proof is given in this paper.
  • ad hoc to paper Feature deviation between draft-generated and target features increases monotonically with draft position.
    This is the central motivation for position specialists; the paper does not directly measure ||f_D - f_T|| but asserts it in Section 3.2 and Figure 3.
  • domain assumption Training on ShareGPT and its distilled version transfers to the six evaluation datasets.
    Training and test data differ; no domain-shift analysis is provided.
  • ad hoc to paper Each position specialist's assigned feature-deviation band is narrow enough that a single Transformer layer can learn it.
    The fixed grouping of positions is a design choice; no evidence proves that this grouping is optimal or that specialists do not interfere.
  • domain assumption Speculative decoding verification preserves the target distribution.
    Standard property of rejection sampling in speculative decoding, inherited from prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of POSS: Position Specialist Generates Better Draft for Speculative Decoding." pith.science (2026). https://pith.science/paper/GUSABTS5

@misc{pith2026250603566,
  author       = {Pith},
  title        = {Pith review of: POSS: Position Specialist Generates Better Draft for Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GUSABTS5}},
  note         = {Machine review of arXiv:2506.03566}
}
read the original abstract

Speculative decoding accelerates Large Language Model (LLM) inference by using a small draft model to predict multiple tokens, and a large target model to verify these tokens in parallel. Recent studies leverage the hidden state of the target model to enhance draft model prediction accuracy. However, existing methods suffer from the degrading quality of draft token predictions at later positions, due to error accumulation in draft model generated features. In this paper, we propose Position Specialists (PosS), which consist of multiple position-specialized draft layers to generate tokens at assigned position(s). Position specialists greatly improve token acceptance rate at later positions per drafting round, as each specialist only needs to focus on handling a certain level of draft model feature deviation. Experiment results on Llama-3-8B-Instruct and Llama-2-13B-chat across six datasets demonstrate that PosS effectively improves over baselines on average acceptance length and speed-up ratio. Our codebase is available at https://github.com/shrango/PosS.

Figures

Figures reproduced from arXiv: 2506.03566 by the authors.

Figure 1
Figure 1. The inference and training stages of EAGLE-2, HASS, and our [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Position-wise acceptance rate (pos￾acc) of the i th token on MT_Bench dataset by various speculative decoding methods. The pos-acc of EAGLE-2 and HASS decays fast as the draft sequence gets longer. Our pro￾posed POSS method keeps a stable and higher pos-acc even at the deepest position (draft model prediction depth L = 6). This chain rule decomposition reveals that the overall acceptance length depends on the multip… view at source ↗
Figure 3
Figure 3. This figure shows a comparison of hidden state (feature) training between [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The position-wise acceptance rate of EAGLE-2, HASS, and variants of POSS. Experi￾ments are conducted on MT-Bench dataset, with base model Llama-3-8B-Instruct and draft depth=8. Three variations of our method maintain a rela￾tively higher pos-acc even at the 8 th positi…
Figure 5
Figure 5. Figure 5: Computation time of different phases on MT-Bench dataset on different models across [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The throughput and average acceptance length of 4 models on different draft depths. The experiments are conducted on MT-Bench dataset. The acceptance length consistently increases as depth rises, while the throughput peaks at depth=5. Recent works apply other methods t…
Figure 7
Figure 7. Figure 7: The Inference-time GPU memory usage of different speculative decoding methods. The [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PreScience: A Dataset and Benchmark for Scientific Forecasting

    cs.AI 2026-02 conditional novelty 6.0 of 10

    A new benchmark tests whether AI can forecast future scientific papers; frontier LLMs score ~5.6/10 on matching real abstracts, and simulated corpora are measurably less diverse and novel than human science.

Reference graph

Works this paper leans on

39 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning, 2022

    work page 2022

  2. [2]

    Sifre, and John M

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, L. Sifre, and John M. Jumper. Accelerating large language model decoding with speculative sampling.ArXiv, abs/2302.01318, 2023

    arXiv 2023

  3. [3]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024

    arXiv 2024

  4. [4]

    EAGLE: Speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. InInternational Conference on Machine Learning, 2024

    2024

  5. [5]

    EAGLE-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. InEmpirical Methods in Natural Language Processing, 2024

    2024

  6. [6]

    Learning harmonized represen- tations for speculative sampling.arXiv preprint arXiv:2408.15766, 2024

    Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized represen- tations for speculative sampling.arXiv preprint arXiv:2408.15766, 2024

    arXiv 2024

  7. [7]

    H. T. Kung and John T. Robinson. On optimistic methods for concurrency control. InTODS, 1979. 10

    work page 1979

  8. [8]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

Show all 39 references
  1. [9]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  2. [10]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021

    2021

  3. [11]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  4. [12]

    Abstractive text summarization using sequence-to-sequence rnns and beyond, 2016

    Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequence rnns and beyond, 2016

    2016

  5. [13]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  6. [14]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding.arXiv preprint arXiv:2402.12374, 2024

    Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding.arXiv preprint arXiv:2402.12374, 2024

    2024 arXiv

  7. [15]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation.arXiv preprint arXiv:2203.16487, 2022

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation.arXiv preprint arXiv:2203.16487, 2022

    2022 arXiv

  8. [16]

    Token-driven gammatune: Adaptive calibration for enchanced speculative decoding.arXiv preprint arXiv:2504.00030, 2025

    Aayush Gautam, Susav Shrestha, and Narasimha Annapareddy. Token-driven gammatune: Adaptive calibration for enchanced speculative decoding.arXiv preprint arXiv:2504.00030, 2025

    2025 arXiv

  9. [17]

    Specdec++: Boosting speculative decoding via adaptive candidate lengths.arXiv preprint arXiv:2405.19715, 2024

    Kaixuan Huang, Xudong Guo, and Mengdi Wang. Specdec++: Boosting speculative decoding via adaptive candidate lengths.arXiv preprint arXiv:2405.19715, 2024

    2024 arXiv

  10. [18]

    Magicdec: Breaking the latency- throughput tradeoff for long context generation with speculative decoding.arXiv preprint arXiv:2408.11049, 2024

    Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency- throughput tradeoff for long context generation with speculative decoding.arXiv preprint arXiv:2408.1...

    2024 arXiv

  11. [19]

    Better & faster large language models via multi-token prediction.arXiv preprint arXiv:2404.19737, 2024

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better & faster large language models via multi-token prediction.arXiv preprint arXiv:2404.19737, 2024

    2024 arXiv

  12. [20]

    Ems-sd: Efficient multi-sample speculative decoding for accelerating large language models.arXiv preprint arXiv:2405.07542, 2024

    Yunsheng Ni, Chuanjian Liu, Yehui Tang, Kai Han, and Yunhe Wang. Ems-sd: Efficient multi-sample speculative decoding for accelerating large language models.arXiv preprint arXiv:2405.07542, 2024

    2024 arXiv

  13. [21]

    Turning trash into treasure: Accelerating inference of large language models with token recycling.arXiv preprint arXiv:2408.08696, 2024

    Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Turning trash into treasure: Accelerating inference of large language models with token recycling.arXiv preprint arXiv:2408.08696, 2024. 11

    2024 arXiv

  14. [22]

    Lossless acceleration of large language model via adaptive n-gram parallel decoding.arXiv preprint arXiv:2404.08698, 2024

    Jie Ou, Yueming Chen, and Wenhong Tian. Lossless acceleration of large language model via adaptive n-gram parallel decoding.arXiv preprint arXiv:2404.08698, 2024

    2024 arXiv

  15. [23]

    Parallel decoding via hidden transfer for lossless large language model acceleration.arXiv preprint arXiv:2404.12022, 2024

    Pengfei Wu, Jiahao Liu, Zhuocheng Gong, Qifan Wang, Jinpeng Li, Jingang Wang, Xunliang Cai, and Dongyan Zhao. Parallel decoding via hidden transfer for lossless large language model acceleration.arXiv preprint arXiv:2404.12022, 2024

    2024 arXiv

  16. [24]

    Parallel speculative decoding with adaptive draft length.arXiv preprint arXiv:2408.11850, 2024

    Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, and Winston Hu. Parallel speculative decoding with adaptive draft length.arXiv preprint arXiv:2408.11850, 2024

    2024 arXiv

  17. [25]

    Fast and accurate language model decoding via parallel token processing

    Zhepei Wei, Wei-Lin Chen, Xinyu Zhu, and Yu Meng. Fast and accurate language model decoding via parallel token processing. InAdaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, 2024

    2024

  18. [26]

    Griffin: Effective token alignment for faster speculative decoding.arXiv preprint arXiv:2502.11018, 2025

    Shijing Hu, Jingyang Li, Xingyu Xie, Zhihui Lu, Kim-Chuan Toh, and Pan Zhou. Griffin: Effective token alignment for faster speculative decoding.arXiv preprint arXiv:2502.11018, 2025

    2025

  19. [27]

    Specinfer: Accelerating large language model serving with tree-based speculative inference and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. InProceedings of the...

    2024

  20. [28]

    Propd: Dynamic token tree pruning and generation for llm parallel decoding

    Shuzhang Zhong, Zebin Yang, Ruihao Gong, Runsheng Wang, Ru Huang, and Meng Li. Propd: Dynamic token tree pruning and generation for llm parallel decoding. InProceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, pages 1–8, 2024

    2024

  21. [29]

    Layerskip: Enabling early exit inference and self-speculative decoding.arXiv preprint arXiv:2404.16710, 2024

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding.arXiv preprint arXiv:2404.16710, 2024

    2024 arXiv

  22. [30]

    Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding.arXiv preprint arXiv:2404.11912, 2024

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding.arXiv preprint arXiv:2404.11912, 2024

    2024 arXiv

  23. [31]

    Judge decoding: Faster speculative sampling requires going beyond model alignment.arXiv preprint arXiv:2501.19309, 2025

    Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment.arXiv preprint arXiv:2501.19309, 2025

    2025 arXiv

  24. [32]

    Specrea- son: Fast and accurate inference-time compute via speculative reasoning.arXiv preprint arXiv:2504.07891, 2025

    Rui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro, Zhihao Jia, and Ravi Netravali. Specrea- son: Fast and accurate inference-time compute via speculative reasoning.arXiv preprint arXiv:2504.07891, 2025

    2025 arXiv

  25. [33]

    Speculative thinking: En- hancing small-model reasoning with large model guidance at inference time.arXiv preprint arXiv:2504.12329, 2025

    Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. Speculative thinking: En- hancing small-model reasoning with large model guidance at inference time.arXiv preprint arXiv:2504.12329, 2025

    2025 arXiv

  26. [34]

    Efficient reasoning for llms through speculative chain-of-thought.arXiv preprint arXiv:2504.19095, 2025

    Jikai Wang, Juntao Li, Lijun Wu, and Min Zhang. Efficient reasoning for llms through speculative chain-of-thought.arXiv preprint arXiv:2504.19095, 2025

    2025 arXiv

  27. [35]

    Efficient test-time scaling via self-calibration.arXiv preprint arXiv:2503.00031, 2025

    Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration.arXiv preprint arXiv:2503.00031, 2025

    2025 arXiv

  28. [36]

    Divide, reweight, and conquer: A logit arithmetic approach for in-context learning.arXiv preprint arXiv:2410.10074, 2024

    Chengsong Huang, Langlin Huang, and Jiaxin Huang. Divide, reweight, and conquer: A logit arithmetic approach for in-context learning.arXiv preprint arXiv:2410.10074, 2024

    2024 arXiv

  29. [37]

    Non-myopic generation of language models for reasoning and planning.arXiv preprint arXiv:2410.17195, 2024

    Chang Ma, Haiteng Zhao, Junlei Zhang, Junxian He, and Lingpeng Kong. Non-myopic generation of language models for reasoning and planning.arXiv preprint arXiv:2410.17195, 2024

    2024 arXiv

  30. [38]

    Toward effective retrieval augmented generative services in 6g networks.IEEE Network, 38(6):459–467, 2024

    Xi Huang, Yinxu Tang, Junling Li, Ning Zhang, and Xuemin Shen. Toward effective retrieval augmented generative services in 6g networks.IEEE Network, 38(6):459–467, 2024. 12

    2024

  31. [39]

    OPT-tree: Speculative decoding with adaptive draft tree structure.Transactions of the Association for Computational Linguistics, 13:188–199, 2025

    Jikai Wang, Yi Su, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, and Min Zhang. OPT-tree: Speculative decoding with adaptive draft tree structure.Transactions of the Association for Computational Linguistics, 13:188–199, 2025. 13 A Limitations POSSrequires slightly...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.