Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

HAMburger: Accelerating LLM Inference via Token Smashing

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Token smashing fuses multiple tokens into one KV cache, giving up to 2× faster LLM decoding while keeping quality.

desk verdict A plausible self-speculative decoder that fuses low-entropy tokens for ~2x speedup on a 1B model; the sub-linear growth claim is overstated and the evaluation needs stronger baselines. read the letter →

arxiv 2505.20438 v1 pith:YTXR37YD submitted 2025-05-26 cs.CL

Jingyu Liu , Ce Zhang This is my paper
classification cs.CL
keywords LLMinferenceKVcachecompressionmulti-tokendecodingspeculativeconditionalentropytokenfusionhierarchicalautoregressivemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that LLM decoding wastes compute and memory by treating every token as equally costly, giving each token its own forward pass and its own KV cache entry. HAMburger instead has the model itself decide which tokens are easy and can be generated confidently from local information, groups those tokens into a single macro-step, and stores only one fused KV cache for the group. The claim is that such token smashing cuts the growth of KV cache and forward FLOPs from linear to sub-linear in output length, delivering up to a 2× reduction in KV cache computation and up to 2× tokens per second while matching or improving quality. If true, this makes long-context and high-throughput serving cheaper without a separate draft model or lossy KV compression.

What carries the argument

The two grafted modules carry the argument. The relative-position-aware compositional embedder, built as a Perceiver-style cross-attention whose query is the mean of the input token embeddings, maps a list of token embeddings into a single hidden state that is permutation-sensitive; it also offsets position information so the base model knows how many tokens were fused. The micro-step decoder is a small stack of transformer layers that conditions on hidden states from middle layers and the last layer to autoregressively emit micro-step tokens, and a binary stop head decides whether another micro-step or a new macro-step should happen. Around them, dynamic data segmentation uses the base model's conditional entropy to split training responses into segments whose non-first tokens have low entropy, so the model learns exactly what dose of information a single KV cache can hold.

What would settle it

Run HAMburger on a generation task where every token is hard to predict from local context — e.g., emitting random strings, exact long identifiers, UUIDs, or adversarially chosen sequences — and measure both quality and the average macro-step length. If the micro-step decoder is forced back to one token per step whenever entropy is high, the claimed sub-linear growth of KV cache and FLOPs will not survive on such inputs.

Watch

Extended reading notes

Core claim

The central discovery is that a base LLM can be taught to self-segment its own output: a compositional embedder fuses an ordered list of tokens from the previous macro-step into a single embedding, and a micro-step decoder rolls out a small run of tokens plus a binary stop decision before another forward pass of the base model is needed. The segmentation is guided by the base model's own conditional entropy, so tokens that the model is confident about are generated micro-step by micro-step from one set of KV caches, while the first token of each segment is handled by the full base-model forward. HAMburger therefore functions as a self-speculative decoder that blindly trusts its own drafts and pays no verification forward pass. The paper reports that this maintains task quality on instruction following, math, reasoning, code, and long-context benchmarks while compressing KV cache computation by up to 2×.

Load-bearing premise

The whole speedup rests on the premise that tokens the base model deems low-entropy can be predicted correctly from local context alone, so that one fused KV cache holds enough information for the micro-step decoder to produce several correct tokens.

Editorial extensions

If this is right

  • KV cache computation and storage drop by up to 2× on short- and long-context generation, with a tunable confidence threshold trading a little quality for more compression.
  • KV cache and forward FLOPs grow sub-linearly with output length instead of linearly, which matters most when generations or contexts are long.
  • The method works with small base models, a regime where speculative decoding struggles because a fast enough draft model is hard to find.
  • Because it is batch-friendly and hardware-agnostic, the speedup can be combined with orthogonal KV compression or offloading techniques.
  • At inference time the speed adapts to the prompt: easy, low-perplexity queries get more tokens per step and hard queries fall back to single-token steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'easy token' hypothesis suggests a deeper architectural change: future LLMs could be trained from scratch with variable-rate decoding, where each KV cache entry is annotated with the amount of information it stores, rather than patched on after the fact.
  • A testable extension would be to use HAMburger's own predicted stop-head confidence as a calibration signal for per-token uncertainty, turning the fused-cache mechanism into a cheap probe for model knowledge or hallucination risk.
  • The reported speedups are on a 1B model; the ratio of base-model cost to micro-step cost suggests larger bases could see larger absolute savings, but also larger risk if the micro-step decoder's errors compound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes HAMburger, an inference-time augmentation of a base LLM with a compositional embedder and a micro-step decoder. At each macro-step, the embedder fuses the tokens generated in the previous step into a single hidden state, the base model performs one forward pass and stores one KV entry for the fused group, and the micro-step decoder autoregressively emits several tokens plus a stop decision. The method is trained by SFT on data segmented according to the base model's own conditional entropy. The authors claim that this shifts KV-cache and forward-FLOP growth from linear to sub-linear in output length, reduces KV cache computation by up to 2x, improves decoding TPS by up to 2.2x, and maintains or improves quality on short- and long-context tasks.

Significance. If the empirical claims held, HAMburger would be a useful contribution: it is a self-speculative scheme with no verification pass, constant drafting overhead, and a KV-compression mechanism that is orthogonal to prior cache-compression and offloading work. The component ablations in Table 2 are informative, and the use of a public data mix is a strength. However, the headline asymptotic claim is unsupported, and the quality-maintenance claim rests on a small number of aggregate point estimates from a single 1B model. After correction, the contribution is best characterized as a constant-factor acceleration and compression method rather than a change in asymptotic complexity.

major comments (4)
  1. [Abstract; §3.7, Eq. (1)] The claim that KV cache and forward FLOPs grow sub-linearly with output length is not supported. In Eq. (1), n is the number of micro-step tokens per macro-step, and training fixes a maximum micro-step size of 4 (Section 4.1). With n bounded by a constant, the number of KV entries is L/n_avg = Θ(L), and total attention FLOPs remain Θ(L^2) with a smaller constant; no sub-linear scaling follows. The paper should either show that the stop head produces an unbounded and growing n in practice, or replace the asymptotic claim with a constant-factor improvement.
  2. [Table 2; §4.2; §5] The central 'maintaining quality' claim is not established. Table 2 reports a per-token accuracy of 0.93 for non-first micro-tokens, but the paper never reports the per-segment exact-match rate or the downstream effect of a wrong non-first token on the final answer. For a 4-token micro-step, independent per-token errors at 7% would put at least one error in roughly 20% of macro-steps; since HAMburger blindly trusts drafted tokens (Section 3.7) and Section 5 admits that the compositional embedder is not perfect, the paper needs to quantify how often fused segments are faithfully reconstructed and how reconstruction errors propagate into subsequent macro-steps.
  3. [§4.1, §4.2, Table 2] The evaluation is too thin to support the cross-task quality claim. All results come from a single Llama-3.2-1B model with greedy decoding and point estimates, with no error bars or multiple seeds. Comparisons to BLT-8B and to StreamingLLM/TOVA/Quest use different base models and, in the latter case, a single sub-task; there is no comparison to Medusa or EAGLE on the same base. Without these, the statement that HAMburger 'maintains and even surpasses' the base model is a point-estimate observation rather than a demonstrated property.
  4. [§3.5, §4.2, Figures 3–5] The confidence threshold is a free parameter that directly sets the measured compression and speedup, and the paper does not specify a single operating point for the headline claims. Figures 3 and 4 show quality varying with the threshold, so 'maintaining quality while achieving up to 2x' requires reporting quality and speed jointly at one preset threshold; otherwise the two numbers may be selected from different thresholds.
minor comments (6)
  1. [§1, §3.1, §3.5] There are several typos and wording issues: 'Hierachically' should be 'Hierarchically', 'switches between two different phrases' should be 'phases', and 'we don not need' should be 'we do not need'.
  2. [§3.5] The dynamic segmentation rule is described only heuristically ('global statistics', 'predefined threshold', 'low entropies relative to that of the first token'). Please provide the exact algorithm or pseudo-code so the data preparation is reproducible.
  3. [Table 2] The metric 'Eval > 1 Token Acc' should be defined precisely, including which tokens are in the denominator and how the average is computed across segments, so the reader can relate it to per-segment exact-match accuracy.
  4. [§4.3, Figure 5] The speculative-decoding baselines are not fully specified; the paper should state the draft model, gamma values, and hardware/software settings for the TPS numbers, since efficiency claims are system-dependent.
  5. [§4.2.2, Figure 4] The comparison to StreamingLLM, TOVA, and Quest uses a 7B model with a 4096-token KV budget against the authors' 1B HAMburger model; this is an apples-to-oranges comparison and should be labeled as such, or matched in base model and budget.
  6. [Figures 3 and 4] The bottom x-axis label 'KV Cache Compression' is presented as a percentage; please clarify whether a value like 187% means 1.87x compression, 187% of baseline KV size, or another convention.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HAMburger's efficiency gains are empirical and its cost model is an algebraic identity, not a fitted prediction.

full rationale

HAMburger's efficiency claims are grounded in external benchmarks (GSM8K, HumanEval, MGSM, LongBench, etc.) and in a cost model in Section 3.7 that is a stated identity, r := n*C(S)/(C(S)+(n-1)*c). The reported speedups and KV-cache reductions in Figures 3-5 are measured values, not predictions derived from this formula; the number n of micro-step tokens is determined by the trained stop head and the confidence threshold, and the downstream quality scores are evaluated independently of that parametrization. Calling the growth 'sub-linear' is a description of the architecture's token-grouping behavior, not a self-defined outcome used as evidence. The only self-citation is [25] (Speculative Prefill, sharing two authors with this paper), used merely as design inspiration for position offset, and it is not load-bearing; no uniqueness theorem, fitted-then-predicted quantity, or ansatz justified solely by the authors' prior work appears. Section 5's admission that the compositional embedder is not perfect is a limitation about reconstruction error, not a circular step. Accordingly, no load-bearing claim reduces to its own input.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central efficiency claim depends on several hand-chosen knobs: the confidence threshold, the entropy segmentation threshold, MAX_STEPS, and the decoder depth. The key domain assumption is that conditional entropy reliably identifies locally predictable tokens, which is not demonstrated with ablations or theoretical justification.

free parameters (4)
  • Confidence threshold = 50-90% (tunable)
    Controls whether the stop head continues micro-step generation; the reported speedup and KV compression are reported as a function of this threshold, so the headline 2x is achieved at lower confidence levels.
  • Entropy segmentation threshold = not disclosed
    Used in Section 3.5 to group tokens into micro-steps; the value of this pre-defined threshold directly determines how many tokens are fused and hence the speedup.
  • MAX_STEPS = 4
    Maximum number of micro-step tokens per macro-step in training (Section 4.1); bounds the per-step speedup at 4x in the best case.
  • Micro-step decoder layers = 4
    Number of transformer layers grafted as the micro-step decoder, chosen by the authors; increases per-step cost c and model capacity.
assumptions (5)
  • domain assumption Low conditional entropy tokens can be generated accurately without global context.
    The key insight stated in Section 1 and operationalized in Section 3.5; the whole method assumes the base model's entropy reliably identifies tokens that do not need full context.
  • domain assumption A single fused KV cache entry retains enough information to predict subsequent micro-step tokens.
    The compositional embedder collapses a list of token embeddings into one hidden state; the paper assumes this lossy fusion does not destroy information needed later (Section 3.2).
  • ad hoc to paper The base model's conditional entropy is a good proxy for token predictability at inference.
    The segmentation heuristic in Section 3.5 uses the base model's entropy on training data to define groups; this is a heuristic with no theoretical justification or ablation on the threshold.
  • ad hoc to paper Reverting to original computation for single-token inputs acts as sufficient implicit regularization.
    Section 3.4 claims this keeps the embedder and decoder close to the base distribution, but no evidence is given that this is necessary or sufficient.
  • standard math Standard transformer operations (self-attention, cross-attention) behave as expected in the grafted modules.
    The architecture assumes standard implementations of these operations in PyTorch/GPT-Fast.
invented entities (1)
  • Virtual token (smashed token)
    purpose: A group of tokens represented by a single KV cache entry to reduce computation and memory
    This is a conceptual abstraction; its validity is measured only through downstream task performance, not an independently falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HAMburger: Accelerating LLM Inference via Token Smashing." pith.science (2026). https://pith.science/paper/YTXR37YD

@misc{pith2026250520438,
  author       = {Pith},
  title        = {Pith review of: HAMburger: Accelerating LLM Inference via Token Smashing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTXR37YD}},
  note         = {Machine review of arXiv:2505.20438}
}
abstract

The growing demand for efficient Large Language Model (LLM) inference requires a holistic optimization on algorithms, systems, and hardware. However, very few works have fundamentally changed the generation pattern: each token needs one forward pass and one KV cache. This can be sub-optimal because we found that LLMs are extremely capable of self-identifying the exact dose of information that a single KV cache can store, and many tokens can be generated confidently without global context. Based on this insight, we introduce HAMburger, a Hierarchically Auto-regressive Model that redefines resource allocation in LLMs by moving beyond uniform computation and storage per token during inference. Stacking a compositional embedder and a micro-step decoder in between a base LLM, HAMburger smashes multiple tokens into a single KV and generates several tokens per step. Additionally, HAMburger functions as a speculative decoding framework where it can blindly trust self-drafted tokens. As a result, HAMburger shifts the growth of KV cache and forward FLOPs from linear to sub-linear with respect to output length, and adjusts its inference speed based on query perplexity and output structure. Extensive evaluations show that HAMburger reduces the KV cache computation by up to 2$\times$ and achieves up to 2$\times$ TPS, while maintaining quality in both short- and long-context tasks. Our method explores an extremely challenging inference regime that requires both computation- and memory-efficiency with a hardware-agnostic design.

Figures

Figures reproduced from arXiv: 2505.20438 by the authors.

Figure 1
Figure 1. How HAMBURGER Merges Tokens: We showcase two examples of dynamically generat￾ing a “unit dose of information” per step instead of a fixed single token (left). The output tokens from the static vocabulary are distinguished by alternating blues and greens. The red dividers separate groups of tokens that are predicted with a single macro-step by HAMBURGER. During data pre￾processing, we rely on model’s own knowledge (i… view at source ↗
Figure 2
Figure 2. HAMBURGER Overview: HAMBURGER stacks the base model with two additional modules that can fuse and predict multiple tokens per iteration for faster decoding. under-utilizing the full potential of hardware accelerators. To solve these problems, prior works take two major routes [43, 7, 16, 39], namely compressing the cache and retrieve only a subset of them for decoding. SnapKV [20] explores special patterns for each … view at source ↗
Figure 3
Figure 3. Standard Task Evaluation: We present the standard task evaluations for our method, which consists of instruction following, math, reasoning and code. The bottom x-axis denotes the KV cache compression rate, which relates to the decoding TPS speedup. The top x-axis shows a tunable parameter that trades-off efficiency and quality. We shade the green area to be the bearable quality loss. In almost all tasks, HAMBURGER … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Long Context Task Evaluation: We showcase the superior long-context performance of HAMBURGER with LongBench suite. With the same graph settings as [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Efficiency Benchmarking: We compare the decoding tokens per second for HAMBURGER against the baselines. For 1B models, HAMBURGER achieves up to 2.2× decoding TPS speedup over the base model. HAMBURGER also beats speculative sampling with 1B INT8 draft model with even >…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

  2. ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling

    cs.LG 2025-07 conditional novelty 3.0 of 10

    ODIA routes 60% of function-calling traffic in a music app to a small 1.3B model, reducing expected latency by 45% and median latency by 78%.

Reference graph

Works this paper leans on

45 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic. com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2025

    work page 2025

  2. [2]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021

    2021

  3. [3]

    Y . Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y . Dong, J. Tang, and J. Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2024

    work page 2024

  4. [4]

    T. Cai, Y . Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024

    work page 2024

  5. [5]

    C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper. Accelerating large language model decoding with speculative sampling, 2023

    work page 2023

  6. [6]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    2021

  7. [7]

    Z. Chen, R. Sadhukhan, Z. Ye, Y . Zhou, J. Zhang, N. Nolte, Y . Tian, M. Douze, L. Bottou, Z. Jia, and B. Chen. Magicpig: Lsh sampling for efficient llm generation, 2024

    work page 2024

  8. [8]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    2018

Show all 45 references
  1. [9]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021

    2021

  2. [10]

    DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, ...

    2025

  3. [11]

    A. R. Fabbri, I. Li, T. She, S. Li, and D. R. Radev. Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model, 2019

    2019

  4. [12]

    T. Feng, C. Jin, J. Liu, K. Zhu, H. Tu, Z. Cheng, G. Lin, and J. You. How far are we from agi: Are llms all we need?, 2024. 10

    2024

  5. [13]

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. The language...

    2024

  6. [14]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru,...

    2024

  7. [15]

    N. Ho, S. Bae, T. Kim, H. Jo, Y . Kim, T. Schuster, A. Fisch, J. Thorne, and S.-Y . Yun. Block transformer: Global-to-local language modeling for fast inference. arXiv preprint arXiv:2406.02657, 2024

    2024 arXiv

  8. [16]

    Hooper, S

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y . S. Shao, K. Keutzer, and A. Gho- lami. Kvquant: Towards 10 million context length llm inference with kv cache quantization, 2024

    2024

  9. [17]

    Huang, S

    L. Huang, S. Cao, N. Parulian, H. Ji, and L. Wang. Efficient attentions for long document summarization, 2021

    2021

  10. [18]

    Jaegle, F

    A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver: General perception with iterative attention, 2021

    2021

  11. [19]

    Leviathan, M

    Y . Leviathan, M. Kalman, and Y . Matias. Fast inference from transformers via speculative decoding, 2023

    2023

  12. [20]

    Y . Li, Y . Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen. Snapkv: Llm knows what you are looking for before generation, 2024

    2024

  13. [21]

    Y . Li, K. Livescu, and J. Zhou. Chunk-distilled language modeling, 2024

    2024

  14. [22]

    Y . Li, F. Wei, C. Zhang, and H. Zhang. Eagle-2: Faster inference of language models with dynamic draft trees, 2024

    2024

  15. [23]

    Y . Li, F. Wei, C. Zhang, and H. Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty, 2025

    2025

  16. [24]

    Liang, R

    P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y . Zhang, D. Narayanan, Y . Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. ...

    2023

  17. [25]

    J. Liu, B. Chen, and C. Zhang. Speculative prefill: Turbocharging ttft with lightweight and training-free token importance estimation, 2025

    2025

  18. [26]

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  19. [27]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019

    2019

  20. [28]

    X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia. Towards efficient generative large language model serving: A survey from algorithms to systems, 2023. 12

    2023

  21. [29]

    Achiam, S

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V . Balcom, P. Bal- tescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner,...

    2024

  22. [30]

    M. Oren, M. Hassid, N. Yarden, Y . Adi, and R. Schwartz. Transformers are multi-state rnns, 2024

    2024

  23. [31]

    Pagnoni, R

    A. Pagnoni, R. Pasunuru, P. Rodriguez, J. Nguyen, B. Muller, M. Li, C. Zhou, L. Yu, J. Weston, L. Zettlemoyer, G. Ghosh, M. Lewis, A. Holtzman, and S. Iyer. Byte latent transformer: Patches scale better than tokens, 2024

    2024

  24. [32]

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, ...

    2025

  25. [33]

    F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. V osoughi, H. W. Chung, Y . Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners, 2022

    2022

  26. [34]

    Z. Shi, Y . Ming, X.-P. Nguyen, Y . Liang, and S. Joty. Discovering the gems in early layers: Accelerating long-context llms with 1000x input token reduction, 2024

    2024

  27. [35]

    Sun, L.-W

    H. Sun, L.-W. Chang, W. Bao, S. Zheng, N. Zheng, X. Liu, H. Dong, Y . Chi, and B. Chen. Shadowkv: Kv cache in shadows for high-throughput long-context llm inference, 2025

    2025

  28. [36]

    J. Tang, Y . Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han. Quest: Query-aware sparsity for efficient long-context llm inference, 2024. 13

    2024

  29. [37]

    L. team, L. Barrault, P.-A. Duquenne, M. Elbayad, A. Kozhevnikov, B. Alastruey, P. Andrews, M. Coria, G. Couairon, M. R. Costa-jussà, D. Dale, H. Elsahar, K. Heffernan, J. M. Janeiro, T. Tran, C. Ropers, E. Sánchez, R. S. Roman, A. Mourachko, S. Saleem, and H. Schwenk. Large c...

    2024

  30. [38]

    H. Xia, Z. Yang, Q. Dong, P. Wang, Y . Li, T. Ge, T. Liu, W. Li, and Z. Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding, 2024

    2024

  31. [39]

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks, 2024

    2024

  32. [40]

    Xiong, R

    Y . Xiong, R. Zhang, Y . Li, T. Wu, and L. Zou. Dyspec: Faster speculative decoding with dynamic token tree structure, 2024

    2024

  33. [41]

    J. Yao, K. Chen, K. Zhang, J. You, B. Yuan, Z. Wang, and T. Lin. Deft: Decoding with flash tree-attention for efficient tree-structured llm inference, 2025

    2025

  34. [42]

    L. Yu, D. Simig, C. Flaherty, A. Aghajanyan, L. Zettlemoyer, and M. Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers, 2023

    2023

  35. [43]

    Zhang, Y

    Z. Zhang, Y . Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y . Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen. H 2o: Heavy-hitter oracle for efficient generative inference of large language models, 2023

    2023

  36. [44]

    Y . Zhao, Z. Xie, C. Liang, C. Zhuang, and J. Gu. Lookahead: An inference acceleration framework for large language model with lossless generation accuracy, 2024

    2024

  37. [45]

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y . Luan, D. Zhou, and L. Hou. Instruction- following evaluation for large language models, 2023. 14 Table 3: HAM BURGER Training Data Mix: We use all public data to create the data mix for training where we select datasets spanni...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.