Pith. sign in

REVIEW 5 major objections 5 minor 10 cited by

TransMLA: Multi-Head Latent Attention Is All You Need

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GQA models can be rewritten as DeepSeek-style MLA with a ~93 percent smaller KV cache and a 10.6x speedup.

desk verdict A genuinely useful GQA-to-MLA conversion pipeline with real speedups, but the expressiveness proof is incomplete and long-context behavior is unvalidated. read the letter →

arxiv 2502.07864 v5 pith:N3HS76Y2 submitted 2025-02-11 cs.LG cs.AI

Fanxu Meng , Pingzhi Tang , Xiaojuan Tang , Zengwei Yao , Xing Sun , Muhan Zhang This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0768T50
keywords multi-headlatentattentiongroup-queryKVcachecompressionRoPEPCArotationmodelconversionLLMinferencelow-rankfactorization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Group-Query Attention, the shared-key/value design used by LLaMA and Qwen models, is strictly less expressive than Multi-Head Latent Attention at the same KV-cache budget, and that any pretrained GQA model can be converted into an MLA model rather than retrained from scratch. The conversion works by rotating queries and keys so that almost all positional (RoPE) information lands in a single attention head, dropping RoPE from the other key heads, and then compressing the remaining keys and values jointly by a norm-balanced low-rank projection. On LLaMA-2-7B the paper reports a 92.97% KV-cache reduction, a measured 10.6x inference speedup at 8K context on one consumer GPU, and benchmark recovery to near-original level after just 6B fine-tuning tokens. If the conversion is as lossless as the perplexity curves suggest, existing GQA deployments could inherit the optimized inference stack built for DeepSeek without retraining the model from scratch.

What carries the argument

The load-bearing identity is the RoPE rotation invariance: if the same orthogonal matrix $U_l$ is applied to the real and imaginary components of the $l$-th RoPE subspace across all attention heads, the inner product $\hat q^R_{t,i}^\top \hat k^R_j$ is unchanged. RoRoPE uses this identity to run joint PCA on calibration key activations and rotate query/key projections so that principal components from all heads concentrate in the first head's dimensions; FreqFold groups nearby RoPE frequencies as identical so that more than one first-head dimension can carry positional signal. Once RoPE is isolated in one head, the remaining keys and values are compressed by a norm-balanced joint PCA (BKV), and the RoPE-free key up-projection can be absorbed into the query projection exactly as in DeepSeek's MLA, which is what makes the converted model load directly into that inference stack.

What would settle it

Take the converted LLaMA-3-8B model with 90% of RoPE removed and run a needle-in-a-haystack retrieval at 32K context plus a code-completion or math benchmark drawn from a distribution unlike WikiText-2; if log-perplexity rises sharply above the reported ~2 or retrieval accuracy collapses, the RoRoPE concentration assumption fails on out-of-calibration positions.

Watch

Extended reading notes

Core claim

The central claim is that MLA with decoupled RoPE subsumes GQA: any GQA layer can be written as an MLA layer with the same KV cache size by introducing an up-projection matrix, while the reverse is not always possible. The paper proves this by showing GQA's key/value generation is a sparse special case of MLA's low-rank latent key/value mechanism, and that the full MLA, with a shared multi-query RoPE key, is strictly more expressive than GQA. The practical consequence is the TransMLA conversion: merge GQA's key heads, use RoRoPE plus FreqFold to concentrate positional information into the first key head, discard RoPE elsewhere, and compress the remaining keys and values with balanced joint PCA. The paper reports that this conversion, applied to LLaMA-2-7B, shrinks the KV cache to 7.03% of its original size, gives up to a 10.6x throughput gain in vLLM at 8K context, and after 6B fine-tuning tokens matches or nearly matches the original model on six commonsense benchmarks.

Load-bearing premise

The conversion is only near-lossless if a PCA rotation learned on a small WikiText-2 calibration set concentrates essentially all RoPE position information into the first attention head for every input the model will see; any positional signal left in the discarded heads is destroyed.

Editorial extensions

If this is right

  • Any GQA or MHA pretrained model (LLaMA, Qwen, Gemma, Mistral) can in principle be converted to MLA and run on DeepSeek-compatible optimized kernels, inheriting vLLM and SGlang support.
  • KV-cache memory per token drops with the compression ratio; at 92.97% compression on LLaMA-2-7B the reported throughput jumps from 85.8 to 832.7 output tokens/s at 8K context on a 165 TFLOPS/24GB GPU.
  • Speedups grow with context length because the saving is proportional to cache size; at 16K context the original model runs out of memory on 24GB while the converted model sustains 414 tokens/s.
  • Fine-tuning with a few hundred million to 6B tokens restores benchmark averages close to the original, making architecture migration cheaper than retraining an MLA model from scratch.
  • Training-free conversion is low-loss at moderate compression (-68.75% drops 1.65 points on LLaMA-2) but degrades more at extreme compression (-92.97% drops about 8 points before fine-tuning).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the RoRoPE concentration generalizes beyond WikiText-2 calibration, the same recipe should transfer to Qwen, Mistral, and Gemma; a cheap test is to run the conversion on one of those models and compare long-context retrieval over positions beyond the calibration range.
  • The paper evaluates conversion quality with perplexity and commonsense benchmarks, but not long-context tasks; because RoPE is a positional mechanism, the most sensitive falsifier is long-range needle-in-haystack or code/math inputs whose position matters more.
  • The 10.6x speedup is a vLLM-specific implementation result that will vary with kernel maturity and hardware; the conversion's architectural benefit (smaller cache) is hardware-independent, so combining it with FP8 quantization or multi-token prediction should compound the gain, as the paper suggests.
  • The expressiveness proof applies to the attention core, not to downstream training dynamics; converted models still need fine-tuning to recover, so 'seamless' should be read as 'weight-compatible with light adaptation' rather than zero-cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents TransMLA, a framework for converting a pretrained GQA-based transformer into the MLA architecture used by DeepSeek, with the stated goal of making existing GQA models directly runnable on DeepSeek's optimized inference stack. The proposed pipeline has four components: an exact merging of all GQA key/value heads into a single latent head; a PCA-based rotation of query and key activations within each RoPE frequency subspace (RoRoPE) that concentrates positional information into the first attention head; a frequency-grouping approximation (FreqFold) that packs more principal components into that head; and a norm-balanced joint low-rank compression of the remaining keys and values (BKV). The paper also claims a theoretical proof that MLA has strictly higher expressive power than GQA under the same KV-cache budget. Experiments on SmolLM-1.7B and LLaMA-2-7B report training-free compression up to 92.97% of the KV cache, recovery of benchmark performance after fine-tuning on 6B tokens, and inference speedups up to 10.6x at 8K context in vLLM. Appendices provide the algebraic proofs, the FreqFold variance argument, and fine-tuning and benchmark details.

Significance. If the conversion were genuinely low-loss, TransMLA would be practically valuable: it would let existing GQA models inherit DeepSeek's MLA inference optimizations, with large KV-cache savings and significant decoding speedups. The RoPE orthogonality invariance in Appendix B is a clean and correctly proven piece of algebra, and the BKV norm-balancing heuristic is well motivated by the observed key/value norm disparity. The reported speedups are large and the fine-tuning token budgets are far below pretraining costs. However, the theoretical expressiveness claim is not established for the actual full MLA with low-rank query, and the low-loss property is validated only on short-context perplexity and six short benchmarks, with no long-context quality evaluation. The 'any GQA model' claim is also broader than the two model families tested. The practical contribution is promising but requires additional validation and a more careful statement of the theoretical and empirical scope.

major comments (5)
  1. [Appendix A, A.2.3 and A.3] The strict-expressiveness argument is not valid as written. In the MQA form derived in Eq. (26), the interaction matrix is (W_Q_i)^T W_UK_i W_DKV, whose rank is at most d because W_UK_i is d x 2gd; the claim that MQA allows rank up to 2gd compares against a generic MQA with a larger query projection, which changes the number of query parameters and is not a controlled comparison. Furthermore, A.3 only establishes expressiveness for MLA_Factorized without low-rank query, while the paper's abstract and Section 4 claim that 'MLA consistently offers higher expressive power than GQA' for the full MLA, which in DeepSeek uses a low-rank query (r_q < d). No proof is given for that full variant. The claim should either be proved for the actual low-rank-query MLA or explicitly narrowed.
  2. [Section 4.2, Appendix B, Eq. (32), Fig. 3a] The PCA objective in Eq. (32) maximizes key-side variance only, but the error introduced by dropping RoPE from non-principal heads depends on both query and key residuals in the discarded components of the attention inner product. Figure 3a reports only key L2 norms, so it does not bound the actual attention-score perturbation. Please report the query-side residual energy across layers, or better, the mean relative error of the attention logits on the calibration set, at the RoPE-removal ratios used in Table 1. Without such evidence, the statement that RoRoPE 'concentrates effectively all positional information' into the first head is not supported.
  3. [Section 5.2, Fig. 3b, Section 5.4] The low-loss validation for RoPE removal and FreqFold consists only of WikiText-2 log-perplexity (Fig. 3b) at an unstated, presumably short, sequence length. FreqFold approximates nearby RoPE frequencies as equal, and the induced phase error grows linearly with the relative position (t-j); the throughput benchmark in Fig. 5 extends to 32K context, but no quality metric is reported at those lengths. Please add long-context quality evaluations (e.g., passkey retrieval, RULER, or LongBench) at 8K and 32K for the converted models, and state the sequence length used in Fig. 3b. This is essential to support the practical claim of seamless conversion for deployed long-context use.
  4. [Table 1, Section 5.2, Appendix E] The calibration protocol for the main LLaMA-2-7B conversions is under-specified. The paper does not report the FreqFold setting (M) used for LLaMA-2-7B, the number of WikiText-2 samples, the context length used for collecting activations, or how the PCA bases were computed per layer. Since Fig. 3b indicates that the optimal FreqFold setting is model-dependent (4D is best for LLaMA-3-8B), these details are necessary for reproducibility of the headline compression and fine-tuning results, and for qualifying the claim that the method applies to any GQA model.
  5. [Section 5.1] The claim of seamless conversion of 'any GQA-based pretrained model' is not supported by the experiments, which cover only SmolLM-1.7B and LLaMA-2-7B, plus an analysis-only study on LLaMA-3-8B. The behavior of RoRoPE and FreqFold may depend on the number of groups, head dimension, and RoPE frequency schedule. Please convert and evaluate at least one additional model family (e.g., Qwen2-7B or Mistral-7B), or explicitly restrict the claim to the tested architectures.
minor comments (5)
  1. [Section 4.2, text after Eq. (19)] The phrase 'the same rotation values ... are applied identically to each dimension' is ambiguous; it should say 'to each head's copy of the l-th RoPE pair' to clarify that the orthogonal transform acts across heads within a fixed frequency subspace.
  2. [Table 1] The row label 'SmolLM-1.7B 1T' is not defined in the caption; please clarify that 1T refers to the original pretraining token count, to avoid confusion with the fine-tuning token counts listed in the same table.
  3. [Section 5.4 and Table 4] The baseline LLaMA-2-7B is evaluated at context lengths up to 32K even though the model was trained with a 4K context; please state explicitly that Fig. 5 and Table 4 report throughput only and that output quality at extrapolated lengths is not assessed.
  4. [Appendix D, Eqs. (34)-(36)] The symbol W_DKV in Eq. (35) denotes the NoPE-only concatenation while Eq. (11) defines W_DKV as the full 2gd x D matrix; using a different symbol, such as W_DKV^{NoPE}, would avoid confusion.
  5. [Abstract] The abstract states '93% of the KV cache' while Table 1 reports 92.97%; please use a consistent number of significant digits.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: expressiveness proof and RoRoPE invariance are self-contained; quality and speed numbers are measured, not fitted.

full rationale

TransMLA's derivation chain is self-contained rather than circular. The GQA-to-MLA containment is an explicit algebraic construction: Appendix A.2.1 sets W_DKV = [W_K; W_V] and uses block selector matrices for W_UK and W_UV (Eqs. 23-24), so the claim that GQA can be embedded in MLA is proven by construction, not assumed. The strictness argument rests on rank and dimensionality differences (A.2.3), which is independent of any fitted parameter. The RoRoPE step is also exact: Eq. 19 and Appendix B prove that any orthogonal U_l applied jointly to the real and imaginary parts inside each RoPE subspace preserves the RoPE inner product; this is a mathematical identity. The subsequent removal of RoPE from non-principal heads is an acknowledged approximation validated by perplexity, not by construction. The PCA bases and balancing scales (Eq. 32, Eq. 33) are fitted to WikiText-2 activations, but the paper's headline numbers—benchmark scores, perplexity, and inference speedups—are measurements on data not used to fit those bases; there is no fitted parameter renamed as a prediction. FreqFold's Proposition 2 is proven via Ky Fan's theorem and does not import any uniqueness claim from the authors' prior work. There are no self-citations used to justify the central claim, and the stated limitation (validation on a broader range of models remains) is a generalizability caveat, not circularity. The only mild methodological note is that FreqFold and compression settings are selected with WikiText-2 validation and then evaluated on the same distribution for Figure 3b, but this is standard model selection rather than a definitional reduction; it does not make the reported conversions equivalent to their inputs.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper introduces several fitted quantities: the FreqFold group size, the MLA latent ranks for each compression target, per-layer BKV scaling factors estimated on WikiText-2, and the PCA bases derived from the same calibration set. These are not free constants used for prediction; they are selected to make the conversion work. The core theoretical results (RoPE invariance, PCA variance inequality) are self-contained. The main domain assumptions are that WikiText-2 activations are representative and that removing RoPE from non-principal heads is nearly lossless.

free parameters (5)
  • FreqFold group size (M) = 2D, 4D, 8D evaluated; 4D selected for LLaMA-3-8B
    Chosen from WikiText-2 log-perplexity; not specified for the LLaMA-2-7B conversion.
  • MLA latent rank rkv = Implied by KV cache reductions 68.75%, 87.5%, 92.97%
    Set by the authors per compression target; central to the quality versus speed trade-off.
  • BKV scaling alpha per layer = Ratio of expected key norm to value norm
    Estimated on calibration data to balance PCA; exact values not reported.
  • PCA calibration dataset and sample count = WikiText-2, unspecified size
    Basis for RoRoPE and joint compression; not fully specified for reproduction.
  • Fine-tuning token budget = 500M to 6B
    Chosen per compression ratio; the recoverability claim depends on these budgets.
assumptions (6)
  • standard math Orthogonal invariance of RoPE inner product (Equation 19)
    Proved in Appendix B; correct.
  • domain assumption WikiText-2 key activations represent the model's general key distribution
    PCA and alpha are estimated on WikiText-2; if unrepresentative, RoPE concentration and KV compression degrade.
  • ad hoc to paper FreqFold's approximation of similar RoPE frequencies as identical is low-error
    Introduced in Section 4.2 and Appendix C; the approximation error is not formally bounded, only empirically probed.
  • ad hoc to paper Removing RoPE from non-principal heads loses negligible position information
    Core to the conversion; justified only by perplexity plots, not by a bound.
  • domain assumption Full MLA with low-rank query remains more expressive than GQA
    Appendix A proves this for MLA_Factorized without query low-rank; extension to full MLA is asserted, not proven.
  • domain assumption Fine-tuning on the SmolLM mixture transfers to general benchmarks
    Used to recover LLaMA-2-7B performance; no long-context or open-ended evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TransMLA: Multi-Head Latent Attention Is All You Need." pith.science (2026). https://pith.science/paper/N3HS76Y2

@misc{pith2026250207864,
  author       = {Pith},
  title        = {Pith review of: TransMLA: Multi-Head Latent Attention Is All You Need},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3HS76Y2}},
  note         = {Machine review of arXiv:2502.07864}
}
read the original abstract

In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with DeepSeek's codebase, allowing these models to fully leverage DeepSeek-specific optimizations such as vLLM and SGlang. By compressing 93% of the KV cache in LLaMA-2-7B, TransMLA achieves a 10.6x inference speedup at an 8K context length while preserving meaningful output quality. Additionally, the model requires only 6 billion tokens for fine-tuning to regain performance on par with the original across multiple benchmarks. TransMLA offers a practical solution for migrating GQA-based models to the MLA structure. When combined with DeepSeek's advanced features, such as FP8 quantization and Multi-Token Prediction, even greater inference acceleration can be realized.

Figures

Figures reproduced from arXiv: 2502.07864 by the authors.

Figure 1
Figure 1. GQA, MLA, and MQA can be equivalently transformed in one direction, illustrating a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of RoRoPE for decoupling RoPE. Blue lines denote real-part dimensions, orange lines denote imaginary-part dimensions. When the keys from multiple heads are concatenated, permuting dimensions does not change the computation, so we gather the same dimension (i.e., the same rotational frequency) across all heads and apply joint principal-component analysis—using the identical procedure for the real and imagina… view at source ↗
Figure 3
Figure 3. Visualization of key norms and RoPE removal results on LLaMA 3 8B model. The top and [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of the norms of keys and values for the first layer of LLaMA 3 8B and [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Inference speedups with TransMLA comparing to the original LLaMA2 7B model on [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Multi-Query Attention (MQA), Group Query Attention (GQA), and Multi [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Pipeline of RoRoPE with FreqFold. RoRoPE encodes the entire frequency spectrum of all [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A post-conversion training objective that reconstructs the original attention function of MLA-converted draft models improves speculative-decoding acceptance in most of 64 matched task cells.

  2. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  3. SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass

    cs.CL 2026-02 unverdicted novelty 6.0 of 10

    SHINE trains a scalable in-context hypernetwork to generate high-quality LoRA adapters from contexts in one pass, enabling efficient LLM adaptation that saves time and compute compared to standard fine-tuning.

  4. Hardware-Efficient Attention for Fast Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.

  5. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

  6. Reinforcement Learning in hyperbolic space for multi-step reasoning

    cs.LG 2025-07 reject novelty 4.0 of 10

    Hyperbolic transformer policies are claimed to beat vanilla transformer policies by 32-45% on a handful of reasoning and control problems, but the evidence is too weak to support the claim.

  7. A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention

    cs.LG 2025-07 reject novelty 4.0 of 10

    Applying Marchenko-Pastur spectral diagnostics to LLaMA-130M variants, the paper reports that sharing a single rotary sub-vector across heads in multi-head latent attention suppresses spectral outlier spikes, while st...

  8. Latent Multi-Head Attention for Small Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    MLA with rotary embeddings at half latent rank keeps validation loss nearly unchanged while cutting KV-cache memory by roughly half on small language models.

  9. MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing

    cs.LG 2025-06 conditional novelty 4.0 of 10

    MoE-GPS shows that predicting only the aggregate token-to-expert distribution, instead of exact token routing, can improve MoE inference latency by more than 23% in simulation, with guidelines for when each strategy wins.

  10. The Rise of Small Language Models in Healthcare: A Comprehensive Survey

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A comprehensive survey of small language models in healthcare, with a taxonomy of building, adapting, and compressing them for clinical NLP tasks.

Reference graph

Works this paper leans on

45 extracted references · 8 canonical work pages · cited by 10 Pith papers

  1. [1]

    Hello GPT-4o , 2024

    OpenAI. Hello GPT-4o , 2024. URL https://openai.com/index/hello-gpt-4o/

    2024

  2. [2]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet

    2024

  3. [3]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024 a

    arXiv 2024

  4. [4]

    Llama 3 model card, 2024

    AI@Meta. Llama 3 model card, 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  5. [5]

    Cheaper, better, faster, stronger: Continuing to push the frontier of ai and making it accessible to all, 2024

    Mistral. Cheaper, better, faster, stronger: Continuing to push the frontier of ai and making it accessible to all, 2024. URL https://mistral.ai/news/mixtral-8x22b

    2024

  6. [6]

    Qwen2.5: A party of foundation models, 2024

    Qwen. Qwen2.5: A party of foundation models, 2024. URL https://qwenlm.github.io/blog/qwen2.5

    work page 2024

  7. [7]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a

    arXiv 2024

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 45 references
  1. [9]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024 b

    2024 arXiv

  2. [10]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024

    2024 arXiv

  3. [11]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  4. [12]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [13]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    2023 arXiv

  6. [14]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  7. [15]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017

    2017

  8. [16]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024

    arXiv

  9. [17]

    Kivi: Quantized key-value representation for efficient long-context transformers

    Zhiyang Liu, Dong Zhang, Xinyi Li, and Ji Wu. Kivi: Quantized key-value representation for efficient long-context transformers. arXiv preprint arXiv:2402.06732, 2024 b

    2024

  10. [18]

    Kvquant: Quantization for efficient key-value caching in transformer models

    James Hooper, Li Dai, Zhen Zhang, and Seung-Hwan Lee. Kvquant: Quantization for efficient key-value caching in transformer models. arXiv preprint arXiv:2402.12345, 2024

    2024 arXiv

  11. [19]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing S...

    2023

  12. [20]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.04434, 2024. URL https://doi.org/10.48550/arXiv.2405.04434

    2024 doi arXiv

  13. [21]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  14. [22]

    Palu: Compressing kv-cache with low-rank projection

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118, 2024

    2024 arXiv

  15. [23]

    Towards economical inference: Enabling deepseek's multi-head latent attention in any transformer-based llms

    Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui. Towards economical inference: Enabling deepseek's multi-head latent attention in any transformer-based llms. arXiv preprint arXiv:2502.14837, 2025

    2025

  16. [24]

    Hardware-efficient attention for fast decoding

    Ted Zadouri, Hubert Strauss, and Tri Dao. Hardware-efficient attention for fast decoding. arXiv preprint arXiv:2505.21487, 2025

    2025 arXiv

  17. [25]

    Tensor product attention is all you need

    Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Zhen Qin, Yang Yuan, Quanquan Gu, and Andrew Chi-Chih Yao. Tensor product attention is all you need. arXiv preprint arXiv:2501.06425, 2025

    2025

  18. [26]

    Lazyllm: Dynamic token pruning for efficient long context llm inference

    Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv preprint arXiv:2407.14057, 2024

    2024 arXiv

  19. [27]

    A2sf: Accumulative attention scoring with forgetting factor for token pruning in transformer decoder

    Hyun-rae Jo and Dongkun Shin. A2sf: Accumulative attention scoring with forgetting factor for token pruning in transformer decoder. arXiv preprint arXiv:2407.20485, 2024

    2024 arXiv

  20. [28]

    Snapkv: Llm knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024

    2024 arXiv

  21. [29]

    You only need one: Efficient kv sharing across transformer layers

    Tian Sun, Li Zhang, and Shuang Wu. You only need one: Efficient kv sharing across transformer layers. Proceedings of the 42nd International Conference on Machine Learning (ICML), 2024

    2024

  22. [30]

    Minicache: Kv cache compression in depth dimension for large language models

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366, 2024 c

    2024 arXiv

  23. [31]

    Mlkv: Multi-layer key-value heads for memory efficient transformer decoding

    Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, and Alham Fikri Aji. Mlkv: Multi-layer key-value heads for memory efficient transformer decoding. arXiv preprint arXiv:2406.09297, 2024

    2024 arXiv

  24. [32]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview...

    2021

  25. [33]

    Think you have solved question answering? try arc, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457

    2018 arXiv

  26. [34]

    PIQA: reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelli...

    2020 doi

  27. [35]

    Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Llu \' s M \` a rquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguisti...

    2019 doi

  28. [36]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empiri...

    2018 doi

  29. [37]

    Winogrande: an adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM , 64 0 (9): 0 99--106, 2021. doi:10.1145/3474381. URL https://doi.org/10.1145/3474381

    2021 doi

  30. [38]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  31. [39]

    Smollm-corpus

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus. 2024. URL https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus

    2024

  32. [40]

    Fineweb-edu: the finest collection of educational content, 2024 a

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024 a . URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

    2024

  33. [41]

    Mixture of lora experts

    Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXiv:2404.13628, 2024

    2024 arXiv

  34. [42]

    Starcoder 2 and the stack v2: The next generation, 2024 b

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024

  35. [43]

    Openwebmath: An open dataset of high-quality mathematical web text, 2023

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023

    2023

  36. [44]

    Stack overflow, 2025

    Stack Overflow . Stack overflow, 2025. URL https://stackoverflow.com. Accessed: 2025-05-21

    2025

  37. [46]

    Smollm2: When smol goes big -- data-centric training of a small language model, 2025 b

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo L...

    2025 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.