Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that pretrained transformer attention heads can be uniformly truncated into a smaller latent space and recovered with about one billion tokens of two-stage training, halving the KV cache with minimal accuracy loss.

desk verdict KV-Latent has a real idea and a useful dvo-vs-dqk finding, but Table 1's central claim needs a same-token continued-pretraining control before it lands. read the letter →

arxiv 2507.11273 v1 pith:M2GCKZZU submitted 2025-07-15 cs.CL

Luohe Shi , Zuchao Li , Lefei Zhang , Guoming Liu , Baoyuan Qi , Hai Zhao This is my paper
classification cs.CL
keywords KVcachecompressionlatentspacedownsamplingrotarypositionembeddingtwo-stagedistillationlow-rankattentiongroupedquerylong-contextinferenceheaddimensionreduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes KV-Latent, a way to shrink the KV cache of a pretrained transformer by reducing the per-head dimension of the key, value, query, and output projections rather than dropping tokens or sharing heads. The claim is that uniform down-sampling of these weight matrices, followed by about 1 billion tokens of two-stage training, lets LLaMA-3-8B and LLaMA-2-7B keep most of their benchmark accuracy while the cache shrinks by half. The paper also argues that RoPE becomes unstable at low head dimensions and fixes this with a frequency-aware sampling rule that keeps low frequencies and drops high frequencies. A reader should care because KV cache size and bandwidth are the main practical bottleneck for long-context LLM inference, and this approach is orthogonal to existing head-level, layer-level, and token-level compression methods.

What carries the argument

The central object is the decoupling of two head dimensions, dqk for query/key heads and dvo for value/output heads, paired with uniform stride sampling of the projection matrices (Eq. 5) so that RoPE's paired-channel rotation structure is preserved. On top of that sit three components: a two-stage training schedule that first matches per-layer hidden states with MSE loss and then trains end-to-end with cross-entropy or KL divergence; LoRA adapters on the FFN layers to let the model adjust to the new attention while retaining pretrained knowledge; and a modified RoPE frequency schedule (Eq. 11) that samples more densely in the low-frequency rotations and avoids high-frequency rotations, which the paper argues stops the numerical approximation of the RoPE decay integral from breaking down at small d. These pieces together carry the claim that a dimension-truncated latent KV cache can be recovered with less than 1% of pretraining compute.

What would settle it

Run the truncation before any training and compare, layer by layer, the hidden states and attention outputs of the truncated model against the original on a fixed text sample; if many layers show large output differences or if the two-stage retraining performs no better than starting from randomly re-initialized heads, the symmetry premise is false.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the constraint that head dimension times number of heads equals hidden dimension can be broken after pretraining: by keeping the same channel positions across W_Q, W_K, W_V, and W_O, a pretrained attention head can be uniformly truncated into a smaller latent space, and a two-stage schedule (per-layer MSE distillation followed by end-to-end next-token prediction or KL distillation) restores the model well enough that LLaMA-3-8B with head dimensions (64,64) averages 41.3 on MMLU/OBQA/ARC versus 42.1 for the base, while the KV cache drops from 491 MB to 245 MB, and LLaMA-2-7B averages 28.3 versus 29.7 with the same 50% cache reduction. It further claims that at (16,16) the model cannot be recovered, that values carry more essential information than keys, and that a frequency-aware RoPE sampling scheme keeps positional encoding stable when head dimensions are small.

Load-bearing premise

The load-bearing premise is that the individual channels inside each pretrained attention head are arranged symmetrically enough that picking the same evenly spaced subset of channels in Q, K, V, and O keeps the information the head needs, and that a short retraining can repair what the pruning removes.

Editorial extensions

If this is right

  • Halving dqk and dvo halves the KV cache footprint in bytes and shortens time-to-first-token in the reported setups, with average benchmark scores within about one point of the base model.
  • Because keys can be compressed more aggressively than values, future models could allocate unequal head dimensions (smaller dqk, larger dvo) as a deliberate architecture choice rather than a post-hoc fix.
  • KV-Latent is compatible with token-level eviction methods; combining it with PyramidInfer at 50% compression further halves the cache to 25% of the original size.
  • There is a practical lower bound near dqk=dvo=16 for these 7-8B models, where benchmark scores and long-context retrieval collapse, so dimension reduction cannot go to zero without additional changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the rotational-symmetry initialization transfers to non-RoPE models or to models trained without grouped-query attention, the same truncate-and-distill recipe could generalize beyond the LLaMA family, but the paper only tests LLaMA-2 and LLaMA-3.
  • A testable extension is to measure how much information the uniform stride actually discards: comparing random initialization, SVD initialization, and uniform truncation before training would quantify the value of the symmetry assumption.
  • The frequency-aware RoPE modification is separable from the cache reduction, so it could be applied directly to other low-dimensional attention designs, such as multi-query or grouped-query variants that use small heads.
  • The larger drop on GQA-based LLaMA-3 than on MHA-based LLaMA-2 hints that models already sharing KV heads have less redundancy left to exploit; confirming this on more models would sharpen the method's expected gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes KV-Latent, a method that reduces the memory footprint of the KV cache in pretrained Transformer LLMs by directly downsampling the per-head dimensions of W_Q, W_K, W_V, and W_O, followed by a two-stage training procedure: per-layer hidden-state distillation (Stage I) and end-to-end next-token prediction or distillation (Stage II). The authors also introduce a 'frequency-aware' modification of RoPE intended to stabilize rotary embeddings at reduced head dimensions. Experiments on LLaMA-3-8B and LLaMA-2-7B report roughly 50% KV-cache reduction with small average drops on MMLU/OBQA/ARC (e.g., 41.3 vs 42.1 for LLaMA-3-8B at dqk=dvo=64), and additional experiments explore different dqk/dvo splits, LoRA ranks, and compatibility with a token-level compression method.

Significance. If the central claim were established, KV-Latent would be a practically interesting way to trade a small amount of post-hoc training for a substantial reduction in KV-cache memory and bandwidth, and the observation that value dimensions are more sensitive than key dimensions would be a useful structural insight. The paper has concrete strengths: it evaluates on both GQA and MHA models, releases code, and demonstrates compatibility with a token-level method. However, the empirical validation as reported is not yet conclusive. The headline results lack a same-token continued-pretraining control, repeated runs or error bars, and controlled ablations of the two main components (channel subsampling and frequency-aware RoPE). The proof in Appendix D.2 also does not support the claim that the modified RoPE is always larger. These gaps are load-bearing for the paper's three stated contributions.

major comments (4)
  1. [Section 4.2, Table 1] The central empirical result is underdetermined because the only comparison is against the untouched base checkpoint. Continued pretraining on 1B tokens of FineWeb-edu with LoRA and the Stage-II recipe can shift 0-shot benchmark scores by several points, so the reported 0.8-point average drop for LLaMA-3-8B (41.3 vs 42.1) cannot be attributed to the downsampling-and-distillation pipeline without a same-token control. Please add a control row that trains the unmodified base model on the same 1B-token subset with the same hyperparameters (learning rate 2e-5, batch size 8, sequence length 4096, LoRA rank 256), and report multiple seeds or evaluation repeats with standard deviations for all rows in Table 1.
  2. [Section 3.2.1, Eq. (5)] The assertion that pretrained attention heads are rotationally symmetric, so that uniform channel subsampling of W_Q, W_K, W_V, and W_O preserves enough information for recovery, is not measured anywhere in the paper. No information-retention statistic is reported, the interaction between uniform subsampling and the GPT-NeoX channel pairing described in Appendix C is not analyzed quantitatively, and there is no ablation against random channel selection or an SVD-style initialization. The Limitations section even concedes that SVD 'could provide the model with additional initial information,' which further indicates that the initialization's adequacy is not independently verified. Please add an explicit information-loss measurement (e.g., attention-output cosine similarity before and after downsampling) and an initialization ablation.
  3. [Section 3.3.3, Appendix D.2] The proof that frequency-aware RoPE is 'always larger' is not valid as written. The step cos(x*theta^{-2j/d}) ≈ 1 for j in (3d/8, d/2) requires the argument to be near zero, but with theta = 10000, j = 3d/8, and x = 4096 (within the context lengths used in the paper), the argument is roughly 4096 * 10000^{-3/4} ≈ 4.1 radians, which is not small. Moreover, Figures 7-10 show only the modified curves, and Table 1 conflates downsampling, training, and the RoPE change, so the claimed stability improvement is not empirically isolated. Please provide a controlled comparison of default RoPE versus frequency-aware RoPE at the same reduced head dimensions and training budget, and either correct the proof or replace it with a numerical verification.
  4. [Section 4.3.1, Table 6] The conclusion that dvo carries more essential information than dqk rests on differences in log perplexity such as 2.86 versus 2.79, with no repeated runs, confidence intervals, or significance test; these differences are small enough to be within noise. In addition, the paper cites Eigen Attention and LoRC but reports no experimental comparison against them or any training-free KV-compression baseline, leaving unclear whether the roughly 1.1B-token training budget is justified relative to the alternative. Please add the missing baselines and uncertainty estimates, or soften the corresponding claims.
minor comments (6)
  1. [General] There are numerous typos and grammatical errors, including 'has gain many attention', 'unables', 'Out work', and 'it's size'; these should be corrected throughout.
  2. [Section 4.3.3, Table 4] Applying PyramidInfer is reported to improve log perplexity from 2.509 to 2.499, which is counterintuitive for an eviction-based compression method; this should be explained or corrected.
  3. [Appendix A, Table 5] The token budgets in Table 5 (0.1B for Stage I, 1B for Stage II Train, 0.25B for Stage II Distill, 0.25B for parameter selection) should be reconciled with the text in Section 4.1, which mentions only 'a 1 billion token subset.'
  4. [Figures 3-10] The RoPE decay figures lack clear axis labels and legends, and in the submitted PDF several labels are unreadable, making it difficult to verify the claimed behavior of the modified RoPE.
  5. [References] The citations to 'bloc97, 2023' and 'Devoto et al.' are incomplete; full references should be provided.
  6. [Appendix D.1] The notation lim_{d/2 -> infinity} is informal; this should be rewritten as a standard limit in d.

Circularity Check

1 steps flagged · score 2.0 of 10

KV-cache reduction is benchmark-grounded; only the frequency-aware RoPE validation is self-confirming.

  1. other [Section 3.3.3, Eq. 11, Figures 7-10; Appendix D.2]
    "We implemented a frequency-aware modification strategy, which involves densifying the sampling of low-frequency rotations and avoiding high-frequency rotation sampling, as described in Formula 11... The results, presented in Figures 7, 8, 9, and 10, demonstrate that our approach achieves enhanced stability while also reducing the occurrence of negative values. (Appendix D.2: 'So RoPEMod − RoPE > 0')"

    The 'stability' outcome is read off the same self-similarity metric RoPEθ,d(x) that the modification was constructed to maximize. Equation 11 selects low-frequency rotation indices whose cosines stay near 1 and discards high-frequency indices that oscillate; Figures 7-10 then plot RoPEθ,d, and Appendix D.2 proves RoPEMod − RoPE > 0. That proof makes the displayed improvement a restatement of the construction rather than an independent experimental finding. The main KV-cache claim, however, is separate and tested on extrinsic benchmarks (MMLU, OBQA, ARC, NIH, latency/size), so this is a minor self-confirmation, not a circular core.

full rationale

The central claim — that head-dimension downsampling plus a small amount of two-stage training reduces KV-cache size while roughly preserving capability — is validated by external benchmarks in Table 1 (MMLU/OBQA/ARC, NIH) and by measured cache-size/latency reductions, so it does not reduce to a fitting artefact. The 'rotational symmetry' initialization premise is unproven and the Limitations section concedes SVD could add initial information, but that is a correctness/robustness gap, not circularity: the paper does not define the target benchmark score in terms of the initialization. The only circular element is the frequency-aware RoPE validation: the schedule in Eq. 11 was hand-designed to keep low-frequency cosine terms near 1 and discard oscillatory high-frequency terms, and the supporting figures show exactly the same RoPEθ,d self-similarity function that was the design objective; Appendix D.2's inequality makes the 'improvement' a mathematical consequence of the construction. Because the main KV-cache contribution is independently benchmark-grounded, the overall circularity is minor. The missing no-downsampling continued-pretraining control in Table 1 is a validity concern about attributing the 0.8-point gap to KV-Latent, but it is not a circularity argument.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central derivation relies on hand-designed parameters (RoPE schedule, head dimensions, token budgets) and untested domain assumptions about symmetry, distillation sufficiency, and benchmark coverage.

free parameters (4)
  • Modified RoPE frequency schedule coefficients = d/8 offset, d/4 boundary, 3d/4 shift in Eq. 11
    Hand-designed so the self-similarity curve stays positive and smooth at low dimensions; the coefficients are not derived from a uniqueness theorem or external constraint.
  • dqk and dvo target head dimensions = (64,64), (16,16), and combinations in Tables 1-2
    Chosen by the authors to define cache-reduction ratios; results are conditional on these choices.
  • Stage-I and Stage-II token budgets = 0.1B, 1B, and 0.25B tokens
    Chosen based on compute limits (Appendix A), not derived from an optimality criterion; the claim of less than 1% of pretraining depends on these budgets.
  • LoRA rank and alpha = rank 256, alpha 512
    Selected for all main runs; Table 3 shows log perplexity changes are small across ranks 16 to 256, so the central result is not highly sensitive to this parameter.
assumptions (4)
  • ad hoc to paper Pretrained attention heads are rotationally symmetric, so uniform subsampling of channels preserves the attention information (Eq. 5, Section 3.2.1).
    The paper asserts random sampling is sufficient without a proof or measurement; the entire recovery depends on the truncated model retaining usable structure.
  • domain assumption Minimizing per-layer MSE between original and modified decoder hidden states is a sufficient surrogate for preserving the original model's behavior (Eq. 7, Section 3.2.2).
    Per-layer closeness at 0.1B tokens is assumed to propagate through the full stack; the authors themselves note deep-layer error amplification motivates Stage II.
  • domain assumption The frequency schedule in Eq. 11 preserves relative position encoding semantics of the pretrained RoPE model.
    The analysis only tracks self-similarity of the all-ones vector; it does not analyze attention logits or downstream position generalization.
  • domain assumption Benchmarks MMLU, OBQA, ARC, and a synthetic needle-in-haystack test are adequate proxies for general language capability.
    No coding, math, instruction-following, or long-generation tasks are included.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding." pith.science (2026). https://pith.science/paper/M2GCKZZU

@misc{pith2026250711273,
  author       = {Pith},
  title        = {Pith review of: KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M2GCKZZU}},
  note         = {Machine review of arXiv:2507.11273}
}
read the original abstract

Large language models (LLMs) based on Transformer Decoders have become the preferred choice for conversational generative AI. Despite the overall superiority of the Decoder architecture, the gradually increasing Key-Value (KV) cache during inference has emerged as a primary efficiency bottleneck, both in aspects of memory consumption and data transfer bandwidth limitations. To address these challenges, we propose a paradigm called KV-Latent. By down-sampling the Key-Value vector dimensions into a latent space, we can significantly reduce the KV Cache footprint and improve inference speed, only with a small amount of extra training, less than 1\% of pre-training takes. Besides, we enhanced the stability of Rotary Positional Embedding applied on lower-dimensional vectors by modifying its frequency sampling mechanism, avoiding noise introduced by higher frequencies while retaining position attenuation. Our experiments, including both models with Grouped Query Attention and those without, have yielded satisfactory results. Finally, we conducted comparative experiments to study the impact of separately reducing Key and Value components on model's performance. Our approach allows for the construction of more efficient language model systems, and opens the new possibility on KV Cache saving and efficient LLMs. Our code is available at https://github.com/ShiLuohe/KV-Latent.

Figures

Figures reproduced from arXiv: 2507.11273 by the authors.

Figure 1
Figure 1. Model preparation process and trainable pa [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Dataflow of two stage training in KV-Latent. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 5
Figure 5. Higher dims only 0 2500 5000 7500 10000 12500 15000 5 0 5 10 15 high 64 Dim mid-high 64 Dim mid-low 64 Dim low 64 Dim [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figures from the paper (2 more)
Figure 8
Figure 8. Figure 8: 64 dim 0 2500 5000 7500 10000 12500 15000 5.0 2.5 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Original Ours [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 11
Figure 11. Figure 11: The cos(θ p ) where θ = 10000 pendix A. We’ve conduct our training on a single node with 8 NVIDIA A100 80G SXM4 GPU. Model wise, we’ve trained two versions of KV￾Latent on LLaMA-3-8B(L3-8B), with (dqk, dvo) = (64, 64) and (16, 16) as a GQA examples, one ver￾sion on LL…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

    cs.AR 2026-07 conditional novelty 7.0 of 10

    A per-request sound upper bound on KV-cache quantization error, implemented as a runtime meter that gates and repairs compression to restore quality.

Reference graph

Works this paper leans on

50 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Joshua Ainslie, James Lee - Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \' o n, and Sumit Sanghai. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.298 GQA: training generalized multi-query transformer models from multi-head checkpoints . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Si...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  2. [2]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean - Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael ...

    arXiv doi:10.48550/arxiv.2312.11805 2023

  3. [3]

    Anthropic . 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family: Opus, sonnet, haiku . Accessed: 2025-05-26

    work page 2024

  4. [4]

    Anthropic . 2025. https://www.anthropic.com/news/claude-4 Introducing claude 4 . Accessed: 2025-05-26

    work page 2025

  5. [5]

    Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. https://doi.org/10.18653/v1/2022.bigscience-1.9 GPT - N eo X -20 B : An open-source autoreg...

    doi:10.18653/v1/2022.bigscience-1.9 2022

  6. [6]

    bloc97. 2023. https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/ NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

    work page 2023

  7. [7]

    William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan - Kelley. 2024. https://doi.org/10.48550/ARXIV.2405.12981 Reducing transformer key-value cache size with cross-layer attention . CoRR, abs/2405.12981

    arXiv doi:10.48550/arxiv.2405.12981 2024

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

Show all 50 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the AI2 reasoning challenge . CoRR, abs/1803.05457

    2018 arXiv

  2. [10]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R\' e . 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf Flashattention: Fast and memory-efficient exact attention with io-awareness . In Advances ...

    2022

  3. [11]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei...

    2024 arXiv

  4. [12]

    Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. 2024. https://doi.org/10.48550/ARXIV.2406.11430 A simple and effective l\( _ 2 \) norm-based strategy for KV cache compression . CoRR, abs/2406.11430

    2024 doi arXiv

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  6. [14]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The P ile: An 800 GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  7. [15]

    Jiaao He and Jidong Zhai. 2024. https://arxiv.org/abs/2403.11421 Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines . Preprint, arXiv:2403.11421

    2024 arXiv

  8. [16]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In 9th International Conference on Learning Representations, ICLR 2021,...

    2021

  9. [17]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...

    2022

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, ...

    2023 doi arXiv

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L \' e lio Renard Lavaud, Lucile Saulnier, Marie...

    2024 doi arXiv

  12. [20]

    Jean Kaddour. 2023. The minipile challenge for data-efficient language models. arXiv preprint arXiv:2304.08442

    2023 arXiv

  13. [21]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. https://doi.org/10.1145/3600006.3613165 Efficient memory management for large language model serving with pagedattention . In Proceedings of the 29...

    2023

  14. [22]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/a452a7c6c463e4ae8fbdc614c6e983e6-Paper-Conference.pdf Scissorhands: Exploiting the...

    2023

  15. [23]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. https://doi.org/10.57967/hf/2497 Fineweb-edu

    2024 doi

  16. [24]

    Ziyang Ma, Zuchao Li, Lefei Zhang, Gui - Song Xia, Bo Du, Liangpei Zhang, and Dacheng Tao. 2025. https://doi.org/10.48550/ARXIV.2503.23924 Model hemorrhage and the robustness limits of large language models . CoRR, abs/2503.23924

    2025 doi arXiv

  17. [25]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018 doi

  18. [26]

    Wong, and Longyue Wang

    Jianhui Pang, Fanghua Ye, Derek F. Wong, and Longyue Wang. 2024. https://doi.org/10.48550/ARXIV.2402.07616 Anchor-based large language models . CoRR, abs/2402.07616

    2024 doi arXiv

  19. [27]

    Guilherme Penedo, Hynek Kydl \' cek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2024. https://doi.org/10.48550/ARXIV.2406.17557 The fineweb datasets: Decanting the web for the finest text data at scale . CoRR, abs/2406.17557

    2024 doi arXiv

  20. [28]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. https://openreview.net/forum?id=wHBfxhZu1u Ya RN : Efficient context window extension of large language models . In The Twelfth International Conference on Learning Representations

    2024

  21. [29]

    Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. 2024 a . https://arxiv.org/abs/2408.05646 Eigen attention: Attention in low-rank space for kv cache compression . Preprint, arXiv:2408.05646

    2024 arXiv

  22. [30]

    Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. 2024 b . https://doi.org/10.48550/ARXIV.2408.05646 Eigen attention: Attention in low-rank space for KV cache compression . CoRR, abs/2408.05646

    2024 doi arXiv

  23. [31]

    Noam Shazeer. 2019. https://arxiv.org/abs/1911.02150 Fast transformer decoding: One write-head is all you need . CoRR, abs/1911.02150

    2019 arXiv

  24. [32]

    Luohe Shi, Yao Yao, Zuchao Li, Lefei Zhang, and Hai Zhao. 2024. http://papers.nips.cc/paper\_files/paper/2024/hash/925869234d3aa2a3aad5f05b643974aa-Abstract-Conference.html Reference trustable decoding: A training-free augmentation paradigm for large language models . In Advan...

    2024

  25. [33]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. https://doi.org/10.1016/j.neucom.2023.127063 Roformer: Enhanced transformer with rotary position embedding . Neurocomputing, 568:127063

    2024

  26. [34]

    Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. 2024. https://doi.org/10.48550/ARXIV.2405.05254 You only cache once: Decoder-decoder architectures for language models . CoRR, abs/2405.05254

    2024 doi arXiv

  27. [35]

    Mingni Tang, Jiajia Li, Lu Yang, Zhiqiang Zhang, Jinhao Tian, Zuchao Li, Lefei Zhang, and Ping Wang. 2025 a . https://aclanthology.org/2025.findings-naacl.399/ NOTA : Multimodal music notation understanding for visual large language model . In Findings of the Association for C...

    2025

  28. [36]

    Zicong Tang, Luohe Shi, Zuchao Li, Baoyuan Qi, Guoming Liu, Lefei Zhang, and Ping Wang. 2025 b . https://openreview.net/forum?id=vxjvu0V8Dy Spindle KV : A novel KV cache reduction method balancing both shallow and deep layers . In The 63rd Annual Meeting of the Association for...

    2025

  29. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  30. [38]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  31. [39]

    Samuel Williams, Andrew Waterman, and David Patterson. 2009. https://doi.org/10.1145/1498765.1498785 Roofline: an insightful visual performance model for multicore architectures . Commun. ACM, 52(4):65–76

    2009

  32. [40]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  33. [41]

    Xin Yan, Zuchao Li, and Lefei Zhang. 2024. https://doi.org/10.1007/978-981-97-8505-6\_33 Centroid-centered modeling for efficient vision transformer pre-training . In Pattern Recognition and Computer Vision - 7th Chinese Conference, PRCV 2024 Urumqi, China, October 18-20, 2024...

    2024 doi

  34. [42]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  35. [43]

    Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024 b . https://doi.org/10.48550/ARXIV.2405.12532 Pyramidinfer: Pyramid KV cache compression for high-throughput LLM inference . CoRR, abs/2405.12532

    2024 doi arXiv

  36. [44]

    Lu Yang, Jiajia Li, En Ci, Lefei Zhang, Zuchao Li, and Ping Wang. 2025. https://aclanthology.org/2025.naacl-long.259/ Label drop for multi-aspect relation modeling in universal information extraction . In Proceedings of the 2025 Conference of the Nations of the Americas Chapte...

    2025

  37. [45]

    Yao Yao, Zuchao Li, and Hai Zhao. 2024. https://doi.org/10.48550/ARXIV.2405.12528 Sirllm: Streaming infinite retentive LLM . CoRR, abs/2405.12528

    2024 doi arXiv

  38. [46]

    Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. 2024. https://doi.org/10.48550/ARXIV.2406.07056 Effectively compress KV heads for LLM . CoRR, abs/2406.07056

    2024 doi arXiv

  39. [47]

    Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Yelong Shen. 2024. https://arxiv.org/abs/2410.03111 Lorc: Low-rank compression for llms kv cache with a progressive compression strategy . Preprint, arXiv:2410.03111

    2024 arXiv

  40. [48]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\' e , Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb2827f...

    2023

  41. [49]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.