Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Power Law Guided Dynamic Sifting for Efficient Attention

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Attention-score quantiles decay as a power law, so filtering can replace top-k selection.

desk verdict A real empirical observation with an incomplete validation: worth refereeing, needs downstream-task fit checks and the missing threshold baselines. read the letter →

arxiv 2506.05300 v1 pith:MK5CBUPW submitted 2025-06-05 cs.LG

Nirav Koley , Prajwal Singhania , Abhinav Bhatele This is my paper
classification cs.LG
keywords attentionsparsificationpower-lawdecayquantilethresholdLLMinferenceGPUmemorybandwidthKVcachetop-kTritonkernel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SiftAttention claims that the tau-th quantile of attention scores in transformer decoders falls off as a predictable power law over generation steps, and that a short warmup of a few hundred steps is enough to fit that curve. If true, the fitted curve can supply a dynamic per-step threshold, so attention can be sparsified by a cheap element-wise comparison instead of an expensive top-k sort. The paper argues this preserves model quality close to full attention while cutting HBM-to-SRAM value-vector traffic, and that the pattern is consistent across models, sizes, and datasets.

What carries the argument

The central object is the power-law quantile model theta_{i,tau} ≈ $\alpha$ * $i^{{-beta}}$, fit in log-log space by closed-form linear regression during a warmup of w generation steps. The fitted curve becomes a time-varying threshold, and the attention weight comparison is an element-wise filter a_S[i] > $\alpha$ * $S^{{-beta}}$, which removes the full-row dependency and thread-synchronization overhead of top-k selection. A fused Triton kernel serves as a proof-of-concept that the filtered value-vector load shrinks with realized sparsity.

What would settle it

On IFEval or MATH prompts, run SiftAttention with a 512-step warmup and compare the predicted tau-th quantile at steps 1024–4096 with the true attention-score quantiles; if the extrapolated thresholds systematically mispredict the quantiles, or realized sparsity deviates from intended sparsity by a large margin on those tasks, the central power-law transfer claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the tau-th quantile of post-softmax attention scores at generation step i follows a power-law decay of the form theta_{i,tau} ≈ $\alpha$ * $i^{{-beta}}$, and that this regularity supports an online, dataset-agnostic, model-agnostic sparsification method. SiftAttention fits $\alpha$ and $\beta$ by log-log linear regression during a warmup phase, then at each later step predicts the quantile as a threshold and keeps only attention scores above it, loading only the corresponding value vectors. The evaluation reports perplexity degradation within the acceptable 0.1 range for most sparsity levels when warmup is 128–512 steps, downstream accuracy within 2–3 percent of full attention on long-context tasks, and a roughly 31 percent reduction in HBM read traffic at context length 8192 as realized sparsity increases.

Load-bearing premise

Power-law parameters fitted during the first 512 generation steps of a perplexity task remain accurate for later steps and for the downstream generation tasks where the threshold is actually applied.

Editorial extensions

If this is right

  • With a warmup of 256–512 steps, SiftAttention keeps perplexity degradation within the accepted 0.1 range for most sparsity levels across Llama, Mistral, and Qwen models.
  • On IFEval and MATH-Hard, SiftAttention with a 128-step warmup matches or beats the top-k baseline at high realized sparsity.
  • On LongGenBench, a 512-step warmup nearly matches full attention while achieving high sparsity, with an average drop of 2–3 percent at 256 warmup steps.
  • Longer warmups bring realized sparsity closer to the intended target, so warmup length acts as a controllable accuracy-sparsity dial.
  • The runtime model predicts SiftAttention's approximate-generation phase is strictly cheaper than top-k attention because thresholding is element-wise and parallel.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The power-law regularity may extend to pre-softmax scores; the authors leave this for future work, but if it holds, thresholds could be applied before softmax, saving that computation as well.
  • The task-dependence of realized versus intended sparsity suggests the fitted exponent beta could serve as a cheap diagnostic of attention saturation during generation, not just a pruning threshold.
  • Because the method is fully online, it could compose with streaming KV-cache compaction or speculative decoding, where per-step thresholds adapt without any offline calibration set.
  • A direct test would be to measure whether the power-law parameters estimated on perplexity prompts transfer to tasks like IFEval and MATH without refitting; the paper's own sparsity gap plots indicate this transfer is only approximate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SiftAttention, an approximate attention method that replaces top-k token selection with an element-wise threshold filter during autoregressive generation. The central empirical claim is that the tau-th quantile of attention scores decays as a power law over generation steps, so the threshold can be predicted from a short warmup fit and used to prune attention without sorting. The authors evaluate the power-law fit on WikiText-2 and C4 perplexity tasks across five models, then evaluate SiftAttention against full attention, Top-k, and H2O on perplexity, IFEval, MATH-Hard, and LongGenBench. They also provide a Triton kernel implementation and measure HBM data transfer. The paper finds that SiftAttention preserves quality within a small margin while reducing data movement, although the Triton kernel is slower than PyTorch's fused SDPA.

Significance. If the power-law transfer from perplexity warmup to downstream tasks holds, SiftAttention would be a novel, sorting-free alternative to top-k sparse attention, with potential bandwidth savings on GPUs. The empirical observation that attention score quantiles decay predictably is interesting and could motivate further work. The paper is also honest in its limitations section about the Triton kernel's performance. However, the load-bearing validation of the power-law mechanism on the tasks where the method is deployed is missing, and the latency claim is not supported by the reported measurements.

major comments (3)
  1. [Section 3.2 / Figure 4 (left) / Appendix B.1] The power-law fit is validated only on perplexity tasks (WikiText-2, C4), with median R2 values of 0.6–0.8 and a 5th percentile around 0.4, but no fit-quality analysis is reported for IFEval, MATH, or LongGenBench, which are the tasks where SiftAttention is actually applied. The realized-versus-intended sparsity deviations shown in Figure 4 (left) and Appendix B.1—PPL undershooting and IFEval/LongGenBench overshooting—are exactly the symptom of a threshold miscalibrated by a power-law that does not transfer across tasks. Since the paper explains its quality preservation through the power-law-guided mechanism, the authors should provide a per-task R2 analysis analogous to Figure 2 on the downstream tasks, or otherwise demonstrate that the realized sparsity deviation does not affect the retained token subset.
  2. [Section 6 / Appendix B.4] The efficiency claim of the paper is not established: the authors admit in Section 4.3 and Appendix B.4 that the Triton kernel is 2–3× slower than PyTorch's fused SDPA, and Figure 4 (middle) shows SiftAttention slower than TorchSDPA at all context lengths. The theoretical runtime comparison in Eq. (7) is only against top-k and is not measured; the only measured benefit is reduced HBM traffic (Figure 4 right), not wall-clock speedup. For a paper titled 'Efficient Attention,' the central performance claim should either be backed by an end-to-end implementation that achieves speedup, or the scope of the claim should be explicitly limited to data-movement reduction rather than latency.
  3. [Section 5 / Figure 6] The practical utility of the warmup is overstated for short-context tasks: IFEval generates up to 1280 tokens and MATH-Hard up to 1024, so a warmup of w=512 consumes 40–50% of the generated sequence, and even w=128 is roughly 10–12%. Additionally, the evaluation discards samples where the number of generated tokens is less than the maximum warmup length (Section 5), which systematically removes the shortest completions and biases the results. The paper should report the fraction of discarded samples and evaluate warmup lengths that are feasible for short tasks, or clearly qualify the method's applicability.
minor comments (5)
  1. [Figure 4 / Figure 12] The legends and captions use 'Sift ( = 0.0)', 'Sift ( = 0.5)', etc., with the tau symbol missing; these should be rendered consistently as 'Sift (τ = ...)'.
  2. [Section 6, Figure 5 discussion] The text says 'warmups of 128 and 512 (green, blue)' but the legend in Figure 5 appears to associate blue with w=128 and green with w=512; the color references should be corrected.
  3. [Algorithm 2] The algorithm does not state whether the retained attention scores a' are renormalized after thresholding; if they are not, the output scale changes, and if they are, the procedure should be described explicitly.
  4. [References] The AxoNN reference is cited inconsistently as [19, 21] in Section 3.1 and as [20, 19] in Section 5; the citation numbers should be unified.
  5. [Appendix A.1] Appendix A.1 states that R2 values 'consistently exceed 0.6 across all evaluated models and datasets,' but the main text reports that the 5th percentile is around 0.4; this overstates the fit quality and should be rephrased to match the reported distribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: warmup-fitted power-law thresholds are evaluated out-of-sample; self-citations are infrastructural and not load-bearing.

full rationale

The paper's derivation chain is self-contained and non-circular. The only fitted quantities are the power-law parameters (alpha, beta) estimated from the tau-th quantile of attention scores during the first w generation steps (Algorithm 1, Line 7), and the only use of that fit is to produce thresholds eta_S = alpha * S^{-beta} at later steps (Algorithm 2, Line 4). Nothing in the warmup fit is defined in terms of the downstream quality metrics or realized sparsity reported in Section 6; those quantities are measured after the fact, and the paper's own Figures 4 and 9 show that realized sparsity can deviate from intended sparsity, which is exactly what one expects when the prediction is not forced by construction. The limitations in Section 7.1 and Appendix A.2 explicitly acknowledge that the power-law analysis was only performed on perplexity tasks, that realized-vs-intended gaps lack a theoretical explanation, and that the fixed warmup requires per-model/per-task tuning; such admissions are evidence that the mechanism is empirically tested rather than definitionally guaranteed. The self-citations (AxoNN, refs. [19-21]; Loki, ref. [22]) are infrastructural or related-work context and are not used to justify the power-law claim or to forbid alternatives. The power-law functional form is assumed rather than derived, but an assumed model is not a circular step: the parameters are estimated from warmup data and then applied out-of-sample on held-out tasks. No equation in the paper reduces to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method's predictive mechanism rests on a fitted two-parameter power-law per layer, head, and prompt, plus hand-chosen tau and w. No independent evidence is provided that the power-law form is unique or that it transfers to downstream tasks; the fit is only evaluated on perplexity. The quality-preserving property of threshold pruning is assumed and tested, not proven.

free parameters (4)
  • Power-law scale alpha (per layer, head, prompt)
    Fitted to warmup quantile scores via log-log linear regression in Section 3.2 and Algorithm 1.
  • Power-law exponent beta (per layer, head, prompt)
    Fitted alongside alpha; governs the decay rate of the quantile.
  • Quantile level tau = 0.5, 0.75, 0.875, 0.95
    Chosen hyperparameter controlling intended sparsity; evaluated over a range.
  • Warmup length w = 16, 32, 64, 128, 256, 512
    Chosen hyperparameter; longer w improves power-law fit but adds overhead.
assumptions (4)
  • domain assumption Attention score quantiles follow a power law of the form theta approximately alpha times i to the minus beta over generation steps.
    Empirically motivated in Section 3.1; fits have median R-squared 0.6 to 0.8 with some poor fits, and no alternative functional form is tested.
  • domain assumption The power-law fit from warmup steps generalizes to later generation steps.
    Assumed in Algorithm 2; with w=512 median R-squared is about 0.6, but with shorter warmup fits degrade to negative R-squared, showing the assumption is fragile.
  • domain assumption Tokens with attention scores below the threshold can be dropped without materially changing the attention output.
    Core sparsification assumption; evaluated empirically in Section 6 but no theoretical guarantee or error bound is provided.
  • standard math Log-log linear regression with multiplicative noise is appropriate for estimating power-law parameters.
    Standard technique for power-law fitting as described in Section 2, Equation 2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Power Law Guided Dynamic Sifting for Efficient Attention." pith.science (2026). https://pith.science/paper/MK5CBUPW

@misc{pith2026250605300,
  author       = {Pith},
  title        = {Pith review of: Power Law Guided Dynamic Sifting for Efficient Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MK5CBUPW}},
  note         = {Machine review of arXiv:2506.05300}
}
abstract

Efficient inference on GPUs using large language models remains challenging due to memory bandwidth limitations, particularly during data transfers between High Bandwidth Memory (HBM) and SRAM in attention computations. Approximate attention methods address this issue by reducing computational and memory overhead but often rely on expensive top-$k$ operations, which perform poorly on GPUs. We propose SiftAttention, a novel approximate attention method that replaces the top-$k$ step with a computationally efficient element-wise filtering operation based on a threshold value. Our intuition for doing this is based on our empirical observation that the $\tau$-th quantile of attention scores follows a predictable power-law over sequential generation steps. Exploiting this insight, our approach dynamically estimates a threshold value per prompt at each generation step. Only attention scores above this threshold and their corresponding value vectors are loaded/used to compute the attention output, reducing data movement between HBM and SRAM. Our evaluation demonstrates that SiftAttention preserves model quality better than existing approximate attention methods while reducing memory bandwidth usage when loading value vectors.

Figures

Figures reproduced from arXiv: 2506.05300 by the authors.

Figure 1
Figure 1. Overview of SiftAttention off accuracy for efficiency. These methods either prune the KV-cache permanently [29] for memory savings or dynamically select a subset of tokens per generation step [18, 10, 22] to reduce the amount of key-value pairs to attend to. The latter set of methods typically use a top-k selection operation to select the top k tokens with the highest attention scores, which is inefficient on GPUs d… view at source ↗
Figure 2
Figure 2. Power-Law fit for attention score quantiles (log y-scale) over generation steps for Llama-3.1 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. R2 of power-law fits for attention score quantiles with warmup. Left: R2 distribution for Llama-3.1-8B-Instruct on WikiText-2, evaluated across warmup sizes of 64, 128, 256, and 512 steps. Right: Fit quality across models with a warmup of 512. Longer warmups consistently yield stronger fits across models. power-law fit performs worse than simply predicting the mean of the log-quantile values across the full sequence… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: (Left) SiftAttention Intended vs. Realized sparsity across tasks and warmup lengths for [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Perplexity evaluation (WikiText-2) comparing SiftAttention with baselines for 3 models [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Short (IFEval, MATH) and long (LongGenBench) generation evaluation across 3 models [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Power-Law fit for attention score quantiles (log y-scale) over generation steps across 5 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: R2 distribution across 5 different models, evaluated across warmup sizes w: 64, 128, 256, and 512 steps. Left columns are samples on WikiText-2; right columns are samples on C4 negative values. These negative values indicate that the power-law fit performs worse than a…
Figure 9
Figure 9. Figure 9: Intended vs. Realized sparsity on SiftAttention across tasks and warmup lengths for 5 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Perplexity evaluation (WikiText-2) comparing SiftAttention with baselines across 5 models [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: SiftAttention short-context task performance vs. Realized Sparsity on IFEval and MATH [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Post-warmup attention latency of our Triton-based SiftAttention implementation compared [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Sparsity masks for τ = 0.5 (top) and τ = 0.9 (bottom) at the 2048th generation step when generating a sequence of 4096 tokens using Llama-3.1-8B-Instruct. The blacks represent the tokens not attended to, while the white represents the tokens that are attended to. C Ge…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 15 canonical work pages

  1. [1]

    https://openai

    Introducing triton: Open-source gpu programming for neural networks. https://openai. com/index/triton/, 2021

    work page 2021

  2. [2]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015

    work page 2015

  3. [3]

    Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding

    Konstantin Berestizshevsky, Renzo Andri, and Lukas Cavigelli. Top-theta attention: Sparsifying transformers by compensated thresholding. arXiv preprint arXiv:2502.08363, 2025

    work page Pith review arXiv 2025

  4. [4]

    Hardmath: A benchmark dataset for challenging problems in applied mathematics

    Jingxuan Fan, Sarah Martinson, Erik Y Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, and Michael P Brenner. Hardmath: A benchmark dataset for challenging problems in applied mathematics. arXiv preprint arXiv:2410.09988, 2024

    arXiv

  5. [5]

    Open llm leaderboard v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_ llm_leaderboard, 2024

    2024

  6. [6]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models, 2024

    2024

  7. [7]

    Memory-efficient transformers via top-k attention

    Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. Memory-efficient transformers via top-k attention. CoRR, abs/2106.06899, 2021

    arXiv 2021

  8. [8]

    Data movement is all you need: A case study on optimizing transformers

    Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: A case study on optimizing transformers. Proceedings of Machine Learning and Systems, 3:711–732, 2021

    work page 2021

Show all 31 references
  1. [9]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  2. [10]

    InfiniGen: Efficient generative inference of large language models with dynamic KV cache management

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages 155–172, Santa Clara, CA, July...

    2024

  3. [11]

    Accelerating attention through gradient-based learned runtime pruning

    Zheng Li, Soroush Ghodrati, Amir Yazdanbakhsh, Hadi Esmaeilzadeh, and Mingu Kang. Accelerating attention through gradient-based learned runtime pruning. In Proceedings of the 49th Annual International Symposium on Computer Architecture , pages 902–915, 2022

    2022

  4. [12]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. arXiv preprint arXiv:2305.17118, 2023

    2023 arXiv

  5. [13]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. CoRR, abs/1609.07843, 2016

    2016 arXiv

  6. [14]

    Linear log-normal attention with unbiased concentration

    Yury Nahshan, Joseph Kampeas, and Emir Haleva. Linear log-normal attention with unbiased concentration. arXiv preprint arXiv:2311.13541, 2023

    2023 arXiv

  7. [15]

    Nvidia nsight compute

    NVIDIA. Nvidia nsight compute. https://developer.nvidia.com/nsight-compute

  8. [16]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017

    2017

  9. [17]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023

    2023

  10. [18]

    Sparq attention: Bandwidth-efficient llm inference, 2023

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficient llm inference, 2023

    2023

  11. [19]

    AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning

    Siddharth Singh and Abhinav Bhatele. AxoNN: An asynchronous, message-driven parallel framework for extreme-scale deep learning. In Proceedings of the IEEE International Parallel & Distributed Processing Symposium, IPDPS ’22. IEEE Computer Society, May 2022

    2022

  12. [20]

    Democratizing AI: Open-source scalable LLM training on GPU-based supercomputers

    Siddharth Singh, Prajwal Singhania, Aditya Ranjan, John Kirchenbauer, Jonas Geiping, Yuxin Wen, Neel Jain, Abhimanyu Hans, Manli Shu, Aditya Tomar, Tom Goldstein, and Abhinav Bhatele. Democratizing AI: Open-source scalable LLM training on GPU-based supercomputers. In Proceedin...

    2024

  13. [21]

    Ranjan, Zack Sating, and Abhinav Bhatele

    Siddharth Singh, Prajwal Singhania, Aditya K. Ranjan, Zack Sating, and Abhinav Bhatele. A 4d hybrid algorithm to scale parallel training to thousands of gpus, 2024

    2024

  14. [22]

    Loki: Low-rank keys for efficient sparse attention

    Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems...

    2024

  15. [23]

    Pytorch 2.0: Our next generation release that is faster, more pythonic and dynamic as ever

    PyTorch Team. Pytorch 2.0: Our next generation release that is faster, more pythonic and dynamic as ever. https://pytorch.org/get-started/pytorch-2.0/ , 2023

    2023

  16. [24]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in neural information processing systems, pages 5998–6008, 2017

    2017

  17. [25]

    Longgenbench: Benchmarking long-form generation in long context llms

    Yuhao Wu, Ming Shan Hee, Zhiqing Hu, and Roy Ka-Wei Lee. Longgenbench: Benchmarking long-form generation in long context llms. arXiv preprint arXiv:2409.02076, 2024

    2024 arXiv

  18. [26]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  19. [27]

    Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation

    Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation. 2023. 11

    2023

  20. [28]

    Parallel top-k algorithms on gpu: A comprehensive study and new methods

    Jingrong Zhang, Akira Naruse, Xipeng Li, and Yong Wang. Parallel top-k algorithms on gpu: A comprehensive study and new methods. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23, New York, NY , USA,

  21. [29]

    H _2 o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H _2 o: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2306.14048, 2023

    2023 arXiv

  22. [30]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. 12 A Extended Power-Law Analysis Results A.1 Power-Law Samples 0 1000 2000 3000 4000 Generation Step 10 ...

    2023

  23. [2023]

    Association for Computing Machinery

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.