pith. sign in

arxiv: 2605.23081 · v1 · pith:TGGUCAKQnew · submitted 2026-05-21 · 💻 cs.LG

ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention

Joe Sharratt This is my paper

Pith reviewed 2026-05-25 05:27 UTC · model grok-4.3

classification 💻 cs.LG
keywords attentionmixed precisionquantizationlong contextFP4online softmaxinference efficiencyheuristic selection
0
0 comments X

The pith

Computing only 5% of query-key blocks in FP16 recovers on average 89.1% of the FP4-to-FP16 quality gap in long-context attention.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that quantisation error in low-bit attention affects final output quality in a highly uneven way, with the largest functional impact concentrated in the query-key interactions involving the most important tokens. A two-stage method first applies a rapid heuristic to identify a small set of these critical blocks, then computes those blocks in FP16 while running the remainder in FP4 and merges the partial results through online softmax. This selective approach delivers most of the quality of full FP16 attention while retaining the speed of FP4 inference. The recovery holds on average across multiple long-context benchmarks and model families, and the relative gain widens as sequence length increases.

Core claim

The output impact of quantisation error is highly non-uniform and increases with the importance of each query-key interaction, concentrating functionally relevant error in a small number of attention blocks that contain the most important tokens. ThriftAttention therefore runs a heuristic that rapidly selects a small number of important query-key block pairs for FP16 precision, computes the selected blocks in FP16 and the remaining blocks in FP4, and merges both paths via online softmax into a single output; across long-context benchmarks and model families this recovers on average 89.1% of the FP4-to-FP16 performance gap when only 5% of blocks are elevated to FP16.

What carries the argument

The two-stage selective mixed-precision mechanism that uses a heuristic to elevate only the most important query-key block pairs to FP16 computation while leaving the rest in FP4 and merges the results with online softmax.

If this is right

  • Quality degradation from uniform FP4 attention is largely eliminated while retaining FP4 inference speed.
  • The quality advantage over uniform FP4 grows as sequence length increases.
  • The recovery percentage is consistent across different model families and long-context benchmarks.
  • Near-FP16 output quality is reached by elevating only 5% of query-key blocks to FP16.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same selective elevation logic could be tested on other low-bit precisions or on different matrix multiplications inside the transformer.
  • Longer contexts could be handled without the quality drop that uniform FP4 currently produces.
  • Refining the selection heuristic might allow an even smaller high-precision fraction while preserving the same recovery level.

Load-bearing premise

The rapid heuristic accurately identifies the small set of query-key block pairs whose quantisation error has the largest functional impact on the final output.

What would settle it

An experiment on the same long-context benchmarks that measures quality recovery well below 89% of the FP4-to-FP16 gap when the 5% high-precision blocks are chosen by the stated heuristic.

Figures

Figures reproduced from arXiv: 2605.23081 by Joe Sharratt.

Figure 1
Figure 1. Figure 1: ThriftAttention approaches FP4 latency while preserving near-FP16 quality. Pareto frontier of negative log-likelihood (NLL) recovery vs inference efficiency at 131k context length (Qwen3-8B). Performance recovery is measured as the percentage of the FP4-to-FP16 NLL gap recovered. In this work we develop ThriftAttention, a training-free mixed-precision attention mechanism that delivers near-FP16 long-contex… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of ThriftAttention Quantised Attention. While post-training quantisation is well-established for linear lay￾ers [Dettmers et al., 2022, Frantar et al., 2022, Lin et al., 2024, Dettmers et al., 2023, Ashkboos et al., 2024, Liu et al., 2025], its extension to attention remains limited. SageAttention [Zhang et al., 2025d,a] accelerates attention via INT8/FP8 quantisation with outlier smoothing, and S… view at source ↗
Figure 3
Figure 3. Figure 3: Typical FP16→FP4 attention quantisation error, e = |PFP16 − PFP4|, by query/key blocks across layers and heads in Qwen3-8B (seq=4096) non-uniform such that tokens with large pre-softmax scores produce large pj through the softmax exponential, amplifying their own quantisation error. Conversely low attention scores dampen the effect of a key token’s quantisation error on o˜ [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
Figure 4
Figure 4. Figure 4: Kernel and end-to-end speedups over FlashAttention-2 for Prefill and Decode. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-token negative log-likelihood increase over the FP16 baseline ( [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
read the original abstract

Efficient attention algorithms are critical to mitigate the quadratic cost of attention in long-context workloads. Prior work utilises block-scaled quantisation techniques on Blackwell GPUs to move attention computation to 4-bit precision to accelerate inference. However, these techniques result in significant quality degradation in long-context settings. We show that the output impact of quantisation error is highly non-uniform and increases with the importance of each query-key interaction, concentrating functionally relevant error in a small number of attention blocks that contain the most important tokens. We propose ThriftAttention, a low-bit attention variant that delivers near-FP16 long-context quality at FP4 inference efficiency. This approach proceeds in two stages. First, a heuristic rapidly selects a small number of important query-key block pairs for FP16 precision. Second, the selected blocks are computed in FP16 and the remaining blocks in FP4, with both paths merged via online softmax into a single output. We demonstrate across long-context benchmarks and model families that by computing only 5% of query-key blocks in FP16, ThriftAttention recovers on average 89.1% of the FP4-to-FP16 performance gap. We show ThriftAttention's advantage grows with sequence length, mitigating the systematic FP4 quality degradation observed at longer contexts. The code is available at https://github.com/joesharratt1229/ThriftAttention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes ThriftAttention, a two-stage mixed-precision attention method for long-context inference. A heuristic first identifies a small set of important query-key block pairs (claimed to contain the most important tokens) for FP16 computation; the remaining blocks are computed in FP4. Both paths are merged using online softmax. The central empirical claim is that retaining only 5% of blocks in FP16 recovers on average 89.1% of the FP4-to-FP16 quality gap across long-context benchmarks and model families, with the advantage growing as sequence length increases. The code is released at the provided GitHub link.

Significance. If the heuristic reliably ranks blocks by the functional impact of their quantization error, the result would be a practical advance for FP4 attention on Blackwell-class GPUs, directly addressing the systematic quality degradation observed in prior block-scaled quantization work at long contexts. The public code release is a clear strength that enables reproducibility.

major comments (2)
  1. [Abstract / §3 (Method)] The paper provides no equation, pseudocode, or explicit definition for the first-stage heuristic that selects query-key blocks by 'importance of each query-key interaction.' Without this, it is impossible to verify whether the proxy actually correlates with output delta under FP4 vs. FP16 (the load-bearing assumption behind the 89.1% recovery figure).
  2. [§4 (Experiments)] No ablation is described that isolates the heuristic's accuracy (e.g., comparing the selected 5% blocks against an oracle selection based on actual per-block output error). This leaves open whether the reported recovery generalizes or depends on the specific heuristic chosen.
minor comments (1)
  1. [Abstract] The abstract states results 'across long-context benchmarks and model families' but does not list the exact models, datasets, or sequence lengths used; this should be stated explicitly in the main text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments point by point below.

read point-by-point responses

  1. Referee: [Abstract / §3 (Method)] The paper provides no equation, pseudocode, or explicit definition for the first-stage heuristic that selects query-key blocks by 'importance of each query-key interaction.' Without this, it is impossible to verify whether the proxy actually correlates with output delta under FP4 vs. FP16 (the load-bearing assumption behind the 89.1% recovery figure).

    Authors: The referee correctly notes that the manuscript does not supply an equation, pseudocode, or formal definition of the heuristic. Section 3 describes the selection criterion only at a high level (blocks whose quantization error has high functional impact on query-key interactions). We will add an explicit mathematical formulation together with pseudocode to §3 in the revision. revision: yes

  2. Referee: [§4 (Experiments)] No ablation is described that isolates the heuristic's accuracy (e.g., comparing the selected 5% blocks against an oracle selection based on actual per-block output error). This leaves open whether the reported recovery generalizes or depends on the specific heuristic chosen.

    Authors: We agree that an oracle ablation (ranking blocks by measured per-block output error under FP4 versus FP16 and comparing overlap with the heuristic) is missing. The current §4 reports only end-to-end recovery; we will insert this controlled ablation in the revised experiments section. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical validation of heuristic method

full rationale

The paper introduces ThriftAttention as a two-stage heuristic for selective FP16/FP4 attention computation and reports its quality recovery as an experimental outcome measured on long-context benchmarks. No equations, fitted parameters, or derivations are described that would make the 89.1% recovery figure equivalent to its inputs by construction. The selection heuristic is presented as a practical proxy without any self-referential definition or self-citation chain that bears the central claim. The result remains falsifiable via external benchmarks and does not reduce to renaming or ansatz smuggling.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

Abstract-only review; the ledger is therefore minimal and provisional.

free parameters (1)
  • fraction of blocks kept in FP16
    The 5% figure is presented as the operating point that achieves the reported recovery; its selection is not derived from first principles in the visible text.
axioms (1)
  • domain assumption Quantisation error impact is highly non-uniform across query-key interactions and concentrates in blocks containing the most important tokens.
    Stated directly in the abstract as the motivation for selective precision.

pith-pipeline@v0.9.0 · 5768 in / 1285 out tokens · 20576 ms · 2026-05-25T05:27:01.393459+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · 9 internal anchors

  1. [1]

    Castro, Andrei Panferov, Soroush Tabesh, Oliver Sieberling, Jiale Chen, Mahdi Nikdan, Saleh Ashkboos, and Dan Alistarh

    Roberto L. Castro, Andrei Panferov, Soroush Tabesh, Oliver Sieberling, Jiale Chen, Mahdi Nikdan, Saleh Ashkboos, and Dan Alistarh. Quartet: Native FP4 training can be optimal for large language models.arXiv preprint arXiv:2505.14669,

    work page arXiv

  2. [2]

    FP4 all the way: Fully quantized training of LLMs

    Brian Chmiel, Maxim Fishman, Ron Banner, and Daniel Soudry. FP4 all the way: Fully quantized training of LLMs.arXiv preprint arXiv:2505.19115,

    work page arXiv

  3. [3]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    RULER: What's the Real Context Size of Your Long-Context Language Models?

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    TurboAttention: Efficient attention approximation for high throughputs LLMs.arXiv preprint arXiv:2412.08585,

    Hao Kang, Srikant Bharadwaj, James Hensman, Tushar Krishna, Victor Ruhle, and Saravan Raj- mohan. TurboAttention: Efficient attention approximation for high throughputs LLMs.arXiv preprint arXiv:2412.08585,

    work page arXiv

  9. [9]

    Efficient Memory Management for Large Language Model Serving with PagedAttention , booktitle =

    doi: 10.1145/3600006.3613165. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. InAdvances in Neural Information Processing Systems,

    work page doi:10.1145/3600006.3613165

  10. [10]

    Ministral 3

    Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, et al. Ministral 3.arXiv preprint arXiv:2601.08584,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Ministral 3

    doi: 10.48550/arXiv.2601.08584. URLhttps: //arxiv.org/abs/2601.08584. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantiza- tion with learned rotations. InInternational Conference on Learning Representations,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2601.08584

  12. [12]

    Pretraining large language models with NVFP4

    NVIDIA, Felix Abecassis, Anjulie Agrusa, Dong Ahn, Jonah Alben, Stefania Alborghetti, et al. Pretraining large language models with NVFP4.arXiv preprint arXiv:2509.25149,

    work page arXiv

  13. [13]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Lev- skaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean

    doi: 10.1109/ISCA59077.2024.00019. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Lev- skaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling trans- former inference. InProceedings of Machine Learning and Systems,

    work page doi:10.1109/isca59077.2024.00019 2024

  14. [14]

    Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models.arXiv preprint arXiv:2401.04658,

    Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models.arXiv preprint arXiv:2401.04658,

    work page arXiv

  15. [15]

    Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537,

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning.arXiv preprint arXiv:2310.10537,

    work page arXiv

  16. [16]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Sch¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG- Bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051,

    work page 2023

  17. [17]

    Linformer: Self-Attention with Linear Complexity

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768,

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  18. [18]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. InInternational Conference on Learning Representations, 2025b. How...

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao

    doi: 10.18653/v1/2025.acl-long.1126. Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, and Tri Dao. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. arXiv preprint arXiv:2603.05451,

    work page doi:10.18653/v1/2025.acl-long.1126 2025

  20. [20]

    For a sequence withn=⌊L/64⌋key blocks, the integer top-kis chosen by (kn−k(k−1)/2)/(n(n+ 1)/2)≈f, rounded and clamped to[1, n]

    Target FP16 budgets are5%,10%, and25%. For a sequence withn=⌊L/64⌋key blocks, the integer top-kis chosen by (kn−k(k−1)/2)/(n(n+ 1)/2)≈f, rounded and clamped to[1, n]. Benchmarks.LongBench v1 is evaluated through lm-eval-harness on the English subset: narrativeqa,qasper,multifieldqa en,hotpotqa,2wikimqa,musique,gov report, qmsum,multi news,trec,triviaqa,sa...

    work page 2048