Output-Aware Rotation for INT2 KV-Cache Quantization

Minsoo Cheong; Murali Annavaram; Sai Praneeth Karimireddy; Sungjoo Yoo; Sunwoo Lee; Vincent-Daniel Yun; Woosang Lim

REVIEW 2 major objections 4 minor 51 references

Output-Aware Rotation for INT2 KV-Cache Quantization

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the right objective for INT2 KV-cache quantization is the attention-output error after the output projection $W_O$, and that learning per-head rotations against it brings 2-bit cache accuracy close to BF16.

desk verdict OptR is a genuinely new output-aware rotation objective for INT2 KV-cache quantization with large, consistent gains; the main caveat is that GPQA-Diamond is used for both calibration and evaluation. read the letter →

arxiv 2608.02691 v2 pith:SZ5F37BQ submitted 2026-08-03 cs.LG cs.AI

Vincent-Daniel Yun , Woosang Lim , Minsoo Cheong , Sunwoo Lee , Murali Annavaram , Sai Praneeth Karimireddy , Sungjoo Yoo This is my paper · ORCID

classification cs.LGcs.AI

keywords KVcachequantizationINT2rotation-basedoutput-awareobjectiveorthogonalrotationrefinementattention-equivalentkeycenteringper-headcalibrationlong-contextLLMinference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Compressing the KV cache of a large language model to 2 bits per element would shrink memory and bandwidth for long-context inference, but INT2 quantization severely degrades accuracy unless the cache is first rotated into a friendlier basis. Existing rotation-based methods pick that basis using cache statistics or the error in reconstructing the keys and values themselves, before the attention readout. This paper argues those are proxy objectives: the model actually feels the error in the attention output after the output projection $W_O$, the signal that enters the residual stream. OptR is the method built on that claim — it decomposes the post-$W_O$ error into a key-induced term and a value-induced term, then learns per-head orthogonal corrections to any base rotation by minimizing that error through the full quantized-attention path, after subtracting a per-channel key mean that leaves the softmax distribution unchanged. The result is INT2 KV-cache accuracy close to BF16 on reasoning and coding benchmarks and sharply better long-context retrieval, at 2.32 effective bits per element and negligible serving overhead.

What carries the argument

The load-bearing identity is the exact decomposition of the post-$W_O$ attention-output error, $\Delta y = \left(\sum_s \Delta p_s\, v_s\right) W_O^\top + \left(\sum_s \tilde{p}_s\, \Delta v_s\right) W_O^\top$, which converts 'choose a good rotation' into a concrete output-space objective. Three mechanisms carry the argument: (1) per-head orthogonal corrections parameterized by skew-symmetric generators $S=A-A^\top$ with $R=R_0\exp(S)\in O(d)$, which keep the corrected rotations orthogonal while remaining freely optimizable; (2) key centering, $k\mapsto k-\mu$, applied before rotation and quantization, which is attention-equivalent because subtracting the same mean from every key adds a query-dependent constant to all logits and thus leaves the softmax distribution unchanged while narrowing the dynamic range the INT2 grid must represent; and (3) a straight-through estimator for the non-differentiable INT2 rounding, so gradients flow through rotation, clipping, grouping, rounding, dequantization, and inverse rotation. The key rotation is optimized before the value rotation because the quantized keys determine the attention distribution that weights the value errors.

What would settle it

Recalibrate OptR on traces from a starkly different distribution — coding prompts or very long synthetic retrieval documents — and compare the learned rotations and downstream accuracy against the GPQA-calibrated ones on both AIME25 and a 128K retrieval task. If the rotations shift substantially and the GPQA-calibrated gains vanish on the held-out tasks, the objective is memorizing the calibration distribution rather than minimizing structural post-$W_O$ error; the paper's account predicts the objective transfers, since post-$W_O$ error is a property of the attention readout rather than of the task.

Watch

Extended reading notes

Core claim

The central claim is that the orthogonal rotation used for INT2 KV-cache quantization should be chosen to minimize the post-$W_O$ attention-output error $E_{\mathrm{out}}=\|\tilde{y}-y\|_2^2$ rather than the cache-reconstruction error $\|K-\tilde{K}\|_F^2+\|V-\tilde{V}\|_F^2$. The paper derives an exact decomposition of $E_{\mathrm{out}}$ into a key-induced term, where quantized keys perturb the softmax attention distribution and the change is then weighted by values and mapped by $W_O$, and a value-induced term, where quantized values are aggregated by the quantized-key attention distribution and mapped by $W_O$. Because these terms weight cache errors by how much attention actually reads them, they can favor different rotations than reconstruction does — that is the objective mismatch the method exploits. OptR learns, per KV head and per layer, an orthogonal correction $R_0\exp(S)$ with skew-symmetric generator $S=A-A^\top$, first for keys (with a KL term that keeps the attention distribution close) and then for values under the fixed quantized-key path, backpropagating through the entire INT2 rotation-clipping-grouping-rounding path with a straight-through estimator. It also centers keys by a per-channel mean before rotation, a reparameterization that provably leaves softmax unchanged while shrinking the range INT2 must cover. On Qwen3-8B this lifts AIME25 accuracy from 17.33% (QuaRot) and 54.67% (OSCAR) to 66.67% and 66.00%, against 68.00% for BF16, at 2.32 effective bits per cache element.

Load-bearing premise

The per-head rotations and the key-centering mean are estimated from a few thousand tokens of GPQA-Diamond decoding traces, and the method assumes those traces are representative enough of all downstream tasks and of context lengths up to 128K tokens that the same fixed rotations transfer without retraining.

Editorial extensions

If this is right

The same 2.32-bit cache that collapses under existing rotations is largely recoverable: with OptR, Qwen3-8B AIME25 reaches 66.67% (from 17.33% with QuaRot) and 66.00% (from 54.67% with OSCAR), against 68.00% for BF16.
Output-space calibration transfers across rotation initializations, so the gain comes from the objective rather than from a particular rotation construction.
Long-context retrieval at 64K to 128K tokens, where plain rotated INT2 falls to near-zero accuracy, keeps most of its BF16-level performance with OptR (e.g., Qwen3-8B 64K RULER accuracy from 0.04-57.54% to 68.65-70.02%, against 84.22% for BF16).
The method stays deployment-compatible: it preserves the paged KV-cache layout, folds value-side rotations into the $W_V$ and $W_O$ weights, and adds no more than about 2% overhead in decode latency, throughput, or prefill time.
Calibration is cheap and stable: a few thousand tokens and 80 Adam steps per head suffice, with AIME25 accuracy roughly flat from 3.1K to 26K calibration tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The decomposition predicts a testable pattern the paper does not report: models with sharper attention distributions should gain more from key-rotation calibration, because key error only matters where softmax weight is large — one could rank heads by attention entropy and correlate with per-head OptR gains.
The output-aware principle generalizes beyond KV caches: the same recipe of measuring quantization error after the consuming operation could be applied to quantized activations or weights inside later layers, but the paper makes no such claim.
Key centering is softmax-equivalent only in full precision; under the four-level INT2 grid, centering changes which values fall into which quantization bins, so part of the gain may come from bin reassignment rather than pure range reduction — the paper does not isolate these two effects.
The GPQA-derived calibration is the main transfer risk: the reported 128K retrieval and coding gains are evidence of transfer but not a guarantee, and calibrating instead on coding or retrieval traces would test whether the learned rotations are task-agnostic.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper up front. First, the core idea is real and new: instead of picking rotations from fixed transforms or proxy reconstruction error, OptR optimizes per-head orthogonal corrections against the post-WO attention-output error, with an exact decomposition into key- and value-induced terms. Second, the experimental gains are large enough that you should care, even after discounting one benchmark column. On Qwen3-8B, QuaRot-INT2 goes from 17.33 to 66.67 on AIME25, and OSCAR+OptR reaches 66.00 against BF16's 68.00. That is not noise.

What the paper does well: the error decomposition in Eq. 8 is clean, the skew-symmetric parameterization keeps the corrections orthogonal throughout training, and the key-centering reparameterization is attention-equivalent. The ablation in Table 4 is the right kind of evidence: it directly compares cache-reconstruction, pre-WO, and post-WO objectives and shows the post-WO objective wins. The long-context RULER results also hold up, with OptR pushing 64K and 128K retrieval well above the base rotations.

Where the soft spots are, in proportion:

The calibration/evaluation overlap on GPQA-Diamond is a real problem, and the paper admits it. Calibration and held-out rotation selection both draw from the 198 GPQA-Diamond prompts, and GPQA-Diamond is also a reported evaluation benchmark. So the GPQA column in Table 1 is partly test-set tuned. This does not kill the paper, because the AIME24/25, MBPP+, LCB, and RULER gains come from data the rotations never saw. But the "five benchmarks" claim is only four benchmarks if you are strict, and the authors need to run a calibration set from a different domain (e.g., general code or dialogue traces) and show the GPQA and AIME gains survive. That is a one-week experiment and should be a revision requirement.

The second weakness is minor: the stated objective is joint post-WO output error, but the implementation optimizes key and value corrections sequentially. The ablation suggests this is still the right direction, and the paper could just soften the claim to "approximately minimizes" or show a joint objective in the appendix.

Third, no code is released yet despite a GitHub placeholder. The implementation details are specific enough that a determined reader could reproduce, but given the calibration issue, code would help the reviewers verify that the GPQA numbers are not an artifact of the selection procedure.

My overall take: the central mechanism is sound, the empirical evidence is strong on the non-calibration benchmarks, and the paper deserves a serious referee. I would send it out with a request for a non-GPQA calibration check and code, not a desk reject. The reader's conditional verdict is about right, though I would put a bit more weight on the AIME/RULER results when judging the method's transferability.

Referee Report

2 major / 4 minor

Summary. The paper proposes OptR, a calibration-based method for INT2 KV-cache quantization. Given an existing base rotation such as QuaRot or OSCAR, OptR first centers the post-RoPE keys with a per-channel mean and then learns per-head orthogonal correction rotations parameterized as exp(A-A^T), propagated through the full INT2 quantization and attention path, using a post-WO attention-output error objective. The exact error decomposition in Eq. (8) splits the post-WO output error into key-induced and value-induced terms; the method calibrates key rotations first with values in BF16 and then value rotations with the quantized-key attention fixed. Experiments on three models report large accuracy gains over both base rotations on AIME24/25, GPQA-Diamond, MBPP+, LiveCodeBench, and RULER-NIAH, with ablations showing that post-WO objectives outperform cache-reconstruction and pre-WO objectives, and with measurements indicating negligible inference overhead.

Significance. If the results hold, OptR is a practically valuable step for INT2 KV-cache serving. The paper provides an exact and useful error decomposition (Eq. 8), demonstrates large and consistent gains over two rotation initializations across several independent benchmarks, shows strong long-context retrieval gains at 64K and 128K, and preserves the standard paged KV-cache layout with negligible runtime overhead. The explicit confidence intervals, the objective ablations (Tables 3 and 4), and the calibration-size sensitivity study (Table B) are welcome strengths. The main caveat is that the GPQA-Diamond column of Table 1 is not an independent generalization measurement for this calibration scheme, because both calibration and held-out rotation selection use GPQA-Diamond prompts. In addition, the algorithm optimizes a sequential surrogate rather than directly minimizing the stated joint post-WO objective, so the paper overstates the objective it actually optimizes.

major comments (2)

[Appendix, Calibration details; Table 1] The calibration and held-out rotation-selection data both come from the GPQA-Diamond evaluation pool, as stated in the Appendix: 'We collect Q/K/V activation statistics from all 198 GPQA-Diamond prompts' and 'their source prompts come from the GPQA-Diamond evaluation pool.' The per-head rotations and the held-out selection step are therefore tuned on the same distribution as the GPQA-Diamond column of Table 1, so that column is not a valid held-out accuracy measurement. Because the central claim is that OptR 'consistently improves' across five benchmarks, the validity of this column is load-bearing. Please add a cross-domain calibration experiment, for example calibrating on AIME or MBPP traces and then evaluating all benchmarks, or explicitly relabel the GPQA-Diamond results as in-distribution performance rather than as evidence of generalization.
[Eqs. (8), (10), (14), (16); Algorithm 1] The paper states that OptR minimizes the post-WO attention-output error E_out (Eq. 10), but the algorithm does not directly optimize E_out. Key calibration (Eq. 14) uses a KL term plus lambda_K times the key-induced error with values in BF16, and value calibration (Eq. 16) then minimizes only the value-induced term with the quantized-key attention distribution fixed. This is a block-coordinate surrogate: the cross term between the key-induced and value-induced contributions in Eq. (8) is never evaluated, and the joint objective in Eq. (10) is not optimized. Please state explicitly that OptR minimizes a sequential approximation of E_out and justify or empirically test why this coordinate-wise ordering is appropriate, or revise the abstract and contribution claims from 'minimizes' to 'reduces via a sequential surrogate.'

minor comments (4)

[Appendix, Calibration details] The sentence 'Key-centering statistics are estimated from 28 full model-generated traces' does not state whether these traces are GPQA-Diamond traces or an independent corpus; please clarify the source and whether they overlap with the calibration or evaluation splits.
[Algorithm 1] The symbol e_K is used both for the effective keys (line 6) and for the key-induced error e_K(t,j) (line 8); renaming one of these quantities would remove a needless source of confusion.
[Section 5.1, Table 2] The long-context results are strong, but the calibration traces appear to be short prefill chunks from GPQA prompts; please state the maximum input length in the calibration traces and explicitly note that the 128K RULER performance is an extrapolation beyond the calibration distribution, not an in-distribution measurement.
[Figure 4 caption] The caption defines 'Output error' and 'Hidden state error' informally; since the appendix gives precise normalization formulas for both metrics, please add a pointer to the appendix in the caption so that readers can interpret the logarithmic y-axes correctly.

Circularity Check

1 steps flagged · score 5.0 of 10

GPQA-Diamond is used both to calibrate and select OptR's rotations and to report a headline benchmark, making that column a fitted result; the remaining benchmarks keep the core claim partially independent.

fitted input called prediction [Appendix A 'Calibration details'; Table 1 GPQA-Diamond column; Section 5.1]
"For each model, we collect a 30K-token pool of BF16 GPQA QKV traces and use disjoint subsets for calibration and held-out rotation selection. ... The calibration and held-out chunks do not overlap, although their source prompts come from the GPQA-Diamond evaluation pool."

OptR's per-head orthogonal corrections and the key-centering mean are optimized on GPQA-Diamond traces, and the held-out selection step chooses the best rotation using held-out GPQA-Diamond chunks. Table 1 then reports GPQA-Diamond accuracy as an independent benchmark, and the paper claims 'consistent' improvements across it. The GPQA-Diamond column therefore measures in-distribution tuning rather than out-of-distribution prediction: the rotations have already been fit and selected on the same prompt pool being scored. This is the fitted-input-called-prediction pattern for that column. The AIME24/25, MBPP+, LiveCodeBench, and RULER results are not calibration sources and remain independent evidence, so the circularity is partial.

full rationale

The mathematical derivation in Sections 3 and 4 is self-contained: the post-WO error decomposition in Eq. (8) is an algebraic identity, and the optimization objective Eout is not derived from benchmark accuracies. The method is evaluated against external benchmarks that are not calibration sources, so the central idea is not circular. The one substantive circular-evidence problem is the calibration and held-out rotation selection both drawing from the GPQA-Diamond evaluation pool, which the appendix explicitly admits. This makes the GPQA-Diamond column in Table 1 a test-set-tuned result rather than an independent generalization measurement. Because four other independent benchmarks and long-context results still support the main claim, the paper is only partially circular, not wholly reducible to its inputs.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central machinery of OptR is a calibration procedure: it fits per-head orthogonal rotations and a key centering mean on a small calibration set. The listed free parameters are the fitted values that the results depend on. The axioms are standard mathematical facts plus domain assumptions about calibration representativeness and gradient estimation. No new physical entities are introduced.

free parameters (6)

per-channel key mean mu = estimated from calibration traces (e.g., 28 full model-generated traces; per-layer/per-head)
Used in key reparameterization; estimated on calibration data, not derived.
key rotation corrections A_K (per head) = optimized matrices exp(A_K - A_K^T) via Adam with STE
Learned to minimize L_K on calibration traces. These are the core fitted objects.
value rotation corrections A_V (per head) = optimized matrices exp(A_V - A_V^T) via Adam with STE
Learned to minimize L_V on calibration traces.
lambda_K loss weight = 1.0 (best for OSCAR; QuaRot best at 10, see Appendix C)
Hand-chosen after sensitivity analysis; affects the objective balance.
clipping ratios c_K, c_V = 0.96 keys, 0.92 values
Adopted from OSCAR configuration, not tuned by OptR.
BF16 sink/recent window sizes = 64 sink tokens, 256 recent tokens
Fixed hyperparameters from the baseline pipeline; affect effective 2.32 BPE.

assumptions (6)

standard math Subtracting a per-channel constant from all cached keys leaves the full-precision softmax distribution unchanged.
Used in Section 4.1 for key reparameterization. Holds only before quantization; after INT2 rounding the distribution changes, which is intended.
standard math The matrix exponential of a skew-symmetric matrix is orthogonal, so R0 exp(A-A^T) is a valid rotation.
Used in Section 4.2 to enforce orthogonality during optimization.
domain assumption The straight-through estimator provides usable gradients for INT2 rounding.
Section 4.5, Eq (18). Standard practice but unverified for this objective.
domain assumption Gradients computed on the final 64 query positions of calibration chunks transfer to all decoding positions.
The objective is computed over the last 64 query positions only (Algorithm 1, line 8). This assumes representativeness.
domain assumption The GPQA-Diamond calibration traces are representative of all evaluation tasks and context lengths.
Calibration details in Appendix. If false, the learned rotations will not generalize.
domain assumption Pre-existing rotation baselines (QuaRot, OSCAR) are correctly implemented by their official codebases.
OptR builds on the official OSCAR codebase; comparisons inherit any baseline implementation issues.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Output-Aware Rotation for INT2 KV-Cache Quantization." pith.science (2026). https://pith.science/paper/SZ5F37BQ

@misc{pith2026260802691,
  author       = {Pith},
  title        = {Pith review of: Output-Aware Rotation for INT2 KV-Cache Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZ5F37BQ}},
  note         = {Machine review of arXiv:2608.02691}
}

abstract

The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important. However, existing rotation-based INT2 methods optimize cache statistics or proxy errors before the complete attention readout, even though the model is ultimately affected by the error propagated through attention and the output projection $W_O$. To address this mismatch, we propose \textit{OptR}, an output-aware rotation method that minimizes post-$W_O$ attention-output error. OptR decomposes the post-$W_O$ attention-output error into key- and value-induced terms and learns per-head orthogonal corrections through the full INT2 quantization and attention path. OptR further applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without changing the softmax distribution. Across three models and five reasoning and coding benchmarks, OptR consistently improves both QuaRot and OSCAR and strengthens long-context retrieval, while preserving the paged KV-cache format with negligible inference overhead.

Figures

Figures reproduced from arXiv: 2608.02691 by the authors.

**Figure 2.** Key magnitude (top) and key-induced attention-output error by cached token (bottom) across five INT2 KV [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Overview of OptR. OptR augments an existing rotation-based KV-cache quantization pipeline with channel [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Per-layer errors under INT2 KV-cache quantization on Qwen3-4B-Thinking-2507. Panels (a,b) show post- [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Output error by layer and KV head relative to naive INT2 (Qwen3-4B-Thinking-2507, GPQA-Diamond). [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Efficiency analysis of OptR integrated into the optimized rotated INT2 KV cache pipeline on an NVIDIA [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 5 linked inside Pith

[1]

2026 , eprint=

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization , author=. 2026 , eprint=

2026
[2]

Softmax is \ 1/2\ -Lipschitz: A tight bound across all \

Pravin Nair , journal=. Softmax is \ 1/2\ -Lipschitz: A tight bound across all \. 2026 , url=

2026
[3]

2018 , eprint =

On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning , author =. 2018 , eprint =

2018
[4]

and Neuhoff, D.L

Gray, R.M. and Neuhoff, D.L. , journal=. Quantization , year=
[5]

2026 , eprint=

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks , author=. 2026 , eprint=

2026
[6]

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

RotateKV: accurate and robust 2-bit KV cache quantization for LLMs via outlier-aware adaptive rotations , author=. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=
[7]

Advances in Neural Information Processing Systems , volume=

Quarot: Outlier-free 4-bit inference in rotated llms , author=. Advances in Neural Information Processing Systems , volume=
[8]

2025 , booktitle =

Yun, Juyoung and Choi, Sol and Rameau, Francois and Kang, Byungkon and Fu, Zhoulai , title =. 2025 , booktitle =

2025

Show all 51 references

[9]

and Ermon, Stefano and Rudra, Atri and R\'

Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R\'. FLASHATTENTION: fast and memory-efficient exact attention with IO-awareness , year =
[10]

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (

Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebr. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (

2023
[11]

H2O: heavy-hitter oracle for efficient generative inference of large language models , year =

Zhang, Zhenyu and Sheng, Ying and Zhou, Tianyi and Chen, Tianlong and Zheng, Lianmin and Cai, Ruisi and Song, Zhao and Tian, Yuandong and R\'. H2O: heavy-hitter oracle for efficient generative inference of large language models , year =
[12]

and Shao, Yakun Sophia and Keutzer, Kurt and Gholami, Amir , title =

Hooper, Coleman and Kim, Sehoon and Mohammadzadeh, Hiva and Mahoney, Michael W. and Shao, Yakun Sophia and Keutzer, Kurt and Gholami, Amir , title =. 2024 , booktitle =

2024
[13]

Efficient Memory Management for Large Language Model Serving with

Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , booktitle =. Efficient Memory Management for Large Language Model Serving with
[14]

and Barrett, Clark and Sheng, Ying , title =

Zheng, Lianmin and Yin, Liangsheng and Xie, Zhiqiang and Sun, Chuyue and Huang, Jeff and Yu, Cody Hao and Cao, Shiyi and Kozyrakis, Christos and Stoica, Ion and Gonzalez, Joseph E. and Barrett, Clark and Sheng, Ying , title =. 2024 , booktitle =

2024
[15]

SpinQuant:

Zechun Liu and Changsheng Zhao and Igor Fedorov and Bilge Soran and Dhruv Choudhary and Raghuraman Krishnamoorthi and Vikas Chandra and Yuandong Tian and Tijmen Blankevoort , booktitle=. SpinQuant:
[16]

International Conference on Machine Learning , pages=

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache , author=. International Conference on Machine Learning , pages=. 2024 , organization=

2024
[17]

Kitty: Accurate and Efficient 2-bit

Haojun Xia and Xiaoxia Wu and Jisen Li and Tsai-chuan Wu and Junxiong Wang and Jue WANG and Chenxi Li and Aman Singhal and Alay Dilipbhai Shah and Alpay Ariyak and Donglin Zhuang and Zhongzhu Zhou and Ben Athiwaratkun and Zhen Zheng and Shuaiwen Leon Song , booktitle=. Kitty: ...

2026
[18]

The Fourteenth International Conference on Learning Representations , year=

TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate , author=. The Fourteenth International Conference on Learning Representations , year=
[19]

Jerry Chee and Yaohui Cai and Volodymyr Kuleshov and Christopher De Sa , booktitle=. Qu. 2023 , url=

2023
[20]

Advances in Neural Information Processing Systems , volume=

NSNQuant: A double normalization approach for calibration-free low-bit vector quantization of KV cache , author=. Advances in Neural Information Processing Systems , volume=
[23]

AIME 2024: American Invitational Mathematics Examination , author =

2024
[24]

AIME 2025: American Invitational Mathematics Examination , author =

2025
[25]

Bowman , booktitle=

David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , booktitle=. 2024 , url=

2024
[26]

Is Your Code Generated by Chat

Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , booktitle =. Is Your Code Generated by Chat. 2023 , url =

2023
[27]

The Thirteenth International Conference on Learning Representations , year=

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author=. The Thirteenth International Conference on Learning Representations , year=
[28]

2024 , url=

Cheng-Ping Hsieh and Simeng Sun and Samuel Kriman and Shantanu Acharya and Dima Rekesh and Fei Jia and Boris Ginsburg , booktitle=. 2024 , url=

2024
[30]

Advances in neural information processing systems , volume=

Attention is all you need , author=. Advances in neural information processing systems , volume=
[31]

arXiv preprint arXiv:2605.19660 , year=

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond , author=. arXiv preprint arXiv:2605.19660 , year=

arXiv
[32]

Fu, Stefano Ermon, Atri Rudra, and Christopher R\' e

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R\' e . Flashattention: fast and memory-efficient exact attention with io-awareness. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022

2022
[33]

GQA : Training generalized multi-query transformer models from multi-head checkpoints

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. GQA : Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

2023
[34]

H2o: heavy-hitter oracle for efficient generative inference of large language models

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\' e , Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: heavy-hitter oracle for efficient generative inference of large language models. In Proceedings of ...

2023
[35]

Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: towards 10 million context length llm inference with kv cache quantization. In Proceedings of the 38th International Conference on Neural Information...

2024
[36]

Oscar: Offline spectral covariance-aware rotation for 2-bit kv cache quantization

Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, and Xiaoxia Wu. Oscar: Offline spectral covariance-aware rotation for 2-bit kv cache quantization. 2026. URL https://arxiv.org/abs/2605.17757

2026 arXiv
[37]

Qu IP : 2-bit quantization of large language models with guarantees

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Qu IP : 2-bit quantization of large language models with guarantees. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=xrk9g5vcXR

2023
[38]

Quarot: Outlier-free 4-bit inference in rotated llms

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37: 0 100213--100240, 2024

2024
[39]

Efficient memory management for large language model serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th Symposium on Operating Systems Principles (...

2023
[40]

Gonzalez, Clark Barrett, and Ying Sheng

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: efficient execution of structured language model programs. In Proceedings of the 38th Interna...

2024
[41]

Rotatekv: accurate and robust 2-bit kv cache quantization for llms via outlier-aware adaptive rotations

Zunhai Su, Hanyu Wei, Zhe Chen, Wang Shen, Linge Li, Huangqi Yu, and Kehong Yuan. Rotatekv: accurate and robust 2-bit kv cache quantization for llms via outlier-aware adaptive rotations. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intellige...

2025
[42]

Kivi: A tuning-free asymmetric 2bit quantization for kv cache

Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In International Conference on Machine Learning, pages 32332--32344. PMLR, 2024

2024
[43]

Kitty: Accurate and efficient 2-bit KV cache quantization with dynamic channel-wise precision boost

Haojun Xia, Xiaoxia Wu, Jisen Li, Tsai chuan Wu, Junxiong Wang, Jue WANG, Chenxi Li, Aman Singhal, Alay Dilipbhai Shah, Alpay Ariyak, Donglin Zhuang, Zhongzhu Zhou, Ben Athiwaratkun, Zhen Zheng, and Shuaiwen Leon Song. Kitty: Accurate and efficient 2-bit KV cache quantization ...

2026
[44]

Nsnquant: A double normalization approach for calibration-free low-bit vector quantization of kv cache

Donghyun Son, Euntae Choi, and Sungjoo Yoo. Nsnquant: A double normalization approach for calibration-free low-bit vector quantization of kv cache. Advances in Neural Information Processing Systems, 38: 0 43124--43159, 2026

2026
[45]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

2017
[46]

Qwen3 technical report

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

2025 arXiv
[47]

Phi-4-reasoning technical report

Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318, 2025

2025 arXiv
[48]

Aime 2025: American invitational mathematics examination

MAA . Aime 2025: American invitational mathematics examination. https://maa.org/math-competitions/aime, 2025

2025
[49]

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA : A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98

2024
[50]

Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrev...

2023
[51]

Livecodebench: Holistic and contamination free evaluation of large language models for code

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learn...

2025
[52]

RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling, 2024

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=kIoBbc76Sy

2024
[53]

Adam: A method for stochastic optimization

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[54]

Turboquant: Online vector quantization with near-optimal distortion rate

Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. Turboquant: Online vector quantization with near-optimal distortion rate. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=tO3ASKZlok

2026

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

2026 , eprint=

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization , author=. 2026 , eprint=

2026

[2] [2]

Softmax is \ 1/2\ -Lipschitz: A tight bound across all \

Pravin Nair , journal=. Softmax is \ 1/2\ -Lipschitz: A tight bound across all \. 2026 , url=

2026

[3] [3]

2018 , eprint =

On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning , author =. 2018 , eprint =

2018

[4] [4]

and Neuhoff, D.L

Gray, R.M. and Neuhoff, D.L. , journal=. Quantization , year=

[5] [5]

2026 , eprint=

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks , author=. 2026 , eprint=

2026

[6] [6]

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

RotateKV: accurate and robust 2-bit KV cache quantization for LLMs via outlier-aware adaptive rotations , author=. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

[7] [7]

Advances in Neural Information Processing Systems , volume=

Quarot: Outlier-free 4-bit inference in rotated llms , author=. Advances in Neural Information Processing Systems , volume=

[8] [8]

2025 , booktitle =

Yun, Juyoung and Choi, Sol and Rameau, Francois and Kang, Byungkon and Fu, Zhoulai , title =. 2025 , booktitle =

2025