Pith. sign in

REVIEW 6 minor 49 references

Adding the gradient covariance to the layer Hessian makes 2-bit LLM weight quantization work where activation-only methods collapse.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 14:37 UTC pith:S4FLMGRF

load-bearing objection Clean engineering win: HG for BiIP + joint-trace mixed precision rescues LLaMA-3-70B 2-bit while keeping the GPTAQ solver intact.

arxiv 2607.07964 v1 pith:S4FLMGRF submitted 2026-07-08 cs.LG

KronQ: LLM Quantization via Kronecker-Factored Hessian

Donghyun Lee , Yuhang Li , Ruokai Yin , Priyadarshini Panda This is my paper
classification cs.LG
keywords post-training quantizationKronecker-factored Hessiangradient covarianceincoherence processingmixed-precision allocationLLM compressionOBS quantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard second-order post-training quantizers for large language models rebuild each layer using only the covariance of its input activations. That choice treats every output channel as equally important. KronQ replaces that assumption with the Kronecker-factored Hessian, which multiplies the activation covariance by the gradient covariance. The product is used in two places: a bidirectional rotation that spreads weight magnitudes evenly across both input and output dimensions, and a joint-trace sensitivity score that ranks sublayers for mixed-precision bit allocation. Because the gradient factor cancels algebraically inside the column-wise weight update, the core quantizer stays as cheap as the best existing activation-only solver. On LLaMA models from 7B to 70B the method yields the largest gains at 2-bit, and on LLaMA-3-70B it produces a usable 7.93 WikiText-2 perplexity while the activation-only baselines diverge above 2000.

Core claim

Under the Kronecker-factored approximation of the weight Hessian, the layer-wise quantization loss depends on both the activation covariance and the gradient covariance. Incorporating the latter through bidirectional incoherence processing and a joint-trace sensitivity score yields stable, low-perplexity 2-bit weight quantizations on LLaMA-scale models where activation-only second-order methods fail.

What carries the argument

The Kronecker-factored Hessian H ≈ HX ⊗ HG. HG is estimated from one backward pass, used to rotate and rescale the output side of the weights, then cancels from the column-wise OBS update, leaving only the GPTAQ-style solver plus a free mixed-precision ranking tr(HG)·tr(HX).

Load-bearing premise

The method assumes that inputs and gradients are independent enough for the Kronecker product to be a faithful Hessian, and that a single backward pass over 128 short calibration sequences gives a reliable gradient covariance for every layer.

What would settle it

Replace the single-pass WikiText-2 HG with either a multi-task average or an identity matrix and re-run the identical 2-bit LLaMA-3-70B pipeline; if perplexity rises back above several hundred, the claimed necessity of the gradient factor is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 6 minor

Summary. KronQ is a post-training quantization method for LLMs that replaces the usual input-only proxy Hessian HX with the Kronecker-factored form H ≈ HX ⊗ HG, where HG is the output-side gradient covariance estimated from a single backward pass. The paper shows that HG cancels algebraically in the column-wise OBS update (Proposition 1), so the solver remains exactly GPTAQ, while HG is used for two complementary purposes: (i) bidirectional incoherence processing (BiIP) that rescales and rotates both input and output dimensions, and (ii) an inter-layer mixed-precision score tr(HG)·tr(HX) that differentiates sublayers sharing the same input. Empirically, KronQ reports state-of-the-art WikiText-2 perplexity and zero-shot accuracy on LLaMA-2/3 (7B–70B) at W2/W3/W4, including the striking result that LLaMA-3-70B W2 reaches 7.93 PPL while GPTQ/GPTAQ diverge or produce degenerate quantizations (>2000 PPL). Supporting material includes rotation-invariance and LDLQ-bound proofs, ablations isolating BiIP directions and diagonal rescaling, outlier analysis of LLaMA-3-70B, and extensions to group, weight-activation, mixed-precision, and newer model families.

Significance. If the results hold, KronQ supplies a practical, low-overhead way to inject output-side second-order information into the dominant GPTQ-style pipeline without changing the online solver. The LLaMA-3-70B W2 rescue (7.93 vs. divergence) and the consistent W2/W3 gains across model scales are of immediate deployment interest, and the mixed-precision score tr(HG)·tr(HX) cleanly breaks the Q/K/V degeneracy that pure-HX metrics cannot resolve. Strengths that raise confidence include clean algebraic cancellation (Prop. 1), rotation invariance (Thm. 1), an extended LDLQ bound (Prop. 2), systematic ablations (Table 7), multi-family evaluation (Tables 1–5, 13–17), and a public code link. The K-FAC independence assumption and single-pass HG estimate are standard modeling choices rather than hidden free parameters; free parameters remain the usual calibration size, damping, and block size.

minor comments (6)
  1. Section 4.1 / Eq. (7): a short empirical check that the single-pass HG (128 WikiText-2 sequences) is stable under different calibration draws or a modest domain shift would strengthen the claim that BiIP and the sensitivity score are robust; this is not load-bearing for the reported numbers but would close the weakest modeling assumption.
  2. Figure 1(a) and Figure 2: axis labels and the precise definition of the normalized diagonal / µ/√dout would benefit from a one-sentence caption clarification so that the coherence drop is immediately readable without the main text.
  3. Appendix D / Figure 6: the CVin = 9.21 outlier analysis for LLaMA-3-70B is persuasive; a brief cross-reference from the main-text discussion of the 70B failure modes (Section 5.2) would help readers locate it.
  4. Table 1 and related tables: a few entries still show “–” or “NaN” without a uniform footnote explaining whether the baseline was not run, diverged, or was unavailable; a single convention would improve readability.
  5. Section 5.4 / Figure 5: the 70B latency comparison notes a two-GPU bf16 baseline; stating the exact multi-GPU communication setting (or marking it as approximate) would avoid over-interpretation of the speedup factor.
  6. Related work: a one-sentence contrast with YAQA’s power-iterated Hessian sketches (already present but brief) could more explicitly highlight the calibration-cost difference reported in Appendix C.2.

Circularity Check

0 steps flagged

No significant circularity: Kronecker factorization, algebraic HG cancellation, BiIP, and tr(HG)·tr(HX) allocation are constructive method steps evaluated on held-out perplexity/zero-shot metrics.

full rationale

The paper’s derivation chain is self-contained and non-circular. The K-FAC step H≈HX⊗HG (Eq. 7) is a standard modeling assumption imported from Martens & Grosse (2015), not defined in terms of the target perplexity. Proposition 1 shows HG cancels algebraically in the column-wise OBS update, reducing the solver exactly to GPTAQ form—an identity, not a fitted identity. Bidirectional incoherence (Eqs. 11–12) and the sensitivity score sℓ=tr(HG)·tr(HX) (Eq. 14) use a once-estimated HG from a single backward pass over calibration data as preprocessing/allocation inputs; they do not “predict” quantities that were fitted into HG. Empirical claims (e.g., LLaMA-3-70B W2 WikiText-2 PPL 7.93 vs GPTQ/GPTAQ >2000) are measured on held-out WikiText-2 and zero-shot suites never used to fit free parameters of a theory. Self-citation of GPTAQ (Li et al., 2025) is ordinary base-solver reuse by overlapping authors and is not load-bearing for uniqueness or for forbidding alternatives. No step reduces a claimed prediction to its inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper rests on standard second-order PTQ machinery (OBS, K-FAC, empirical Fisher) plus a small set of engineering knobs (damping, α, calibration size). No new physical entities are postulated; the only modeling assumptions that are load-bearing are the K-FAC independence and the representativeness of a single backward-pass HG.

free parameters (3)
  • calibration set size / context = 128 × 2048
    Fixed at 128 WikiText-2 sequences of length 2048; standard but still a free experimental choice that affects HG quality.
  • damping λ and GPTAQ scaling α
    Inherited from GPTAQ; values affect numerical stability of the Cholesky and the asymmetric correction term.
  • block size B = 128
    Column-block size for the OBS loop (commonly 128); engineering parameter.
axioms (4)
  • domain assumption K-FAC independence x ⊥⊥ g so that E[xx⊤ ⊗ gg⊤] ≈ HX ⊗ HG
    Invoked in Eq. 7 and throughout Section 4; standard in the K-FAC literature but known to be approximate.
  • domain assumption Empirical Fisher / second-order Taylor expansion of the layer-wise reconstruction loss
    Background for all OBS-style PTQ methods; used without re-derivation.
  • standard math Column-wise OBS update remains optimal after the asymmetric GPTAQ correction
    Proposition 1 shows HG cancels; the remaining update is exactly GPTAQ’s.
  • domain assumption Randomized Hadamard transforms render both HX and HG incoherent with high probability
    Taken from QuIP#; used for BiIP.

pith-pipeline@v1.1.0-grok45 · 32807 in / 2513 out tokens · 25861 ms · 2026-07-10T14:37:15.814390+00:00 · methodology

0 comments
read the original abstract

Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Existing second-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), KronQ achieves 7.93 perplexity.

Figures

Figures reproduced from arXiv: 2607.07964 by Donghyun Lee, Priyadarshini Panda, Ruokai Yin, Yuhang Li.

Figure 1
Figure 1. Figure 1: (a) Normalized diagonal entries of the gradient covariance [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) µ-incoherence of HG before and after incoherence preprocessing across sublay￾ers of LLaMA-2-7B. (b) Weight magnitude distribution of Q proj under three configurations: original weights, after input-side incoherence (HX only), and after bidirectional incoherence (HX + HG), where CVin and CVout denote the coefficient of variation of column and row norms, respectively. The derivation of Proposition 1 is p… view at source ↗
Figure 3
Figure 3. Figure 3: Sublayer sensitivity rankings under the KronQ score [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: WikiText-2 perplexity vs. aver￾age bit-width on LLaMA-2-7B, comparing KRONQ with prior mixed precision works. 5.3 Results on Mixed-precision We evaluate the mixed-precision allocation strategy by incrementally upgrading the most sensitive sublayers from W2 to W3 according to the sensitivity score s = tr(HG) · tr(HX) in Equation (14), applying each upgrade across all transformer layers [PITH_FULL_IMAGE:fig… view at source ↗
Figure 5
Figure 5. Figure 5: Inference efficiency of KronQ quantized models. (a) Peak VRAM and (b) decoding [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Weight magnitude distribution of the Q projection in the first layer under three [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 49 canonical work pages · 23 internal anchors

  1. [1]

    Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Quantization error propagation: Revisiting layer-wise post-training quantization.arXiv preprint arXiv:2504.09629,

    Yamato Arai and Yuma Ichikawa. Quantization error propagation: Revisiting layer-wise post-training quantization.arXiv preprint arXiv:2504.09629,

    work page arXiv

  3. [3]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    work page 1901

  4. [4]

    Channel-Wise Mixed-Precision Quantization for Large Language Models

    Zihan Chen, Bike Xie, Jundong Li, and Cong Shen. Channel-wise mixed-precision quantiza- tion for large language models.arXiv preprint arXiv:2410.13056,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers), ...

    work page 2019

  6. [6]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    The language model evaluation harness, 07

    11 Published as a conference paper at COLM 2026 Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite,...

    work page 2026

  9. [9]

    Gemma Team

    URL https://zenodo.org/ records/12608602. Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    work page arXiv

  10. [10]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  13. [13]

    OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting

    Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting.arXiv preprint arXiv:2501.13987,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

    Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojuan Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models.arXiv preprint arXiv:2405.14917,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Representations, volume 2025, pp. 58791–58831,

    work page 2025

  16. [16]

    Mistral 7B

    AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de Las Casas, F Bres- sand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv 2023.arXiv preprint arXiv:2310.06825,

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Guidedquant: Large language model quantization via exploiting end loss guidance.arXiv preprint arXiv:2505.07004,

    Jinuk Kim, Marwa El Halabi, Wonpyo Park, Clemens JS Schaefer, Deokjae Lee, Yeonhong Park, Jae W Lee, and Hyun Oh Song. Guidedquant: Large language model quantization via exploiting end loss guidance.arXiv preprint arXiv:2505.07004,

    work page arXiv

  18. [18]

    BoA: Attention-aware Post-training Quantization without Backpropagation

    Junhan Kim, Ho-young Kim, Eulrang Cho, Chungman Lee, Joonyoung Kim, and Yongk- weon Jeon. Boa: Attention-aware post-training quantization without backpropagation. arXiv preprint arXiv:2406.13474,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Optimal brain damage.Advances in neural information processing systems, 2,

    12 Published as a conference paper at COLM 2026 Yann LeCun, John Denker, and Sara Solla. Optimal brain damage.Advances in neural information processing systems, 2,

    work page 2026

  20. [20]

    Amq: Enabling automl for mixed-precision weight-only quantization of large language models

    Sangjun Lee, Seung-taek Woo, Jun-gyu Jin, Changhun Lee, and Eunhyeok Park. Amq: Enabling automl for mixed-precision weight-only quantization of large language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 35520–35538,

    work page 2025

  21. [21]

    GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration

    Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda. Gp- taq: Efficient finetuning-free quantization for asymmetric calibration.arXiv preprint arXiv:2504.02692,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models

    Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? an empirical study on quantized reasoning models.arXiv preprint arXiv:2504.04823,

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    SpinQuant: LLM quantization with learned rotations

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations.arXiv preprint arXiv:2405.16406,

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    Higgs: Pushing the limits of large language model quantization via the linearity theorem

    Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, Han Guo, Peter Richt ´arik, and Dan Alistarh. Higgs: Pushing the limits of large language model quantization via the linearity theorem. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long P...

    work page 2025

  25. [25]

    AIME 2024 dataset

    Maxwell-Jia. AIME 2024 dataset. https://huggingface.co/datasets/Maxwell-Jia/AIME 2024,

    work page 2024

  26. [26]

    Pointer Sentinel Mixture Models

    Hugging Face dataset. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pp. 2381–2391,

    work page 2018

  28. [28]

    The Uniqueness of LLaMA3-70B Series with Per-Channel Quantization

    Contact: qubitium@modelcloud.ai. Minghai Qin. The uniqueness of llama3-70b series with per-channel quantization.arXiv preprint arXiv:2408.15301,

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark.arXiv preprint arXiv:2311.12022,

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106,

    13 Published as a conference paper at COLM 2026 Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106,

    work page 2026

  31. [31]

    OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models.arXiv preprint arXiv:2308.13137,

    work page internal anchor Pith review Pith/arXiv arXiv

  32. [32]

    FlatQuant: Flatness Matters for LLM Quantization

    Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for llm quantization.arXiv preprint arXiv:2410.09426,

    work page internal anchor Pith review Pith/arXiv arXiv

  33. [33]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    work page internal anchor Pith review Pith/arXiv arXiv

  34. [34]

    Model-Preserving Adaptive Rounding

    Albert Tseng, Zhaofeng Sun, and Christopher De Sa. Model-preserving adaptive rounding. arXiv preprint arXiv:2505.22988,

    work page internal anchor Pith review Pith/arXiv arXiv

  35. [35]

    The LLM Surgeon

    Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, Yuki M Asano, and Tijmen Blankevoort. The llm surgeon.arXiv preprint arXiv:2312.17244,

    work page internal anchor Pith review Pith/arXiv arXiv

  36. [36]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar...

    work page 2020

  37. [37]

    URL https://www.aclweb.org/ anthology/2020.emnlp-demos.6

    Association for Computational Linguistics. URL https://www.aclweb.org/ anthology/2020.emnlp-demos.6. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. InInternational conference on machine learning, pp. 38087–38099. PMLR,

    work page 2020

  38. [38]

    Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis

    Jiaqi Zhao, Ming Wang, Miao Zhang, Yuzhang Shang, Xuebo Liu, Yaowei Wang, Min Zhang, and Liqiang Nie. Benchmarking post-training quantization in llms: Comprehensive taxonomy, unified evaluation, and comparative analysis.arXiv preprint arXiv:2502.13178,

    work page internal anchor Pith review Pith/arXiv arXiv

  39. [39]

    14 Published as a conference paper at COLM 2026 A Proofs A.1 Proof of Proposition 1 (KronQ Weight Compensation) The general OBS optimal weight update (Hassibi & Stork,

    work page 2026

  40. [40]

    Note that ∆W′ =W ′ −cW′ =U∆WV ⊤

    yields (10).□ A.2 Proof of Theorem 1 (Rotation Invariance) Denote the transformed quantities after (12) as W′ =UWV ⊤, eHG =UH GU⊤, eHX = VHXV⊤, and (∆XX ⊤)′ =V∆XX ⊤V⊤. Note that ∆W′ =W ′ −cW′ =U∆WV ⊤. We verify invariance of each term separately. For the first term: tr h eHG ∆W′eHX (∆W′)⊤ i =tr h UHGU⊤ ·U∆WV ⊤ ·VH XV⊤ ·V∆W ⊤U⊤ i =tr h UHG∆WHX∆W⊤U⊤ i =tr h...

    work page 2026

  41. [41]

    KronQ+BiIP improves this bound by the factor µ 2 GkG/dout, which can be substantially less than one when HG is approximately low-rank

    bound µ 2 XkX tr(HX)/(4din). KronQ+BiIP improves this bound by the factor µ 2 GkG/dout, which can be substantially less than one when HG is approximately low-rank. Table 9: Worst-case proxy loss under LDLQ rounding. Method Worst-case proxy loss Nearest rounding 1 4 tr(HX)·tr(H G) GPTQ-LDLQ (HG =I) 1 4 tr(DX)·d out KronQ-LDLQ (ours) 1 4 tr(DX)·tr(D G) 16 P...

    work page 2026

  42. [42]

    9:return W,H X,∆XX ⊤,S X,S G,U,V Algorithm 2KronQ – Quantization Loop Require: W∈R m×n, HX ∈R n×n, ∆XX⊤ ∈R n×n, SX, SG, U, V (from Alg. 1), block size B, damp- ingλ, scalingα 1:H X ←H X +λ·mean(diag(H X))I▷damping 2:L←Inverse Cholesky(HX) 3:P←α (∆XX ⊤ ·L)⊙M U L⊤ ▷GPTAQ correction;M U: upper-triangular mask 4:Q←0 m×n;E←0 m×B 5:fori=0,B, 2B, . . .do 6:forj=...

    work page 2026

  43. [43]

    As shown in Table 12, KronQ matches or exceeds its LNQ-based results with a plain uniform per-channel grid

    weights the input-side Hessian by per-output-channel saliency, missing the cross-output-channel structure thatHG captures. As shown in Table 12, KronQ matches or exceeds its LNQ-based results with a plain uniform per-channel grid. Table 11: YAQA setup (LLaMA- 3.1-8B-Instruct, WikiText-2, ctx 8192, INT4 per-channel). Method WikiText-2 PPL BF16 6.50 LDLQ 6....

    work page 2026

  44. [44]

    Table 14: Weight-only group quantization (WxA16, g=128) WikiText-2 perplexity

    results are taken from their papers. Table 14: Weight-only group quantization (WxA16, g=128) WikiText-2 perplexity. LLaMA-2-7B LLaMA-2-13B LLaMA-2-70B Method W4 W3 W2 W4 W3 W2 W4 W3 W2 RTN 5.72 6.66 4.2e3 4.98 5.51 122.08 3.46 3.97 27.27 GPTQ 5.57 6.02 274.00 4.95 5.22 8.40 3.39 3.64 5.22 AWQ 5.60 6.24 NaN 4.97 5.32 NaN 3.41 3.74 – OmniQuant 5.58 6.03 11....

    work page 2026

  45. [45]

    All models are calibrated on 128 WikiText-2 sequences with no fine-tuning

    and Gemma-3-12B-IT (Gemma Team, 2025), each quantized to W4 (per-channel, asymmetric, weight-only) with KronQ and, as baselines, with GPTQ and GPTAQ under identical settings. All models are calibrated on 128 WikiText-2 sequences with no fine-tuning. MMLU, GPQA-Diamond, and AIME-2024 are evaluated through the lm-evaluation-harness (Gao et al.,

    work page 2025

  46. [46]

    in the zero-shot setting, while LiveCodeBench uses its official runner with vLLM. We report log-likelihood accuracy on MMLU (57 subjects) and GPQA-Diamond (198 questions), exact-match accuracy on AIME-2024 (30 problems), and pass@1 on LiveCodeBench ( release v5, code-generation scenario, 32,768 maximum new tokens). For the reasoning model DeepSeek-R1-Dist...

    work page 2024

  47. [47]

    Table 18: Sublayer sensitivity rankings and WikiText-2 perplexity on LLaMA-3-8B and LLaMA-2-13B

    and CMPQ (Chen et al., 2024), consistent with the LLaMA-2-7B results in the main paper. Table 18: Sublayer sensitivity rankings and WikiText-2 perplexity on LLaMA-3-8B and LLaMA-2-13B. Each row cumulatively up- grades one additional sublayer to W3 across all layers. Score Ranking Avg bits Wiki2↓ LLaMA-3-8B baseline W2 - 2.00 11.92 tr(HG)·tr(HX) 1:down pro...

    work page 2024

  48. [48]

    For the allocation comparisons, we hold the base quantizer fixed and vary only the allocation

    that KronQ uses for HG, so the comparison isolates the allocation methodology from calibration-data access. For the allocation comparisons, we hold the base quantizer fixed and vary only the allocation. Table 19 holds GPTQ fixed (LLaMA-2-7B, context 2048), where KronQ’s allocation beats AMQ’s data-aware NSGA-II search at a matched 3.1-bit budget. Table 20...

    work page 2048

  49. [49]

    𝑪𝑽𝒊𝒏=𝟎.𝟑𝟕𝑪𝑽𝒐𝒖𝒕=𝟎.𝟔𝟒 𝑪𝑽𝒊𝒏=𝟎.𝟐𝟓𝑪𝑽𝒐𝒖𝒕=𝟎.𝟓𝟏 𝑪𝑽𝒊𝒏=𝟎.𝟐𝟕𝑪𝑽𝒐𝒖𝒕=𝟎.𝟐𝟓 (a) Weight distributions of LLaMA-2-70B Original𝐻!only 𝐻!+𝐻

    and compares against Q-Palette’s data-aware actual-loss allocation. KronQ’s allocation achieves lower perplexity at every budget. Table 21 instead isolates the quantizer: against the data-aware GPTQ+HIGGS configuration (LLaMA- 2-7B, context 4096), KronQ’s scalar grid leads at every bit-width despite HIGGS using a vector quantizer. Crucially, KronQ’s alloc...

    work page 2026