Pith. sign in

REVIEW 3 major objections 7 minor 4 cited by

Augmented residual channels let 4-bit LLM inference reach 8-bit accuracy while keeping a strictly unified NVFP4 hardware path.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:04 UTC pith:ELFPNCT2

load-bearing objection ARCQuant has a genuinely neat mechanism—folding residual compensation into the K dimension of a single NVFP4 GEMM—but its headline error-bound claim is only proven for selected outlier channels, and the threshold that picks them is asserted, not analyzed. the 3 major comments →

arxiv 2601.07475 v2 pith:ELFPNCT2 submitted 2026-01-12 cs.LG cs.AI

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

Haoqian Meng , Yilun Luo , Yafei Zhao , Wenyuan Liu , Peng Zhang , Xindian Ma This is my paper
classification cs.LG cs.AI
keywords post-training quantizationNVFP4microscaling formatsLLM inferenceactivation outliersresidual error compensationunified-precision GEMMBlackwell GPU
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ARCQuant claims that the accuracy gap between 4-bit and 8-bit activation quantization in LLMs can be closed while keeping a strictly 4-bit hardware path. The trick is to append extra columns to the activation matrix containing quantized residuals of the most extreme channels, and duplicate the corresponding weight columns, so that a single standard GEMM computes both the quantized product and the residual correction. The paper argues this yields worst-case error comparable to MXFP8, because two stages of NVFP4 (E2M1) quantization give ε4^2 = ε8. Experiments on LLaMA and Qwen models show perplexity and downstream-task scores close to full precision, with only a few percent latency overhead and 2–3.5x speedups over FP16 on Blackwell GPUs. If right, this means W4A4 inference can approach W4A8 quality without mixed-precision kernels.

Core claim

The paper's central claim is that outlier-dominated activation quantization error can be repaired within the NVFP4 format rather than by moving to higher precision or rotating the tensor. ARCQuant identifies outlier channels offline by a threshold τ=2^{-3}M, quantizes the residual error of those channels in NVFP4 again, and concatenates the residuals onto the activation matrix along the reduction dimension, with matching duplicated weight columns. The key identity, Y ≈ sXaug·QXaug (sWaug·QWaug)^T, turns the two-stage computation into one unified-precision GEMM. Because the residual is itself quantized, the worst-case error for compensated channels is (α1α2) M ε8 with α1α2 ≈ 1.266 vs MXFP8's

What carries the argument

The Augmented Residual Channel mechanism: after reordering channels by magnitude, a fixed threshold τ=2^{-3}M (reflecting the 3-bit exponent gap between the E5M2 reference and the E2M1 element) selects the top-S outlier channels. Their first-stage NVFP4 residual Ro = Xo - sXo·QXo is quantized again, and QXaug = [QX | QRo] is formed with combined scales. The identity that carries the argument is Eq. (2), which distributes the correction term Q(Ro)Q(Wo)^⊤ into the extended reduction dimension, so a standard unified-precision GEMM performs both stages. The supporting identity is ε4^2 = ε8, which lets dual-stage NVFP4 match single-stage MXFP8 resolution.

Load-bearing premise

The claim that worst-case NVFP4 error matches MXFP8 depends on the fixed threshold τ=2^{-3}M marking the true set of outlier channels, and on those channels keeping their magnitudes during inference; if the threshold misclassifies or the outliers shift, the dual-stage bound does not cover the channels that dominate the error.

What would settle it

A concrete falsification test: after quantization, measure the actual maximum absolute error per layer on a held-out set with an adversarial perturbation that increases an initially small channel above τ. If the layer's worst-case error exceeds the MXFP8 bound, the central theoretical claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • W4A4 inference on Blackwell hardware can reach accuracy comparable to W4A8 baselines and near FP16, without any mixed-precision or rotated kernels.
  • The residual-compensation principle transfers to INT4 and MXFP4, where the paper reports consistent gains over round-to-nearest baselines.
  • Total latency overhead stays in the 3–9% range relative to uncompensated NVFP4, and end-to-end prefill speedups over FP16 reach 2–3.5x with 1.5–2.8x memory reduction.
  • Because compensation lives entirely in the input data space, any standard GEMM kernel works; porting to a new microscaling format only requires updating the quantization kernel.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'comparable to MXFP8' error bound is proven for the compensated top-S channels only; unselected channels keep single-stage NVFP4 error, which is 4x MXFP8's bound, so the global worst-case guarantee is weaker than it first reads.
  • The fixed threshold τ=2^{-3}M implicitly assumes outlier magnitudes and positions are stable across inputs; if they drift in long-tailed or adversarial distributions, the offline-chosen S could miss the channels that need compensation.
  • One natural extension the paper leaves implicit is hierarchical residuals: quantizing the residual's residual could extend the same trick toward sub-4-bit formats or further tighten the bound.
  • Since the method duplicates weight columns rather than computing true residuals on the weight side, it effectively doubles the GEMM reduction dimension for the outlier channels; the speedup numbers suggest this is cheaper than a second kernel, but the memory trade-off for very large S is not fully explored.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes ARCQuant, a post-training quantization method for NVFP4 that keeps a strictly unified NVFP4 data path by augmenting the activation matrix with quantized residual channels. Outlier channels are identified from calibration data, their residuals are quantized to NVFP4, and the primary and residual tensors are concatenated along the reduction dimension so that a single standard GEMM computes both the main product and the error compensation term (Eq. 2). The paper claims a theoretical worst-case error bound comparable to MXFP8 for this dual-stage scheme, reports accuracy improvements over existing W4A4 methods on LLaMA and Qwen models, and demonstrates up to 3.5x prefill speedup over FP16 on RTX 5090 and RTX PRO 6000 GPUs. Code is provided.

Significance. If the theoretical guarantee and the empirical gains hold, ARCQuant would be a practically useful contribution: it shows a way to mitigate activation outliers in fine-grained 4-bit formats without resorting to mixed precision or rotation transforms, and it maps cleanly onto existing tensor-core GEMM kernels. The paper includes a machine-checkable algebraic identity (Eq. 2), an explicit error-bound calculation (Eq. 4), real-hardware benchmarks, and an anonymous code release, which are strengths. The empirical results, especially the NVFP4-only comparison in Table 2, suggest that residual-channel compensation can improve accuracy while preserving hardware uniformity. However, the central theoretical claim as stated in the abstract and conclusion is broader than what the analysis actually proves, and the threshold on which the method relies is not ablated or derived.

major comments (3)
  1. [Section 3.4 / Abstract / Conclusion] The statement that the dual-stage NVFP4 scheme has a worst-case error bound 'comparable to that of standard 8-bit formats such as MXFP8' is made globally, but Eq. (4) bounds only the compensated outlier channels. For all non-selected channels the error remains the single-stage NVFP4 bound |e| ≤ α M ε4 = α M·2^{-2}, which is 4× the MXFP8 bound with ε8=2^{-4} (even with the smaller alignment factor α ≤ 1.125 vs. α_mx < 2, the ratio is about 2.25). The threshold τ=2^{-3}M in Section 3.2 does not remove this gap: a channel whose block maximum is below τ has absolute error at most τ ε4 = 2^{-5}M, whereas MXFP8 on the same block has error at most τ ε8 = 2^{-7}M, still a factor of 4. Additionally, the threshold rationale invokes 'the reference E5M2 (5 bits)' while the theoretical comparison uses MXFP8 E4M3; these are different baselines. Please scope the claim to compensated channels or provide
  2. [Section 3.2 / Section 4.4] The selection threshold τ=2^{-3}M and the resulting layer-wise outlier count S are load-bearing: they determine which channels receive second-stage compensation, and the theoretical claim in Section 3.4 only applies to those channels. Yet the paper does not provide a derivation of τ, nor does it report an ablation varying τ or S. Section 4.4 tests calibration robustness by changing the calibration dataset, but not by changing the threshold. Given that both accuracy and the theoretical guarantee depend on this threshold, please add a sensitivity study (e.g., sweep τ across a range and report accuracy/error) and either justify τ analytically or present empirical evidence that the method is robust to its choice.
  3. [Table 1 / Section 4.1] The headline comparison in Table 1 mixes quantization formats. Atom and FlatQuant are run in their original configurations, which the text states are 'structurally incompatible with NVFP4', and the 'W4A8 + RTN' baseline uses MXFP4 weights with MXFP8 activations. Thus the claim of being the 'best among W4A4 methods' is not a controlled comparison within a single format. The NVFP4-only comparison in Table 2 is more appropriate. Please either re-implement the baselines in NVFP4 (where possible) or qualify the Table 1 conclusions as cross-format rather than a same-format state-of-the-art result.
minor comments (7)
  1. [Tables 1-2] The column header 'Lamba' should be 'Lambada'.
  2. [Abstract / Section 4.3] The abstract says 'up to 3× speedup over FP16', but Section 4.3 reports 3.5× for Llama 3.1-8B on RTX 5090. Please align the numbers.
  3. [Section 3.4] Eq. (4) uses sup α1α2 = 1.125² ≈ 1.266. It would be helpful to state explicitly that α1 and α2 are independent worst-case scale alignment factors and that the product assumes both scales can be at their maximum simultaneously.
  4. [Section 3.2 / Eq. (2)] The notation sXaug is used both as a vector of block scales and as a single scalar in Eq. (2). Please define the elementwise/scalar aspect clearly to avoid confusion.
  5. [Figure 8] The caption uses 'S≤512' while the text says 'S ≤ 512'; make the notation consistent.
  6. [Introduction] The claim 'first PTQ framework optimized for NVFP4' is strong; given prior work such as FGMP that also targets NVFP4, consider softening to 'among the first' or clarifying the specific sense in which this is the first.
  7. [Appendix D] The interleaved layout groups channels into blocks of 16; when S is not a multiple of 16, the implementation details (padding or partial blocks) are not specified. Please clarify.

Circularity Check

0 steps flagged

No significant circularity: the GEMM fusion is an exact algebraic identity, accuracy is measured against external baselines, and the error-bound analysis is a format-level calculation rather than a fitted prediction.

full rationale

ARCQuant's central derivation chain is self-contained rather than circular. Equation (2) states an exact algebraic identity: after augmenting the activation and weight matrices with residual and duplicate channels, one GEMM computes the primary quantized product plus the residual correction. This is a construction, not a prediction, and it does not assume the conclusion. The accuracy claims are supported by empirical benchmarks on standard tasks (perplexity, MMLU, HumanEval, etc.) against FP16, W4A8, and other W4A4 methods, so they do not reduce to fitted inputs. The theoretical error-bound comparison in Section 3.4 is a direct calculation from the format parameters (epsilon_4^2 = epsilon_8 and the scale-alignment factors alpha_1 alpha_2 < alpha_mx), not a fitted parameter renamed as a prediction. The only self-citation (MicroMix) appears in related work and is not load-bearing. The threshold tau = 2^{-3}M is a heuristic selection rule, not a quantity fitted to the very data being predicted, and the acknowledged limitation about static outlier assumptions is an unproven assumption rather than a circular reduction. The abstract's global phrasing of the MXFP8-comparable bound is broader than the Section 3.4 claim, which explicitly restricts the bound to compensated outlier channels; that is a correctness or overclaim concern, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

Central claim rests on threshold choice, calibration data, and hardware-format assumptions plus a standard GEMM linearity identity. No new physical entities are postulated.

free parameters (3)
  • Outlier selection threshold coefficient = 2^{-3}
    τ=2^{-3}M; chosen from the exponent-width difference between E5M2 and E2M1; no sensitivity analysis reported.
  • Layer-wise outlier count S = per-layer, data-dependent
    Number of augmented residual channels per layer; derived from calibration data and threshold; governs accuracy/compute tradeoff.
  • Calibration configuration = 128×2048 WikiText2, seed 0
    Used to determine reordering and S; robustness tested across C4/HumanEval/WikiText2 but not for threshold or seed.
axioms (6)
  • standard math Quantization error bound |e| ≤ sε for E*M* formats, with ε=2^{-p}
    Used in Section 3.4 to derive Bmx and Barc; standard unit-roundoff argument, though the paper's definition of M/s is informal.
  • domain assumption E4M3 scale alignment overhead sup α=1.125 for NVFP4; hence sup α1α2≈1.266
    Section 3.4; assumes block scales are rounded to E4M3 with ≤1/8 relative overhead; not proven in text.
  • domain assumption NVFP4 hardware format: block size 16, E2M1 elements, E4M3 block scale, per-tensor FP32 scale
    Appendix A; hardware spec that the method targets; if wrong, the method's hardware claims fail.
  • domain assumption Activation outlier channels are stable between calibration and inference
    Section 3.2 and Limitations; the static reordering and S assume distribution does not drift.
  • standard math GEMM linearity: splitting K dimension and duplicating QWo computes the residual correction exactly
    Eq. (2); relies on exact accumulation in the MMA.
  • ad hoc to paper τ=2^{-3}M marks the range where NVFP4 precision is comparable to MXFP8, so uncompensated channels are safe
    Section 3.2; asserted without derivation; the actual relative precision of NVFP4 at |x|≤M/8 is still ε4=2^-2, not ε8.

pith-pipeline@v1.3.0-alltime-deepseek · 13908 in / 18819 out tokens · 183243 ms · 2026-08-03T11:04:45.651020+00:00 · methodology

0 comments
read the original abstract

The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference. However, it is difficult to adapt existing Post-Training Quantization (PTQ) strategies to these formats: rotation-based methods compromise fine-grained block isolation; smoothing techniques struggle with significant 4-bit quantization errors; and mixed-precision approaches often conflict with hardware constraints on unified-precision computation. To address these challenges, we propose ARCQuant, a framework that boosts NVFP4 performance via Augmented Residual Channels. Distinct from methods that compromise block isolation or hardware uniformity, ARCQuant maintains a strictly unified NVFP4 format by augmenting the activation matrix with quantized residual channels. This design integrates the error compensation process directly into the matrix reduction dimension, enabling the use of standard, highly optimized GEMM kernels with minimal overhead. Theoretical analysis confirms that the worst-case error bound of our dual-stage NVFP4 quantization is comparable to that of standard 8-bit formats such as MXFP8. Extensive experiments on LLaMA and Qwen models demonstrate that ARCQuant achieves state-of-the-art accuracy, comparable to full-precision baselines in perplexity and downstream tasks. Furthermore, deployment on RTX 5090 and RTX PRO 6000 GPUs confirms practical benefits, achieving up to 3x speedup over FP16. Our code is available at https://github.com/actypedef/ARCQuant.

Figures

Figures reproduced from arXiv: 2601.07475 by Haoqian Meng, Peng Zhang, Wenyuan Liu, Xindian Ma, Yafei Zhao, Yilun Luo.

Figure 1
Figure 1. Figure 1: ARCQuant closes the NVFP4 accuracy gap while preserving high throughput on Blackwell platforms. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Activation magnitudes (blue) and quantization errors (red) on Llama 3.1-8B [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: o_proj in Llama 3.1-8B, ARCQuant consis￾tently suppressed MSE on NVFP4 across all layers. scaling factor sX are calculated as: QX = round  X sX  , sX = max(|X|) qmax . (1) The dequantized approximation is given by Q(X) = sX · QX. Our objective is to minimize the reconstruction error ∥Y − Q(X)Q(W) ⊤∥F . Block-scaled Quantization. NVIDIA Blackwell architecture introduces hardware acceleration for block-sca… view at source ↗
Figure 4
Figure 4. Figure 4: ARCQuant fuses main and residual computations by mapping them into the extended dimension. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: ARCQuant on a Transformers block in LLM. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prefill efficiency for various models with 2048 sequence length on RTX 5090 and PRO 6000. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Outlier channels S across Qwen2.5-7B layers. 2.0×–2.5× speedup (PRO 6000) and Llama 3.1- 8B reaches 3.5× (RTX 5090), while memory usage drops by 1.5×–2.8×. Crucially, residual overhead is marginal: compared to uncompensated NVFP4, latency increases by only 3%–9%. 4.4 Ablation Studies Generalizability. Although tailored for the fine￾grained NVFP4 format, ARCQuant demonstrates robust extensibility to INT4 an… view at source ↗
Figure 9
Figure 9. Figure 9: Qwen2.5-Math-7B-Instruct on GSM8K and CMATH of 2048, as the calibration set. Based on the de￾rived reordering indices and the layer-specific out￾lier counts S, we perform weight quantization. Ta￾ble 4 reports the calibration latency, the resulting quantized model size, and the total quantization time. B.2 Results Breakdown Calibration Robustness. To assess the robust￾ness of ARCQuant to calibration data va… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

    cs.LG 2026-07 conditional novelty 7.0

    In end-to-end 4-bit RL post-training, rollout activation underflow, not training quantization, is the main accuracy killer; a sparse residual correction closes most of the gap to BF16.

  2. Quantizing Recursive Reasoning Models

    cs.LG 2026-06 conditional novelty 6.0

    Per-tensor 4-bit quantization of weight-tied recursive reasoners induces accumulating activation bias that kills accuracy; per-block scaling (MXInt4) restores it.

  3. SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

    cs.LG 2026-06 unverdicted novelty 6.0

    SharQ combines input-adaptive N:M sparsity and FP4 quantization via sparse backbone plus dense residual, recovering 43-63% of the NVFP4-to-FP16 accuracy gap on Llama and Qwen models without calibration or retraining.

  4. SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization

    cs.LG 2026-05 unverdicted novelty 6.0

    SOAR improves NVFP4 post-training quantization accuracy for LLMs by analytically solving joint scale optimization and searching decoupled scales.

Reference graph

Works this paper leans on

37 extracted references · 22 linked inside Pith · cited by 4 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. 2023. Quik: Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.09259

    Pith/arXiv arXiv 2023

  4. [4]

    Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. https://openreview.net/forum?id=dfqsW38v1X Quarot: Outlier-free 4-bit inference in rotated LLM s . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  5. [5]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. https://arxiv.org/abs/2108.07732 Program synthesis with large language models . Preprint, arXiv:2108.07732

    Pith/arXiv arXiv 2021

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. https://arxiv.org/abs/2107.03374 Evaluating large lang...

    Pith/arXiv arXiv 2021

  7. [7]

    Brian Chmiel, Maxim Fishman, Ron Banner, and Daniel Soudry. 2025. Fp4 all the way: Fully quantized training of llms. arXiv preprint arXiv:2505.19115

    Pith/arXiv arXiv 2025

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    Pith/arXiv arXiv 2018

  9. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    Pith/arXiv arXiv 2021

  10. [10]

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, and 14 others. 2023. https://arxiv.org/abs/2310.10537 M...

    Pith/arXiv arXiv 2023

  11. [11]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. https://arxiv.org/abs/2208.07339 Llm.int8(): 8-bit matrix multiplication for transformers at scale . Preprint, arXiv:2208.07339

    Pith/arXiv arXiv 2022

  12. [12]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. https://arxiv.org/abs/2210.17323 Gptq: Accurate post-training quantization for generative pre-trained transformers . Preprint, arXiv:2210.17323

    Pith/arXiv arXiv 2023

  13. [13]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. https://doi.org/10.5281/zenodo.12608602 The languag...

    doi:10.5281/zenodo.12608602 2024

  14. [14]

    Akhil Mathur

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and et al. Akhil Mathur. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    Pith/arXiv arXiv 2024

  15. [15]

    Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, and Hao Yu. 2024. Aptq: Attention-aware post-training mixed-precision quantization for large language models. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pages 1--6

    2024

  16. [16]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  17. [17]

    Coleman Hooper, Charbel Sakr, Ben Keller, Rangharajan Venkatesan, Kurt Keutzer, Sophia Shao, and Brucek Khailany. 2025. https://arxiv.org/abs/2504.14152 Fgmp: Fine-grained mixed-precision weight and activation quantization for hardware-accelerated llm inference . Preprint, arXiv:2504.14152

    Pith/arXiv arXiv 2025

  18. [18]

    Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, and Song Han. 2024. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05007

    arXiv 2024

  19. [19]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for llm compression and acceleration. In MLSys

    2024

  20. [20]

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2025. https://arxiv.org/abs/2405.04532 Qserve: W4a8kv4 quantization and system co-design for efficient llm serving . Preprint, arXiv:2405.04532

    Pith/arXiv arXiv 2025

  21. [21]

    Wenyuan Liu, Haoqian Meng, Yilun Luo, Peng Zhang, and Xindian Ma. 2025. Micromix: Efficient mixed-precision quantization with microscaling formats for large language models. arXiv preprint arXiv:2508.02343

    arXiv 2025

  22. [22]

    Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. https://arxiv.org/abs/2103.13009 Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark . Preprint, arXiv:2103.13009

    Pith/arXiv arXiv 2021

  23. [23]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. https://arxiv.org/abs/1609.07843 Pointer sentinel mixture models . CoRR, abs/1609.07843

    Pith/arXiv arXiv 2016

  24. [24]

    Nvidia. 2024. https://resources.nvidia.com/en-us-blackwell-architecture Nvidia blackwell architecture technical brief

    2024

  25. [25]

    NVIDIA Corporation . 2024. cuDNN Frontend API v1.14.0: Block-Scaling Operation . https://docs.nvidia.com/deeplearning/cudnn/frontend/v1.14.0/operations/BlockScaling.html. Accessed: 2025-09-16

    2024

  26. [26]

    Denis Paperno, Germ\' a n Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. http://www.aclweb.org/anthology/P16-1144 The LAMBADA dataset: Word prediction requiring a broad discourse context . In Proceedings of the 54th Annual Meeting of the Association for Computati...

    2016

  27. [27]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    Pith/arXiv arXiv 2025

  28. [28]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . CoRR, abs/1910.10683

    Pith/arXiv arXiv 2019

  29. [29]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. https://arxiv.org/abs/1907.10641 Winogrande: An adversarial winograd schema challenge at scale . Preprint, arXiv:1907.10641

    Pith/arXiv arXiv 2019

  30. [30]

    Utkarsh Saxena, Sayeh Sharify, Kaushik Roy, and Xin Wang. 2025. https://arxiv.org/abs/2412.14363 Resq: Mixed-precision quantization of large language models with low-rank residuals . Preprint, arXiv:2412.14363

    Pith/arXiv arXiv 2025

  31. [31]

    Yuantian Shao, Peisong Wang, Yuanteng Chen, Chang Xu, Zhihui Wei, and Jian Cheng. 2025. Block rotation is all you need for mxfp4 quantization. arXiv preprint arXiv:2511.04214

    arXiv 2025

  32. [32]

    Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, and 1 others. 2024. Flatquant: Flatness matters for llm quantization. arXiv preprint arXiv:2410.09426

    Pith/arXiv arXiv 2024

  33. [33]

    Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. https://openreview.net/forum?id=9BrydUVcoe Qu IP \ \#\ : Even better LLM quantization with hadamard incoherence and lattice codebooks . In Forty-first International Conference on Machine Learning

    2024

  34. [34]

    Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. 2023. https://arxiv.org/abs/2306.16636 Cmath: Can your language model pass chinese elementary school math test? Preprint, arXiv:2306.16636

    Pith/arXiv arXiv 2023

  35. [35]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2024. https://arxiv.org/abs/2211.10438 Smoothquant: Accurate and efficient post-training quantization for large language models . Preprint, arXiv:2211.10438

    Pith/arXiv arXiv 2024

  36. [36]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    Pith/arXiv arXiv 2019

  37. [37]

    Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. https://proceedings.mlsys.org/paper_files/paper/2024/file/5edb57c05c81d04beb716ef1d542fe9e-Paper-Conference.pdf Atom: Low-bit quantization for efficient and accurate llm serving . In Proceedings of Machine Learning a...

    2024