Pith. sign in

REVIEW 3 major objections 4 minor 56 references

Rotation and fine-grained group quantization can be made cooperative, giving 4-bit LLM inference near-full-precision accuracy while keeping dequantization fully integer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:10 UTC pith:S6UIPL7P

load-bearing objection GyRot makes rotation and fine-grained group quantization cooperate via a genuinely new R=2^g·G decoupling plus harmonic-aware permutation, with strong accuracy evidence; the main gap is that the load-bearing HAP mechanism lacks a specified selection algorithm and per-block outlier budget analysis. the 3 major comments →

arxiv 2607.27694 v1 pith:S6UIPL7P submitted 2026-07-30 cs.AR cs.LG

GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

Sangjin Kim , Yuseon Choi , Byeongcheol Kim , Jungjun Oh , Hoi-jun Yoo This is my paper
classification cs.AR cs.LG
keywords 4-bit quantizationLLM inferencerotation-based quantizationgroup quantizationHadamard transformasymmetric quantizationinteger dequantizationhardware accelerator
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that rotation and fine-grained group quantization — previously thought incompatible for low-bit LLM inference — can be made cooperative, yielding 4-bit accuracy close to full precision without floating-point dequantization. The key is to apply rotation at a coarser scope than the quantization group (CoRFiG, e.g., rotation over 1024 channels while groups are 32) and to permute high-magnitude outlier channels onto the harmonic +1/−1 rows of the Hadamard matrix (HAP), so each group's distribution stays tightly bounded after rotation. This boundedness lets scaling factors and zero-points be quantized to INT8, and a reformulated asymmetric quantization with ceiling-based zero-point rounding keeps dequantization fully integer. The consequence, if right, is that 4-bit weight/activation LLM inference can be both accurate and hardware-efficient — the paper reports LLaMA-3-8B zero-shot accuracy of 72.98% versus 74.19% for FP16, with up to 3.4x speedup and 3.6x energy savings.

Core claim

On its own terms, the paper establishes that the incompatibility between rotation and fine-grained group quantization is not fundamental but stems from applying rotation globally while group scaling acts locally. GyRot's CoRFiG chooses a rotation scope R = 2^g · G that is a power-of-two multiple of the group size G, so outliers spread across R channels — large enough to flatten distributions, small enough to preserve group-level coherence; the paper's default is R = 1024, G = 32. HAP then permutes selected outlier channels onto the harmonic rows of each local Hadamard block, rows that are constant +1 or −1 across the group, making each group's post-rotation values cluster tightly around a sh

What carries the argument

The paper's central mechanism is the pair CoRFiG and HAP. CoRFiG (coarse rotation, fine grouping) sets the rotation scope R to a power-of-two multiple of the quantization group G (e.g., R=1024, G=32), decoupling the two granularities so rotation flattens within a bounded window while groups keep local scaling. HAP (harmonic-aligned permutation) permutes high-magnitude outlier channels onto the harmonic rows of the local Hadamard matrix — rows that are all-+1 or all-−1 over each group — so each outlier contributes a constant sign within its group, producing tightly bounded, shifted-biased group distributions. This tightness is what lets scales and zero-points be INT8. A reformulated asymmetri

Load-bearing premise

The whole scheme rests on the premise that a small set of outlier channels can be identified once, per layer, and stably aligned to the harmonic rows across all tokens; if the significant outliers are token-dependent or more numerous than the harmonic rows, the tight per-group ranges and the INT8 scale/zero-point representation fall apart.

What would settle it

Give the model a set of inputs engineered so that the largest activation channels in a layer are not among the channels HAP permuted onto harmonic rows; if W4A4 zero-shot accuracy drops sharply relative to the paper's reported numbers, the static-outlier premise fails. Alternatively, measure the per-token stability of top-channel identities in the first QKV layer before rotation: if the top channels vary across tokens, HAP's one-time permutation cannot bound group ranges for every token.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • 4-bit weight/activation LLM inference becomes practical for edge and datacenter deployment: near-full-precision accuracy without floating-point dequantization overhead.
  • Small group sizes (e.g., 32) are no longer prohibitive in hardware, since dequantization stays in the integer domain with INT8 metadata.
  • Rotation-based quantization can be combined with fine-grained group quantization, reversing the accuracy degradation previous combined schemes reported.
  • The same recipe extends to more aggressive 3-bit weight settings (W3A4), suggesting headroom below 4 bits.
  • HAP permutations fuse into the weight matrix at zero runtime cost, so the accuracy benefit comes without per-token overhead.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Ed.: The harmonic-alignment principle may generalize beyond Hadamard matrices — any orthogonal transform with sign-constant rows over a group could serve as the alignment target, opening a family of rotation–group co-designs.
  • Ed.: The paper selects outlier channels statically; a dynamic selection that adapts per layer or per input distribution could widen the method's robustness to models with token-dependent outlier patterns.
  • Ed.: Fully integer dequantization suggests a path for GPU software stacks to keep more of the low-bit GEMM in integer tensor-core paths, if INT8 scale/zero-point handling is exposed.
  • Ed.: The saturation of perplexity improvement around R=1024 hints at a design rule relating rotation scope to model hidden width; sweeping R as a fraction of width across models would test that rule.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. GyRot proposes a quantization framework and accelerator that combine rotation with fine-grained group quantization (G=32) by decoupling the rotation scope from the group size (CoRFiG, R=2^g·G) and permuting outlier channels onto harmonic Hadamard rows (HAP). A reformulated asymmetric quantization and a ceiling-based zero-point rounding strategy are introduced to allow INT8 scale/zero-point metadata and a fully integer dequantization datapath. The paper reports W4A4 perplexity, zero-shot accuracy, and MT-bench results close to FP16 on LLaMA-family models, plus a 28nm RTL implementation of an 8×8×32 tensor PE array with a 5-stage FHT unit, claiming up to 3.4× speedup and 3.6× energy efficiency over baseline accelerators.

Significance. If the claims hold, GyRot addresses a real and current problem: the known incompatibility between Hadamard rotation and fine group sizes such as 32. The reformulation in Eqs. (6)–(7) is algebraically consistent, and the evaluation is broader than perplexity alone, including zero-shot tasks and MT-bench. The hardware contribution is concrete: RTL synthesis at 28nm, iso-compute-area comparisons, and an ablation study that attributes accuracy recovery specifically to HAP and to the reformulated asymmetric quantization. The main risks are that HAP's selection mechanism and capacity are underspecified, and that the INT8 zero-point quantization is not fully defined in the equations. These are load-bearing because CoRFiG alone degrades catastrophically when SF is quantized to INT8 (Table IV: PPL 364.17 vs 6.80 with HAP).

major comments (3)
  1. [Sec. IV-A (Fig. 3/4; Table IV)] HAP's capacity and selection rule are not specified. For the default G=32, R=1024, only R/G=32 harmonic rows per rotation block are sign-constant within each group. Section IV-A refers only to 'globally selected high-magnitude outlier channels' without giving a threshold, an algorithm, or the expected number of outliers per block. If any 1024-channel block has more than 32 significant outliers, the excess channels cannot be aligned to harmonic rows, and the bounded-group-range argument behind INT8 SF/ZP breaks. Table IV shows that CoRFiG without HAP yields PPL 364.17 with INT8 SF, so HAP is the linchpin of the fully integer claim. The paper should report per-layer/per-block outlier-count statistics, define the selection criterion, and provide a sensitivity analysis showing robustness when the outlier count exceeds 32.
  2. [Sec. IV-B / Sec. V-A (Eqs. (6)–(7); Fig. 5)] The quantization of the zero-point to INT8 is not fully specified. Eq. (6) defines z_x = ceil(-min(x_g)), an integer that can easily exceed the INT8 range for large activation ranges. Figure 5's caption mentions that zero-points are 'normalized with per-layer power-of-two scale,' but Eq. (7) uses z_x directly as if it were exact. If z_x is stored as INT8 with a scale z_s, the dequantization should include that scale (e.g., replace z_x by z_s·z_q or fold z_s into s_x). The paper should state how Z_X is normalized and quantized, and update Eq. (7) and the PE datapath description accordingly. This is essential to the fully integer dequantization claim.
  3. [Sec. VI-B (Table III; default configuration)] The default hyperparameters (G=32, R=1024) are selected using WikiText-2 perplexity on LLaMA-3-8B and then the same model/benchmark family is used for the main zero-shot and MT-bench evaluations. Table III itself shows R=512 giving slightly lower PPL (6.89) than the selected R=1024 (6.91). To support the state-of-the-art claim, the authors should either select the configuration on a separate validation split or a different model and then transfer, or report sensitivity of the final accuracy to this choice across all evaluated models. Without this, there is a risk that the reported gains are partially overfit to the evaluation setup.
minor comments (4)
  1. [Sec. V-B] The paragraph describing the FHT unit ('We implement a 5-stage, 32-way FHT unit...') is duplicated verbatim, with only minor wording differences. Remove one copy.
  2. [Throughout] Typos and inconsistencies: 'Institue' in the author affiliation, 'LLaMa3-8B' in Fig. 1, 'Equa. 6' in the Fig. 7 caption, and inconsistent use of 'R= 2 g ·G' spacing.
  3. [Table V] The GQ-only row shows INT8 ZP PPL (7.21) identical to FP16 and much better than FP8 (19.37). This is surprising and should be explained or corrected, as it does not follow the trend in the other rows.
  4. [Fig. 5 caption] The mention of per-layer power-of-two normalization for zero-points is only in the caption and not in the body; integrate this into the algorithm description or define it clearly.

Circularity Check

0 steps flagged

No load-bearing circularity: the core quantization/accuracy claims are empirically measured and the HAP range-reduction mechanism follows from Hadamard structure; only non-load-bearing self-citation/tuning noted.

full rationale

The central chain—CoRFiG/HAP produce tighter per-group ranges, enabling INT8 scale/zero-point dequantization while preserving accuracy—is not circular. The HAP mechanism is a constructive mathematical argument: permuting an outlier onto a Hadamard row that is constant over each quantization group makes that outlier contribute a constant shift within each group, reducing intra-group range. The paper does not define 'outlier' as 'a channel that fits into a harmonic row'; it states that 'globally selected high-magnitude outlier channels' are aligned, and whether R/G=32 rows suffice is an empirical robustness question, not an identity. The reformulated asymmetric quantization (Eq. 6) is a genuine alternative quantizer rather than a restatement of Eq. 4; its accuracy with INT8 SF/ZP is measured on WikiText-2, zero-shot tasks, and MT-Bench rather than derived from the formula. The INT8 SF/ZP results are benchmark measurements, not fitted predictions forced by construction. Self-citations to LightRot [19] appear as background, motivation, and baseline comparisons; they do not supply a uniqueness theorem or load-bearing premise that forces GyRot's design, so they are not circular. The choice of G32/R1024 is made after a design-space exploration on LLaMA-3-8B, which is a mild tuning concern, but the same configuration is applied across all models and the claims do not reduce to a single fitted constant. No equation in the paper can be exhibited as identical to an input by construction, so no circular step rises to the level required by the rubric.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The ledger shows that GyRot's accuracy story leans on standard math (Hadamard) plus two empirically-tuned components: the (G,R) design point and the HAP outlier-selection procedure. The fully-integer dequantization claim is a design choice validated only in the combined configuration, not a consequence of the mathematics alone.

free parameters (4)
  • Group size G (default 32) = 32
    Selected by design-space exploration in Table III/VI on LLaMA-3-8B; fine group size is central to the method.
  • Rotation scope R (default 1024) = 1024
    Default R=2^5·32 selected from Table III; R is a free hyperparameter controlling CoRFiG, and Table III shows PPL varies from 30.12 to 6.89 across R.
  • HAP outlier selection (top-k/threshold) = unspecified
    The number and identity of outlier channels permuted to harmonic rows are not specified; this is a hidden tuning choice affecting INT8 SF/ZP accuracy.
  • Scale factor and zero-point bit widths = INT8
    INT8 quantization of metadata is a design choice; the paper shows it only works after CoRFiG+HAP.
axioms (5)
  • standard math Hadamard matrices are orthogonal; rotation can be folded into weights because H^T H = I
    Invoked in Sec. II-B to justify fused rotation; follows Quarot [1].
  • ad hoc to paper Harmonic rows of the local Hadamard matrix align outlier signs so group-wise ranges stay bounded
    Introduced in Sec. IV-A / Fig. 4; no proof or formal bound, only empirical support.
  • domain assumption Outlier channels can be identified offline from calibration data and are token-invariant
    Sec. V-B fuses HAP permutation into weights; if outliers vary per token, the permutation would need runtime changes.
  • domain assumption WikiText-2 perplexity is a valid proxy for choosing default hyperparameters and reporting final accuracy
    Table III uses WikiText-2 PPL for G/R selection; Table I reports PPL on the same dataset.
  • domain assumption INT8 scale/zero-point dequantization preserves accuracy when combined with CoRFiG+HAP
    Supported by Tables IV-V but only empirically; no error bound is derived.

pith-pipeline@v1.3.0-daily-deepseek · 22961 in / 16738 out tokens · 159208 ms · 2026-08-01T03:10:17.964165+00:00 · methodology

0 comments
read the original abstract

Low-bit quantization is essential for efficient LLM inference, and both rotation and fine-grained group quantization have shown individual promise. However, their combination often leads to accuracy degradation or hardware overhead due to a mismatch between the global nature of rotation and the localized behavior of group scaling. We propose GyRot, a quantization framework and hardware accelerator that bridges this gap through algorithm-hardware co-design. GyRot introduces Coarse Rotation, Fine Grouping (CoRFiG) and Harmonic-Aligned Permutation (HAP) to enable cooperative integration of rotation and group quantization, enhancing quantizability while relaxing scaling factor precision. To further reduce hardware cost, we reformulate asymmetric quantization and introduce a zero-point rounding strategy that enables fully integer dequantization. Implemented on an INT4-based tensor PE architecture, GyRot achieves state-of-the-art 4-bit accuracy across LLaMA-family models, while delivering up to 3.4x speedup and 3.6x energy efficiency over baseline LLM accelerators. These results validate GyRot's practical effectiveness for scalable and energy-efficient LLM deployment.

Figures

Figures reproduced from arXiv: 2607.27694 by Byeongcheol Kim, Hoi-Jun Yoo, Jungjun Oh, Sangjin Kim, Yuseon Choi.

Figure 1
Figure 1. Figure 1: Effect of data rotation with different quantization granularities. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Hardware cost with different quantization granularity. to worse accuracy when rotation is applied. Similar findings are reported in AMXFP [22], where applying rotation with a group size of 32 resulted in accuracy degradation, ultimately leading to the removal of rotation. This discrepancy stems from a fundamental mismatch between the two techniques: rotation globally redistributes values across all channel… view at source ↗
Figure 3
Figure 3. Figure 3: Proposed quantization algorithm: Coarse-Rotation, Fine Grouping (CoRFiG) with Harmonic-Aligned Permutation (HAP). (G = 8, R = 32 case.) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of HAP on activation distribution after rotation. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of zero-point distributions in conventional and reformu [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of zero-point quantization according to the sign of quantization [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: GyRot PE. (a) Operation flow change for reformulated asymmetric quantization (Equa. 6).(b) Microarchitecture. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: shows the architectural overview of the GyRot accelerator. The accelerator adopts an 8 × 8 systolic PE array, where each PE supports 32-way 4-bit dot products, with group quantization applied. This configuration allows the 8×8×32 tensor array to perform 2048 operations in parallel. The systolic array operates in an output-stationary manner. Each PE performs dot products for intra-group accumulation and app… view at source ↗
Figure 9
Figure 9. Figure 9: Evaluation with LLM-as-a-Judge on MT-Bench [56], (a) Comparison [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: PE area and energy comparison on LLaMA-1-7B/13B, matching or surpassing prior rotation￾based methods, while GyRot-INT shows only marginal degra￾dation despite using fully integer scaling and zero-points. These results confirm that the cooperative rotation–group quantization design of GyRot remains effective even when bit precision is aggressively reduced, demonstrating its applica￾bility beyond the standa… view at source ↗
Figure 11
Figure 11. Figure 11: Speedup and energy efficiency comparison across accelerators on WikiText2 with various bit configurations. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Energy breakdown of GyRot in contrast with the baseline accelera [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 22 linked inside Pith

  1. [1]

    Quarot: Outlier-free 4-bit inference in rotated llms,

    S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “Quarot: Outlier-free 4-bit inference in rotated llms,”Advances in Neural Information Processing Systems, vol. 37, pp. 100 213–100 240, 2024

    2024

  2. [2]

    Piqa: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, J. Gao, Y . Choiet al., “Piqa: Reasoning about physical commonsense in natural language,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, no. 05, 2020, pp. 7432– 7439

    2020

  3. [3]

    Bitmod: Bit-serial mixture-of- datatype llm acceleration,

    Y . Chen, A. F. AbouElhamayed, X. Dai, Y . Wang, M. Andronic, G. A. Constantinides, and M. S. Abdelfattah, “Bitmod: Bit-serial mixture-of- datatype llm acceleration,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1082–1097

    2025

  4. [4]

    Fp4 all the way: Fully quantized training of llms,

    B. Chmiel, M. Fishman, R. Banner, and D. Soudry, “Fp4 all the way: Fully quantized training of llms,”arXiv preprint arXiv:2505.19115, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Boolq: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,”arXiv preprint arXiv:1905.10044, 2019

    Pith/arXiv arXiv 1905

  6. [6]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,”arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  7. [7]

    Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference,

    S. Dai, R. Venkatesan, M. Ren, B. Zimmer, W. Dally, and B. Khailany, “Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference,” inProceedings of Machine Learning and Systems (MLSys), vol. 3, 2021, pp. 873–884

    2021

  8. [8]

    Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point,

    B. Darvish Rouhani, D. Lo, R. Zhao, M. Liu, J. Fowers, K. Ovtcharov, A. Vinogradsky, S. Massengill, L. Yang, R. Bittneret al., “Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point,”Advances in neural information processing systems, vol. 33, pp. 10 271–10 281, 2020

    2020

  9. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprint arXiv:1810.04805, 2018

    Pith/arXiv arXiv 2018

  10. [10]

    Training dnns with hybrid block floating point,

    M. Drumond, T. Lin, M. Jaggi, and B. Falsafi, “Training dnns with hybrid block floating point,”Advances in Neural Information Processing Systems, vol. 31, 2018

    2018

  11. [11]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,”arXiv preprint arXiv:2210.17323, 2022

    Pith/arXiv arXiv 2022

  12. [12]

    The language model evaluation harness,

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “The language model evaluation harness,” 07 2024. [Online]. Available: https://zenodo.org/re...

    arXiv 2024

  13. [13]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Mar...

    2024

  14. [14]

    Olive: Accelerating large language models via hardware- friendly outlier-victim pair quantization,

    C. Guo, J. Tang, W. Hu, J. Leng, C. Zhang, F. Yang, Y . Liu, M. Guo, and Y . Zhu, “Olive: Accelerating large language models via hardware- friendly outlier-victim pair quantization,” inProceedings of the Interna- tional Symposium on Computer Architecture (ISCA), 2023, pp. 1–15

    2023

  15. [15]

    Ant: Exploiting adaptive numerical data type for low-bit deep neural network quantization,

    C. Guo, C. Zhang, J. Leng, Z. Liu, F. Yang, Y . Liu, M. Guo, and Y . Zhu, “Ant: Exploiting adaptive numerical data type for low-bit deep neural network quantization,” inProceedings of the 55th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’22. IEEE Press, 2023, p. 1414–1433

    2023

  16. [16]

    M-ant: Efficient low-bit group quantization for llms via mathematically adaptive numerical type,

    W. Hu, H. Zhang, C. Guo, Y . Feng, R. Guan, Z. Hua, Z. Liu, Y . Guan, M. Guo, and J. Leng, “M-ant: Efficient low-bit group quantization for llms via mathematically adaptive numerical type,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1112–1126

    2025

  17. [17]

    Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting,

    X. Hu, Y . Cheng, D. Yang, Z. Xu, Z. Yuan, J. Yu, C. Xu, Z. Jiang, and S. Zhou, “Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting,” arXiv preprint arXiv:2501.13987, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Panacea: Novel dnn accelerator using accuracy-preserving asymmetric quantiza- tion and energy-saving bit-slice sparsity,

    D. Kam, M. Yun, S. Yoo, S. Hong, Z. Zhang, and Y . Lee, “Panacea: Novel dnn accelerator using accuracy-preserving asymmetric quantiza- tion and energy-saving bit-slice sparsity,” inProceedings - 2025 IEEE International Symposium on High Performance Computer Architecture, HPCA 2025, 2025, pp. 701–715

    2025

  19. [19]

    Lightrot: A light- weighted rotation scheme and architecture for accurate low-bit large language model inference,

    S. Kim, Y . Choi, J. Oh, B. Kim, and H.-J. Yoo, “Lightrot: A light- weighted rotation scheme and architecture for accurate low-bit large language model inference,”IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2025

    2025

  20. [20]

    23.3 edgediff: 418.4mj/inference multi-modal few-step diffusion model accelerator with mixed-precision and reordered group quantization,

    S. Kim, J. Oh, J. So, Y . Choi, S. Kim, D. Im, G. Park, and H.-J. Yoo, “23.3 edgediff: 418.4mj/inference multi-modal few-step diffusion model accelerator with mixed-precision and reordered group quantization,” in2025 IEEE International Solid-State Circuits Conference (ISSCC), vol. 68, 2025, pp. 1–3

    2025

  21. [21]

    Tender: Accelerating large language models via tensor decomposition and runtime requantization,

    J. Lee, W. Lee, and J. Sim, “Tender: Accelerating large language models via tensor decomposition and runtime requantization,” inProceedings of the 51st Annual International Symposium on Computer Architecture (ISCA), 2024

    2024

  22. [22]

    Amxfp4: Taming activation outliers with asymmetric microscaling floating-point for 4-bit llm inference,

    J. Lee, J. Park, J. Kim, Y . Kim, J. Oh, J. Oh, and J. Choi, “Amxfp4: Taming activation outliers with asymmetric microscaling floating-point for 4-bit llm inference,”arXiv preprint arXiv:2411.09909, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Duquant: Distributing outliers via dual transformation makes stronger quantized llms,

    H. Lin, H. Xu, Y . Wu, J. Cui, Y . Zhang, L. Mou, L. Song, Z. Sun, and Y . Wei, “Duquant: Distributing outliers via dual transformation makes stronger quantized llms,”Advances in Neural Information Processing Systems, vol. 37, pp. 87 766–87 800, 2024

    2024

  24. [24]

    Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,”Proceedings of machine learning and systems, vol. 6, pp. 87–100, 2024

    2024

  25. [25]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,

    Y . Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han, “Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,”arXiv preprint arXiv:2405.04532, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

    Pith/arXiv arXiv 1907

  27. [27]

    Spinquant: Llm quantization with learned rotations,

    Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V . Chandra, Y . Tian, and T. Blankevoort, “Spinquant: Llm quantization with learned rotations,” inThe Thirteenth International Conference on Learning Representations, 2024

    2024

  28. [28]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” inProceedings of the International Conference on Learning Representations (ICLR), 2016

    2016

  29. [29]

    Dram power calculator,

    Micron Technology, Inc., “Dram power calculator,” https://www.micron. com/sales-support/design-tools/dram-power-calculator, accessed: 2025- 08-01

    2025

  30. [30]

    Nvidia tensorrt-llm,

    NVIDIA, “Nvidia tensorrt-llm,” https://docs.nvidia.com/tensorrt-llm, ac- cessed: 2025-10-20

    2025

  31. [31]

    Energy-efficient neural network ac- celerator based on outlier-aware low-precision computation,

    E. Park, D. Kim, and S. Yoo, “Energy-efficient neural network ac- celerator based on outlier-aware low-precision computation,” in2018 ACM/IEEE 45th Annual International Symposium on Computer Archi- tecture (ISCA). IEEE, 2018, pp. 688–698

    2018

  32. [32]

    Microscopiq: Acceler- ating foundational models through outlier-aware microscaling quantiza- tion,

    A. Ramachandran, S. Kundu, and T. Krishna, “Microscopiq: Acceler- ating foundational models through outlier-aware microscaling quantiza- tion,” inProceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 1193–1209

    2025

  33. [33]

    With shared microexponents, a little shifting goes a long way,

    B. D. Rouhani, R. Zhao, V . Elango, R. Shafipour, M. Hall, M. Mes- makhosroshahi, A. More, L. Melnick, M. Golub, G. Varatkaret al., “With shared microexponents, a little shifting goes a long way,” in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1–13

    2023

  34. [34]

    Microscaling data formats for deep learning,

    B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolfet al., “Microscaling data formats for deep learning,”arXiv preprint arXiv:2310.10537, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,”Communications of the ACM, vol. 64, no. 9, pp. 99–106, 2021

    2021

  36. [36]

    Resq: Mixed-precision quantization of large language models with low-rank residuals,

    U. Saxena, S. Sharify, K. Roy, and X. Wang, “Resq: Mixed-precision quantization of large language models with low-rank residuals,”arXiv preprint arXiv:2412.14363, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Omniquant: Omnidirectionally calibrated quan- tization for large language models,

    W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y . Qiao, and P. Luo, “Omniquant: Omnidirectionally calibrated quan- tization for large language models,”arXiv preprint arXiv:2308.13137, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    Glu variants improve transformer,

    N. Shazeer, “Glu variants improve transformer,”arXiv preprint arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  39. [39]

    Roformer: En- hanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: En- hanced transformer with rotary position embedding,”Neurocomputing, vol. 568, p. 127063, 2024

    2024

  40. [40]

    Flatquant: Flatness matters for llm quantization,

    Y . Sun, R. Liu, H. Bai, H. Bao, K. Zhao, Y . Li, J. Hu, X. Yu, L. Hou, C. Yuanet al., “Flatquant: Flatness matters for llm quantization,”arXiv preprint arXiv:2410.09426, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Design compiler: Timing, area, power, & test op- timization,

    Synopsys, “Design compiler: Timing, area, power, & test op- timization,” https://www.synopsys.com/implementation-and-signoff/rtl- synthesis-test/dc-ultra.html, accessed: 2025-08-01

    2025

  42. [42]

    Gemma: Open models based on gemini research and technology,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Riviere, M. Kale, J. C. Love, P. D. Tafti, L. Hussenot, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. H’eliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippolito, D. Reid...

    Pith/arXiv arXiv 2024

  43. [43]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V . Kerkez, M. Khabsa, I. Kloumann, A. Koren...

    Pith/arXiv arXiv 2023

  45. [45]

    Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks,

    A. Tseng, J. Chee, Q. Sun, V . Kuleshov, and C. De Sa, “Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks,”arXiv preprint arXiv:2402.04396, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Dfrot: Achieving outlier-free and massive activation-free for rotated llms with refined rotation,

    J. Xiang and S. Q. Zhang, “Dfrot: Achieving outlier-free and massive activation-free for rotated llms with refined rotation,”arXiv preprint arXiv:2412.00648, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Duarot: Dual rotation for advanced outlier mitigation in rotated llms,

    J. Xiang, Y . Zhang, C. Ma, Y . Wang, W. Lin, Y . Liuet al., “Duarot: Dual rotation for advanced outlier mitigation in rotated llms,” http://openreview.net/, 2024

    2024

  48. [48]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 38 087–38 099

    2023

  49. [49]

    Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation,

    Z. Yao, X. Wu, C. Li, S. Youn, and Y . He, “Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation,”arXiv preprint arXiv:2303.08302, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Rptq: Reorder-based post-training quantization for large language models,

    Z. Yuan, L. Niu, J. Liu, W. Liu, X. Wang, Y . Shang, G. Sun, Q. Wu, J. Wu, and B. Wu, “Rptq: Reorder-based post-training quantization for large language models,”arXiv preprint arXiv:2305.00000, 2023

    arXiv 2023

  51. [51]

    Gobo: Quan- tizing attention-based nlp models for low latency and energy efficient inference,

    A. H. Zadeh, I. Edo, O. M. Awad, and A. Moshovos, “Gobo: Quan- tizing attention-based nlp models for low latency and energy efficient inference,” inProceedings of the International Symposium on Microar- chitecture (MICRO), 2020, pp. 811–824

    2020

  52. [52]

    Hel- laswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?”arXiv preprint arXiv:1905.07830, 2019

    Pith/arXiv arXiv 1905

  53. [53]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Linet al., “Opt: Open pre-trained transformer language models,”arXiv preprint arXiv:2205.01068, 2022

    Pith/arXiv arXiv 2022

  54. [54]

    Afpq: Asymmetric floating point quantization for llms,

    Y . Zhang, S. Zhang, S. Cao, D. Du, J. Wei, T. Cao, and N. Xu, “Afpq: Asymmetric floating point quantization for llms,” inFindings of the Association for Computational Linguistics ACL 2024, 2024, pp. 28–36

    2024

  55. [55]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,”Proceedings of Machine Learning and Systems, vol. 6, pp. 196–209, 2024

    2024

  56. [56]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023

    2023