Pith. sign in

REVIEW 4 major objections 4 minor 28 references

LightRot claims a rotation scheme and 28nm accelerator deliver 4-bit LLaMA inference near full precision at 27.4 TOPS/W.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:57 UTC pith:L4LNKUE5

load-bearing objection The paper's GLR + ODA algorithmic additions are plausible and the hierarchical FHT hardware is a genuine contribution, but the WikiText-2 evaluation may be circular and the TOPS/W comparison looks wrong (ref [18] is 27.5), so the headline accuracy and efficiency claims are not fully supported as written. the 4 major comments →

arxiv 2607.27704 v1 pith:L4LNKUE5 submitted 2026-07-30 cs.AR cs.LG

LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

Sangjin Kim , Yuseon Choi , Jungjun Oh , Byeongcheol Kim , Hoi-Jun Yoo This is my paper
classification cs.AR cs.LG
keywords quantizationlarge language modelsrotationHadamard transformhardware acceleratoroutlier handlinglow-bit inferenceenergy efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to make rotation-based quantization practical for low-bit large language model inference by removing two obstacles: the computational cost of rotating activations and the leftover distortion from extreme outliers. It claims that Grouped Local Rotation (GLR) cuts rotation operations by 9–36x with Fast Hadamard Transform units, and Outlier Direction Aligning (ODA) pre-permutes the largest outliers to the all-ones row of the Hadamard matrix, reducing quantization range by an average of 3.87x in the worst groups. Combined with group quantization, the method reaches WikiText-2 perplexities of 5.73, 5.08, and 6.98 for LLaMA2-7B, LLaMA2-13B, and LLaMA3-8B, close to the FP16 baselines of 5.47, 4.88, and 6.14. A 28nm accelerator implementing this scheme achieves 27.4 TOPS/W for 4-bit inference, and per-block energy for LLaMA3-8B falls to 4.0 mJ, a 3.9x improvement over an 8-bit no-rotation baseline. If true, this means near-full-precision-quality LLMs can run at very low energy on dedicated hardware.

Core claim

On the paper's own terms, the discovery is that the two practical barriers to rotation-based low-bit LLM inference — the high compute cost of rotations when dimensions are not powers of two, and the bimodal distributions caused by extreme outliers that survive rotation — can both be removed. Grouped Local Rotation restricts rotations to power-of-two-size groups (128 channels in the experiments), enabling a small hierarchical Fast Hadamard Transform to replace large or matrix-multiplication-based rotation units. Outlier Direction Aligning permutes the largest outlier in each group to the row of the Hadamard matrix that is all +1, so that outstanding outliers are not split into two modes; a gr

What carries the argument

Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA). GLR partitions the activation vector into groups of g channels and applies a g×g Hadamard rotation inside each group, so a Fast Hadamard Transform unit works even when the model dimension is not a power of two; the hardware uses a two-step hierarchical FHT (16-way then 8-way) around a transposable register file to keep area and energy low. ODA uses the Sylvester construction of Hadamard matrices, whose first row is all +1, and permutes the largest outlier in each group to that row before rotation; this concentrates the outlier in one direction, prevents a bimodal post-rotation distribution, and enables a group-shared asymmetr

Load-bearing premise

The load-bearing premise is that outlier channel indices and their relative magnitudes stay consistent enough across tokens and batches that a permutation learned on one calibration set (WikiText-2) remains correct for the full inference run; if those indices drift on unseen data, the aligned outlier is no longer placed on the all-ones row, and the range-reduction benefit of ODA collapses.

What would settle it

Run a LLaMA2-7B model quantized with LightRot's full recipe on a corpus whose outlier channels are substantially re-ordered after calibration, and check whether the perplexity gap to FP16 grows beyond the 0.26 reported on WikiText-2; a sharp degradation would indicate that the ODA permutation is not transferable across distributions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If LightRot's claims hold, 4-bit quantized LLaMA2-13B and LLaMA3-8B can reach WikiText-2 perplexity within 0.26 and 0.84 of FP16, respectively, while all IA, W, and KV are INT4; this makes low-bit inference viable for production-quality chat models.
  • Rotation energy overhead, previously over 35.7% of layer energy in LLaMA2-7B with naive rotation, drops to a negligible level after GLR+ODA and the hierarchical FHT unit, enabling 3.9x per-block energy reduction versus an 8-bit no-rotation baseline for LLaMA3-8B.
  • The accelerator's 27.4 TOPS/W exceeds the 1.91–22.9 TOPS/W range of prior processors, according to the paper's comparison, without relying on mixed precision or arbitrary quantization.
  • Because ODA improves quantizability independently of GLR, the two techniques can in principle be applied together or separately to other models and other low-bit formats.
  • The MT-Bench win-rate results suggest the accuracy benefit transfers to long-form conversational outputs, not just perplexity on short text slices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's quality numbers depend on the stability of outlier channel indices across tokens and batches, since ODA's permutation is fitted on a calibration set (WikiText-2) and then applied at inference; a natural extension is to test whether the permutation generalizes to held-out domains such as code or medical text, and whether re-calibrating on a small sample of the target domain recovers acc
  • The hierarchical FHT design with a transposable register file is a general hardware pattern for any rotation-based scheme in non-power-of-two dimensions, so it could benefit other quantization methods that use Hadamard rotations, not only LightRot.
  • ODA introduces an asymmetric quantization bias; the paper argues the 128-sized group keeps the overhead low. A testable extension is to combine the same alignment idea with smaller-group formats such as microscaling (MX) to see whether the accuracy gain survives when group size is reduced.
  • Since the 27.4 TOPS/W is demonstrated on a 28nm process at 250 MHz with a relatively small 4.86 mm² die, porting the same architecture to a more advanced node would likely raise efficiency further, though the paper does not claim this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. LightRot proposes a low-bit LLM inference scheme combining Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA) with a hierarchical Fast Hadamard Transform (FHT) unit. The algorithm is evaluated on WikiText-2 perplexity and MT-Bench, reporting 4-bit results close to FP16 for LLaMA2-7B/13B and LLaMA3-8B. A 28nm accelerator implementation is described, claiming 27.4 TOPS/W peak energy efficiency for 4-bit inference and positioning this as surpassing prior state-of-the-art.

Significance. If the accuracy results hold under a clean calibration/evaluation protocol, the algorithmic contribution would be a useful step toward making rotation-based quantization hardware-efficient for LLMs. The GLR idea is straightforward and the hardware implementation with hierarchical FHT appears non-trivial. The paper also goes beyond GPT-2 evaluations by testing on LLaMA-class models and MT-Bench, which is commendable. However, the two main claims — near-FP16 perplexity and superiority over prior accelerators — are not currently supported by the evidence as presented.

major comments (4)
  1. [§III-B and §IV-A] The ODA permutation order is stated to be 'pre-determined using a calibration set, such as WikiText-2 perplexity benchmarks' (§III-B), yet Table 1 reports final perplexity on WikiText-2 without disclosing any calibration/evaluation split. If the same token subset is used to select outlier channel indices and to compute PPL, the results are leaked and not an independent measure. The authors must specify the calibration data (e.g., a separate split or a distinct corpus) and report evaluation on a disjoint test set. This is load-bearing because the near-FP16 PPL numbers in Table 1 are the principal algorithmic claim.
  2. [§III-B] The stability assumption that 'outlier indices remain consistent across batches' is asserted but never tested. No comparison between calibration-set outlier ranks and evaluation-set outlier ranks is reported, nor any cross-corpus experiment. If the assumption fails on unseen data, the ODA permutation order may misalign and the range-reduction benefit shown in Fig. 8 would not transfer, invalidating the generalization of the Table 1 PPL improvements. Please add a robustness experiment, e.g., calibrating on one corpus and evaluating on another, or a quantitative outlier-rank correlation analysis.
  3. [§VI / Table II / Abstract] The paper claims 27.4 TOPS/W 'surpassing prior state-of-the-art designs.' However, the cited prior work [18] (ISSCC 2022) reports 27.5 TOPS/W in the same 28nm process. The comparison table lists prior efficiencies ranging from 1.91–22.9 TOPS/W, excluding [18] and thereby making the 'surpassing' claim internally inconsistent. Please either include [18] in the comparison and qualify the claim (e.g., 'comparable' or 'better under a different metric'), or provide a fairer normalized comparison accounting for precision, model, and measurement conditions.
  4. [§IV-B / Fig. 9] The MT-Bench win/loss rates are reported without confidence intervals, number of prompts, or any significance test. Since Fig. 9(c) shows the proposed method is 'relatively competitive' against FP16 but the win rates are all below 50%, the statistical basis for the 'robust applicability' conclusion is unclear. Please report the number of MT-Bench prompts and include confidence intervals or a significance test, especially because the win-rate differences between configurations appear small (e.g., between +GQ and +GQ+GLR in Fig. 9(a) by visual inspection).
minor comments (4)
  1. [Captions and text] Several figure references are inconsistent. §V-C refers to 'Fig. 13' for the Gathering Unit, but the Gathering Unit appears in Fig. 12, while Fig. 13 is the chip summary. Also, §III-C is mentioned in the text but ODA is in §III-B.
  2. [Fig. 8 caption] The caption says 'Evaluated on the top 5% Outlier Group' — should probably be 'top 5% outlier groups' to match the description in the text.
  3. [Reference [22]] The reference for EdgeDiff lists the venue as ISSCC 2024 but the date as Feb. 2025; the year and page information should be corrected.
  4. [Typos] Fig. 14 caption has 'LLaMMA2-7B' (extra M). Fig. 9 caption has 'opeartion' typo. The name 'QuaRot' is spelled inconsistently as both 'Quarot' and 'QuaRot'.

Circularity Check

1 steps flagged

ODA's permutation order is calibrated on WikiText-2 and PPL is reported on WikiText-2 with no disclosed split; Table 1 may be leaked.

specific steps
  1. fitted input called prediction [Section III-B (ODA) and Section IV-A, Table I]
    "outlier indices remain consistent across batches, allowing the permutation order to be pre-determined using a calibration set, such as WikiText-2 perplexity benchmarks [16]. ... Table 1 presents the perplexity (PPL) results on the WikiText-2 dataset [16] for various configurations, demonstrating the effectiveness of the proposed features (+GLR and +ODA) in improving quantization quality while maintaining low-bit inference efficiency."

    The ODA permutation order is a fitted parameter: it is selected on a calibration set and then carried into inference. The headline accuracy is WikiText-2 perplexity, and the calibration set is named as 'WikiText-2 perplexity benchmarks' while the evaluation is also on the WikiText-2 dataset. As written, no disjoint calibration/test split is disclosed, so the near-FP16 PPL values in Table 1 can measure the quantizer's fit to the evaluation text itself rather than an independent prediction. No stability check of outlier ranks on held-out data is reported.

full rationale

The only load-bearing circularity is the ODA calibration/evaluation coupling. Section III-B states outlier indices are stable and the permutation order can be pre-determined using a calibration set 'such as WikiText-2 perplexity benchmarks'; Section IV-A then reports the final perplexity on the WikiText-2 dataset without stating that calibration and evaluation token sets are disjoint. On this text, the permutation order is fitted to the same named benchmark that is scored, so the near-FP16 results in Table 1 are not independently established. The rest of LightRot builds on QuaRot's external rotation framework, and the hardware-efficiency claim (28nm, 27.4 TOPS/W) is self-contained from the chip implementation and energy analysis. Self-citations to [21] and [22] are minor and not load-bearing. Score 6 rather than higher because the GLR/FHT hardware contributions and the accelerator energy claims remain independent; only the core accuracy evaluation is partly reduced to its calibration input.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central claim rests on standard Hadamard math plus a domain assumption about outlier stability. The main fitted quantities are the group size, the per-group scale/bias, the outlier threshold, and the permutation order. No new physical entities are introduced.

free parameters (4)
  • GLR group size g = 128
    Chosen by hand to match the group size of group quantization used in prior works (QuaRot/Atom); not derived from any optimization. Introduced in Section III-A.
  • Group-wise quantization scale IAscale and bias IAbias = per-group values from calibration data
    Estimated from calibration activations; the ODA accuracy results depend on these fitted values (equations in Section III-B).
  • ODA outstanding-outlier threshold = not stated
    The paper distinguishes 'outstanding outliers' (thousands to tens of thousands of times larger than inliers) but gives no explicit rule for how many outliers per group are selected or the exact magnitude cutoff (Section III-B, Fig. 6-8).
  • ODA permutation order (outlier channel indices) = per-model, from WikiText-2 calibration
    Outlier indices are determined from calibration and frozen; this fitted structure directly affects the PPL evaluation and is a key source of potential leakage (Section III-B).
axioms (6)
  • standard math Hadamard rotation preserves the network output when the inverse rotation is folded into weights
    Inherited from QuaRot [11]; used as the correctness basis for all rotation operations (Section II-B).
  • domain assumption Outlier channels of LLM input activations are stable across tokens and batches
    Needed so ODA's permutation, calibrated on a small set, remains valid at inference (Section III-B).
  • standard math Sylvester construction yields Hadamard matrices whose first row is all +1
    Basis of ODA's all-1-row alignment and of the FHT factorization (Section III-B).
  • standard math The 128-point Hadamard transform can be factored into a two-stage 16-way then 8-way FHT via a transpose
    Underpins the hierarchical FHT hardware design (Section V-B).
  • domain assumption Group quantization with the same group size as GLR compensates the scale misalignment introduced by local rotation
    Used to explain why GLR only improves accuracy when combined with GQ (Section IV-A).
  • domain assumption The calibration set is representative of the inference distribution
    Required for ODA permutation and scale/bias fitting to transfer to the test benchmark (Section III-B).

pith-pipeline@v1.3.0-daily-deepseek · 14748 in / 12302 out tokens · 121382 ms · 2026-08-01T02:57:00.137357+00:00 · methodology

0 comments
read the original abstract

As large language models (LLMs) continue to demonstrate exceptional capabilities across various domains, the challenge of achieving energy-efficient and accurate inference becomes increasingly critical. This work presents LightRot, a lightweight rotation scheme and dedicated hardware accelerator designed for low-bit LLM inference. The proposed architecture integrates Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA) algorithms with a hierarchical Fast Hadamard Transform (FHT)-based rotation unit to address key challenges in low-bit quantization, including the energy overhead of rotation operations. The proposed accelerator, implemented in a 28nm CMOS process, achieves a peak energy efficiency of 27.4 TOPS/W for 4-bit inference, surpassing prior state-of-the-art designs. Unlike conventional approaches that rely on higher-precision inference or evaluate on basic language modeling tasks like GPT-2, LightRot is optimized for advanced models such as LLaMA2-13B and LLaMA3-8B. Its performance is further validated on MT-Bench, demonstrating robust applicability to real-world conversational scenarios and redefining benchmarks for chat-based AI systems. By synergizing algorithmic innovations and hardware efficiency, this work sets a new paradigm for scalable, low-bit LLM inference, paving the way for sustainable AI advancements.

Figures

Figures reproduced from arXiv: 2607.27704 by Byeongcheol Kim, Hoi-Jun Yoo, Jungjun Oh, Sangjin Kim, Yuseon Choi.

Figure 1
Figure 1. Figure 1: (a) Challenge of LLM Quantization by Outlier and (b) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Applying Rotation in LLM Model with Online/offline Rotation. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Challenges of Rotation-based Quantization [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Concept of Grouped Local Rotation [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Motivation of Outlier Direction Aligning. [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Concept of Outlier Direction Aligning. groups and applying smaller rotation matrices to each group instead of using a single large rotation matrix. The principle of rotation-based quantization relies on redistributing outliers across channels to enhance quantizability. GLR achieves suf￾ficient quantizability by dispersing outliers locally within each group rather than across all channels. To implement this… view at source ↗
Figure 8
Figure 8. Figure 8: Range Reduction with the ODA. Evaluated on the top [PITH_FULL_IMAGE:figures/full_fig_p005_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Evaluation with LLM-as-a-Judge [17]. Win/Loss Rate [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Overall Architecture. required for rotation tasks. These components are intercon￾nected by a high-bandwidth interconnect network. The Global Memory handles intermediate data storage for activations and weights, while the Top Controller manages pipeline scheduling and synchronization between the NPU Core and the Rotation Unit. The NPU Core is a highly optimized integer-based pro￾cessing unit designed for l… view at source ↗
Figure 11
Figure 11. Figure 11: Hierarchical FHT Operation with Transposable-RF [PITH_FULL_IMAGE:figures/full_fig_p009_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Gathering Unit for ODA. Group ID Counter and Table Counter. The Inlier Address Generator handles the sequential generation of inlier addresses by filtering out addresses that correspond to outlier channels. The data flow begins with the Channel Counter, which checks whether the current channel is the first in the group. For the first channel in a group, the Group ID Counter directs the Outlier Address Gen… view at source ↗
Figure 13
Figure 13. Figure 13: (a) Chip Performance Summary (b) Energy Consump [PITH_FULL_IMAGE:figures/full_fig_p010_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Accuracy-Energy Comparison of Different Quantiza [PITH_FULL_IMAGE:figures/full_fig_p010_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 1 canonical work pages

  1. [1]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., ”Language Models are Few-Shot Learners,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877–1901, 2020

    1901

  2. [2]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, et al., ”LLaMA 2: Open Foundation and Fine-Tuned Chat Models,” inarXiv preprint, arXiv:2307.09288, 2023. [Online]. Available: https://arxiv.org/abs/2307.09288

    Pith/arXiv arXiv 2023

  3. [3]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, et al., ”The Llama 3 Herd of Mod- els,” inarXiv preprint, arXiv:2407.21783, 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  4. [4]

    A. Q. Jianget al., ”Mixtral of Experts,”arXiv preprint arXiv:2401.04088, Jan. 2024. [Online]. Available: https://arxiv.org/abs/2401.04088

    Pith/arXiv arXiv 2024

  5. [5]

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, ”AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration,” inProceedings of Machine Learning and Systems (MLSys), vol. 6, 2024

    2024

  6. [6]

    S. Kim, C. R. C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, ”SqueezeLLM: Dense-and-Sparse Quan- tization,” presented atICML 2024 Poster Session, May 2024. [Online]. Available: https://arxiv.org/abs/2309.05662

    Pith/arXiv arXiv 2024

  7. [7]

    C. Lee, J. Jin, T. Kim, H. Kim, and E. Park, ”OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 12, pp. 13355–13364, 2024. [Online]. Available: https://doi.org/10.1609/aaai.v38i12.29237

    work page doi:10.1609/aaai.v38i12.29237 2024

  8. [8]

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, ”SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models,” inProceedings of the 40th International Con- ference on Machine Learning (ICML), PMLR, vol. 202, pp. 38087–38099, 2023

    2023

  9. [9]

    Zhao, C.-Y

    Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, ”Atom: Low-Bit Quantization for Efficient and Accurate LLM Serving,” inProceedings of Machine Learning and Systems (MLSys), vol. 6, 2024

    2024

  10. [10]

    Y . Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han, ”QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving,” inarXiv preprint, arXiv:2405.04532, May 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2405.04532

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.04532 2024

  11. [11]

    Ashkboos, A

    S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, ”QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs,” inarXiv preprint, arXiv:2404.00456, Apr

    Pith/arXiv arXiv

  12. [12]

    Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V . Chandra, Y . Tian, and T. Blankevoort, ”SpinQuant: LLM Quantization with Learned Rotations,” inarXiv preprint, arXiv:2405.16406, May 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2405.16406

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.16406 2024

  13. [13]

    Y . Sun, R. Liu, H. Bai, H. Bao, K. Zhao, Y . Li, J. Hu, X. Yu, L. Hou, C. Yuan, X. Jiang, W. Liu, and J. Yao, ”FlatQuant: Flatness Matters for LLM Quantization,” inarXiv preprint, arXiv:2410.09426, Oct. 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2410.09426

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.09426 2024

  14. [14]

    Tseng, J

    A. Tseng, J. Chee, Q. Sun, V . Kuleshov, and C. De Sa, ”QuIP: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks,” presented atICML 2024 Poster Session, May 2024. [Online]. Available: https://arxiv.org/abs/2405.04532

    Pith/arXiv arXiv 2024

  15. [15]

    Shazeer, ”GLU Variants Improve Transformer,” inarXiv preprint, arXiv:2002.05202, Feb

    N. Shazeer, ”GLU Variants Improve Transformer,” inarXiv preprint, arXiv:2002.05202, Feb. 2020. [Online]. Available: https://arxiv.org/abs/2002.05202

    Pith/arXiv arXiv 2002

  16. [16]

    Merity, C

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, ”Pointer Sentinel Mixture Models,” inProceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017. [Online]. Available: https://arxiv.org/abs/1609.07843 IEEE JOURNAL ON EMERGING AND SELECTED TOPICS IN CIRCUITS AND SYSTEMS, VOL. XX, NO. X, XX, 2025 13

    Pith/arXiv arXiv 2017

  17. [17]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, ”Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” in Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, vol. 36, 2023

    2023

  18. [18]

    Y . Wang, Y . Qin, D. Deng, J. Wei, Y . Zhou, Y . Fan, T. Chen, H. Sun, L. Liu, S. Wei, and S. Yin, ”A 28nm 27.5 TOPS/W approximate computing- based transformer processor with asymptotic sparsity speculating and out-of-order computing,” inProceedings of the 2022 IEEE International Solid-State Circuits Conference (ISSCC), IEEE, pp. 1-3, 2022

    2022

  19. [19]

    T. Tambeet al., ”22.9 A 12nm 18.1TFLOPs/W Sparse Transformer Processor with Entropy-Based Early Exit, Mixed-Precision Predication and Fine-Grained Power Management,” inProceedings of the 2023 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA, pp. 342-344, 2023

    2023

  20. [20]

    Qinet al., ”Ayaka: A Versatile Transformer Accelerator With Low- Rank Estimation and Heterogeneous Dataflow,”IEEE Journal of Solid- State Circuits, pp

    Y . Qinet al., ”Ayaka: A Versatile Transformer Accelerator With Low- Rank Estimation and Heterogeneous Dataflow,”IEEE Journal of Solid- State Circuits, pp. 1–15, 2024

    2024

  21. [21]

    S. Kim, S. Kim, W. Jo, S. Kim, S. Hong, and H.-J. Yoo, ”C-Transformer: A 2.6-18.1µJ/Token Homogeneous DNN-Transformer/Spiking- Transformer Processor with Big-Little Network and Implicit Weight Generation for Large Language Models,” inProceedings of the 2024 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA, Feb. 2024, doi:...

    arXiv 2024

  22. [22]

    S. Kim, J. Oh, J. So, Y . Choi, S. Kim, D. Im, G. Park, and H.-J. Yoo, ”EdgeDiff: 418.4mJ/inference Multi-modal Few-step Diffusion Model Accelerator with Mixed-Precision and Reordered Group-Quantization,” inProceedings of the 2024 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA, Feb. 2025

    2024

  23. [23]

    B. D. Rouhaniet al., ”Microscaling data formats for deep learning.” in arXiv preprint arXiv:2310.10537, 2023b

    Pith/arXiv arXiv

  24. [24]

    Leeet al., ”AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference” inarXiv preprint arXiv:2411.09909, 2024

    J. Leeet al., ”AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference” inarXiv preprint arXiv:2411.09909, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Kelleret al., ”A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nm,” inIEEE Journal of Solid-State Circuits, vol

    B. Kelleret al., ”A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nm,” inIEEE Journal of Solid-State Circuits, vol. 58, no. 4, pp. 1129-1141, April 2023, doi: 10.1109/JSSC.2023.3234893

    arXiv 2023

  26. [26]

    S. Moon, H. -G. Mun, H. Son and J. -Y . Sim, ”Multipurpose Deep- Learning Accelerator for Arbitrary Quantization With Reduction of Stor- age, Logic, and Latency Waste,” inIEEE Journal of Solid-State Circuits, vol. 59, no. 1, pp. 143-156, Jan. 2024, doi: 10.1109/JSSC.2023.3312615. Sangjin Kim(Member, IEEE) received his B.S., M.S., and Ph.D. degrees in Elec...

    arXiv 2024

  27. [28]

    degree in Graduate School of AI Semiconductor, Ko- rea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

    He is currently pursuing an M.S. degree in Graduate School of AI Semiconductor, Ko- rea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea. His current re- search interests include hardware-aware optimiza- tion algorithms and energy-efficient deep learning processors. Byeongcheol Kim(Graduate Student Member, IEEE) received the B.S....

    2018

  28. [2024]

    Available: https://doi.org/10.48550/arXiv.2404.00456

    [Online]. Available: https://doi.org/10.48550/arXiv.2404.00456

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.00456