Pith. sign in

REVIEW 3 major objections 4 minor 25 references

Turning LLM Activations Quantization-Friendly

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Applying channel-wise scaling before a Hadamard rotation (Smooth Rotation) lowers 4-bit layer-wise quantization error in LLaMA2-7B, with the largest gains on layers that have massive activation outliers.

desk verdict Scaling before Hadamard rotation is a sensible trick for 4-bit down-projection layers, but the in-sample evaluation and a normalization slip in Eq (9) leave the mechanism unproven, and the positioning against DuQuant is missing. read the letter →

arxiv 2506.01967 v1 pith:NCUQNQMQ submitted 2025-05-11 cs.LG cs.AIcs.CL

Patrik Czakó , Gábor Kertész , Sándor Szénási This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords activationquantization4-bitoutliersHadamardrotationchannel-wisesmoothingdifficultyLLMinferenceweight-activation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a cheap preprocessing step—divide each activation channel by a scale computed from that channel's maximum weight magnitude, then apply a Hadamard rotation—makes LLaMA2-7B activations substantially easier to quantize to 4 bits without changing the model's learned weights. The authors call this Smooth Rotation and argue it gets the best of both existing fixes: channel-wise scaling flattens activations but makes weights harder to quantize, while rotation redistributes outliers but fails when a single token contains a few values far larger than the rest. They support the claim with a mathematical model of a token with massive outliers, plus layer-wise 4-bit quantization-error measurements across all modules. If the claim holds, it is a practical step toward W4A4 inference, where both weights and activations use 4-bit integers, with no additional training or codebook.

What carries the argument

The load-bearing object is the Smooth Rotation transform: first apply a diagonal scale $s_j = \sqrt{\max|X_j| / \max|W_j|}$ to activations and its inverse to weights, then multiply both by the same Hadamard matrix $R$ whose entries are $\pm 1/\sqrt{d}$, so the linear map remains exactly unchanged. The argument runs through Equation (9): after smoothing, the largest entry of a rotated token that has massive outliers on a small set $\mathcal{O}$ of channels is approximately $\sum_{i \in \mathcal{O}} \sqrt{|o_i|\, \max|W_i|} \,/\, d$, versus $\sum_{i\in\mathcal{O}} |o_i| / \sqrt{d}$ for rotation alone. It is that replacement of $|o_i|$ by $\sqrt{|o_i|\, \max|W_i|}$ and the denominator changing from $\sqrt{d}$ to $d$ that carries the paper's case that smoothing before rotation shrinks the quantization step size. A secondary device is the quantization difficulty metric, the standard deviation of channel magnitudes, which the paper uses to explain why rotation and smoothing behave differently on weights versus activations.

What would settle it

Record real activation tokens from a down-projection layer, identify channels whose values are far outside the noise floor, and compute the largest absolute value after smoothing plus Hadamard rotation for each token; if a substantial fraction of tokens have a maximum far above the Equation (9) estimate, then the paper's mathematical explanation is not what is driving the observed error reduction.

Watch

Extended reading notes

Core claim

On LLaMA2-7B with 4-bit symmetric round-to-nearest quantization, applying channel-wise scaling with smoothing strength $\alpha = 0.5$ before a Hadamard rotation (Smooth Rotation) yields the lowest layer-wise squared Frobenius quantization error among the original model, scaling alone, rotation alone, and the hybrid, across nearly every module. The paper shows that rotation alone can be worse than no transformation on down-projection layers with massive outliers, because the rotated token's largest magnitude is approximately the sum of the outlier magnitudes divided by $\sqrt{d}$. After smoothing, the largest magnitude becomes approximately $\sum_{i\in\mathcal{O}} \sqrt{|o_i|\, \max|W_i|} \,/\, d$, so the outlier contribution is both dampened by the weight magnitudes and spread over $d$ dimensions. The paper also introduces a quantization-difficulty metric—the standard deviation of channel magnitudes—and reports a correlation above 0.97 with layer-wise error once a few outlier-heavy layers are excluded. The authors are careful to claim Smooth Rotation is currently best justified on down-projection layers and recommend end-to-end perplexity tests for future work.

Load-bearing premise

The derivation assumes that massive outliers sit in a small fixed set of channels while all other entries in a token are independent noise with one variance, and that some column of the Hadamard rotation has signs aligned with all of those outliers at once; if either fails on real activations, the stated proof of Smooth Rotation's benefit no longer goes through.

Editorial extensions

If this is right

  • Smooth Rotation gives the lowest layer-wise quantization error among original, smoothing-only, rotation-only, and the hybrid across almost all LLaMA2-7B modules, with the largest margin on down-projection layers that contain massive outliers.
  • Because the transform is an exact reparameterization of each linear layer, it changes no model outputs before quantization; only a Hadamard application to activations and a rescaled weight matrix are added to inference.
  • The mechanism directly targets the quantization step size: after smoothing and rotation, the maximum absolute value that sets the 4-bit grid is roughly $\sum \sqrt{|o_i|\max|W_i|}/d$ rather than $\sum |o_i|/\sqrt{d}$, shrinking the step in exactly the outlier-heavy layers where rotation alone fails.
  • The paper limits its current recommendation to down-projection layers; attention-output and gate projections may need a tuned $\alpha$ above 0.5 before smoothing helps, as the experiments show for those modules.
  • The method uses fixed, online-computed scaling with $\alpha = 0.5$ and no learned rotation, so the extra machinery is cheap and does not require quantizer training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Equation (9) is the real mechanism, the benefit of Smooth Rotation should be largest when each massive outlier $|o_i|$ is much larger than the corresponding $\max|W_i|$; measuring the gain per layer against that ratio would test the explanation directly.
  • The paper measures only layer-wise error, not generative quality; a natural next experiment is W4A4 perplexity after Smooth Rotation, which the paper itself lists as future work.
  • The same smoothing-before-rotation pattern could plausibly help the value and query projections (which share the key projection's input tensor) or even weight-only quantization, though the paper does not claim this.
  • The sign-alignment assumption suggests that rotations with a deliberately balanced first column, or random orthogonal rotations, could behave differently from the Sylvester Hadamard matrix on the same layers; comparing rotations would reveal whether the benefit is specific to Hadamard alignment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies activation outliers in LLaMA2-7B and proposes "Smooth Rotation," a hybrid preprocessing method that applies channel-wise scaling (SmoothQuant-style, alpha=0.5) before a Hadamard rotation. It introduces a "quantization difficulty" metric based on the standard deviation of channel magnitudes, reports layer-wise 4-bit quantization errors for original, smoothed, rotated, and smooth-rotated activations, and provides an analytical expression (Eq. (9)) intended to explain why smoothing before rotation reduces the maximum rotated activation magnitude. The main empirical claim is that Smooth Rotation gives the lowest layer-wise squared Frobenius quantization error, especially in down-projection layers that contain massive token-specific outliers.

Significance. If the claim holds, Smooth Rotation is a cheap and simple preprocessing step for W4A4 quantization, and the proposed quantization-difficulty metric could be a useful diagnostic tool. The paper also usefully documents the failure mode of rotation alone on massive-outlier layers. The strengths are the clear layer-wise experimental setup, the explicit connection to SmoothQuant and QuaRot, the promised code release, and the attempt to give a mathematical mechanism rather than only an empirical observation. However, the significance is currently limited by two load-bearing issues: the smoothing scales are computed in-sample, which makes the comparison to rotation unequal and not deployment-equivalent for pre-quantized weights, and Eq. (9) appears to be mis-normalized, undermining the theoretical narrative.

major comments (3)
  1. [Section IV-E, Eq. (9)] The normalization in Eq. (9) is inconsistent with Eqs. (7)-(8). For rotation alone, Eq. (8) gives max(|t_hat|) = (sum_{i in O} |o_i|)/sqrt(d) + |epsilon|. If smoothing replaces each |o_i| with sqrt(|o_i| max|W_i|), the same derivation yields max(|tilde t|) ≈ (sum_{i in O} sqrt(|o_i| max|W_i|))/sqrt(d), not the printed denominator d. As written, Eq. (9) is a factor of sqrt(d) too small, which appears to be the source of the "doubling the number of dimensions" claim. Please provide a corrected derivation or clarify why the denominator differs.
  2. [Section III-C and Fig. 4] The scaling factors s_j are computed online from the current data, whereas rotation is a fixed transform. Because the weight transform diag(s)W is tied to s, an online s_j would require re-quantizing W for every inference batch; the experiment therefore does not measure the method in the W4A4 deployment setting described in the Introduction. The paper should rerun the comparison with s_j fixed from a separate calibration set and report performance on distinct evaluation data, and should reconcile this with Section V's statement that the method "depends on calibration."
  3. [Section IV-D, Eqs. (6)-(8)] The derivation assumes outliers occupy a fixed small set O and all other coordinates are independent Gaussian noise, and it relies on the existence of a Hadamard column whose signs align with all outliers. Real activations exhibit correlated and token-varying outliers, and the first Sylvester-Hadamard column is not mean-zero, so the conditions for Eq. (8) should be verified empirically; for example, report how often an aligning column actually exists in the measured down-projection tokens. Without this check, the mathematical explanation is a hypothesis rather than a validated mechanism.
minor comments (4)
  1. [Section IV-C] The phrase "By substituting the formula (3) with (4)" is confusing; it should read "substituting Eq. (4) into Eq. (3)" or similar.
  2. [Section III-D] The statement that the constructed Hadamard columns "have a mean of 0" is not correct for the first column of the Sylvester construction, which is all +1; please rephrase to "all but the first column" or explain why the first column does not affect the argument.
  3. [Section IV-B] The reported correlation "over 0.97" is not accompanied by a numerical coefficient, a scatter plot, or a statement of the fitting procedure; please report the Pearson or Spearman r with and without the excluded layers and justify the exclusions.
  4. [Sections V and VI] The limitations section appropriately restricts the practical recommendation to down-projection layers, but the Conclusion states the method "consistently offers the lowest errors in other cases as well" without repeating the calibration caveat; please align the two sections.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the hybrid method's gains are measured directly with a fixed alpha; the analytical explanation is approximate and not fitted to the outcome.

full rationale

The paper's central claim is empirical: applying SmoothQuant-style per-channel scaling with a fixed alpha of 0.5 before a Hadamard rotation lowers the measured 4-bit layer-wise squared Frobenius quantization error in LLaMA2-7B (Fig. 4). No parameter is fitted to the target error: alpha is taken from SmoothQuant, the Hadamard matrix is fixed, and the scaling factors are computed online from the current activations. This online, in-sample computation is a validity limitation rather than a circular reduction, because the reported error is a direct measurement, not a prediction derived from the fitted scales. The paper itself notes the calibration dependence in Section V. The quantization-difficulty metric (standard deviation of channel magnitudes) is descriptive and used for visualization and correlation, not as an input that defines the quantization error. Equation (9) is an approximate explanatory formula, not fitted to data, and is not equivalent to its assumptions by construction; it does contain a normalization slip relative to Eq. (8) (d vs. sqrt(d)), which is a correctness concern, not circularity. There are no load-bearing self-citations: the authors cite external work for SmoothQuant, QuaRot, and QuIP#, and the proposed combination is evaluated directly. Overall, the derivation chain is self-contained and the score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the layer-wise error proxy, on the simplified outlier token model, and on the availability of balanced Hadamard matrices. None of these is independently verified beyond the single-experiment setup; alpha is an inherited hyperparameter, not a fitted value. No new physical entity or mediator is introduced.

free parameters (1)
  • alpha (migration strength) = 0.5 (fixed, from SmoothQuant, not tuned)
    Controls how much quantization difficulty is shifted from activations to weights. Sec IV-C notes per-module optimal values around 0.7 (out proj) and 0.65 (gate proj), so the fixed value is a hand-set hyperparameter affecting all comparison results.
assumptions (5)
  • domain assumption Layer-wise squared Frobenius error (Eq 2) is a sufficient proxy for practical quantized-model quality; end-to-end metrics such as perplexity are not required.
    All recommendations are based on layer-wise error alone; the authors state in Sec V that they did not measure perplexity.
  • standard math Symmetric uniform quantization noise is uniformly distributed in [-Delta/2, Delta/2] with variance Delta^2/12.
    Used in Sec II-B to argue that reducing tensor norms and step sizes reduces layer-wise quantization error.
  • ad hoc to paper The token model of Eq (6): massive outliers at a fixed small set of dimensions O and independent Gaussian noise elsewhere captures real LLM activation structure.
    Introduced in Sec IV-D to derive Eqs (7)-(9); it ignores correlations and non-Gaussian structure in real activations.
  • domain assumption Hadamard rotation matrices used here, including the 172x172 block for d=11008, have columns with near-zero mean and contain the sign-alignment pattern required by Eq (8).
    Stated in Sec III-D; the first column is all +1, so the zero-mean property has exceptions that are waved aside, and existence of the 172x172 Hadamard matrix is assumed from QuIP#.
  • domain assumption SmoothQuant's alpha=0.5 formula (Eq 4) is a valid fixed smoothing transformation; no calibration search is needed.
    The paper applies smoothing online with alpha=0.5 and acknowledges that this may be suboptimal in some modules (Sec IV-C).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Turning LLM Activations Quantization-Friendly." pith.science (2026). https://pith.science/paper/NCUQNQMQ

@misc{pith2026250601967,
  author       = {Pith},
  title        = {Pith review of: Turning LLM Activations Quantization-Friendly},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCUQNQMQ}},
  note         = {Machine review of arXiv:2506.01967}
}
read the original abstract

Quantization effectively reduces the serving costs of Large Language Models (LLMs) by speeding up data movement through compressed parameters and enabling faster operations via integer arithmetic. However, activating integer arithmetic requires quantizing both weights and activations, which poses challenges due to the significant outliers in LLMs that increase quantization error. In this work, we investigate these outliers with an emphasis on their effect on layer-wise quantization error, then examine how smoothing and rotation transform the observed values. Our primary contributions include introducing a new metric to measure and visualize quantization difficulty based on channel magnitudes, as well as proposing a hybrid approach that applies channel-wise scaling before rotation, supported by a mathematical formulation of its benefits.

Figures

Figures reproduced from arXiv: 2506.01967 by the authors.

Figure 1
Figure 1. Input activation magnitudes measured at the second key projection layer (k [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Input activation magnitudes measured at the second to last down projection layer (down [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Layer-wise statistics measured at different modules of LLaMA2-7B. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Layer-wise statistics measured at down projection layers of LLaMA2-7B with different transformations applied. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Absolute value distribution and effective quantization bins in massive [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 14 canonical work pages

  1. [1]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    H. Touvron, L. Martin, K. Stone, et al. “Llama 2: Open Foundation and Fine-Tuned Chat Models.” arXiv: 2307. 09288 [cs]. (Jul. 19, 2023), pre-published

    work page 2023

  2. [2]

    The Llama 3 Herd of Models

    A. Grattafiori, A. Dubey, A. Jauhri, et al. “The Llama 3 Herd of Models.” arXiv: 2407.21783 [cs]. (Nov. 23, 2024), pre-published

    arXiv 2024

  3. [3]

    Training Experimental Language Models with Low Resources, for the Hungar- ian Language,

    Z. G. H. Yang and T. V ´aradi, “Training Experimental Language Models with Low Resources, for the Hungar- ian Language,” Acta Polytechnica Hungarica , vol. 20, no. 5, 2023

    work page 2023

  4. [4]

    Fake News Detection System, based on CBOW and BERT,

    T. H. V o, I. Felde, and K. C. Ninh, “Fake News Detection System, based on CBOW and BERT,” Acta Polytechnica Hungarica, vol. 22, no. 1, 2025

    work page 2025

  5. [5]

    Model Compression and Efficient Inference for Large Language Models: A Survey

    W. Wang, W. Chen, Y . Luo, et al. “Model Compression and Efficient Inference for Large Language Models: A Survey.” arXiv: 2402 . 09748 [cs]. (Feb. 15, 2024), pre-published

    work page 2024

  6. [6]

    A Survey on Model Compression for Large Language Models

    X. Zhu, J. Li, Y . Liu, C. Ma, and W. Wang. “A Survey on Model Compression for Large Language Models.” arXiv: 2308.07633v4. (Jul. 30, 2024), pre-published

    arXiv 2024

  7. [7]

    Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Only Inference

    B. Jacob, S. Kligys, B. Chen, et al. “Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Only Inference.” arXiv: 1712.05877 [cs, stat]. (Dec. 15, 2017), pre-published

    arXiv 2017

  8. [8]

    LLM.int8(): 8-bit Matrix Multiplication for Transform- ers at Scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transform- ers at Scale,” presented at the Advances in Neural Information Processing Systems, vol. 35, 2022

    work page 2022

Show all 25 references
  1. [9]

    SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models,” presented at the Proceedings of Machine Learning Re- search, vol. 202, 2023, pp. 38 087–38 099

    2023

  2. [10]

    Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs

    J. Yang, H. Kim, and Y . Kim. “Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs.” arXiv: 2405.14428v1. (May 23, 2024), pre-published

    2024 arXiv

  3. [11]

    DuQuant: Distribut- ing Outliers via Dual Transformation Makes Stronger Quantized LLMs

    H. Lin, H. Xu, Y . Wu, et al. “DuQuant: Distribut- ing Outliers via Dual Transformation Makes Stronger Quantized LLMs.” arXiv: 2406 . 01721v2. (Sep. 24, 2024), pre-published

    2024

  4. [12]

    Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

    Y . Zhao, C.-Y . Lin, K. Zhu, et al. “Atom: Low-bit Quantization for Efficient and Accurate LLM Serving.” arXiv: 2310.19102v3. (Apr. 16, 2024), pre-published

    2024 arXiv

  5. [13]

    QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

    S. Ashkboos, A. Mohtashami, M. L. Croci, et al. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.” arXiv: 2404 . 00456v2. (Oct. 29, 2024), pre- published

    2024

  6. [14]

    OmniQuant: Omnidirectionally calibrated quantization for large lan- guage models,

    W. Shao, M. Chen, Z. Zhang, et al. , “OmniQuant: Omnidirectionally calibrated quantization for large lan- guage models,” presented at the 12th International Con- ference on Learning Representations, ICLR 2024, 2024

    2024

  7. [15]

    SpinQuant: LLM quantization with learned rotations

    Z. Liu, C. Zhao, I. Fedorov, et al. “SpinQuant: LLM quantization with learned rotations.” arXiv: 2405 . 16406v3. (Oct. 7, 2024), pre-published

    2024

  8. [16]

    SPQR: A sparse-quantized representation for near- lossless llm weight compression,

    T. Dettmers, R. Svirschevski, V . Egiazarian, et al. , “SPQR: A sparse-quantized representation for near- lossless llm weight compression,” presented at the 12th International Conference on Learning Representations, ICLR 2024, 2024

    2024

  9. [17]

    SqueezeLLM: Dense-and-Sparse Quantization,

    S. Kim, C. Hooper, A. Gholami, et al. , “SqueezeLLM: Dense-and-Sparse Quantization,” presented at the Pro- ceedings of Machine Learning Research, vol. 235, 2024, pp. 23 901–23 923

    2024

  10. [18]

    QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks,

    A. Tseng, J. Chee, Q. Sun, V . Kuleshov, and C. De Sa, “QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks,” presented at the Proceedings of Machine Learning Research, vol. 235, 2024, pp. 48 630–48 656

    2024

  11. [19]

    EfficientQAT: Effi- cient Quantization-Aware Training for Large Language Models

    M. Chen, W. Shao, P. Xu, et al. “EfficientQAT: Effi- cient Quantization-Aware Training for Large Language Models.” arXiv: 2407 . 11062v2. (Oct. 2, 2024), pre- published

    2024

  12. [20]

    OPTQ: Accurate quantization for generative pre- trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alis- tarh, “OPTQ: Accurate quantization for generative pre- trained transformers,” presented at the 11th International Conference on Learning Representations, ICLR 2023, 2023

    2023

  13. [21]

    FlatQuant: Flatness Matters for LLM Quantization

    Y . Sun, R. Liu, H. Bai, et al. “FlatQuant: Flatness Matters for LLM Quantization.” arXiv: 2410.09426v1. (Oct. 12, 2024), pre-published

    2024 arXiv

  14. [22]

    Pointer Sentinel Mixture Models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer Sentinel Mixture Models,” presented at the International Conference on Learning Representations, Feb. 6, 2017

    2017

  15. [23]

    Transformers: State- of-the-Art Natural Language Processing,

    T. Wolf, L. Debut, V . Sanh, et al., “Transformers: State- of-the-Art Natural Language Processing,” in Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Q. Liu and D. Schlangen, Eds., Online: Association for Compu...

    2020 doi

  16. [24]

    PyTorch: An im- perative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, et al., “PyTorch: An im- perative style, high-performance deep learning library,” in Proceedings of the 33rd International Conference on Neural Information Processing Systems , 721, Red Hook, NY , USA: Curran Associates Inc., Dec. 8, 2019, pp. 8026–8037

    2019

  17. [25]

    Mistral 7B

    A. Q. Jiang, A. Sablayrolles, A. Mensch, et al. “Mistral 7B.” arXiv: 2310.06825 [cs]. (Oct. 10, 2023), pre- published

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.