Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the activation spikes ruining 8-bit quantization in LLaMA models are confined to two or three projection layers, and that quantizing only those in FP8 restores near-FP16 quality.

desk verdict A practical, architecture-specific mixed-precision recipe with a clean control experiment, but the fixed layer selection and thin baselines make the general claim about LLaMA-family stability under-supported. read the letter →

arxiv 2504.21553 v1 pith:WHMB4SBN submitted 2025-04-30 cs.CL

Lucas Maisonnave , Cyril Moineau , Olivier Bichler , Fabrice Rastello This is my paper · ORCID
classification cs.CL
keywords largelanguagemodelsquantizationactivationspikesmixedprecisionper-tensorFP8LLaMApost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the activation outliers that crash 8-bit per-tensor quantization in LLaMA-style models are not a model-wide problem. They are concentrated in two or three projection layers: the first and last MLP down projections, plus in some models the final attention output projection. Keeping only those projections in FP16 or FP8 while quantizing the rest of the model to 8-bit integers gives near-FP16 perplexity and large zero-shot accuracy gains across LLaMA2, LLaMA3, and Mistral. For LLaMA3-8B, per-tensor perplexity on WikiText2 falls from 40.45 (naive) and 44.74 (SmoothQuant) to 8.24, against 6.14 for FP16. The paper argues that architecture-specific quantization beats general-purpose outlier-handling methods.

What carries the argument

The carrier of the argument is a projection-level spike map: a table of per-layer, per-projection maximum absolute activation values, computed with the model in FP16 on calibration text, which identifies exactly which linear projections host the spikes that inflate the quantization scale. The recipe built on it is a static mixed-precision assignment: the spike projections run in FP16 or FP8 E5M2 (chosen because E5M2's maximum representable value of 57344 exceeds the observed spikes, which reach about 2500), while all other linear layers are quantized symmetrically per tensor to 8-bit integers. The random-control experiment is what rules out the alternative that any high-precision layers would help: random FP16 placement recovers none of the performance, so the location is what matters.

What would settle it

Run the paper's fixed layer recipe on a LLaMA-family model outside its table, for instance LLaMA3-70B, and measure 8-bit per-tensor WikiText2 perplexity against FP16; if any projection outside the chosen two or three shows comparable max-absolute spikes, or if the perplexity gap stays large, the locality claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that in LLaMA-style transformers the activation values that destroy per-tensor quantization are not distributed across the model: they appear as spikes in the MLP down projection of the first and last layers, specifically layers 2 and 32 for LLaMA3-8B, and additionally the attention output projection of the final layer for LLaMA2 and Mistral models. A per-tensor symmetric quantizer with scale $\Delta = \max|X|/(2^{b-1}-1)$ is dominated by a single spike, yet the paper shows that leaving just those two or three projections in FP16 or FP8 E5M2 and quantizing everything else to 8-bit integers recovers near-FP16 perplexity. The evidence includes max-absolute-value traces across layers, a random-layer control (random FP16 on the same number of projections gives perplexity 39.76 versus 8.24 for targeted placement on LLaMA3-8B), and tables across four models showing 8-bit per-tensor PPL of 8.24 (LLaMA3-8B), 6.27 (LLaMA2-7B), 8.38 (LLaMA2-13B), and 10.14 (Mistral-7B) against FP16 values around 5 to 6.

Load-bearing premise

The load-bearing premise is that the spike-bearing projections found in the four tested models are the same two or three projections in every LLaMA-style model, so the fixed recipe transfers without per-model re-detection.

Editorial extensions

If this is right

  • Eight-bit per-tensor quantization becomes deployable for LLaMA-family models with only two or three FP16 or FP8 projections, with no specialized outlier quantizer required.
  • On LLaMA3-8B, per-tensor perplexity drops from 40.45 (naive) and 44.74 (SmoothQuant) to 8.24, and zero-shot accuracy rises from about 51.5 to 65.5, approaching FP16 baselines.
  • FP8 E5M2 on the spike layers matches FP16 in most settings, so the recipe can ride on hardware with fast FP8 support.
  • Randomizing the placement of high precision does not recover the lost quality, meaning the gains come from hitting the specific spike-bearing projections rather than from extra precision anywhere.
  • The same recipe makes 6-bit per-tensor quantization usable, though with some instability, so the spike-localization idea extends to tighter bit budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The spike-to-layer map is probably a training artifact, so the fixed recipe should be re-derived on each new checkpoint rather than assumed; the paper does not test a LLaMA model outside its four-model table.
  • The BOT-token result hints that part of the spike is an input-format effect; feeding a long prefix or a different beginning-of-text token before quantization could shrink the spike without mixed precision.
  • The method suggests a cheap diagnostic for any LLaMA-like model: one forward pass plotting per-layer max-abs activations tells you which two or three layers need FP8, turning a one-size-fits-all quantizer into a calibration-by-architecture scheme.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a post-training mixed-precision quantization strategy for LLaMA-family LLMs. Based on an analysis of activation magnitudes in LLaMA2-7B/13B, LLaMA3-8B, and Mistral-7B, the authors claim that activation spikes are concentrated in a small number of projection layers (mostly the first and last MLP down projections, plus the attention output projection for some models). Their method keeps those selected layers in FP16 or FP8 while quantizing all other linear layers to 8-bit or 6-bit per-tensor uniform quantization. The paper reports WikiText2 perplexity and seven zero-shot task accuracies, comparing against SmoothQuant and naive uniform quantization, and includes a random-projection control that shows targeted layer selection matters.

Significance. If the spike-localization claim holds beyond the four tested checkpoints, the method is a simple and practical way to improve 8-bit per-tensor quantization for LLaMA-like models without specialized outlier quantizers. The random-projection ablation in Table 5 is a valuable control because it directly supports the claim that the specific choice of layers, rather than the mere addition of FP16 capacity, is responsible for the perplexity gains. The modeling of the problem as architecture-specific rather than universal is also a useful framing. However, the paper's central generalization claim is not yet fully supported: the layer indices in Table 2 vary across the four models, and there is no held-out LLaMA derivative or fully specified calibration protocol to show the spike-to-layer mapping is stable.

major comments (4)
  1. [§4.1, Table 2; §3.4] The central claim that spike locations are a stable property of the LLaMA architecture is not established. Table 2 itself shows the selected layers vary across the four models: the first down_proj is layer 2 for LLaMA3-8B, LLaMA2-7B, and Mistral-7B but layer 4 for LLaMA2-13B; the out_proj is absent for LLaMA3-8B but present at layers 32, 40, and 32 for the other models; and Mistral requires an extra down_proj at layer 32. Section 3.4 explicitly attributes spike emergence to training hyperparameters and notes that Mistral, despite being LLaMA-based, behaves differently. As presented, the method is a hardcoded list for four checkpoints rather than a procedure for locating spike layers in a new model, so the claimed transfer to other LLaMA derivatives is unsupported. Please add a held-out experiment on a different LLaMA-family checkpoint (for example LLaMA-3.1-8B, a fine-tuned model, or a different context length), or provide and validate a fully specified per-model spike-localization protocol that does not use the evaluation set.
  2. [§4.1, Figures 3–6; §5.1] The text does not state which corpus is used to compute the max-abs activation maps that lead to Table 2 and the layer selections. If the same WikiText2 test sequence is used both to select the FP16/FP8 layers and to report perplexity, the layer choice is calibrated on the evaluation data, making the comparisons in Tables 3–6 circular. Please state the calibration text, its length, and confirm that it is disjoint from the WikiText2 evaluation set; if it is not, repeat the selection on a separate calibration set.
  3. [Tables 3 and 6] The same condition, '8 per_tensor Mix', reports different perplexities in the two tables: LLaMA2-13B has 8.38 in Table 3 but 8.31 in Table 6, and Mistral-7B has 10.14 in Table 3 but 9.97 in Table 6. In addition, the standalone 'FP8' row of Table 6 is undefined in the caption, and the per-token Mix+FP8 result for LLaMA3-8B (5.32) is below the FP16 perplexity (6.136), which is surprising and needs an explanation such as a typo, a different evaluation setup, or a documented regularization effect. Please reconcile the numbers and define each row of Table 6 precisely.
  4. [§5.2, Tables 3–4] The abstract and conclusion claim the proposed method 'outperforms general-purpose methods' designed for outliers, but the only general-purpose baseline is SmoothQuant. This baseline set is too thin to support the comparative claim, especially because the paper itself cites LLM.int8(), per-channel/group quantization, and other outlier-aware methods. Please add at least one additional strong baseline, for example LLM.int8() or per-channel/group post-training quantization, and report the same metrics for that baseline.
minor comments (6)
  1. [Table 6] The header 'Bits Type Mix/FP8' and the standalone 'FP8' row need a caption or footnote explaining what is quantized in each row; as printed, a reader cannot tell whether 'FP8' means full-model FP8 or FP8 only for the retained spike projections.
  2. [§4.1] The threshold of 100 for deciding whether to keep an out projection in FP16 is introduced without sensitivity analysis or a rationale; please report how the results change when the threshold is varied.
  3. [Table 1] The 'W8A8 w/o BOT' condition should specify operationally how a token is excluded from quantization, for example computed in FP16 and re-injected, and whether this is applied at every layer.
  4. [§3.3] The statement that spikes are 'generated by the down projection of the second layer' conflicts with Table 2, where the early down projection is at layer 2 for three models but layer 4 for LLaMA2-13B; please rephrase as 'early layer' or make the layer indexing explicit.
  5. [Table 5] The random mixed-precision control should state the exact random sampling procedure, including which layers are candidates, how many are kept, and the three seeds used, because the comparison is only interpretable if the random draws have the same budget as the targeted selection.
  6. [Throughout] Minor wording and typographical issues: 'consummation' should be 'consumption' (Section 1), 'Langage' should be 'Language' in the keywords, and 'teqCO2' should be 'tCO2e' or defined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: spike localization is empirical, validated by a random-projection control, and no load-bearing derivation reduces to its inputs.

full rationale

The paper makes no formal derivation: it observes activation spikes in specific projections of four LLaMA-family models, selects those layers for FP16/FP8, and reports perplexity. The only possible circularity would be if the reported perplexity were forced by the way the layers were chosen, but the paper provides an independent control: Table 5 shows that randomly chosen down/out projections do not reproduce the gains, so the targeted selection is not a tautology. The method is calibrated on the same model architectures on which it is evaluated, and the calibration text for the spike figures is not specified, so generalization to unseen LLaMA variants is a robustness concern rather than a logical reduction. Section 3.4 even concedes that training hyperparameters alter spike emergence and that Mistral behaves differently from LLaMA despite architectural similarity, which weakens the universality claim but does not make it circular. There are no self-citations, no imported uniqueness theorems, and no fitted parameter renamed as a prediction. The quantization equation (1) and the E5M2/E4M3 choice are standard, externally grounded components. The paper's central result is an empirical finding that keeping two or three spiking projections at higher precision preserves perplexity, supported by the random ablation; this is self-contained measurement rather than derivation-from-assumption.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard quantization formulas, an empirical spike-localization observation, and the untested transfer of the layer selection to other LLaMA models. The only hand-fitted numeric value is the 100-threshold for the out projection; the layer indices are discrete choices made from inspection.

free parameters (2)
  • Spike threshold for out projection FP16 retention = 100 (absolute activation value)
    Section 4.1: 'we decided to keep this specific projection in full precision when spike values exceed 100'. This threshold is chosen by inspecting the max-abs plots and is not derived from theory.
  • Layer selections for mixed precision (down/out projections per model) = Table 2: e.g., LLaMA3-8B down 2,32; Mistral-7B down 2,31,32 plus out 32
    The specific layers kept in high precision are selected by visual inspection of per-layer max-abs activation plots for each model (Figures 3-6, Appendix A). They are effectively hand-picked constants that define the method.
assumptions (5)
  • standard math Symmetric uniform quantization with absolute-maximum range (Eq. 1)
    Used throughout as the quantization scheme; standard in PTQ literature.
  • domain assumption The absolute maximum activation value determines the quantization scale for each tensor
    Standard for per-tensor quantization; can be suboptimal for skewed distributions but is the baseline assumption.
  • domain assumption Activation spike patterns observed on calibration text generalize to evaluation text
    The max-abs plots (Figures 3-6) are based on some input sample; the method assumes these locations are stable across WikiText2 and zero-shot tasks.
  • ad hoc to paper The selected layer indices in Table 2 transfer to other LLaMA-family models
    The paper does not test the recipe on a LLaMA model not used in the analysis; this is the paper's own generalization assumption.
  • domain assumption RMSNorm reduces spike amplitude in LLaMA architectures
    Empirical claim from Section 3.4/Figures 3-6; used to argue spikes stay confined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models." pith.science (2026). https://pith.science/paper/WHMB4SBN

@misc{pith2026250421553,
  author       = {Pith},
  title        = {Pith review of: Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WHMB4SBN}},
  note         = {Machine review of arXiv:2504.21553}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks. However, their size presents significant challenges for deployment and inference. This paper investigates the quantization of LLMs, focusing on the LLaMA architecture and its derivatives. We challenge existing assumptions about activation outliers in LLMs and propose a novel mixed-precision quantization approach tailored for LLaMA-like models. Our method leverages the observation that activation spikes in LLaMA architectures are predominantly concentrated in specific projection layers. By applying higher precision (FP16 or FP8) to these layers while quantizing the rest of the model to lower bit-widths, we achieve superior performance compared to existing quantization techniques. Experimental results on LLaMA2, LLaMA3, and Mistral models demonstrate significant improvements in perplexity and zero-shot accuracy, particularly for 8-bit per-tensor quantization. Our approach outperforms general-purpose methods designed to handle outliers across all architecture types, highlighting the benefits of architecture-specific quantization strategies. This research contributes to the ongoing efforts to make LLMs more efficient and deployable, potentially enabling their use in resource-constrained environments. Our findings emphasize the importance of considering model-specific characteristics in developing effective quantization pipelines for state-of-the-art language models by identifying and targeting a small number of projections that concentrate activation spikes.

Figures

Figures reproduced from arXiv: 2504.21553 by the authors.

Figure 1
Figure 1. Causal architecture of OPT and LLaMA model. There are 2 main differ [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Layer 19 activations values of (a) the first RMSNorm’s input in a [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Maximum absolute value over layers for a LLaMA3-8B. Each color rep [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Maximum absolute value over layers for a LLaMA2-7B. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Maximum absolute value over layers for a LLaMA2-13B. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Maximum absolute value over layers for a Mistral-7B. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs

    cs.LG 2025-04 conditional novelty 6.0 of 10

    Gradual Binary Search over per-projection clipping ratios, combined with Hadamard rotations and dimension expansion, enables 3-bit WAKV quantization with better benchmark accuracy than QuaRot.

Reference graph

Works this paper leans on

21 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Advances in Neural Information Processing Systems36, 34278–34294 (2023)

    Ahmadian, A., Dash, S., Chen, H., Venkitesh, B., Gou, Z.S., Blunsom, P., Üstün, A., Hooker, S.: Intriguing properties of quantization at scale. Advances in Neural Information Processing Systems36, 34278–34294 (2023)

    work page 2023

  2. [2]

    arXiv preprint arXiv:2109.12948 (2021)

    Bondarenko, Y., Nagel, M., Blankevoort, T.: Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109.12948 (2021)

    arXiv 2021

  3. [3]

    int8 (): 8-bit matrix multiplicationfortransformersatscale.AdvancesinNeuralInformationProcessing Systems 35, 30318–30332 (2022)

    Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L.: Gpt3. int8 (): 8-bit matrix multiplicationfortransformersatscale.AdvancesinNeuralInformationProcessing Systems 35, 30318–30332 (2022)

    2022

  4. [4]

    Advances in Neural Information Processing Systems36 (2024)

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient fine- tuning of quantized llms. Advances in Neural Information Processing Systems36 (2024)

    2024

  5. [5]

    In: International Conference on Machine Learning

    Du, N., Huang, Y., Dai, A.M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A.W., Firat, O., et al.: Glam: Efficient scaling of language models with mixture-of-experts. In: International Conference on Machine Learning. pp. 5547–

    work page

  6. [6]

    arXiv preprint arXiv:2210.17323 (2022)

    Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D.: Gptq: Accurate post- training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    arXiv 2022

  7. [7]

    He, B., Noci, L., Paliotta, D., Schlag, I., Hofmann, T.: Understanding and min- imisingoutlierfeaturesinneuralnetworktraining.arXivpreprintarXiv:2405.19279 (2024)

    arXiv 2024

  8. [8]

    HuggingFace: Meta-llama-3-8b (2024), https://huggingface.co/meta-llama/ Meta-Llama-3-8B

    work page 2024

Show all 21 references
  1. [9]

    arXiv preprint arXiv:2310.06825 (2023)

    Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  2. [10]

    Le Scao, T., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., Gallé, M., et al.: Bloom: A 176b-parameter open-access multilingual language model (2023)

    2023

  3. [11]

    arXiv preprint arXiv:2402.17764 (2024)

    Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., Wei, F.: The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764 (2024)

    2024 arXiv

  4. [12]

    arXiv preprint arXiv:2209.05433 (2022)

    Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al.: Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433 (2022)

    2022 arXiv

  5. [13]

    arXiv preprint arXiv:2405.20835 (2024)

    Paglieri, D., Dash, S., Rocktäschel, T., Parker-Holder, J.: Outliers and calibra- tion sets have diminishing effect on quantization of modern llms. arXiv preprint arXiv:2405.20835 (2024)

    2024 arXiv

  6. [14]

    Proceedings of Machine Learning and Sys- tems 6, 483–498 (2024)

    Shen, H., Mellempudi, N., He, X., Gao, Q., Wang, C., Wang, M.: Efficient post- training quantization with fp8 formats. Proceedings of Machine Learning and Sys- tems 6, 483–498 (2024)

    2024

  7. [15]

    arXiv preprint arXiv:2406.12016 (2024)

    Son, S., Park, W., Han, W., Kim, K., Lee, J.: Prefixing attention sinks can mit- igate activation outliers for large language model quantization. arXiv preprint arXiv:2406.12016 (2024)

    2024 arXiv

  8. [16]

    arXiv preprint arXiv:2402.17762 (2024) Precision Where It Matters 15

    Sun, M., Chen, X., Kolter, J.Z., Liu, Z.: Massive activations in large language models. arXiv preprint arXiv:2402.17762 (2024) Precision Where It Matters 15

    2024 arXiv

  9. [17]

    https://doi.org/10.5281/zenodo.10256836, https://doi.org/10

    Sutawika, L., Gao, L., Schoelkopf, H., Biderman, S., Tow, J., Abbasi, B., ben fattori, Lovering, C., farzanehnakhaee70, Phang, J., Thite, A., Fazz, Aflah, Muen- nighoff, N., Wang, T., sdtblck, nopperl, gakada, tttyuntian, researcher2, Chris, Etxaniz, J., Kasner, Z., Khalid, Hs...

    2023 doi

  10. [18]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  11. [19]

    arXiv preprint arXiv:2310.11453 (2023)

    Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y., Wei, F.: Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453 (2023)

    2023 arXiv

  12. [20]

    In: International Conference on Machine Learning

    Xiao,G.,Lin,J.,Seznec,M.,Wu,H.,Demouth,J.,Han,S.:Smoothquant:Accurate and efficient post-training quantization for large language models. In: International Conference on Machine Learning. pp. 38087–38099. PMLR (2023)

    2023

  13. [21]

    arXiv preprint arXiv:2405.14428 (2024)

    Yang, J., Kim, H., Kim, Y.: Mitigating quantization errors due to activation spikes in glu-based llms. arXiv preprint arXiv:2405.14428 (2024)

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.