{"id":"8f2a2cae-27b7-4607-ab95-73e7ebaa26e5","arxiv_id":"2504.21553","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Keeping two or three spike-heavy projections in FP16 or FP8 while quantizing the rest to 8 bits improves LLaMA-family quantization over SmoothQuant in the per-tensor regime.","lead":"The paper finds that in LLaMA-family language models, activation spikes during quantization appear mainly in a few specific projection layers, and proposes keeping those layers in higher precision while quantizing the rest to 8 bits. This simple recipe improves perplexity and zero-shot accuracy over SmoothQuant for 8-bit per-tensor quantization on several LLaMA and Mistral models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Layer-selection is fixed after four models and one calibration text; no adaptive mechanism or held-out LLaMA variant is tested, so the claimed architecture-level stability of spike locations is unestablished.","rationale":"The reader's verdict (CONDITIONAL) seems right. The paper has real strengths: it documents a reproducible spike pattern, includes a control (random layers) that supports specificity, and reports results for four models with consistent trends. However, the central claim -- that the spike locations are an architectural property of LLaMA-like models -- is the load-bearing premise for the method's transferability. The paper's own Section 3.4 undermines this by stating that training hyperparameters and model lineage change outlier behavior. Table 2 shows the layer indices are not uniformly 'first and last' (first down is 2 vs 4; Mistral adds layers). The method is a hardcoded list, not an adaptive selection algorithm, and the calibration text used to build the figures is unspecified. If a user applies the table to a new checkpoint without re-running spike detection, there is no guarantee the right layers are protected. I considered the suspicious SmoothQuant baselines (which are sometimes worse than naive) as an alternative concern, but that affects the comparative claim, not the internal mechanism; the most load-bearing issue is whether the recipe generalizes. A concrete test on a held-out LLaMA variant with disjoint calibration/eval text would settle it. If the fixed recipe fails that test, the paper's 'architecture-specific' claim must be weakened to 'checkpoint-specific'. The verdict remains CONDITIONAL pending this robustness check.","tokens_in":9566,"tokens_out":6969,"duration_ms":66343,"concrete_test":"Take a held-out LLaMA-family checkpoint not in Table 2 (e.g., Meta-Llama-3.1-8B-Instruct or Llama-2-70B) and two disjoint text sources: a calibration set (e.g., 128 C4 sequences) and the WikiText2 test set. First, compute per-projection max-abs activations on the calibration set and on the test set; if the ranking of top-spike layers changes between the two sources, the fixed recipe is not input-robust. Second, evaluate W8A8 per-tensor PPL on WikiText2 under (a) the nearest Table 2 recipe (e.g., down 2/32 for a Llama-3.x model) and (b) an adaptive recipe selecting layers from the calibration-set spike ranking; if (a) has PPL more than 2x higher than (b), the fixed recipe does not transfer to a new checkpoint, confirming the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim treats the spike-to-layer mapping as an architectural invariant, but the paper's own evidence undermines that. Table 2 shows the 'same' pattern is not identical: the first spiking down_proj is layer 2 for LLaMA3-8B, LLaMA2-7B, and Mistral-7B but layer 4 for LLaMA2-13B; the last down_proj is 32, 31, 39, varying with depth; and Mistral requires an extra down (32) plus an attention output projection. Section 3.4 explicitly states that training hyperparameters (weight decay, gradient clipping, encoding format, dropout) change outlier emergence, and that Mistral differs from LLaMA despite sharing the architecture. The proposed method (Section 4.1) is not a procedure for locating spikes on a new checkpoint; it is a hardcoded list of layers for four models. The figures used to build Table 2 (Figures 3-6 and Appendix A) do not state which text produced the activations or whether it is disjoint from the WikiText2 test set, so it is possible the layer choices are calibrated on the same text they are evaluated on. If the mapping shifts with training run, checkpoint size, or input distribution, applying the fixed recipe to a new LLaMA derivative (e.g., LLaMA-3.1-8B, a fine-tuned model, or a longer context) can select the wrong layers and the near-FP16 perplexity claim fails. There is no ablation showing the recipe is robust to these shifts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-training mixed-precision quantization strategy for LLaMA-family LLMs. Based on an analysis of activation magnitudes in LLaMA2-7B/13B, LLaMA3-8B, and Mistral-7B, the authors claim that activation spikes are concentrated in a small number of projection layers (mostly the first and last MLP down projections, plus the attention output projection for some models). Their method keeps those selected layers in FP16 or FP8 while quantizing all other linear layers to 8-bit or 6-bit per-tensor uniform quantization. The paper reports WikiText2 perplexity and seven zero-shot task accuracies, comparing against SmoothQuant and naive uniform quantization, and includes a random-projection control that shows targeted layer selection matters.","tokens_in":9896,"tokens_out":6274,"duration_ms":61371,"significance":"If the spike-localization claim holds beyond the four tested checkpoints, the method is a simple and practical way to improve 8-bit per-tensor quantization for LLaMA-like models without specialized outlier quantizers. The random-projection ablation in Table 5 is a valuable control because it directly supports the claim that the specific choice of layers, rather than the mere addition of FP16 capacity, is responsible for the perplexity gains. The modeling of the problem as architecture-specific rather than universal is also a useful framing. However, the paper's central generalization claim is not yet fully supported: the layer indices in Table 2 vary across the four models, and there is no held-out LLaMA derivative or fully specified calibration protocol to show the spike-to-layer mapping is stable.","major_comments":[{"comment":"The central claim that spike locations are a stable property of the LLaMA architecture is not established. Table 2 itself shows the selected layers vary across the four models: the first down_proj is layer 2 for LLaMA3-8B, LLaMA2-7B, and Mistral-7B but layer 4 for LLaMA2-13B; the out_proj is absent for LLaMA3-8B but present at layers 32, 40, and 32 for the other models; and Mistral requires an extra down_proj at layer 32. Section 3.4 explicitly attributes spike emergence to training hyperparameters and notes that Mistral, despite being LLaMA-based, behaves differently. As presented, the method is a hardcoded list for four checkpoints rather than a procedure for locating spike layers in a new model, so the claimed transfer to other LLaMA derivatives is unsupported. Please add a held-out experiment on a different LLaMA-family checkpoint (for example LLaMA-3.1-8B, a fine-tuned model, or a different context length), or provide and validate a fully specified per-model spike-localization protocol that does not use the evaluation set.","section":"§4.1, Table 2; §3.4"},{"comment":"The text does not state which corpus is used to compute the max-abs activation maps that lead to Table 2 and the layer selections. If the same WikiText2 test sequence is used both to select the FP16/FP8 layers and to report perplexity, the layer choice is calibrated on the evaluation data, making the comparisons in Tables 3–6 circular. Please state the calibration text, its length, and confirm that it is disjoint from the WikiText2 evaluation set; if it is not, repeat the selection on a separate calibration set.","section":"§4.1, Figures 3–6; §5.1"},{"comment":"The same condition, '8 per_tensor Mix', reports different perplexities in the two tables: LLaMA2-13B has 8.38 in Table 3 but 8.31 in Table 6, and Mistral-7B has 10.14 in Table 3 but 9.97 in Table 6. In addition, the standalone 'FP8' row of Table 6 is undefined in the caption, and the per-token Mix+FP8 result for LLaMA3-8B (5.32) is below the FP16 perplexity (6.136), which is surprising and needs an explanation such as a typo, a different evaluation setup, or a documented regularization effect. Please reconcile the numbers and define each row of Table 6 precisely.","section":"Tables 3 and 6"},{"comment":"The abstract and conclusion claim the proposed method 'outperforms general-purpose methods' designed for outliers, but the only general-purpose baseline is SmoothQuant. This baseline set is too thin to support the comparative claim, especially because the paper itself cites LLM.int8(), per-channel/group quantization, and other outlier-aware methods. Please add at least one additional strong baseline, for example LLM.int8() or per-channel/group post-training quantization, and report the same metrics for that baseline.","section":"§5.2, Tables 3–4"}],"minor_comments":[{"comment":"The header 'Bits Type Mix/FP8' and the standalone 'FP8' row need a caption or footnote explaining what is quantized in each row; as printed, a reader cannot tell whether 'FP8' means full-model FP8 or FP8 only for the retained spike projections.","section":"Table 6"},{"comment":"The threshold of 100 for deciding whether to keep an out projection in FP16 is introduced without sensitivity analysis or a rationale; please report how the results change when the threshold is varied.","section":"§4.1"},{"comment":"The 'W8A8 w/o BOT' condition should specify operationally how a token is excluded from quantization, for example computed in FP16 and re-injected, and whether this is applied at every layer.","section":"Table 1"},{"comment":"The statement that spikes are 'generated by the down projection of the second layer' conflicts with Table 2, where the early down projection is at layer 2 for three models but layer 4 for LLaMA2-13B; please rephrase as 'early layer' or make the layer indexing explicit.","section":"§3.3"},{"comment":"The random mixed-precision control should state the exact random sampling procedure, including which layers are candidates, how many are kept, and the three seeds used, because the comparison is only interpretable if the random draws have the same budget as the targeted selection.","section":"Table 5"},{"comment":"Minor wording and typographical issues: 'consummation' should be 'consumption' (Section 1), 'Langage' should be 'Language' in the keywords, and 'teqCO2' should be 'tCO2e' or defined.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about fixed layer selection lands: the paper itself shows variation in Table 2 and acknowledges training-dependent behavior in §3.4, so the generalization claim needs a held-out model. The Table 3 vs Table 6 inconsistency is also a concrete reproducibility problem that should be fixed before publication. If the authors add a disjoint calibration protocol and one held-out LLaMA variant, the contribution would be a solid practical result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear R.,\n\nThe one thing to know: this paper has a genuinely useful empirical observation—in the four LLaMA-like models they test, the activation spikes that break 8-bit per-tensor quantization sit in two or three specific projections (mostly the first and last MLP down projections, sometimes the attention output projection). Keeping just those layers in FP16 (or FP8) while quantizing the rest per-tensor to 8 bits brings perplexity close to FP16 on WikiText2. Their random-layer control (Table 5) is the right experiment, and it lands: random selections give PPL close to naive, targeted ones much better. That alone separates the paper from a laundry-list of quantization tricks.\n\nWhat is actually new: the per-model mapping of spike locations in LLaMA-family models, and the demonstration that a tiny number of layers need special treatment. The quantization itself is standard absmax, as they acknowledge. The paper honestly reviews prior outlier claims and shows some don't reproduce on LLaMA.\n\nWhere it is soft, in proportion: first, the layer choice is hand-picked per model after inspecting activation plots. There is no procedure for a new checkpoint, and the paper itself says training details (weight decay, clipping, etc.) change outlier emergence and that Mistral differs despite sharing the LLaMA base. The stress-test worry that the mapping might shift for LLaMA-3.1, a fine-tuned model, or different input lengths is real and untested. Second, the baselines are thin: SmoothQuant and naive only. No comparison to per-channel or per-group methods, no error bars on the main PPL tables (the random ablation has them; the main numbers don't), and no speed or memory numbers, so the deployment benefit is asserted, not measured. Third, the calibration text used to build the spike plots (Figures 3–6) isn't described; if it overlaps with the WikiText2 test set, the layer choices could be overfit to the eval text. None of these undermine the core observation, but they limit the strength of the \"architecture-level invariant\" framing.\n\nWho should read it: anyone doing post-training quantization for LLaMA-family deployment, and people studying where activation outliers actually live in modern decoders. I'd bring it to reading group. For peer review, yes—it deserves a serious referee despite the soft spots. A competent reviewer can push for a robustness experiment (e.g., a LLaMA-3.1 or fine-tuned variant, and separation of calibration and eval text), and the paper would be stronger for it.\n\nAll the best.","headline":"A practical, architecture-specific mixed-precision recipe with a clean control experiment, but the fixed layer selection and thin baselines make the general claim about LLaMA-family stability under-supported.","tokens_in":10425,"tokens_out":2009,"would_cite":true,"duration_ms":19832,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the activation spikes ruining 8-bit quantization in LLaMA models are confined to two or three projection layers, and that quantizing only those in FP8 restores near-FP16 quality.","keywords":["large language models","quantization","activation spikes","mixed precision","per-tensor quantization","FP8","LLaMA","post-training quantization"],"falsifier":"Run the paper's fixed layer recipe on a LLaMA-family model outside its table, for instance LLaMA3-70B, and measure 8-bit per-tensor WikiText2 perplexity against FP16; if any projection outside the chosen two or three shows comparable max-absolute spikes, or if the perplexity gap stays large, the locality claim fails.","tokens_in":9359,"feed_emoji":"⚡","tokens_out":7326,"duration_ms":67173,"temperature":0.7,"pith_summary":"The paper sets out to show that the activation outliers that crash 8-bit per-tensor quantization in LLaMA-style models are not a model-wide problem. They are concentrated in two or three projection layers: the first and last MLP down projections, plus in some models the final attention output projection. Keeping only those projections in FP16 or FP8 while quantizing the rest of the model to 8-bit integers gives near-FP16 perplexity and large zero-shot accuracy gains across LLaMA2, LLaMA3, and Mistral. For LLaMA3-8B, per-tensor perplexity on WikiText2 falls from 40.45 (naive) and 44.74 (SmoothQuant) to 8.24, against 6.14 for FP16. The paper argues that architecture-specific quantization beats general-purpose outlier-handling methods.","feed_headline":"Two hidden projections break LLaMA's 8-bit quantization","feed_subtitle":"Leaving just the MLP down-projections in FP8 brings per-tensor perplexity from 40 to 8 on LLaMA3-8B.","key_machinery":"The carrier of the argument is a projection-level spike map: a table of per-layer, per-projection maximum absolute activation values, computed with the model in FP16 on calibration text, which identifies exactly which linear projections host the spikes that inflate the quantization scale. The recipe built on it is a static mixed-precision assignment: the spike projections run in FP16 or FP8 E5M2 (chosen because E5M2's maximum representable value of 57344 exceeds the observed spikes, which reach about 2500), while all other linear layers are quantized symmetrically per tensor to 8-bit integers. The random-control experiment is what rules out the alternative that any high-precision layers would help: random FP16 placement recovers none of the performance, so the location is what matters.","core_discovery":"The paper's central claim is that in LLaMA-style transformers the activation values that destroy per-tensor quantization are not distributed across the model: they appear as spikes in the MLP down projection of the first and last layers, specifically layers 2 and 32 for LLaMA3-8B, and additionally the attention output projection of the final layer for LLaMA2 and Mistral models. A per-tensor symmetric quantizer with scale $\\Delta = \\max|X|/(2^{b-1}-1)$ is dominated by a single spike, yet the paper shows that leaving just those two or three projections in FP16 or FP8 E5M2 and quantizing everything else to 8-bit integers recovers near-FP16 perplexity. The evidence includes max-absolute-value traces across layers, a random-layer control (random FP16 on the same number of projections gives perplexity 39.76 versus 8.24 for targeted placement on LLaMA3-8B), and tables across four models showing 8-bit per-tensor PPL of 8.24 (LLaMA3-8B), 6.27 (LLaMA2-7B), 8.38 (LLaMA2-13B), and 10.14 (Mistral-7B) against FP16 values around 5 to 6.","pith_inferences":["The spike-to-layer map is probably a training artifact, so the fixed recipe should be re-derived on each new checkpoint rather than assumed; the paper does not test a LLaMA model outside its four-model table.","The BOT-token result hints that part of the spike is an input-format effect; feeding a long prefix or a different beginning-of-text token before quantization could shrink the spike without mixed precision.","The method suggests a cheap diagnostic for any LLaMA-like model: one forward pass plotting per-layer max-abs activations tells you which two or three layers need FP8, turning a one-size-fits-all quantizer into a calibration-by-architecture scheme."],"forward_implications":["Eight-bit per-tensor quantization becomes deployable for LLaMA-family models with only two or three FP16 or FP8 projections, with no specialized outlier quantizer required.","On LLaMA3-8B, per-tensor perplexity drops from 40.45 (naive) and 44.74 (SmoothQuant) to 8.24, and zero-shot accuracy rises from about 51.5 to 65.5, approaching FP16 baselines.","FP8 E5M2 on the spike layers matches FP16 in most settings, so the recipe can ride on hardware with fast FP8 support.","Randomizing the placement of high precision does not recover the lost quality, meaning the gains come from hitting the specific spike-bearing projections rather than from extra precision anywhere.","The same recipe makes 6-bit per-tensor quantization usable, though with some instability, so the spike-localization idea extends to tighter bit budgets."],"supporting_citations":[{"why":"Documents that training hyperparameters change outlier emergence, the premise the paper uses to question universal outlier claims.","marker":"[1]"},{"why":"Supplies the first statistical definition of transformer outliers that the paper contrasts with its spike concept.","marker":"[2]"},{"why":"Defines the 6.0-magnitude outlier criterion and locates spikes in OPT, the contrast case for LLaMA.","marker":"[3]"},{"why":"Defines the FP8 E5M2 and E4M3 formats the paper uses for spike-carrying projections.","marker":"[12]"},{"why":"Introduces the massive-activations or spike terminology that the paper adopts.","marker":"[16]"},{"why":"Provides the SmoothQuant baseline and the channel-outlier view that the paper's LLaMA observations contradict.","marker":"[20]"},{"why":"Attributes quantization errors to activation spikes in GLU-based LLMs and motivates keeping spike layers in higher precision.","marker":"[21]"},{"why":"Supplies the evaluation harness and benchmark suite used for the zero-shot accuracy numbers.","marker":"[17]"}],"fun_headline_variants":["Two MLP down-projection spikes wreck LLaMA 8-bit per-tensor quant","Keep just two LLaMA projections in FP8 to fix 8-bit quantization","Architecture-specific quantization: two FP8 projections fix LLaMA 8-bit","Spikes in LLaMA's MLP down-projections explain 8-bit quantization failure","Why LLaMA 8-bit fails: spikes in two MLP down-projections"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the spike-bearing projections found in the four tested models are the same two or three projections in every LLaMA-style model, so the fixed recipe transfers without per-model re-detection.","fun_headline_variants_meta":{"raw":{"variants":["Two MLP down-projection spikes wreck LLaMA 8-bit per-tensor quant","Keep just two LLaMA projections in FP8 to fix 8-bit quantization","Architecture-specific quantization: two FP8 projections fix LLaMA 8-bit","Spikes in LLaMA's MLP down-projections explain 8-bit quantization failure","Why LLaMA 8-bit fails: spikes in two MLP down-projections"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001608,"raw_usage":{"total_tokens":6471,"prompt_tokens":1077,"completion_tokens":5394,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":5283}},"tokens_in":693,"tokens_out":5394,"duration_ms":41000,"temperature":1.0,"reasoning_tokens":5283,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:00:04.847784+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's fixed layer recipe on a LLaMA-family model outside its table, for instance LLaMA3-70B, and measure 8-bit per-tensor WikiText2 perplexity against FP16; if any projection outside the chosen two or three shows comparable max-absolute spikes, or if the perplexity gap stays large, the locality claim fails.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems36, 34278–34294 (2023)","cited_arxiv_id":null,"evidence_quote":"Documents that training hyperparameters change outlier emergence, the premise the paper uses to question universal outlier claims."},{"cited_title":"In: International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Provides the SmoothQuant baseline and the channel-outlier view that the paper's LLaMA observations contradict."}],"review_version":1}