{"id":"9587e241-4c33-4315-ad34-f1b758a29037","arxiv_id":"2505.21670","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.","lead":"This paper analyzes where outlier activations in large language models come from and claims that most are copies propagated through residual connections. It proposes that removing these copies does not hurt model accuracy, which could simplify quantization for LLMs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's own numbers contradict the 'no measurable effect' claim: setting y6 TMAs to mean/zero raises LLaMA3.2-3B C4 PPL from 10.790 to 14.423/14.767, so the central claim is unverified and, on the printed data, likely false.","rationale":"I read the paper as trying to establish a root-cause taxonomy (TMA/FMA) and to show that the propagated 'fake' massive activations can be safely removed, which would simplify quantization smoothing. The practical claim depends on the quantitative assertion that removing the y6 TMAs leaves downstream accuracy essentially unchanged. Table 2 is the only direct evidence for that assertion, and it is the weakest link: the table is garbled, and the legible entries for LLaMA3.2-3B and LLaMA3.1-8B show PPL increases that are far from negligible. This is an internal-consistency problem rather than a disagreement with external consensus, so it is especially damaging. The reader's weakest assumption about other mechanism regenerating MAs is related and plausible, but I would put the Table 2 contradiction first: even granting the propagation story, the printed numbers do not support the central claim. The Limitations section describing the paper as 'this survey' and the absence of released code or data further reduce verifiability. A clean rerun of the y6 intervention would settle whether the concern lands: if the rerun reproduces 14.423 on C4 for LLaMA3.2-3B, the claim fails; if it yields a tiny PPL change, the concern is resolved, though the manuscript would still need to fix its corrupted tables and provide the missing causal evidence. Since the reader's verdict was already REJECT and the concern reinforces rather than redirects that verdict, I recommend no change to the verdict.","tokens_in":14064,"tokens_out":6278,"duration_ms":62895,"concrete_test":"Reproduce the Section 3.3 intervention on LLaMA3.2-3B and LLaMA3.1-8B: for 100 WikiText and 100 C4 sequences, compute baseline PPL; then repeat with every activation whose magnitude exceeds 100 and is at least 1000x the tensor median at y6 of the first-layer FFN set to the mean (and, in a second arm, to zero) of that y6 tensor, continuing inference through the remaining layers. Report exact PPLs for all five models. If LLaMA3.2-3B C4 PPL exceeds the printed 10.790 baseline by more than 0.5 PPL, the 'no measurable effect' claim fails. Also re-profile top-3 magnitudes in x1 of layers 2, 25, and 39 after the y6 intervention to test the all-FMAs-eliminated assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central no-accuracy-loss claim is contradicted by its own Table 2 as printed. For LLaMA3.2-3B, replacing the y6 TMAs with the tensor mean changes WikiText PPL from 5.567 to 6.053 (+8.7%) and C4 PPL from 10.790 to 14.423 (+33.7%); replacing them with zeros gives 6.237 and 14.767. LLaMA3.1-8B C4 moves from 9.046 to 10.046 with the mean replacement and 10.255 with zeros. These are not 'no measurable effect', and they are far larger than any threshold one would accept for a quantization-motivated smoothing intervention. For LLaMA2-13B and GPT-2, the printed y6 rows are identical to the baseline row, which strongly suggests table corruption rather than a genuine null result, so even the supporting nulls cannot be trusted. Separately, the paper never directly shows, after the y6 intervention, that all later-layer massive activations are absent; the all-FMAs-eliminated claim rests on the residual-removal figures (Figures 4 and 5) and a one-layer appendix table, not on a causal re-profiling of every layer across all models. Thus the load-bearing mitigation recommendation 'ignore or remove propagated MAs' is unverified and, on the readable entries, empirically contradicted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the formation and propagation of massive activations (MAs) and channel-wise outliers in LLMs, and proposes interventions to remove them with minimal impact on accuracy. The authors distinguish 'true' massive activations (TMAs), generated at y6 of the first-layer FFN, from 'fake' massive activations (FMAs), which are propagations of TMAs through residual connections. They claim that removing TMAs at y6 eliminates all MAs downstream with negligible perplexity change, and that channel-wise outliers arise from normalization rescaling and from a small set of 'Outlier Triggering Channels' (OTCs) in weight matrices. The paper concludes by recommending that quantization smoothing focus only on the small number of TMAs. The central evidence is presented in Table 2 (PPL after interventions), Figures 4-6 (MA propagation), Table 3 (OTC ablations), and several appendix tables.","tokens_in":14381,"tokens_out":3000,"duration_ms":33661,"significance":"If the central claims were correct, the paper would offer a useful, low-cost recipe for outlier smoothing in PTQ: removing only the true MAs at y6 would simplify activation quantization without retraining. The proposed TMA/FMA taxonomy and the OTC ablation could inform quantization-aware preprocessing. However, the manuscript as written does not support these claims: Table 2 contains corrupted rows and values that directly contradict the 'no measurable effect' conclusion, Figure 1 is an unrelated diagram from a different paper, the appendix provides only a single-layer, single-input profiling table in support of the propagation story, and no error bars or code are provided. The paper's self-description in the Limitations section as a 'survey' is also inconsistent with the claimed contributions. The potential practical significance is therefore not realized in this version.","major_comments":[{"comment":"Table 2 directly contradicts the claim that removing TMAs at y6 has 'no measurable effect' on accuracy. For LLaMA3.2-3B, replacing y6 TMAs with the tensor mean raises WikiText PPL from 5.567 to 6.053 (+8.7%) and C4 PPL from 10.790 to 14.423 (+33.7%); replacing with zeros gives 6.237 and 14.767. For LLaMA3.1-8B, C4 PPL goes from 9.046 to 10.046 (mean) and 10.255 (zeros). These are substantial degradations for a quantization-motivated intervention, and the text's assertion in Section 3.3 that 'the results remain comparable to the original LLM' is not supported by the printed numbers. The y7 rows are also unparsable concatenations of digits (e.g., '1124111.7521046.8221281.491301562.251301562.256469.42'), so the reader cannot verify the claimed contrast between y6 and y7 interventions.","section":"Section 3.3, Table 2"},{"comment":"Table 2 shows strong evidence of table corruption. The 'TMAs to mean at y6' and 'TMAs to zeroes at y6' rows for LLaMA2-13B and GPT-2 are numerically identical to the 'Original' row (4.355/6.405 and 14.795/19.460), which is implausible even for a null result given that the same interventions produce visible changes in other models. The y7 rows for LLaMA2-13B and GPT-2 are also identical to the original row, while other models show catastrophic PPL increases. These internal inconsistencies mean that no trustworthy inference can be drawn from the paper's central experimental table.","section":"Section 3.3, Table 2"},{"comment":"Figure 1 is not the LLM quantization pipeline described in the caption. The figure contains labels such as 'FSNet Architecture', 'Foveated Segmentation', 'Saliency DNN', and 'Gaze direction', which are unrelated to outlier smoothing or post-training quantization. This is a serious presentation error: it undermines confidence in the provenance and careful construction of the other figures, and it must be corrected or explained before the manuscript can be evaluated.","section":"Figure 1"},{"comment":"The claim that removing TMAs at y6 'effectively eliminates all TMAs and FMAs' is not causally established. The evidence in Figures 4 and 5 compares the model with and without all residual connections, which is a much more aggressive intervention than replacing y6 activations; it does not show that a y6-only replacement prevents the regeneration of large activations at every later layer. The appendix (Tables 4 and 5) profiles only layer 2 of LLaMA-3.2-3B under a single WikiText input. To support the paper's propagation story, the authors would need to re-profile all layers and all evaluated models after the y6 intervention, showing that no activation exceeding the MA threshold appears downstream.","section":"Section 3.2, Figures 4 and 5"},{"comment":"The TMA/FMA distinction is defined by the same residual-removal operation used to test it: FMAs are called 'MAs caused by the residual link', and are then identified by removing residual connections. This makes the conclusion 'most MAs are caused by residual connections' partly circular. The classification could be made informative by an independent operational definition that does not presuppose the residual mechanism, or by a falsifiable prediction, such as a quantitative statement about which channels and token positions should carry FMAs in models with modified residual structures.","section":"Section 3.2, taxonomy definitions"},{"comment":"All PPL measurements are reported as single numbers from '100 random samples' with no variance, confidence intervals, or statistical significance tests. Given that the key claim is 'no measurable effect,' the absence of any measure of measurement error is load-bearing. A null result requires some statement of the resolution of the experiment, e.g., standard error or a confidence interval on the PPL difference; without it, the reader cannot distinguish a genuine null from an underpowered or noisy evaluation.","section":"Section 3.1 and Section 4.1"}],"minor_comments":[{"comment":"The Limitations section begins 'While this survey offers a comprehensive overview...', but the paper is not a survey; it is presented as an empirical study with new interventions. This mislabeling should be fixed to accurately describe the scope and contribution.","section":"Limitations"},{"comment":"The first bullet contains a grammatical error: 'massive activations (MAs) are predominantly generated in the initial layers' is missing a preposition; it should be 'initial layers of the model.'","section":"Section 1, bullet list"},{"comment":"The text uses inconsistent capitalization for the y7 intervention, referring to 'Y7' in one place and 'y7' in others; additionally, the sentence 'Thus, we show that most TMAs can be safely removed... Consequently, no TMAs can appear at Y7 or propagate via the residual connection' is logically unclear, since y7 follows y6 and would be directly affected by the y6 replacement.","section":"Section 3.3"},{"comment":"There is a typographical error in the sentence 'anoutlier channel if it satisfies' - 'anoutlier' should be 'an outlier'.","section":"Section 4.1"},{"comment":"In the sentence 'a great strategy is to fine-tune the rescaling factors gamma to reduce their variation,' the phrase 'a great strategy' is vague; the authors likely mean 'a promising strategy.'","section":"Section 4.4"},{"comment":"The table caption says 'Table A.3' while the table number is displayed as 'Table 6'; the mismatch should be resolved.","section":"Appendix B, Table 6"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to have been assembled with significant errors: an unrelated figure from a different paper is used as Figure 1, the main quantitative table is internally inconsistent, and the central claim is contradicted by the table's own readable entries. The self-description as a survey further suggests the text has not been carefully aligned with the experiments. I recommend rejection; if the authors can produce a corrected dataset, reproducible code, and a careful re-analysis, a fresh submission might be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the paper is stronger on nomenclature than on evidence. The TMA/FMA distinction — true massive activations that originate in the first-layer FFN and fake ones that ride along residual connections — is a plausible and potentially useful framing. The OTC idea (weight channels that look unremarkable in isolation but trigger outlier activations) is also interesting. If these held up, they'd guide where to spend smoothing effort. That is the paper's real value.\n\nThe problem is that the central empirical claim — removing propagated MAs has 'no measurable effect' on accuracy — is contradicted by the paper's own Table 2. For LLaMA3.2-3B, replacing y6 TMAs with the tensor mean lifts C4 PPL from 10.790 to 14.423; with zeros to 14.767. That is a 33–37% degradation, not 'no measurable effect.' LLaMA3.1-8B C4 moves from 9.046 to 10.046/10.255. The null rows for LLaMA2-13B and GPT-2 are numerically identical to baseline, which looks like corruption rather than a genuine null. So the load-bearing claim is unsupported, and on the readable entries it is false.\n\nThe manuscript also has structural defects that make verification impossible: Figure 1 is a mislabeled diagram from an unrelated foveation paper, Table 2 and Table 3 are garbled, and the Limitations section calls the work 'this survey.' No code or data is provided, no error bars, and the paper overstates novelty by claiming no prior channel-wise outlier study exists when SmoothQuant and others directly target that problem. These are not cosmetic issues; they undermine trust in every reported number.\n\nWhat the paper does well is the profiling method: look at where MAs originate, strip residual connections to separate propagated from original, and test interventions. That methodology could be reused. But the execution is not there.\n\nBottom line: this is a rough draft, not a reliable paper. If the authors clean up the tables, release code, and re-check the central claim on more models — and if it turns out that propagated MAs are indeed safe to remove in most cases — then it could be a modest but useful contribution to quantization research. As is, I would reject and not send it to peer review. There is nothing here that should be cited as evidence.","headline":"The TMA/FMA distinction is worth thinking about, but the paper's own table contradicts its central 'no accuracy loss' claim, and the manuscript is too garbled to trust.","tokens_in":14933,"tokens_out":3963,"would_cite":false,"duration_ms":36551,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that most massive activations in LLMs are propagated copies of a few true outliers and can be removed with no measurable accuracy loss.","keywords":["massive activations","channel-wise outliers","post-training quantization","residual connections","outlier smoothing","LLM inference","normalization rescaling","outlier-triggering channels"],"falsifier":"Run the paper's y6 ablation (set TMAs to zero at y6 of layer 1) on a model outside the tested set—say, a Mixture-of-Experts or long-context model—and measure the top activation magnitudes at every layer. If any later layer exhibits a new massive activation at a channel or token position that was not present in the y6 output, the propagated-copy story fails for that architecture.","tokens_in":13813,"feed_emoji":"⚡","tokens_out":6006,"duration_ms":55569,"temperature":0.7,"pith_summary":"The paper investigates where the two known kinds of LLM outliers come from. It argues that the vast majority of massive activations are not generated by most layers at all: they are copies of a handful of true outliers produced at the first feed-forward layer's y6 tensor, carried forward by residual connections. Removing those copied activations leaves perplexity almost unchanged across LLaMA, GPT-2, and Qwen models. The paper also traces channel-wise outliers to the rescaling step inside normalization layers and to specific outlier-triggering rows in weight matrices. If correct, outlier smoothing for quantization can be restricted to a small set of true outliers, lowering the cost of methods like Hadamard transforms.","feed_headline":"Removing copied outliers does not hurt LLM accuracy","feed_subtitle":"Most massive activations are residual-propagation copies of a few true outliers; quantizers can ignore them.","key_machinery":"The central object is the distinction between True Massive Activations (TMAs) and Fake Massive Activations (FMAs): TMAs are the handful of activations (magnitude >100 and 1000× the median) that originate at y6 of the first-layer FFN (or y4 in classic MLPs), while FMAs are copies of them carried across layers by residual connections. For channel-wise outliers, the key mechanisms are the rescaling factor γ inside LayerNorm/RMSNorm, whose channel-specific scaling multiplies pre-existing variation into outlier channels, and Outlier-Triggering Channels (OTCs), roughly 1% of rows in a weight matrix that produce outlier activations when multiplied with the input even though they look unremarkable in isolation. The paper uses ablation-style interventions—setting activations, rescaling factors, or weight rows to zero or the mean—to show which components actually carry accuracy-relevant information.","core_discovery":"On the paper's own terms: massive activations first appear inside the feed-forward network of the first layer—at y6, the gating product in gated MLPs, or right after GeLU in classic MLPs—and are then propagated through the model by residual connections. The paper calls these propagated copies Fake Massive Activations (FMAs) and the original few True Massive Activations (TMAs). Replacing the TMAs at y6 with zero or the tensor mean removes all subsequent FMAs, and the model's perplexity stays essentially unchanged on WikiText-2 and C4; replacing activations at y7 instead degrades performance dramatically. Channel-wise outliers, separately, first arise during the rescaling step of normalization, and a small set of weight rows (about 1% of channels, called Outlier-Triggering Channels) can create new outlier channels in the query and key projections. The paper concludes that most massive activations can be ignored by quantizers, and that mitigating channel-wise outliers should target rescaling factors and OTCs.","pith_inferences":["If the propagated-copy finding holds across architectures, outlier handling for any residual network could be reduced to a first-layer problem, not just for the tested LLaMA, GPT-2, and Qwen families.","The paper's static profiling on 100 samples leaves open whether FMAs reappear at different token positions under long-context or adversarial inputs; a causal intervention study across more layers and models would test whether regeneration ever occurs.","A natural extension is a quantizer that only protects the y6 positions (and the γ/OTC rows) and quantizes everything else aggressively; the paper's numbers suggest this could reach lower bit-widths than current smoothing pipelines."],"forward_implications":["Outlier smoothing for quantization can ignore residual-propagated massive activations entirely, since they are copies with no measurable effect on perplexity.","The cost of invariance-based outlier smoothing (e.g., Hadamard rotations) can be confined to the few True Massive Activations at y6 rather than applied to all outlier positions.","Channel-wise outliers in inputs x2 and y2 can be reduced by adjusting the rescaling factors γ of normalization layers, though setting them to fixed values outright hurts accuracy.","Fine-tuning that specifically removes or regularizes Outlier-Triggering Channels in Q and K projections should suppress channel-wise outliers in intermediate activations with less damage than random channel changes.","Existing quantizers that already handle massive activations (e.g., mixed-precision or prefix grouping) may be over-engineering the problem, since most such activations do not need special treatment."],"supporting_citations":[{"why":"Provides the definition of massive activations (magnitude >100 and 1000× the median) that the paper adopts and refines.","marker":"Sun et al., 2024"},{"why":"Establishes the presence of outlier activations and KV-vector outliers in LLMs, motivating the mixed-precision decomposition baseline.","marker":"Dettmers et al., 2022"},{"why":"SmoothQuant, the scale-invariance method for channel-wise outlier smoothing that the paper's CO analysis builds on.","marker":"Xiao et al., 2023"},{"why":"QuaRot, an invariance transform for outlier-free quantized LLMs; the paper suggests limiting such transforms to the few true MAs.","marker":"Ashkboos et al., 2024b"},{"why":"PrefixQuant, which groups tokens with massive activations for quantization; the paper's result implies most grouped MAs are fake copies.","marker":"Chen et al., 2024"},{"why":"Olive, a hybrid quantization scheme that treats massive activations separately, providing a baseline that the FMA finding would simplify.","marker":"Guo et al., 2023"},{"why":"QuIP#, a Hadamard-incoherence quantization scheme; cited as a representative invariance-transform approach whose cost could be reduced.","marker":"Tseng et al., 2024"}],"fun_headline_variants":["Most LLM outliers are just copies; ignore them","LLM outliers: most are fake, safe to ignore","Remove fake outliers, keep LLM accuracy","LLMs: only a few true outliers matter","Ignore copied outliers, retain LLM performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's core claim rests on the assumption that stripping the top activations at y6 of the first-layer feed-forward network removes all later massive activations, meaning no other layer can regenerate a fresh one; the evidence given is profiling of a few layers and models without a causal check.","fun_headline_variants_meta":{"raw":{"variants":["Most LLM outliers are just copies; ignore them","LLM outliers: most are fake, safe to ignore","Remove fake outliers, keep LLM accuracy","LLMs: only a few true outliers matter","Ignore copied outliers, retain LLM performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1475,"prompt_tokens":894,"completion_tokens":581,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":509}},"tokens_in":510,"tokens_out":581,"duration_ms":5943,"temperature":1.0,"reasoning_tokens":509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:25:13.712516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's y6 ablation (set TMAs to zero at y6 of layer 1) on a model outside the tested set—say, a Mixture-of-Experts or long-context model—and measure the top activation magnitudes at every layer. If any later layer exhibits a new massive activation at a channel or token position that was not present in the y6 output, the propagated-copy story fails for that architecture.","supporting_citations":[],"review_version":1}