Pith. sign in

REVIEW 6 major objections 6 minor 48 references

Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that most massive activations in LLMs are propagated copies of a few true outliers and can be removed with no measurable accuracy loss.

desk verdict The TMA/FMA distinction is worth thinking about, but the paper's own table contradicts its central 'no accuracy loss' claim, and the manuscript is too garbled to trust. read the letter →

arxiv 2505.21670 v1 pith:JI4XPD72 submitted 2025-05-27 cs.CL cs.AI

Rahul Raman , Khushi Sharma , Sai Qian Zhang This is my paper
classification cs.CLcs.AI
keywords massiveactivationschannel-wiseoutlierspost-trainingquantizationresidualconnectionsoutliersmoothingLLMinferencenormalizationrescalingoutlier-triggeringchannels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates where the two known kinds of LLM outliers come from. It argues that the vast majority of massive activations are not generated by most layers at all: they are copies of a handful of true outliers produced at the first feed-forward layer's y6 tensor, carried forward by residual connections. Removing those copied activations leaves perplexity almost unchanged across LLaMA, GPT-2, and Qwen models. The paper also traces channel-wise outliers to the rescaling step inside normalization layers and to specific outlier-triggering rows in weight matrices. If correct, outlier smoothing for quantization can be restricted to a small set of true outliers, lowering the cost of methods like Hadamard transforms.

What carries the argument

The central object is the distinction between True Massive Activations (TMAs) and Fake Massive Activations (FMAs): TMAs are the handful of activations (magnitude >100 and 1000× the median) that originate at y6 of the first-layer FFN (or y4 in classic MLPs), while FMAs are copies of them carried across layers by residual connections. For channel-wise outliers, the key mechanisms are the rescaling factor γ inside LayerNorm/RMSNorm, whose channel-specific scaling multiplies pre-existing variation into outlier channels, and Outlier-Triggering Channels (OTCs), roughly 1% of rows in a weight matrix that produce outlier activations when multiplied with the input even though they look unremarkable in isolation. The paper uses ablation-style interventions—setting activations, rescaling factors, or weight rows to zero or the mean—to show which components actually carry accuracy-relevant information.

What would settle it

Run the paper's y6 ablation (set TMAs to zero at y6 of layer 1) on a model outside the tested set—say, a Mixture-of-Experts or long-context model—and measure the top activation magnitudes at every layer. If any later layer exhibits a new massive activation at a channel or token position that was not present in the y6 output, the propagated-copy story fails for that architecture.

Watch

Extended reading notes

Core claim

On the paper's own terms: massive activations first appear inside the feed-forward network of the first layer—at y6, the gating product in gated MLPs, or right after GeLU in classic MLPs—and are then propagated through the model by residual connections. The paper calls these propagated copies Fake Massive Activations (FMAs) and the original few True Massive Activations (TMAs). Replacing the TMAs at y6 with zero or the tensor mean removes all subsequent FMAs, and the model's perplexity stays essentially unchanged on WikiText-2 and C4; replacing activations at y7 instead degrades performance dramatically. Channel-wise outliers, separately, first arise during the rescaling step of normalization, and a small set of weight rows (about 1% of channels, called Outlier-Triggering Channels) can create new outlier channels in the query and key projections. The paper concludes that most massive activations can be ignored by quantizers, and that mitigating channel-wise outliers should target rescaling factors and OTCs.

Load-bearing premise

The paper's core claim rests on the assumption that stripping the top activations at y6 of the first-layer feed-forward network removes all later massive activations, meaning no other layer can regenerate a fresh one; the evidence given is profiling of a few layers and models without a causal check.

Editorial extensions

If this is right

  • Outlier smoothing for quantization can ignore residual-propagated massive activations entirely, since they are copies with no measurable effect on perplexity.
  • The cost of invariance-based outlier smoothing (e.g., Hadamard rotations) can be confined to the few True Massive Activations at y6 rather than applied to all outlier positions.
  • Channel-wise outliers in inputs x2 and y2 can be reduced by adjusting the rescaling factors γ of normalization layers, though setting them to fixed values outright hurts accuracy.
  • Fine-tuning that specifically removes or regularizes Outlier-Triggering Channels in Q and K projections should suppress channel-wise outliers in intermediate activations with less damage than random channel changes.
  • Existing quantizers that already handle massive activations (e.g., mixed-precision or prefix grouping) may be over-engineering the problem, since most such activations do not need special treatment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the propagated-copy finding holds across architectures, outlier handling for any residual network could be reduced to a first-layer problem, not just for the tested LLaMA, GPT-2, and Qwen families.
  • The paper's static profiling on 100 samples leaves open whether FMAs reappear at different token positions under long-context or adversarial inputs; a causal intervention study across more layers and models would test whether regeneration ever occurs.
  • A natural extension is a quantizer that only protects the y6 positions (and the γ/OTC rows) and quantizes everything else aggressively; the paper's numbers suggest this could reach lower bit-widths than current smoothing pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper studies the formation and propagation of massive activations (MAs) and channel-wise outliers in LLMs, and proposes interventions to remove them with minimal impact on accuracy. The authors distinguish 'true' massive activations (TMAs), generated at y6 of the first-layer FFN, from 'fake' massive activations (FMAs), which are propagations of TMAs through residual connections. They claim that removing TMAs at y6 eliminates all MAs downstream with negligible perplexity change, and that channel-wise outliers arise from normalization rescaling and from a small set of 'Outlier Triggering Channels' (OTCs) in weight matrices. The paper concludes by recommending that quantization smoothing focus only on the small number of TMAs. The central evidence is presented in Table 2 (PPL after interventions), Figures 4-6 (MA propagation), Table 3 (OTC ablations), and several appendix tables.

Significance. If the central claims were correct, the paper would offer a useful, low-cost recipe for outlier smoothing in PTQ: removing only the true MAs at y6 would simplify activation quantization without retraining. The proposed TMA/FMA taxonomy and the OTC ablation could inform quantization-aware preprocessing. However, the manuscript as written does not support these claims: Table 2 contains corrupted rows and values that directly contradict the 'no measurable effect' conclusion, Figure 1 is an unrelated diagram from a different paper, the appendix provides only a single-layer, single-input profiling table in support of the propagation story, and no error bars or code are provided. The paper's self-description in the Limitations section as a 'survey' is also inconsistent with the claimed contributions. The potential practical significance is therefore not realized in this version.

major comments (6)
  1. [Section 3.3, Table 2] Table 2 directly contradicts the claim that removing TMAs at y6 has 'no measurable effect' on accuracy. For LLaMA3.2-3B, replacing y6 TMAs with the tensor mean raises WikiText PPL from 5.567 to 6.053 (+8.7%) and C4 PPL from 10.790 to 14.423 (+33.7%); replacing with zeros gives 6.237 and 14.767. For LLaMA3.1-8B, C4 PPL goes from 9.046 to 10.046 (mean) and 10.255 (zeros). These are substantial degradations for a quantization-motivated intervention, and the text's assertion in Section 3.3 that 'the results remain comparable to the original LLM' is not supported by the printed numbers. The y7 rows are also unparsable concatenations of digits (e.g., '1124111.7521046.8221281.491301562.251301562.256469.42'), so the reader cannot verify the claimed contrast between y6 and y7 interventions.
  2. [Section 3.3, Table 2] Table 2 shows strong evidence of table corruption. The 'TMAs to mean at y6' and 'TMAs to zeroes at y6' rows for LLaMA2-13B and GPT-2 are numerically identical to the 'Original' row (4.355/6.405 and 14.795/19.460), which is implausible even for a null result given that the same interventions produce visible changes in other models. The y7 rows for LLaMA2-13B and GPT-2 are also identical to the original row, while other models show catastrophic PPL increases. These internal inconsistencies mean that no trustworthy inference can be drawn from the paper's central experimental table.
  3. [Figure 1] Figure 1 is not the LLM quantization pipeline described in the caption. The figure contains labels such as 'FSNet Architecture', 'Foveated Segmentation', 'Saliency DNN', and 'Gaze direction', which are unrelated to outlier smoothing or post-training quantization. This is a serious presentation error: it undermines confidence in the provenance and careful construction of the other figures, and it must be corrected or explained before the manuscript can be evaluated.
  4. [Section 3.2, Figures 4 and 5] The claim that removing TMAs at y6 'effectively eliminates all TMAs and FMAs' is not causally established. The evidence in Figures 4 and 5 compares the model with and without all residual connections, which is a much more aggressive intervention than replacing y6 activations; it does not show that a y6-only replacement prevents the regeneration of large activations at every later layer. The appendix (Tables 4 and 5) profiles only layer 2 of LLaMA-3.2-3B under a single WikiText input. To support the paper's propagation story, the authors would need to re-profile all layers and all evaluated models after the y6 intervention, showing that no activation exceeding the MA threshold appears downstream.
  5. [Section 3.2, taxonomy definitions] The TMA/FMA distinction is defined by the same residual-removal operation used to test it: FMAs are called 'MAs caused by the residual link', and are then identified by removing residual connections. This makes the conclusion 'most MAs are caused by residual connections' partly circular. The classification could be made informative by an independent operational definition that does not presuppose the residual mechanism, or by a falsifiable prediction, such as a quantitative statement about which channels and token positions should carry FMAs in models with modified residual structures.
  6. [Section 3.1 and Section 4.1] All PPL measurements are reported as single numbers from '100 random samples' with no variance, confidence intervals, or statistical significance tests. Given that the key claim is 'no measurable effect,' the absence of any measure of measurement error is load-bearing. A null result requires some statement of the resolution of the experiment, e.g., standard error or a confidence interval on the PPL difference; without it, the reader cannot distinguish a genuine null from an underpowered or noisy evaluation.
minor comments (6)
  1. [Limitations] The Limitations section begins 'While this survey offers a comprehensive overview...', but the paper is not a survey; it is presented as an empirical study with new interventions. This mislabeling should be fixed to accurately describe the scope and contribution.
  2. [Section 1, bullet list] The first bullet contains a grammatical error: 'massive activations (MAs) are predominantly generated in the initial layers' is missing a preposition; it should be 'initial layers of the model.'
  3. [Section 3.3] The text uses inconsistent capitalization for the y7 intervention, referring to 'Y7' in one place and 'y7' in others; additionally, the sentence 'Thus, we show that most TMAs can be safely removed... Consequently, no TMAs can appear at Y7 or propagate via the residual connection' is logically unclear, since y7 follows y6 and would be directly affected by the y6 replacement.
  4. [Section 4.1] There is a typographical error in the sentence 'anoutlier channel if it satisfies' - 'anoutlier' should be 'an outlier'.
  5. [Section 4.4] In the sentence 'a great strategy is to fine-tune the rescaling factors gamma to reduce their variation,' the phrase 'a great strategy' is vague; the authors likely mean 'a promising strategy.'
  6. [Appendix B, Table 6] The table caption says 'Table A.3' while the table number is displayed as 'Table 6'; the mismatch should be resolved.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the study is a set of direct model interventions, and its central claim, while potentially incorrect on the printed data, is not equivalent to its inputs by construction.

full rationale

The paper is an empirical study rather than a derivation-from-principles paper. Its main causal chain is: (1) profile hidden states to locate massive activations; (2) identify y6 of the first-layer FFN as the earliest source; (3) classify residual-propagated MAs as FMAs and the residual-independent remainder as TMAs; (4) ablate TMAs by replacing them with mean or zero and measure perplexity. Each step is directly executed as an intervention on the model, and none of the reported perplexity numbers is obtained by fitting a parameter and then predicting a closely related quantity. The TMA/FMA terminology is defined through the residual-removal experiment, and the y6 location is established by profiling; although this makes the taxonomy intervention-relative, it does not make the perplexity comparison equivalent to the definition by construction. There are no load-bearing self-citations, no imported uniqueness theorem, and no fitted parameter disguised as a prediction. Table 2 is internally inconsistent and appears to contradict the 'no measurable effect' conclusion for LLaMA3.2-3B and LLaMA3.1-8B, but that is an empirical correctness problem, not circularity. Therefore, no circular step can be identified.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The paper's conclusions depend on a small number of hand-chosen thresholds (MA magnitude, m, beta), a small sampling budget, and unstated assumptions about the additivity of residual propagation and the stability of hidden-state interventions. The conceptual entities TMA, FMA, and OTC are introduced without independent falsifiable handles, and the empirical evidence for them is contained entirely within the paper.

free parameters (5)
  • MA magnitude threshold = 100 (absolute); 1000x median
    Adopted from Sun et al. 2024 for defining massive activations; not justified in this paper.
  • Channel outlier mean threshold m = 4 (default; varied in Figure 7)
    Set to 4 'without loss of generality'; the number of outlier channels changes materially with m.
  • Channel outlier std threshold beta = 1/3
    Fixed to 1/3 to enforce similar magnitudes within a channel; arbitrary.
  • Number of top activations profiled = 2 or 3
    Top-1, top-2, top-3 values are used to characterize MAs; the choice is not justified.
  • Sample count for PPL = 100
    All experiments average over 100 random samples; no variance reported.
assumptions (5)
  • domain assumption Top-2/top-3 activation magnitudes are sufficient to capture massive-activation behavior.
    Used throughout Section 3.2 and Table 1; no argument that lower-magnitude activations cannot be massive or important.
  • domain assumption Residual connections are the sole propagation path for massive activations between layers.
    The claim that removing y6 MAs removes all downstream MAs assumes no other path (e.g., attention, weight multiplication) regenerates them.
  • domain assumption Intervening on a hidden state by setting MAs to a constant does not alter other computations except through the removed outlier.
    The intervention test's validity depends on this stability assumption; no causal analysis is provided.
  • domain assumption Perplexity on 100 random samples from WikiText/C4 is a stable estimate of model accuracy.
    All PPL tables use 100 samples with no error bars; the conclusion that FMAs can be removed 'without impact' is based on these small samples.
  • ad hoc to paper The chosen outlier thresholds correctly identify channel-wise outliers.
    m=4 and beta=1/3 are set by the authors and varied only in a sensitivity plot; the observations are threshold-dependent.
invented entities (3)
  • True Massive Activations (TMA)
    purpose: Massive activations generated spontaneously in a layer, as opposed to those carried by residual connections.
    The TMA/FMA split is defined by the residual-removal procedure itself; no external prediction is given that could validate the taxonomy outside this paper.
  • Fake Massive Activations (FMA)
    purpose: Massive activations propagated via residual connections from earlier layers.
    Identified only in the paper's own interventions; the claim that they can be removed with no PPL change is not reproduced or tested on other tasks/models beyond the listed ones.
  • Outlier Triggering Channels (OTC)
    purpose: Approximately 1% of weight-matrix rows that, when multiplied by activation channels, create new channel-wise outliers in the output.
    OTCs are identified post hoc by comparing outlier channels in x3 vs x2, and their importance is tested by removal, which is an in-sample ablation, not an independent falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking the Outlier Distribution in Large Language Models: An In-depth Study." pith.science (2026). https://pith.science/paper/JI4XPD72

@misc{pith2026250521670,
  author       = {Pith},
  title        = {Pith review of: Rethinking the Outlier Distribution in Large Language Models: An In-depth Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JI4XPD72}},
  note         = {Machine review of arXiv:2505.21670}
}
read the original abstract

Investigating outliers in large language models (LLMs) is crucial due to their significant impact on various aspects of LLM performance, including quantization and compression. Outliers often cause considerable quantization errors, leading to degraded model performance. Identifying and addressing these outliers can enhance the accuracy and efficiency of the quantization process, enabling smoother deployment on edge devices or specialized hardware. Recent studies have identified two common types of outliers in LLMs: massive activations and channel-wise outliers. While numerous quantization algorithms have been proposed to mitigate their effects and maintain satisfactory accuracy, few have thoroughly explored the root causes of these outliers in depth. In this paper, we conduct a comprehensive investigation into the formation mechanisms of these outliers and propose potential strategies to mitigate their occurrence. Ultimately, we introduce some efficient approaches to eliminate most massive activations and channel-wise outliers with minimal impact on accuracy.

Figures

Figures reproduced from arXiv: 2505.21670 by the authors.

Figure 1
Figure 1. Given a pretrained LLM, techniques are first [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Architecture of a LLM decoder block. (b), (c) and (d) show the architectures of self-attention block, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) One example of massive activation pre [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Left: TMAs and FMAs within the input of LLaMA-13B across each layer. Right: after removing the MAs in residual connection, only TMA left. WikiText (Merity et al., 2016) and C4 (Hugging￾face, 2022). Each experiment is averaged over 100 random samples from the dataset. L…
Figure 5
Figure 5. Figure 5: Left: TMAs and FMAs within the input of GPT-2 across each layer. Right: after removing the MAs in residual connection, only TMA left. emerge in y6, the product of y4 and y5, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Changes on number of channel-wise outlier af [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Changes on number of channel-wise outliers within a normalization block of SA. Outlier channels denoted [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The blue line shows the number of outlier [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 12 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024 a . Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024

    arXiv 2024

  3. [3]

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024 b . Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456

    arXiv 2024

  4. [4]

    Lorenzo Bini, Marco Sorbi, and Stephane Marchand-Maillet. 2024. Characterizing massive activations of attention mechanism in graph neural networks. arXiv preprint arXiv:2409.03463

    work page Pith review arXiv 2024

  5. [5]

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. https://arxiv.org/abs/2306.12929 Quantizable transformers: Removing outliers by helping attention heads do nothing . Preprint, arXiv:2306.12929

    arXiv 2023

  6. [6]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  7. [7]

    Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13169--13178

    work page 2020

  8. [8]

    Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. 2024. Prefixquant: Static quantization beats dynamic through prefixed outliers in llms. arXiv preprint arXiv:2410.05265

    arXiv 2024

Show all 48 references
  1. [9]

    Xiang Deng, Vasilisa Bashlovkina, Feng Han, Simon Baumgartner, and Michael Bendersky. 2023. Llms to the moon? reddit market sentiment analysis with large language models. In Companion Proceedings of the ACM Web Conference 2023, pages 1014--1019

    2023

  2. [10]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318--30332

    2022

  3. [11]

    Elias Frantar and Dan Alistarh. 2022. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475--4488

    2022

  4. [12]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323--10337. PMLR

    2023

  5. [13]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

    2022 arXiv

  6. [14]

    Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. 2023. Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Co...

    2023

  7. [15]

    Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415

    2016 arXiv

  8. [16]

    Hui Huang, Shuangzhi Wu, Xinnian Liang, Bing Wang, Yanrui Shi, Peihao Wu, Muyun Yang, and Tiejun Zhao. 2023. Towards making the most of llm for translation quality estimation. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 375--386....

    2023

  9. [17]

    Huggingface . 2022. C4 dataset. https://huggingface.co/datasets/legacy-datasets/c4

    2022

  10. [18]

    Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara Jane Ericson, David Weintrop, and Tovi Grossman. 2023. How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment. In Proceedings of the 23rd Koli Calling International Confe...

    2023

  11. [19]

    Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. 2024. https://arxiv.org/abs/2402.18158 Evaluating quantized large language models . Preprint, arXiv:2402.18158

    2024 arXiv

  12. [20]

    Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426

    2021 arXiv

  13. [21]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024 a . Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, ...

    2024

  14. [22]

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024 b . Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532

    2024 arXiv

  15. [23]

    Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. 2021. Pay attention to mlps. Advances in neural information processing systems, 34:9204--9215

    2021

  16. [24]

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024. Spinquant--llm quantization with learned rotations. arXiv preprint arXiv:2405.16406

    2024 arXiv

  17. [25]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. https://arxiv.org/abs/1609.07843 Pointer sentinel mixture models . Preprint, arXiv:1609.07843

    2016 arXiv

  18. [26]

    Md Saef Ullah Miah, Md Mohsin Kabir, Talha Bin Sarwar, Mejdl Safran, Sultan Alfarhood, and MF Mridha. 2024. A multimodal approach to cross-lingual sentiment analysis with ensemble of transformer and llm. Scientific Reports, 14(1):9603

    2024

  19. [27]

    Andre Nakkab, Sai Qian Zhang, Ramesh Karri, and Siddharth Garg. 2024. Rome was not built in a single step: Hierarchical prompting for llm-based chip design. In Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD, pages 1--11

    2024

  20. [28]

    Alec Radford. 2018. Improving language understanding by generative pre-training

    2018

  21. [29]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  22. [30]

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2023. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137

    2023 arXiv

  23. [31]

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762

    2024 arXiv

  24. [32]

    Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. 2024. Verigen: A large language model for verilog code generation. ACM Transactions on Design Automation of Electronic Systems, 29(3):1--31

    2024

  25. [33]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  26. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  27. [35]

    Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. https://arxiv.org/abs/2402.04396 Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks . Preprint, arXiv:2402.04396

    2024 arXiv

  28. [36]

    Zeyu Wang, Yue Zhu, Shuyao He, Hao Yan, and Ziyi Zhu. 2024. Llm for sentiment analysis in e-commerce: A deep dive into customer feedback. Applied Science and Engineering Journal for Advanced Research, 3(4):8--13

    2024

  29. [37]

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. 2023. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145

    2023 arXiv

  30. [38]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087--38099. PMLR

    2023

  31. [39]

    Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Lingpeng Kong, et al. 2024. Uncomp: Uncertainty-aware long-context compressor for efficient large language model inference. arXiv preprint arXiv:2410.03090

    2024

  32. [40]

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417

    2024 arXiv

  33. [41]

    An Yang and et al. 2024. https://arxiv.org/abs/2407.10671 Qwen2 technical report . Preprint, arXiv:2407.10671

    2024 arXiv

  34. [42]

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168--27183

    2022

  35. [43]

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414

    2022 arXiv

  36. [44]

    Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. 2024 a . Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069

    2024 arXiv

  37. [45]

    Ying Zhang, Peng Zhang, Mincong Huang, Jingyang Xiang, Yujie Wang, Chao Wang, Yineng Zhang, Lei Yu, Chuan Liu, and Wei Lin. 2024 b . Qqq: Quality quattuor-bit quantization for large language models. arXiv preprint arXiv:2406.09904

    2024 arXiv

  38. [46]

    Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXiv:2304.04675

    2023 arXiv

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.