Pith. sign in

REVIEW 4 major objections 6 minor 31 references

SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By applying channel-wise scaling before a Hadamard rotation, SmoothRot makes 4-bit quantized LLMs perform closer to their FP16 originals, recovering about 10–30% of the performance gap on language generation and zero-shot tasks without…

desk verdict A solid, honest incremental paper: scaling before rotation works end-to-end under RTN, but the alpha tuning and missing latency measurement keep it from being a slam dunk. read the letter →

arxiv 2506.05413 v2 pith:XRVZSC4J submitted 2025-06-04 cs.CL cs.AIcs.LG

Patrik Czakó , Gábor Kertész , Sándor Szénási This is my paper
classification cs.CLcs.AIcs.LG
keywords post-trainingquantizationactivationoutliersHadamardtransformchannel-wisescaling4-bitlargelanguagemodelsefficientinferenceLLMcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the outliers that wreck 4-bit quantization of large language models can be tamed by doing channel-wise scaling before a Hadamard rotation, not just rotation alone. The proposed method, SmoothRot, pushes quantization error down so that 4-bit weight-activation-key-value quantized LLaMA2 7B, LLaMA3.1 8B, and Mistral 7B get perplexity and zero-shot accuracy closer to their FP16 originals than QuaRot achieves, shrinking the gap by roughly 10–30% with no added inference latency. The paper's central contribution is showing that a layer-wise recommendation about the order of operations survives end-to-end.

What carries the argument

The central mechanism is the composition $A = (H\Lambda)^{-1}$, where $\Lambda = \mathrm{diag}(s)$ is a channel-wise scaling matrix and $H$ is a Hadamard matrix. The scaling transfers part of the quantization difficulty from activations to weights; the rotation distributes the residual outlier mass across channels. Because the scaling is applied inside the feed-forward module before the Hadamard transform, and the scaling factors are fused into neighboring weight matrices, no extra latency is introduced. The machinery's work is to convert massive activation outliers (values above 1000) into values below roughly 0.5, a more than 30-fold reduction.

What would settle it

Re-tune the migration strength alpha on C4 instead of Wikitext-2 and compare C4 perplexity with SmoothRot's reported Wikitext-2-tuned alpha; if the improvement over QuaRot shrinks or reverses, the assumption that one per-model alpha transfers across distributions is falsified.

Watch

Extended reading notes

Core claim

SmoothRot combines channel-wise scaling with Hadamard rotation to preprocess the input activations of down-projection layers in GLU-based feed-forward modules. The channel-wise scaling, with per-channel factors $s_j = \max(|X_j|)^\alpha / \max(|W_j|)^{1-\alpha}$, is absorbed into the up-projection weights so there is no runtime cost, and the inverse scaling is folded into the down-projection weights. After this smoothing, the Hadamard rotation redistributes any remaining outliers across channels. The result is that, under W4A4KV4 RTN quantization, SmoothRot consistently beats rotation-only QuaRot on Wikitext-2 and C4 perplexity and on six zero-shot tasks across three 7–8B models, recovering about 10–30% of the gap to FP16.

Load-bearing premise

The single smoothing strength per model is tuned on one dataset, Wikitext-2, and the paper assumes the same value stays near-optimal on C4 and all six zero-shot tasks; if that transfer fails, the reported 10–30% gap reductions could shrink or even reverse on the unmeasured distributions.

Editorial extensions

If this is right

  • If SmoothRot holds, 4-bit RTN quantization with W4A4KV4 becomes usable for deployment without retraining or mixed-precision fallbacks.
  • The 10–30% reduction in gap to FP16 translates to roughly 0.8–1.2 perplexity improvement on Wikitext-2 and about 2–4 percentage points higher average zero-shot accuracy across the tested models.
  • Since no latency is added, SmoothRot can replace QuaRot in serving pipelines with no runtime tradeoff.
  • The method is calibration-light: 512 sentences and a few seconds of calibration suffice, so it is practical for quick deployment.
  • The paper's finding that SmoothRot does not help with GPTQ weight quantization suggests its use is best limited to RTN weight quantization unless further tuning is done.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The observation that SmoothRot does not help with GPTQ suggests that when weights are already accurately quantized, the benefit of smoothing diminishes; a natural extension is to adapt the migration strength based on the weight quantization error rather than fixing it for RTN.
  • The optimal alpha values cluster around 0.5, so a default alpha of 0.5 with no per-model search may be a robust starting point, a hypothesis that could be tested on models beyond LLaMA and Mistral.
  • SpinQuant's optimized rotation matrices were not re-optimized after smoothing was added; re-optimizing them for the smoothed activations may recover or exceed SpinQuant's standalone performance.
  • The method targets GLU down-projection spikes; extending SmoothRot to attention modules or to non-GLU architectures could broaden its applicability, but the paper does not test that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SmoothRot, a post-training quantization method for 4-bit weight-activation-KV-cache quantization of LLaMA-style LLMs. It combines channel-wise scaling, following SmoothQuant, with the Hadamard-rotation framework of QuaRot, applying scaling before rotation specifically at down-projection inputs in FFN blocks. Scaling factors are calibrated offline from Wikitext-2 and absorbed into adjacent weights so that no extra online operations are introduced. The method is evaluated on LLaMA2 7B, LLaMA3.1 8B, and Mistral 7B using Wikitext-2 and C4 perplexity plus six zero-shot tasks, with ablations covering calibration set, migration strength, GPTQ, and SpinQuant. The authors report that SmoothRot consistently narrows the gap to FP16 by about 10-30% relative to QuaRot under RTN quantization.

Significance. Modest but useful contribution: a simple, computationally invariant preprocessing step that improves 4-bit RTN quantization on three 7-8B models. The construction is sound, code is released, and Table III's calibration-dataset ablation is a useful robustness check. The claimed zero-latency property is plausible from the design but unmeasured, and the evaluation is narrow (single seed, no latency benchmarks, three models). The GPTQ and SpinQuant ablations, which show no benefit or slight degradation, are reported honestly but conflict with the framing of these as compatibility results. If alpha-transfer and latency are addressed, the RTN claim can be considered supported.

major comments (4)
  1. [Section IV-A, Fig. 3, Table I] The per-model migration strength α is selected by linear search on Wikitext-2 perplexity, which is also one of the two generation benchmarks in Table I. The paper shows α sweeps only for Wikitext-2 (Fig. 3) and reports no sweep over C4 or the zero-shot average. Consequently, the Wikitext-2 column is exposed to selection bias, and the transfer of the chosen α to C4 and zero-shot tasks is under-supported; Table III does not address α transfer because it fixes α and varies only the data used to estimate max(|X_j|) and max(|W_j|). Please provide α sweeps or a sensitivity analysis on at least one held-out distribution to support the 'consistent 10-30% gap reduction' claim.
  2. [Section IV-A and Conclusion] The abstract and conclusion state that SmoothRot introduces no additional inference latency, but no wall-clock, throughput, or memory benchmark is reported. The architectural argument (absorbed scaling matrices and fused transformations) makes zero overhead plausible, but the paper should either report a latency/throughput comparison against QuaRot or qualify the claim as 'no additional online operations' rather than an empirically verified latency property.
  3. [Section IV-C, Tables IV and V] Under GPTQ weight quantization, Table IV shows SmoothRot slightly worse than QuaRot on most entries (e.g., LLaMA3.1 8B Wiki 8.24 vs 8.12, C4 13.33 vs 13.16); under SpinQuant rotations, Table V shows SmoothRot worse on all reported metrics for LLaMA2 7B. The text acknowledges these as needing further investigation, but this is in tension with the contribution claiming compatibility with GPTQ and SpinQuant. The central claim should be scoped explicitly to RTN weight quantization (as the conclusion partially does), and the GPTQ/SpinQuant sections should be framed as negative results rather than compatibility demonstrations.
  4. [Table II] The per-task zero-shot results in Table II include several cases where SmoothRot is worse than QuaRot (e.g., LLaMA3.1 8B WinoGrande 65.27 vs 66.38; Mistral 7B PIQA 79.11 vs 79.22 and WinoGrande 68.11 vs 68.82). The 'consistent' improvement therefore holds for averages, not for all tasks; please define the scope of the consistency claim explicitly (e.g., average over tasks) or report statistical significance or confidence intervals.
minor comments (6)
  1. [Throughout] There are several typos that should be corrected: 'compatiblity' (Section I and II), 'post-trainig' (Section V), 'exhibinting' and 'magntidues' (Fig. 1 caption), 'withing' (Section II-A), and 'quantizations errors' (Section II-B).
  2. [Table II] The header 'PICA' should read 'PIQA', matching the task name used in the text.
  3. [Section IV-C, Fig. 3] The sentence 'Although LLaMA models only achieve better results than QuaRot within an α range of 0.35 − 0.7 to outperform QuaRot' is garbled; please rephrase to clarify the range and the comparison.
  4. [Section IV-A] The calibration procedure states that smoothing factors use '512 sentences (512 tokens each) randomly selected from Wikitext-2'; please specify which split (train or validation) is used and whether the same samples are used for all three models, to aid reproducibility.
  5. [Fig. 1] The caption describes activations 'under different quantization schemes', but panels (b) and (c) show transformed activations before quantization; the caption should say 'under different pre-quantization transformations' to avoid confusion.
  6. [Section II-A] The definition of massive activations says they are 'approximately 1000 times larger than the median magnitude of their hidden states'; consider reporting the exact criterion used (e.g., threshold on median ratio) since Fig. 1 and the conclusion both refer to magnitude reductions.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central 10-30% gap-reduction claim is measured against external baselines; the [10] self-citation is motivational, and the Wikitext-2 alpha search is a selection-bias concern rather than a circular reduction.

full rationale

SmoothRot is not circular. Its pipeline composes two published transformations (SmoothQuant channel-wise scaling [11] and QuaRot Hadamard rotation [12]) and then measures end-to-end performance against QuaRot, SpinQuant, GPTQ, and FP16 baselines; the claimed 10-30% gap reduction is an empirical comparison, not a value forced by an equation. The only author-overlapping citation is [10], which supplies the design hypothesis that scaling before rotation helps; that citation is motivational rather than load-bearing, because the paper's own Tables I-V, including cases where SmoothRot is slightly worse than GPTQ/SpinQuant baselines, constitute the actual evidence. The closest circularity-adjacent issue is hyperparameter selection: the per-model migration strength alpha is chosen by linear search on Wikitext-2 perplexity ('The optimal migration strength (α) is determined individually performing a linear search per model: 0.6 (LLaMA2 7B), 0.5 (LLaMA3.1 8B), and 0.45 (Mistral 7B)'), and the same metric is then reported in Table I. This creates selection bias in the Wikitext-2 column, but it does not reduce by construction: alpha is a discrete scalar hyperparameter, the C4 and zero-shot results are outside the tuning objective, and the calibration-set ablation in Table III indicates robustness to the data used. Under the hard rules, no quoted equation is equivalent to its input by definition, and no fitted parameter is renamed as a prediction; the honest verdict is no significant circularity, with a minor methodological caveat about test-set-aligned hyperparameter selection.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central method rests on standard linear-algebra identities and the empirical assumption that massive outliers appear at down-projection inputs. No new physical entities are introduced. The main model-specific slack is the migration strength alpha and the clipping ratios.

free parameters (5)
  • migration strength alpha = 0.6 (LLaMA2 7B), 0.5 (LLaMA3.1 8B), 0.45 (Mistral 7B)
    Tuned by linear search on Wikitext-2 PPL per model; controls the trade-off in the SmoothQuant scaling formula s_j = max(|X_j|)^alpha / max(|W_j|)^(1-alpha).
  • weight clipping ratios = per-channel, searched over squared error
    Used in RTN weight quantization; determined by linear search, standard in QuaRot but still tuned.
  • activation clipping ratio = 0.9
    Constant for per-token symmetric activation quantization, inherited from QuaRot.
  • KV clipping ratio = 0.95
    Constant for KV asymmetric quantization, inherited from QuaRot.
  • calibration set size = 512 sentences x 512 tokens
    Choice of calibration data size; not tuned, but a manual selection.
assumptions (4)
  • domain assumption Massive activation outliers are concentrated in specific channels of the input to down-projection layers in GLU-based LLMs.
    Basis for targeting FFN down-projections; from [16] and the authors' [10].
  • standard math RMSNorm is rotation-invariant: RMSNorm(X) = RMSNorm(XQ^T)Q for orthogonal Q, allowing fusion of the diagonal RMSNorm scaling Gamma with adjacent weight matrices.
    Used in Section III to preserve computational invariance during rotation.
  • standard math Elementwise nonlinearity (SiLU) commutes with diagonal scaling, so the scaling matrix Lambda can be absorbed into the up-projection weights.
    Enables zero-latency smoothing of the down-projection input.
  • domain assumption The SmoothQuant scaling formula with a single alpha adequately balances activation and weight quantization difficulty across the tested models.
    The alpha values are tuned per model, but the form of the formula is assumed from [11].

how reviews work

0 comments
Cite this review

Pith. "Pith review of SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs." pith.science (2026). https://pith.science/paper/XRVZSC4J

@misc{pith2026250605413,
  author       = {Pith},
  title        = {Pith review of: SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XRVZSC4J}},
  note         = {Machine review of arXiv:2506.05413}
}
read the original abstract

We present SmoothRot, a novel post-training quantization technique to enhance the efficiency of 4-bit quantization in Large Language Models (LLMs). SmoothRot addresses the critical challenge of massive activation outliers, by integrating channel-wise scaling with Hadamard transformations. Our technique effectively transforms extreme outliers into quantization-friendly activations, significantly improving quantization accuracy. Experiments conducted on popular LLMs (LLaMA2 7B, LLaMA3.1 8B, and Mistral 7B) demonstrate that SmoothRot consistently reduces the performance gap between quantized and FP16 models by approximately 10-30\% across language generation and zero-shot reasoning tasks, without introducing additional inference latency. Code is available at https://github.com/czakop/smoothrot.

Figures

Figures reproduced from arXiv: 2506.05413 by the authors.

Figure 1
Figure 1. Input activation magnitudes of the second down-projection layer of LLaMA2 7B under different quantization schemes. The original model (a) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SmoothRot applied to a LLaMA-style FFN. Following QuaRot [12], a diagonal matrix ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Effect of migration strength (α) on Wikitext-2 perplexity. The dashed horizontal lines indicate the perplexity achieved by QuaRot [12] for each model. Lower perplexity values correspond to better language modeling quality. and Arc-Challenge [29]. Evaluations are conducted using the LM Evaluation Harness [30] with default parameters. Our main results compare SmoothRot against QuaRot [12], highlighting the improvement… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 18 canonical work pages

  1. [10]

    Turning llm ac- tivations quantization-friendly,

    P. Czak ´o, G. Kert´esz, and S. Sz ´en´asi, “Turning llm ac- tivations quantization-friendly,” in 2025 IEEE 19th In- ternational Symposium on Applied Computational In- telligence and Informatics (SACI) , 2025, pp. 000 211– 000 216. DOI: 10.1109/SACI66288.2025.11030191

    arXiv 2025

  2. [1]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    H. Touvron, L. Martin, K. Stone, et al. “Llama 2: Open Foundation and Fine-Tuned Chat Models.” arXiv: 2307.09288 [cs]. (Jul. 19, 2023), pre-published

    arXiv 2023

  3. [2]

    The Llama 3 Herd of Models

    A. Grattafiori, A. Dubey, A. Jauhri, et al. “The Llama 3 Herd of Models.” arXiv: 2407.21783 [cs]. (Nov. 23, 2024), pre-published

    arXiv 2024

  4. [3]

    Mistral 7B

    A. Q. Jiang, A. Sablayrolles, A. Mensch, et al. “Mistral 7B.” arXiv: 2310.06825 [cs]. (Oct. 10, 2023), pre- published

    arXiv 2023

  5. [4]

    Training Experimen- tal Language Models with Low Resources, for the Hungarian Language,

    Z. G. H. Yang and T. V ´aradi, “Training Experimen- tal Language Models with Low Resources, for the Hungarian Language,” Acta Polytechnica Hungarica , vol. 20, no. 5, 2023

    work page 2023

  6. [5]

    Model Compression and Efficient Inference for Large Language Models: A Survey

    W. Wang, W. Chen, Y . Luo, et al. “Model Compression and Efficient Inference for Large Language Models: A Survey.” arXiv: 2402.09748 [cs]. (Feb. 15, 2024), pre-published

    arXiv 2024

  7. [6]

    Quantization and training of neural networks for efficient integer- arithmetic-only inference,

    B. Jacob, S. Kligys, B. Chen, et al. , “Quantization and training of neural networks for efficient integer- arithmetic-only inference,” Jun. 2018, pp. 2704–2713

    work page 2018

  8. [7]

    SqueezeLLM: Dense-and-Sparse Quantization,

    S. Kim, C. Hooper, A. Gholami, et al., “SqueezeLLM: Dense-and-Sparse Quantization,” presented at the Pro- ceedings of Machine Learning Research, vol. 235, 2024, pp. 23 901–23 923

    work page 2024

Show all 31 references
  1. [8]

    QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks,

    A. Tseng, J. Chee, Q. Sun, V . Kuleshov, and C. De Sa, “QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks,” pre- sented at the Proceedings of Machine Learning Re- search, vol. 235, 2024, pp. 48 630–48 656

    2024

  2. [9]

    Addressing activation outliers in llms: A systematic review of post- training quantization techniques,

    P. Czak ´o, G. Kert ´esz, and S. Sz ´en´asi, “Addressing activation outliers in llms: A systematic review of post- training quantization techniques,” IEEE Access, 2025. DOI: 10.1109/ACCESS.2025.3568702

    2025

  3. [11]

    SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models,” presented at the Proceedings of Machine Learning Research, vol. 202, 2023, pp. 38 087–38 099

    2023

  4. [12]

    QuaRot: Outlier-free 4-bit inference in rotated LLMs,

    S. Ashkboos, A. Mohtashami, M. L. Croci, et al., “QuaRot: Outlier-free 4-bit inference in rotated LLMs,” in Proceedings of the 38th International Con- ference on Neural Information Processing Systems , ser. NIPS ’24, vol. 37, Red Hook, NY , USA: Curran Associates Inc., Jun. 5,...

    2025

  5. [13]

    OPTQ: Accurate quantization for generative pre- trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “OPTQ: Accurate quantization for generative pre- trained transformers,” presented at the 11th Interna- tional Conference on Learning Representations, ICLR 2023, 2023

    2023

  6. [14]

    SpinQuant: LLM quantization with learned rotations

    Z. Liu, C. Zhao, I. Fedorov, et al. “SpinQuant: LLM quantization with learned rotations.” arXiv: 2405 . 16406v3. (Oct. 7, 2024), pre-published

    2024

  7. [15]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettle- moyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” presented at the Advances in Neural Information Processing Systems, vol. 35, 2022

    2022

  8. [16]

    Massive Activations in Large Language Models

    M. Sun, X. Chen, J. Z. Kolter, and Z. Liu. “Massive Activations in Large Language Models.” arXiv: 2402. 17762 [cs]. (Aug. 14, 2024), pre-published

    2024

  9. [17]

    Mitigating Quantiza- tion Errors Due to Activation Spikes in GLU-Based LLMs

    J. Yang, H. Kim, and Y . Kim. “Mitigating Quantiza- tion Errors Due to Activation Spikes in GLU-Based LLMs.” arXiv: 2405.14428v1. (May 23, 2024), pre- published

    2024 arXiv

  10. [18]

    Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization

    S. Son, W. Park, W. Han, K. Kim, and J. Lee. “Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization.” arXiv: 2406.12016v2. (Oct. 4, 2024), pre-published

    2024 arXiv

  11. [19]

    FlatQuant: Flatness Matters for LLM Quantization

    Y . Sun, R. Liu, H. Bai, et al. “FlatQuant: Flatness Matters for LLM Quantization.” arXiv: 2410.09426v1. (Oct. 12, 2024), pre-published

    2024 arXiv

  12. [20]

    AffineQuant: Affine Transformation Quantization for Large Lan- guage Models,

    Y . Ma, H. Li, X. Zheng, et al. , “AffineQuant: Affine Transformation Quantization for Large Lan- guage Models,” presented at the 12th International Conference on Learning Representations, ICLR 2024, 2024

    2024

  13. [21]

    DuQuant: Distribut- ing Outliers via Dual Transformation Makes Stronger Quantized LLMs

    H. Lin, H. Xu, Y . Wu, et al. “DuQuant: Distribut- ing Outliers via Dual Transformation Makes Stronger Quantized LLMs.” arXiv: 2406 . 01721v2. (Sep. 24, 2024), pre-published

    2024

  14. [22]

    DFRot: Achieving Outlier- Free and Massive Activation-Free for Rotated LLMs with Refined Rotation

    J. Xiang and S. Q. Zhang. “DFRot: Achieving Outlier- Free and Massive Activation-Free for Rotated LLMs with Refined Rotation.” arXiv: 2412 . 00648 [cs]. (Dec. 3, 2024), pre-published

    2024

  15. [23]

    Pointer Sentinel Mixture Models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer Sentinel Mixture Models,” presented at the International Conference on Learning Representations, Feb. 6, 2017

    2017

  16. [24]

    Exploring the limits of transfer learning with a unified text-to- text transformer,

    C. Raffel, N. Shazeer, A. Roberts, et al. , “Exploring the limits of transfer learning with a unified text-to- text transformer,” J. Mach. Learn. Res., vol. 21, no. 1, 140:5485–140:5551, Jan. 1, 2020, ISSN : 1532-4435

    2020

  17. [25]

    PIQA: Reasoning about Physical Commonsense in Natural Language,

    Y . Bisk, R. Zellers, R. Le Bras, J. Gao, and Y . Choi, “PIQA: Reasoning about Physical Commonsense in Natural Language,” in Proceedings of the AAAI Con- ference on Artificial Intelligence, vol. 34, Apr. 3, 2020, pp. 7432–7439. DOI: 10.1609/aaai.v34i05.6239

    2020 doi

  18. [26]

    WinoGrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “WinoGrande: An adversarial winograd schema challenge at scale,” Commun. ACM , vol. 64, no. 9, pp. 99–106, Aug. 24, 2021, ISSN : 0001-0782. DOI: 10.1145/3474381

    2021 doi

  19. [27]

    HellaSwag: Can a Machine Really Finish Your Sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “HellaSwag: Can a Machine Really Finish Your Sentence?” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, Florence, Italy: Association for Computational Linguistics, 2019, ...

    2019 doi

  20. [28]

    The LAMBADA dataset: Word prediction requiring a broad discourse context,

    D. Paperno, G. Kruszewski, A. Lazaridou, et al. , “The LAMBADA dataset: Word prediction requiring a broad discourse context,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , K. Erk and N. A. Smith, Eds., Berl...

    2016 doi

  21. [29]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    P. Clark, I. Cowhey, O. Etzioni, et al. “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.” arXiv: 1803 . 05457 [cs]. (Mar. 14, 2018), pre-published

    2018

  22. [30]

    L. Gao, J. Tow, B. Abbasi, et al. , A framework for few-shot language model evaluation , version v0.4.3, Jul. 2024. DOI: 10.5281/zenodo.12608602. [Online]. Available: https://zenodo.org/records/12608602

    2024

  23. [31]

    Building a Large Annotated Corpus of English: The Penn Treebank,

    M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a Large Annotated Corpus of English: The Penn Treebank,” Computational Linguistics , vol. 19, no. 2, J. Hirschberg, Ed., pp. 313–330, 1993

    1993

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.