Pith. sign in

REVIEW 3 major objections 5 minor 24 references

Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the functionally important learned structure in transformer attention weights is concentrated in a few spectral outliers above the Marchenko-Pastur threshold, so zeroing them collapses model competence to chance.

desk verdict Useful observational mapping of attention-weight spectral structure, but the headline causal claim is not supported by the experiments as run; the paper's own V-bulk ablation undercuts it. read the letter →

arxiv 2608.07921 v1 pith:2QIJJL74 submitted 2026-08-08 cs.LG cs.AIcs.CL

Kasun Dewage , Marianna Pensky , Suranadi De Silva , T. H. Bandara This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords Marchenko-PasturrandommatrixtheorytransformerinterpretabilityattentionmechanismsspectraloutliersmodelpruningLoRAlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the functionally important learned structure in transformer attention weights is concentrated in a small number of spectral outliers—singular values that stand above the Marchenko-Pastur threshold expected from a random matrix. Zeroing all such outliers in Mistral-7B drops HellaSwag from 0.814 to 0.256, MMLU from 0.627 to 0.269, and PIQA from 0.823 to 0.508, close to random guessing, while zeroing a count-matched set of below-threshold singular values does far less damage. A sympathetic reader would care because this offers a principled, parameter-free way to decide which parts of a transformer's weights carry knowledge and which are near-noise, with direct implications for pruning and low-rank fine-tuning. The paper also documents systematic patterns across 11 models: Q projections carry the most outliers, V projections under grouped-query attention largely lose clean signal, and specific residual-stream dimensions persist as outlier "highways" across layers.

What carries the argument

The central object is the Marchenko-Pastur upper edge estimate λ+ = σ̂²(1+√γ)², with noise variance estimated from the median squared singular value and aspect ratio γ = max(m,n)/min(m,n). Singular values squared above λ+ are classified as spectral outliers (signal); those below form the bulk. The argument is carried by a causal ablation design: zeroing outlier singular values versus zeroing a count-matched random subset of bulk singular values, evaluated with HellaSwag, MMLU, and PIQA, plus entry-level outlier heatmaps and cross-layer residual-stream alignment to show the outliers are spatially organized.

What would settle it

A single concrete observation would settle the claim: if there exists a count-matched subset of below-threshold singular values in some attention projection of Mistral-7B whose zeroing collapses HellaSwag or MMLU as severely as zeroing the spectral outliers does, then the outliers are not uniquely dominant. The paper's own LLaMA-3-8B V projection already approaches this, with 7,013 bulk singular values driving MMLU from 0.660 to 0.254; a systematic search across projections for similarly catastrophic bulk subsets would either confirm the dominant-outlier claim or falsify it.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that attention projection matrices in pretrained transformers can be separated by the Marchenko-Pastur law into a random-like bulk and a sparse set of spectral outliers, and that this split is causally meaningful: deleting the outliers destroys the model's competence, while deleting an equal number of bulk components is comparatively benign. The ablation zeroing roughly 112,000 outlier singular values across all Q, K, V, O projections of Mistral-7B drives three benchmarks to near chance, and per-component ablations on LLaMA-3-8B give a criticality order K > Q ≫ V. A caution the paper itself flags: below-threshold components are not pure noise, because MMLU drops 6–13 points when bulk singular values are removed, and in one LLaMA-3 V-projection ablation zeroing only 7,013 bulk singular values collapses MMLU to 0.254.

Load-bearing premise

The load-bearing premise is that the Marchenko-Pastur threshold, computed with a median-based noise estimator, cleanly separates genuine learned signal from random-like bulk, so singular values below the threshold are near-noise and safe to use as a control—an assumption the paper explicitly calls approximate and that its own LLaMA-3 V-bulk ablation shows can fail.

Editorial extensions

If this is right

  • If the claim is right, a substantial fraction of a transformer's attention weights are near-random and can be pruned or quantized aggressively without touching the outlier subspace.
  • Parameter-efficient fine-tuning methods like LoRA should allocate rank budget to outlier-rich projections (Q, and K under GQA) rather than uniformly across layers and matrices.
  • The persistent residual-stream "highway" dimensions in K and O are natural targets for structured pruning or for identifying which directions carry information across layers.
  • The K > Q ≫ V criticality ordering, if it holds more broadly, means value projections deserve less adaptation budget than query and key projections.
  • The V-bulk catastrophe under GQA warns that uniform singular-value thresholding across projections is unsafe; each projection type has its own signal/noise regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to run the same zero-outlier ablation across many more tasks and models; if the collapse is universal, spectral outlier fraction could become a cheap proxy for a model's usable capacity.
  • The paper's own V-bulk result suggests the MP threshold may systematically misclassify spectrally diffuse but real structure in V under GQA; a heavy-tailed RMT model, as the authors note, might fix this and is a concrete next step.
  • The cross-component persistent dimensions (e.g., LLaMA-2 dimension 2533 appearing in all four projections) hint at a small set of privileged residual-stream channels; if confirmed by activation-based analysis, these could serve as interpretable 'bus' directions that mediate cross-layer communication.
  • The entry-level row/column band structure in Q and O suggests that attention heads are not interchangeable: a few heads dominate outlier structure, so head-level pruning based on spectral outlier density may outperform magnitude pruning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript applies a Marchenko–Pastur (MP) upper-edge threshold to the squared singular values of Transformer attention projection matrices, labeling super-threshold singular values as spectral outliers and below-threshold values as bulk. After documenting outlier counts, energy ratios, entry-level outlier maps, and cross-layer persistent residual-stream dimensions across 11 models, the authors run zeroing ablations: removing all MP outliers in Mistral-7B collapses HellaSwag, MMLU, and PIQA to near chance, while count-matched random bulk removals in LLaMA-1/2 show smaller degradation. The paper concludes that spectral outliers encode a dominant component of learned structure, identifies recurring patterns (Q dominance, V sparsity under GQA, row/column band structure, persistent residual-stream channels), and discusses implications for fine-tuning and pruning.

Significance. The paper is potentially valuable as a weight-space diagnostic: the code is released, the model coverage is broad (six architecture families, 11 checkpoints), and the observational findings—Q-dense/V-sparse spectra under GQA, row-bands in Q, column-bands in O, persistent residual-stream dimensions—are concrete and reproducible. The causal ablation idea is a good way to test the decomposition. However, the central causal claim as stated is not currently supported: the key comparison is confounded by model identity and by spectral energy, and the paper's own V-bulk ablation on LLaMA-3 contradicts the low-importance-bulk assumption. If the authors add a same-model, energy-matched control and restrict the dominance claim to the components and architectures for which it holds, the paper could make a sound contribution.

major comments (3)
  1. [IV.B; Table VI] The headline comparison mixes models and is not energy-matched. The all-outlier ablation is reported only for Mistral-7B (N=112,458), while the random-bulk controls are reported for LLaMA-1-7B and LLaMA-2-7B; Table VI contains no Mistral random-bulk row. The controls are count-matched only in the loose sense of using each model's own outlier count (with f=0.75; N_zero differs across models), and they are not energy-matched. From Table II, the zeroed Mistral outliers carry 87.5% of Q, 74.7% of K, and 84.6% of O spectral energy, whereas random draws from the bulk select low-energy singular values. The observed collapse can therefore be attributed to removing a large fraction of spectral mass rather than specifically to MP-identified 'learned structure.' Please add a same-model Mistral ablation that removes the same number of bulk singular values with matched total spectral energy, and/or report per-unit-energy damage.
  2. [IV.C; Table VI; Section VII] The below-threshold bulk is not a low-importance control in the paper's own data. The LLaMA-3-8B V random-bulk ablation zeros only 7,013 below-threshold V singular values and drops MMLU from 0.660 to 0.254 and HellaSwag from 0.821 to 0.649, damage comparable to the Mistral all-outlier collapse. This is a core projection of a GQA model, not a negligible edge case. Section VII correctly acknowledges that GQA V lacks a clean signal/noise separation, but the abstract, Section IV.B, and Pattern 5 present the dominance claim without this exception. The manuscript needs to either restrict the claim to Q/K/O in architectures where bulk ablation is benign, or show quantitatively why the V-bulk catastrophe is compatible with the headline 'spectral outliers encode a dominant component of learned structure.'
  3. [II.A, II.D, Eq. (1)] The causal design does not isolate the MP threshold from singular-value magnitude. Since the signal/noise split is defined by s_i^2 > λ_+, a comparison of 'large singular values removed' versus 'small singular values removed' may simply reflect a monotone relationship between singular value magnitude and functional importance; a random bulk draw selects low-energy components by construction. The MP threshold would then be a useful ranking heuristic but not a validated discovery of 'learned structure' vs 'noise.' To support the stronger conclusion, compare outlier removal against removal of the largest below-threshold bulk singular values matched on count and on total energy, and add a shuffled-weight or randomly initialized control (the latter is already listed as future work in Section VII).
minor comments (5)
  1. [II.A / Eq. (1)] The median-based noise estimator σ̂² = median(s_i²)/(1+γ) is an approximation to the median of the MP law; the Limitation section notes this, but the Method section should state it at the point of definition and cite the exact median correction for finite γ.
  2. [IV.B / Table VI] Calling the random-bulk MMLU degradation 'smaller but non-negligible' understates LLaMA-1's f=0.75 row, where MMLU drops from 0.352 to 0.292, leaving only 4.2 points above the 0.250 chance baseline; please report relative degradation and discuss this in the same paragraph as the 'preserves much of the model's capability' sentence.
  3. [II.D / Table VI] The random-bulk experiments appear to use a single draw with no seed; report at least 5 random draws with mean and standard deviation, as Section VII already says is needed.
  4. [II.B / Section VII] The entry-level threshold |w_ij| > 4σ_W is applied without a null-model baseline; since isolated extremes are expected in large matrices, include a shuffled-weight or random-Gaussian control to support the claim that observed row/column bands are learned, not threshold artifacts.
  5. [Table II / III] Table II leaves Q/K/V blank for Phi-3-mini and Table III omits two models; add a note in each caption or a supplementary table with the full data so the cross-model comparisons are not obscured by page-limit omissions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is supported by an independent causal ablation, and the paper's conclusion is not forced by its definitions.

full rationale

The paper's derivation chain is empirical rather than definitional. It defines spectral outliers as singular values above the MP upper edge (Eq. 1) and then causally intervenes by zeroing those singular values (Section IV.B). The observed collapse on Mistral-7B is not entailed by the definition of an outlier; the ablation could have failed, and indeed partially does in the LLaMA-3-8B V-bulk case reported in Table VI and Section IV.C. The count-matched bulk control is run on different models (LLaMA-1/LLaMA-2) and is not energy-matched, so the 'dominant learned structure' claim is not as strongly supported as the paper suggests; however, that is a confound or validity limitation, not a circular reduction. The only self-citation, CRAFT [24], appears in the conclusion as a complementary pointer and is not load-bearing. No equation is defined in terms of the conclusion, no fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The limitations section (Section VII) explicitly acknowledges that the MP bulk is not pure noise and that the causal claim concerns dominance rather than exclusivity, further indicating the claim is an empirical hypothesis rather than a definitional tautology.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on the MP threshold and on the control comparison. Both are approximate and partly hand-chosen. The most consequential choices are the median-based noise estimator, the 4-sigma entry threshold, the mu+4-sigma band threshold, the at-least-3-layers persistence cut-off, and the f=0.75/1.0 bulk matching fractions. No new entities are postulated.

free parameters (5)
  • Noise variance estimator constant = 1/(1+gamma) scaling of median(s^2)
    sigma_hat^2 = median(s^2)/(1+gamma) is an approximate MP median formula. Section VII acknowledges that the exact median of the MP distribution differs slightly with gamma, which can shift lambda_plus and the outlier count.
  • Entry-level outlier threshold = 4 sigma_W
    Positions with |W_ij| > 4 sigma_W are flagged as entry outliers. This threshold is chosen by hand, and Section VII notes that isolated extreme values can occur by chance in large random matrices.
  • Band outlier threshold = mu + 4 sigma
    Columns or rows whose norm exceeds the mean plus four standard deviations are flagged as band outliers. This is a hand-chosen threshold with no calibration.
  • Persistence threshold = >= 3 layers
    A residual-stream dimension is called persistent if it appears as a band outlier in at least 3 layers. This cut-off is arbitrary and affects the persistence tables.
  • Bulk matching fraction f = 0.75 and 1.00
    Random bulk subsets are matched to 75% or 100% of the outlier count. This is hand-chosen and count-based, not energy-matched.
assumptions (4)
  • domain assumption Marchenko-Pastur law and the spiked model describe the squared singular value spectrum of attention weight matrices as a random bulk plus a few outliers.
    Invoked in Section II.A (Eq. 1) to set lambda_plus. The paper cites Martin and Mahoney [6] showing real DNN spectra are heavy-tailed, so the MP bulk assumption is an acknowledged approximation.
  • domain assumption The median of the squared singular values estimates the noise variance of the bulk.
    Used in Eq. (1) via sigma_hat^2 = median(s^2)/(1+gamma). This assumes the bulk dominates the spectral median so that outliers do not bias the estimate.
  • domain assumption Zeroing a random subset of bulk singular values is an appropriate control for removing outliers.
    Underlies the causal comparison in Section IV.B. The control is count-matched but not energy-matched, and it is run on different models than the outlier collapse, so the comparison is confounded.
  • domain assumption Zeroing singular values in a fixed pretrained weight matrix before evaluation is a valid measure of the functional importance of those components.
    The ablation methodology in Section II.D modifies the weight matrix before zero-shot evaluation, assuming the SVD-based removal reflects the component's role in the forward pass.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention." pith.science (2026). https://pith.science/paper/2QIJJL74

@misc{pith2026260807921,
  author       = {Pith},
  title        = {Pith review of: Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QIJJL74}},
  note         = {Machine review of arXiv:2608.07921}
}
read the original abstract

We apply Marchenko-Pastur (MP) random matrix theory to pre-trained attention weights in order to separate each projection matrix into a random-like bulk and a set of spectral outliers. We validate this decomposition causally: zeroing the MP-identified outliers (signal) in Mistral-7B drives HellaSwag, MMLU, and PIQA close to random-chance performance, whereas zeroing a count-matched subset of bulk singular values causes smaller but non-negligible degradation. Across 11 pre-trained transformers we identify five recurring patterns: spectral outliers encode a dominant component of the learned structure; Q projections carry the most outliers; V projections under grouped-query attention lack a clean signal/noise separation; entry-level outliers form structured row-bands in Q and column-bands in O; and specific residual-stream dimensions persist as band outliers across layers in K and O. We close by outlining how these observations could inform parameter-efficient fine-tuning and structured pruning.

Figures

Figures reproduced from arXiv: 2608.07921 by the authors.

Figure 1
Figure 1. Spectral density of transformer weight matrices illustrating the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Mistral-7B (left) and Qwen2.5-7B (right). Both GQA models [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. 3D attention stack for BERT-base. Persistent dimensions concentrate [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: 3D attention stack for LLaMA-2-7B. Dense persistent dimensions [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 8 canonical work pages

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 5998–6008

    2017

  2. [2]

    A mathematical framework for transformer circuits,

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. Mc- Candlish, and C. Olah, “A mathematical framework for transformer circuits,”Transformer Ci...

    work page 2021

  3. [3]

    In-context learning and induction heads,

    C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah, “In-context learning and induction heads,” arXiv prepri...

    arXiv 2022

  4. [4]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inProc. Int. Conf. Learn. Represent. (ICLR), 2022

    2022

  5. [5]

    Distribution of eigenvalues for some sets of random matrices,

    V . A. Marchenko and L. A. Pastur, “Distribution of eigenvalues for some sets of random matrices,”Mathematics of the USSR-Sbornik, vol. 1, no. 4, pp. 457–483, 1967, doi: 10.1070/SM1967v001n04ABEH001994

    doi:10.1070/sm1967v001n04abeh001994 1967

  6. [6]

    Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning,

    C. H. Martin and M. W. Mahoney, “Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning,”Journal of Machine Learning Research, vol. 22, no. 165, pp. 1–73, 2021

    2021

  7. [7]

    A spectral condition for feature learning,

    G. Yang, J. B. Simon, and J. Bernstein, “A spectral condition for feature learning,”arXiv preprint arXiv:2310.17813, 2023

    arXiv 2023

  8. [8]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” inProc. NAACL-HLT, 2019, pp. 4171–4186

    work page 2019

Show all 24 references
  1. [9]

    RoBERTa: A robustly optimized BERT pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  2. [10]

    OPT: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “OPT: Open pre-trained transformer language models,”arXiv preprint a...

    2022 arXiv

  3. [11]

    LLaMA: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “LLaMA: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  4. [12]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  5. [13]

    The Llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The Llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    Mistral 7B,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed, “Mistral 7B,”arXiv preprint arXiv:2310.06825, 2023

    arXiv

  7. [15]

    Qwen2.5 technical report,

    Qwen Team: A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia...

    2024 arXiv

  8. [16]

    Phi-3 technical report: A highly capable language model locally on your phone,

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behlet al., “Phi-3 technical report: A highly capable language model locally on your phone,”arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  9. [17]

    GQA: Training generalized multi-query transformer models from multi-head checkpoints,

    J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebr ´on, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” inProc. Conf. Empirical Methods in Natural Language Processing (EMNLP), 2023, pp. 4895–4901

    2023

  10. [18]

    HellaSwag: Can a machine really finish your sentence?,

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “HellaSwag: Can a machine really finish your sentence?,” inProc. 57th Annu. Meeting Assoc. Comput. Linguistics (ACL), 2019, pp. 4791–4800

    2019

  11. [19]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2021

    2021

  12. [20]

    PIQA: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, R. Le Bras, J. Gao, and Y . Choi, “PIQA: Reasoning about physical commonsense in natural language,” inProc. AAAI Conf. Artif. Intell., vol. 34, no. 05, 2020, pp. 7432–7439

    2020

  13. [21]

    A framework for few-shot language model evaluation,

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muen- nighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framewo...

    2023 doi

  14. [22]

    What does BERT learn about the structure of language?,

    G. Jawahar, B. Sagot, and D. Seddah, “What does BERT learn about the structure of language?,” inProc. 57th Annu. Meeting Assoc. Comput. Linguistics (ACL), Florence, Italy, 2019, pp. 3651–3657

    2019

  15. [23]

    Toy models of superposition,

    N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. Mc- Candlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah, “Toy models of superposition,”arXiv preprint arXiv:2209.10652, 2022

    2022 arXiv

  16. [24]

    LORA-CRAFT: Cross-layer rank adaptation via frozen Tucker decomposition of pre- trained attention weights,

    K. Dewage, M. Pensky, S. De Silva, and S. Mondal, “LORA-CRAFT: Cross-layer rank adaptation via frozen Tucker decomposition of pre- trained attention weights,”arXiv preprint arXiv:2602.17510, 2026

    2026

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.