Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Coarse spectral matching of pretrained weights does not improve GPT-2-style pretraining, even though residual-writing matrices share clear depth trends.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 04:42 UTC pith:CABDRD3N

load-bearing objection Clean negative result: shared residual-writing spectra are real across GPT-2-style models, but coarse magnitude/spectrum init does not beat standard or weight reuse. the 3 major comments →

arxiv 2607.09204 v1 pith:CABDRD3N submitted 2026-07-10 cs.CL cs.LG

Complexity-Guided Component-wise Initialization for Language Model Pretraining

Konstantin Garbers , Nicholas Oh This is my paper
classification cs.CL cs.LG
keywords language model pretrainingtransformer initializationlayerwise diagnosticsspectral analysisFrobenius normeffective-rank entropyGPT-2
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Trained GPT-2-style models repeatedly develop similar layerwise and component-wise weight spectra. The paper asks whether those recurring patterns of scale and singular-value concentration can be copied into a new model's initialization so pretraining starts closer to a trained solution. Across eleven checkpoints that differ in size, language, tokenizer, and corpus, residual-writing matrices grow in total energy with depth while their spectra become more concentrated. Initializers that impose those magnitudes and spectral shapes do change the model's spectral profiles, and some differences survive training, but they do not produce better validation or downstream numbers than ordinary Gaussian initialization. Direct reuse of pretrained weights remains competitive despite a tokenizer and language mismatch. The practical claim is therefore negative but useful: pretrained spectra diagnose real structure, yet coarse component-wise scale and singular-value shape alone are not enough to accelerate or improve pretraining.

Core claim

Shared component-wise spectral trends exist across diverse GPT-2-style checkpoints, especially increasing Frobenius norm and stronger spectral concentration in residual-writing matrices, but initialization schemes that imitate only those magnitudes and singular-value shapes do not yield a corresponding performance advantage over standard initialization, whereas direct pretrained-weight reuse remains competitive.

What carries the argument

Component-wise spectral initialization: layer- and subcomponent-specific target magnitudes for the attention and MLP projections, optionally with a depth-dependent exponential decay of singular values followed by Frobenius rescaling, used to test whether pretrained-like scale and effective-rank profiles are useful starting signals.

Load-bearing premise

That a single training seed, one primary dataset, and one 24-layer GPT-2-medium run per initializer is enough to treat the lack of gain as evidence that coarse spectral matching is not a reliable optimization strategy rather than a setup-specific or underpowered result.

What would settle it

Rerun the same magnitude-plus-spectrum and realistic-scale initializers against the standard GPT-2 baseline across several seeds and longer training horizons, and check whether any of them reliably beat standard initialization on validation perplexity or BLiMP.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper asks whether recurring spectral structure in pretrained GPT-2-style weights can be reused as an initialization signal for pretraining. It first measures Frobenius norm and effective-rank entropy on eleven GPT-2-style checkpoints that vary in size, language, tokenizer, and corpus, and reports shared depth trends, especially rising scale and stronger spectral concentration in residual-writing matrices (W_O, W_down). It then builds component-wise initializers that match pretrained magnitude profiles and, in some variants, singular-value concentration (Magnitude, Mag.+spectrum, Realistic scale), and compares them to Standard, High-std, No-residual-scaling, and Pretrained-reuse baselines on a single 24-layer GPT-2-medium run (90k steps, SlimPajama-6B). The initializers visibly alter spectral diagnostics at init and after training, but do not improve validation/held-out perplexity or downstream accuracy relative to Standard; pretrained-weight reuse remains competitive. The authors conclude that pretrained spectra are useful diagnostics but that coarse scale and singular-value shape alone are not a reliable optimization strategy.

Significance. If the negative result holds, it is a useful, falsifiable contribution: it separates descriptive spectral regularities from causal usefulness for initialization, and it cautions against treating coarse spectral matching as a free lunch for pretraining. The multi-checkpoint diagnostic analysis (eleven models, component-wise residual-writing trends) is a clear strength and is of independent interest for mechanistic and complexity-oriented work on Transformers. The paper is also explicit about what is not claimed (exact spectral replicas, multi-architecture transfer) and about internal-validity limits. The main limitation on significance is that the load-bearing optimization claim rests on single-seed, single-setup full runs, so the result is currently more of a carefully documented non-result than a definitive null.

major comments (3)
  1. §3.2 and Table 3: The central claim that “coarse spectral matching alone is not a reliable optimization strategy” rests on one 90k-step run per initializer (seed 1, SlimPajama-6B, fixed 24-layer GPT-2-medium). Gaps are modest (e.g., Val PPL 22.24 / 22.06 vs Standard 21.68; Magnitude BLiMP 76.2). §4.1 already flags this internal-validity threat. Without multi-seed or multi-dataset confirmation, the negative claim is underpowered relative to how strongly it is stated in the abstract and conclusion. At minimum, report 2–3 seeds for Standard vs Mag.+spectrum / Realistic scale, or reframe the claim as setup-specific evidence rather than a general reliability statement.
  2. §3.1 / Table 2: The Mag.+spectrum rule multiplies singular values by exp(−λ(z) i/n) with λ(z)=5(1−z) before Frobenius rescaling. This decay schedule and the component-wise target-std depth profiles are free design choices (not fit to the cohort curves beyond qualitative imitation). The paper should either (i) ablate λ and the depth profiles, or (ii) show that the imposed init spectra actually match the pretrained cohort in a quantitative sense (e.g., depth-wise correlation or KL on normalized singular-value mass), so that a null result can be attributed to “coarse spectral matching” rather than to a poorly tuned closed form.
  3. §2.1 footnote and component grouping: Treating W_QKV jointly while analyzing W_O separately is acknowledged as a limitation (W_V is functionally closer to W_O). Because the strongest shared trends and the residual-writing interpretation drive both the diagnostics and the initializers, this grouping is load-bearing. A split of Q/K vs V (or QKV vs O with V reported separately) would strengthen the claim that residual-writing matrices are the primary shared pattern.
minor comments (5)
  1. Figures 2–5: Axis labels and panel titles are hard to read in the manuscript text; ensure published figures have legible metric names, depth axes, and strategy legends without relying on color alone.
  2. Table 1 interpretive associations are useful but should be clearly marked as qualitative (as the text later does) so they are not read as causal claims from the spectral plots alone.
  3. §3.2 evaluation suite: BLiMP and ARC/WinoGrande are appropriate; briefly note whether evaluation is zero-shot and whether multiple-choice scoring is length-normalized, for reproducibility.
  4. Related work §5: The connection to residual-depth scaling (GPT-2) and to spectral complexity / memorization work is good; a short pointer to recent depth-dependent init or μP-style scale transfer would situate the component-wise idea more clearly.
  5. Typos / polish: “poem-generation, and story-generation models” list is fine; check consistency of “pre-trained” vs “pretrained” and of W_QKV notation across text and figures.

Circularity Check

0 steps flagged

No circularity: spectral diagnostics from external checkpoints are imposed as initializers and tested on independent held-out metrics; the negative claim is not forced by construction.

full rationale

The paper's chain is observational then experimental, not definitional. Section 2 measures Frobenius norm and effective-rank entropy on eleven external Hugging Face GPT-2-style checkpoints and reports shared depth trends, especially in residual-writing matrices W_O and W_down. Section 3 then constructs coarse Magnitude / Mag.+spectrum / Realistic-scale initializers that imitate those component-wise profiles, trains a fixed 24-layer GPT-2-medium on SlimPajama-6B, and evaluates validation/held-out perplexity plus BLiMP, ARC, and WinoGrande (Table 3). The central claim—that coarse spectral matching is not a reliable optimization strategy—follows from those independent performance numbers (e.g., Mag.+spectrum Val PPL 22.24 vs Standard 21.68), not from re-labeling the fitted spectral targets as success. Pretrained-weight reuse is a separate baseline, not a self-citation uniqueness theorem. There are no self-citations by Garbers/Oh that load-bear the argument, no uniqueness theorem imported from the authors, no fitted parameter renamed as a prediction of a closely related quantity, and no ansatz smuggled in via self-citation. Designing initializers from the diagnosed cohort trends is ordinary hypothesis construction; the evaluation against external benchmarks is self-contained and falsifiable. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on standard spectral diagnostics, a fixed GPT-2 training recipe, and hand-designed closed-form magnitude/spectrum rules derived from observed cohort trends. No new physical entities are introduced; the main free choices are the spectrum-shaping schedule and the experimental setup used to judge reliability.

free parameters (4)
  • spectrum decay schedule lambda(z)=5(1-z)
    Hand-chosen depth-dependent singular-value decay used in Mag.+spectrum; not derived from a uniqueness theorem or external benchmark.
  • component-wise target standard-deviation depth profiles
    Per-layer/per-component magnitude targets for WQKV, Wup, WO, and Wdown are designed to imitate pretrained trends rather than fixed by theory.
  • cohort-derived absolute-scale multipliers
    Realistic-scale rescaling uses multipliers fitted to the pretrained cohort Frobenius RMS per subcomponent.
  • training hyperparameters (lr 3e-4, 90k steps, batch recipe, seed 1)
    The reliability conclusion depends on this specific optimization setup; different horizons or seeds could change rankings among close runs.
axioms (4)
  • domain assumption Frobenius norm and effective-rank entropy are informative static diagnostics of weight-matrix scale and spectral concentration.
    Invoked throughout §2 as the basis for comparing pretrained checkpoints and initializers.
  • domain assumption GPT-2 residual-branch scaling and Gaussian initialization form a fair standard baseline for pretraining comparisons.
    Used as the main control in §3 and Table 2.
  • ad hoc to paper Coarse closed-form matching of component-wise magnitude and singular-value shape is a meaningful test of whether pretrained spectra can be reused as initialization signals.
    Stated design choice in §3.1 and defended as a construct-validity limit in §4.1.
  • domain assumption Shared trends across eleven GPT-2-style checkpoints of varying size/language/tokenizer/corpus indicate architecture-and-optimization regularities rather than corpus-specific accidents.
    Underpins RQ1 and the interpretation in §2.3.
invented entities (2)
  • Mag.+spectrum initializer no independent evidence
    purpose: Impose pretrained-like component magnitudes and stronger spectral concentration via depth-dependent singular-value decay plus Frobenius rescaling.
    A paper-specific construction rather than a previously standard method; evaluated as one proposed initializer in Table 2.
  • Realistic scale initializer no independent evidence
    purpose: Match not only relative spectral shape but also cohort-derived absolute Frobenius RMS per subcomponent.
    Another paper-specific construction used to test whether absolute scale closes the performance gap.

pith-pipeline@v1.1.0-grok45 · 17222 in / 2784 out tokens · 23713 ms · 2026-07-13T04:42:14.199587+00:00 · methodology

0 comments
read the original abstract

Pretrained language models often exhibit structured weight spectra, suggesting that training may repeatedly produce similar layerwise and component-wise organization. We ask whether these recurring spectral patterns can be reused as an initialization signal for GPT-2-style language-model pretraining. First, we analyze eleven pretrained GPT-2-style checkpoints that vary in size, language, tokenizer, and training corpus, measuring Frobenius norm and effective-rank entropy across layers and Transformer subcomponents. The checkpoints show shared depth trends, especially increasing scale and stronger spectral concentration in residual-writing matrices. We then construct initialization schemes that imitate the component-wise magnitudes and spectral profiles of pretrained models, and compare them with several weight initialization methods. These initializers visibly change the model's structural spectral patterns, but the evaluation results do not show a corresponding performance advantage. Pretrained-weight reuse remains competitive, while coarse spectral matching alone is not a reliable optimization strategy. Our results suggest that pretrained spectra are useful diagnostics of trained model structure, but that effective reuse likely requires preserving richer information than component-wise scale and singular-value shape.

Figures

Figures reproduced from arXiv: 2607.09204 by Konstantin Garbers, Nicholas Oh.

Figure 1
Figure 1. Figure 1: Pre-layer-norm GPT-2 block notation. Attention and MLP [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pretrained layerwise Frobenius-norm and effective-entropy trends. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pretrained component-wise Frobenius-norm and effective-entropy trends for attention and MLP projections. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Initialization diagnostic grid. Rows correspond to [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Strategy-matched initialization and final-checkpoint diagnostics. Rows correspond to the initialization strategy used for training. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 2 canonical work pages

  1. [1]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question An- swering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457 [cs.AI] https://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  2. [2]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Das- Sarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah....

    2021

  3. [3]

    Gabriel Franco, Carson Loughridge, and Mark Crovella. 2026. Singular Vectors of Attention Heads Align with Features. arXiv:2602.13524 [cs.LG] doi:10.48550/ arXiv.2602.13524 To be published in ICML 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi...

    doi:10.18653/v1/2022.emnlp-main.3 2022

  5. [5]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. InProceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana...

    doi:10.18653/v1/2021.emnlp-main.446 2021

  6. [6]

    Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 9), Yee Whye Teh and Mike Titterington (Eds.). PMLR, Chia Laguna Resort, Sardinia, Italy, 249–25...

    2010

  7. [7]

    Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText Corpus. http:// Skylion007.github.io/OpenWebTextCorpus. Konstantin Garbers and Nicholas Oh

    2019

  8. [8]

    Pierre Guillou. 2020. GPorTuguese-2 (Portuguese GPT-2 small): a Lan- guage Model for Portuguese text generation (and more NLP tasks...). https://huggingface.co/pierreguillou/gpt2-small-portuguese

    2020

  9. [9]

    Chengxi Guo. 2023. Mymusise/Gpt2-Medium-Chinese · Hugging Face. https://huggingface.co/mymusise/gpt2-medium-chinese

    2023

  10. [10]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InProceedings of the IEEE international conference on computer vision. ICCV, Santiago, Chile, 1026–1034. doi:10.1109/ICCV.2015.123

    doi:10.1109/iccv.2015.123 2015

  11. [11]

    Ting Hua, Xiao Li, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2023. Dynamic Low-rank Estimation for Transformer-based Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9275–9287. doi:10.18653/v1/20...

    work page doi:10.18653/v1/2023.findings-emnlp.621 2023

  12. [12]

    Youcheng Huang, Chen Huang, Duanyu Feng, Wenqiang Lei, and Jiancheng Lv

  13. [13]

    Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austri...

    2025

  14. [14]

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. 2024. Posi- tion: The Platonic Representation Hypothesis. InProceedings of the 41st Interna- tional Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix...

    2024

  15. [15]

    H Toprak Kesgin, M Kaan Yuce, Eren Dogan, M Egemen Uzun, Atahan Uz, H Emre Seyrek, Ahmed Zeer, and M Fatih Amasyali. 2024. Introducing cosmos- GPT: Monolingual training for Turkish language models. In2024 International Conference on INnovations in Intelligent SysTems and Applications (INISTA). IEEE, INISTA, Craiova, Romania, 1–6

    2024

  16. [16]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. FineWeb-Edu: the Finest Collection of Educational Content. doi:10.57967/hf/2497

    doi:10.57967/hf/2497 2024

  17. [17]

    Charles H Martin and Michael W Mahoney. 2021. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.Journal of Machine Learning Research22, 165 (2021), 1–73

    2021

  18. [18]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locat- ing and Editing Factual Associations in GPT.Advances in Neural Information Processing Systems35 (Dec. 2022), 17359–17372

    2022

  19. [19]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer Sentinel Mixture Models. arXiv:1609.07843 [cs.CL]

    Pith/arXiv arXiv 2016

  20. [20]

    Narmeen Fatimah Oozeer, Dhruv Nathawani, Nirmalendu Prakash, Michael Lan, Abir Harrasse, and Amir Abdullah. 2025. Activation space interventions can be transferred between large language models. InProceedings of the 42nd International Conference on Machine Learning(Vancouver, Canada)(ICML’25). JMLR.org, Vancouver, Canada, Article 1890, 75 pages

    2025

  21. [21]

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds....

    doi:10.52202/079017-0970 2024

  22. [22]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog1, 8 (2019), 9

    2019

  23. [23]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi

  24. [24]

    arXiv:1907.10641 [cs.CL] https://arxiv.org/abs/1907.10641

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.10641 [cs.CL] https://arxiv.org/abs/1907.10641

    Pith/arXiv arXiv 1907

  25. [25]

    Kei Sawada, Tianyu Zhao, Makoto Shing, Kentaro Mitsui, Akio Kaga, Yukiya Hono, Toshiaki Wakatsuki, and Koh Mitsuda. 2024. Release of Pre-Trained Models for the Japanese Language. InProceedings of the 2024 Joint International Confer- ence on Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024), Nicoletta Calzolari, Min-Yen Kan, ...

    2024

  26. [26]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hest- ness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and dedupli- cated version of RedPajama. https://www.cerebras.net/blog/slimpajama-a-627b- token-cleaned-and-deduplicated-version-of-redpajama. https://huggingface. co/datasets/cerebras/SlimPajama-627B

    2023

  27. [27]

    Max Staats, Matthias Thamm, and Bernd Rosenow. 2026. Small singular values matter: A random matrix analysis of transformer models.Advances in Neural Information Processing Systems38 (2026), 153545–153573

    2026

  28. [28]

    Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End- to-end memory networks. InProceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2(Montreal, Canada)(NIPS’15). MIT Press, Cambridge, MA, USA, 2440–2448

    2015

  29. [29]

    Pranav Vadrevu. 2021. Pranavpsv/Gpt2-Genre-Story-Generator · Hugging Face. https://huggingface.co/pranavpsv/gpt2-genre-story-generator

    2021

  30. [30]

    Nha Nguyen Van. 2025. NlpHUST/Gpt2-Vietnamese · Hugging Face. https://huggingface.co/NlpHUST/gpt2-vietnamese

    2025

  31. [31]

    Wenxuan Wang and Zhaopeng Tu. 2020. Rethinking the Value of Transformer Components. InProceedings of the 28th International Conference on Computational Linguistics, Donia Scott, Nuria Bel, and Chengqing Zong (Eds.). International Committee on Computational Linguistics, Barcelona, Spain (Online), 6019–6029. doi:10.18653/v1/2020.coling-main.529

    doi:10.18653/v1/2020.coling-main.529 2020

  32. [32]

    Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. 2025. SVD- LLM V2: Optimizing Singular Value Truncation for Large Language Model Com- pression. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), Luis Chiruzzo, Alan...

    2025

  33. [33]

    doi:10.18653/v1/2025.naacl-long.217

    doi:10.18653/v1/2025.naacl-long.217 2025

  34. [34]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng- Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English.Transactions of the Association for Computational Linguistics8 (2020), 377–392. arXiv:https://doi.org/10.1162/tacl_a_00321 doi:10. 1162/tacl_a_00321

    doi:10.1162/tacl_a_00321 2020

  35. [35]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement De- langue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-th...

    doi:10.18653/v1/2020.emnlp-demos.6 2020

  36. [36]

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. 2025. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. arXiv:2310.05175 [cs.LG] https://arxiv.org/abs/2310.05175

    Pith/arXiv arXiv 2025

  37. [37]

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Rama- puram, Yizhe Zhang, Jiatao Gu, and Josh Susskind. 2023. Stabilizing transformer training by preventing attention entropy collapse. InProceedings of the 40th International Conference on Machine Learning (ICML’23). JMLR.org, Honolulu, Hawaii, USA, Article 1709, 34 pages

    2023

  38. [38]

    Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang, Yaoyu Zhang, and Zhi-Qin John Xu. 2026. Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers.IEEE Transactions on Pattern Analysis and Machine Intelligence48, 4 (2026), 4336–4349. doi:10.1109/TPAMI.2025.3646483

    work page doi:10.1109/tpami.2025.3646483 2026

  39. [39]

    Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang, Yaoyu Zhang, and Zhi-Qin John Xu. 2024. Initialization is critical to whether transformers fit composite functions by reasoning or memorizing. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, A...

    2024

  40. [40]

    Tianyu Zhao and Kei Sawada. 2021. rinna/japanese-gpt2-medium. https:// huggingface.co/rinna/japanese-gpt2-medium

    2021

  41. [41]

    Zhe Zhao, Yudong Li, Cheng Hou, Jing Zhao, Rong Tian, Weijie Liu, Yiren Chen, Ningyuan Sun, Haoyan Liu, Weiquan Mao, et al. 2023. Tencentpretrain: A scalable and flexible toolkit for pre-training models of different modalities. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). ACL,...

    2023

  42. [42]

    Markov, Vladislav Mikhailov, and Alena Fenogenova

    Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, and Alena Fenogenova. 2024. A Family of Pretrained Transformer Language Models for Russian. InProceedings of the 2024 Joint International Con...

    2024