Pith. sign in

REVIEW 2 major objections 2 minor 228 references

Larger transformer language models delay the onset of plasticity loss but do not prevent it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Plasticity loss in GPT-style transformers on multilingual tasks persists from 5M to 314M parameters, follows a sublinear scaling law with model size, and occurs in both continual and stationary settings.

T0 review reviewed 2026-06-25 challenge →

load-bearing objection Plasticity loss in transformers scales sublinearly and persists in stationary training, but the evidence depends on a probe whose validity as a plasticity measure is unclear. the 2 major comments →

arxiv 2606.24752 v1 pith:YGCU5UV5 submitted 2026-06-23 cs.AI

Can Scale Save Us From Plasticity Loss in Large Language Models?

J. Fernando Hernandez-Garcia , Tom\'as Figliolia , Beren Millidge This is my paper
classification cs.AI
keywords plasticity losslarge language modelscontinual learningscaling lawstransformersmultilingual trainingstationary trainingadaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether simply making GPT-style models bigger solves the long-known problem of networks losing the ability to learn new information after extended training. Researchers train models from 5 million to 314 million non-embedding parameters on multilingual data and measure how well they can still adapt by tracking performance drops on a held-out Vietnamese task. Plasticity loss appears at all tested sizes, with its appearance delayed according to a sublinear scaling law. The same loss occurs even when the training data distribution stays fixed with no abrupt task switches. These findings indicate that parameter scaling by itself will not keep large language models adaptable after sufficiently long training.

Core claim

In GPT-style Transformer models trained on a multilingual continual learning problem, evidence of plasticity loss appears across scales from 5M to 314M non-embedding parameters as measured by deterioration on a held-out Vietnamese probing task. The onset of this loss follows a predictable scaling law that grows sublinearly with model size. The same deterioration is observed under stationary multilingual training without task changes, indicating that the phenomenon is not limited to abrupt distributional shifts.

What carries the argument

Deterioration on a held-out Vietnamese probing task, used to quantify plasticity loss, together with the sublinear scaling law that describes when this deterioration begins as a function of model size.

Load-bearing premise

Drops in performance on the held-out Vietnamese task serve as a sufficient and representative indicator of plasticity loss, and the multilingual regimes and model sizes examined reflect typical large language model training.

What would settle it

Training models substantially larger than 314 million parameters on the same multilingual setup and finding no performance deterioration on the Vietnamese probing task after long training would contradict the claim that scale alone cannot eliminate plasticity loss.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Even models with hundreds of millions of parameters will eventually require additional mechanisms to maintain efficient adaptation after prolonged training.
  • Plasticity loss arises in both continual learning with task changes and in stationary training on a fixed data distribution.
  • Increasing parameter count postpones the measurable onset of plasticity loss according to a sublinear relationship.
  • Natural-language transformers will lose the capacity for efficient adaptation to new data after sufficiently long training regardless of scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Methods other than pure scaling, such as architectural changes or regularization techniques, will likely be needed to sustain long-term adaptability in deployed language models.
  • The sublinear delay pattern suggests that practical training runs of current-generation models may already encounter adaptation limits before reaching the largest feasible sizes.
  • Similar measurements on non-language domains could reveal whether the scaling behavior is specific to text or applies more broadly to neural networks.
  • Monitoring probing-task performance during pretraining could serve as an early warning signal for when plasticity begins to degrade.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper studies plasticity loss in GPT-style Transformer language models (5M–314M non-embedding parameters) trained on multilingual continual learning. It reports consistent deterioration on a held-out Vietnamese probing task as evidence of plasticity loss, identifies a sublinear scaling law governing the onset of this loss with model size, and finds similar deterioration under stationary multilingual training, concluding that scale delays but does not eliminate the problem.

Significance. If the Vietnamese probe is shown to measure reduced adaptation capacity rather than other forms of drift, the sublinear scaling result and the stationary-training observation would indicate that parameter count alone is unlikely to solve plasticity loss in natural-language domains. This would strengthen the case for targeted interventions beyond scale in continual-learning LLM research.

major comments (2)
  1. [Abstract] Abstract and §3 (experimental setup): the central claim equates deterioration on the held-out Vietnamese probing task with plasticity loss, yet the standard definition requires demonstrating slower adaptation rates on a new task after prolonged training; static evaluation on never-seen data may instead reflect representation drift or multilingual interference, rendering the scaling law and stationary-training conclusions ambiguous.
  2. [§4] §4 (results): the reported sublinear scaling of plasticity-loss onset with model size is presented without error bars, confidence intervals, or controls for training-procedure confounds, so it is unclear whether the functional form is robust or an artifact of the specific multilingual regime and model sizes (5M–314M).
minor comments (2)
  1. [Abstract] The abstract states the models have 5M–314M non-embedding parameters but does not specify embedding sizes or total parameter counts, which would aid reproducibility.
  2. No reference is made to prior work that directly measures adaptation rates (e.g., via fine-tuning curves) after long pre-training; adding such citations would clarify how the probe differs from standard plasticity metrics.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract] Abstract and §3 (experimental setup): the central claim equates deterioration on the held-out Vietnamese probing task with plasticity loss, yet the standard definition requires demonstrating slower adaptation rates on a new task after prolonged training; static evaluation on never-seen data may instead reflect representation drift or multilingual interference, rendering the scaling law and stationary-training conclusions ambiguous.

    Authors: We appreciate the referee's point on definitional precision. Our manuscript uses deterioration on the held-out Vietnamese probe as a practical proxy for plasticity loss in the multilingual setting, following the measurement approach in the cited prior work on language models. We acknowledge that this static evaluation does not directly demonstrate slower adaptation rates on a new task and could be influenced by drift or interference. To resolve the ambiguity, we will revise the abstract and §3 to explicitly frame the probe result as a proxy measure, add a limitations paragraph discussing alternative interpretations, and note that future work could include direct adaptation-rate experiments. This is a partial revision. revision: partial

  2. Referee: [§4] §4 (results): the reported sublinear scaling of plasticity-loss onset with model size is presented without error bars, confidence intervals, or controls for training-procedure confounds, so it is unclear whether the functional form is robust or an artifact of the specific multilingual regime and model sizes (5M–314M).

    Authors: We agree that the scaling-law figure and analysis would be strengthened by statistical rigor. In the revised manuscript we will recompute the onset points with error bars and confidence intervals obtained from multiple independent runs, and we will add a paragraph in §4 discussing controls for training-procedure variables (e.g., learning-rate schedules, data ordering) to show that the sublinear functional form is not an artifact of the particular regime. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical measurements on held-out task with independent scaling observation

full rationale

The paper reports direct empirical measurements of performance deterioration on a held-out Vietnamese probing task across model sizes, plus an observed sublinear scaling of onset with parameter count. No equations, self-citations, or ansatzes are invoked that define the target quantity in terms of itself or rename a fitted parameter as a prediction. The central claims rest on external data evaluation rather than any reduction to the paper's own inputs by construction. This is the standard non-circular outcome for an empirical scaling study.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests on the empirical observation of performance deterioration on the Vietnamese probing task and the interpretation of that deterioration as plasticity loss. No free parameters, axioms, or invented entities are explicitly introduced in the abstract; the scaling law is described as predictable but its functional form and fitted constants are not detailed.

reviewed 2026-06-25 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Scale Save Us From Plasticity Loss in Large Language Models?." pith.science (2026). https://pith.science/paper/YGCU5UV5

@misc{pith2026260624752,
  author       = {Pith},
  title        = {Pith review of: Can Scale Save Us From Plasticity Loss in Large Language Models?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGCU5UV5}},
  note         = {Machine review of arXiv:2606.24752}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains. To determine whether loss of plasticity remains a problem in the modern transformer-based LLM paradigm, we study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem. Consistent with prior work, we find evidence of plasticity loss across models ranging from 5M to 314M non-embedding parameters, as measured by deterioration on a held-out Vietnamese probing task. We further find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size. These results suggest that larger models may delay the measurable effects of plasticity loss, but that increasing parameter count alone is likely to be insufficient to completely prevent it. We also find evidence of plasticity loss under stationary multilingual training, challenging the view that the phenomenon is exclusive to continual learning with abrupt task changes. Overall, our results suggest that even large Transformer language models trained on natural-language will eventually lose the ability to efficiently adapt to new data after sufficiently long training, in both continual and stationary settings.

Figures

Figures reproduced from arXiv: 2606.24752 by Beren Millidge, J. Fernando Hernandez-Garcia, Tom\'as Figliolia.

Figure 1
Figure 1. Figure 1: Multilingual continual learning problem. Models are [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Percentage change in validation-loss AUC on the probing task relative to the first cycle for models ranging from 5M to [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Percentage change in validation-loss AUC on the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Log-log model predicting the onset of plasticity loss, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average parameter magnitude and percentage of dormant units for the (a) 12M model, (b) 53M model, and (c) 106M [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Percentage of dormant units per layer in the (a) 53M [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Top panels: Percentage of collapsed and lazy attention heads for the (a) 12M model, (b) 53M model, and (c) 106M [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Diagram of the specific Transformer architecture used [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Extended probing in the Multilingual Continual Learning Problem. Each point is the AUC of the validation loss during [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparison between different models for predicting the onset of plasticity loss. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Average weight magnitude, percentage of dormant units, percentage of collapsed heads, and percentage of lazy heads [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Correlates of loss of plasticity for the 5M, 12M, and 27M models trained on the Multilingual Stationary Learning [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

228 extracted references · 8 canonical work pages · 2 internal anchors

  1. [1]

    , author=

    Age of acquisition effects in adult lexical processing reflect loss of plasticity in maturing systems: insights from connectionist networks. , author=. 2000 , publisher=

    2000

  2. [2]

    The Early Word Catches the Weights , url =

    Smith, Mark and Cottrell, Garrison and Anderson, Karen , booktitle =. The Early Word Catches the Weights , url =

  3. [3]

    On Warm-Starting Neural Network Training , year =

    Ash, Jordan and Adams, Ryan P , booktitle =. On Warm-Starting Neural Network Training , year =

  4. [4]

    39th International Conference on Machine Learning , pages=

    The Primacy Bias in Deep Reinforcement Learning , author=. 39th International Conference on Machine Learning , pages=

  5. [5]

    10th International Conference on Learning Representations , year=

    Understanding and Preventing Capacity Loss in Reinforcement Learning , author=. 10th International Conference on Learning Representations , year=

  6. [6]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Understanding Plasticity in Neural Networks , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

  7. [7]

    Nature , volume=

    Loss of plasticity in deep continual learning , author=. Nature , volume=. 2024 , publisher=

    2024

  8. [8]

    41st International Conference on Machine Learning , year=

    Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks , author=. 41st International Conference on Machine Learning , year=

  9. [9]

    2024 , note=

    Curvature Explains Loss of Plasticity , author=. 2024 , note=

    2024

  10. [10]

    Normalization and effective learning rates in reinforcement learning , author=

  11. [11]

    12th International Conference on Learning Representations , year=

    Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning , author=. 12th International Conference on Learning Representations , year=

  12. [12]

    Rupam Mahmood , title=

    Mohamed Elsayed and Qingfeng Lan and Clare Lyle and A. Rupam Mahmood , title=. RLJ , volume=. 2024 , cdate=

    2024

  13. [13]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Parseval Regularization for Continual Reinforcement Learning , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  14. [14]

    3rd Conference on Lifelong Learning Agents , year=

    Disentangling the causes of plasticity loss in neural networks , author=. 3rd Conference on Lifelong Learning Agents , year=

  15. [15]

    3rd Conference on Lifelong Learning Agents , year=

    Maintaining plasticity in continual learning via regenerative regularization , author=. 3rd Conference on Lifelong Learning Agents , year=

  16. [16]

    4th Conference on Lifelong Learning Agents , year=

    Reinitializing weights vs units for maintaining plasticity in neural networks , author=. 4th Conference on Lifelong Learning Agents , year=

  17. [17]

    The Thirteenth International Conference on Learning Representations , year=

    Self-Normalized Resets for Plasticity in Continual Learning , author=. The Thirteenth International Conference on Learning Representations , year=

  18. [18]

    The Thirteenth International Conference on Learning Representations , year=

    Learning Continually by Spectral Regularization , author=. The Thirteenth International Conference on Learning Representations , year=

  19. [19]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  20. [20]

    The Fourteenth International Conference on Learning Representations , year=

    Forget Forgetting: Continual Learning in a World of Abundant Memory , author=. The Fourteenth International Conference on Learning Representations , year=

  21. [21]

    Forty-second International Conference on Machine Learning , year=

    Overtrained Language Models Are Harder to Fine-Tune , author=. Forty-second International Conference on Machine Learning , year=

  22. [22]

    Do Neural Networks Lose Plasticity in a Gradually Changing World? , author=

  23. [23]

    The Fourteenth International Conference on Learning Representations , year=

    Activation Function Design Sustains Plasticity in Continual Learning , author=. The Fourteenth International Conference on Learning Representations , year=

  24. [24]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Plasticity as the Mirror of Empowerment , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  25. [25]

    Resetting the Optimizer in Deep

    Kavosh Asadi and Rasool Fakoor and Shoham Sabach , booktitle=. Resetting the Optimizer in Deep. 2023 , url=

    2023

  26. [26]

    2026 , publisher =

    Han, Tessa and Bordt, Sebastian and Zhang, Hanlin and Kakade, Sham , title=. 2026 , publisher =

    2026

  27. [27]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism , author=

  28. [28]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  29. [29]

    Language Models are Unsupervised Multitask Learners , author =

  30. [30]

    2018 , publisher=

    Improving language understanding by generative pre-training , author=. 2018 , publisher=

    2018

  31. [31]

    , title =

    Raffel, Colin and Shazeer, Noam and Roberts, Adam and Lee, Katherine and Narang, Sharan and Matena, Michael and Zhou, Yanqi and Li, Wei and Liu, Peter J. , title =. J. Mach. Learn. Res. , month = jan, articleno =. 2020 , issue_date =

    2020

  32. [32]

    Qwen3 technical report , author=

  33. [33]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  34. [34]

    Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =

    Yang, Ge and Hu, Edward and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , booktitle =. Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =

  35. [35]

    When Attention Collapses: How Degenerate Layers in

    Sunny Sanyal and Ravid Shwartz-Ziv and Alex Dimakis and sujay sanghavi , journal=. When Attention Collapses: How Degenerate Layers in. 2026 , url=

    2026

  36. [36]

    Proceedings of the 40th International Conference on Machine Learning , articleno =

    Zhai, Shuangfei and Likhomanenko, Tatiana and Littwin, Etai and Busbridge, Dan and Ramapuram, Jason and Zhang, Yizhe and Gu, Jiatao and Susskind, Josh , title =. Proceedings of the 40th International Conference on Machine Learning , articleno =. 2023 , publisher =

    2023

  37. [37]

    The Twelfth International Conference on Learning Representations , year=

    Efficient Streaming Language Models with Attention Sinks , author=. The Twelfth International Conference on Learning Representations , year=

  38. [38]

    The Twelfth International Conference on Learning Representations , year=

    Vision Transformers Need Registers , author=. The Twelfth International Conference on Learning Representations , year=

  39. [39]

    A Comprehensive Survey of Continual Learning: Theory, Method and Application , year=

    Wang, Liyuan and Zhang, Xingxing and Su, Hang and Zhu, Jun , journal=. A Comprehensive Survey of Continual Learning: Theory, Method and Application , year=

  40. [40]

    Frontiers in psychology , volume=

    The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects , author=. Frontiers in psychology , volume=. 2013 , publisher=

    2013

  41. [41]

    Psychology of Learning and Motivation , volume=

    Catastrophic interference in connectionist networks: The sequential learning problem , author=. Psychology of Learning and Motivation , volume=. 1989 , publisher=

    1989

  42. [42]

    French , keywords =

    Robert M. French , keywords =. Catastrophic forgetting in connectionist networks , journal =. 1999 , issn =

    1999

  43. [43]

    Proceedings of the national academy of sciences , volume=

    Overcoming catastrophic forgetting in neural networks , author=. Proceedings of the national academy of sciences , volume=. 2017 , publisher=

    2017

  44. [44]

    and Nguyen, Thien Huu

    Nguyen, Thuat and Nguyen, Chien Van and Lai, Viet Dac and Man, Hieu and Ngo, Nghia Trung and Dernoncourt, Franck and Rossi, Ryan A. and Nguyen, Thien Huu. C ultura X : A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resou...

    2024

  45. [45]

    An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks , author=

  46. [46]

    International conference on machine learning , pages=

    Continual learning through synaptic intelligence , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  47. [47]

    Proceedings of the European conference on computer vision (ECCV) , pages=

    Memory aware synapses: Learning what (not) to forget , author=. Proceedings of the European conference on computer vision (ECCV) , pages=

  48. [48]

    Nature Machine Intelligence , volume=

    Three types of incremental learning , author=. Nature Machine Intelligence , volume=. 2022 , publisher=

    2022

  49. [49]

    INFORMS Journal on Applied Analytics , volume=

    A better match for drivers and riders: Reinforcement learning at lyft , author=. INFORMS Journal on Applied Analytics , volume=. 2024 , publisher=

    2024

  50. [50]

    Machine Learning , pages=

    GVFs in the real world: making predictions online for water treatment , author=. Machine Learning , pages=. 2023 , publisher=

    2023

  51. [51]

    Proceedings of machine learning and systems , volume=

    What is the state of neural network pruning? , author=. Proceedings of machine learning and systems , volume=

  52. [52]

    and Pechenizkiy, Mykola and Mocanu, Decebal Constantin , title =

    Grooten, Bram and Sokar, Ghada and Dohare, Shibhansh and Mocanu, Elena and Taylor, Matthew E. and Pechenizkiy, Mykola and Mocanu, Decebal Constantin , title =. Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages =. 2023 , isbn =

    2023

  53. [53]

    The Twelfth International Conference on Learning Representations , year=

    Dynamic Sparse Training with Structured Sparsity , author=. The Twelfth International Conference on Learning Representations , year=

  54. [54]

    Deep Rewiring: Training very sparse deep networks , author=

  55. [55]

    2018 , publisher=

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science , author=. 2018 , publisher=

    2018

  56. [56]

    International conference on machine learning , pages=

    Rigging the lottery: Making all tickets winners , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  57. [57]

    Advances in neural information processing systems , volume=

    Deconstructing lottery tickets: Zeros, signs, and the supermask , author=. Advances in neural information processing systems , volume=

  58. [58]

    2019 , url=

    Namhoon Lee and Thalaiyasingam Ajanthan and Philip Torr , booktitle=. 2019 , url=

    2019

  59. [59]

    Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages =

    Understanding the difficulty of training deep feedforward neural networks , author =. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages =. 2010 , volume =

    2010

  60. [60]

    IEEE International Conference on Computer Vision , pages=

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification , author=. IEEE International Conference on Computer Vision , pages=

  61. [61]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Improving Transformer Optimization Through Better Initialization , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    2020

  62. [62]

    Neural machine translation by jointly learning to align and translate , author=

  63. [63]

    Proceedings of the IEEE , volume=

    Gradient-based learning applied to document recognition , author=. Proceedings of the IEEE , volume=. 2002 , publisher=

    2002

  64. [64]

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages=

    2019

  65. [65]

    Advances in neural information processing systems , year=

    Language models are few-shot learners , author=. Advances in neural information processing systems , year=

  66. [66]

    9th International Conference on Learning Representations , year=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. 9th International Conference on Learning Representations , year=

  67. [67]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  68. [68]

    ICML Workshop on Deep Learning for Audio, Speech and Language Processing , year=

    Rectifier nonlinearities improve neural network acoustic models , author=. ICML Workshop on Deep Learning for Audio, Speech and Language Processing , year=

  69. [69]

    Gaussian Error Linear Units (GELUs)

    Gaussian Error Linear Units (GELUs) , author=. arXiv preprint arXiv:1606.08415 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  70. [70]

    Searching for Activation Functions

    Swish: a self-gated activation function , author=. arXiv preprint arXiv:1710.05941 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  71. [71]

    IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Deep residual learning for image recognition , author=. IEEE Conference on Computer Vision and Pattern Recognition , pages=

  72. [72]

    Weiaicunzai , title =

  73. [73]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Scratching visual transformer's back with uniform attention , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  74. [74]

    AAAI , volume=

    Adapting bias by gradient descent: An incremental version of delta-bar-delta , author=. AAAI , volume=. 1992 , organization=

    1992

  75. [75]

    arXiv preprint arXiv:2402.02342 , year=

    Metaoptimize: A framework for optimizing step sizes and other meta-parameters , author=. arXiv preprint arXiv:2402.02342 , year=

    work page arXiv

  76. [76]

    Improving neural networks by preventing co-adaptation of feature detectors , author=

  77. [77]

    The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks , author=

  78. [78]

    Scaling laws for neural language models , author=

  79. [79]

    2004 , publisher=

    Convex Optimization , author=. 2004 , publisher=

    2004

  80. [80]

    On the Origin of Implicit Regularization in Stochastic Gradient Descent , author=

Showing first 80 references.

This paper was first reviewed by grok-4.3 on June 25, 2026.