Pith. sign in

REVIEW 2 major objections 2 minor 228 references

Can Scale Save Us From Plasticity Loss in Large Language Models?

T0 review · 2 major / 2 minor · reviewed 2026-06-25 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Larger transformer language models delay the onset of plasticity loss but do not prevent it.

desk verdict Plasticity loss in transformers scales sublinearly and persists in stationary training, but the evidence depends on a probe whose validity as a plasticity measure is unclear. read the letter →

arxiv 2606.24752 v1 pith:YGCU5UV5 submitted 2026-06-23 cs.AI

J. Fernando Hernandez-Garcia , Tomás Figliolia , Beren Millidge This is my paper
classification cs.AI
keywords plasticitylosslargelanguagemodelscontinuallearningscalinglawstransformersmultilingualtrainingstationaryadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether simply making GPT-style models bigger solves the long-known problem of networks losing the ability to learn new information after extended training. Researchers train models from 5 million to 314 million non-embedding parameters on multilingual data and measure how well they can still adapt by tracking performance drops on a held-out Vietnamese task. Plasticity loss appears at all tested sizes, with its appearance delayed according to a sublinear scaling law. The same loss occurs even when the training data distribution stays fixed with no abrupt task switches. These findings indicate that parameter scaling by itself will not keep large language models adaptable after sufficiently long training.

What carries the argument

Deterioration on a held-out Vietnamese probing task, used to quantify plasticity loss, together with the sublinear scaling law that describes when this deterioration begins as a function of model size.

What would settle it

Training models substantially larger than 314 million parameters on the same multilingual setup and finding no performance deterioration on the Vietnamese probing task after long training would contradict the claim that scale alone cannot eliminate plasticity loss.

Watch

Extended reading notes

Core claim

In GPT-style Transformer models trained on a multilingual continual learning problem, evidence of plasticity loss appears across scales from 5M to 314M non-embedding parameters as measured by deterioration on a held-out Vietnamese probing task. The onset of this loss follows a predictable scaling law that grows sublinearly with model size. The same deterioration is observed under stationary multilingual training without task changes, indicating that the phenomenon is not limited to abrupt distributional shifts.

Load-bearing premise

Drops in performance on the held-out Vietnamese task serve as a sufficient and representative indicator of plasticity loss, and the multilingual regimes and model sizes examined reflect typical large language model training.

Editorial extensions

If this is right

  • Even models with hundreds of millions of parameters will eventually require additional mechanisms to maintain efficient adaptation after prolonged training.
  • Plasticity loss arises in both continual learning with task changes and in stationary training on a fixed data distribution.
  • Increasing parameter count postpones the measurable onset of plasticity loss according to a sublinear relationship.
  • Natural-language transformers will lose the capacity for efficient adaptation to new data after sufficiently long training regardless of scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Methods other than pure scaling, such as architectural changes or regularization techniques, will likely be needed to sustain long-term adaptability in deployed language models.
  • The sublinear delay pattern suggests that practical training runs of current-generation models may already encounter adaptation limits before reaching the largest feasible sizes.
  • Similar measurements on non-language domains could reveal whether the scaling behavior is specific to text or applies more broadly to neural networks.
  • Monitoring probing-task performance during pretraining could serve as an early warning signal for when plasticity begins to degrade.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper studies plasticity loss in GPT-style Transformer language models (5M–314M non-embedding parameters) trained on multilingual continual learning. It reports consistent deterioration on a held-out Vietnamese probing task as evidence of plasticity loss, identifies a sublinear scaling law governing the onset of this loss with model size, and finds similar deterioration under stationary multilingual training, concluding that scale delays but does not eliminate the problem.

Significance. If the Vietnamese probe is shown to measure reduced adaptation capacity rather than other forms of drift, the sublinear scaling result and the stationary-training observation would indicate that parameter count alone is unlikely to solve plasticity loss in natural-language domains. This would strengthen the case for targeted interventions beyond scale in continual-learning LLM research.

major comments (2)
  1. [Abstract] Abstract and §3 (experimental setup): the central claim equates deterioration on the held-out Vietnamese probing task with plasticity loss, yet the standard definition requires demonstrating slower adaptation rates on a new task after prolonged training; static evaluation on never-seen data may instead reflect representation drift or multilingual interference, rendering the scaling law and stationary-training conclusions ambiguous.
  2. [§4] §4 (results): the reported sublinear scaling of plasticity-loss onset with model size is presented without error bars, confidence intervals, or controls for training-procedure confounds, so it is unclear whether the functional form is robust or an artifact of the specific multilingual regime and model sizes (5M–314M).
minor comments (2)
  1. [Abstract] The abstract states the models have 5M–314M non-embedding parameters but does not specify embedding sizes or total parameter counts, which would aid reproducibility.
  2. No reference is made to prior work that directly measures adaptation rates (e.g., via fine-tuning curves) after long pre-training; adding such citations would clarify how the probe differs from standard plasticity metrics.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract] Abstract and §3 (experimental setup): the central claim equates deterioration on the held-out Vietnamese probing task with plasticity loss, yet the standard definition requires demonstrating slower adaptation rates on a new task after prolonged training; static evaluation on never-seen data may instead reflect representation drift or multilingual interference, rendering the scaling law and stationary-training conclusions ambiguous.

    Authors: We appreciate the referee's point on definitional precision. Our manuscript uses deterioration on the held-out Vietnamese probe as a practical proxy for plasticity loss in the multilingual setting, following the measurement approach in the cited prior work on language models. We acknowledge that this static evaluation does not directly demonstrate slower adaptation rates on a new task and could be influenced by drift or interference. To resolve the ambiguity, we will revise the abstract and §3 to explicitly frame the probe result as a proxy measure, add a limitations paragraph discussing alternative interpretations, and note that future work could include direct adaptation-rate experiments. This is a partial revision. revision: partial

  2. Referee: [§4] §4 (results): the reported sublinear scaling of plasticity-loss onset with model size is presented without error bars, confidence intervals, or controls for training-procedure confounds, so it is unclear whether the functional form is robust or an artifact of the specific multilingual regime and model sizes (5M–314M).

    Authors: We agree that the scaling-law figure and analysis would be strengthened by statistical rigor. In the revised manuscript we will recompute the onset points with error bars and confidence intervals obtained from multiple independent runs, and we will add a paragraph in §4 discussing controls for training-procedure variables (e.g., learning-rate schedules, data ordering) to show that the sublinear functional form is not an artifact of the particular regime. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical measurements on held-out task with independent scaling observation

full rationale

The paper reports direct empirical measurements of performance deterioration on a held-out Vietnamese probing task across model sizes, plus an observed sublinear scaling of onset with parameter count. No equations, self-citations, or ansatzes are invoked that define the target quantity in terms of itself or rename a fitted parameter as a prediction. The central claims rest on external data evaluation rather than any reduction to the paper's own inputs by construction. This is the standard non-circular outcome for an empirical scaling study.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The central claim rests on the empirical observation of performance deterioration on the Vietnamese probing task and the interpretation of that deterioration as plasticity loss. No free parameters, axioms, or invented entities are explicitly introduced in the abstract; the scaling law is described as predictable but its functional form and fitted constants are not detailed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Scale Save Us From Plasticity Loss in Large Language Models?." pith.science (2026). https://pith.science/paper/YGCU5UV5

@misc{pith2026260624752,
  author       = {Pith},
  title        = {Pith review of: Can Scale Save Us From Plasticity Loss in Large Language Models?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGCU5UV5}},
  note         = {Machine review of arXiv:2606.24752}
}
read the original abstract

The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains. To determine whether loss of plasticity remains a problem in the modern transformer-based LLM paradigm, we study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem. Consistent with prior work, we find evidence of plasticity loss across models ranging from 5M to 314M non-embedding parameters, as measured by deterioration on a held-out Vietnamese probing task. We further find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size. These results suggest that larger models may delay the measurable effects of plasticity loss, but that increasing parameter count alone is likely to be insufficient to completely prevent it. We also find evidence of plasticity loss under stationary multilingual training, challenging the view that the phenomenon is exclusive to continual learning with abrupt task changes. Overall, our results suggest that even large Transformer language models trained on natural-language will eventually lose the ability to efficiently adapt to new data after sufficiently long training, in both continual and stationary settings.

Figures

Figures reproduced from arXiv: 2606.24752 by the authors.

Figure 1
Figure 1. Multilingual continual learning problem. Models are [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Percentage change in validation-loss AUC on the probing task relative to the first cycle for models ranging from 5M to [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Percentage change in validation-loss AUC on the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (9 more)
Figure 3
Figure 3. Figure 3: Log-log model predicting the onset of plasticity loss, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]
Figure 5
Figure 5. Figure 5: Average parameter magnitude and percentage of dormant units for the (a) 12M model, (b) 53M model, and (c) 106M [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Percentage of dormant units per layer in the (a) 53M [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Top panels: Percentage of collapsed and lazy attention heads for the (a) 12M model, (b) 53M model, and (c) 106M [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Diagram of the specific Transformer architecture used [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Extended probing in the Multilingual Continual Learning Problem. Each point is the AUC of the validation loss during [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Comparison between different models for predicting the onset of plasticity loss. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Average weight magnitude, percentage of dormant units, percentage of collapsed heads, and percentage of lazy heads [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Correlates of loss of plasticity for the 5M, 12M, and 27M models trained on the Multilingual Stationary Learning [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

228 extracted references · 8 canonical work pages

  1. [1]

    , author=

    Age of acquisition effects in adult lexical processing reflect loss of plasticity in maturing systems: insights from connectionist networks. , author=. 2000 , publisher=

    2000

  2. [2]

    The Early Word Catches the Weights , url =

    Smith, Mark and Cottrell, Garrison and Anderson, Karen , booktitle =. The Early Word Catches the Weights , url =

  3. [3]

    On Warm-Starting Neural Network Training , year =

    Ash, Jordan and Adams, Ryan P , booktitle =. On Warm-Starting Neural Network Training , year =

  4. [4]

    39th International Conference on Machine Learning , pages=

    The Primacy Bias in Deep Reinforcement Learning , author=. 39th International Conference on Machine Learning , pages=

  5. [5]

    10th International Conference on Learning Representations , year=

    Understanding and Preventing Capacity Loss in Reinforcement Learning , author=. 10th International Conference on Learning Representations , year=

  6. [6]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Understanding Plasticity in Neural Networks , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

  7. [7]

    Nature , volume=

    Loss of plasticity in deep continual learning , author=. Nature , volume=. 2024 , publisher=

    2024

  8. [8]

    41st International Conference on Machine Learning , year=

    Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks , author=. 41st International Conference on Machine Learning , year=

Show all 228 references
  1. [9]

    2024 , note=

    Curvature Explains Loss of Plasticity , author=. 2024 , note=

    2024

  2. [10]

    Normalization and effective learning rates in reinforcement learning , author=

  3. [11]

    12th International Conference on Learning Representations , year=

    Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning , author=. 12th International Conference on Learning Representations , year=

  4. [12]

    Rupam Mahmood , title=

    Mohamed Elsayed and Qingfeng Lan and Clare Lyle and A. Rupam Mahmood , title=. RLJ , volume=. 2024 , cdate=

    2024

  5. [13]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Parseval Regularization for Continual Reinforcement Learning , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  6. [14]

    3rd Conference on Lifelong Learning Agents , year=

    Disentangling the causes of plasticity loss in neural networks , author=. 3rd Conference on Lifelong Learning Agents , year=

  7. [15]

    3rd Conference on Lifelong Learning Agents , year=

    Maintaining plasticity in continual learning via regenerative regularization , author=. 3rd Conference on Lifelong Learning Agents , year=

  8. [16]

    4th Conference on Lifelong Learning Agents , year=

    Reinitializing weights vs units for maintaining plasticity in neural networks , author=. 4th Conference on Lifelong Learning Agents , year=

  9. [17]

    The Thirteenth International Conference on Learning Representations , year=

    Self-Normalized Resets for Plasticity in Continual Learning , author=. The Thirteenth International Conference on Learning Representations , year=

  10. [18]

    The Thirteenth International Conference on Learning Representations , year=

    Learning Continually by Spectral Regularization , author=. The Thirteenth International Conference on Learning Representations , year=

  11. [19]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  12. [20]

    The Fourteenth International Conference on Learning Representations , year=

    Forget Forgetting: Continual Learning in a World of Abundant Memory , author=. The Fourteenth International Conference on Learning Representations , year=

  13. [21]

    Forty-second International Conference on Machine Learning , year=

    Overtrained Language Models Are Harder to Fine-Tune , author=. Forty-second International Conference on Machine Learning , year=

  14. [22]

    Do Neural Networks Lose Plasticity in a Gradually Changing World? , author=

  15. [23]

    The Fourteenth International Conference on Learning Representations , year=

    Activation Function Design Sustains Plasticity in Continual Learning , author=. The Fourteenth International Conference on Learning Representations , year=

  16. [24]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Plasticity as the Mirror of Empowerment , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  17. [25]

    Resetting the Optimizer in Deep

    Kavosh Asadi and Rasool Fakoor and Shoham Sabach , booktitle=. Resetting the Optimizer in Deep. 2023 , url=

    2023

  18. [26]

    2026 , publisher =

    Han, Tessa and Bordt, Sebastian and Zhang, Hanlin and Kakade, Sham , title=. 2026 , publisher =

    2026

  19. [27]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism , author=

  20. [28]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  21. [29]

    Language Models are Unsupervised Multitask Learners , author =

  22. [30]

    2018 , publisher=

    Improving language understanding by generative pre-training , author=. 2018 , publisher=

    2018

  23. [31]

    , title =

    Raffel, Colin and Shazeer, Noam and Roberts, Adam and Lee, Katherine and Narang, Sharan and Matena, Michael and Zhou, Yanqi and Li, Wei and Liu, Peter J. , title =. J. Mach. Learn. Res. , month = jan, articleno =. 2020 , issue_date =

    2020

  24. [32]

    Qwen3 technical report , author=

  25. [33]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  26. [34]

    Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =

    Yang, Ge and Hu, Edward and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , booktitle =. Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =

  27. [35]

    When Attention Collapses: How Degenerate Layers in

    Sunny Sanyal and Ravid Shwartz-Ziv and Alex Dimakis and sujay sanghavi , journal=. When Attention Collapses: How Degenerate Layers in. 2026 , url=

    2026

  28. [36]

    Proceedings of the 40th International Conference on Machine Learning , articleno =

    Zhai, Shuangfei and Likhomanenko, Tatiana and Littwin, Etai and Busbridge, Dan and Ramapuram, Jason and Zhang, Yizhe and Gu, Jiatao and Susskind, Josh , title =. Proceedings of the 40th International Conference on Machine Learning , articleno =. 2023 , publisher =

    2023

  29. [37]

    The Twelfth International Conference on Learning Representations , year=

    Efficient Streaming Language Models with Attention Sinks , author=. The Twelfth International Conference on Learning Representations , year=

  30. [38]

    The Twelfth International Conference on Learning Representations , year=

    Vision Transformers Need Registers , author=. The Twelfth International Conference on Learning Representations , year=

  31. [39]

    A Comprehensive Survey of Continual Learning: Theory, Method and Application , year=

    Wang, Liyuan and Zhang, Xingxing and Su, Hang and Zhu, Jun , journal=. A Comprehensive Survey of Continual Learning: Theory, Method and Application , year=

  32. [40]

    Frontiers in psychology , volume=

    The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects , author=. Frontiers in psychology , volume=. 2013 , publisher=

    2013

  33. [41]

    Psychology of Learning and Motivation , volume=

    Catastrophic interference in connectionist networks: The sequential learning problem , author=. Psychology of Learning and Motivation , volume=. 1989 , publisher=

    1989

  34. [42]

    French , keywords =

    Robert M. French , keywords =. Catastrophic forgetting in connectionist networks , journal =. 1999 , issn =

    1999

  35. [43]

    Proceedings of the national academy of sciences , volume=

    Overcoming catastrophic forgetting in neural networks , author=. Proceedings of the national academy of sciences , volume=. 2017 , publisher=

    2017

  36. [44]

    and Nguyen, Thien Huu

    Nguyen, Thuat and Nguyen, Chien Van and Lai, Viet Dac and Man, Hieu and Ngo, Nghia Trung and Dernoncourt, Franck and Rossi, Ryan A. and Nguyen, Thien Huu. C ultura X : A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages. Proceedings of the ...

    2024

  37. [45]

    An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks , author=

  38. [46]

    International conference on machine learning , pages=

    Continual learning through synaptic intelligence , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  39. [47]

    Proceedings of the European conference on computer vision (ECCV) , pages=

    Memory aware synapses: Learning what (not) to forget , author=. Proceedings of the European conference on computer vision (ECCV) , pages=

  40. [48]

    Nature Machine Intelligence , volume=

    Three types of incremental learning , author=. Nature Machine Intelligence , volume=. 2022 , publisher=

    2022

  41. [49]

    INFORMS Journal on Applied Analytics , volume=

    A better match for drivers and riders: Reinforcement learning at lyft , author=. INFORMS Journal on Applied Analytics , volume=. 2024 , publisher=

    2024

  42. [50]

    Machine Learning , pages=

    GVFs in the real world: making predictions online for water treatment , author=. Machine Learning , pages=. 2023 , publisher=

    2023

  43. [51]

    Proceedings of machine learning and systems , volume=

    What is the state of neural network pruning? , author=. Proceedings of machine learning and systems , volume=

  44. [52]

    and Pechenizkiy, Mykola and Mocanu, Decebal Constantin , title =

    Grooten, Bram and Sokar, Ghada and Dohare, Shibhansh and Mocanu, Elena and Taylor, Matthew E. and Pechenizkiy, Mykola and Mocanu, Decebal Constantin , title =. Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages =. 2023 , isbn =

    2023

  45. [53]

    The Twelfth International Conference on Learning Representations , year=

    Dynamic Sparse Training with Structured Sparsity , author=. The Twelfth International Conference on Learning Representations , year=

  46. [54]

    Deep Rewiring: Training very sparse deep networks , author=

  47. [55]

    2018 , publisher=

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science , author=. 2018 , publisher=

    2018

  48. [56]

    International conference on machine learning , pages=

    Rigging the lottery: Making all tickets winners , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  49. [57]

    Advances in neural information processing systems , volume=

    Deconstructing lottery tickets: Zeros, signs, and the supermask , author=. Advances in neural information processing systems , volume=

  50. [58]

    2019 , url=

    Namhoon Lee and Thalaiyasingam Ajanthan and Philip Torr , booktitle=. 2019 , url=

    2019

  51. [59]

    Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages =

    Understanding the difficulty of training deep feedforward neural networks , author =. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages =. 2010 , volume =

    2010

  52. [60]

    IEEE International Conference on Computer Vision , pages=

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification , author=. IEEE International Conference on Computer Vision , pages=

  53. [61]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Improving Transformer Optimization Through Better Initialization , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    2020

  54. [62]

    Neural machine translation by jointly learning to align and translate , author=

  55. [63]

    Proceedings of the IEEE , volume=

    Gradient-based learning applied to document recognition , author=. Proceedings of the IEEE , volume=. 2002 , publisher=

    2002

  56. [64]

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages=

    2019

  57. [65]

    Advances in neural information processing systems , year=

    Language models are few-shot learners , author=. Advances in neural information processing systems , year=

  58. [66]

    9th International Conference on Learning Representations , year=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. 9th International Conference on Learning Representations , year=

  59. [67]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  60. [68]

    ICML Workshop on Deep Learning for Audio, Speech and Language Processing , year=

    Rectifier nonlinearities improve neural network acoustic models , author=. ICML Workshop on Deep Learning for Audio, Speech and Language Processing , year=

  61. [69]

    arXiv preprint arXiv:1606.08415 , year=

    Gaussian Error Linear Units (GELUs) , author=. arXiv preprint arXiv:1606.08415 , year=

    arXiv

  62. [70]

    arXiv preprint arXiv:1710.05941 , year=

    Swish: a self-gated activation function , author=. arXiv preprint arXiv:1710.05941 , year=

    arXiv

  63. [71]

    IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Deep residual learning for image recognition , author=. IEEE Conference on Computer Vision and Pattern Recognition , pages=

  64. [72]

    Weiaicunzai , title =

  65. [73]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Scratching visual transformer's back with uniform attention , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  66. [74]

    AAAI , volume=

    Adapting bias by gradient descent: An incremental version of delta-bar-delta , author=. AAAI , volume=. 1992 , organization=

    1992

  67. [75]

    arXiv preprint arXiv:2402.02342 , year=

    Metaoptimize: A framework for optimizing step sizes and other meta-parameters , author=. arXiv preprint arXiv:2402.02342 , year=

  68. [76]

    Improving neural networks by preventing co-adaptation of feature detectors , author=

  69. [77]

    The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks , author=

  70. [78]

    Scaling laws for neural language models , author=

  71. [79]

    2004 , publisher=

    Convex Optimization , author=. 2004 , publisher=

    2004

  72. [80]

    On the Origin of Implicit Regularization in Stochastic Gradient Descent , author=

  73. [81]

    Implicit Regularization in Deep Learning May Not Be Explainable by Norms , volume =

    Razin, Noam and Cohen, Nadav , booktitle =. Implicit Regularization in Deep Learning May Not Be Explainable by Norms , volume =

  74. [82]

    7th International Conference on Learning Representations , year=

    Harnessing structures for value-based planning and reinforcement learning , author=. 7th International Conference on Learning Representations , year=

  75. [83]

    Ian Goodfellow and Yoshua Bengio and Aaron Courville , publisher=

  76. [84]

    Understanding Batch Normalization , volume =

    Bjorck, Nils and Gomes, Carla P and Selman, Bart and Weinberger, Kilian Q , booktitle =. Understanding Batch Normalization , volume =

  77. [85]

    On the difficulty of training recurrent neural networks , author=

  78. [86]

    IEEE transactions on neural networks , volume=

    Learning long-term dependencies with gradient descent is difficult , author=. IEEE transactions on neural networks , volume=. 1994 , publisher=

    1994

  79. [87]

    Is normalization indispensable for training deep neural network? , author=

  80. [88]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift , author=

  81. [89]

    2016 , eprint=

    Layer Normalization , author=. 2016 , eprint=

    2016

  82. [90]

    How does batch normalization help optimization? , author=

  83. [91]

    Communications in Computational Physics , year =

    Lu, Lu and Yeonjong, Shin and Yanhui, Su and Karniadakis, George, Em , title =. Communications in Computational Physics , year =. doi:https://doi.org/10.4208/cicp.OA-2020-0165 , url =

    2020 doi

  84. [92]

    2020 , pages=

    Trainability of relu networks and data-dependent initialization , author=. 2020 , pages=

    2020

  85. [93]

    2017 , archivePrefix=

    The exploding gradient problem demystified-definition, prevalence, impact, origin, tradeoffs, and solutions , author=. 2017 , archivePrefix=

    2017

  86. [94]

    IEEE Transactions on Neural Networks and Learning Systems , volume=

    Spurious local minima are common for deep neural networks with piecewise linear activations , author=. IEEE Transactions on Neural Networks and Learning Systems , volume=. 2022 , publisher=

    2022

  87. [95]

    Measuring Saturation in Neural Networks , year=

    Rakitianskaia, Anna and Engelbrecht, Andries , booktitle=. Measuring Saturation in Neural Networks , year=

  88. [96]

    2012 , publisher=

    Montavon, Gr. 2012 , publisher=

    2012

  89. [97]

    International Conference on Learning Representations , year=

    Sharpness-aware Minimization for Efficiently Improving Generalization , author=. International Conference on Learning Representations , year=

  90. [98]

    Mechanization of Thought Processes, Proceedings of a Symposium Held at the National Physical Laboratory , pages=

    Pandemonium: A Paradigm for Learning , author=. Mechanization of Thought Processes, Proceedings of a Symposium Held at the National Physical Laboratory , pages=. 1958 , address=

    1958

  91. [99]

    Klopf, A. H. and Gose, Earl , journal=. An Evolutionary Pattern Recognition Network , year=

  92. [100]

    SIGART Bull

    Cognitive systems based on adaptive algorithms , author=. SIGART Bull. , volume=. 1977 , publisher=

    1977

  93. [101]

    Advances in neural information processing systems , volume=

    The cascade-correlation learning architecture , author=. Advances in neural information processing systems , volume=

  94. [102]

    IEEE transactions on neural networks and learning systems , volume=

    Deep cascade learning , author=. IEEE transactions on neural networks and learning systems , volume=. 2018 , publisher=

    2018

  95. [103]

    Progressive neural networks , author=

  96. [104]

    1993 , publisher=

    Learning in Embedded Systems , author=. 1993 , publisher=

    1993

  97. [105]

    and Adams, A

    Waugh, S. and Adams, A. , booktitle=. Pruning within cascade-correlation , year=

  98. [106]

    , author=

    Representation Search through Generate and Test. , author=. AAAI Workshop: Learning Rich Representations from Low-Level Sensors , year=

  99. [107]

    Nature Reviews Neuroscience , volume=

    Spine dynamics in the brain, mental disorders and artificial neural networks , author=. Nature Reviews Neuroscience , volume=. 2021 , publisher=

    2021

  100. [108]

    arXiv preprint arXiv:2109.00267 , year=

    The impact of reinitialization on generalization in convolutional neural networks , author=. arXiv preprint arXiv:2109.00267 , year=

  101. [109]

    I Can't Believe It's Not Better! - Understanding Deep Learning Through Empirical Falsification

    When Does Re-initialization Work? , author =. Proceedings on "I Can't Believe It's Not Better! - Understanding Deep Learning Through Empirical Falsification" at NeurIPS 2022 Workshops , pages =. 2023 , editor =

    2022

  102. [110]

    International Conference on Learning Representations , year=

    Fortuitous Forgetting in Connectionist Networks , author=. International Conference on Learning Representations , year=

  103. [111]

    Knowledge Evolution in Neural Networks , year=

    Taha, Ahmed and Shrivastava, Abhinav and Davis, Larry , booktitle=. Knowledge Evolution in Neural Networks , year=

  104. [112]

    International conference on machine learning , pages=

    Rifle: Backpropagation in depth for deep transfer learning through re-initializing the fully-connected layer , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  105. [113]

    Nature communications , volume=

    Concept transfer of synaptic diversity from biological to artificial neural networks , author=. Nature communications , volume=. 2025 , publisher=

    2025

  106. [114]

    A study on the plasticity of neural networks , author=

  107. [115]

    Parallel distributed processing: Explorations in the microstructure of cognition , volume=

    State-dependent factors influencing neural plasticity: A partial account of the critical period , author=. Parallel distributed processing: Explorations in the microstructure of cognition , volume=. 1986 , publisher=

    1986

  108. [116]

    Age of Acquisition Effects in Word Reading and Other Tasks , journal =

    Jason D Zevin and Mark S Seidenberg , keywords =. Age of Acquisition Effects in Word Reading and Other Tasks , journal =. 2002 , issn =

    2002

  109. [117]

    The influence of age of acquisition in word reading and other tasks: A never ending story? , journal =

    Patrick Bonin and Christopher Barry and Alain Méot and Marylène Chalard , keywords =. The influence of age of acquisition in word reading and other tasks: A never ending story? , journal =. 2004 , issn =

    2004

  110. [118]

    Lambon Ralph and Sheeba Ehsan , title =

    Matthew A. Lambon Ralph and Sheeba Ehsan , title =. 2006 , publisher =

    2006

  111. [119]

    Thomas, Michael S. C. and Johnson, Mark H. , title =

  112. [120]

    Proceedings of the 31th Annual Conference of the Cognitive Science Society , pages=

    Frequency trajectory gives rise to an age-limited learning effect as a function of input-output mapping in connectionist networks , author=. Proceedings of the 31th Annual Conference of the Cognitive Science Society , pages=. 2009 , organization=

    2009

  113. [121]

    15th European Conference on Computer Vision , pages=

    Riemannian walk for incremental learning: Understanding forgetting and intransigence , author=. 15th European Conference on Computer Vision , pages=

  114. [122]

    6th International Conference on Learning Representations , year=

    Critical learning periods in deep networks , author=. 6th International Conference on Learning Representations , year=

  115. [123]

    The Interplay of Search and Gradient Descent in Semi-stationary Learning Problems , author=

  116. [124]

    Toward Generate-and-Test Algorithms for Continual Feature Discovery , author=

  117. [125]

    Real-time Reinforcement Learning for Achieving Goals in Big Worlds , author=

  118. [126]

    Transient Non-stationarity and Generalisation in Deep Reinforcement Learning , author=

  119. [127]

    Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning , author=

  120. [128]

    Continual

    Dohare, Shibhansh and Sutton, Richard S and Mahmood, A Rupam , year=. Continual

  121. [129]

    2022 , howpublished =

    Maintaining Plasticity in Deep Continual Learning , author =. 2022 , howpublished =

    2022

  122. [130]

    2023 , note=

    Maintaining plasticity in deep continual learning , author=. 2023 , note=

    2023

  123. [131]

    40th International Conference on Machine Learning , pages =

    The Dormant Neuron Phenomenon in Deep Reinforcement Learning , author =. 40th International Conference on Machine Learning , pages =

  124. [132]

    2nd Conference on Lifelong Learning Agents , year=

    Loss of Plasticity in Continual Deep Reinforcement Learning , author=. 2nd Conference on Lifelong Learning Agents , year=

  125. [133]

    Deep Reinforcement Learning with Plasticity Injection , url =

    Nikishin, Evgenii and Oh, Junhyuk and Ostrovski, Georg and Lyle, Clare and Pascanu, Razvan and Dabney, Will and Barreto, Andre , booktitle =. Deep Reinforcement Learning with Plasticity Injection , url =

  126. [134]

    Advances in Neural Information Processing Systems , volume=

    Plastic: Improving input and label plasticity for sample efficient reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  127. [135]

    Same accuracy, twice as fast: continuous training surpasses retraining from scratch , author=

  128. [136]

    ATT Labs [Online]

    MNIST handwritten digit database , author=. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , volume=

  129. [137]

    ImageNet: A large-scale hierarchical image database , year=

    Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Kai Li and Li Fei-Fei , booktitle=. ImageNet: A large-scale hierarchical image database , year=

  130. [138]

    Tiny imagenet visual recognition challenge , author=

  131. [139]

    Datasets: A Community Library for Natural Language Processing

    Lhoest, Quentin and Villanova del Moral, Albert and Jernite, Yacine and Thakur, Abhishek and von Platen, Patrick and Patil, Suraj and Chaumond, Julien and Drame, Mariama and Plu, Julien and Tunstall, Lewis and Davison, Joe and S a s ko, Mario and Chhablani, Gunjan and Malik, B...

    2021

  132. [140]

    2009 , institution=

    Learning multiple layers of features from tiny images , author=. 2009 , institution=

    2009

  133. [141]

    Proximal policy optimization algorithms , author=

  134. [142]

    , title =

    Han, Song and Pool, Jeff and Tran, John and Dally, William J. , title =. Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1 , pages =. 2015 , publisher =

    2015

  135. [143]

    International Conference on Learning Representations , year=

    Lookahead: A Far-sighted Alternative of Magnitude-based Pruning , author=. International Conference on Learning Representations , year=

  136. [144]

    International Conference on Learning Representations , year=

    Pruning Convolutional Neural Networks for Resource Efficient Inference , author=. International Conference on Learning Representations , year=

  137. [145]

    DepGraph: Towards Any Structural Pruning , year=

    Fang, Gongfan and Ma, Xinyin and Song, Mingli and Bi Mi, Michael and Wang, Xinchao , booktitle=. DepGraph: Towards Any Structural Pruning , year=

  138. [146]

    Optimal Brain Damage , url =

    LeCun, Yann and Denker, John and Solla, Sara , booktitle =. Optimal Brain Damage , url =

  139. [147]

    and Stork, D.G

    Hassibi, B. and Stork, D.G. and Wolff, G.J. , booktitle=. Optimal Brain Surgeon and general network pruning , year=

  140. [148]

    Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =

    Dong, Xin and Chen, Shangyu and Pan, Sinno Jialin , title =. Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =. 2017 , isbn =

    2017

  141. [149]

    Advances in neural information processing systems , volume=

    Skeletonization: A technique for trimming the fat from a network via relevance assessment , author=. Advances in neural information processing systems , volume=

  142. [150]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Importance estimation for neural network pruning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  143. [151]

    Information Fusion , volume=

    A review of deep learning techniques for speech processing , author=. Information Fusion , volume=. 2023 , publisher=

    2023

  144. [152]

    2014 , volume =

    Nitish Srivastava and Geoffrey Hinton and Alex Krizhevsky and Ilya Sutskever and Ruslan Salakhutdinov , title =. 2014 , volume =

    2014

  145. [153]

    Saxe and James L

    Andrew M. Saxe and James L. Mcclelland and Surya Ganguli , title =

  146. [154]

    The effective rank: A measure of effective dimensionality , year=

    Roy, Olivier and Vetterli, Martin , booktitle=. The effective rank: A measure of effective dimensionality , year=

  147. [155]

    2018 , url=

    Gradients explode - Deep Networks are shallow - ResNet explained , author=. 2018 , url=

    2018

  148. [156]

    Bertsekas, Dimitri , year=

  149. [157]

    Transactions of Machine Learning Research , year=

    Representation Alignment in Neural Networks , author=. Transactions of Machine Learning Research , year=

  150. [158]

    1986 , publisher=

    Learning representations by back-propagating errors , author=. 1986 , publisher=

    1986

  151. [159]

    1998 , publisher=

    Lifelong learning algorithms , author=. 1998 , publisher=

    1998

  152. [160]

    Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E , booktitle =

  153. [161]

    Silver, David and Huang, Aja and Maddison, Chris J. and Guez, Arthur and Sifre, Laurent and van den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and Dieleman, Sander and Grewe, Dominik and Nham, John and Kalchbr...

    doi

  154. [162]

    and Koop, Anna and Silver, David , title =

    Sutton, Richard S. and Koop, Anna and Silver, David , title =. 2007 , isbn =. doi:10.1145/1273496.1273606 , booktitle =

    2007 doi

  155. [163]

    Towards continual reinforcement learning: A review and perspectives , author=

  156. [164]

    and Al-Shedivat, Maruan and Whiteson, Shimon and Abbeel, Pieter and Mordatch, Igor , title =

    Foerster, Jakob and Chen, Richard Y. and Al-Shedivat, Maruan and Whiteson, Shimon and Abbeel, Pieter and Mordatch, Igor , title =. 2018 , publisher =

    2018

  157. [165]

    Training and Tracking in Robotics , author=

  158. [166]

    1998 , publisher=

    CHILD: A first step towards continual learning , author=. 1998 , publisher=

    1998

  159. [167]

    Proceedings of the 34th International Conference on Machine Learning , pages =

    Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , author =. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , volume =

    2017

  160. [168]

    Growing a brain: Fine-tuning by increasing model capacity , author=

  161. [169]

    Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning , author=

  162. [170]

    Fahlman, Scott E and others , year=

  163. [171]

    Sutton , title =

    Richard S. Sutton , title =

  164. [172]

    and Whitehead, Steven D

    Sutton, Richard S. and Whitehead, Steven D. , title =. 1993 , booktitle =

    1993

  165. [173]

    2009 , volume=

    Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Kai Li and Li Fei-Fei , booktitle=. 2009 , volume=

    2009

  166. [174]

    , title =

    Nair, Vinod and Hinton, Geoffrey E. , title =. 2010 , isbn =

    2010

  167. [175]

    Maas and Awni Y

    Andrew L. Maas and Awni Y. Hannun and Andrew Y. Ng , title =

  168. [176]

    Fast and

    Djork. Fast and. 2016 , timestamp =

    2016

  169. [177]

    Le , title =

    Prajit Ramachandran and Barret Zoph and Quoc V. Le , title =. 2018 , timestamp =

    2018

  170. [178]

    Kingma and Jimmy Ba , title =

    Diederik P. Kingma and Jimmy Ba , title =. 3rd International Conference on Learning Representations , year =

  171. [179]

    Network trimming: A data-driven neuron pruning approach towards efficient deep architectures , author=

  172. [180]

    Erwin Coumans and Yunfei Bai , title =

  173. [181]

    Speech recognition with deep recurrent neural networks , year=

    Graves, Alex and Mohamed, Abdel-rahman and Hinton, Geoffrey , booktitle=. Speech recognition with deep recurrent neural networks , year=

  174. [182]

    and Koutník, Jan and Steunebrink, Bas R

    Greff, Klaus and Srivastava, Rupesh K. and Koutník, Jan and Steunebrink, Bas R. and Schmidhuber, Jürgen , journal=. LSTM: A Search Space Odyssey , year=

  175. [183]

    1943 , publisher=

    A logical calculus of the ideas immanent in nervous activity , author=. 1943 , publisher=

    1943

  176. [184]

    Measuring catastrophic forgetting in neural networks , author=

  177. [185]

    Optimal brain damage , author=

  178. [186]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding , author=

  179. [187]

    Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers , author=

  180. [188]

    What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study , author=

  181. [189]

    and Bottou, L

    Lecun, Y. and Bottou, L. and Bengio, Y. and Haffner, P. , journal=. Gradient-based learning applied to document recognition , year=

  182. [190]

    2013 , volume =

    On the importance of initialization and momentum in deep learning , author =. 2013 , volume =

    2013

  183. [191]

    Using additive noise in back-propagation training , author=

  184. [192]

    Adding gradient noise improves learning for very deep networks , author=

  185. [193]

    Journal of Machine Learning Research , year =

    Shimon Whiteson and Peter Stone , title =. Journal of Machine Learning Research , year =

  186. [194]

    Online incremental feature learning with denoising autoencoders , author=

  187. [195]

    A downsampled variant of imagenet as an alternative to the cifar datasets , author=

  188. [196]

    2021 , volume =

    Zero-Shot Text-to-Image Generation , author =. 2021 , volume =

    2021

  189. [197]

    2015 , publisher=

    Human-level control through deep reinforcement learning , author=. 2015 , publisher=

    2015

  190. [198]

    2015 , publisher=

    Imagenet large scale visual recognition challenge , author=. 2015 , publisher=

    2015

  191. [199]

    Squeeze-and-excitation networks , author=

  192. [200]

    Chasing sparsity in vision transformers: An end-to-end exploration , author=

  193. [201]

    Dynamic sparse training for deep reinforcement learning , author=

  194. [202]

    Evaluating pruning methods , author=

  195. [203]

    40th International Conference on Machine Learning , pages=

    Understanding Plasticity in Neural Networks , author=. 40th International Conference on Machine Learning , pages=

  196. [204]

    Pytorch: An imperative style, high-performance deep learning library , author=

  197. [205]

    IEEE Signal Processing Magazine , volume=

    Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups , author=. IEEE Signal Processing Magazine , volume=. 2012 , publisher=

    2012

  198. [206]

    Highly accurate protein structure prediction with

    Jumper, John and Evans, Richard and Pritzel, Alexander and Green, Tim and Figurnov, Michael and Ronneberger, Olaf and Tunyasuvunakool, Kathryn and Bates, Russ and. Highly accurate protein structure prediction with. Nature , volume=. 2021 , publisher=

    2021

  199. [207]

    A general reinforcement learning algorithm that masters

    Silver, David and Hubert, Thomas and Schrittwieser, Julian and Antonoglou, Ioannis and Lai, Matthew and Guez, Arthur and Lanctot, Marc and Sifre, Laurent and Kumaran, Dharshan and Graepel, Thore and others , journal=. A general reinforcement learning algorithm that masters. 20...

    2018

  200. [208]

    Morav. Deep. Science , volume=. 2017 , publisher=

    2017

  201. [209]

    3rd International Conference on Learning Representations , year =

    Dzmitry Bahdanau and Kyunghyun Cho and Yoshua Bengio , title =. 3rd International Conference on Learning Representations , year =

  202. [210]

    Nature , volume=

    Champion-level drone racing using deep reinforcement learning , author=. Nature , volume=. 2023 , publisher=

    2023

  203. [211]

    Outracing champion

    Wurman, Peter R and Barrett, Samuel and Kawamoto, Kenta and MacGlashan, James and Subramanian, Kaushik and Walsh, Thomas J and Capobianco, Roberto and Devlic, Alisa and Eckert, Franziska and Fuchs, Florian and others , journal=. Outracing champion. 2022 , publisher=

    2022

  204. [212]

    Journal of Field Robotics , volume=

    A survey of deep learning techniques for autonomous driving , author=. Journal of Field Robotics , volume=. 2020 , publisher=

    2020

  205. [213]

    Maintaining plasticity via regenerative regularization , author=

  206. [214]

    Proceedings of the IEEE International Conference on Computer Vision , pages=

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification , author=. Proceedings of the IEEE International Conference on Computer Vision , pages=

  207. [215]

    Advances in Neural Information Processing Systems 36 , pages=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems 36 , pages=

  208. [216]

    The International Journal of Robotics Research , volume=

    Learning dexterous in-hand manipulation , author=. The International Journal of Robotics Research , volume=. 2020 , publisher=

    2020

  209. [217]

    Dota 2 with Large Scale Deep Reinforcement Learning , author=

  210. [218]

    Rebuffi, Sylvestre-Alvise and Kolesnikov, Alexander and Sperl, Georg and Lampert, Christoph H , booktitle=. i

  211. [219]

    Mathematical Programming , volume=

    Restart procedures for the conjugate gradient method , author=. Mathematical Programming , volume=. 1977 , publisher=

    1977

  212. [220]

    Evolutionary Computation , volume=

    Evolving neural networks through augmenting topologies , author=. Evolutionary Computation , volume=. 2002 , publisher=

    2002

  213. [221]

    1992 , publisher=

    Adaptation in Natural and Artificial Systems , author=. 1992 , publisher=

    1992

  214. [222]

    1995 , publisher=

    Without Miracles: Universal Selection Theory and the Second Darwinian Revolution , author=. 1995 , publisher=

    1995

  215. [223]

    Dennett, Daniel C , journal=. Why the. 1978 , publisher=

    1978

  216. [224]

    Thorndike, E L , title =

  217. [225]

    Self-Organizing Systems , volume=

    Blind variation and selective survival as a general strategy in knowledge-processes , author=. Self-Organizing Systems , volume=. 1960 , publisher=

    1960

  218. [226]

    International Conference on Learning Representations , year=

    Efficient Streaming Language Models with Attention Sinks , author=. International Conference on Learning Representations , year=

  219. [227]

    International Conference on Learning Representations , year=

    When Attention Sink Emerges in Language Models: An Empirical View , author=. International Conference on Learning Representations , year=

  220. [228]

    2025 , eprint=

    DeepSeek-V3 Technical Report , author=. 2025 , eprint=

    2025

Pith tools

Reviewed June 25, 2026 · model on record in the stance chip above.