Optimisation Is Not What You Need

Alfredo Ibias

REVIEW 3 major objections 5 minor 64 references

Optimisation Is Not What You Need

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that catastrophic forgetting is not a fixable flaw of neural networks but a mathematical consequence of tuning weights to minimise a loss, so weight-based optimisation cannot be the route to artificial general…

desk verdict A well-intentioned position paper whose formal no-go theorem rests on an unstated injectivity assumption; the central claim is not proved. read the letter →

arxiv 2507.03045 v1 pith:JAQVHBPA submitted 2025-07-03 cs.LG cs.AI

Alfredo Ibias This is my paper · ORCID

classification cs.LGcs.AI

keywords catastrophicforgettingoptimisationmethodsoverfittingartificialgeneralintelligenceworld-modellingweightedmappingscontinuallearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to prove that catastrophic forgetting is inherent to any learning method that solves problems by adjusting weights to minimise a loss function, not a bug that better architectures can remove. It formalises an optimiser as a weighted mapping $M_w(i)=i\circ w$ and shows that two different tasks demand different weights, so retraining on a new task must change the outputs on the old task. The same formal setup is used to argue that overfitting is also a necessary consequence of the optimisation framing. To make the point constructive, the paper reports small experiments with a representation-based world-modelling algorithm that keeps its accuracy after sequential training on two datasets, and concludes that general intelligence requires moving outside optimisation.

What carries the argument

The load-bearing object is the weighted mapping $M_w: I \to_w O$ with $M_w(i)=i\circ w$, where $\circ$ is any operation that combines an input $i$ with a weight vector $w$. Theorems 1 and 2 establish an injectivity principle for this object: for two different optimisation problems, distinct weights imply distinct outputs on every input, and equal outputs across both training sets force equal weights. Theorem 3 applies that principle to sequential training, and Definition 6 defines catastrophic forgetting as a change in previously learned input assignments; the proof of Theorem 3 is what converts 'the weights must change' into 'the old outputs must change'.

What would settle it

Train a linear model $f(x)=w^\top x$ on task A whose inputs span a subspace $S$, then train on a disjoint task B while projecting every gradient step onto the orthogonal complement of $S$, so outputs on task A's inputs never change. If the model learns B without losing accuracy on A, a weight-based optimisation method has avoided catastrophic forgetting, contradicting Theorem 3; the same test can be done on an overparameterised neural network by projecting updates onto the null space of the Jacobian of task-A outputs.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is Theorem 3: any optimisation method that produces a weighted mapping $M_w$ and is trained on a new task $Q'\neq Q$ on a disjoint input set will suffer catastrophic forgetting as defined in Definition 6. The proof runs through Theorem 1, which says that two weighted solutions to two different problems must have different weights, and therefore that a weight update forced by the new task must alter the mapping on the old task's inputs. The same machinery yields Theorem 5, which states that a model that optimally solves one training set cannot simultaneously solve a disjoint but similar test set, making overfitting inherent too. The paper presents these results as the formal reason why mitigation strategies, from elastic weight consolidation to rehearsal to architectural changes, have only reduced but never eliminated the problem, and as evidence that the field needs non-optimisation, world-modelling approaches.

Load-bearing premise

The proof depends on the assumption that two different weight vectors always give different outputs on every single input, but real overparameterised networks routinely have many weight vectors that implement the same function on a given finite set of inputs.

Editorial extensions

If this is right

A single weight-based model trained by loss minimisation cannot learn a second, disjoint task without degrading its performance on the first, so continual learning with one network is impossible in principle.
Existing mitigation strategies such as elastic weight consolidation, rehearsal, and architecture changes can only reduce the damage; if the proof is right, none of them can eliminate it.
Overfitting is not a training pathology but a consequence of the same setup: the better a model solves its training problem, the further it sits from the slightly different test problem.
Any AGI built from a single loss-guided weight-tuning learner is blocked; the viable direction is modular or representation-based systems where each component handles one task.
A set of single-task optimisation methods can still solve multiple tasks, as long as no single model is retrained on a different problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The proof's hidden injectivity assumption means the result covers parametric families where weights are identifiable from the mapping on the training inputs; overparameterised networks, where many weights give identical outputs, are not covered by the argument as written.
If non-identifiability is the escape route, then overparameterisation and null-space-constrained updates are what could make continual learning work for neural networks, the opposite of the paper's dismissal of weight-based methods.
The empirical demonstration uses two small numeric datasets with largely non-overlapping representations; a natural extension is to run the same sequential protocol on image or language tasks where representations overlap, where the claimed immunity of world-modelling methods is not yet shown.
The theorem applies to loss-guided weight tuning, not to learning in general; lookup tables, non-parametric methods, and prototype-based classifiers fall outside it, which suggests the real distinction is not optimisation versus world-modelling but global weight identifiability versus other forms of memory.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper's formal core does not hold. The new thing is the attempt to prove catastrophic forgetting is inherent to all optimisation methods, but the proof assumes what it needs to show. I agree with the stress-test note: Theorem 1 uses a hidden injectivity/uniqueness assumption, i∘w ≠ i∘w' for every i whenever Q ≠ Q', and that is false for overparameterised networks and even for linear maps. The concrete counterexample with R^2 and two collinear training points is right. Theorem 2 mirrors the problem; Theorem 3 applies Theorem 1 after explicitly ruling out one of its premises; Theorem 5 assumes its own conclusion. So the claimed inevitability is built into the definitions rather than derived.

What the paper does well: it is an honest survey of the forgetting literature, and the observation that most mitigation methods reduce rather than remove forgetting is worth saying. The author also explicitly flags the small scale of the world-modelling experiments, which is more than many papers do. The definitions are clear enough to locate the flaw.

Where it is soft: the empirical section is two runs on two UCI datasets with the author's own framework, no code, no baselines, no error bars. That cannot support the positive claim about world-modelling methods. The self-citations are not a problem in themselves, but they do not substitute for independent evidence. The framing from 'most current methods suffer' to 'all optimisation methods must suffer' is exactly the leap the proofs were supposed to close, and they do not.

Who it is for: someone following AGI critique may read it as a position statement, but I would not cite it for the formal claims. It is not a solid contribution to continual learning theory.

Recommendation: desk reject in current form, or send back with an invitation to resubmit as a non-technical essay. I would not send this version to referees; the main theorem needs to be either rebuilt under honest assumptions or dropped.

Referee Report

3 major / 5 minor

Summary. The paper claims to formally prove that catastrophic forgetting is inherent to all optimisation methods that learn by tuning weights, and that overfitting is similarly inherent. It introduces a formal framework of weighted mappings and machine-learning optimisation problems, states four theorems (Theorems 1–4 on forgetting, Theorem 5 on overfitting), and argues that world-modelling frameworks such as the author's Unsupervised Cognition avoid both problems. Two small experiments on UCI datasets (Wisconsin Breast Cancer and Pima Indians Diabetes) are reported as empirical support for the world-modelling alternative.

Significance. If the claimed formal result were correct, it would be a far-reaching negative result for continual learning and for the optimisation-based approach to artificial general intelligence. The paper also attempts a positive contribution by pointing to world-modelling alternatives, which is a direction worth exploring. However, the formal core is the load-bearing contribution, and it is invalid: Theorem 1 relies on an unstated injectivity assumption that is false for overparameterised networks, and the later theorems inherit this flaw. The empirical evidence is too limited to carry the general claims. The paper is therefore not publishable in its current form, although the framework and the discussion of alternative approaches may contain seeds for future work.

major comments (3)

[Section III, Theorem 1] The proof of the 'only if' direction asserts that if w≠w' and Q≠Q', then i◦w≠i◦w' for every i∈I. This is a substantive injectivity/uniqueness assumption, not a consequence of Definitions 2–4. It fails for overparameterised networks, where many weight vectors implement the same function on a finite training set. For example, with I=R^2, M_w(x)=w·x, L={(1,0)}, L'={(2,0)}, w=(1,1), w'=(1,2), both weight vectors achieve zero loss on both tasks and give identical outputs on L∪L', yet w≠w'. This directly falsifies the claimed 'only if' direction of Theorem 1, which is then used as the foundation for Theorem 3.
[Section IV, Theorem 3] The proof invokes Theorem 1 to conclude that y_L=y'_L implies w=w'. Theorem 1, however, requires equality of the two mappings on both L and L'. The proof has explicitly assumed y_{L'}≠y'_{L'}, so the condition of Theorem 1 is not met. The earlier step 'that implies that w≠w′ because i◦w≠i◦w′ ∀i∈L′' again relies on the same unstated injectivity assumption. Consequently Theorem 3 does not establish that every weight-based optimisation method must suffer catastrophic forgetting; it establishes a consequence of the paper's hidden axiom, not of the general definitions.
[Section V, Theorem 5] The proof states that 'by Theorem 1, P cannot solve Q too', but Theorem 1 concerns two different mappings that solve two different problems and says nothing about a single mapping solving two optimisation problems. Moreover, Definition 7 already defines overfitting as failing to minimise on a new input set, so the theorem's conclusion is essentially built into its premise. The deduction is therefore not a proof that overfitting is inherent to optimisation methods.

minor comments (5)

[Throughout] The paper writes "training set L ∈ I", but L is a subset of the input set I; it should read L ⊆ I.
[Definition 1] The notation "∃o ∈ O∥M(i)=o" uses ∥ rather than a proper separator such as "such that"; the definition is also missing a colon after ∀i∈I.
[Section III, Theorems 1–2] The symbol ≡ is used for equivalence of optimisation problems (Q' ≡ Q), but no definition of equivalence is given; this matters because the proofs rely on it.
[Section VI(A)] There is a typo: "Hence. the learning" should read "Hence, the learning".
[Section VI] The empirical section reports only two datasets and does not provide model hyperparameters, dataset split details, or measures of variability; as presented, it is too limited to support the general claim that world-modelling methods avoid catastrophic forgetting and overfitting.

Circularity Check

2 steps flagged · score 8.0 of 10

Theorem 3's inevitability proof assumes pointwise output-distinctness for different weights, which is exactly the catastrophic-forgetting conclusion, and the empirical alternative evidence rests on the author's own Unsupervised Cognition system.

other [Section III, Theorem 1 proof (used in Theorem 3, Section IV)]
"Now, as Mw solves Q, and Mw′ solves Q′, with Q ≠ Q′, then i ◦ w ≠ i ◦ w′ ∀i ∈ I, where ◦ is the operations performed by the mapping M."

Theorem 1 is meant to prove that solving two different optimisation problems forces different outputs on the old training set. Instead, the proof assumes the pointwise version of that claim: whenever w≠w′, the outputs i◦w and i◦w′ differ for every input i, including the old inputs L. Under that assumption, Mw(L)≠Mw′(L) follows immediately, so the only-if direction is the assumption itself, not a consequence of Definitions 2–4. The assumption is false in general: with M_w(x)=w·x, w=(1,1), w′=(1,2), and x=(1,0), the outputs coincide despite w≠w′. Since Theorem 3 derives the inevitability of catastrophic forgetting directly from Theorem 1, the central impossibility result reduces to this unstated injectivity condition rather than to the definitions.
self citation load bearing [Section VI (World-modelling alternatives), Section VI.A, references [23]–[25]]
"This framework has been implemented into an algorithm called Unsupervised Cognition [23], that has obtained astounding results, not only getting similar results to those obtained by optimisation methods, but actually becoming the state-of-the-art in some cases [23], [25]."

The paper's positive conclusion—that world-modelling methods avoid catastrophic forgetting and overfitting—is supported by the author's own Unsupervised Cognition/Synthetic Cognition line of work (refs [23], [24], and [25] are by the same author) and by two small experiments run on that same implementation. No independent implementation or external benchmark is used. Thus the claim that the field should look outside optimisation rests on self-citations that are not independently verified in this paper. This is load-bearing for the overall recommendation, although it is not the basis of Theorem 3.

full rationale

The formal derivation chain collapses at Theorem 1: the only-if direction is obtained by assuming i◦w≠i◦w′ for every input whenever w≠w′. That pointwise injectivity condition is precisely a stronger form of the catastrophic-forgetting conclusion for weighted mappings, so Theorem 3's claim that any optimisation method based on weights will suffer catastrophic forgetting is not proved from the stated definitions; it is assumed. The same assumption reappears in Theorem 3's proof ('i ◦ w ≠ i ◦ w′ ∀i ∈ L′'), and Theorem 5's overfitting result also invokes Theorem 1. The empirical world-modelling evidence is additionally self-referential, since it evaluates only the author's own Unsupervised Cognition algorithm and cites the author's prior papers for its state-of-the-art claims. Because the central theorem reduces to an unstated assumption equivalent to the target result, the circularity score is high; however, the failure is also a straightforward mathematical gap, so the finding should be read as 'not proven' rather than as a demonstration that the conclusion is false.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The formal results rest on hidden assumptions rather than explicit free parameters: injectivity of the weight-to-function map, uniqueness of solutions, and the reduction of all optimisation methods to weighted mappings. No free numerical parameters are reported for the experiments; the algorithm's configuration is not given.

assumptions (3)

ad hoc to paper For distinct weight assignments w and w', the weighted mapping satisfies i◦w ≠ i◦w' for every input i.
Used in Theorem 1 and Theorem 3 (Sections III and IV). Not generally true; overparameterized neural networks can have many weight vectors with identical input-output behavior.
ad hoc to paper If two mappings both solve the same optimisation problem Q, they agree on every input in I.
Used in Theorem 2 (Section III). False when the loss has multiple global minima or when the training data do not identify a unique function.
domain assumption Every optimisation method of interest produces a single weighted mapping Mw, and learning a new task necessarily changes the weights.
Definition 5 restricts the proof to weighted mappings, but the abstract and conclusions generalize to all optimisation methods; this scope reduction is used without justification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Optimisation Is Not What You Need." pith.science (2026). https://pith.science/paper/JAQVHBPA

@misc{pith2026250703045,
  author       = {Pith},
  title        = {Pith review of: Optimisation Is Not What You Need},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JAQVHBPA}},
  note         = {Machine review of arXiv:2507.03045}
}

read the original abstract

The Artificial Intelligence field has focused on developing optimisation methods to solve multiple problems, specifically problems that we thought to be only solvable through cognition. The obtained results have been outstanding, being able to even surpass the Turing Test. However, we have found that these optimisation methods share some fundamental flaws that impede them to become a true artificial cognition. Specifically, the field have identified catastrophic forgetting as a fundamental problem to develop such cognition. This paper formally proves that this problem is inherent to optimisation methods, and as such it will always limit approaches that try to solve the Artificial General Intelligence problem as an optimisation problem. Additionally, it addresses the problem of overfitting and discuss about other smaller problems that optimisation methods pose. Finally, it empirically shows how world-modelling methods avoid suffering from either problem. As a conclusion, the field of Artificial Intelligence needs to look outside the machine learning field to find methods capable of developing an artificial cognition.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 59 canonical work pages

[1]

J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024: Student Research Workshop, St. Julian’s, Malta, March 21-22, 2024, pages 225–

work page 2024
[2]

E. L. Aleixo, J. G. Colonna, M. Cristo, and E. Fernandes. Catastrophic forgetting in deep learning: A comprehensive taxonomy. J. Braz. Comput. Soc., 30(1), 2024. IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. XX, NO. X, AUGUST 2025 8

work page 2024
[3]

Atkinson, B

C. Atkinson, B. McCane, L. Szymanski, and A. V . Robins. Pseudo- rehearsal: Achieving deep reinforcement learning without catastrophic forgetting. Neurocomputing, 428:291–307, 2021

work page 2021
[4]

R. A. Bafghi, N. Harilal, C. Monteleoni, and M. Raissi. Parameter effi- cient fine-tuning of self-supervised vits without catastrophic forgetting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 - Workshops, Seattle, WA, USA, June 17-18, 2024 , pages 3679–3684. IEEE, 2024

work page 2024
[5]

Bardes, J

A. Bardes, J. Ponce, and Y . LeCun. MC-JEPA: A joint-embedding predictive architecture for self-supervised learning of motion and content features. CoRR, abs/2307.12698, 2023

arXiv 2023
[6]

Y . Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade - Second Edition, volume 7700 of Lecture Notes in Computer Science, pages 437–

work page
[7]

Bhatt, J

G. Bhatt, J. Ross, and L. Sigal. Preventing catastrophic forgetting through memory networks in continuous detection. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29- October 4, 2024, Proceedings, Part LXXXIV , volume 15142 of Lecture Notes in Computer Science , pages 442–458. Springer, 2024

work page 2024
[8]

C. Biever. Chatgpt broke the turing test-the race is on for new ways to assess ai. Nature, 619(7971):686–689, 2023

work page 2023

Show all 64 references

[9]

Carta, A

A. Carta, A. Cossu, F. Errica, and D. Bacciu. Catastrophic forgetting in deep graph networks: A graph classification benchmark. Frontiers Artif. Intell., 5:824655, 2022

2022
[10]

C. Chen, R. Li, Y . Hu, Y . Chen, C. Qin, and Q. Zhang. Overcoming catastrophic forgetting by exemplar selection in task-oriented dialogue system. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 ...

2024
[11]

Chen and Y

W. Chen and Y . Zhou. Make domain shift a catastrophic forgetting alleviator in class-incremental learning. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA , pages 15866–15874. AAAI Press, 2025

2025
[12]

A. A. Chuganskaya, A. K. Kovalev, and A. Panov. The problem of concept learning and goals of reasoning in large language models. In Hybrid Artificial Intelligent Systems - 18th International Conference, HAIS 2023, Salamanca, Spain, September 5-7, 2023, Proceedings , volume 140...

2023
[13]

Cortes and V

C. Cortes and V . Vapnik. Support-vector networks. Mach. Learn. , 20(3):273–297, 1995

1995
[14]

D. R. Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statistical Methodology , 20(2):215– 232, 1958

1958
[15]

L. Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. , 29(6):141–142, 2012

2012
[16]

Dua and C

D. Dua and C. Graff. UCI machine learning repository, 2017

2017
[17]

Fernando, D

C. Fernando, D. Banarse, C. Blundell, Y . Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734, 2017

2017 arXiv
[18]

Gao and W

R. Gao and W. Liu. Defying catastrophic forgetting via influence function. Artif. Intell., 339:104261, 2025

2025
[19]

Greco, B

C. Greco, B. Plank, R. Fern ´andez, and R. Bernardi. Measuring catastrophic forgetting in visual question answering. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction - 10th International Workshop on Spoken Dialogue Systems, IWSDS 2019, Syracuse, Sicily,...

2019
[20]

Huang, M

H. Huang, M. B. Mustafa, and A. Asemi. An experimental study on dynamic lifelong learning with GPT for mitigating catastrophic forgetting in aspect-based sentiment analysis. IEEE Access, 13:90316– 90332, 2025

2025
[21]

Huang, X

J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024

2024
[22]

Huang, L

J. Huang, L. Cui, A. Wang, C. Yang, X. Liao, L. Song, J. Yao, and J. Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

2024
[23]

Ibias, H

A. Ibias, H. Antona, G. Ramirez-Miranda, E. Guinovart, and E. Alarc ´on. Unsupervised cognition. CoRR, abs/2409.18624, 2024

2024 arXiv
[24]

Ibias, G

A. Ibias, G. Ramirez-Miranda, E. Guinovart, and E. Alarc ´on. From manifestations to cognitive architectures: A scalable framework. In Artificial General Intelligence - 17th International Conference, AGI 2024, Seattle, WA, USA, August 13-16, 2024, Proceedings , volume 14951 of...

2024
[25]

Ibias, M

A. Ibias, M. Rodriguez-Galindo, H. Antona, G. Ramirez-Miranda, and E. Guinovart. Beating transformers using synthetic cognition. CoRR, abs/2504.07619, 2025

2025 arXiv
[26]

T. Kalb. Principles of Catastrophic Forgetting for Continual Semantic Segmentation in Automated Driving . PhD thesis, Karlsruhe Institute of Technology, Germany, 2024

2024
[27]

Koloski, B

B. Koloski, B. Skrlj, M. Robnik-Sikonja, and S. Pollak. Measuring catastrophic forgetting in cross-lingual classification: Transfer paradigms and tuning strategies. IEEE Access, 13:33509–33520, 2025

2025
[28]

Krizhevsky, I

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December...

2012
[29]

Kuipers and R

M. Kuipers and R. Prasad. Journey of artificial intelligence. Wireless Personal Communications, pages 1–16, 2022

2022
[30]

Kumar and S

A. Kumar and S. Agarwal. Handling catastrophic forgetting using cross- domain order in incremental deep learning. J. Electronic Imaging, 32(2), 2023

2023
[31]

A. Lee, H. M. Gomes, Y . Zhang, and W. B. Kleijn. Kolmogorov- arnold networks still catastrophically forget but differently from MLP. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA, pa...

2025
[32]

C. Liu, G. Sun, W. Liang, J. Dong, C. Qin, and Y . Cong. Museummaker: Continual style customization without catastrophic forgetting. IEEE Trans. Image Process., 34:2499–2512, 2025

2025
[33]

Z. Liu, Y . Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Soljacic, T. Y . Hou, and M. Tegmark. KAN: kolmogorov-arnold networks. CoRR, abs/2404.19756, 2024

2024 arXiv
[34]

Soulier, K

J: Lov ´on-Melgarejo, L. Soulier, K. Pinel-Sauvagnat, and L. Tamine. Studying catastrophic forgetting in neural ranking models. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part...

2021
[35]

Masarczyk, K

W. Masarczyk, K. Deja, and T. Trzcinski. On robustness of generative representations against catastrophic forgetting. In Neural Information Processing - 28th International Conference, ICONIP 2021, Sanur, Bali, Indonesia, December 8-12, 2021, Proceedings, Part VI , volume 1517 ...

2021
[36]

McCloskey and N

M. McCloskey and N. J. Cohen. Catastrophic interference in connec- tionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109–165. Elsevier, 1989

1989
[37]

W. S. McCulloch and W. H. Pitts. A logical calculus of the ideas im- manent in nervous activity. In The Philosophy of Artificial Intelligence , Oxford readings in philosophy, pages 22–39. Oxford University Press, 1990

1990
[38]

J. Park, A. Ji, M. Park, M. S. Rahman, and S. E. Oh. Malcl: Leveraging gan-based generative replay to combat catastrophic forgetting in malware classification. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025,...

2025
[39]

Pf ¨ulb, A

B. Pf ¨ulb, A. Gepperth, S. Abdullah, and A. Kilian. Catastrophic forgetting: Still a problem for dnns. In Artificial Neural Networks and Machine Learning - ICANN 2018 - 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings...

2018
[40]

Ranaldi, G

L. Ranaldi, G. Pucci, B. Haddow, and A. Birch. Empowering multi- step reasoning across languages via program-aided language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, page...

2024
[41]

Ranjan and S

S. Ranjan and S. K. Singh. Overcoming catastrophic forgetting in molecular property prediction using continual learning of sequential episodes. Expert Syst. Appl. , 267:125997, 2025. IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. XX, NO. X, AUGUST 2025 9

2025
[42]

A. V . Robins. Catastrophic forgetting in neural networks: the role of rehearsal mechanisms. In First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems, ANNES ’93, Dunedin, New Zealand, November 24-26, 1993, pages 65–68. IEEE, 1993

1993
[43]

A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. CoRR, abs/1606.04671, 2016

2016 arXiv
[44]

E. S. Savage-Rumbaugh, R. A Sevcik, D.M. Rumbaugh, and E. Rubert. The capacity of animals to acquire language: do species differences have anything to say to us? Philosophical Transactions of the Royal Society of London. B, Biological Sciences , 308(1135):177–185, 1985

1985
[45]

S. P. Sawant. Understanding catastrophic forgetting for adaptive deep learning. In Proceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD), Mumbai, India, January 4-7, 2023 , pages 282–283. ACM, 2023

2023
[46]

Serr `a, D

J. Serr `a, D. Suris, M. Miron, and A. Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm¨assan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Pro...

2018
[47]

F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. V osoughi, H. W. Chung, Y . Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, ...

2023
[48]

Shojaee, I

P. Shojaee, I. Mirzadeh, K. Alizadeh, M. Horton, S. Bengio, and M. Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025

2025
[49]

Shorten and T

C. Shorten and T. M. Khoshgoftaar. A survey on image data augmen- tation for deep learning. J. Big Data , 6:60, 2019

2019
[50]

Singampalli, D

A. Singampalli, D. Gufran, and S. Pasricha. CIELO: class- incremental continual learning for overcoming catastrophic forgetting with smartphone-based indoor localization. IEEE Access , 13:68536– 68546, 2025

2025
[51]

W Smith, J

J. W Smith, J. E. Everhart, W.C. Dickson, W. C. Knowler, and R. S. Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care , page 261. American Medical Informatics As...

1988
[52]

Srivastava, G

N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. , 15(1):1929–1958, 2014

1929
[53]

A. M. Turing. Computing machinery and intelligence. Mind, LIX(236):433–460, 1950

1950
[54]

Varis and O

D. Varis and O. Bojar. Unsupervised pretraining for neural machine translation using elastic weight consolidation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August 2, 2019, Volume 2: Student Res...

2019
[55]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long B...

2017
[56]

Y . Wen, X. Liu, and H. Yu. Adaptive tree-like neural network: Over- coming catastrophic forgetting to classify streaming data with concept drifts. Knowl. Based Syst. , 293:111636, 2024

2024
[57]

Wolberg, W

W. Wolberg, W. Street, and O. Mangasarian. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1995

1995
[58]

R. Yang, T. Yang, Z. Yan, T. Krajn ´ık, and Y . Ruichek. Preventing catastrophic forgetting in continuous online learning for autonomous driving. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2024, Abu Dhabi, United Arab Emirates, October 14-18, ...

2024
[59]

S. Yang, E. Gribovskaya, N. Kassner, M. Geva, and S. Riedel. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 1...

2024
[60]

Zenke, B

F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research , pages 398...

2017
[61]

Zhong, Q

L. Zhong, Q. Sui, Y . Todo, J. Tang, and S. Gao. Mitigating catastrophic forgetting through knowledge transfer and weighted loss integration in continual learning. In IEEE International Conference on Networking, Sensing and Control, ICNSC 2024, Hangzhou, China, October 18-20, ...

2024
[62]

Y . Zhou, R. Jiao, D. Wang, J. Mu, and J. Li. Catastrophic forgetting problem in semi-supervised semantic segmentation. IEEE Access , 10:48855–48864, 2022

2022
[63]

L. Zhu, D. Jeon, W. Sun, L. Yang, C. Y . Xie, and S. Niu. Flexible memory rotation (FMR): rotated representation with dynamic regular- ization to overcome catastrophic forgetting in continual knowledge graph learning. In IEEE International Conference on Big Data, BigData 2024,...

2024
[237]

Association for Computational Linguistics, 2024

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024: Student Research Workshop, St. Julian’s, Malta, March 21-22, 2024, pages 225–

work page 2024

[2] [2]

E. L. Aleixo, J. G. Colonna, M. Cristo, and E. Fernandes. Catastrophic forgetting in deep learning: A comprehensive taxonomy. J. Braz. Comput. Soc., 30(1), 2024. IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, VOL. XX, NO. X, AUGUST 2025 8

work page 2024

[3] [3]

Atkinson, B

C. Atkinson, B. McCane, L. Szymanski, and A. V . Robins. Pseudo- rehearsal: Achieving deep reinforcement learning without catastrophic forgetting. Neurocomputing, 428:291–307, 2021

work page 2021

[4] [4]

R. A. Bafghi, N. Harilal, C. Monteleoni, and M. Raissi. Parameter effi- cient fine-tuning of self-supervised vits without catastrophic forgetting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 - Workshops, Seattle, WA, USA, June 17-18, 2024 , pages 3679–3684. IEEE, 2024

work page 2024

[5] [5]

Bardes, J

A. Bardes, J. Ponce, and Y . LeCun. MC-JEPA: A joint-embedding predictive architecture for self-supervised learning of motion and content features. CoRR, abs/2307.12698, 2023

arXiv 2023

[6] [6]

Y . Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade - Second Edition, volume 7700 of Lecture Notes in Computer Science, pages 437–

work page

[7] [7]

Bhatt, J

G. Bhatt, J. Ross, and L. Sigal. Preventing catastrophic forgetting through memory networks in continuous detection. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29- October 4, 2024, Proceedings, Part LXXXIV , volume 15142 of Lecture Notes in Computer Science , pages 442–458. Springer, 2024

work page 2024

[8] [8]

C. Biever. Chatgpt broke the turing test-the race is on for new ways to assess ai. Nature, 619(7971):686–689, 2023

work page 2023