Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning

Bartosz Krawczyk; Jan Wasilewski; J\k{e}drzej Kozal; Micha{\l} Wo\'zniak

arxiv: 2606.20431 · v1 · pith:HLILHVWJ · submitted 2026-06-18 · cs.LG

Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning

Jan Wasilewski , J\k{e}drzej Kozal , Micha{\l} Wo\'zniak , Bartosz Krawczyk This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 17:58 UTCgrok-4.3pith:HLILHVWJ record.json open to challenge →

classification cs.LG

keywords continual learningcatastrophic forgettingsuperpositionsparsityrepresentation retentionmechanistic study

0 comments

The pith

In synthetic continual learning tasks, higher superposition does not inevitably cause forgetting when representations remain strong.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper constructs a controlled synthetic setup to separate the factors that cause forgetting in continual learning systems. Tasks are generated with adjustable sparsity in latent features and measurable overlap between them. Tracking how representation strength changes over time shows that superposition grows gradually but drops at task boundaries. Forgetting is reduced when representations stay strong, even with high overlap, and sparser features lead to using more capacity across tasks as measured by effective rank. The results indicate that overlap interacts with strength and allocation rather than directly driving loss of prior knowledge.

Core claim

Using a synthetic generator-separator pipeline that defines ground-truth latent features with tunable sparsity and overlap, the authors introduce measurable quantities for representation strength and superposition as directional overlap. They fit sparse dynamical relations via SINDy to model retention dynamics from exposure history and analyze task-level capacity allocation via effective rank. Experiments show superposition tends to increase over time with transient dips at task boundaries, higher sparsity induces more superposition yet reduces forgetting when representations remain strong, and effective rank grows with sparsity indicating broader capacity usage. These outcomes demonstrate t

What carries the argument

The generator-separator pipeline for creating tasks with ground-truth features of controllable sparsity and overlap, together with quantities for representation strength and superposition as directional overlap among features.

If this is right

Superposition grows over time but shows temporary reductions at new task boundaries due to interference.
Higher feature sparsity produces more superposition without necessarily increasing forgetting if representation strength is preserved.
Task-level capacity allocation expands under sparser regimes as shown by growth in effective rank.
Retention of prior knowledge can be maintained through strong representations even when feature overlap is present.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The boundary-specific dips in superposition point to studying how models handle task switches as a way to reduce interference.
The same strength and overlap metrics could be computed on real datasets to check whether the observed interactions hold outside the synthetic setup.
Techniques that promote sparsity might help maintain retention without the forgetting penalty usually expected from increased overlap.

Load-bearing premise

The synthetic generator-separator pipeline with tunable sparsity and overlap faithfully isolates the mechanisms that drive forgetting in real continual-learning datasets.

What would settle it

Applying the same measurements of representation strength, superposition, and effective rank to a standard continual learning benchmark and finding that stronger representations fail to reduce forgetting despite high overlap would challenge the central claim.

Figures

Figures reproduced from arXiv: 2606.20431 by Bartosz Krawczyk, Jan Wasilewski, J\k{e}drzej Kozal, Micha{\l} Wo\'zniak.

**Figure 1.** Figure 1: Diagram of the experimental setup. The generator and separator are randomly initialized frozen neural networks () that generate data sample and label. The encoder, classifier, and decoder modules are trainable (), and learning is performed sequentially with a replay buffer. 3 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Figure 2: Heatmap of superposition and representation of the fea [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Dynamics of superposition and representation in time [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Figure 4: Evolution of feature retention as a function of representation strength and superposition. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

**Figure 5.** Figure 5: Sum of effective dimensions of all the tasks in time vs [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

**Figure 6.** Figure 6: Evolution of effective dimensions of tasks embeddings. The task is mapped into region with similar effective rank no matter if [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

**Figure 7.** Figure 7: Dynamics of superposition and representation in time [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

**Figure 8.** Figure 8: Dynamics of superposition and representation in time [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗

**Figure 9.** Figure 9: Effective dimensions of tasks vs sparsity-settings 2 [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗

**Figure 10.** Figure 10: Effective dimensions of tasks vs sparsity-settings 3 [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗

**Figure 11.** Figure 11: Instability of the method [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗

**Figure 12.** Figure 12: Instability of the method. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗

read the original abstract

Continual learning (CL) systems often forget previously acquired knowledge, yet the mechanisms driving forgetting remain hard to isolate in practice because real datasets entangle many factors. We present a controlled, toy-world framework that makes these mechanisms observable and testable. Using a synthetic generator-separator pipeline, we define ground-truth latent features, build tasks with tunable sparsity and overlap, and introduce measurable quantities for representation strength and superposition (directional overlap among features). We then study retention dynamics-the temporal change of representation strength by fitting sparse dynamical relations (via SINDy) between retention, superposition, and exposure history. A complementary task-level analysis based on effective rank characterizes how representational capacity is allocated across tasks. Our controlled experiments yield three takeaways. (1) Superposition tends to increase over time with transient dips at task boundaries, suggesting boundary-specific interference rather than steady drift. (2) Higher feature sparsity induces more superposition yet does not inevitably cause forgetting; when representations remain strong, forgetting can be reduced despite overlap. (3) Task-level effective rank grows with sparsity, indicating broader capacity usage under sparse regimes. Together, these results nuance the common intuition that more superposition leads to more forgetting by showing that overlap interacts with representation strength and capacity allocation. Our toy analysis provides falsifiable hypotheses and diagnostic tools for CL.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper introduces measurable quantities for representation strength and directional superposition in a synthetic generator-separator setup, then uses SINDy to track retention dynamics and shows overlap interacts with strength rather than always increasing forgetting.

read the letter

The core contribution is a controlled synthetic pipeline that lets them define ground-truth latents, tune sparsity and overlap, and track how representation strength changes over tasks. They add directional superposition as a measurable overlap quantity and fit sparse dynamical models with SINDy to retention, superposition, and exposure. A task-level effective rank analysis shows how capacity gets allocated.

This setup produces three concrete observations: superposition grows over time but dips at boundaries, higher sparsity increases superposition without forcing forgetting when strength stays high, and effective rank rises with sparsity. Those relations are new in the CL literature they cite and give falsifiable hypotheses plus diagnostic tools.

The experiments stay inside the toy generator-separator world. The abstract describes the pipeline and SINDy fits at a high level, so it is hard to judge robustness, exact library terms, or whether error bars back the claims. The stress-test concern holds: real continual learning mixes semantic structure, non-stationary gradients, and optimizer effects that the separable synthetic features do not include. The observed interactions could be artifacts of the separation assumptions.

The work is aimed at researchers who want mechanistic diagnostics inside controlled regimes. It is worth a serious referee because the framework is explicit, the quantities are reproducible in principle, and the claims are scoped to the model rather than overclaimed for real networks. Minor revisions could tighten the experimental reporting and add a clearer discussion of when the toy results are expected to transfer.

Referee Report

2 major / 0 minor

Summary. The manuscript introduces a controlled synthetic generator-separator pipeline to isolate mechanisms of representation retention and forgetting in continual learning. Ground-truth latent features are defined with tunable sparsity and overlap; measurable quantities for representation strength and directional superposition are introduced; retention dynamics are modeled by fitting sparse dynamical relations via SINDy; and capacity allocation is characterized via task-level effective rank. The central empirical takeaways are that superposition grows over time with transient dips at task boundaries, higher sparsity induces more superposition without inevitable forgetting when representations remain strong, and effective rank grows with sparsity, together showing that overlap interacts with strength and capacity allocation to nuance the superposition-forgetting intuition.

Significance. If the synthetic pipeline's separation assumptions hold, the work supplies a falsifiable, toy-world testbed that generates concrete hypotheses (boundary-specific interference, strength-modulated forgetting) and diagnostic tools (SINDy fits, effective-rank tracking) for continual-learning research. The controlled variation of sparsity/overlap and the complementary dynamical and rank analyses are strengths that could help disentangle factors otherwise entangled in real datasets.

major comments (2)

[Abstract] Abstract (paragraph describing the framework): the claim that the generator-separator pipeline 'faithfully isolates the mechanisms that drive forgetting in real continual-learning datasets' is load-bearing for all three takeaways yet rests on an untested assumption; no validation against real CL benchmarks, no ablation of entangled factors (semantic structure, non-stationary gradients, optimizer effects), and no demonstration that the observed superposition-strength relations survive when those factors are re-introduced.
[Abstract] Abstract (takeaway 2): the statement that 'when representations remain strong, forgetting can be reduced despite overlap' is presented as a general nuance, but the manuscript provides no quantitative threshold or cross-condition test showing that the strength-superposition interaction survives outside the synthetic separation assumptions.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We agree that the original abstract phrasing overstated the generality of the synthetic pipeline and will revise it to emphasize that the work provides controlled, falsifiable hypotheses rather than claiming direct isolation of mechanisms from real datasets. Below we respond point by point.

read point-by-point responses

Referee: [Abstract] Abstract (paragraph describing the framework): the claim that the generator-separator pipeline 'faithfully isolates the mechanisms that drive forgetting in real continual-learning datasets' is load-bearing for all three takeaways yet rests on an untested assumption; no validation against real CL benchmarks, no ablation of entangled factors (semantic structure, non-stationary gradients, optimizer effects), and no demonstration that the observed superposition-strength relations survive when those factors are re-introduced.

Authors: We accept this criticism. The manuscript positions the pipeline as a controlled toy-world testbed for generating hypotheses, not as a faithful replica of real CL. The original wording was imprecise. We will revise the abstract to remove the phrase 'faithfully isolates the mechanisms that drive forgetting in real continual-learning datasets' and replace it with language stating that the framework 'makes these mechanisms observable and testable in a controlled setting, yielding hypotheses for real CL.' No real-benchmark validation or ablation of semantic/optimizer factors will be added, as that would exceed the stated scope of isolating factors via synthetic control; the paper already notes its toy nature in the final paragraph. revision: yes
Referee: [Abstract] Abstract (takeaway 2): the statement that 'when representations remain strong, forgetting can be reduced despite overlap' is presented as a general nuance, but the manuscript provides no quantitative threshold or cross-condition test showing that the strength-superposition interaction survives outside the synthetic separation assumptions.

Authors: The takeaway is derived from the controlled variation of representation strength and overlap within the synthetic generator-separator setup; multiple sparsity and overlap conditions demonstrate that strong representations mitigate forgetting even under higher superposition. We do not claim a universal quantitative threshold, as strength is a continuous variable in the model. We agree the interaction has not been tested outside the synthetic assumptions. We will add a clarifying clause to the abstract takeaway stating that the interaction is observed 'under the controlled synthetic conditions' and will expand the discussion section to explicitly note the scope limitation and the need for future work to test the hypothesis in real-data settings. revision: partial

Circularity Check

0 steps flagged

No significant circularity; empirical observations on synthetic data

full rationale

The paper constructs a synthetic generator-separator pipeline to generate tasks with explicit sparsity/overlap parameters, measures representation strength and superposition directly from the generated latents, fits SINDy models to the resulting time series, and reports observational patterns (e.g., superposition growth with boundary dips, effective-rank growth with sparsity). None of these steps reduce by the paper's own equations to a fitted parameter that is then relabeled as a prediction, nor do they rely on self-citations, uniqueness theorems, or ansatzes imported from prior work by the same authors. The central claims are falsifiable experimental outcomes from the controlled toy world rather than tautological restatements of inputs. This is a standard empirical mechanistic study whose derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claims rest on the assumption that the synthetic pipeline isolates the relevant mechanisms; SINDy fitting introduces free parameters whose values are not reported in the abstract.

free parameters (1)

SINDy sparsity threshold and library terms
SINDy requires choice of library functions and sparsity parameter to produce the dynamical relations between retention, superposition, and exposure.

axioms (1)

domain assumption SINDy can recover the governing dynamical relations from observed time series of representation strength and superposition
Invoked when fitting sparse dynamical relations to retention dynamics.

pith-pipeline@v0.9.1-grok · 5786 in / 1223 out tokens · 20767 ms · 2026-06-26T17:58:53.889567+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

24 extracted references · 2 canonical work pages · 1 internal anchor

[1]

Network dissection: Quantifying inter- pretability of deep visual representations, 2017

David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying inter- pretability of deep visual representations, 2017

2017
[2]

Mechanistic inter- pretability for ai safety – a review, 2024

Leonard Bereska and Efstratios Gavves. Mechanistic inter- pretability for ai safety – a review, 2024

2024
[3]

Brunton, Joshua L

Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identi- fication of nonlinear dynamical systems.Proceedings of the National Academy of Sciences, 113(15):3932–3937, March 2016

2016
[4]

New insights on reducing abrupt representation change in online continual learning, 2022

Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuyte- laars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning, 2022

2022
[5]

Curve detectors.Dis- till, 2020

Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schu- bert, Michael Petrov, and Chris Olah. Curve detectors.Dis- till, 2020. https://distill.pub/2020/circuits/curve-detectors

2020
[6]

Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. Continual learning with tiny episodic memories.CoRR, abs/1902.10486, 2019

work page internal anchor Pith review Pith/arXiv arXiv 1902
[7]

Zhiyuan Chen and B. Liu. Lifelong machine learning, sec- ond edition.Synthesis Lectures on Artificial Intelligence and Machine Learning, 2018

2018
[8]

Online bias correction for task-free continual learning

Aristotelis Chrysakis and Marie-Francine Moens. Online bias correction for task-free continual learning. InThe Eleventh International Conference on Learning Representa- tions, 2023

2023
[9]

Probing representation for- getting in supervised and unsupervised continual learning, 2022

MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Probing representation for- getting in supervised and unsupervised continual learning, 2022

2022
[10]

Pure: Turn- ing polysemantic neurons into pure features by identifying relevant circuits.arXiv preprint arXiv:2404.06453, 2024

Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojciech Samek, and Sebastian Lapuschkin. Pure: Turn- ing polysemantic neurons into pure features by identifying relevant circuits.arXiv preprint arXiv:2404.06453, 2024

work page arXiv 2024
[11]

Toy models of superposition, 2022

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022

2022
[12]

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Das- Sarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah....

2021
[13]

Catastrophic forgetting in connectionist net- works.Trends in cognitive sciences, 3:128–135, 05 1999

Robert French. Catastrophic forgetting in connectionist net- works.Trends in cognitive sciences, 3:128–135, 05 1999

1999
[14]

Multimodal neurons in artificial neural net- works.Distill, 2021

Gabriel Goh, Nick Cammarata †, Chelsea V oss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural net- works.Distill, 2021. https://distill.pub/2021/multimodal- neurons. 9

2021
[15]

What variables affect out- of-distribution generalization in pretrained models?, 2024

Md Yousuf Harun, Kyungbok Lee, Jhair Gallardo, Giri Kr- ishnan, and Christopher Kanan. What variables affect out- of-distribution generalization in pretrained models?, 2024

2024
[16]

Knowledge accumulation in continually learned representations and the issue of feature forgetting, 2024

Timm Hess, Eli Verwimp, Gido M van de Ven, and Tinne Tuytelaars. Knowledge accumulation in continually learned representations and the issue of feature forgetting, 2024

2024
[17]

Coarse-to-fine grained classification

Yuqi Huo, Yao Lu, Yulei Niu, Zhiwu Lu, and Ji-Rong Wen. Coarse-to-fine grained classification. InProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, page 1033–1036, New York, NY , USA, 2019. Association for Computing Machinery

2019
[18]

Adversar- ial examples are not bugs, they are features

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversar- ial examples are not bugs, they are features. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019

2019
[19]

The tunnel effect: Building data representations in deep neural networks, 2023

Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miło ´s, and Tomasz Trzci ´nski. The tunnel effect: Building data representations in deep neural networks, 2023

2023
[20]

Zoom in: An introduction to circuits

Christopher Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. 2020

2020
[21]

The effective rank: A mea- sure of effective dimensionality

Olivier Roy and Martin Vetterli. The effective rank: A mea- sure of effective dimensionality. In2007 15th European Sig- nal Processing Conference, pages 606–610, 2007

2007
[22]

Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning

Grzegorz Rype ´s´c, Sebastian Cygert, Tomasz Trzci ´nski, and Bartł omiej Twardowski. Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 63268–63289...

2024
[23]

Large scale in- cremental learning, 2019

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale in- cremental learning, 2019. 10

2019
[24]

SINDy Sparse Identification of Nonlinear Dynamical Systems (SINDy) is a data-driven method for discovering governing equations from time-series data

Appendix 9.1. SINDy Sparse Identification of Nonlinear Dynamical Systems (SINDy) is a data-driven method for discovering governing equations from time-series data. Given a system with state variablesx(t), SINDy assumes that their evolution follows a sparse set of nonlinear interactions: dx dt =f(x).(7) To identifyf(x), SINDy constructs a library of candi-...

2000

[1] [1]

Network dissection: Quantifying inter- pretability of deep visual representations, 2017

David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying inter- pretability of deep visual representations, 2017

2017

[2] [2]

Mechanistic inter- pretability for ai safety – a review, 2024

Leonard Bereska and Efstratios Gavves. Mechanistic inter- pretability for ai safety – a review, 2024

2024

[3] [3]

Brunton, Joshua L

Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identi- fication of nonlinear dynamical systems.Proceedings of the National Academy of Sciences, 113(15):3932–3937, March 2016

2016

[4] [4]

New insights on reducing abrupt representation change in online continual learning, 2022

Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuyte- laars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning, 2022

2022

[5] [5]

Curve detectors.Dis- till, 2020

Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schu- bert, Michael Petrov, and Chris Olah. Curve detectors.Dis- till, 2020. https://distill.pub/2020/circuits/curve-detectors

2020

[6] [6]

Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. Continual learning with tiny episodic memories.CoRR, abs/1902.10486, 2019

work page internal anchor Pith review Pith/arXiv arXiv 1902

[7] [7]

Zhiyuan Chen and B. Liu. Lifelong machine learning, sec- ond edition.Synthesis Lectures on Artificial Intelligence and Machine Learning, 2018

2018

[8] [8]

Online bias correction for task-free continual learning

Aristotelis Chrysakis and Marie-Francine Moens. Online bias correction for task-free continual learning. InThe Eleventh International Conference on Learning Representa- tions, 2023

2023

[9] [9]

Probing representation for- getting in supervised and unsupervised continual learning, 2022

MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Probing representation for- getting in supervised and unsupervised continual learning, 2022

2022

[10] [10]

Pure: Turn- ing polysemantic neurons into pure features by identifying relevant circuits.arXiv preprint arXiv:2404.06453, 2024

Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojciech Samek, and Sebastian Lapuschkin. Pure: Turn- ing polysemantic neurons into pure features by identifying relevant circuits.arXiv preprint arXiv:2404.06453, 2024

work page arXiv 2024

[11] [11]

Toy models of superposition, 2022

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022

2022

[12] [12]

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Das- Sarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah....

2021

[13] [13]

Catastrophic forgetting in connectionist net- works.Trends in cognitive sciences, 3:128–135, 05 1999

Robert French. Catastrophic forgetting in connectionist net- works.Trends in cognitive sciences, 3:128–135, 05 1999

1999

[14] [14]

Multimodal neurons in artificial neural net- works.Distill, 2021

Gabriel Goh, Nick Cammarata †, Chelsea V oss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural net- works.Distill, 2021. https://distill.pub/2021/multimodal- neurons. 9

2021

[15] [15]

What variables affect out- of-distribution generalization in pretrained models?, 2024

Md Yousuf Harun, Kyungbok Lee, Jhair Gallardo, Giri Kr- ishnan, and Christopher Kanan. What variables affect out- of-distribution generalization in pretrained models?, 2024

2024

[16] [16]

Knowledge accumulation in continually learned representations and the issue of feature forgetting, 2024

Timm Hess, Eli Verwimp, Gido M van de Ven, and Tinne Tuytelaars. Knowledge accumulation in continually learned representations and the issue of feature forgetting, 2024

2024

[17] [17]

Coarse-to-fine grained classification

Yuqi Huo, Yao Lu, Yulei Niu, Zhiwu Lu, and Ji-Rong Wen. Coarse-to-fine grained classification. InProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, page 1033–1036, New York, NY , USA, 2019. Association for Computing Machinery

2019

[18] [18]

Adversar- ial examples are not bugs, they are features

Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversar- ial examples are not bugs, they are features. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019

2019

[19] [19]

The tunnel effect: Building data representations in deep neural networks, 2023

Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miło ´s, and Tomasz Trzci ´nski. The tunnel effect: Building data representations in deep neural networks, 2023

2023

[20] [20]

Zoom in: An introduction to circuits

Christopher Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. 2020

2020

[21] [21]

The effective rank: A mea- sure of effective dimensionality

Olivier Roy and Martin Vetterli. The effective rank: A mea- sure of effective dimensionality. In2007 15th European Sig- nal Processing Conference, pages 606–610, 2007

2007

[22] [22]

Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning

Grzegorz Rype ´s´c, Sebastian Cygert, Tomasz Trzci ´nski, and Bartł omiej Twardowski. Task-recency bias strikes back: Adapting covariances in exemplar-free class incremental learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 63268–63289...

2024

[23] [23]

Large scale in- cremental learning, 2019

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale in- cremental learning, 2019. 10

2019

[24] [24]

SINDy Sparse Identification of Nonlinear Dynamical Systems (SINDy) is a data-driven method for discovering governing equations from time-series data

Appendix 9.1. SINDy Sparse Identification of Nonlinear Dynamical Systems (SINDy) is a data-driven method for discovering governing equations from time-series data. Given a system with state variablesx(t), SINDy assumes that their evolution follows a sparse set of nonlinear interactions: dx dt =f(x).(7) To identifyf(x), SINDy constructs a library of candi-...

2000