Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learning

Jihie Kim; Zahid Ullah

REVIEW 4 major objections 6 minor 45 references

Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KAN-based ViTs reduce catastrophic forgetting in continual learning

desk verdict The paper's own Table 3 contradicts its central claim, and the experimental numbers are too low to be credible; this needs a desk reject. read the letter →

arxiv 2507.04020 v1 pith:4BZKMT3G submitted 2025-07-05 cs.CV

Zahid Ullah , Jihie Kim This is my paper

classification cs.CV

keywords continuallearningcatastrophicforgettingKolmogorov-Arnoldnetworksvisiontransformerssplineactivationslocalplasticityincrementalimageclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that replacing the MLP layers inside a vision transformer with Kolmogorov-Arnold network (KAN) layers reduces catastrophic forgetting in continual image classification. The proposed mechanism is local plasticity: because KAN activations are B-splines, a training sample is said to update only the spline coefficients near its active region, leaving distant coefficients intact, unlike the global weight updates of MLPs. On sequential MNIST and CIFAR100 tasks, KAN-based ViTs report higher average incremental accuracy than MLP-based ViTs, with lower forgetting on MNIST and mixed results on CIFAR100. If this holds, a simple architectural substitution, rather than replay or regularization, could make ViTs more stable in settings where tasks arrive over time.

What carries the argument

The load-bearing component is the KAN module: an MLP replacement whose edge functions are learnable B-splines, with activation $\phi(x) = w_b\, \mathrm{silu}(x) + w_s\, \mathrm{spline}(x)$, where $\mathrm{spline}(x) = \sum_i c_i B_i(x)$. The B-spline basis functions $B_i$ have compact local support, so in principle each input sample updates only a subset of coefficients $c_i$. This locality is the entire mechanism: it is what converts ordinary gradient descent into a mostly local update rule that preserves previously learned spline regions while new tasks adapt the rest.

What would settle it

Track, during the second task, the per-coefficient gradient norms in the KAN layers of a trained ViT-KAN; if coefficients far from a sample's active spline regions receive non-negligible updates, or if a parameter-matched MLP with equally sparse effective updates shows the same retention, the claim that spline locality causes the forgetting reduction is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that KANs inherit a form of local plasticity from their spline-based activations, and that embedding them in ViTs—replacing every MLP block with a KAN module while keeping self-attention intact—produces a model that forgets less when trained on a sequence of tasks. The authors ground this in the Kolmogorov-Arnold representation theorem and in the earlier observation that spline bases are local: a new sample affects only nearby spline coefficients, so previously learned regions of the parameter space remain approximately unchanged. Their experiments compare MLP-based ViTs with KAN-based ViTs under task-incremental splits of MNIST (5 tasks) and CIFAR100 (10 tasks), with and without replay, reporting improved average incremental accuracy for the KAN variant. They also report that the advantage is most visible in the early stages of incremental learning and that, with replay on CIFAR100, KAN-ViT achieves higher last-task accuracy while showing slightly higher average global forgetting.

Load-bearing premise

The paper assumes, without measurement, that the local support of B-splines translates into localized gradient updates in a fully trained ViT, so that new tasks leave old spline coefficients untouched.

Editorial extensions

If this is right

On the MNIST split, EfficientKAN outperforms MLP on all three continual-learning metrics, with average incremental accuracy 52.2 percent versus 45.8 percent and average global forgetting 71.0 versus 95.2.
On the same split, ViT-KAN achieves higher average incremental accuracy than ViT-MLP (18.44 percent versus 17.70 percent) and lower average global forgetting (33.47 percent versus 35.16 percent), though lower last-task accuracy.
On CIFAR100, ViT-KAN again reports higher average incremental accuracy (15.49 percent versus 13.63 percent), but average global forgetting is higher for ViT-KAN (49.85 percent versus 44.67 percent).
Adding replay on CIFAR100 raises both models' incremental and last-task accuracy, and ViT-KAN with replay edges out ViT-MLP on both metrics.
In all ViT comparisons, the KAN advantage is concentrated in early learning stages; performance converges to that of MLP-based ViTs as tasks accumulate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If spline locality is the true cause, then explicitly measuring per-sample gradient support over spline coefficients would show sparsity; that measurement is absent from the paper and would test the mechanism directly.
The same mechanism suggests a cheap, testable extension: freeze or consolidate the coefficients active for earlier tasks while training later ones, which could amplify retention beyond what the paper reports.
Because the ViT comparisons used different training schedules and the CIFAR100 result shows higher forgetting, the paper's architecture-level conclusion is not yet disentangled from optimizer and schedule effects; controlled parameter-matched runs would isolate the KAN-specific contribution.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the paper is a direct combination of existing ideas—swap ViT MLPs for KANs and hope spline locality reduces forgetting—with no new mechanism or theory. Second, the experimental evidence does not support the abstract's claim of significant mitigation. Table 3 shows ViT-KAN has worse average global forgetting than ViT-MLP on CIFAR-100 (49.85 vs 44.67) and worse last-task accuracy on MNIST (4.67 vs 6.66). The only favorable metric is average incremental accuracy, which is not the metric that 'mitigating catastrophic forgetting' names. The abstract and conclusion say 'consistently outperform,' while Section 5 itself concedes convergence and trade-offs. That is an internal contradiction, not a minor gap.

What the paper does do well: it runs both MNIST and CIFAR-100, reports three CL metrics, and includes a replay variant. The standalone EfficientKAN vs MLP comparison in Table 2 goes in the expected direction. So the authors did run experiments, and the question they ask—does KAN locality survive inside a fully-trained ViT where all parameters get gradient updates—is legitimate.

The soft spots are load-bearing. The numbers themselves are absurd: last-task accuracies of 4-7% on a 10-class task are below random guessing, and the 10-class baseline would be 10%. That suggests a broken task/evaluation protocol, not a meaningful comparison. There are no error bars or seeds, so we cannot even tell if the small deltas are noise. The spline-locality mechanism is asserted from Liu et al. and never measured—no gradient locality analysis, no ablation. The citation pattern is sloppy: reference [30] is attributed to Liu et al. but is actually Ramasesh et al., and several other citations point to the wrong first authors. None of this is fatal to the underlying hypothesis, but it is fatal to this manuscript's ability to support it.

Who is this for? Possibly someone building on KAN architectures in continual learning might skim it, but no one should take its conclusions at face value. I would not bring it to reading group, and I would not cite it. It deserves a desk reject, not referee time, because the core evidence is self-contradictory and the experimental setup is not credible. If the authors fix the protocol, run multiple seeds, and honestly report the trade-offs, the question could be worth a proper study—but this version is not that study.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes replacing the MLP blocks in vision transformers (ViTs) with Kolmogorov-Arnold Network (KAN) layers to reduce catastrophic forgetting in continual learning. The authors introduce the architecture, describe experiments on MNIST and CIFAR-100 with class-sequential task splits (5 tasks and 10 tasks), and report average incremental accuracy, last-task accuracy, and average global forgetting for standalone MLPs/KANs and ViT-MLP versus ViT-KAN. The stated conclusion is that KAN-based ViTs significantly mitigate catastrophic forgetting and outperform MLP-based ViTs. The paper also claims a mechanistic explanation based on the local plasticity of spline activations.

Significance. If the central claim were supported, replacing MLPs with KANs would be a simple architectural intervention with practical relevance for continual learning with transformers. The paper does describe a clear experimental framework and provides a GitHub link, though the repository is stated as a future release rather than currently available code. However, the significance is undercut by the paper's own data: the principal comparison table shows mixed and partly contrary results, and the reported accuracies are at or below chance. The paper also does not provide error bars, multiple seeds, or direct evidence for the spline-locality mechanism. The manuscript therefore does not currently establish the claimed effect.

major comments (4)

[Section 5.1, Table 3] The abstract's claim that KAN-based ViTs 'significantly mitigate catastrophic forgetting, outperforming traditional MLP-based ViTs in knowledge retention' is contradicted by the paper's own table. On CIFAR-100, ViT-KAN has higher average global forgetting (49.85%) than ViT-MLP (44.67%); on MNIST, ViT-KAN has lower last-task accuracy (4.67%) than ViT-MLP (6.66%). The only consistently favorable metric is average incremental accuracy, which is not the metric that the phrase 'mitigating catastrophic forgetting' names. The central empirical claim is therefore not supported by the reported results.
[Section 5.1, Table 3 and Section 3.3] The reported last-task accuracies (4.67-6.66% on MNIST; 4.75-4.77% on CIFAR-100) are at or below the random baseline for the respective classification problems. Such values indicate that the training or evaluation protocol is unreliable, and numerical comparisons between models built on these numbers are difficult to interpret as evidence about forgetting.
[Section 4.2 and Tables 3-4] No standard deviations, number of seeds, or error bars are reported. Differences in average incremental accuracy (e.g., 18.44 vs 17.70 on MNIST; 17.23 vs 16.58 in Table 4) are small and could easily arise from run-to-run variation. Without repeated runs, the conclusion that KAN-based ViTs provide a meaningful advantage is unsupported.
[Section 2.8 and Section 1] The claimed mechanism, that spline locality ensures only a few coefficients are updated per sample, is asserted but never measured. The paper does not report any analysis of which B-spline coefficients change across tasks, nor does it compare gradient localization between KAN and MLP. Since this proposed mechanism is given as the causal explanation for the claimed improvement, its absence is a load-bearing gap. A concrete test would be to measure per-task changes in spline coefficients or gradient norms for task-specific samples.

minor comments (6)

[Throughout] The dataset name is inconsistently typeset as 'CIFAR100' and 'CIF AR100'; please standardize the spelling.
[Section 3.3] Equations (5)-(9) are improperly formatted and the definitions of A^b, LA, and F_G are unclear as typeset (for example, Eq. (8) omits task indices).
[Abstract and Section 1] The text says the code 'will be openly shared after publication' while also providing a GitHub URL; please clarify whether the code is currently available and ensure the link is to a populated repository.
[Section 2.4.2] The citation to Liu et al. [30] for KANs in continual learning appears mismatched with the reference list, where [30] is Ramasesh et al.; the KAN paper is [26]. Please correct the reference numbering.
[Section 2.7.2] The sentence 'This work aims to advance the state of task-based incremental learning by proposing novel replay-based mechanisms' is not supported by the actual contribution, which consists of a fixed replay loss scaling factor; this appears to be leftover boilerplate.
[Figure 3] Figure 3 is labeled as conceptual only; please either provide quantitative evidence for the depicted MLP-versus-KAN forgetting behavior or clearly mark the figure as illustrative without implying empirical support.

Circularity Check

0 steps flagged · score 0.0 of 10

No constructional circularity: the KAN-ViT claim rests on an empirical architecture comparison, not on a fitted parameter or definitional identity.

full rationale

The paper's derivation chain is: the Kolmogorov-Arnold theorem motivates spline-based KANs; spline locality, imported from Liu et al. [26], motivates replacing MLPs with KANs in ViTs; and experiments compare ViT-MLP with ViT-KAN on MNIST and CIFAR-100. No equation defines a prediction in terms of a fitted quantity, no parameter is fitted to a subset of data and then reported as a prediction on a closely related quantity, and the paper contains no self-citations by Ullah/Kim that carry a load-bearing uniqueness or ansatz argument. The central claim is an empirical comparison of reported metrics in Tables 2-4, so even though those tables partially contradict the abstract (e.g., CIFAR-100 average global forgetting is higher for ViT-KAN at 49.85 vs 44.67 for ViT-MLP, and MNIST last-task accuracy is lower at 4.67 vs 6.66), that is a correctness and internal-consistency problem, not a circularity problem. The spline-locality mechanism from [26] is imported as an unverified explanatory premise and is never directly measured; the conclusion attributes the outcome to this premise, which is a validation gap, but it is not a definitional reduction of the result to its inputs. The paper explicitly frames the work as empirical validation ('This research aims to empirically validate this claim', Section 2.8), and it concedes convergence limitations in Section 3.1.4 ('its performance converges with ViTs in later stages') and Section 5 ('the performance of KAN-ViT converged with that of MLP-ViT'). Under the constructional circularity standard, a score of 0 is appropriate.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an imported mechanism (spline locality), a set of hand-chosen hyperparameters, and an untested evaluation protocol. No new entities are introduced, but the key assumptions are asserted rather than verified.

free parameters (5)

Replay loss scaling factor = 0.5
Loss for replayed prior-task data is scaled by 0.5 to balance old and new knowledge; no sensitivity analysis is reported and the value is chosen by hand in Section 4.1.3.
Epoch schedule per task = MNIST: 7 then 5; CIFAR-100: 25 then 10
Training epochs for task 1 and subsequent tasks are set without justification or ablation; affects retention versus plasticity trade-off reported in Tables 2-4.
KAN grid size and spline order = not reported
The KAN module's B-spline grid size, spline order, and hidden widths are not specified, yet these control the locality and capacity of the activation functions.
ViT architectural hyperparameters = not reported
Patch size, depth, embedding width, and pre-training details of the chosen ViT are not given, making the comparison ill-defined.
Optimizer hyperparameters = not reported
Adam is mentioned, but learning rate, beta values, weight decay, and scheduling are absent, leaving the training procedure irreproducible.

assumptions (4)

standard math Kolmogorov-Arnold representation theorem
Invoked in Sections 1 and 2.4 to justify KANs' expressive power; assumed true as a classical result.
domain assumption Local plasticity of spline bases prevents catastrophic forgetting
Borrowed from Liu et al. [26] and stated as a premise in Sections 1 and 2.8; the paper never measures which parameters actually update, and this premise is the mechanism behind the central claim.
domain assumption KAN modules are drop-in replacements for ViT MLPs without altering training dynamics
Section 3.1.7 asserts dimension matching and similar computational footprint, but no stability or convergence analysis is provided.
domain assumption The evaluation protocol correctly measures task-incremental accuracy
The reported last-task accuracy of 4.67% on MNIST (Table 3) is below chance for 10 classes, indicating the protocol or task setup may be flawed; the paper does not check or discuss this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learning." pith.science (2026). https://pith.science/paper/4BZKMT3G

@misc{pith2026250704020,
  author       = {Pith},
  title        = {Pith review of: Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4BZKMT3G}},
  note         = {Machine review of arXiv:2507.04020}
}

read the original abstract

Continual learning (CL), the ability of a model to learn new tasks without forgetting previously acquired knowledge, remains a critical challenge in artificial intelligence, particularly for vision transformers (ViTs) utilizing Multilayer Perceptrons (MLPs) for global representation learning. Catastrophic forgetting, where new information overwrites prior knowledge, is especially problematic in these models. This research proposes replacing MLPs in ViTs with Kolmogorov-Arnold Network (KANs) to address this issue. KANs leverage local plasticity through spline-based activations, ensuring that only a subset of parameters is updated per sample, thereby preserving previously learned knowledge. The study investigates the efficacy of KAN-based ViTs in CL scenarios across benchmark datasets (MNIST, CIFAR100), focusing on their ability to retain accuracy on earlier tasks while adapting to new ones. Experimental results demonstrate that KAN-based ViTs significantly mitigate catastrophic forgetting, outperforming traditional MLP-based ViTs in knowledge retention and task adaptation. This novel integration of KANs into ViTs represents a promising step toward more robust and adaptable models for dynamic environments.

Figures

Figures reproduced from arXiv: 2507.04020 by the authors.

**Figure 2.** Decision tree for the three CL scenarios. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Catastrophic forgetting MLP vs KAN. The MLP waveforms become distorted as [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Our proposed methodology replaces the MLP layers in the selected ViT model [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

**Figure 5.** Figure 5: Images from MNIST dataset. classes. Similar to the MNIST setup, the tasks were arranged sequentially, with the first task involving classes 0 to 9, the second task covering classes 10 to 19, and so on, until all 100 classes were utilized. This division introduced a gre…

**Figure 6.** Figure 6: Images from CIFAR100 dataset. internal mechanisms to mitigate catastrophic forgetting. This experimental design provided a controlled and rigorous environment to assess the potential of KAN and its integration with ViT for CL applications. 3.3. Evaluation Metrics We ha…

**Figure 7.** Figure 7: Base level accuracy of MLP and efficientKAN on CIFAR-100 dataset. [PITH_FULL_IMAGE:figures/full_fig_p033_7.png]

**Figure 8.** Figure 8: Base level loss of MLP and efficientKAN on CIFAR-100 dataset. [PITH_FULL_IMAGE:figures/full_fig_p033_8.png]

**Figure 9.** Figure 9: Base level accuracy of MLP and efficientKAN on MNIST dataset. [PITH_FULL_IMAGE:figures/full_fig_p034_9.png]

**Figure 10.** Figure 10: Base level loss of MLP and efficientKAN on MNIST dataset. [PITH_FULL_IMAGE:figures/full_fig_p035_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 33 canonical work pages

[30]

V. V. Ramasesh, E. Dyer, M. Raghu, Anatomy of catastrophic for- getting: Hidden representations and task semantics, arXiv preprint arXiv:2007.07400 (2020). 39

arXiv 2020
[1]

L. Wang, X. Zhang, H. Su, J. Zhu, A comprehensive survey of contin- ual learning: Theory, method and application, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

work page 2024
[2]

G. M. Van de Ven, T. Tuytelaars, A. S. Tolias, Three types of incre- mental learning, Nature Machine Intelligence 4 (2022) 1185–1197

work page 2022
[3]

Personalized Artificial General Intelligence (AGI) via Neuroscience-Inspired Continuous Learning Systems

R. Gupta, S. Gupta, R. Parikh, D. Gupta, A. Javaheri, J. S. Shaktawat, Personalized artificial general intelligence (agi) via neuroscience-inspired continuous learning systems, arXiv preprint arXiv:2504.20109 (2025)

work page Pith review arXiv 2025
[4]

Understanding Continual Learning Settings with Data Distribution Drift Analysis

T. Lesort, M. Caccia, I. Rish, Understanding continual learning settings with data distribution drift analysis, arXiv preprint arXiv:2104.01678 (2021)

work page Pith review arXiv 2021
[5]

Lesort, V

T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, N. D ´ ıaz- Rodr ´ ıguez, Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges, Information fusion 58 (2020) 52–68

work page 2020
[6]

X. Xu, J. Chen, D. Thakur, D. Hong, Multi-modal disease segmentation with continual learning and adaptive decision fusion, Information Fusion (2025) 102962

work page 2025
[7]

G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, S. Wermter, Continual lifelong learning with neural networks: A review, Neural networks 113 (2019) 54–71

2019

Show all 45 references

[8]

Aljundi, P

R. Aljundi, P. Chakravarty, T. Tuytelaars, Expert gate: Lifelong learn- ing with a network of experts, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3366–3375

2017
[9]

De Lange, R

M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, T. Tuytelaars, A continual learning survey: Defying for- getting in classification tasks, IEEE transactions on pattern analysis and machine intelligence 44 (2021) 3366–3385

2021
[10]

Z. Ke, B. Liu, H. Xu, L. Shu, Classic: Continual and contrastive learning of aspect sentiment classification tasks, arXiv preprint arXiv:2112.02714 (2021). 37

2021 arXiv
[11]

M. J. Mirza, M. Masana, H. Possegger, H. Bischof, An efficient domain- incremental learning approach to drive in all weather conditions, in: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, 2022, pp. 3001–3011

2022
[12]

Uguroglu, J

S. Uguroglu, J. Carbonell, Feature selection for transfer learning, in: Joint European Conference on Machine Learning and Knowledge Dis- covery in Databases, Springer, 2011, pp. 430–442

2011
[13]

Mounsaveng, D

S. Mounsaveng, D. Vazquez, I. B. Ayed, M. Pedersoli, Adversarial learn- ing of general transformations for data augmentation, arXiv preprint arXiv:1909.09801 (2019)

2019 arXiv
[14]

D. Li, T. Wang, J. Chen, K. Kawaguchi, C. Lian, Z. Zeng, Multi-view class incremental learning, Information Fusion 102 (2024) 102021

2024
[15]

X. Tao, X. Hong, X. Chang, S. Dong, X. Wei, Y. Gong, Few-shot class- incremental learning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 12183–12192

2020
[16]

Masana, X

M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, J. Van De Weijer, Class-incremental learning: survey and performance evalu- ation on image classification, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (2022) 5513–5533

2022
[17]

Cossu, G

A. Cossu, G. Graffieti, L. Pellegrini, D. Maltoni, D. Bacciu, A. Carta, V. Lomonaco, Is class-incremental enough for continual learning?, Fron- tiers in Artificial Intelligence 5 (2022) 829842

2022
[18]

Masip, P

S. Masip, P. Rodriguez, T. Tuytelaars, G. M. van de Ven, Continual learning of diffusion models with generative distillation, arXiv preprint arXiv:2311.14028 (2023)

2023 arXiv
[19]

L. Liu, X. Li, M. Thakkar, X. Li, S. Joty, L. Si, L. Bing, Towards robust low-resource fine-tuning with multi-view compressed representa- tions, arXiv preprint arXiv:2211.08794 (2022)

2022 arXiv
[20]

H. Song, M. Kim, D. Park, Y. Shin, J.-G. Lee, Learning from noisy labels with deep neural networks: A survey, IEEE transactions on neural networks and learning systems 34 (2022) 8135–8153. 38

2022
[21]

Dahan, L

S. Dahan, L. Z. Williams, A. Fawaz, D. Rueckert, E. C. Robinson, Sur- face analysis with vision transformers, arXiv preprint arXiv:2205.15836 (2022)

2022 arXiv
[22]

S. A. Grosz, A. K. Jain, Afr-net: Attention-driven fingerprint recogni- tion network, IEEE Transactions on biometrics, behavior, and identity science 6 (2023) 30–42

2023
[23]

Kheddar, Transformers and large language models for efficient in- trusion detection systems: A comprehensive survey, Information Fusion (2025) 103347

H. Kheddar, Transformers and large language models for efficient in- trusion detection systems: A comprehensive survey, Information Fusion (2025) 103347

2025
[24]

Dosovitskiy, L

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 (2020)

2020 arXiv
[25]

Wu, Powerful design of small vision transformer on cifar10, arXiv preprint arXiv:2501.06220 (2025)

G. Wu, Powerful design of small vision transformer on cifar10, arXiv preprint arXiv:2501.06220 (2025)

2025 arXiv
[26]

Z. Liu, Y. Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Soljaˇ ci´ c, T. Y. Hou, M. Tegmark, Kan: Kolmogorov-arnold networks, arXiv preprint arXiv:2404.19756 (2024)

2024 arXiv
[27]

Y. Shen, P. Guo, J. Wu, Q. Huang, N. Le, J. Zhou, S. Jiang, M. Un- berath, Movit: Memorizing vision transformers for medical image anal- ysis, in: International Workshop on Machine Learning in Medical Imag- ing, Springer, 2023, pp. 205–213

2023
[28]

A. N. Kolmogorov, On the representations of continuous functions of many variables by superposition of continuous functions of one variable and addition, in: Dokl. Akad. Nauk USSR, volume 114, 1957, pp. 953– 956

1957
[29]

Krizhevsky, G

A. Krizhevsky, G. Hinton, et al., Learning multiple layers of features from tiny images (2009)

2009
[31]

A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, R. Hadsell, Progressive neural networks, arXiv preprint arXiv:1606.04671 (2016)

2016 arXiv
[32]

Rebuffi, A

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, C. H. Lampert, icarl: Incre- mental classifier and representation learning, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 2001–2010

2017
[33]

Lopez-Paz, M

D. Lopez-Paz, M. Ranzato, Gradient episodic memory for continual learning, Advances in neural information processing systems 30 (2017)

2017
[34]

X. Mao, G. Qi, Y. Chen, X. Li, R. Duan, S. Ye, Y. He, H. Xue, Towards robust vision transformer, in: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2022, pp. 12042–12051

2022
[35]

Aldahdooh, W

A. Aldahdooh, W. Hamidouche, O. Deforges, Reveal of vision transformers robustness against adversarial attacks, arXiv preprint arXiv:2106.03734 (2021)

2021 arXiv
[36]

Zhang, R

Z. Zhang, R. Xu, J. Zhou, C. Wang, X. Pei, W. Xu, J. Zhang, L. Guo, L. Gao, W. Xu, et al., Image recognition with online lightweight vision transformer: A survey, arXiv preprint arXiv:2505.03113 (2025)

2025
[37]

G. A. Pereira, M. Hussain, A review of transformer-based models for computer vision tasks: Capturing global context and spatial relation- ships, arXiv preprint arXiv:2408.15178 (2024)

2024 arXiv
[38]

Lee, D.-W

J. Lee, D.-W. Choi, Lossless token merging even without fine-tuning in vision transformers, arXiv preprint arXiv:2505.15160 (2025)

2025
[39]

M. Shi, Y. Zhou, R. Yu, Z. Li, Z. Liang, X. Zhao, X. Peng, S. R. Vedantam, W. Zhao, K. Wang, et al., Faster vision mamba is rebuilt in minutes via merged token re-training, arXiv preprint arXiv:2412.12496 (2024)

2024 arXiv
[40]

Sharma, V

N. Sharma, V. Jain, A. Mishra, An analysis of convolutional neural networks for image classification, Procedia computer science 132 (2018) 377–384. 40

2018
[41]

A. Basu, M. Sathya, Handwritten digit recognition using improved bounding box recognition technique, arXiv preprint arXiv:2111.05483 (2021)

2021 arXiv
[42]

Picek, M

L. Picek, M. ˇSulc, J. Matas, T. S. Jeppesen, J. Heilmann-Clausen, T. Læssøe, T. Frøslev, Danish fungi 2020-not just another image recog- nition dataset, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 1525–1535

2020
[43]

ParisiGerman, L

I. ParisiGerman, L. PartJose, et al., Continual lifelong learning with neural networks (2019)

2019
[44]

Theotokis, Human brain inspired artificial intelligence neural net- works, Journal of Integrative Neuroscience 24 (2025) 26684

P. Theotokis, Human brain inspired artificial intelligence neural net- works, Journal of Integrative Neuroscience 24 (2025) 26684

2025
[45]

Rahman, W

S. Rahman, W. Lee, Out of distribution performance of state of art vision model, arXiv preprint arXiv:2301.10750 (2023). 41

2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [30]

V. V. Ramasesh, E. Dyer, M. Raghu, Anatomy of catastrophic for- getting: Hidden representations and task semantics, arXiv preprint arXiv:2007.07400 (2020). 39

arXiv 2020

[2] [1]

L. Wang, X. Zhang, H. Su, J. Zhu, A comprehensive survey of contin- ual learning: Theory, method and application, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

work page 2024

[3] [2]

G. M. Van de Ven, T. Tuytelaars, A. S. Tolias, Three types of incre- mental learning, Nature Machine Intelligence 4 (2022) 1185–1197

work page 2022

[4] [3]

Personalized Artificial General Intelligence (AGI) via Neuroscience-Inspired Continuous Learning Systems

R. Gupta, S. Gupta, R. Parikh, D. Gupta, A. Javaheri, J. S. Shaktawat, Personalized artificial general intelligence (agi) via neuroscience-inspired continuous learning systems, arXiv preprint arXiv:2504.20109 (2025)

work page Pith review arXiv 2025

[5] [4]

Understanding Continual Learning Settings with Data Distribution Drift Analysis

T. Lesort, M. Caccia, I. Rish, Understanding continual learning settings with data distribution drift analysis, arXiv preprint arXiv:2104.01678 (2021)

work page Pith review arXiv 2021

[6] [5]

Lesort, V

T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, N. D ´ ıaz- Rodr ´ ıguez, Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges, Information fusion 58 (2020) 52–68

work page 2020

[7] [6]

X. Xu, J. Chen, D. Thakur, D. Hong, Multi-modal disease segmentation with continual learning and adaptive decision fusion, Information Fusion (2025) 102962

work page 2025

[8] [7]

G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, S. Wermter, Continual lifelong learning with neural networks: A review, Neural networks 113 (2019) 54–71

2019