Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Autoencoder-Based Hybrid Replay for Class-Incremental Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An autoencoder that stores replay exemplars as compressed latent codes matches or beats existing class-incremental learning methods while using less memory.

desk verdict A useful empirical recipe with a broken headline claim: the O(0.1t) memory bound is not a valid asymptotic statement, and the experiments never test it, but the fixed-budget comparison of lossy latent replay is a real result worth refereeing. read the letter →

arxiv 2505.05926 v3 pith:OOXARSFZ submitted 2025-05-09 cs.LG

Milad Khademi Nori , Il-Min Kim , Guanghui Wang This is my paper · ORCID
classification cs.LG
keywords class-incrementallearningcatastrophicforgettingexemplarreplaygenerativehybridautoencoderlatentspacecompressionrepulsiveforcealgorithm
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Class-incremental learning systems must remember old classes while absorbing new ones, usually by replaying stored exemplars; this paper argues that storing those exemplars in the compressed latent space of a purpose-built autoencoder, rather than as raw images, gives the accuracy of exemplar replay with a fraction of the memory. The proposed strategy, AHR, pairs a hybrid autoencoder (HAE) with a repulsive-force placement of class centroids so that the same network both reconstructs old data for replay and classifies new data by nearest centroid. On five benchmarks and against ten baselines, the paper reports that AHR outperforms recent replay, generative, and hybrid methods under matched memory and compute budgets. If the strategy holds up, it would let continual learners keep more exemplars per class and scale to longer task streams on memory-constrained devices.

What carries the argument

The central object is the hybrid autoencoder (HAE), an encoder-decoder network whose objective is the sum of an $L^2$ reconstruction term and a clustering term that pulls each latent code toward its class centroid embedding (CCE). The class centroid embeddings are placed and kept apart by the repulsive force algorithm (RFA), which treats centroids as charged particles and minimizes Coulomb-style potential energy through Euler-Lagrange equations; this determines where each new class's centroid sits before training. The episodic memory holds latent codes rather than raw data, and the decoder's job is deliberately memorization rather than generalization, so decoded exemplars are near-lossless. At test time the encoder plus nearest-centroid matching performs classification. The same machinery therefore carries both replay and discrimination.

What would settle it

Keep the number of stored exemplars per class constant while increasing the number of tasks $t$, and measure the actual memory bytes consumed by AHR's latent exemplars plus its decoder against raw exemplar storage; if the total grows with the same slope as raw storage, or if the decoder's fixed cost dominates on short task streams, the central memory-complexity claim is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that replay memory for class-incremental learning can be compressed without losing the benefits of exact exemplar replay. Where exemplar replay stores raw samples and generative replay fabricates pseudo-data, AHR encodes each exemplar into a low-dimensional latent vector, about ten times smaller than the input in the reported setups, stores those vectors in an episodic memory, and decodes them on demand during training. The same hybrid autoencoder is trained with a loss that simultaneously minimizes reconstruction error and clusters same-class samples around fixed class-centroid embeddings; the centroids are placed incrementally with the charged-particle-system energy minimization and repulsive-force algorithm so that new classes push apart from old ones in latent space. At inference, classification is nearest-centroid in the latent space, with no task identifiers. The paper claims this yields state-of-the-art performance across MNIST, SVHN, CIFAR-10, CIFAR-100, and miniImageNet while cutting worst-case memory from $O(t)$ to $O(0.1t)$ at the same $O(t)$ compute.

Load-bearing premise

The memory-saving case assumes a tenfold compression of stored exemplars is a real asymptotic reduction, yet $0.1t$ grows exactly like $t$, and the experiments hold total memory fixed instead of letting it grow with the number of tasks.

Editorial extensions

If this is right

  • AHR can be grafted onto existing exemplar-replay pipelines, replacing stored raw images with stored latent codes and decoding them during rehearsal.
  • Under a fixed memory budget, the compression lets the learner store roughly ten times more exemplars per class, increasing exemplar diversity and reducing catastrophic forgetting.
  • Because classification happens directly in latent space by nearest centroid, the approach does not need task identifiers at test time and can operate in task-free settings.
  • The reported gains are largest when memory is small, suggesting compressed latent replay is most useful for long task streams and low-memory devices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The asymptotic claim $O(0.1t)$ should be read as a constant-factor compression: in standard big-O semantics $0.1t$ and $t$ have the same growth rate, so the real promise is a tenfold constant-factor saving, not a new complexity class.
  • The evaluation fixes the total exemplar budget (200 or 2000) across all task counts, so the advertised memory reduction as $t$ grows is not directly measured; a test that grows $t$ while holding exemplars per class fixed would isolate it.
  • Because the decoder is a fixed cost that does not grow with $t$, the break-even point depends on task-stream length; on very short streams the decoder overhead could outweigh memory savings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes AHR, a class-incremental learning strategy that stores exemplars not as raw data but as compressed latent vectors produced by a hybrid autoencoder (HAE), and uses a charged-particle-system-inspired repulsive force algorithm (RFA) to place class centroids in the latent space. The authors claim that AHR reduces memory complexity from O(t) to O(0.1t) at the worst case, while keeping compute complexity at O(t), and report state-of-the-art accuracy across five benchmarks compared with ten baselines. The paper includes an extensive experimental section with ablations on compression, latent-space structuring methods, and resource consumption.

Significance. If the central complexity claim were valid, AHR would be a notable contribution: it would show that compressed latent replay can match or exceed raw exemplar replay while using an order of magnitude less memory. The empirical study is broad: five benchmarks, many baselines, ablation of lossy versus lossless replay, comparisons of latent-space structuring methods (RFA versus contrastive versus GMM), and resource-consumption experiments. The practical finding that, at a fixed memory budget, storing more lossy latent exemplars can outperform storing fewer raw exemplars is consistent with prior compressed-replay work (REMIND, i-CTRL) and is a useful confirmation. However, the advertised asymptotic memory reduction is not a valid complexity claim, and the experimental protocol, which fixes the total exemplar budget, does not measure memory scaling with the number of tasks.

major comments (3)
  1. [Abstract, Table 1, Appendix A.4] The claim that AHR achieves O(0.1t) memory complexity is not meaningful under standard asymptotic notation: O(0.1t) = O(t), since constant factors are subsumed by the definition of big-O. The derivation in Appendix A.4, which concludes that a roughly 10x compression ratio 'becomes O(0.1t)', confuses a constant-factor saving at a fixed task count with a change in growth rate. At most, the authors can claim a constant-factor memory reduction at a given t; the asymptotic complexity of both AHR and exemplar replay remains O(t). This affects the abstract, Table 1, Section 1, Section 2, and Appendix A.4, and is the paper's headline contribution.
  2. [Section 4, Table 2] The experimental evaluation cannot support the advertised memory reduction because it never enters the regime in which the O(t) versus O(0.1t) distinction would matter. Table 2 fixes the total exemplar budget (#Total Exemplars 200/2000), and the text explicitly states that 'all the strategies always follow the fixed exemplar memory, not growing exemplar memory'. Under this protocol, both AHR and the exemplar-replay baselines have O(1) memory with respect to the number of tasks t. To substantiate the asymptotic claim, the authors would need to vary t while holding the number of exemplars per class (or per-task memory) constant and report memory usage in bytes as t grows. No such experiment is provided.
  3. [Appendix A.4, Section 4] The 'derivation' of O(0.1t) in Appendix A.4 simply asserts that the encoder compresses the input '10 times at the very least in our experiments' and then concludes the memory complexity becomes O(0.1t). This is not a derivation; it is a constant-factor observation with no asymptotic content. Additionally, the claim of state-of-the-art performance is not fully supported because the baseline set omits several recent strong CIL methods (e.g., FeCAM, RanPAC, or other compressed-replay approaches from 2023 onward). The paper's results demonstrate competitiveness with the included baselines, but the 'SOTA' claim should be tempered or the comparison set updated.
minor comments (5)
  1. [Table 1] The table uses the nonstandard notation O(cte) for constant memory; this should be written O(1) to avoid confusion.
  2. [Equation (3)] The Lagrangian in Eq. (3) contains a garbled summation index 'k=i,j'; the double sum over tasks and classes should be written with explicit indices (e.g., sum over i and j).
  3. [Algorithm 4] Line 3 of Algorithm 4 has a typo: 'R ANK' should be 'RANK'. The formula for ε is also ambiguous: it should be the total memory divided by the current number of classes, and the notation should be clarified.
  4. [Algorithm 3] The loss expression on lines 8–10 is not a complete equation; it mixes the HAE loss and distillation losses without clear notation. Please present the full objective function.
  5. [General presentation] The paper states that the source code is included in the supplementary material, but no code was available in the reviewed version; if code is truly provided, the authors should ensure it is accessible to reviewers and readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AHR is evaluated against external benchmarks and the only self-citation is background; the O(0.1t) concern is a complexity-notation error, not a circular derivation.

full rationale

The paper's central derivation chain is self-contained. AHR's latent-replay approach is implemented and measured against published baselines on standard benchmarks, and the HAE loss, the RFA-based centroid placement, and the herding-based memory selection are all specified independently of the reported test accuracies; no hyperparameter is fit to the final accuracies and then renamed as a prediction. The only self-citation, Nori et al. (2025), appears in the literature review as one of several compressed-replay works and is not load-bearing for the method's correctness. The advertised O(0.1t) memory complexity is not a circular step: it is a compression-ratio claim that is conceptually misstated, since O(0.1t) equals O(t) in Landau notation, and the experiments fix total memory so both AHR and exemplar baselines operate at O(1) memory in t. That is a correctness and asymptotic-analysis issue, not a case of the derivation reducing to its own inputs. The empirical finding that more lossy latent exemplars can outperform fewer raw exemplars is plausible and consistent with prior compressed-replay work, but it is presented as an experimental result rather than as a theorem forced by definition. Therefore the paper exhibits no circularity under the stated criteria.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central empirical method relies on an autoencoder losing some fidelity, on a physics-inspired heuristic for arranging class centroids, and on a nonstandard use of big-O notation for the headline complexity claim. No external formal proof or released artifact is provided.

free parameters (6)
  • λ (latent clustering weight)
    Weights the latent-space centroid loss in Eq. 1; no value is reported in the main text or Table 4.
  • ζ (repulsive constant)
    Scales the repulsive force in Algorithm 2; value not reported.
  • m (particle mass)
    Mass in the velocity update in Algorithm 2; value not reported.
  • Δt (time step)
    Time step in the CCE placement simulation; value not reported.
  • τ (simulation duration)
    Number of simulation steps in Algorithm 2; value not reported.
  • Latent dimension = 20 (MNIST), 307 (SVHN/CIFAR), 2117 (miniImageNet)
    Chosen to give about 10x compression; a per-benchmark design choice.
assumptions (4)
  • domain assumption The autoencoder decoder can memorize training data well enough that decoded exemplars are nearly as effective as perfect exemplars.
    Section 2 states the decoder is 'designed to memorize the training data as opposed to being designed to generalize'; AHR relies on this to avoid hazy pseudo-data.
  • domain assumption Charged particle system energy minimization and repulsive force equations from Nazmitdinov et al. (2017) are applicable to laying out class centroids in the latent space.
    Section 2 introduces CPSEM/RFA as the mechanism for incremental centroid embedding; no justification is given for why this physics model is suitable for latent representations.
  • ad hoc to paper Big-O notation can express constant-factor memory reductions, i.e., O(0.1t) is a smaller class than O(t).
    Appendix A.4 concludes memory complexity becomes O(0.1t); under standard definitions O(0.1t) = O(t), so this premise is false.
  • domain assumption Representing memory as growing with the number of tasks (dozens of exemplars per class) is the right model despite the experiments using a fixed total memory budget.
    Appendix A.3 and Algorithm 4 conflict with Table 2: the complexity model grows memory with t, while experiments fix total memory (#Total Exemplars 200/2000).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Autoencoder-Based Hybrid Replay for Class-Incremental Learning." pith.science (2026). https://pith.science/paper/OOXARSFZ

@misc{pith2026250505926,
  author       = {Pith},
  title        = {Pith review of: Autoencoder-Based Hybrid Replay for Class-Incremental Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOXARSFZ}},
  note         = {Machine review of arXiv:2505.05926}
}
abstract

In class-incremental learning (CIL), effective incremental learning strategies are essential to mitigate task confusion and catastrophic forgetting, especially as the number of tasks $t$ increases. Current exemplar replay strategies impose $\mathcal{O}(t)$ memory/compute complexities. We propose an autoencoder-based hybrid replay (AHR) strategy that leverages our new hybrid autoencoder (HAE) to function as a compressor to alleviate the requirement for large memory, achieving $\mathcal{O}(0.1 t)$ at the worst case with the computing complexity of $\mathcal{O}(t)$ while accomplishing state-of-the-art performance. The decoder later recovers the exemplar data stored in the latent space, rather than in raw format. Additionally, HAE is designed for both discriminative and generative modeling, enabling classification and replay capabilities, respectively. HAE adopts the charged particle system energy minimization equations and repulsive force algorithm for the incremental embedding and distribution of new class centroids in its latent space. Our results demonstrate that AHR consistently outperforms recent baselines across multiple benchmarks while operating with the same memory/compute budgets. The source code is included in the supplementary material and will be open-sourced upon publication.

Figures

Figures reproduced from arXiv: 2505.05926 by the authors.

Figure 1
Figure 1. (a) Usage of RFA for the latent space. (b) Adop [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Task-based and task-free. plar replay and generative replay. Specifically, AHR does not store the exemplars in the input space like exemplar replay which would require a large memory O(t); it rather stores the data samples in the latent space after they are encoded O(0.1t). Hence, AHR has characteristics that leverage the advantages of both exemplar and generative replay. AHR can decode data samples when they are ne… view at source ↗
Figure 3
Figure 3. The impact of the memory size (first row). The required resources to achieve a target performance (second row). [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Images produced by the decoder at different tasks (for the decoder size of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performances for various decoder/memory sizes. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 15 canonical work pages

  1. [2]

    K., Torr, P

    Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajan- than, T., Dokania, P. K., Torr, P. H., and Ranzato, M. A. On tiny episodic memories in continual learning. arXiv:1902.10486,

    arXiv 1902

  2. [4]

    Extended Literature Review Task-based or task-free

    C. Extended Literature Review Task-based or task-free. Incremental learning literature features various learning scenarios that present their own unique challenges, and accordingly, diverse strategies have been developed (Parisi et al., 2019; De Lange et al., 2021; Yu et al., 2024; Elsayed & Mahmood, 2024). In the first place, there are two learning scena...

    work page 2019

  3. [6]

    Less-forgetting learning in deep neural networks

    Jung, H., Ju, J., Jung, M., and Kim, J. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122,

    arXiv

  4. [10]

    Energy-Based Models for Continual Learning

    Li, S., Du, Y ., Ven, G. M., and Mordatch, I. Energy-based models for continual learning. arXiv:2011.12216,

    work page Pith review arXiv 2011

  5. [11]

    D., and van de Weijer, J

    10 Autoencoder-Based Hybrid Replay for Class-Incremental Learning Masana, M., Liu, X., Twardowski, B., Menta, M., Bag- danov, A. D., and van de Weijer, J. Class-incremental learning: Survey and performance evaluation on image classification. arXiv:2010.15277,

    arXiv 2010

  6. [13]

    K., Kim, I.-M., and Wang, G

    Nori, M. K., Kim, I.-M., and Wang, G. Federated class- incremental learning: A hybrid approach using latent exemplars and data-free techniques to address local and global forgetting. arXiv:2501.15356,

    arXiv

  7. [14]

    Latent replay for real-time continual learning

    Pellegrini, L., Graffieti, G., Lomonaco, V ., and Maltoni, D. Latent replay for real-time continual learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10203–10209. IEEE,

    work page 2020

  8. [15]

    Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001–2010,

    work page 2001

Show all 28 references
  1. [16]

    In- cremental learning of structured memory via closed-loop transcription

    Tong, S., Dai, X., Wu, Z., Li, M., Yi, B., and Ma, Y . In- cremental learning of structured memory via closed-loop transcription. arXiv:2202.05411,

    arXiv

  2. [17]

    S., and King, I

    11 Autoencoder-Based Hybrid Replay for Class-Incremental Learning Yu, D., Zhang, X., Chen, Y ., Liu, A., Zhang, Y ., Yu, P. S., and King, I. Recent advances of multimodal continual learning: A comprehensive survey. arXiv:2410.05352,

  3. [18]

    Zaj ˛ ac, M., Tuytelaars, T., and van de Ven, G. M. Prediction error-based classification for class-incremental learning. arXiv preprint arXiv:2305.18806,

    arXiv

  4. [19]

    A model or 603 exemplars: Towards memory-efficient class- incremental learning

    Zhou, D.-W., Wang, Q.-W., Ye, H.-J., and Zhan, D.-C. A model or 603 exemplars: Towards memory-efficient class- incremental learning. arXiv:2205.13218,

    arXiv

  5. [20]

    uses a memory denoted by MEM that stores dozens of exemplars per class so that each time a new task arrives a representative minibatch of data consisting of all previous tasks is fed into the discriminative model denoted by DIS(). In this strategy, both the memory MEM has to g...

    2020

  6. [22]

    and task-free (Ven et al., 2021; Aljundi et al., 2019b). In task-based, the model receives the data in the form of tasks: The task-based scenario is divided into two popular scenarios: task-incremental learning (TIL) and class-incremental learning (CIL). Whereas in TIL the mod...

    2021

  7. [23]

    Conversely, in the online scenario, the model visits the data only once as they arrive and cannot iterate on them

    or online (Zaj ˛ ac et al., 2023; Wang et al., 2024; Zhuang et al., 2024b; Raghavan et al., 2024), where in the offline learning scenario, the data of each task can be fed to the model multiple times before moving on to the next task. Conversely, in the online scenario, the mo...

    2023

  8. [24]

    When learning new tasks, the importance coefficients help in minimizing weight drift

    mitigates weight drift of the parameters optimized for the previous tasks by assigning an importance coefficient for each parameter in the network (assuming the independence of weights) after learning each task (Kirkpatrick et al., 2017; Zenke et al., 2017; Nguyen et al., 2018...

    2017

  9. [25]

    maintains and updates per-parameter importance measures in an online manner. Data regularization is the second regularization strategy, aimed at preventing activation drift through knowledge distillation (Buciluundefined et al., 2006; Hinton et al., 2015), originally designed ...

    2006

  10. [26]

    LT often improves the performance when added on top of other strategies (Wu et al., 2019)

    is a rehearsal-free bias-correcting algorithm that prevents negative bias on past tasks. LT often improves the performance when added on top of other strategies (Wu et al., 2019). However,because AHR separates representation learning from classification similar to (Rebuffi et ...

    2019

  11. [28]

    (popular in data mining (Kim et al., 2011; Pang et al., 2005)) is thought to be another form of generative classifier (Ven et al., 2021); however, it prevents representation learning. 15

    2011

  12. [2011]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv:1412.6980,

    arXiv

  13. [2013]

    SLDA (Hayes & Kanan,

    and importance sampling (Burda et al., 2016). SLDA (Hayes & Kanan,

    2016

  14. [2014]

    Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv:1312.6114,

    arXiv

  15. [2016]

    Distilling the knowledge in a neural network

    Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv

  16. [2017]

    Y ., et al

    Netzer, Y ., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y ., et al. Reading digits in natural images with unsu- pervised feature learning. In Proc. Advances in Neural Information Processing Systems, volume 2011, pp. 7,

    2011

  17. [2018]

    Online continual learning with maximally interfered retrieval

    Aljundi, R., Caccia, L., Belilovsky, E., Caccia, M., Lin, M., Charlin, L., and Tuytelaars, T. Online continual learning with maximally interfered retrieval. arXiv:1908.04742, 2019a. Aljundi, R., Kelchtermans, K., and Tuytelaars, T. Task-free continual learning. In Proc. of the...

    1908 arXiv

  18. [2019]

    and Mahmood, A

    Elsayed, M. and Mahmood, A. Addressing catastrophic forgetting and loss of plasticity in neural networks. In Proc. 12th International Conference on Learning Repre- sentations (ICLR, 2024),

    2024

  19. [2020]

    and Kanan, C

    Kemker, R. and Kanan, C. Fearnet: Brain-inspired model for incremental learning. arXiv:1711.10563,

    arXiv

  20. [2024]

    and Gal, Y

    Farquhar, S. and Gal, Y . Towards robust evaluations of continual learning. arXiv:1805.09733, 2018a. 9 Autoencoder-Based Hybrid Replay for Class-Incremental Learning Farquhar, S. and Gal, Y . A unifying bayesian view of contin- ual learning. In Proc. International Conference o...

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.