REVIEW 3 major objections 5 minor 21 references
Class Incremental Learning for Algorithm Selection
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Storing 100 exemplars from earlier classes lets a bin-packing algorithm selector learn new solver classes with only about a 6% accuracy loss compared with a model trained on all data at once.
desk verdict A solid first benchmark of CIL for algorithm selection, with a clear rehearsal win, but the optimization claim rests on an unmeasured solver-performance metric. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the rehearsal buffer: a fixed set of 100 stored instances from earlier classes that the model re-trains on when learning a new task. In Experience Replay this buffer is directly mixed into the gradient updates for the new task, so the decision boundaries separating old solver classes are re-exercised instead of overwritten. The comparison is built from six two-task streams formed by pairing four deterministic solvers, with 30 runs per configuration, and the Falkenauer metric defines which heuristic wins an instance and therefore becomes the target class label. The same architecture, epochs, and training sizes are used across all eight methods, making the rehearsal buffer the main controlled difference.
What would settle it
Re-run the same six stream orders on the bin-packing dataset and, for the Replay, GEM, cumulative, and oracle models, record the Falkenauer packing score of the heuristic actually selected on each held-out instance; if the Replay model's mean selected-solver packing score stays within a small margin of the cumulative model despite the 6% accuracy gap, the viability claim is confirmed on the metric that defines the wins, while a large packing gap would collapse the claim.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that class-incremental learning for algorithm selection is feasible provided the classifier stores a small buffer of old instances. In the bin-packing benchmark, Experience Replay with 100 exemplars achieves 82.3% mean accuracy over four solver classes at the end of the stream, compared with 84.8% for a cumulative model that keeps all data and 88.3% for an oracle that sees all classes at once. First-task accuracy after learning the second task drops to 0.739 for Replay, and the overall gap to the cumulative benchmark is about 6%. By contrast, EWC, MAS, SI, LwF, and Feature Replay retain almost nothing from the first task after the second task is learned, with first-task accuracy below 0.05; GEM and A-GEM fall between, at 0.683 and 0.442. The paper therefore claims that the decisive factor for streaming algorithm selection is not the continual-learning mechanism in general but whether the method stores and replays exemplars.
Load-bearing premise
The load-bearing premise is that algorithm-selection quality is faithfully measured by the classifier's accuracy at naming the winning heuristic; if a wrong label still selects a solver with almost the same packing performance, the reported forgetting overstates the practical loss, and if it selects a much worse solver, it understates it.
Editorial extensions
If this is right
- If rehearsal-based CIL holds, an algorithm selector in a streaming environment can be updated with a small buffer, avoiding the storage and privacy costs of keeping all historical instances.
- Parameter regularization and knowledge distillation approaches, as configured here, are not suitable for streaming algorithm selection because they catastrophically forget early solver classes.
- The gap to a cumulative model is small enough that retaining all data in a stream may not be worth the extra cost, at least for this bin-packing setting.
- The order in which solver classes appear matters: some pairs, such as FF and WF, show larger forgetting, so stream scheduling could meaningfully affect final selector quality.
- The results are demonstrated on one dataset with four deterministic heuristics, so the viability claim remains to be tested on other portfolios and instance distributions.
Reading between the lines
- The paper evaluates quality through label accuracy, but the true optimisation cost is the packing quality of the heuristic actually selected; reporting the Falkenauer score of the selected solver, not just the class label, would show whether the 6% accuracy gap translates into a meaningful packing-quality gap.
- Feature Replay's failure while exemplar replay succeeds suggests that for tabular instance data, raw stored examples carry information that feature-level consolidation cannot recover; this could be tested on other optimisation datasets with similar tabular features.
- Because the buffer was fixed at 100 exemplars, the limits of rehearsal remain open; comparing buffer sizes from one exemplar per class to the full first-task dataset would reveal where the advantage begins and whether a single prototype per class is enough.
- The class-pair effects suggest that individual solver pairs have very different confusability, pointing toward a practical extension where buffer allocation and stream ordering are tuned to protect the most easily forgotten pairs.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies Class Incremental Learning (CIL) in the context of per-instance algorithm selection for online bin-packing. Using a dataset with four deterministic heuristics (FF, BF, NF, WF), the authors construct six two-task streams, train eight CIL methods (EWC, MAS, SI, LwF, GEM, AGEM, Feature Replay, Experience Replay) over five training-set sizes, and aggregate across 30 runs. The evaluation measures top-1 classification accuracy on the winning-solver label. The central empirical finding is that Experience Replay substantially outperforms the other CIL methods, reaching 82.3% final four-class accuracy, close to a Cumulative baseline of 84.8%, while parameter regularization, knowledge distillation, and Feature Replay suffer near-total forgetting on D1. The paper concludes that rehearsal-based methods are viable for streaming optimisation scenarios.
Significance. If the result holds, this is a useful first benchmark connecting the continual-learning literature to the algorithm-selection community. The paper uses a publicly available dataset, provides code and data via Zenodo, and evaluates a broad set of standard CIL baselines, which is a strength for reproducibility and comparability. The study is small in scope (one dataset, two tasks per stream, four solvers), but it establishes an empirical baseline that could inform future work. The main positive is the clear, aggregated comparison in Table 1; the main weakness is that success is measured solely by label accuracy, not by the quality of the solver actually selected, which is the quantity that matters for the paper's viability claim.
major comments (3)
- [Section 3.2.1 / Sections 4.1 and 5] The evaluation metric is top-1 accuracy of predicting the winning solver, but the paper's central claim is that CIL methods are viable for streaming optimisation. Since the win labels are defined by the Falkenauer metric (Section 3.1), the optimization-quality question is whether the selected solver produces good packing, not merely whether it is the single best heuristic. Heuristic performances are likely correlated: on some instances First-Fit and Best-Fit may be nearly interchangeable, while on others Worst-Fit may be much worse. Consequently, label accuracy can both understate and overstate the realized optimization cost. For example, Table 1 reports Replay's D1 accuracy dropping to 0.739 after D2, and Table 2 shows per-class D1 accuracy falling to 0.61 for FF in the [FF,WF] stream; without knowing which solver is chosen instead and the resulting Falkenauer value, this forgetting cannot be interpreted as a corresponding loss in packing quality. Please add an evaluation that, for each test instance, applies the solver selected by the learned model and reports the mean Falkenauer score (or regret relative to the oracle and the best available solver), alongside the accuracy results. This is load-bearing for the conclusion that these methods are viable for streaming optimisation.
- [Abstract and Section 4.1] The abstract states that 'the loss is small at around 7%', but this does not describe the recommended method. Table 1 shows Experience Replay's D1 accuracy dropping from 0.928 to 0.739, a loss of 18.9 percentage points; the roughly 7% loss is the cumulative baseline's D1 forgetting (0.907 to 0.834). The final four-class gap between Replay and Cumulative is 2.5 points (0.823 vs 0.848), while Section 5's 'loss compared to this benchmark is 6%' appears to refer to the Oracle (0.883). Please make these comparisons explicit and internally consistent, attributing each loss to the correct reference model, and revise the abstract's 7% claim if it refers to a benchmark rather than to the CIL method.
- [Section 2.1 and Section 4.1 / Table 1] The claim that 'rehearsal-based methods significantly outperform other CIL methods' is broader than the evidence. Feature Replay is listed as a data-replay method in Section 2.1, yet Table 1 shows it achieves D1 accuracy of only 0.043 after D2, comparable to the parameter-regularization and knowledge-distillation methods. Either restrict the conclusion to methods that replay stored raw exemplars (Experience Replay and arguably GEM's episodic memory), or provide an explanation for why feature replay fails despite being categorized as rehearsal-based. In addition, 'significantly' is used without inferential statistics; please report confidence intervals or paired statistical tests to support the comparative claim, especially for the Replay versus GEM comparison (0.823 vs 0.798 with overlapping standard deviations).
minor comments (5)
- [General] There are frequent typos and misspellings, including 'per per-instance' in the abstract, 'Falkanauer' for Falkenauer, 'signifcant', 'acccross', 'magnitdue', 'classifer', 'A verage', 'distiniguish', and 'taxonomoy'.
- [Section 2.1] The description of Feature Replay as a data-replay method that visits exemplars from previous tasks appears inconsistent with reference [14], which is titled 'Elastic feature consolidation for cold start exemplar-free incremental learning'. Please clarify the actual mechanism and how it fits the stated taxonomy.
- [Figure 1] The caption says 'line length indicates the magnitude of the loss', but the loss is never defined. Please state explicitly that the loss is the difference in accuracy between the final Replay model and the Cumulative model, or define it in the text.
- [Table 3] The caption says 'Difference is expressed as D1 accuracy minus D2 accuracy', but the entries are negative in many cases. Please clarify the sign convention and add a sentence interpreting negative versus positive values.
- [Section 3.2] The architecture, hyperparameters, and number of training epochs are relegated to the supplementary information; given that the study's reproducibility depends on these choices, please include a concise table in the main text or ensure the Zenodo link resolves to a complete, versioned record.
Circularity Check
No significant circularity: the benchmark compares fixed external CIL methods against fixed baselines, and the reported accuracies are direct empirical measurements rather than fitted predictions.
full rationale
The paper's derivation chain is an empirical benchmark: it defines class labels via the Falkenauer metric for four fixed solvers, trains eight CIL methods on a fixed bin-packing dataset, and reports classifier accuracy on those labels. No quantity is fitted to a subset of the data and then relabeled as a prediction; the Oracle, Cumulative, and per-method results are all measured accuracies under the stated protocol. The only self-citations are the dataset source [1] and the supplementary material [15]; the dataset is a fixed benchmark, not a parameter fitted to the conclusions, and the supplementary material is not used to justify the outcome. The central comparison—Replay reaching 82.3% overall accuracy versus Cumulative's 84%—is a direct empirical observation, not a quantity that reduces to its own inputs by construction. The concern that accuracy on winning-solver labels is not validated against realized Falkenauer packing performance is a substantive validity limitation, but it is a correctness risk rather than a circularity in the derivation chain. Accordingly, the paper exhibits no significant circularity.
Assumptions & free parameters
free parameters (1)
- Exemplar buffer size for rehearsal methods =
100
assumptions (4)
- domain assumption Classification accuracy on the winning solver label is an adequate proxy for algorithm selection quality.
- domain assumption The class-incremental scenario with B=2 tasks and 2 labels per task is representative of streaming optimisation settings.
- domain assumption The Avalanche library implementations of the eight CIL methods are faithful to their original papers.
- domain assumption The bin-packing dataset and the four deterministic heuristics define a meaningful algorithm selection problem.
Cite this review
Pith. "Pith review of Class Incremental Learning for Algorithm Selection." pith.science (2026). https://pith.science/paper/XK6XXP3Y
@misc{pith2026250601545,
author = {Pith},
title = {Pith review of: Class Incremental Learning for Algorithm Selection},
year = {2026},
howpublished = {\url{https://pith.science/paper/XK6XXP3Y}},
note = {Machine review of arXiv:2506.01545}
}
read the original abstract
Algorithm selection is commonly used to predict the best solver from a portfolio per per-instance. In many real scenarios, instances arrive in a stream: new instances become available over time, while the number of class labels can also grow as new data distributions arrive downstream. As a result, the classification model needs to be periodically updated to reflect additional solvers without catastrophic forgetting of past data. In machine-learning (ML), this is referred to as Class Incremental Learning (CIL). While commonly addressed in ML settings, its relevance to algorithm-selection in optimisation has not been previously studied. Using a bin-packing dataset, we benchmark 8 continual learning methods with respect to their ability to withstand catastrophic forgetting. We find that rehearsal-based methods significantly outperform other CIL methods. While there is evidence of forgetting, the loss is small at around 7%. Hence, these methods appear to be a viable approach to continual learning in streaming optimisation scenarios.
Figures
Reference graph
Works this paper leans on
-
[1]
Mohamad Alissa, Kevin Sim, and Emma Hart. 2019. Algorithm selection us- ing deep learning without feature extraction. In Proceedings of the Genetic and Evolutionary Computation Conference. 198–206
work page 2019
-
[2]
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2017. Memory Aware Synapses: Learning what (not) to forget. (11 2017). http://arxiv.org/abs/1711.09601
arXiv 2017
-
[3]
A Bifet, R Gavalda, G Holmes, and B Pfahringer. 2018. Machine learningfor data streams: with practical examples in MOA
work page 2018
-
[4]
Antonio Carta, Lorenzo Pellegrini, Andrea Cossu, Scuola Normale Superiore, Hamed Hemati, Vincenzo Lomonaco, and Joaquin Vanschoren. 2023. A valanche: A PyTorch Library for Deep Continual Learning . Technical Report. 1–6 pages. https://avalanche.continualai.org
work page 2023
-
[5]
Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed El- hoseiny. 2018. Efficient Lifelong Learning with A-GEM. (12 2018). http: //arxiv.org/abs/1812.00420
arXiv 2018
-
[6]
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence 44, 7 (2021), 3366–3385
work page 2021
-
[7]
E. Falkenauer and A. Delchambre. 1992. A genetic algorithm for bin packing and line balancing. In Proceedings 1992 IEEE International Conference on Robotics and Automation. 1186–1192 vol.2. https://doi.org/10.1109/ROBOT.1992.220088
-
[8]
Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet
Show all 21 references
-
[9]
Pascal Kerschke, Holger H Hoos, Frank Neumann, and Heike Trautmann. 2019. Automated algorithm selection: Survey and perspectives. Evolutionary computa- tion 27, 1 (2019), 3–45
2019
-
[10]
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forget...
2016 doi
-
[11]
Georg Krempl, Indre Žliobaite, Dariusz Brzeziński, Eyke Hüllermeier, Mark Last, Vincent Lemaire, Tino Noack, Ammar Shaker, Sonja Sievi, Myra Spiliopoulou, et al. 2014. Open challenges for data stream mining research. ACM SIGKDD explorations newsletter 16, 1 (2014), 1–10
2014
-
[12]
Zhizhong Li and Derek Hoiem. 2016. Learning without Forgetting. (6 2016). http://arxiv.org/abs/1606.09282
2016 arXiv
-
[13]
David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient Episodic Memory for Continual Learning. (6 2017). http://arxiv.org/abs/1706.08840
2017 arXiv
-
[14]
Simone Magistri, Tomaso Trinci, Albin Soutif-Cormerais, Joost van de Weijer, and Andrew D Bagdanov. 2024. Elastic feature consolidation for cold start exemplar- free incremental learning. arXiv preprint arXiv:2402.03917 (2024)
2024 arXiv
-
[15]
Mate Botond Nemeth, Emma Hart, Kevin Sim, and Quentin Renau. 2025. Class Incremental Learning for Algorithm Selection. https://doi.org/10.5281/zenodo. 15184096
2025 doi
-
[16]
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural networks 113 (2019), 54–71
2019
-
[17]
Roger Ratcliff. 1990. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review 97, 2 (1990), 285
1990
-
[18]
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. 2022. Three types of incremental learning. Nature Machine Intelligence 4, 12 (2022), 1185–1197
2022
-
[19]
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual Learning Through Synaptic Intelligence. (3 2017). http://arxiv.org/abs/1703.04200
2017 arXiv
-
[20]
Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. 2024. Class-incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
2024
-
[2017]
ACM Com- puting Surveys (CSUR) 50, 2 (2017), 1–36
A survey on ensemble learning for data stream classification. ACM Com- puting Surveys (CSUR) 50, 2 (2017), 1–36
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.