Pith. sign in

REVIEW 3 major objections 5 minor 21 references

Class Incremental Learning for Algorithm Selection

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Storing 100 exemplars from earlier classes lets a bin-packing algorithm selector learn new solver classes with only about a 6% accuracy loss compared with a model trained on all data at once.

desk verdict A solid first benchmark of CIL for algorithm selection, with a clear rehearsal win, but the optimization claim rests on an unmeasured solver-performance metric. read the letter →

arxiv 2506.01545 v1 pith:XK6XXP3Y submitted 2025-06-02 cs.LG

classification cs.LG
keywords ClassIncrementalLearningAlgorithmSelectionCatastrophicForgettingExperienceReplayBinPackingStreamingDataSolverPortfolioContinual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a classifier that selects heuristics for bin-packing can be updated as new solver classes arrive without losing old knowledge. It benchmarks eight continual-learning methods on streams of two tasks, each introducing two new solver labels. It claims that rehearsal-based replay, with only 100 stored exemplars, nearly matches a model trained on all data at once: 82.3% accuracy over four classes versus 84.8% for a cumulative model and 88.3% for an oracle. Other method families, including parameter regularization, knowledge distillation, and feature replay, suffer near-total loss of the first task, with first-task accuracy dropping below 0.05. The conclusion is that rehearsal-based class-incremental learning is a viable way to keep selectors current in streaming optimisation.

What carries the argument

The central mechanism is the rehearsal buffer: a fixed set of 100 stored instances from earlier classes that the model re-trains on when learning a new task. In Experience Replay this buffer is directly mixed into the gradient updates for the new task, so the decision boundaries separating old solver classes are re-exercised instead of overwritten. The comparison is built from six two-task streams formed by pairing four deterministic solvers, with 30 runs per configuration, and the Falkenauer metric defines which heuristic wins an instance and therefore becomes the target class label. The same architecture, epochs, and training sizes are used across all eight methods, making the rehearsal buffer the main controlled difference.

What would settle it

Re-run the same six stream orders on the bin-packing dataset and, for the Replay, GEM, cumulative, and oracle models, record the Falkenauer packing score of the heuristic actually selected on each held-out instance; if the Replay model's mean selected-solver packing score stays within a small margin of the cumulative model despite the 6% accuracy gap, the viability claim is confirmed on the metric that defines the wins, while a large packing gap would collapse the claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that class-incremental learning for algorithm selection is feasible provided the classifier stores a small buffer of old instances. In the bin-packing benchmark, Experience Replay with 100 exemplars achieves 82.3% mean accuracy over four solver classes at the end of the stream, compared with 84.8% for a cumulative model that keeps all data and 88.3% for an oracle that sees all classes at once. First-task accuracy after learning the second task drops to 0.739 for Replay, and the overall gap to the cumulative benchmark is about 6%. By contrast, EWC, MAS, SI, LwF, and Feature Replay retain almost nothing from the first task after the second task is learned, with first-task accuracy below 0.05; GEM and A-GEM fall between, at 0.683 and 0.442. The paper therefore claims that the decisive factor for streaming algorithm selection is not the continual-learning mechanism in general but whether the method stores and replays exemplars.

Load-bearing premise

The load-bearing premise is that algorithm-selection quality is faithfully measured by the classifier's accuracy at naming the winning heuristic; if a wrong label still selects a solver with almost the same packing performance, the reported forgetting overstates the practical loss, and if it selects a much worse solver, it understates it.

Editorial extensions

If this is right

  • If rehearsal-based CIL holds, an algorithm selector in a streaming environment can be updated with a small buffer, avoiding the storage and privacy costs of keeping all historical instances.
  • Parameter regularization and knowledge distillation approaches, as configured here, are not suitable for streaming algorithm selection because they catastrophically forget early solver classes.
  • The gap to a cumulative model is small enough that retaining all data in a stream may not be worth the extra cost, at least for this bin-packing setting.
  • The order in which solver classes appear matters: some pairs, such as FF and WF, show larger forgetting, so stream scheduling could meaningfully affect final selector quality.
  • The results are demonstrated on one dataset with four deterministic heuristics, so the viability claim remains to be tested on other portfolios and instance distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates quality through label accuracy, but the true optimisation cost is the packing quality of the heuristic actually selected; reporting the Falkenauer score of the selected solver, not just the class label, would show whether the 6% accuracy gap translates into a meaningful packing-quality gap.
  • Feature Replay's failure while exemplar replay succeeds suggests that for tabular instance data, raw stored examples carry information that feature-level consolidation cannot recover; this could be tested on other optimisation datasets with similar tabular features.
  • Because the buffer was fixed at 100 exemplars, the limits of rehearsal remain open; comparing buffer sizes from one exemplar per class to the full first-task dataset would reveal where the advantage begins and whether a single prototype per class is enough.
  • The class-pair effects suggest that individual solver pairs have very different confusability, pointing toward a practical extension where buffer allocation and stream ordering are tuned to protect the most easily forgotten pairs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies Class Incremental Learning (CIL) in the context of per-instance algorithm selection for online bin-packing. Using a dataset with four deterministic heuristics (FF, BF, NF, WF), the authors construct six two-task streams, train eight CIL methods (EWC, MAS, SI, LwF, GEM, AGEM, Feature Replay, Experience Replay) over five training-set sizes, and aggregate across 30 runs. The evaluation measures top-1 classification accuracy on the winning-solver label. The central empirical finding is that Experience Replay substantially outperforms the other CIL methods, reaching 82.3% final four-class accuracy, close to a Cumulative baseline of 84.8%, while parameter regularization, knowledge distillation, and Feature Replay suffer near-total forgetting on D1. The paper concludes that rehearsal-based methods are viable for streaming optimisation scenarios.

Significance. If the result holds, this is a useful first benchmark connecting the continual-learning literature to the algorithm-selection community. The paper uses a publicly available dataset, provides code and data via Zenodo, and evaluates a broad set of standard CIL baselines, which is a strength for reproducibility and comparability. The study is small in scope (one dataset, two tasks per stream, four solvers), but it establishes an empirical baseline that could inform future work. The main positive is the clear, aggregated comparison in Table 1; the main weakness is that success is measured solely by label accuracy, not by the quality of the solver actually selected, which is the quantity that matters for the paper's viability claim.

major comments (3)
  1. [Section 3.2.1 / Sections 4.1 and 5] The evaluation metric is top-1 accuracy of predicting the winning solver, but the paper's central claim is that CIL methods are viable for streaming optimisation. Since the win labels are defined by the Falkenauer metric (Section 3.1), the optimization-quality question is whether the selected solver produces good packing, not merely whether it is the single best heuristic. Heuristic performances are likely correlated: on some instances First-Fit and Best-Fit may be nearly interchangeable, while on others Worst-Fit may be much worse. Consequently, label accuracy can both understate and overstate the realized optimization cost. For example, Table 1 reports Replay's D1 accuracy dropping to 0.739 after D2, and Table 2 shows per-class D1 accuracy falling to 0.61 for FF in the [FF,WF] stream; without knowing which solver is chosen instead and the resulting Falkenauer value, this forgetting cannot be interpreted as a corresponding loss in packing quality. Please add an evaluation that, for each test instance, applies the solver selected by the learned model and reports the mean Falkenauer score (or regret relative to the oracle and the best available solver), alongside the accuracy results. This is load-bearing for the conclusion that these methods are viable for streaming optimisation.
  2. [Abstract and Section 4.1] The abstract states that 'the loss is small at around 7%', but this does not describe the recommended method. Table 1 shows Experience Replay's D1 accuracy dropping from 0.928 to 0.739, a loss of 18.9 percentage points; the roughly 7% loss is the cumulative baseline's D1 forgetting (0.907 to 0.834). The final four-class gap between Replay and Cumulative is 2.5 points (0.823 vs 0.848), while Section 5's 'loss compared to this benchmark is 6%' appears to refer to the Oracle (0.883). Please make these comparisons explicit and internally consistent, attributing each loss to the correct reference model, and revise the abstract's 7% claim if it refers to a benchmark rather than to the CIL method.
  3. [Section 2.1 and Section 4.1 / Table 1] The claim that 'rehearsal-based methods significantly outperform other CIL methods' is broader than the evidence. Feature Replay is listed as a data-replay method in Section 2.1, yet Table 1 shows it achieves D1 accuracy of only 0.043 after D2, comparable to the parameter-regularization and knowledge-distillation methods. Either restrict the conclusion to methods that replay stored raw exemplars (Experience Replay and arguably GEM's episodic memory), or provide an explanation for why feature replay fails despite being categorized as rehearsal-based. In addition, 'significantly' is used without inferential statistics; please report confidence intervals or paired statistical tests to support the comparative claim, especially for the Replay versus GEM comparison (0.823 vs 0.798 with overlapping standard deviations).
minor comments (5)
  1. [General] There are frequent typos and misspellings, including 'per per-instance' in the abstract, 'Falkanauer' for Falkenauer, 'signifcant', 'acccross', 'magnitdue', 'classifer', 'A verage', 'distiniguish', and 'taxonomoy'.
  2. [Section 2.1] The description of Feature Replay as a data-replay method that visits exemplars from previous tasks appears inconsistent with reference [14], which is titled 'Elastic feature consolidation for cold start exemplar-free incremental learning'. Please clarify the actual mechanism and how it fits the stated taxonomy.
  3. [Figure 1] The caption says 'line length indicates the magnitude of the loss', but the loss is never defined. Please state explicitly that the loss is the difference in accuracy between the final Replay model and the Cumulative model, or define it in the text.
  4. [Table 3] The caption says 'Difference is expressed as D1 accuracy minus D2 accuracy', but the entries are negative in many cases. Please clarify the sign convention and add a sentence interpreting negative versus positive values.
  5. [Section 3.2] The architecture, hyperparameters, and number of training epochs are relegated to the supplementary information; given that the study's reproducibility depends on these choices, please include a concise table in the main text or ensure the Zenodo link resolves to a complete, versioned record.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark compares fixed external CIL methods against fixed baselines, and the reported accuracies are direct empirical measurements rather than fitted predictions.

full rationale

The paper's derivation chain is an empirical benchmark: it defines class labels via the Falkenauer metric for four fixed solvers, trains eight CIL methods on a fixed bin-packing dataset, and reports classifier accuracy on those labels. No quantity is fitted to a subset of the data and then relabeled as a prediction; the Oracle, Cumulative, and per-method results are all measured accuracies under the stated protocol. The only self-citations are the dataset source [1] and the supplementary material [15]; the dataset is a fixed benchmark, not a parameter fitted to the conclusions, and the supplementary material is not used to justify the outcome. The central comparison—Replay reaching 82.3% overall accuracy versus Cumulative's 84%—is a direct empirical observation, not a quantity that reduces to its own inputs by construction. The concern that accuracy on winning-solver labels is not validated against realized Falkenauer packing performance is a substantive validity limitation, but it is a correctness risk rather than a circularity in the derivation chain. Accordingly, the paper exhibits no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical result rests on the training protocol, the chosen buffer size, and the accuracy proxy. No new entities are introduced. Method-specific hyperparameters are deferred to the supplementary repository and cannot be audited from the manuscript.

free parameters (1)
  • Exemplar buffer size for rehearsal methods = 100
    Chosen by hand in Section 3.2 ('We set this number to 100 as this corresponds to the minimum number of instances associated with a class'). The comparison could depend on this value, but only one buffer size is tested.
assumptions (4)
  • domain assumption Classification accuracy on the winning solver label is an adequate proxy for algorithm selection quality.
    All conclusions about the viability of CIL for optimization are drawn from classifier accuracy in Section 3.2.1 and Table 1, not from the Falkenauer metric that determines the win labels. Misclassification may still yield a good solver, or a bad one.
  • domain assumption The class-incremental scenario with B=2 tasks and 2 labels per task is representative of streaming optimisation settings.
    The paper fixes B=2 and Y_b=2 in Section 3. It generalizes to 'streaming optimisation scenarios' in the conclusion without testing longer streams.
  • domain assumption The Avalanche library implementations of the eight CIL methods are faithful to their original papers.
    Section 3.2 states all methods are implemented using Avalanche [4]; implementation fidelity is not verified in the main text.
  • domain assumption The bin-packing dataset and the four deterministic heuristics define a meaningful algorithm selection problem.
    The dataset from [1] and the Falkenauer win definition are taken as given in Section 3.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Class Incremental Learning for Algorithm Selection." pith.science (2026). https://pith.science/paper/XK6XXP3Y

@misc{pith2026250601545,
  author       = {Pith},
  title        = {Pith review of: Class Incremental Learning for Algorithm Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XK6XXP3Y}},
  note         = {Machine review of arXiv:2506.01545}
}
read the original abstract

Algorithm selection is commonly used to predict the best solver from a portfolio per per-instance. In many real scenarios, instances arrive in a stream: new instances become available over time, while the number of class labels can also grow as new data distributions arrive downstream. As a result, the classification model needs to be periodically updated to reflect additional solvers without catastrophic forgetting of past data. In machine-learning (ML), this is referred to as Class Incremental Learning (CIL). While commonly addressed in ML settings, its relevance to algorithm-selection in optimisation has not been previously studied. Using a bin-packing dataset, we benchmark 8 continual learning methods with respect to their ability to withstand catastrophic forgetting. We find that rehearsal-based methods significantly outperform other CIL methods. While there is evidence of forgetting, the loss is small at around 7%. Hence, these methods appear to be a viable approach to continual learning in streaming optimisation scenarios.

Figures

Figures reproduced from arXiv: 2506.01545 by the authors.

Figure 1
Figure 1. Comparison of Cumulative and Replay accuracy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Accuracy over all 4 classes separated by the order in which pairs of classes are presented. Results are shown for the cumulative method as context, and the best performing CIL methods Replay and GEM. on D2. We note that after training on D2, all of the CIL methods result in better accuracy with respect to D2 than the Cumulative model (a 9% difference in accuracy between the best CIL model FR and the reference model)… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

  1. [1]

    Mohamad Alissa, Kevin Sim, and Emma Hart. 2019. Algorithm selection us- ing deep learning without feature extraction. In Proceedings of the Genetic and Evolutionary Computation Conference. 198–206

  2. [2]

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2017. Memory Aware Synapses: Learning what (not) to forget. (11 2017). http://arxiv.org/abs/1711.09601

  3. [3]

    A Bifet, R Gavalda, G Holmes, and B Pfahringer. 2018. Machine learningfor data streams: with practical examples in MOA

  4. [4]

    Antonio Carta, Lorenzo Pellegrini, Andrea Cossu, Scuola Normale Superiore, Hamed Hemati, Vincenzo Lomonaco, and Joaquin Vanschoren. 2023. A valanche: A PyTorch Library for Deep Continual Learning . Technical Report. 1–6 pages. https://avalanche.continualai.org

  5. [5]

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed El- hoseiny. 2018. Efficient Lifelong Learning with A-GEM. (12 2018). http: //arxiv.org/abs/1812.00420

  6. [6]

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence 44, 7 (2021), 3366–3385

  7. [7]

    Falkenauer and A

    E. Falkenauer and A. Delchambre. 1992. A genetic algorithm for bin packing and line balancing. In Proceedings 1992 IEEE International Conference on Robotics and Automation. 1186–1192 vol.2. https://doi.org/10.1109/ROBOT.1992.220088

  8. [8]

    Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet

Show all 21 references
  1. [9]

    Pascal Kerschke, Holger H Hoos, Frank Neumann, and Heike Trautmann. 2019. Automated algorithm selection: Survey and perspectives. Evolutionary computa- tion 27, 1 (2019), 3–45

  2. [10]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forget...

  3. [11]

    Georg Krempl, Indre Žliobaite, Dariusz Brzeziński, Eyke Hüllermeier, Mark Last, Vincent Lemaire, Tino Noack, Ammar Shaker, Sonja Sievi, Myra Spiliopoulou, et al. 2014. Open challenges for data stream mining research. ACM SIGKDD explorations newsletter 16, 1 (2014), 1–10

  4. [12]

    Zhizhong Li and Derek Hoiem. 2016. Learning without Forgetting. (6 2016). http://arxiv.org/abs/1606.09282

  5. [13]

    David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient Episodic Memory for Continual Learning. (6 2017). http://arxiv.org/abs/1706.08840

  6. [14]

    Simone Magistri, Tomaso Trinci, Albin Soutif-Cormerais, Joost van de Weijer, and Andrew D Bagdanov. 2024. Elastic feature consolidation for cold start exemplar- free incremental learning. arXiv preprint arXiv:2402.03917 (2024)

  7. [15]

    Mate Botond Nemeth, Emma Hart, Kevin Sim, and Quentin Renau. 2025. Class Incremental Learning for Algorithm Selection. https://doi.org/10.5281/zenodo. 15184096

  8. [16]

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural networks 113 (2019), 54–71

  9. [17]

    Roger Ratcliff. 1990. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review 97, 2 (1990), 285

  10. [18]

    Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. 2022. Three types of incremental learning. Nature Machine Intelligence 4, 12 (2022), 1185–1197

  11. [19]

    Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual Learning Through Synaptic Intelligence. (3 2017). http://arxiv.org/abs/1703.04200

  12. [20]

    Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. 2024. Class-incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

  13. [2017]

    ACM Com- puting Surveys (CSUR) 50, 2 (2017), 1–36

    A survey on ensemble learning for data stream classification. ACM Com- puting Surveys (CSUR) 50, 2 (2017), 1–36

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.