{"id":"a33bd045-9754-4f6a-9640-b5c7d45f3e59","arxiv_id":"2412.06642","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A distribution-matching, class-balanced selection method (CBS) improves incremental learning from unlabeled pools, beating random and standard active learning baselines on five datasets.","lead":"This paper introduces Active Class-Incremental Learning, a setting where an algorithm picks a small number of images from a large unlabeled pool to label for each new batch of classes. It proposes a class-balanced, distribution-matching selection strategy that beats random sampling and standard active learning baselines on five image datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Protocol confound: CBS selects and trains once (50 epochs), while AL baselines use a multi-round 20-sample loop with up to 130 epochs, so reported gains may reflect training protocol rather than selection quality.","rationale":"The reader's verdict is CONDITIONAL and identifies cluster-class alignment as the weakest assumption. I agree that alignment is unvalidated, but I view the single most load-bearing issue as the training-protocol confound between CBS and the AL baselines. The central claim is an empirical superiority claim; the evidence must isolate the selection strategy. The paper's own implementation details show the two arms use different selection/training loops, with baselines receiving substantially more total epochs (130 vs 50 for B=100). This could easily account for differences in Avg accuracy, making the tables and figures inconclusive. A matched-protocol rerun is the decisive experiment. The protocol confound does not invalidate the method's internal logic; therefore I recommend keeping the verdict CONDITIONAL (UNCHANGED) rather than rejecting. The reader's cluster concern remains valid: k-means with k=|C_t| is not guaranteed to produce class-aligned clusters, and the random discard step in Alg. 2 line 16 further weakens the class-balance guarantee. However, even if that issue were fixed, the empirical claim would still be unsupported until the protocol confound is addressed. Thus agreement_with_reader is 'partial': the reader's rationale mentions the confound, but the weakest_assumption field names a different specific link.","tokens_in":24671,"tokens_out":7068,"duration_ms":72183,"concrete_test":"Re-run the Table 1 experiments (B=100, three CIL methods, five datasets) with a matched single-shot protocol: each AL baseline selects B samples at once using its acquisition function evaluated on the same pretrained feature extractor / current session model, then trains exactly 50 epochs once, identical to CBS. If CBS's Mean Avg advantage over random and the baselines shrinks or reverses, the reported gains are attributable to the training protocol rather than to CBS. As a complementary check, run CBS in the multi-round 20-sample/round protocol with the same epoch schedule as the baselines; if CBS then fails to beat random, the selection strategy itself is not the cause of the original gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline comparison (Table 1, Fig. 2) pits CBS, which selects all B samples at once (Alg. 1 line 4, Alg. 2) and trains once for 50 epochs, against AL baselines run in their native multi-round protocol: select 20 samples, label, train, repeat R = B/20 times. Implementation Details (Sec. 4.1) state that baselines train 20 epochs in each of the first R-1 rounds and 50 epochs in the R-th round; for B=100 this is 4x20+50 = 130 epochs versus CBS's 50. The two arms therefore differ in training schedule, cumulative epochs per sample, and whether the acquisition model is updated during selection. Any observed Avg advantage for CBS could be caused by single-shot training rather than by CBS's sample-selection strategy. The central claim that 'CBS outperforms both random selection and other SOTA active learning approaches' is thus not established by the reported experiments. The cluster-alignment assumption flagged by the reader is a real secondary risk, but the protocol confound is more directly damaging: even if k-means clusters were perfectly class-aligned, the current evidence would still not isolate the effect of the selection strategy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Active Class-Incremental Learning (ACIL), where each incremental session has a large unlabeled pool and a labeling budget. It proposes Class-Balanced Selection (CBS), which extracts features with a frozen pretrained encoder, clusters the unlabeled pool into |C_t| groups via k-means, and then greedily selects samples within each cluster so that the Gaussian distribution of the selected features matches the Gaussian distribution of the cluster. When used with LP-DiF, the method also exploits the unselected unlabeled data to improve Gaussian replay distributions. The method is evaluated on five datasets with three prompt-based CIL backbones (L2P, DualPrompt, LP-DiF) and compared against random selection and seven active learning baselines; the paper reports that CBS outperforms both random and the baselines.","tokens_in":24881,"tokens_out":3477,"duration_ms":36446,"significance":"If the empirical claims hold, ACIL is a useful problem formulation and CBS is a practical, model-agnostic selection strategy that could reduce annotation cost while preserving incremental learning accuracy. The strengths include the public code release, evaluation across five datasets and three CIL backbones, and an ablation (Table 2) that separates the clustering step from the greedy selection step. However, the central empirical claim is currently undermined by a protocol confound in the comparison with active learning baselines, and the paper's supporting analysis of class balance relies on an unvalidated cluster-alignment assumption and a circular KL-divergence demonstration. The core idea remains plausible, but the evidence as reported does not yet isolate the effect of the selection strategy.","major_comments":[{"comment":"The comparison between CBS and the active learning baselines is confounded by training protocol. CBS selects all B samples at once and trains for 50 epochs, whereas the AL baselines are run in a multi-round loop: 20 samples selected, labeled, and trained per round, with 20 epochs in each of the first R-1 rounds and 50 epochs in the final round. For B=100, this gives the baselines 130 epochs versus CBS's 50 epochs. The reported Avg gains for CBS could therefore be caused by the single-shot training schedule rather than by the selection quality. This directly impacts the headline claim in the Abstract and Section 1 that CBS outperforms existing AL approaches. The authors should rerun the comparison with the training schedule equalized across both arms (e.g., train all selected sets for the same total number of epochs, or give CBS the same multi-round training as the baselines) and re-report Table 1 and Fig. 2 accordingly.","section":"Sec. 4.1, Implementation Details; Table 1; Fig. 2"},{"comment":"The class-balance guarantee of CBS rests on the assumption that k-means clustering with k = |C_t| produces clusters that align with the true session classes. The paper provides no evidence of this alignment: the 'class-imbalanced ratio' analysis in the appendix is computed using ground-truth labels of the selected samples, not on the cluster composition, and no cluster-purity metric is reported. If the clusters do not correspond to classes, proportional sampling per cluster does not guarantee class balance, and the KL objective only matches the selected set to the cluster distribution, not to the class distribution. Please report cluster purity (e.g., adjusted Rand index against the session's class labels) or vary k and show the sensitivity; without this, the mechanism linking CBS to class balance is unsupported.","section":"Alg. 2, line 4; Sec. 3.2; Sec. A of the appendix"},{"comment":"The KL-divergence comparison in Fig. 7 is circular as a demonstration of representativeness, because CBS greedily minimizes exactly this KL divergence between the selected set's Gaussian and the full cluster's Gaussian (Eq. 2 in Sec. 3.2). Reporting that CBS achieves lower KL than random is therefore a direct consequence of the optimization objective, not independent evidence that the selected samples are more representative. If this figure is meant as a sanity check of the optimizer, it should be labeled as such; otherwise, please provide an external metric not aligned with the objective, such as nearest-neighbor classification accuracy on the selected set or coverage of the feature space.","section":"Appendix D, Eq. (4) and Fig. 7"}],"minor_comments":[{"comment":"The sentence 'CIFAR-100 consists of 100 general classes, each of which contains 50,000 training images' is incorrect as written (CIFAR-100 has 50,000 training images in total); it should say each class contains about 500 training images.","section":"Appendix A"},{"comment":"The cross-reference to Table 4 appears as 'Tab. ??' in the text; please fix the reference.","section":"Appendix D, Table 4"},{"comment":"The phrase 'our CBS outperform Balance random for all CIL methods on most datasets' contains a typo ('outperform' should be 'outperforms') and the term 'Balance random' is not defined in the main text; the table uses 'Balanced random (FSCIL)'.","section":"Sec. 4.2, after Table 1"},{"comment":"The class-imbalanced ratio divides the count of the most-selected class by the count of the least-selected class; please state explicitly how ties and classes with zero selected samples are handled, since those cases affect the ratio significantly when the budget is small.","section":"Sec. A of the appendix, 'class-imbalanced ratio' definition"},{"comment":"The ablation that replaces the greedy selection with Entropy, Coreset, and BADGE within each cluster is reported only on CUB-200 with LP-DiF at B=100; adding at least one more dataset/backbone would strengthen the claim that the greedy step, not the clustering, is responsible for the improvement.","section":"Sec. 4.3, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The protocol confound is the main issue: the paper's experimental design gives CBS 50 epochs and the AL baselines 130 epochs (for B=100), so the superiority claim is not yet established. The appendix's CIFAR-100-LT discussion (Sec. C) actually acknowledges the multi-round versus single-round difference, but the main paper does not address it as a confound. This is fixable with a controlled experimental rerun, and the cluster-alignment validation is also a reasonable addition, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper is worth reading for the task formulation—active class-incremental learning—and for the observation that standard AL methods produce class-imbalanced samples and often underperform random in this setting. The proposed CBS (cluster features, then greedily match per-cluster Gaussian statistics) is simple, plausible, and code is provided. But the central empirical claim, that CBS outperforms both random selection and other SOTA active learning approaches, is not actually established by the experiments as reported, because the two arms are not trained under the same protocol.\n\nThe details matter. For CBS, you select all B samples at once and train once for 50 epochs. For the AL baselines, the authors keep their native multi-round pipeline: select 20, label, train 20 epochs, repeat R = B/20 times, with the last round going 50 epochs. For B=100 that's 4x20 + 50 = 130 epochs per session. So the baselines get more training updates, a different batch composition over time, and an acquisition model that is updated during selection. CBS gets 50 epochs and a static selection. Any reported advantage could come from the training schedule or the one-shot protocol, not from the selection strategy per se.\n\nThe authors are aware of this—they state the schedule difference in Appendix B, and in the CIFAR-100-LT section they attribute CBS's margin partly to the baselines overfitting in their multi-round paradigm. That is a direct admission that the comparison conflates protocol with selection quality. The fix is straightforward: either give all methods the same total number of epochs and the same single-shot selection loop, or run CBS in the same multi-round fashion.\n\nSecondary issues: the k-means step assumes clusters align with the true classes. The paper provides some indirect evidence that this works in practice (the class-imbalance ratio and classes discovery ratio in Fig. 5/6), but it never validates the alignment directly. Also missing: error bars. Some gains over Balanced random (the FSCIL reference) are fractions of a point on CIFAR-100, and without variance it's hard to tell if that's meaningful. And the KL-divergence analysis in Fig. 7 is circular—it's the objective CBS optimizes, so of course it's lower.\n\nWhat's good: the task itself is a natural and useful benchmark for low-annotation continual learning, and the paper makes a compelling case that class balance matters. The method is simple and efficient, and the plug-and-play design across L2P, DualPrompt, and LP-DiF is a nice contribution. If the evaluation is redone with a matched protocol, I expect the core idea to survive, though the margins may shrink.\n\nBottom line: this deserves a serious referee (or a revision), but the empirical claim as written is conditional. I would not desk-reject it, and I'd like to see the re-run.","headline":"A genuinely new task (ACIL) and a simple class-balance-driven selector, but the main comparison is confounded by mismatched training schedules: CBS gets 50 epochs once, baselines get up to 130+ epochs in multi-round loops.","tokens_in":25439,"tokens_out":7175,"would_cite":true,"duration_ms":65903,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Class-balanced selection outperforms random and active learning baselines in the new Active Class-Incremental Learning protocol.","keywords":["Active Class-Incremental Learning","class-balanced selection","active learning","few-shot class-incremental learning","prompt tuning","k-means clustering","Gaussian distribution matching","class-imbalanced ratio"],"falsifier":"Build an incremental session from two classes whose pretrained features overlap heavily, run CBS with the same budget, and measure the class-imbalanced ratio of the selected set and the session's accuracy. If the ratio is no better than random and accuracy drops to the random baseline, the cluster-alignment premise fails.","tokens_in":24445,"feed_emoji":"⚖️","tokens_out":9278,"duration_ms":88162,"temperature":0.7,"pith_summary":"This paper proposes Active Class-Incremental Learning (ACIL), in which each incremental session begins with a large pool of unlabeled images and a small labeling budget, and the goal is to choose which images to label so that the incremental learner performs as well as possible. The authors report that standard active learning heuristics—uncertainty, diversity, and hybrid scores—select class-imbalanced batches under this protocol and can do worse than random sampling, because an imbalanced training set hurts the incremental learner. They propose Class-Balanced Selection (CBS), which clusters the unlabeled features into as many groups as there are classes and then greedily picks samples from each cluster so that the selected samples' Gaussian distribution matches the cluster's distribution. Across five datasets and three prompt-tuning incremental methods, CBS beats random selection, the active learning baselines, and forced balanced random selection at the same labeling budget, with the largest gains at small budgets.","feed_headline":"Class-balanced selection beats active learning on incremental tasks","feed_subtitle":"Matching labeled samples to the unlabeled pool's feature distribution lifts accuracy on five datasets.","key_machinery":"The core object is the Class-Balanced Selection (CBS) routine. At each session it extracts normalized features with the pretrained encoder, runs k-means with k equal to the session's class count, allocates selections to each cluster in proportion to cluster size, and then greedily adds samples that minimize the KL divergence between the Gaussian fitted to the selected samples and the Gaussian fitted to the entire cluster. This distribution-matching step buys both class balance, through proportional cluster budgets, and representativeness, through per-cluster greedy matching. For LP-DiF, a second mechanism estimates per-class Gaussians from labeled plus pseudo-labeled unselected data, so replay pseudo-features better resist catastrophic forgetting.","core_discovery":"The central claim is that class balance, not uncertainty or diversity per se, is what makes an annotated subset useful for incremental learning under the ACIL protocol. The paper shows empirically that existing active learning methods (Entropy, Margin, Coreset, BADGE, Typiclust, ProbCover, DropQuery) produce class-imbalanced selections that are sometimes worse than random selection, and that their accuracy trails random selection as a result. CBS fixes this by making the selected set's feature distribution mirror the whole unlabeled pool: k-means splits the pool into as many clusters as there are classes, each cluster contributes a number of samples proportional to its size, and within each cluster a greedy rule adds the sample that most reduces the KL divergence between the Gaussian of the selected samples and the Gaussian of the cluster. When plugged into L2P, DualPrompt, and LP-DiF, CBS raises average session accuracy over random and all baselines on five datasets; in LP-DiF, the paper further uses the unselected unlabeled data to estimate class Gaussians for pseudo-feature replay, which specifically improves old-class accuracy.","pith_inferences":["Beyond the paper's non-overlapping-class assumption, the method's balance guarantee needs the clusters to track true classes; an open-world ACIL setting would require estimating the number of clusters or filtering old-class features before running CBS.","The greedy step is pure distribution matching on frozen features, so the same routine could be reused for any budgeted subset-selection task where representativeness matters, such as coreset construction, without retraining.","The paper's link between class-imbalanced ratio and accuracy suggests a cheap diagnostic: measure the imbalance ratio of any proposed selection before labeling, and if it is worse than random, expect the learner to underperform random.","The LP-DiF gain from pseudo-labeling the unused pool suggests active selection and semi-supervised distribution estimation are complementary; combining CBS with confidence-thresholded pseudo-labeling is a testable extension."],"forward_implications":["At a labeling budget of $B=100$, CBS raises mean average accuracy over five datasets to 81.42 for L2P, 83.01 for DualPrompt, and 82.03 for LP-DiF, versus 78.19, 79.77, and 80.45 for random selection.","The advantage over random and prior active learning grows at small budgets ($B=40$ or $60$), where class imbalance is most damaging and CBS's classes-discovery ratio stays highest.","CBS is model-agnostic for prompt-tuning incremental learners: it swaps into L2P, DualPrompt, and LP-DiF without changing their training procedures.","For LP-DiF, incorporating unselected unlabeled data into Gaussian replay improves old-class accuracy (from 66.21% to 67.66% on the last CUB-200 session) while leaving new-class accuracy unchanged.","CBS selects a batch in one pass, so it is cheaper than multi-round active learners: 42 seconds per session versus 149 for DropQuery on CUB-200."],"supporting_citations":[{"why":"Supplies the k-means clustering step that partitions unlabeled features into as many groups as there are classes in the session.","marker":"[36]"},{"why":"LP-DiF is the main prompt-tuning incremental learner CBS is plugged into, and its Gaussian-replay estimation is extended with unused unlabeled data.","marker":"[26]"},{"why":"L2P serves as one of the three prompt-tuning CIL backbones used to test CBS.","marker":"[56]"},{"why":"DualPrompt serves as one of the three prompt-tuning CIL backbones used to test CBS.","marker":"[55]"},{"why":"Entropy is an uncertainty-based active learning baseline that CBS must beat.","marker":"[24]"},{"why":"Margin is an uncertainty-based active learning baseline that CBS must beat.","marker":"[43]"},{"why":"BADGE is a hybrid uncertainty-diversity baseline that CBS must beat.","marker":"[7]"},{"why":"Coreset is a diversity-based baseline that CBS must beat.","marker":"[47]"},{"why":"Typiclust is a density-based baseline that CBS must beat.","marker":"[22]"},{"why":"ProbCover is a density-based baseline that CBS must beat.","marker":"[65]"}],"fun_headline_variants":["Balanced picks beat clever picks for incremental learning","Why class balance trumps uncertainty in active learning","Matching feature spread boosts incremental learning accuracy","Active learning fails without class balance, new method fixes it","CBS: selecting samples to mirror the pool lifts incremental learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes k-means clusters of pretrained features line up with the true classes in each session's unlabeled pool, so that sampling proportionally from each cluster yields a class-balanced labeled set.","fun_headline_variants_meta":{"raw":{"variants":["Balanced picks beat clever picks for incremental learning","Why class balance trumps uncertainty in active learning","Matching feature spread boosts incremental learning accuracy","Active learning fails without class balance, new method fixes it","CBS: selecting samples to mirror the pool lifts incremental learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2582,"prompt_tokens":1020,"completion_tokens":1562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1487}},"tokens_in":636,"tokens_out":1562,"duration_ms":11931,"temperature":1.0,"reasoning_tokens":1487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:26:56.045643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an incremental session from two classes whose pretrained features overlap heavily, run CBS with the same budget, and measure the class-imbalanced ratio of the selected set and the session's accuracy. If the ratio is no better than random and accuracy drops to the random baseline, the cluster-alignment premise fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the k-means clustering step that partitions unlabeled features into as many groups as there are classes in the session."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DualPrompt serves as one of the three prompt-tuning CIL backbones used to test CBS."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Entropy is an uncertainty-based active learning baseline that CBS must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Margin is an uncertainty-based active learning baseline that CBS must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Coreset is a diversity-based baseline that CBS must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Typiclust is a density-based baseline that CBS must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ProbCover is a density-based baseline that CBS must beat."}],"review_version":1}