{"id":"a5d62a8a-f73f-4e51-999d-a489f7d0de48","arxiv_id":"2412.09701","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CUAL combines ambiguity-based active querying with confidence-filtered pseudo-labeling so a continual learner can handle unlabeled streams containing both old and novel classes under a tiny labeling budget.","lead":"This paper introduces CUAL, a system that lets a deployed image classifier keep learning from unlabeled data that contains both familiar and brand-new classes, using only a tiny labeling budget. It actively asks for labels only on the most ambiguous samples and confidently auto-labels the rest, and tests the idea on several image datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-label precision is the load-bearing assumption: CUAL's reported lead over ER/PseudoER/CCIC depends on the top-20% 'confident' pseudo-labels (Eq. 2, Appendix 4.1.1) being largely correct, yet no pseudo-label accuracy or oracle-pseudo-label control is reported.","rationale":"Reviewed the full text, including the appendix. The central claim is that CUAL's combination of uncertainty-guided active labeling of ambiguous samples and pseudo-labeling of confident novel samples outperforms ER, PseudoER, and CCIC under tiny labeling budgets. The ablation in Fig. 2 right shows pseudo-labeling contributes substantially to the reported accuracy, so the correctness of those pseudo-labels is load-bearing. The reader's weakest assumption identifies exactly this point, and I agree. The manuscript provides no pseudo-label accuracy metric, no error bars, and no oracle-pseudo-label control, leaving a concrete, testable gap. I do not see an internal contradiction or evidence of fraud; the concern is an unverified assumption, not a demonstrated failure. Therefore the conditional verdict remains appropriate: acceptance should depend on the proposed pseudo-label precision check or an equivalent validation.","tokens_in":10613,"tokens_out":7175,"duration_ms":80668,"concrete_test":"On a held-out labeled subset of U(t), run CUAL's normal inner loop, record the indices selected for pseudo-labeling (top α=20% of Si(u) above Tinner), and compute precision@20% against ground truth for the predicted novel class. Then run an 'oracle-pseudo-label' variant that uses the same selected indices but replaces the predicted labels with ground-truth labels before the Eq. (4) ER update. If oracle-pseudo-label accuracy exceeds CUAL by more than ~5 points on Im21K, or if precision@20% is below ~80%, the pseudo-labeling assumption is not secure and the headline gains need re-benchmarking against a less optimistic baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2.2 defines pseudo-labeling as selecting the top α=20% of samples with the highest Si(u) (Eq. 2), and Appendix 4.1.1 fixes α=20%. These pseudo-labels are trained into A_s and later into A via the ER loss in Eq. (4) with weight γ=0.25. The ablation row 'only-AL' (Fig. 2 right) shows a large drop (Im21K: 86.6 vs 79.1) when pseudo-labeling is removed, so this branch is a primary driver of the reported gains. Yet the paper reports no precision or accuracy for these pseudo-labels. The score Si is a ratio of old-class FRE to predicted-novel-class FRE; a high ratio is interpreted as high confidence, but it can also arise if an old-class sample is poorly reconstructed by all old classes and accidentally matches a novel class's PCA subspace, or if a novel sample is assigned to the wrong novel class but still yields a low denominator. The paper itself warns that 'wrong pseudo-labels ... can lead to severe degradation of performance' (Section 2.2.2), but provides no measurement of label noise. Consequently, the comparison against ER/PseudoER/CCIC is not yet a clean test of the proposed ambiguity-plus-pseudo-labeling mechanism; this is the most load-bearing assumption because the central claim relies on the selected pseudo-labels being sufficiently trustworthy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CUAL, a continual active learning method for a setting where unlabeled post-deployment data contains both unseen old-class samples and samples from novel classes, under a tiny labeling budget. CUAL builds a score S_i(u) that combines a feature-reconstruction-error (FRE) distance to old classes in the numerator and to a pseudo-predicted novel class in the denominator. Samples are partitioned into confident-novel, confident-old, and ambiguous; the method actively labels the ambiguous ones and pseudo-labels the top α=20% most confident novel samples, updating a short-term novel-class head and the long-term classifier via experience replay. Experiments compare CUAL with ER, PseudoER, and CCIC on four reported datasets (Imagenet21K-OOD, Places365, Eurosat, Cifar100-superclasses) and two frozen backbones, with ablations and budget sweeps, reporting accuracy advantages such as 86.6 vs. 76.1 on Im21K with ResNet50.","tokens_in":10959,"tokens_out":3981,"duration_ms":40675,"significance":"If the reported results are reliable, CUAL addresses a realistic and under-explored problem: continual active learning without the oracle assumption that new and old classes do not co-occur. The paper is among the first to propose ambiguity-based querying in continual AL, and the combination of active labeling and confidence-gated pseudo-labeling is a sensible mechanism. Strengths include the clear problem formulation, the use of a published uncertainty metric (FRE) with prior external benchmarking, ablations isolating the active-labeling and pseudo-labeling branches, and experiments across two backbones and multiple datasets. The main weaknesses that constrain the significance are the absence of any error bars or significance tests and the lack of any direct measurement or control for pseudo-label correctness, which is the mechanism behind the largest ablation gap.","major_comments":[{"comment":"No error bars or statistical significance tests are reported for any of the central comparisons. All claims that CUAL outperforms ER, PseudoER, and CCIC rest on single point estimates per dataset and backbone. Given that the margins in the left table of Figure 2 range from about 1.7 points (Eurosat ViT) to 11.7 points (Im21K R50), it is unknown whether the smallest gaps are within run-to-run noise. The authors should report results over multiple seeds (at least 3, preferably 5) with standard deviations, and where feasible a paired significance test, for the main tables and the ablations in Figure 2.","section":"Section 3.1, Figure 2"},{"comment":"The pseudo-labeling branch is the load-bearing component of the reported gains, but its reliability is never measured. The only-AL ablation in Figure 2 (right) shows a large drop relative to the default CUAL (e.g., Im21K R50: 86.6 vs. 79.1; Eurosat R50: 95.8 vs. 74.8), which indicates that the advantage largely depends on the correctness of the top-α pseudo-labels selected via Eq. (2) with α=20%. However, the paper reports no pseudo-label accuracy or precision, and it does not include a control experiment that replaces pseudo-labels with ground-truth novel-class labels. Since Section 2.2.2 itself warns that wrong pseudo-labels can cause severe degradation, the central claim requires at least (i) a measurement of pseudo-label precision on the selected α=20% samples, and (ii) an oracle-pseudo-label experiment to establish the ceiling of the pseudo-labeling branch.","section":"Section 2.2.2, Eq. (2), Appendix 4.1.1"},{"comment":"There is a mismatch between the listed datasets and the reported experiments. The text states 'We test on 4 diverse datasets' but then enumerates five: Imagenet21K-OOD, Places365-OOD, Eurosat, iNaturalist-Plants-20 (Plants), and Cifar100-superclasses. All result tables (Figure 1 and Figure 2) report only four datasets, and iNaturalist-Plants-20 does not appear in any table or figure. Either report the Plants results in the main paper or in the appendix, or correct the dataset list in the text to match the experiments actually presented.","section":"Section 3.1, Section 3.2"},{"comment":"The description of how the ER and PseudoER baselines select 'ambiguous' samples is unclear and potentially not a fair comparison. The text says that, similar to CUAL, the baselines actively label ambiguous samples 'according to the same formula as outlined in appendix 4.1.2', but that formula is defined specifically in terms of CUAL's S_i(u) score, which uses FRE and the pseudo-labeling head. For entropy-based ER and PseudoER, it is not clear how this formula is instantiated. Please specify the exact ambiguity score used for each baseline, and clarify whether the baselines have access to the same T_inner and PCA machinery; if they do not, the comparison may be giving CUAL an advantage in query selection that is not a property of the proposed method alone.","section":"Section 4.2.3"}],"minor_comments":[{"comment":"The phrase 'We test on 4 diverse datasets' should be revised to either 'five' or the list shortened to four, since the mismatch with the enumerated dataset list is confusing.","section":"Section 3.1"},{"comment":"In Eq. (2), the notation F RE_{m}^{i-1}(u) should be defined more precisely: the superscript i-1 indexes the PCA transform from the previous inner-loop iteration, while the subscript m is the pseudo-label predicted by A_s^cl(u; i-1, t). Make the dependence of the transform T_m^{t,i-1} explicit to avoid confusion.","section":"Equation (2)"},{"comment":"The left table in Figure 2 is referred to as 'Default CUAL and baselines' but the rows include 'Oracle', 'CUAL', 'ER-Ent', 'ER-Rand', etc. The caption would be clearer if it stated that the table reports average accuracy over all tasks for each dataset and backbone.","section":"Figure 2 caption"},{"comment":"The thresholds T_inner and α are described as 'not highly sensitive', but no quantitative evidence or sensitivity analysis is provided. A short table or figure showing performance for a few values of α (e.g., 10%, 20%, 30%) and for T_inner at 1, 2, and 3 standard deviations would substantiate this claim.","section":"Section 4.1.1"},{"comment":"There is a minor grammar issue: 'the latter Acl is same for all baselines' should be 'the latter Acl is the same for all baselines'.","section":"Section 4.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical submission with a clearly motivated problem and a broadly sensible method, but the missing statistical reliability measures and the untested pseudo-labeling assumption are substantial enough that the current evidence does not fully warrant acceptance. The authors should be encouraged to add multi-seed results, pseudo-label precision measurements, and an oracle-pseudo-label control. Also, the missing iNaturalist-Plants-20 results should either be added or removed from the text; the current inconsistency is visible to any reader. These are fixable within the scope of the manuscript, so major revision seems appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution to continual active learning with mixed old/new unlabeled pools. The setup removes the common oracle assumption that post-deployment data contains only novel classes, and the method—ambiguity-based active querying plus confident pseudo-labeling of novel-class samples—is new as a combination. The empirical story is internally consistent: CUAL beats ER, PseudoER, and CCIC on four datasets with two frozen backbones, and the ablations (AL-Top, AL-Rand, only-AL) line up with the claimed mechanism. Credit where due: the evaluation is thorough, the baselines are reasonable adaptations, and the authors are honest that wrong pseudo-labels can hurt.\n\nThe soft spot the stress test flags is real and is the main thing I'd want pinned down. Pseudo-labeling is load-bearing: removing it drops Im21K accuracy from 86.6 to 79.1, and an 8x larger AL budget still doesn't recover. Yet the paper never reports pseudo-label precision, and there's no oracle-pseudo-label control that would tell you how much of the gain comes from the pseudo-labels being correct versus from the ER training dynamics. The Si(u) ratio can be high for reasons other than true confidence—an old-class sample poorly reconstructed by old classes, or a novel sample assigned to the wrong novel class. So the magnitude of CUAL's advantage over PseudoER is not fully established. This is a genuine limitation, not a manufactured one.\n\nOther issues are minor by comparison. No error bars or significance tests anywhere. The paper lists five datasets (including Plants) but reports results for four and never explains where Plants went. PCA subspace dimension is unspecified. Code is promised but not released. These are fixable in revision.\n\nThe centrality of pseudo-labeling means I'd want to see precision numbers and an oracle-pseudo-label ablation before believing the exact margins. But the central empirical claim is plausible and consistent, and the setting is valuable enough that it deserves referee time. I'd send this to review with a request for the additional validation rather than desk-reject it. If you work on continual learning or active learning, it's worth a read; I'd cite it as evidence that ambiguity querying plus confident pseudo-labeling works in this mixed setting.","headline":"Solid and genuinely new continual active learning recipe, but the pseudo-labeling branch that drives much of the gain is validated only indirectly—worth refereeing, not desk-rejecting.","tokens_in":11472,"tokens_out":1808,"would_cite":true,"duration_ms":19030,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CUAL claims that when old and new classes co-occur in an unlabeled stream, the most informative samples to label are the ambiguous ones, not the most confidently novel, and that confident pseudo-labels can stretch a tiny labeling budget.","keywords":["continual learning","active learning","novel class discovery","uncertainty estimation","pseudo-labeling","out-of-distribution detection","feature reconstruction error","experience replay"],"falsifier":"Track the precision of CUAL's pseudo-labels on each task's novel-class pool. If the top $\\alpha=20\\%$ of highest-score samples are frequently mislabeled (for example, more than a small fraction are actually old-class outliers), or if training the same experience-replay procedure with corrupted pseudo-labels at that rate reproduces CUAL's accuracy drop, the uncertainty-scoring premise is falsified. A sharp test is to run CUAL with $\\alpha$ varied from 0% to 50% and check whether accuracy peaks near the claimed confident region or degrades immediately.","tokens_in":10398,"feed_emoji":"🎯","tokens_out":9334,"duration_ms":86512,"temperature":0.7,"pith_summary":"This paper tackles a realistic operational problem: a deployed classifier receives a stream of unlabeled images that mix old, previously learned classes with entirely new classes, and it has only a tiny labeling budget to keep learning. CUAL, the proposed method, actively asks a human to label the ambiguous samples—the ones that sit near the boundary between novel and familiar—rather than the most confidently novel ones, and simultaneously pseudo-labels the small fraction of samples it is most certain are new. The paper reports that this combination beats the usual continual-learning baselines (experience replay with entropy sampling, replay with pseudo-labeling, and the semi-supervised continual learner CCIC) across four datasets and two frozen feature backbones, at labeling budgets of 0.5–2.5%. The central claim is that ambiguity, not novelty confidence, is the right criterion for spending a scarce labeling budget when old and new classes co-occur.","feed_headline":"CUAL learns novel classes with 0.5–2.5% labels","feed_subtitle":"Ambiguity-guided queries and confident pseudo-labels beat standard replay at tiny labeling budgets.","key_machinery":"The load-bearing object is the uncertainty score $$S_i(u)=\\frac{\\min_{j\\in\\mathcal{C}_{\\text{old}}}\\mathrm{FRE}$_j^{0}$(u)}{\\mathrm{FRE}_{m}^{i-1}(u)},\\quad m=A^s_{cl}(u,i-1,t)\\in\\mathcal{C}^t_{\\text{new}},$$ where $\\mathrm{FRE}_m$ is a per-class feature reconstruction error: the $\\ell^2$ distance between a deep feature and its reconstruction after projection into a class-specific PCA subspace and back. FRE is built for each old class at task 0 and for each discovered novel class from the actively labeled samples of the previous inner-loop iteration. The score separates unlabeled samples into confidently novel (high), confidently old (low), and ambiguous (near the inner-loop threshold $T_{\\text{inner}}$); ambiguity is formalized as $1/\\lVert S_i(u)-T_{\\text{inner}}\\rVert^2$, so active queries select the samples closest to the old/new decision boundary, while the top $\\alpha=20\\%$ of high scores are pseudo-labeled. The short-term head $A^s_{cl}$ maps features to novel-class labels and routes the FRE selection; the long-term head is updated at task end by experience replay.","core_discovery":"The paper's central claim is that when an unlabeled continual-learning stream mixes old and new classes, the most valuable labels come from samples whose uncertainty score falls in the middle band: far enough from the old classes to look novel but close enough to the predicted novel class to be uncertain. CUAL computes, for each sample and each old class, a per-class feature reconstruction error (FRE), then forms a score $S_i(u)$ as the ratio of the minimum old-class error to the error under the novel class the pseudo-label head predicts. High ratios mark confidently novel samples, low ratios mark familiar samples, and samples near the threshold are ambiguous. The labeling budget is spent on the ambiguous band, while the top $\\alpha=20\\%$ of confidently novel samples receive pseudo-labels and both sets are used to retrain the heads and update the long-term classifier via experience replay. Across ImageNet21K-OOD, Places365, Eurosat, and Cifar100-superclasses, with frozen ResNet50 and ViT-S/16 features, the paper reports average accuracies such as 86.6% versus 76.1% for entropy-based experience replay on ImageNet21K-OOD; ablations show that removing pseudo-labeling, or querying the most novel or random samples, drops performance substantially, and that an eight-fold larger labeling budget without pseudo-labeling still trails the full method.","pith_inferences":["An implication left implicit is that the uncertainty scoring function does not depend on FRE being the only possible metric; the paper states CUAL is agnostic to the elemental uncertainty measure, so substituting a different per-class distance (for instance a nearest-classifier distance or a calibrated softmax score) would be a direct test of the method's generality.","The reported scaling with class increments (90.7% for one new class per task down to 83.6% for seven on ImageNet21K-OOD) suggests the confident-novel set becomes noisier as more classes arrive at once; a stress experiment with many novel classes per task and a fixed small budget would reveal whether the pseudo-labeling branch needs an early correction.","Because all experiments mix old and new samples at a fixed 2:1 ratio, a natural next test is to vary the novelty rate, especially making novel classes rare; in that regime the ambiguity band may contain mostly old-class samples, and CUAL's advantage over random labeling could narrow.","The paper explicitly leaves distribution shift of old classes for future work, so the current method should not be read as handling appearance changes in previously learned classes inside the unlabeled stream."],"forward_implications":["If the results hold, continual learners can operate in mixed old/new unlabeled streams with tiny labeling budgets (0.5–2.5%) instead of assuming the oracle condition that only novel classes appear.","Spending the labeling budget on ambiguous samples—those near the old/new boundary—matters more than selecting the most novel-looking samples; ablations show AL-Top and AL-Rand lag CUAL by 7–11 points on ImageNet21K-OOD.","Pseudo-labeling the most confident novel samples at $\\alpha=20\\%$ adds more value than multiplying the active labeling budget by eight without pseudo-labeling (83.6 vs 86.6 on ImageNet21K-OOD).","The advantage is reproducible across different frozen feature extractors (ResNet50 and ViT-S/16), indicating the method does not depend on one backbone's geometry.","Semi-supervised continual baselines that assume old and new classes do not co-occur (CCIC) fall well behind when that assumption is dropped."],"supporting_citations":[{"why":"Defines the feature reconstruction error (FRE) metric that CUAL's per-class uncertainty scoring is built on.","marker":"[16]"},{"why":"Shows FRE can be extended to continual novelty detection, providing the per-class PCA transform approach and the benchmarking setup CUAL adapts.","marker":"[17]"},{"why":"Supplies the experience replay algorithm and training-loss structure used to update the long-term classifier with active and pseudo-labeled samples.","marker":"[18, 3]"},{"why":"The CCIC semi-supervised continual learning baseline that CUAL must outperform when old and new classes co-occur.","marker":"[5]"},{"why":"Source of the pseudo-labeling idea used both by the PseudoER baseline and by CUAL's confident pseudo-label branch.","marker":"[29]"},{"why":"Demonstrates entropy/margin active querying in continual learning, the adaptation used to construct the ER and PseudoER baselines.","marker":"[13]"},{"why":"Provides the distinction between most-surely and least-surely uncertain samples that motivates querying ambiguous rather than confidently novel samples.","marker":"[15]"},{"why":"SwAV pretraining for the frozen ResNet50 feature extractor used in the primary experiments.","marker":"[20]"}],"fun_headline_variants":["Uncertainty band unlocks novel classes with tiny budgets","Ask the uncertain, pseudo-label the confident to learn new classes","CUAL: querying ambiguity beats entropy replay with 0.5% labels","Tiny label budget? Target middle uncertainty, pseudo-label the rest","Novel class mastery from 0.5% labels via uncertainty targeting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the most confident fifth of the samples it flags as novel are genuinely from a novel class, even though the classifier assigning those pseudo-labels was trained on only a tiny set of human-labeled examples; if those pseudo-labels are wrong, training on them can degrade performance rather than help.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty band unlocks novel classes with tiny budgets","Ask the uncertain, pseudo-label the confident to learn new classes","CUAL: querying ambiguity beats entropy replay with 0.5% labels","Tiny label budget? Target middle uncertainty, pseudo-label the rest","Novel class mastery from 0.5% labels via uncertainty targeting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1304,"prompt_tokens":982,"completion_tokens":322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":231}},"tokens_in":598,"tokens_out":322,"duration_ms":4014,"temperature":1.0,"reasoning_tokens":231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:50:28.229374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track the precision of CUAL's pseudo-labels on each task's novel-class pool. If the top $\\alpha=20\\%$ of highest-score samples are frequently mislabeled (for example, more than a small fraction are actually old-class outliers), or if training the same experience-replay procedure with corrupted pseudo-labels at that rate reproduces CUAL's accuracy drop, the uncertainty-scoring premise is falsified. A sharp test is to run CUAL with $\\alpha$ varied from 0% to 50% and check whether accuracy peaks near the claimed confident region or degrades immediately.","supporting_citations":[{"cited_title":"Most-surely vs. least-surely uncertain","cited_arxiv_id":null,"evidence_quote":"Provides the distinction between most-surely and least-surely uncertain samples that motivates querying ambiguous rather than confidently novel samples."},{"cited_title":"Unsupervised learning of visual features by contrasting cluster as- signments","cited_arxiv_id":null,"evidence_quote":"SwAV pretraining for the frozen ResNet50 feature extractor used in the primary experiments."}],"review_version":1}