{"id":"785496b0-c492-4c5b-bb4c-07ee259aadf4","arxiv_id":"2412.10473","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CONCLAD combines iterative PCA-based uncertainty scores, a small active-labeling budget, and pseudo-labeling to continuously separate and learn multiple novel classes from old classes.","lead":"This paper presents a method that lets a deployed AI spot new classes in unlabeled data as they arrive, using a tiny amount of human labeling and automatic pseudo-labels. It addresses a gap in continual learning, where most methods assume an oracle tells the model when new classes appear or assume only one new class arrives at a time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of multi-class novel separation is unmeasured: AUROC only tests old-vs-new, and the 0.3% initial query can miss entire novel classes.","rationale":"The reader's verdict is CONDITIONAL and identifies per-class PCA reliability as the weakest assumption. My analysis agrees that erroneous pseudo-labels can corrupt PCA subspaces, but it sharpens the concern in two ways. First, the failure can occur before PCA reliability even matters: at i=0 the active-query budget may simply not include samples from every novel class, so no PCA is initialized for that class at all. Section 2.2.2 explicitly acknowledges that 'it is possible that not all true novel classes are found in this initial iteration,' but the paper provides no mechanism or evaluation showing that this missing-class scenario is rare or harmlessly corrected. Second, the evaluation never measures the multi-class part of the claim. AUROC is a binary old-vs-new measure; it cannot confirm that CONCLAD discriminates among multiple novel classes. The paper's own ablation 'No-Pseudo' shows pseudo-labeling matters, but the reported metric would improve even if pseudo-labels merely grouped all novel samples into one coarse cluster, as long as that cluster was distinct from old classes. A concrete test with class-coverage counts and novel-class clustering purity would settle whether the method actually delivers the 'several new classes at a time' capability promised in the abstract. Because the missing evidence is obtainable and the verdict is already CONDITIONAL, I do not change the verdict, but the acceptance condition should explicitly require these multi-class metrics.","tokens_in":10180,"tokens_out":5477,"duration_ms":63100,"concrete_test":"Run CONCLAD on the Im21K-OOD task with 5-class increments and a 0.3% supervision budget. After the i=0 active query, record which true novel classes received at least one label. On the held-out test set, assign each novel-class sample to the class with minimum final FRE and compute the adjusted Rand index (or cluster purity) against the true novel-class labels. If any true class has zero initial labels, or if novel-class purity is not high (e.g., greater than 0.8), the central claim of separating multiple novel classes is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 2.2.2, at the first inner iteration (i=0) the detector labels a tiny subset of U(t) (0.3%–1.25%) by sampling uniformly among samples with S0(u) greater than the mean S0(u). For the central claim to hold, this labeled set must cover every class in C_new^t, because per-class PCA transforms and the pseudo-labeler are initialized only from these labels. No coverage guarantee is provided. With 5 novel classes in Im21K-OOD and a 0.3% budget, a novel class can easily be absent from the high-S0 tail. Eq. (2) then routes samples of that missed class through the nearest initialized novel class m; the denominator FRE_m is large, so S_i is low and the samples are not recognized as novel. If they are confidently misrouted, they are pseudo-labeled as the wrong class, corrupting that class's PCA and the pseudo-labeler—the exact error-propagation loop the paper claims to minimize. Moreover, the only reported metric is AUROC (Fig. 2 left), which is binary old-vs-new and cannot verify the stated ability to 'discriminate between the different novel classes themselves.' No clustering purity, per-class recall, or novel-class discovery rate is reported anywhere in the manuscript. The missing Fig. 1 and the placeholder 'figure X' further remove the visual confirmation of the claimed large margins. Thus the most load-bearing part of the claim—multi-class novel separation under tiny supervision—is not actually measured, and the mechanism intended to enable it has no coverage guarantee.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CONCLAD, a continual novelty detection method that combines per-class PCA feature reconstruction error (FRE), an iterative uncertainty scoring function, active supervision from a tiny labeling budget, and pseudo-labeling of confident novel samples. The method is evaluated on four datasets with frozen ResNet50 and ViT backbones, and compared against continual novelty detection baselines and ablations. The central claim is that CONCLAD continuously separates old-class samples from multiple novel classes with high AUROC while using only 0.3% to 1.25% of each task's unlabeled pool for supervision.","tokens_in":10555,"tokens_out":3168,"duration_ms":36950,"significance":"If the claimed results hold, CONCLAD addresses a realistic and under-explored problem: continual novel class detection when several new classes can appear at once and no task oracle is available. The use of frozen feature extractors, the independent held-out test evaluation, the inclusion of official baseline implementations, and the ablations isolating the iterative and pseudo-labeling components are strengths. The main significance gap is that the paper's distinctive claim of discriminating among multiple novel classes is not directly measured: all reported metrics are binary old-versus-new AUROC, which would also be high for a method that collapses all novel classes into one. The lack of uncertainty quantification and missing reproducibility details further limit the strength of the numerical claims.","major_comments":[{"comment":"The central claim that CONCLAD discriminates between different novel classes themselves is not supported by any reported metric. AUROC is a binary old-versus-new measure; it does not test whether samples from two different novel classes are routed to distinct per-class PCA models or pseudo-labels. Since Eq. (2) depends on the pseudo-labeler assigning each sample to the correct novel class, the paper needs a direct inter-novel-class metric, such as per-class recall, clustering purity, or a confusion matrix among novel classes. Without this, the reported AUROC advantages could be obtained by a method that treats all novel samples as one class, which is exactly the failure mode the paper attributes to incDFM.","section":"Section 3.2 and Fig. 2 (left)"},{"comment":"The initial supervision query samples uniformly among instances with S0(u) above the mean S0(u), and these labels initialize both the pseudo-labeler and the per-novel-class PCA transforms. With a 0.3% budget and several novel classes per task, a novel class can easily be absent from this high-S0 tail. The paper acknowledges that not all novel classes may be found at i=0, but no recovery mechanism is described: at subsequent iterations the pseudo-labeler has output nodes only for already-detected classes, and Eq. (2) routes a sample of a missed class through the nearest initialized class, making its score low and preventing detection. The authors should either provide a coverage guarantee or a concrete mechanism for later iterations to discover skipped classes, and report per-class discovery rates.","section":"Section 2.2.2, first inner iteration"},{"comment":"The manuscript reports no error bars, no number of independent runs, and no standard deviations, even though several reported margins are small (e.g., Plants R50: CONCLAD 73.6 vs. incDFM 68.7; Plants ViT: CONCLAD 58.8 vs. incDFM 58.2). In addition, the per-class PCA subspace dimension is never stated, although it is a critical parameter of the FRE scoring function in Eq. (3). The authors should report means and variances over multiple runs, and state the PCA dimension and the hyperparameter selection procedure, to substantiate the claim of consistent and large margins.","section":"Section 3.1 and Appendix 4.2.1"}],"minor_comments":[{"comment":"The center panel of Fig. 1 is referenced as 'figure X', and Fig. 1 itself appears to be missing from the manuscript; this placeholder must be fixed before the paper can be properly reviewed.","section":"Section 3.2"},{"comment":"The supervision budget is stated inconsistently as 0.3% to 1.25% in the abstract, 0.3% to 2.5% in Section 2.2, and as varying from 0.32% to 5% in Section 3.1; please unify these numbers and clarify the budget used for each experiment.","section":"Abstract, Section 2.2, Section 3.1"},{"comment":"The pseudo-labeling head is denoted N_pl in Section 2.2.2 but N_ps and Acl_s in Section 3.1; please use consistent notation throughout.","section":"Section 3.1 and Section 2.2.2"},{"comment":"The inner-loop stopping threshold Tinner and the ambiguity formulation depend on an in-distribution validation set, but the paper does not state the size of this validation set or how its 0.1% subset is selected; please provide these details.","section":"Section 4.1.1"},{"comment":"The caption for Fig. 3 lists panels D/E for Plants and Cifar100, but the corresponding dataset selection and task design are not described as precisely as for the main datasets; please add the same setup details as in Section 3.1.","section":"Appendix 4.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper tackles a real gap: continual novel class detection when multiple novel classes arrive at once, without an oracle. The iterative multi-class uncertainty score (ratio of old-class FRE to predicted-novel-class FRE), combined with ambiguous-sample querying and pseudo-labeling, is a genuine capability expansion over the authors' earlier incDFM, which only handled one novel class per task. The ablations are the strongest part: No-Pseudo, No-Iters, Sup-Top, and Sup-Rand each isolate a design choice and show meaningful drops, so the components do seem to matter. The comparison to incDFM and DFM using official implementations is also fair, and the consistent AUROC margins across four datasets and two backbones are suggestive.\n\nThat said, the paper is not ready in its current form. The biggest issue is that the central claim, that CONCLAD can discriminate between the different novel classes themselves, is never actually measured. The only metric is AUROC, which is binary old-vs-new. There is no clustering purity, no per-class recall, no novel-class discovery rate anywhere in the manuscript. So the headline result is about separating novel from old, not about separating novel classes from each other.\n\nThe stress-test concern about coverage is also real, not hypothetical. At iteration 0, the method samples uniformly from the high-S0 tail of the unlabeled pool. With a 0.3% budget and five novel classes, a class can easily be missing from that tail. If that happens, samples from the missed class get routed through the nearest initialized novel class; the denominator in Eq. (2) becomes large, the score drops, and they are not flagged as novel. Confident misrouting then corrupts that class's PCA and the pseudo-labeler, which is the exact error-propagation loop the paper claims to minimize. The paper acknowledges that not all classes may be found initially, but it gives no coverage analysis and no fallback for missed classes.\n\nOther soft spots are more minor but still matter: no error bars or run counts, no released code, the PCA subspace dimension is unstated, and there are missing figures and placeholder references. These are all fixable. Self-citation is not a problem here, since extending incDFM requires citing it and the paper does compare against it.\n\nWho is this for? Researchers in continual learning, OOD detection, and active learning. They will find the problem framing and the iterative uncertainty formulation useful, but they should not take the multi-class separation claim at face value yet. The paper deserves a serious referee, but it needs major revision: report per-class separation metrics, address the coverage gap explicitly, release code, and clean up the manuscript. I would send it back, not desk-reject it.","headline":"A promising extension of incDFM to multi-class novelties with real ablations, but the central claim of separating novel classes is not actually measured and the tiny initial query has no coverage guarantee.","tokens_in":752,"tokens_out":792,"would_cite":false,"duration_ms":26763,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CONCLAD detects and learns several new classes at each task using only 0.3%–1.25% of the unlabeled pool for supervision, beating earlier continual novelty detectors by large margins.","keywords":["continual novelty detection","multiple novel classes","feature reconstruction error","pseudo-labeling","active learning","uncertainty estimation","continual learning","out-of-distribution detection"],"falsifier":"For a dataset with several novel classes per task, train each class's PCA subspace using only the samples CONCLAD would pseudo-label from a 0.3% budget, then measure the mean FRE of held-out samples from that same class versus held-out samples from other classes; the central ordering claim is falsified if, for any true novel class, held-out samples of that class do not have the lowest mean FRE under that class's own subspace.","tokens_in":10008,"feed_emoji":"🆕","tokens_out":12705,"duration_ms":116608,"temperature":0.7,"pith_summary":"This paper tackles the problem of a deployed classifier that must keep learning from a stream of unlabeled data in which several brand-new classes appear together, alongside familiar classes. CONCLAD is a method that, at each task, iteratively scores every incoming sample by a ratio of old-class to novel-class feature reconstruction errors, uses that score to ask a human to label the most ambiguous samples and auto-label the most confidently novel samples, then updates its per-class models. The paper claims that with a supervision budget of only 0.3% to 1.25% of each unlabeled pool, CONCLAD separates multiple novel classes from old classes and from each other, and that it does so consistently better than existing continual novelty detectors on four datasets and two frozen feature backbones. The result matters because it removes the unrealistic oracle assumption that someone tells the model when new data is novel, while also handling more than one new class at a time, which previous continual novelty detection could not.","feed_headline":"Detector spots multiple new classes on a 0.3% label budget","feed_subtitle":"Outperforms earlier continual novelty detectors on four datasets with minimal labeling.","key_machinery":"The central object is the iterative multi-class uncertainty score S_i(u) in Equation 2, built on per-class Feature Reconstruction Error (FRE). FRE for a class m is the ℓ2 norm of the difference between a deep feature vector and its reconstruction after applying that class's PCA transform and its inverse; it measures how far a sample is from the class's feature subspace. In S_i(u), the numerator is the minimum FRE over all old classes and the denominator is the FRE of the pseudo-predicted novel class m, so the ratio orders samples from confidently novel (large numerator, small denominator) to confidently old (small numerator, large denominator). This single scalar score drives three operations: selecting ambiguous samples for the tiny active label budget, thresholding confidently novel samples for pseudo-labeling, and updating per-class PCA transforms and the pseudo-labeling head at each inner-loop iteration.","core_discovery":"The paper's central claim is that generalized continual novel class detection—multiple new classes arriving together at each task, mixed with unseen old-class samples and no task oracle—becomes tractable if uncertainty is measured iteratively by a ratio of feature reconstruction errors. Starting from per-class PCA models of the old classes, CONCLAD first labels a few ambiguous samples, builds a rough PCA subspace and a pseudo-labeling head for each detected novel class, then recomputes the score S_i(u) = min_{j in old} FRE_j(u) / FRE_m(u) for the pseudo-predicted novel class m. High scores mark confidently novel samples for pseudo-labeling; low scores mark old-class samples; mid-range scores are actively queried. Over inner-loop iterations the per-class subspaces and the pseudo-labeler improve, and the final models are folded into the detector so that the new classes become old classes in the next task. Averaged AUROC numbers reported for ImageNet-21K-OOD, Eurosat, iNaturalist-Plants-20, and Cifar100 place CONCLAD above incDFM, DFM, ER, and PseudoER across almost all settings, with ablations indicating that the iterative loop, the pseudo-labeling, and the ambiguous-sample querying all contribute to the gain.","pith_inferences":["Because the score design is agnostic to the underlying uncertainty metric, one could re-instantiate Equation 2 with Mahalanobis distances or likelihood ratios instead of FRE and test whether the ordering property persists with fewer labeled samples.","The method's reliance on a frozen, high-quality feature extractor suggests the reported gains may depend on representation quality; a natural stress test is to run CONCLAD with a smaller or fine-tuned backbone and see how much of the AUROC gap remains.","The first inner iteration has no novel-class models, so the initial ambiguity sampling is inherently heuristic; an adaptive querying rule that expands candidate coverage as novel classes are discovered might improve early-task accuracy on very large pools.","Since the authors list old-class distribution shift as future work, a testable extension is to augment the FRE ratio with a drift-aware term so the detector can distinguish a genuinely new class from an old class whose appearance has shifted."],"forward_implications":["With a supervision budget as small as 0.3% of the unlabeled pool, several new classes can be discovered, separated, and learned within one task instead of just one new class per task.","Because each novel class receives its own PCA transform, updates are incremental: classes discovered at task t are added without disturbing older transforms and become old classes at task t+1.","Querying ambiguous samples near the detection threshold contributes to the gains; the ablations show it beats querying the most-confident novel samples (Sup-Top) by 2.4% average AUROC.","The iterative inner loop is essential: removing it costs an average 11.2% AUROC, and removing pseudo-labeling costs 16.8%, indicating both serve to limit continual error propagation."],"supporting_citations":[{"why":"Defines the prior single-class continual novelty detector that CONCLAD extends and compares against; CONCLAD targets its multi-class collapse.","marker":"[11]"},{"why":"Introduces the feature reconstruction error (FRE) and per-class PCA transforms that Equation 2 is built on.","marker":"[14]"},{"why":"Originally proposed the FRE measure for static novelty detection; used as a continual baseline adapted with per-task PCA transforms.","marker":"[26]"},{"why":"Provides the experience replay baseline (ER) adapted for semi-supervised continual learning with entropy-based active labeling.","marker":"[27]"},{"why":"Defines the pseudo-labeling baseline (PseudoER) that CONCLAD outperforms, and motivates the need for a reliable confidence score.","marker":"[29]"},{"why":"Supplies the SwAV-pretrained ResNet50 backbone used as the frozen feature extractor for most experiments.","marker":"[31]"},{"why":"Supplies the DINO-pretrained ViT backbone used as the second feature extractor in experiments.","marker":"[33]"},{"why":"Provides the Cifar100-superclasses benchmark that allows direct comparison with incDFM.","marker":"[25]"}],"fun_headline_variants":["CONCLAD: finds many new classes with only 0.3% labels","No oracle: CONCLAD still spots novel classes on the fly","Iterative uncertainty detection nails multiple new classes","Pseudo-labels and active queries beat prior novelty detectors","0.3% supervision, many new classes: CONCLAD's trick"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a per-class PCA transform estimated from a tiny handful of labeled and pseudo-labeled samples (as few as 0.3% of the pool) faithfully represents the class's true feature distribution, so that the ratio in Equation 2 orders confident novel from confident old; if a pseudo-labeled sample is misassigned, its polluted subspace can systematically misroute later samples.","fun_headline_variants_meta":{"raw":{"variants":["CONCLAD: finds many new classes with only 0.3% labels","No oracle: CONCLAD still spots novel classes on the fly","Iterative uncertainty detection nails multiple new classes","Pseudo-labels and active queries beat prior novelty detectors","0.3% supervision, many new classes: CONCLAD's trick"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3506,"prompt_tokens":946,"completion_tokens":2560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2471}},"tokens_in":562,"tokens_out":2560,"duration_ms":19612,"temperature":1.0,"reasoning_tokens":2471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:43:33.790311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a dataset with several novel classes per task, train each class's PCA subspace using only the samples CONCLAD would pseudo-label from a 0.3% budget, then measure the mean FRE of held-out samples from that same class versus held-out samples from other classes; the central ordering claim is falsified if, for any true novel class, held-out samples of that class do not have the lowest mean FRE under that class's own subspace.","supporting_citations":[{"cited_title":"incdfm: Incremental deep feature modeling for continual novelty detec- tion","cited_arxiv_id":null,"evidence_quote":"Defines the prior single-class continual novelty detector that CONCLAD extends and compares against; CONCLAD targets its multi-class collapse."},{"cited_title":"Subspace Modeling for Fast Out- Of-Distribution and Anomaly Detection","cited_arxiv_id":null,"evidence_quote":"Originally proposed the FRE measure for static novelty detection; used as a continual baseline adapted with per-task PCA transforms."},{"cited_title":"Experience replay for continual learning","cited_arxiv_id":null,"evidence_quote":"Provides the experience replay baseline (ER) adapted for semi-supervised continual learning with entropy-based active labeling."},{"cited_title":"Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks","cited_arxiv_id":null,"evidence_quote":"Defines the pseudo-labeling baseline (PseudoER) that CONCLAD outperforms, and motivates the need for a reliable confidence score."},{"cited_title":"Unsupervised learning of visual features by contrasting cluster as- signments","cited_arxiv_id":null,"evidence_quote":"Supplies the SwAV-pretrained ResNet50 backbone used as the frozen feature extractor for most experiments."},{"cited_title":"Emerging properties in self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the DINO-pretrained ViT backbone used as the second feature extractor in experiments."},{"cited_title":"Learning Multiple Layers of Features from Tiny Images","cited_arxiv_id":null,"evidence_quote":"Provides the Cifar100-superclasses benchmark that allows direct comparison with incDFM."}],"review_version":1}