{"id":"36fe24fd-11c0-47f0-9b5d-910486566770","arxiv_id":"2411.17941","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CRAB is a new active learning strategy that combines positive and negative label correlation matrices with beta scoring rules to improve multi-label text classification on imbalanced datasets.","lead":"This paper proposes a new way to choose which unlabeled documents to annotate first in multi-label text classification, where each document can have multiple labels. It uses correlations between labels plus a scoring rule that handles imbalanced data, and reports better accuracy than existing methods on legal, medical, and news datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'consistently superior' empirical claim is unsupported: all micro-F1 curves are averaged over 5 seeds with no error bars, CIs, or significance tests, despite many dataset-specific hyperparameter choices. A per-seed paired analysis is required before the claim can stand.","rationale":"The Reader's weakest_assumption was that the correlation matrices estimated from 100 labeled documents with up to 102 labels are noisy and unvalidated. That is a genuine mechanism-level risk, and it could amplify seed variance. However, the central claim is not primarily about the internal accuracy of A and NegA; it is about comparative performance. If the reported superiority is not statistically reproducible, the paper's main advertised benefit fails regardless of whether the correlation matrices are well estimated; if the superiority is statistically solid, the matrix-noise concern becomes a secondary robustness issue. I therefore place the load-bearing concern on the absence of per-seed statistical evidence and on the dataset-specific hyperparameter choices, both of which directly undermine the word 'consistently.' The reader's rationale already listed 'missing error bars and significance tests' as a weakness, so our concerns partially overlap, but the reader's weakest_assumption field identified a different issue. Conditional acceptance remains appropriate: the experiments are broad and the framework is coherent, but the headline claim needs the paired analysis above before it can be read as established. If the test shows non-significance, the verdict should move toward REJECT or UNVERDICTED; if it confirms significance and a sensitivity analysis is added, the paper could move toward ACCEPT.","tokens_in":13206,"tokens_out":7893,"duration_ms":73015,"concrete_test":"Rerun the four benchmark datasets times three backbones using the released code, preserving per-seed checkpoints. For each (dataset, backbone) configuration, compute the 5 per-seed micro-F1 deltas (CRAB minus best baseline) at the final acquisition budget and form a paired bootstrap 95% CI for the mean delta by resampling seeds with replacement. Also record how many of the 5 seeds are positive. If, for any configuration, the bootstrap CI includes zero or fewer than 4 of 5 seeds favor CRAB, the abstract's 'consistently' claim is not established and should be weakened to 'competitive on average'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is comparative: CRAB 'consistently achieves more reliable and superior performance' against several established methods (Abstract; Section 4.3). The only quantitative support is Figs. 2-5, whose captions state 'averaged results with 5 random seeds' and report no standard deviation, confidence interval, or significance test. With only 5 seeds, the reported micro-F1 gaps may lie within seed-level noise. This matters more than any single modeling assumption because it is the claim itself that is underdetermined. The risk is amplified by the number of protocol choices made on the evaluation data: Section 4.1 sets hard-to-learn query sizes to 300/200 and per-label sizes to 50/100 per dataset; Eq. 14 fixes gamma=2; Sections 3.3.1-3.3.2 set correlation thresholds to sigma and 2sigma; alpha=0.1 and beta=3 are inherited from BESRA's greedy search. No sensitivity analysis is reported for these choices on all datasets, and Fig. 8 examines only the hard-to-learn size/decay, apparently on a single dataset. Thus the observed advantage could be a consequence of favorable hyperparameter choices rather than the proposed mechanism. The correlation-matrix noise identified in the Reader's verdict is a plausible mechanism for seed variance, but the decisive check is whether the CRAB advantage survives a paired statistical analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRAB, a multi-label active learning strategy that maintains dynamically updated positive and negative correlation matrices over the label space, uses ensemble pseudo-labels to refine the unlabeled pool into label-wise, negatively correlated, and hard-to-learn subsets, and then scores the refined pool with a beta scoring rule augmented by an attention-style correlation term. The authors claim that CRAB consistently outperforms several established multi-label active learning baselines on four benchmark datasets across three backbone classifiers, and that it remains robust on synthetic datasets with varying imbalance ratios. The paper also includes ablations for the correlation-aware sampling components, an analysis of hard-to-learn query sizes and decay schedules, and a comparison of clustering-based versus random refinement of the unlabeled pool.","tokens_in":13581,"tokens_out":3904,"duration_ms":37544,"significance":"If the central empirical claim were fully supported, CRAB would be a useful contribution to multi-label active learning: it explicitly models negative and asymmetric label correlations without training additional correlation models, and it combines uncertainty, diversity, and imbalance-aware scoring in a single acquisition function. The paper has several strengths: the code is publicly released, the experimental comparison spans four datasets, three backbones, and five baselines, and the ablation study isolates several components of the method. However, the headline claim that CRAB 'consistently achieves more reliable and superior performance' is not yet supported by the reported evidence, because the experiments lack error bars, confidence intervals, and significance tests, and several key hyperparameters are either inherited from prior work or chosen per dataset without a corresponding sensitivity analysis. The contribution is therefore promising but needs additional empirical validation before the comparative claim can be accepted.","major_comments":[{"comment":"The central claim of the paper is comparative: CRAB 'consistently achieves more reliable and superior performance' against established methods (Abstract; Section 4.3). The only quantitative support is the averaged micro-F1 curves in Figures 2–5, whose captions state 'averaged results with 5 random seeds' but report no standard deviation, confidence interval, or significance test. With only five seeds, the reported gaps between CRAB and BESRA or AUDI may be within seed-level noise. The authors should provide per-seed paired comparisons, confidence intervals, or a statistical test (e.g., paired bootstrap or Wilcoxon signed-rank test) across the learning curves or at final acquisition budgets. Without this, the 'consistently superior' claim is underdetermined.","section":"Section 4.3, Figures 2–5"},{"comment":"The positive and negative correlation matrices are estimated from the labeled pool by simple conditional frequency counts. The labeled pool starts at only 100 documents (Section 4.1), while RCV1 has 102 labels; many entries of A(m,n) and NegA(m,n) will be estimated from zero or very few observations, and entries with zero denominator are undefined. These matrices are used directly in all three sampling heuristics (Sections 3.3.1–3.3.3) and in the attention score in Eq. (13). The paper provides no smoothing, shrinkage, or uncertainty quantification for these estimates, nor any sensitivity analysis for early active-learning iterations. The authors should analyze the noise in these matrices and demonstrate that the method's behavior is not driven by unreliable early estimates.","section":"Section 3.2, Eqs. (9)–(10)"},{"comment":"Several decisive hyperparameters are fixed per dataset or inherited from prior work, and the reported sensitivity analysis is not sufficient to establish that the observed gains come from the proposed mechanism. These include the hard-to-learn query size Z (300 for benchmark, 200 for synthetic), the per-label query size N (50 for RCV1, 100 for others), the attention normalization gamma=2 in Eq. (14), the correlation thresholds sigma and 2sigma in Sections 3.3.1–3.3.2, and alpha=0.1, beta=3 from BESRA. Figure 8 only studies the hard-to-learn size and decay on a single configuration, and no sensitivity analysis is reported for gamma, sigma, alpha, beta, or the per-label query size across all datasets. The authors should add a sensitivity study and clarify which choices, if any, were made after seeing evaluation-set performance.","section":"Sections 3.3–3.4 and 4.1"},{"comment":"The theoretical framework in Section 3.1, culminating in Eqs. (3)–(5), describes a Bayesian expected increase in a proper scoring rule, but the actual acquisition score used in Algorithm 1 is the attention-style beta score SAB in Eq. (13). The connection between the two is not established: Eq. (13) is introduced as a heuristic combination of per-label beta scores weighted by the positive correlation matrix, and it is not shown to approximate or upper-bound the expected score reduction in Eq. (3). The authors should either derive Eq. (13) from the expected-loss-reduction formulation under an explicit approximation, or state clearly that Eq. (13) is a heuristic acquisition score and adjust the claims accordingly.","section":"Sections 3.1 and 3.4, Eq. (13)"}],"minor_comments":[{"comment":"There is a typo in 'exployed' (should be 'employed'), and in Section 4.3 'generaliztion' should be 'generalization'.","section":"Section 4.1"},{"comment":"The caption 'Averaged micro-F1 score on DistilBERT' is awkward; it should be 'Averaged micro-F1 score using DistilBERT' or 'with DistilBERT'.","section":"Figure 2 caption"},{"comment":"The definition of the threshold sigma as 'the standard deviation of a two-tailed normal distribution' is vague, and Eq. (14) normalizes by the column maximum max(A(:, n)) while the text loosely refers to preserving each label's original significance; the normalization convention should be stated precisely.","section":"Section 3.3.1 and Eq. (14)"},{"comment":"The relation between the query size N, the per-label query size N, and the hard-to-learn query size Z is not fully specified: line 12 acquires only N centers after selecting N label-wise and N negative-correlated samples per label plus Z hard samples, and it is unclear how these quantities interact when the per-label sampling produces more than N candidates.","section":"Algorithm 1"},{"comment":"The notation switches between a continuous parameter space theta and a finite ensemble Theta_E without explaining how P(theta | L) is represented in the ensemble approximation; a brief clarification would help readers follow the derivation.","section":"Section 3.1, Eqs. (4)–(5)"}],"recommendation":"major_revision","confidential_remarks":"The paper falls within the scope of an applied machine learning journal and the authors provide a public code repository, which is commendable. My main concern is not the method's plausibility but the strength of the comparative evidence: the 'consistently superior' claim rests on five-seed averages without error bars or significance tests, and several hyperparameters are tuned per dataset. These issues are fixable in a revision, so I recommend major revision rather than rejection. I did not find evidence of problematic citation practices; the use of BESRA's alpha and beta values is disclosed, though the authors should make clear whether these values were selected on the evaluation datasets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before deciding whether to spend time on it. First, the core idea is a reasonable new combination: it couples dynamically updated positive and negative label-correlation matrices with beta scoring rules, ensemble pseudo-labeling, and k-means diversity in one multi-label active learning strategy. That specific combination is not in the prior literature, as far as I can tell, and the code is public. Second, the paper's central claim that CRAB 'consistently achieves more reliable and superior performance' is not backed by the statistics. Every micro-F1 curve is an average over 5 seeds with no error bars, confidence intervals, or paired significance tests. With 5 seeds, the reported gaps could be noise. The paper does several things well. The experimental setup is broad: four benchmark text datasets, three backbone classifiers, ablation studies for the main components, and synthetic imbalance variants. The authors are explicit that the beta-scoring parameters come from their own earlier BESRA paper, which is fine when the method is inherited. The ablations showing the benefit of negative-correlation sampling and the correlation-aware attention score are useful. The method also avoids extra trained modules, which is a practical plus. The soft spots are real but addressable. Besides missing error bars, several decisive hyperparameters are tuned on the evaluation datasets: alpha/beta from BESRA's greedy search, gamma=2, the sigma and 2sigma thresholds, and per-dataset query sizes. The sensitivity analysis in Figure 8 covers only the hard-to-learn query size and decay schedule, apparently on a single dataset. More importantly, the correlation matrices in Eqs. (9)-(10) are simple conditional frequencies computed from a labeled pool that starts at 100 documents, with up to 102 labels in RCV1. Early in active learning those estimates will be very noisy, and none of the three sampling heuristics or the attention score accounts for that. Finally, the theory in Section 3.1 derives an expected proper-score increment, but the implemented acquisition score in Eq. (13) is a heuristic attention-weighted sum that is never derived from that framework. That disconnect should be acknowledged or fixed. This is not a fatal paper. The mechanism is coherent, the experiments are wide, and the weaknesses are fixable with a per-seed paired analysis, error bars, and a clearer statement that Eq. (13) is a heuristic. I would not cite it in my own work until the empirical claim is tightened, but I would send it to peer review and ask for those revisions. If you work in active learning, it is worth a reading-group slot to see whether the correlation-based sampling idea has legs beyond the current evidence.","headline":"A plausible new combination for multi-label active learning that deserves peer review, but its 'consistently superior' claim is statistically unproven without per-seed analysis.","tokens_in":795,"tokens_out":1823,"would_cite":false,"duration_ms":30782,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Active learning that reads label links beats five baselines","keywords":["multi-label active learning","label correlation","negative correlation","beta scoring rules","ensemble pseudo labeling","data imbalance"],"falsifier":"Run CRAB on RCV1 with the correlation matrices replaced by random matrices (or by identity matrices) each iteration, keeping the rest of the pipeline unchanged. If micro-F1 does not drop substantially relative to the true matrices, the correlation component is not the source of the claimed advantage; alternatively, if performance is highly sensitive to a single early random seed when the labeled pool is 100, the matrix estimates may be too noisy.","tokens_in":101,"feed_emoji":"🎯","tokens_out":2362,"duration_ms":84651,"temperature":0.7,"pith_summary":"This paper proposes CRAB, a multi-label active learning strategy that uses progressively updated positive and negative label-correlation matrices to guide which unlabeled documents to query for annotation. The central claim is that incorporating inter-label relationships—both co-occurrence and mutual exclusivity—into a beta-scoring-rule acquisition function yields more reliable and higher micro-F1 performance than established multi-label active learning methods. The authors argue this matters because real-world multi-label datasets are large, imbalanced, and have complex label dependencies, so a query strategy that exploits those dependencies can reduce annotation cost while maintaining accuracy.","feed_headline":"Label correlations guide active learning to steady gains","feed_subtitle":"A new query strategy reads label links to beat five baselines on four text benchmarks, easing data imbalance.","key_machinery":"The core mechanism is the pair of correlation matrices defined in Eqs. (9) and (10): A(m,n) = P(y_m=1|y_n=1), the empirical probability that label m co-occurs with label n, and NegA(m,n) = P(y_m=-1|y_n=1), the empirical probability that label m is absent given label n is present. These matrices are recomputed after each acquisition iteration from the labeled pool, and they drive both the refined subset selection (label-wise, negative-correlated, hard-to-learn) and the correlation-aware beta score in Eq. (13), where each label's beta score is weighted by a normalized attention coefficient derived from A. This lets the acquisition function treat labels as interrelated rather than isolated, capturing asymmetric and hierarchical dependencies.","core_discovery":"The paper's central discovery is that a multi-label active learning acquisition function can be made correlation-aware without training additional correlation models. CRAB maintains two dynamic matrices: a positive co-occurrence matrix A and a negative anti-correlation matrix NegA, each computed from the currently labeled pool via conditional frequency counts. These matrices feed three correlated sampling heuristics—label-wise sampling that removes asymmetrically dependent labels, negative-correlated sampling that targets mutually exclusive label pairs, and hard-to-learn sampling—and also weight the beta scoring rule through an attention-like term. The authors report that this strategy consistently outperforms five baselines, including random sampling, MMC, AUDI, ADAPTIVE, BESRA, and CMAL, across four benchmark multi-label text datasets and synthetic variants with varying imbalance ratios.","pith_inferences":["A natural extension would be to quantify uncertainty in the correlation matrix estimates (e.g., with Dirichlet smoothing or Laplace smoothing) early in the learning process, when the labeled pool is only 100 documents and RCV1 has 102 labels; the paper does not test this sensitivity.","The same correlation-aware attention weighting could be plugged into other acquisition functions beyond beta scoring, such as entropy or margin-based learners, to test whether the benefit is tied to the specific scoring rule.","One could design a falsification experiment: shuffle the rows/columns of A and NegA after each iteration while keeping the rest of CRAB fixed. If performance does not degrade significantly relative to the true matrices, the correlation component would not be the source of the reported gains.","The paper implicitly assumes label correlations are stable across the unlabeled distribution, but in temporally drifting datasets (e.g., evolving legal text), recomputing matrices from only the labeled pool may not capture shifts; a streaming or exponential-decay update would be a natural test."],"forward_implications":["If CRAB's reported gains hold, multi-label active learning can leverage label correlation at negligible extra computational cost, since the matrices are simple frequency counts.","The explicit modeling of negative correlations offers a principled way to resolve mutually exclusive label predictions, which should help in domains with strict label hierarchies, such as legal or medical coding.","Because the beta scoring rules are tunable, the framework can be adjusted to different imbalance regimes, potentially extending beyond the four text benchmarks to image or sensor data.","The refined-pool sampling stage reduces the number of candidates scored by the acquisition function, which could make large-pool active learning more tractable."],"supporting_citations":[{"why":"Supplies the beta scoring rule framework and its hyperparameters (alpha=0.1, beta=3) that CRAB adopts as its base acquisition function.","marker":"[Tan et al., 2024]"},{"why":"Provides the AUDI baseline that combines uncertainty and diversity in multi-label active learning, which CRAB extends with correlation awareness.","marker":"[Huang and Zhou, 2013]"},{"why":"Introduces CMAL, a baseline that uses a global label correlation matrix; CRAB aims to improve on this by adding negative correlations and dynamic updates.","marker":"[Yu et al., 2020]"},{"why":"Inspires the attention mechanism used in Eq. (13) to weight label beta scores by correlation coefficients.","marker":"[Vaswani et al., 2017]"},{"why":"Establishes the theory of proper scoring rules that justifies using expected score reduction as an acquisition objective.","marker":"[Gneiting and Raftery, 2007]"},{"why":"Provides the RCV1 benchmark dataset used to evaluate CRAB and to create synthetic imbalance variants.","marker":"[Lewis et al., 2004]"}],"fun_headline_variants":["Correlation-aware queries boost multi-label active learning","Active learning leverages label co-occurrence for better picks","Multi-label active learning uses label links to counter imbalance","Label dependencies guide active sampling for reliable gains","Inter-label relationships sharpen active learning strategies"],"cache_read_input_tokens":16128,"weakest_assumption_plain":"The correlation matrices are computed from the currently labeled pool using plain frequency counts, and the paper assumes these are reliable guides for sample selection even when the pool starts at just 100 documents, including a dataset with 102 labels where most conditional counts will be zero or near-zero, and no smoothing is applied.","fun_headline_variants_meta":{"raw":{"variants":["Correlation-aware queries boost multi-label active learning","Active learning leverages label co-occurrence for better picks","Multi-label active learning uses label links to counter imbalance","Label dependencies guide active sampling for reliable gains","Inter-label relationships sharpen active learning strategies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1398,"prompt_tokens":839,"completion_tokens":559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":489}},"tokens_in":455,"tokens_out":559,"duration_ms":5837,"temperature":1.0,"reasoning_tokens":489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:40:45.961302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CRAB on RCV1 with the correlation matrices replaced by random matrices (or by identity matrices) each iteration, keeping the rest of the pipeline unchanged. If micro-F1 does not drop substantially relative to the true matrices, the correlation component is not the source of the claimed advantage; alternatively, if performance is highly sensitive to a single early random seed when the labeled pool is 100, the matrix estimates may be too noisy.","supporting_citations":[{"cited_title":"Harnessing the power of beta scoring in deep active learning for multi-label text classification","cited_arxiv_id":null,"evidence_quote":"Supplies the beta scoring rule framework and its hyperparameters (alpha=0.1, beta=3) that CRAB adopts as its base acquisition function."},{"cited_title":"Active query driven by uncertainty and diversity for incremental multi-label learning","cited_arxiv_id":null,"evidence_quote":"Provides the AUDI baseline that combines uncertainty and diversity in multi-label active learning, which CRAB extends with correlation awareness."},{"cited_title":"Rcv1: A new benchmark collection for text categorization research","cited_arxiv_id":null,"evidence_quote":"Provides the RCV1 benchmark dataset used to evaluate CRAB and to create synthetic imbalance variants."}],"review_version":1}