{"id":"9b76154b-f816-4f98-b59f-6b2a5671816c","arxiv_id":"2508.21263","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Simple uncertainty-based active learning with a weighted loss matched full-data COVID-19 CXR severity classifiers using 15.4% (binary) and 23.1% (multi-class) of training labels.","lead":"This retrospective study tested whether active learning can train a COVID-19 lung severity classifier from chest X-rays using only a fraction of the usual radiologist labels. Simple uncertainty-based sampling matched full-data accuracy with about 15% of labels in binary and 23% in three-class tasks, though methodological questions about the data split remain open.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central label-efficiency claim rests on an unverified patient-exclusive split; if serial CXRs of the same patient cross train/test, all reported metrics and label-savings percentages are inflated.","rationale":"The reader's weakest_assumption is exactly the concern I would put first. The central claim is an empirical label-efficiency result, and every quantitative anchor in the abstract depends on a clean test set. The manuscript's own description of the split as image-level and stratified by class label, combined with multiple CXRs per patient, makes patient leakage a concrete possibility rather than a hypothetical. If leakage exists, both the reported active-learning performance and the baselines are inflated, so the claim that the method 'maintains or exceeding diagnostic performance' could be an artifact. I agree with the reader that this warrants a conditional verdict: the result is plausible and in-family with active-learning literature, but the split must be verified. I did not find a more load-bearing concern. The secondary issues—data-count inconsistencies and the use of the test set for acquisition-function selection without significance testing—are real but less central; they would weaken the strength of the comparison among acquisition functions, not the fundamental label-efficiency claim as directly. Since the reader already identified this concern and issued CONDITIONAL, my stress-test pass does not change the verdict.","tokens_in":14563,"tokens_out":2789,"duration_ms":29579,"concrete_test":"Re-run the two decisive pipelines (binary Entropy Sampling and multi-class Mean STD) and the full-data baselines on a patient-exclusive split: randomly assign the 963 patients into 70/10/20 partitions, keep every CXR of a patient in exactly one partition, and stratify by patient-level class proportion to preserve label balance as much as possible. Compare accuracy, AUROC, and AUPRC at the same 15.4%/23.1% training-data fractions and the full-data baselines. If either the active-learning metrics or the baselines shift by more than roughly 2 percentage points, the leakage concern is confirmed and the label-savings claim needs substantial revision. As a cheaper diagnostic first step, report the number of patients whose images appear in both the current train and test sets; any nonzero count demonstrates leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that Entropy Sampling reaches baseline accuracy with 15.4% of training data (binary) and Mean STD with 23.1% (multi-class)—is meaningful only if no patient's images appear in both the training pool and the test set. Section 2.1 describes only a 70/10/20 split, and Section 2.3 says the split was 'stratified across class labels'; neither states that patients were separated. This is critical because the dataset comprises 963 patients with 1–30 CXRs per patient, and multiple images per admission are common. An image-level stratified split will, with high probability, place different CXRs from the same patient into train and test. The model can then exploit patient-specific similarities—shared anatomy, same imaging device, adjacent acquisition times—to inflate test accuracy, AUROC, and AUPRC. The full-data baselines are inflated as well, so the 'maintains or exceeds baseline' comparison is not a safeguard. Worse, the active-learning stopping criterion itself is evaluated on the test set, so the reported 15.4%/23.1% fractions are also leakage-dependent. Internal inconsistencies in the manuscript (2,199 vs 2,319 CXRs in the results text; 'no missing data' in §2.2 vs 120 CXRs lacking demographic information in §3) reinforce the need for careful verification, but the patient-split question is the single load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using deep active learning with MC-dropout uncertainty (a BNN approximation) and a weighted loss to reduce labeling effort for lung disease severity classification from chest X-rays under class imbalance. Using a ResNet50 on a COVID-19 CXR dataset (963 patients, multiple images per patient), the authors compare seven acquisition functions in binary and multi-class settings. The central claim is that Entropy Sampling reaches full-data baseline accuracy with 15.4% of the training data in binary classification and Mean STD reaches it with 23.1% in multi-class classification, while also improving AU ROC/AU PRC in some cases.","tokens_in":14745,"tokens_out":4062,"duration_ms":42569,"significance":"If the quantitative claims were valid, the contribution would be practically significant: it would show that simple uncertainty-based acquisition functions combined with a weighted loss can substantially reduce annotation costs in an imbalanced medical-imaging task. The paper has strengths: it evaluates seven acquisition functions, runs ten random seeds, reports IQRs, includes timing analysis, and analyzes how active sampling changes class distributions. However, the two major methodological concerns below directly affect the validity of the headline label-efficiency numbers, so the central claim is not yet established.","major_comments":[{"comment":"The dataset consists of 963 patients with 1–30 CXRs per patient, but the train/validation/test split is described only as a 70/10/20 split 'stratified across class labels.' There is no statement that images from the same patient were kept within a single split. With serial CXRs, an image-level stratified split will, with high probability, place images from the same patient in both training and test sets, allowing the model to exploit patient-specific factors (anatomy, device, acquisition time) and inflating all reported metrics, including the full-data baselines. The authors must either state that a patient-exclusive split was used or re-run the experiments with such a split and report whether the 15.4%/23.1% figures and the AUROC/AUPRC comparisons change.","section":"§2.1, §2.3"},{"comment":"The active-learning stopping criterion is the full-data baseline accuracy measured on the test set (93.75% and 70.25% in Online Resource Tables 1 and 2). The loop in Fig. 2 continues until 'baseline accuracy has been achieved,' and the results report the percentage of training data at which this test-based baseline is first reached. This means the test set is used to decide when to stop, so the reported label-efficiency percentages are optimistic estimates of a test-set-optimized stopping point. In addition, the 'optimal acquisition function' is selected by comparing the same test-set metrics. The authors should use a held-out validation set for stopping and for selecting acquisition functions, and report test-set performance only once at the end.","section":"§2.4, Fig. 2, §3.1–3.2"}],"minor_comments":[{"comment":"The abstract and §2.1 report 2,319 CXRs, but §3 and Table 1 report 2,199 CXRs. Moreover, the class counts in §3 (1,157 + 836 + 326) sum to 2,319, not 2,199. This internal inconsistency needs clarification.","section":"Abstract vs. §3, Table 1"},{"comment":"§2.2 states 'There was no missing data,' but §3 reports that 120 CXRs lacked demographic information and were omitted from Table 1. Clarify whether the 120 images lack demographic metadata only and whether they are included in the imaging dataset.","section":"§2.2 vs. §3"},{"comment":"Several acquisition-function formulas are incompletely rendered in the provided supplementary text (e.g., the entropy definition, BatchBALD mutual-information expression, and variation-ratio formula). The equations should be typeset correctly.","section":"Online Resource §2"},{"comment":"The dash notation is under-specified: e.g., Variation Ratios multi-class IQR entries such as '[23.11, −]' appear in Tables 2 and 3. The text explains the dash for the median but not for IQR upper bounds; clarify what '−' means in the upper-bound position.","section":"Tables 2 and 3"},{"comment":"Reported differences between acquisition functions are mostly based on medians and IQRs over ten seeds; no formal statistical comparison (e.g., paired tests or confidence intervals) is provided. Given the overlap in many IQRs, the claim that one acquisition function 'outperforms' another should be tempered or supported statistically.","section":"§2.6, §3.3"}],"recommendation":"major_revision","confidential_remarks":"The patient-level split issue is the single most important concern. If the authors cannot demonstrate that no patient appears in both training and test sets, the label-efficiency claims are not interpretable. The test-set-based stopping criterion is also a serious methodological flaw that should be corrected. Both are fixable in a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a competent empirical comparison of standard acquisition functions on a private COVID-19 CXR dataset. The headline—entropy and mean-std reaching baseline accuracy with 15–23% of labels—is believable in direction and consistent with prior active learning work, but the paper does not actually establish it. Section 2.1 describes a 70/10/20 split and Section 2.3 says it was stratified across class labels; nowhere does it say patients were separated. With 963 patients contributing 1–30 CXRs each, an image-level stratified split will almost certainly put multiple images from the same patient in both train and test. The model could then exploit patient-specific features—shared anatomy, same machine, adjacent acquisition times—and inflate every reported metric, including the full-data baselines. The \"matches baseline\" comparison is not a safeguard because the baseline is inflated too. This is the load-bearing issue.\n\nWhat the paper does well: the comparison of seven acquisition functions with ten seeds and IQR reporting is solid. The finding that simple uncertainty methods beat BatchBALD is a legitimate empirical data point. The timing analysis and the class-distribution shift toward minority classes are nice details. The authors also honestly acknowledge inter-reader variability in their limitations.\n\nSoft spots beyond the split: there are small internal inconsistencies (2,199 vs 2,319 CXRs; \"no missing data\" vs 120 CXRs lacking demographic info). Also, the optimal acquisition function is selected using the same test set that defines the baseline, and there are no significance tests across the ten seeds—the per-metric data percentages are read off runs stopped at the accuracy baseline. These are milder but worth addressing.\n\nThe stress-test note on patient leakage is the central concern and it holds up on reading. The paper is not incoherent or dishonest—it just omits a critical methodological detail. The authors likely can rerun with patient-exclusive splits, which would either confirm or kill the specific percentages.\n\nWho is this for? A reader wanting quick qualitative priors about AL on imbalanced CXRs might find it useful, but no one should rely on the 15.4% and 23.1% figures until the split question is resolved. I would not cite the quantitative claims as-is. The paper deserves serious peer review because the research question is important and the flaw is addressable. Send it to reviewers with a clear request to verify patient-level independence and add cross-seed statistical testing.","headline":"A plausible label-efficiency result that is not yet established, because the train/test split may not separate patients who contribute up to 30 serial CXRs each; worth a rigorous referee, not a desk reject.","tokens_in":15401,"tokens_out":1822,"would_cite":false,"duration_ms":19696,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that off-the-shelf uncertainty sampling — not specialized acquisition algorithms — can match full-data chest X-ray severity classification using 15.4% and 23.1% of labeled images.","keywords":["deep active learning","chest X-ray classification","COVID-19 severity grading","class imbalance","Bayesian neural networks","Monte Carlo dropout","acquisition functions","label efficiency"],"falsifier":"Re-run the same active-learning protocol with a strict patient-level split (every image of a given patient placed in the same fold) and with a same-size randomly drawn subset for the random baseline. If the fraction of data needed to reach baseline rises materially above 15.4%/23.1%, or if the full-data baselines drop, the label-efficiency claim rests on train/test leakage or on an unfavorable random comparison.","tokens_in":14309,"feed_emoji":"🩻","tokens_out":12663,"duration_ms":114742,"temperature":0.7,"pith_summary":"The paper sets out to show that active learning — letting the model choose which images get labeled — can cut the radiologist labeling burden for COVID-19 chest X-ray severity grading without losing diagnostic accuracy, even though the severity classes are imbalanced. Using a ResNet50 with Monte Carlo Dropout as a Bayesian uncertainty estimator and a class-weighted loss, the authors iteratively purchased labels with seven acquisition functions. Entropy Sampling matched the full-data baseline (93.7% accuracy) in the normal-vs-diseased binary task using only 15.4% of the training images, and Mean STD matched 70.3% accuracy in the normal/moderate/severe three-class task using 23.1%; both beat more complex functions such as BatchBALD and beat random sampling on the same subset. If right, this means standard, inexpensive uncertainty estimators can replace most manual labeling for this kind of diagnostic model — a practical result because radiologist time is the scarce resource. The authors themselves caution that inter- and intra-reader variability is significant, so the reference standard is a median of three to six independent reads.","feed_headline":"Samplers match full-data CXR accuracy using only 15–23% of labels","feed_subtitle":"Entropy and Mean STD sampling beat costlier methods, cutting radiologist labeling for COVID-19 CXR grading by ~80%.","key_machinery":"The load-bearing mechanism is the active-learning loop, not any single network. A ResNet50 with Monte Carlo Dropout estimates predictive uncertainty by averaging several stochastic forward passes; acquisition functions then score every unlabeled image. The two winners are Entropy Sampling (maximize the entropy of the averaged predictive distribution) and Mean STD (maximize the standard deviation of the MC-dropout probabilities) — both cheap to compute (~41–44 s per batch). A weighted loss with inverse class frequencies counteracts the ~14% normal / 36% moderate / 50% severe imbalance during training, and the loop stops when the model reaches the full-data baseline accuracy. The argument is c","core_discovery":"The central claim: a ResNet50 with Monte Carlo Dropout — a stand-in for a Bayesian neural network — trained under a class-weighted loss, can be driven by simple uncertainty-based acquisition functions to match or exceed the accuracy of a model trained on all labels. On chest X-rays from 963 COVID-19 patients labeled by three to six radiologists (median read as reference), Entropy Sampling reached 93.7% accuracy (AU ROC 0.91) in the binary normal-vs-diseased task using 15.4% of the training set, and hit the AU ROC baseline with just 8.0%. Mean STD reached 70.3% accuracy (AU ROC 0.86) in the three-class normal/moderate/severe task using 23.1% of the data, slightly surpassing the full-data AU R","pith_inferences":["A patient-level re-run — keeping every scan of a patient in one fold — is the test this paper does not report; it would reveal how much of the 15.4%/23.1% label savings survives when serial scans cannot straddle the train–test boundary.","The random baseline is trained on the actively selected subset rather than a same-size random draw, so the reported active-vs-random gap conflates the selection rule with the subset's composition; a true random subset would isolate the rule's contribution.","Because the winning samplers oversample the minority class, some of the label-efficiency gain may be implicit rebalancing rather than informativeness; ablating the weighted loss while holding acquisition fixed would separate the two effects.","The MC-dropout uncertainty used to buy labels could be reused at deployment to flag low-confidence CXRs for radiologist review, turning the acquisition machinery into a reading-worklist prioritizer — an operational consequence the paper motivates but does not develop."],"forward_implications":["Label budgets for severity-grading models can shrink to roughly a sixth to a quarter of the full dataset: a radiology team could annotate about 15–23% of available CXRs and still match a fully labeled model.","Complex acquisition functions lose: BatchBALD and Variation Ratios consistently needed more data and more compute (60–70 s per batch vs. ~41–44 s), so deployment cost and latency favor the simple samplers.","Oversampling of the minority class is built into the winning samplers (35% normal in binary, 30% normal in multi-class vs ~14% prevalence), so active learning doubles as an imbalance remedy without explicit resampling.","The framework transfers: nothing in the pipeline is COVID-specific, and the authors propose applying it to other imbalanced medical imaging datasets and modalities.","Metric-specific deployment: the optimal-sampler table gives clinicians a choice — entropy sampling maximizes F1/AU ROC/sensitivity in binary tasks, mean STD maximizes precision/specificity and multi-class accuracy — so the acquisition function can be matched to the clinical priority."],"supporting_citations":[{"why":"Supplies Monte Carlo Dropout as a Bayesian approximation, the mechanism whose predictive uncertainty drives every acquisition function.","marker":"[20]"},{"why":"BatchBALD, the most complex acquisition baseline the paper must beat; it consistently requires more labeled data and compute.","marker":"[24]"},{"why":"ResNet50, the backbone architecture the active-learning loop trains.","marker":"[19]"},{"why":"The multireader composite severity-score database and reader-variability analysis that this dataset and its median-label reference standard build on.","marker":"[16]"},{"why":"Prior severity-assessment study on a subset of these patients whose dataset and model setup this work expands.","marker":"[15]"},{"why":"Prior COVID-19 deep active learning result (86.6% accuracy with 42% of labels) that frames the label-efficiency comparison.","marker":"[13]"},{"why":"Origin of entropy-based uncertainty sampling, the winning binary acquisition function.","marker":"[22]"},{"why":"Origin of the Mean STD acquisition criterion via Bayesian deep learning uncertainty, the winning multi-class function.","marker":"[25]"}],"fun_headline_variants":["Active learning cuts CXR labeling by 80% with same accuracy","Simple uncertainty sampling beats complex deep active learning","15–23% of labels match full dataset for COVID-19 CXR grading","Entropy sampling tops deep active learning on imbalanced CXRs","Less data, same result: uncertainty sampling on chest X-rays"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper never states that every chest X-ray of a given patient is confined to a single train/validation/test fold, even though patients contribute anywhere from 1 to 30 serial scans; if the split is by individual image rather than by patient, the same patient's scans can appear on both sides of the train–test boundary, which would inflate the reported accuracies and the label-savings percentages.","fun_headline_variants_meta":{"raw":{"variants":["Active learning cuts CXR labeling by 80% with same accuracy","Simple uncertainty sampling beats complex deep active learning","15–23% of labels match full dataset for COVID-19 CXR grading","Entropy sampling tops deep active learning on imbalanced CXRs","Less data, same result: uncertainty sampling on chest X-rays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000639,"raw_usage":{"total_tokens":2851,"prompt_tokens":889,"completion_tokens":1962,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":1875}},"tokens_in":633,"tokens_out":1962,"duration_ms":14279,"temperature":1.0,"reasoning_tokens":1875,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:26:26.176637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same active-learning protocol with a strict patient-level split (every image of a given patient placed in the same fold) and with a same-size randomly drawn subset for the random baseline. If the fraction of data needed to reach baseline rises materially above 15.4%/23.1%, or if the full-data baselines drop, the label-efficiency claim rests on train/test leakage or on an unfavorable random comparison.","supporting_citations":[{"cited_title":"van Assen, M","cited_arxiv_id":null,"evidence_quote":"The multireader composite severity-score database and reader-variability analysis that this dataset and its median-label reference standard build on."},{"cited_title":"Zandehshahvar, M","cited_arxiv_id":null,"evidence_quote":"Prior severity-assessment study on a subset of these patients whose dataset and model setup this work expands."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior COVID-19 deep active learning result (86.6% accuracy with 42% of labels) that frames the label-efficiency comparison."},{"cited_title":"Schohn, D","cited_arxiv_id":null,"evidence_quote":"Origin of entropy-based uncertainty sampling, the winning binary acquisition function."}],"review_version":1}