{"id":"147fa6b1-1759-44de-8e90-08a8ac938f1b","arxiv_id":"1908.10508","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"O-MedAL combines entropy-based candidate filtering with centroid-distance sampling and incremental training to reduce labeling effort and computation for medical image classification.","lead":"The authors extend their MedAL active learning framework into an online version, O-MedAL, that selects medical images by uncertainty plus distance in feature space and updates the model incrementally. The paper claims a 6.3% accuracy gain over a baseline and reaching baseline accuracy with about a quarter of the labels, which matters because expert medical labeling is costly.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 is false: maximizing average Euclidean distance to labeled examples is not rank-equivalent to maximizing Euclidean distance to their centroid, so O-MedAL's advertised sampling rule and O(M) exact speedup are not established.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw. The paper advertises as its first contribution a sampling rule that maximizes average distance to all labeled examples, and Theorem 1 is the only justification for replacing that rule with a centroid-distance computation. The proof's step removing the outer square root before summing over the labeled set is invalid: the order of sums of square roots is not preserved by squaring the summands. The one-dimensional counterexample shows the rankings genuinely reverse, so the implemented sampler is not the proposed sampler. This is independently sufficient to reject the paper's central methodological claim, even before considering the experimental concerns about test-set selection and the changed Messidor split. Those concerns are real and further support rejection, but the theorem flaw is decisive and does not require re-running experiments to verify. The reader's REJECT verdict remains appropriate; no adjustment is needed.","tokens_in":22405,"tokens_out":8812,"duration_ms":94531,"concrete_test":"Recompute the two scores from Eq. (2) and Eq. (4) on the one-dimensional set with labeled embeddings {0,10,10} and candidates x=6 and x=8. If the argmax of average Euclidean distance (x=6) differs from the argmax of distance to the centroid (x=8), Theorem 1 is false. This is a direct arithmetic check requiring no training or code changes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central methodological claim is that Eq. (2), s(X) = 1/N * sum_i d(f(X_i), f(X)) with d the Euclidean distance, can be maximized in O(M) by maximizing d(f(X), Y), where Y is the centroid of labeled embeddings. Theorem 1 is supposed to prove rank preservation. The proof breaks at the transition from Eq. (5)/(6) to Eq. (7): it replaces each term sqrt(sum_j (f(X_i)^(j) - f(X)^(j))^2) by its squared value and then sums over i. For nonnegative scalars, z -> z^2 does not preserve the order of sums; rank(sum_i sqrt(z_i)) is not generally equal to rank(sum_i z_i). A concrete one-dimensional counterexample: labeled embeddings {0,10,10}, candidates x=6 and x=8. Eq. (2) gives s(6)=14/3 > s(8)=4, while the centroid distances are |6-20/3|=2/3 < |8-20/3|=4/3. The rankings are reversed. Consequently, a sampler using centroid distance does not implement the average-distance rule described in the abstract; it implements a different, unproven heuristic. The equivalence would hold for squared Euclidean distances, but the paper explicitly selects Euclidean distance. The claimed quadratic-to-linear speedup, and the statement that Theorem 1 makes O-MedAL tractable, rest on a false identity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the authors' MedAL active learning framework to an online setting, O-MedAL, which keeps model weights between iterations, trains on newly labeled examples plus a random subset of previously labeled data, and selects examples with a distance-based sampling rule. The central claims are: (i) maximizing the average Euclidean distance from an unlabeled example to all labeled examples can be replaced, without changing the ranking, by maximizing the Euclidean distance to the centroid of the labeled examples, giving an O(M) sampler; (ii) online training increases accuracy over a ResNet18 baseline by 6.30% while using only 25.29% of the labeled data to reach baseline accuracy; and (iii) these gains generalize across binary/multi-class and balanced/imbalanced medical image tasks. The experiments are conducted on Messidor, BACH breast cancer, and ISIC skin lesion datasets.","tokens_in":22732,"tokens_out":7193,"duration_ms":75224,"significance":"If the central claims were valid, the linear-time exact reformulation of the sampling rule would be a practically useful contribution to active learning for deep medical image analysis, and the reported label-efficiency gains would be of interest to the clinical machine-learning community. Strengths include the use of three public medical datasets and the release of public code. However, the main theoretical equivalence is not proven as stated, and the headline results appear to be obtained under evaluation protocols that do not support them. The significance of the contributions is therefore not established by this manuscript.","major_comments":[{"comment":"The proof of rank preservation is invalid. The text moves from the scalar property that squaring a nonnegative quantity preserves its order to the claim that the order of sums of square roots is preserved after squaring each term before summing; this is not true. A one-dimensional counterexample is labeled embeddings {0,10,10} with candidates x=6 and x=8. The average Euclidean distances are s(6)=14/3 and s(8)=4, so x=6 is the maximizer, while the distances to the centroid 20/3 are 2/3 and 4/3, so x=8 is the maximizer. The rankings are reversed. Consequently, the O(M) exact evaluation of the average-distance rule in Eq. (2) is not established, and the sampler described in Figure 1, which selects examples farthest from the centroid, is a different heuristic from the one in Eq. (2). The equivalence would hold for squared Euclidean distances, but the paper explicitly selects Euclidean distance.","section":"Proposed Method, Theorem 1 and Eqs. (5)-(9)"},{"comment":"The hyperparameters p and the patience values appear to be selected on the same Messidor test set that produces the headline claims. The text states that the p=0.125 and p=0.875 models were further evaluated with patience values of 5, 10, and 20 'as these two values of p were best performing,' and no validation split for O-MedAL is described. Because the reported table gives maximum test accuracy, the headline 92.02%, 25.29%, and 6.30% figures are likely optimistic selections on the test set. This is a circular evaluation of the main accuracy claims.","section":"O-MedAL Evaluation, Figure 5A"},{"comment":"The paper states that for Messidor the 80/20 split is 're-computed each time a model is trained.' This means the test set is not fixed across the compared models or across iterations, so the test accuracies in Figures 4-5 and Table 5A are not measured on a common test set. The comparison between O-MedAL, MedAL, and the ResNet18 baseline is therefore not statistically clean, and the reported +6.30% improvement over the baseline is undermined.","section":"O-MedAL Implementation Details, Messidor split"},{"comment":"The comparison between O-MedAL and MedAL is confounded by a change of base architecture: O-MedAL uses ResNet18 while MedAL uses InceptionV3, with the paper justifying the change on the grounds that ResNet18 is smaller. Claims that O-MedAL is 'more accurate than MedAL' and that it reduces wall time by 4.5-6 times over MedAL therefore conflate the effect of the online active learning method with the effect of the architecture change. The comparison would need to hold the base architecture fixed or explicitly model architecture as a separate factor.","section":"O-MedAL Implementation Details and Table 5A"}],"minor_comments":[{"comment":"The abstract and Eq. (2) describe the sampler as maximizing the average distance to all training examples, while Figure 1's caption and the implementation section describe selecting examples farthest from the centroid. These are different rules; the paper should state explicitly which rule was actually used in the experiments.","section":"Abstract and Figure 1"},{"comment":"The wall-time comparisons are not directly interpretable because O-MedAL models are trained with up to 150 epochs per iteration and validation every epoch, while the ResNet18 baseline is trained for a fixed 80 epochs; the authors partially acknowledge this, but the 'Wall Time' column should not be read as a clean efficiency comparison.","section":"Table 5A, wall time"},{"comment":"Equation (10) is introduced as the cumulative number of example images used for backpropagation, but the following sentence says the cumulative number of weight updates is N_t/b; using the same symbol N_t for both images and updates is confusing and should be clarified.","section":"Eq. (10)"},{"comment":"The phrase 'we believe between 2005 and 2006' for Messidor collection dates is informal; it should be replaced with a citation or removed.","section":"Dataset Description, Messidor"},{"comment":"The statement that 'assuming overfitting does not decrease test accuracy' is an unverified assumption used to interpret the MedAL curve; it should be explicitly flagged as an assumption rather than implied.","section":"Figure 4 discussion"}],"recommendation":"reject","confidential_remarks":"The paper's own Figure 1 caption already implements the centroid rule, which is internally inconsistent with Eq. (2) and Theorem 1. The false theorem is load-bearing for the advertised linear-time speedup and for the identity of the proposed sampling method, and the experimental protocol has test-set selection and split instability issues. A resubmission would need to reformulate the method, fix the evaluation protocol, and rerun the experiments; these are not local revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. First, the headline result—that you can replace the average Euclidean distance to all labeled examples with the distance to their centroid and get the same ranking in O(M)—is false. The proof removes square roots before summing, and that operation does not preserve order. A simple one-dimensional counterexample (labeled {0,10,10}, candidates 6 and 8) reverses the ranking. So the claimed quadratic-to-linear speedup is not established. Second, the actual algorithm in the experiments uses centroid distance, not average distance, so the abstract overstates what the method does.\n\nWhat is genuinely new here is the combination: entropy pruning to a candidate set, then distance-to-centroid sampling in a learned feature space, plus online fine-tuning on only the newly labeled examples and a random subset of previously labeled ones. That combination is not verbatim in the cited literature, and the authors ship code. They also test on three public medical datasets (Messidor, BACH, ISIC) and show consistent gains over random and entropy-only baselines, with some attention to class imbalance.\n\nThe soft spots beyond the false theorem are mostly in the experiments. MedAL uses Inception v3 while O-MedAL uses ResNet18, so the comparison between the two is not clean. The Messidor split is changed (stratified across hospitals, duplicates removed), so previous numbers aren't directly comparable. The choice of p and patience appears to be made on the same test set that produces the headline 6.30% number, and there are no error bars. The '25% of labels to reach baseline accuracy' claim comes from a single selected run.\n\nThe paper is not a waste of time. The empirical pattern suggests that entropy plus diversity sampling and online training can be label-efficient for medical images, and that result is worth testing independently. But as written, the main theoretical justification is wrong, and the experiments are too loose to support the quantitative claims. I would send it to a mathematically careful referee only if the authors are willing to fix the theorem (or replace it with a claim about squared distances) and tighten the evaluation. Otherwise, I'd let it go.","headline":"The paper's central speedup theorem is false, so the headline claim is unsupported, though the underlying active-learning combination might be worth a careful second look.","tokens_in":23288,"tokens_out":4236,"would_cite":false,"duration_ms":42943,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An online active-learning method reaches its baseline accuracy with about a quarter of the labels by querying images farthest from labeled examples in feature space.","keywords":["active learning","online learning","deep learning","medical image analysis","feature embeddings","uncertainty sampling","class imbalance","diabetic retinopathy"],"falsifier":"Compute the average Euclidean distance and the centroid distance for a one-dimensional example with labeled points $\\{0,1,10\\}$ and candidates $\\{2,5\\}$. The average-distance score ranks 5 above 2 ($14/3 > 11/3$), while the centroid-distance score ranks 2 above 5 ($5/3 > 4/3$). This is a direct counterexample to the claimed rank preservation for plain Euclidean distance; if the actual sampler uses plain Euclidean distance, the $O(M)$ shortcut can select a different image than the stated criterion.","tokens_in":22217,"feed_emoji":"🩺","tokens_out":9696,"duration_ms":91829,"temperature":0.7,"pith_summary":"O-MedAL is an online active learning method for medical image classification. It claims that an unlabeled image is worth labeling when it is far in the network's learned feature space from everything already labeled, and that this criterion can be queried cheaply: selecting the example farthest from the centroid of the labeled examples preserves the ranking of the average-distance score. The paper reports that this lets a ResNet-18 reach its fully-supervised baseline accuracy with only about 25% of the dataset labeled, improve on that baseline by 6.30% when more labels are used, and cut backpropagated images by up to roughly 68%. The practical stake is that medical experts labeling images can interact with the model in near-real-time instead of waiting for full retraining between batches.","feed_headline":"Active learning hits baseline with a quarter of the labels","feed_subtitle":"Picks maximally distant images in feature space, so doctors label less and the model still beats its baseline.","key_machinery":"The central object is the score $s(X) = \\frac{1}{N}\\sum_{i=1}^N d(f(X_i), f(X))$, the average Euclidean distance from an unlabeled image's feature embedding to all labeled embeddings. The linchpin is Theorem 1, which replaces the pairwise sums by a single distance to the centroid $Y=\\frac{1}{N}\\sum_i f(X_i)$: the argmax over candidates is claimed to be unchanged, turning an $O(MN)$ pairwise computation into $O(M)$ distances. A second mechanism is the online training schedule: the weights are never reset, and each iteration trains on the newly labeled batch plus a $p$-fraction of previously labeled examples, so the model improves incrementally rather than repeatedly relearning from scratch.","core_discovery":"The paper claims that active learning for deep networks can be made both label-efficient and computationally cheap by combining predictive-entropy screening with a distance criterion in feature space. At each active-learning iteration, the top-$M$ unlabeled images by predictive entropy form a candidate set $C_M$; among those, O-MedAL selects the image maximizing the average Euclidean distance to all labeled examples in the embedding $f(\\cdot)$. Theorem 1 states that this argmax can be evaluated as the distance to the centroid of the labeled embeddings, reducing the per-iteration cost from $O(MN)$ to $O(M)$ distance evaluations. The online component then fine-tunes the same network on the newly labeled images plus a random subset of previously labeled images, without resetting weights between iterations. On Messidor diabetic retinopathy data, O-MedAL with $p=0.875$ and early stopping reaches 92.02% test accuracy versus an 85.71% ResNet-18 baseline, reaches the baseline accuracy with 25.29% of labels, and the most computation-efficient configuration processes about 68% fewer backpropagated images than the baseline.","pith_inferences":["As an extension beyond the paper, applying the same farthest-from-centroid sampling to non-medical image datasets would test whether the method is a general active-learning recipe or specific to medical imaging.","The observation that subsets of labels can beat the fully supervised baseline suggests O-MedAL could be turned into a data-subset selection tool for deep learning generally, identifying redundant or harmful training examples.","Because the model is updated online without resets, the framework could also serve streaming or continuously arriving data, where new classes or distributions arrive over time; that setting is not the paper's focus.","Batch size can shrink toward single-image queries without quadratic cost, which in practice would let an annotator label one image, see the model update, and immediately receive the next query."],"forward_implications":["If the centroid-ranking claim holds, the sampling step scales to large unlabeled pools, and batch size can be reduced toward single-image queries without quadratic cost, enabling near-real-time human-in-the-loop annotation.","If O-MedAL's empirical results generalize, a medical imaging team can obtain a network's fully supervised accuracy after labeling a quarter of the data, with a human expert spending far less time on annotation.","The online fine-tuning schedule means the model is updated incrementally, so annotators do not wait between active-learning iterations; this is the practical condition for using active learning in a clinical workflow.","Consistent gains over uncertainty-only and random sampling on balanced, imbalanced, binary, and multi-class medical datasets suggest the method is not specialized to one disease or imaging modality.","Because selected subsets can outperform the fully supervised baseline, the method also points toward identifying training examples that are redundant or unhelpful, not merely toward spending fewer labels."],"supporting_citations":[{"why":"Defines pool-based active learning, the setting O-MedAL extends, and surveys sampling strategies.","marker":"Settles, 2008"},{"why":"Introduces MedAL and the empirical choice of feature layer and Euclidean distance that O-MedAL builds on.","marker":"Smailagic et al., 2018"},{"why":"Shows continuous fine-tuning at each active-learning iteration in biomedical imaging, which O-MedAL generalizes with a random subset of previous labels.","marker":"Zhou et al., 2017"},{"why":"Supplies the deep Bayesian active learning comparison used in the skin-lesion AUC experiment.","marker":"Gal et al., 2017"},{"why":"Provides ResNet-18, the baseline network whose accuracy O-MedAL claims to improve.","marker":"He et al., 2015"},{"why":"Supplies the ISIC skin lesion dataset used for imbalanced binary melanoma classification.","marker":"Gutman et al., 2016"},{"why":"Supplies the BACH breast cancer histology dataset used for balanced multi-class grading.","marker":"Aresta et al., 2019"},{"why":"Provides evidence that deep networks fit random labels, motivating the departure from uncertainty-only sampling.","marker":"Zhang et al., 2016"}],"fun_headline_variants":["Quarter labels, baseline accuracy in medical imaging","O-MedAL: 25% labels, 67% fewer backprop images","Active learning lifts accuracy 6.3% on retina data","Entropy screening plus max-distance picks key images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that distance in the learned feature space measures how much an unlabeled image will improve the classifier, and that ranking examples by average Euclidean distance to all labeled examples is the same as ranking them by distance to the centroid; the proof establishes this exactly for squared Euclidean distances, while the paper states the score with plain Euclidean distance.","fun_headline_variants_meta":{"raw":{"variants":["Quarter labels, baseline accuracy in medical imaging","O-MedAL: 25% labels, 67% fewer backprop images","Active learning lifts accuracy 6.3% on retina data","Entropy screening plus max-distance picks key images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000354,"raw_usage":{"total_tokens":1896,"prompt_tokens":884,"completion_tokens":1012,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":944}},"tokens_in":500,"tokens_out":1012,"duration_ms":10499,"temperature":1.0,"reasoning_tokens":944,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:42:36.752677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the average Euclidean distance and the centroid distance for a one-dimensional example with labeled points $\\{0,1,10\\}$ and candidates $\\{2,5\\}$. The average-distance score ranks 5 above 2 ($14/3 > 11/3$), while the centroid-distance score ranks 2 above 5 ($5/3 > 4/3$). This is a direct counterexample to the claimed rank preservation for plain Euclidean distance; if the actual sampler uses plain Euclidean distance, the $O(M)$ shortcut can select a different image than the stated criterion.","supporting_citations":[{"cited_title":"(2008) Curious machines: Active learning with structured instances","cited_arxiv_id":null,"evidence_quote":"Defines pool-based active learning, the setting O-MedAL extends, and surveys sampling strategies."},{"cited_title":"Y., Walawalkar, D., Khandelwal, K., Galdran, A., Mirshekari, M., Fagert, J., Xu, S., Zhang, P","cited_arxiv_id":null,"evidence_quote":"Introduces MedAL and the empirical choice of feature layer and Euclidean distance that O-MedAL builds on."},{"cited_title":"and Liang, J","cited_arxiv_id":null,"evidence_quote":"Shows continuous fine-tuning at each active-learning iteration in biomedical imaging, which O-MedAL generalizes with a random subset of previous labels."},{"cited_title":"and Ghahramani, Z","cited_arxiv_id":null,"evidence_quote":"Supplies the deep Bayesian active learning comparison used in the skin-lesion AUC experiment."},{"cited_title":"S., Safwan, M., Alex, V., Marami, B., Prastawa, M., Chan, M., Donovan, M., Fernandez, G., Zeineh, J., Kohl, M., Walz, C., Ludwig, F., Braunewell, S., Baust, M., Vu, Q","cited_arxiv_id":null,"evidence_quote":"Supplies the BACH breast cancer histology dataset used for balanced multi-class grading."}],"review_version":1}