{"id":"5744be20-ee27-4551-8bdf-06bff72799fd","arxiv_id":"2507.05815","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A human-AI framework trains segmentation models from binary 'better or worse' feedback, with a clicking agent and feature-based label propagation, achieving improving Dice scores across rounds on three public datasets.","lead":"This paper presents a method to train medical image segmentation models using only simple 'better or worse' feedback from a human instead of requiring detailed pixel-by-pixel annotation. The key idea is an AI clicking agent that learns where to click and what to label, guided by those binary preferences, to generate training masks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Dice-based oracle in Section 3.1 is the sole learning signal, yet no evidence shows real human 'better/worse' judgments carry the same information; the claim of eliminating manual annotation is therefore not yet demonstrated.","rationale":"The reader's weakest_assumption identifies precisely the load-bearing concern: the simulated Dice oracle may not represent real human 'better or worse' judgments. My stress-test agrees and sharpens it: the oracle is not just a noise-free stand-in for human feedback; it converts dense pixel-level ground truth into a scalar reward that exactly ranks masks by Dice. This gives the clicking agent a globally consistent training signal that a human, comparing two imperfect masks, may not reliably provide. The paper's own Limitations section explicitly concedes this, so the argument is internally honest but externally unverified. The strongest claim—'competitive performance using only binary preference feedback'—rests entirely on this oracle assumption. A human study, or at least a noisy-reward ablation, would be needed to support the claimed elimination of manual annotation. Since the reader already returned CONDITIONAL with moderate confidence, and since the paper's conclusion is appropriately hedged in Section 4, I do not see a reason to move the verdict: the concern is real but does not invalidate the proof-of-concept; it narrows what can be claimed.","tokens_in":6305,"tokens_out":3119,"duration_ms":41258,"concrete_test":"Run a small human-in-the-loop pilot on one dataset (e.g., 100 BUSI or ISIC images) with 2–3 clinicians: for each click update, show the previous and proposed segmentation masks without any ground truth, ask only 'better or worse', and use those answers as rewards in Algorithm 1. Compare the final Dice curves over 5 rounds against the oracle-simulated run. If the human-driven run loses more than 5 DSC points or fails to show monotonic improvement across rounds, the oracle assumption is the load-bearing gap in the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a segmentation network can be trained to competitive performance using only binary 'better or worse' feedback, eliminating manual pixel-level or region-level annotation. The only support for this claim is the simulated oracle described in Section 3.1: the ground-truth mask is used to compute Dice before and after each click, and the sign of the Dice change is converted into a +1/−1 reward. This oracle is not merely a convenient simulation; it is a dense, globally consistent, pixel-level evaluator. A real clinician comparing two segmentations may judge based on clinical relevance, boundary plausibility, or local structure, and their binary judgment can be noisy, inconsistent, or only weakly correlated with Dice. The REINFORCE-trained clicking agent receives a reward that, in the experiments, is exactly derived from ground-truth overlap; a human-provided reward would have different and likely much noisier statistics. The paper itself acknowledges this in Section 4, noting that the oracle 'oversimplifies real-world expert judgments.' That limitation is exactly where the central claim hinges: without evidence that human binary preference provides a usable learning signal, the demonstrated improvement across rounds only shows that a policy can be trained when an omniscient evaluator supplies dense ground-truth-derived rewards. The phrase 'eliminating the need for any manual pixel-level and/or region-level annotation' is therefore too strong relative to the evidence. The concern is not internal inconsistency; the method works as described under the oracle. The concern is external validity: the mechanism that makes the framework succeed in the paper may disappear when the oracle is replaced by a human.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a human-AI collaborative framework for medical image segmentation that aims to train a segmentation network using only binary \"better or worse\" expert feedback, thereby eliminating manual pixel-level or region-level annotation. The framework combines a DINOv2+LoRA feature extractor, feature-similarity-based label propagation, a REINFORCE-trained clicking agent, and multi-round fine-tuning of a PVT-based segmentation network (HSNet) on pseudo-labels. Experiments on BUSI, ISIC 2018, and combined polyp datasets report consistent improvement in Dice and HD95 over five annotation rounds. However, the feedback is simulated by an oracle that computes Dice improvement from ground-truth masks, the evaluation protocol is not clearly described, no baseline comparisons are provided, and the paper's own Section 4 acknowledges that the oracle oversimplifies real-world expert judgments.","tokens_in":6635,"tokens_out":5524,"duration_ms":60510,"significance":"If validated with real human feedback and a rigorous evaluation protocol, the core idea of training a clicking agent from binary preference feedback would be a valuable contribution to reducing annotation burden in medical imaging. The integration of a foundation model with preference-driven pseudo-labeling is timely and the multi-round bootstrapping concept is interesting. The paper is also transparent about the oracle limitation, which is a strength. However, the current experimental evidence is not yet sufficient to support the central claims of achieving \"competitive segmentation performance\" or eliminating the need for manual annotation. The absence of held-out evaluation, baselines, and ablations, together with the reliance on a ground-truth-based oracle, make the quantitative results hard to interpret.","major_comments":[{"comment":"The simulated oracle that generates the reward is computed directly from ground-truth Dice: the reward is +1 if the new mask's Dice against ground truth increases, and −1 otherwise. This is the sole learning signal for the clicking agent and the same metric is used for evaluation. The paper acknowledges in Section 4 that this \"oversimplifies real-world expert judgments,\" but that acknowledgment actually highlights the load-bearing weakness: the central claim is that binary \"better or worse\" feedback from a human expert suffices, yet the only evidence is with an omniscient, dense, pixel-level evaluator. A real clinician's binary judgment may be noisy, locally inconsistent, or correlated with clinical relevance rather than global Dice. The authors should provide a real human study or a carefully justified noise model for human feedback that preserves the qualitative improvement over rounds. Without this, the claim of \"eliminating the need for any manual pixel-level and/or region-level annotation\" is not demonstrated.","section":"§3.1, Algorithm 1 line 12; §4"},{"comment":"The evaluation protocol is not described. For BUSI the text says \"we utilize the complete dataset for our experiments\" and for ISIC and polyp datasets no train/test split is mentioned. If the reported Dice and HD95 values are computed on the same images that were used to generate pseudo-labels and fine-tune the segmentation network, then the results reflect training-set fit, not generalization, and the multi-round improvement could be a trivial consequence of fitting the training data. The authors must specify the exact train/validation/test protocol (e.g., a held-out test set or cross-validation) and evaluate the final model on images not used for pseudo-label generation or model fine-tuning.","section":"§3.2, Fig. 2 and Fig. 3"},{"comment":"The claim of \"competitive segmentation performance\" is unsupported because no baseline comparisons are reported. There is no comparison to a fully supervised model trained on the original ground-truth masks, to an interactive segmentation method such as SAM with clicks, to a random-click policy, or to existing weakly supervised approaches. Without such baselines, the reader cannot judge whether the proposed framework actually reaches competitive accuracy or whether the improvement over rounds is meaningful. Additionally, REINFORCE is stochastic, but no standard deviations over multiple runs or statistical significance tests are provided; the \"consistent improvement\" in Fig. 2 could be within one standard deviation and would be more persuasive with error bars.","section":"§3.2, Fig. 2"},{"comment":"Several key hyperparameters are fixed without sensitivity analysis or ablation: the number of rounds R=5, interaction steps per image T=5, similarity threshold τ=0.8, top-K filtering percentage, temperature parameter, and REINFORCE learning rate. In particular, the label-propagation threshold τ and the number of clicks T directly determine the quality of the pseudo-labels and the difficulty of the clicking agent's task. Ablation studies are needed to show that the framework's performance is not an artifact of a particular hyperparameter choice, and to isolate the contribution of each component (adaptable foundation model, clicking agent, label propagation, multi-round training). Without these, it is unclear how robust the method is across datasets and settings.","section":"Algorithm 1"}],"minor_comments":[{"comment":"The phrase \"especially for complex 3D anatomies or subtle pathologies\" is duplicated in the second paragraph of the introduction.","section":"§1"},{"comment":"The captions of Figures 2 and 3 list the dataset order as \"polyp (left), skin lesion (middle), and ultrasound (right)\", while the text in Section 3.2 states the order as \"ultrasound (left), skin lesion (middle), and polyp (right)\". Please align the text and figure captions.","section":"Fig. 2 and Fig. 3 captions"},{"comment":"The algorithm uses \"DINO-based label propagation\" while the text discusses DINOv2; please unify the terminology to avoid confusion.","section":"§2.2, Algorithm 1"},{"comment":"The temperature parameter for action sampling and the top-K filtering percentage are mentioned but never specified. Provide the exact values or state that they are set to defaults and list those defaults.","section":"§3.1"},{"comment":"Please report the training details for the main segmentation network HSNet, including optimizer, learning rate, number of epochs, and the LoRA rank for the adaptable foundation model, to enable reproducibility.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a promising and novel idea, but the experimental validation is currently too weak to support the strong claims in the abstract and introduction. The most severe issue is the unclear evaluation protocol: if the reported metrics are computed on the training set, the results are uninterpretable. The reliance on a ground-truth-based oracle is acknowledged, but the implications for the central claim are downplayed. I recommend major revision rather than rejection because the pipeline is coherent and could be credibly validated with a proper held-out test split, baselines, ablations, and either a human study or a realistic feedback-noise model. The paper would also benefit from code release to support reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper shows you can train a medical segmentation network from binary 'better or worse' feedback, as long as the feedback is supplied by a ground-truth Dice oracle. The idea is genuinely new, and the multi-round bootstrapping works on three public datasets. The catch is that the evaluation never replaces that oracle with a human, and there are no baselines, so 'competitive performance' is asserted rather than shown.\n\nWhat's new: the combination of a preference-driven clicking agent with DINOv2 feature propagation and pseudo-label training is a real contribution. The paper is clearly written, and Algorithm 1 makes the method reproducible in principle. The consistent improvement over five rounds on polyp, skin lesion, and ultrasound data is a legitimate proof of concept.\n\nThe soft spots are mostly in the experimental design. First, the oracle in Section 3.1 computes Dice before and after each click and converts the sign into a reward. That is dense, pixel-level supervision, just with a threshold. The claim that this eliminates manual annotation is too strong; it replaces pixel-level annotation with ground-truth-derived reward, which in practice would come from a human. The paper acknowledges this in the limitations, but the abstract and introduction don't soften the claim accordingly.\n\nSecond, no comparisons. No random-click baseline, no scribble/box supervision baseline, no full-supervision upper bound, no existing interactive method. 'Competitive' cannot be evaluated without those. Third, no ablations: does the learned clicking agent beat a fixed point-sampling strategy? Does LoRA adaptation actually matter? Fourth, no error bars or multiple seeds; the figures show trends but not variance.\n\nIf the authors add baselines, ablations, and at least a small human pilot or a simulation with noisy feedback, the paper would make a strong case. As it stands, it is a promising proof of concept that overclaims its evidence. I would send it to review, but with a clear expectation of major revision.\n\nRead it if you care about preference-based learning or interactive segmentation; it is worth twenty minutes.","headline":"Promising preference-based segmentation framework, but the Dice oracle and missing baselines leave the central claim undemonstrated.","tokens_in":7103,"tokens_out":2226,"would_cite":false,"duration_ms":25686,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A medical image segmentation network can be trained to competitive accuracy using only binary 'better or worse' expert feedback on AI-generated masks, with no manual pixel-level or region-level annotation.","keywords":["medical image segmentation","preference learning","better or worse feedback","clicking agent","pseudo-labeling","foundation model","reinforcement learning","label propagation"],"falsifier":"Run the same five-round protocol with clinicians giving real better/worse feedback on the same three datasets and compare final Dice and HD95 with the simulated-oracle results; the central claim is falsified if real feedback fails to produce round-over-round improvement or falls well short of the oracle-guided performance.","tokens_in":6145,"feed_emoji":"🩺","tokens_out":8183,"duration_ms":84387,"temperature":0.7,"pith_summary":"Medical-image segmentation normally depends on experts tracing pixel-level ground truth, which is slow and costly. This paper claims that a segmentation network can instead be trained from a much weaker signal: an expert simply states whether a new AI-generated mask is better or worse than the previous one. The authors build a loop in which a clicking agent, rewarded by these binary comparisons, learns where to place corrective clicks, a foundation model propagates each click into a dense pseudo-mask, and a standard segmentation network is fine-tuned on the resulting pseudo-labels over several rounds. On three public datasets—polyp, skin lesion, and breast ultrasound—the reported performance improves round over round and reaches competitive Dice scores. If the claim holds, the human's role in annotation shifts from careful pixel drawing to high-level comparison.","feed_headline":"Better-or-worse feedback alone trains a segmentation AI","feed_subtitle":"An expert compares two masks; the system learns where to click and reaches competitive accuracy on three medical datasets.","key_machinery":"The load-bearing machinery is a preference-driven clicking loop. A lightweight U-Net policy, trained by the REINFORCE policy-gradient algorithm, observes a state made of the image plus the current predicted mask and outputs a softmax over click locations; the binary better/worse reward is the only learning signal. Each accepted click is converted from a single point to a dense pseudo-label by comparing patch features from a DINOv2-based foundation model (adapted with LoRA) under a cosine-similarity threshold, and only masks judged better replace the current one. Pseudo-labels from all images are filtered by quality and used to fine-tune the main segmentation network, which then seeds the next round's clicks, creating a bootstrapping loop.","core_discovery":"The paper's central claim is that binary preference feedback—'better' or 'worse' relative to the previous mask—is sufficient supervision to train a medical image segmentation model without any manual pixel-level or region-level annotation. The mechanism works by treating each corrective click as a reinforcement-learning action: the clicking agent samples a coordinate, the click is propagated into a dense mask through cosine similarity in a LoRA-adapted DINOv2 feature space, and the expert's binary judgment is returned as a +1/−1 reward. The updated mask is kept only when the feedback is positive, and the accumulated pseudo-masks are used to fine-tune the segmentation network, whose improved outputs initialize the next round. Across the BUSI ultrasound, ISIC 2018 skin lesion, and combined Kvasir-SEG/CVC-ClinicDB polyp datasets, the authors report consistent gains in Dice and HD95 over five annotation rounds, with final distributions concentrated around high Dice scores.","pith_inferences":["Editorial inference: real clinician feedback will be noisier than a Dice oracle, so practical deployment may need reward smoothing, repeated comparisons, or confidence thresholds to keep the loop stable.","Editorial inference: the same binary-comparison supervision could extend beyond segmentation to detection, registration, or image enhancement, wherever a global quality judgment is easier to give than a spatial correction.","Editorial inference: the decisive next experiment is a human study in which clinicians supply the better/worse judgments on the same datasets, to see whether the round-over-round gains survive real-world variability."],"forward_implications":["An expert can train or refine a segmenter just by comparing masks, avoiding pixel tracing and reducing cognitive load.","The clicking agent's policy improves across rounds, so later rounds need fewer corrective interactions for the same quality.","Because pseudo-labels are filtered by quality before training, a fraction of bad interactions does not derail the final model.","The framework's structure is not tied to one imaging modality; with a suitable foundation model it should transfer to other medical segmentation tasks."],"supporting_citations":[{"why":"Cited as the DINOv2 feature extractor; its patch-level features carry cosine-similarity label propagation.","marker":"[10]"},{"why":"Low-rank adaptation is used to fine-tune the feature extractor on pseudo-labeled data.","marker":"[7]"},{"why":"REINFORCE provides the policy-gradient update that trains the clicking agent from binary rewards.","marker":"[20]"},{"why":"HSNet is the segmentation architecture trained on pseudo-labels and evaluated across rounds.","marker":"[21]"},{"why":"BUSI supplies the breast ultrasound images and ground truth used to simulate better/worse feedback.","marker":"[1]"},{"why":"The ISIC 2018 training set supplies the skin lesion images and masks used for evaluation.","marker":"[17]"},{"why":"Kvasir-SEG supplies polyp images and masks that form part of the combined polyp dataset.","marker":"[9]"},{"why":"CVC-ClinicDB supplies the other half of the combined polyp dataset.","marker":"[2]"}],"fun_headline_variants":["Binary 'better/worse' clicks train medical segmentation","'Better or worse' feedback replaces manual annotation","Segmentation AI trains from just 'better or worse' input","No pixel labels: AI learns from binary preference","Expert says 'better/worse' and the AI learns to segment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simulated oracle—'better' when ground-truth Dice rises and 'worse' when it falls—behaves like a real expert's judgment; if human better/worse decisions diverge from Dice changes, the clicking agent's reward signal may be misleading.","fun_headline_variants_meta":{"raw":{"variants":["Binary 'better/worse' clicks train medical segmentation","'Better or worse' feedback replaces manual annotation","Segmentation AI trains from just 'better or worse' input","No pixel labels: AI learns from binary preference","Expert says 'better/worse' and the AI learns to segment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1441,"prompt_tokens":932,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":431}},"tokens_in":548,"tokens_out":509,"duration_ms":5016,"temperature":1.0,"reasoning_tokens":431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:16:39.085357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five-round protocol with clinicians giving real better/worse feedback on the same three datasets and compare final Dice and HD95 with the simulated-oracle results; the central claim is falsified if real feedback fails to produce round-over-round improvement or falls well short of the oracle-guided performance.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF International Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"Cited as the DINOv2 feature extractor; its patch-level features carry cosine-similarity label propagation."},{"cited_title":"ICLR1(2), 3 (2022) 10 Y","cited_arxiv_id":null,"evidence_quote":"Low-rank adaptation is used to fine-tune the feature extractor on pseudo-labeled data."},{"cited_title":"Computers in biology and medicine150, 106173 (2022)","cited_arxiv_id":null,"evidence_quote":"HSNet is the segmentation architecture trained on pseudo-labels and evaluated across rounds."}],"review_version":1}