REVIEW 4 major objections 5 minor 21 references
Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only "Better or Worse" Expert Feedback
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A medical image segmentation network can be trained to competitive accuracy using only binary 'better or worse' expert feedback on AI-generated masks, with no manual pixel-level or region-level annotation.
desk verdict Promising preference-based segmentation framework, but the Dice oracle and missing baselines leave the central claim undemonstrated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a preference-driven clicking loop. A lightweight U-Net policy, trained by the REINFORCE policy-gradient algorithm, observes a state made of the image plus the current predicted mask and outputs a softmax over click locations; the binary better/worse reward is the only learning signal. Each accepted click is converted from a single point to a dense pseudo-label by comparing patch features from a DINOv2-based foundation model (adapted with LoRA) under a cosine-similarity threshold, and only masks judged better replace the current one. Pseudo-labels from all images are filtered by quality and used to fine-tune the main segmentation network, which then seeds the next round's clicks, creating a bootstrapping loop.
What would settle it
Run the same five-round protocol with clinicians giving real better/worse feedback on the same three datasets and compare final Dice and HD95 with the simulated-oracle results; the central claim is falsified if real feedback fails to produce round-over-round improvement or falls well short of the oracle-guided performance.
Extended reading notes
Core claim
The paper's central claim is that binary preference feedback—'better' or 'worse' relative to the previous mask—is sufficient supervision to train a medical image segmentation model without any manual pixel-level or region-level annotation. The mechanism works by treating each corrective click as a reinforcement-learning action: the clicking agent samples a coordinate, the click is propagated into a dense mask through cosine similarity in a LoRA-adapted DINOv2 feature space, and the expert's binary judgment is returned as a +1/−1 reward. The updated mask is kept only when the feedback is positive, and the accumulated pseudo-masks are used to fine-tune the segmentation network, whose improved outputs initialize the next round. Across the BUSI ultrasound, ISIC 2018 skin lesion, and combined Kvasir-SEG/CVC-ClinicDB polyp datasets, the authors report consistent gains in Dice and HD95 over five annotation rounds, with final distributions concentrated around high Dice scores.
Load-bearing premise
The load-bearing premise is that the simulated oracle—'better' when ground-truth Dice rises and 'worse' when it falls—behaves like a real expert's judgment; if human better/worse decisions diverge from Dice changes, the clicking agent's reward signal may be misleading.
Editorial extensions
If this is right
- An expert can train or refine a segmenter just by comparing masks, avoiding pixel tracing and reducing cognitive load.
- The clicking agent's policy improves across rounds, so later rounds need fewer corrective interactions for the same quality.
- Because pseudo-labels are filtered by quality before training, a fraction of bad interactions does not derail the final model.
- The framework's structure is not tied to one imaging modality; with a suitable foundation model it should transfer to other medical segmentation tasks.
Reading between the lines
- Editorial inference: real clinician feedback will be noisier than a Dice oracle, so practical deployment may need reward smoothing, repeated comparisons, or confidence thresholds to keep the loop stable.
- Editorial inference: the same binary-comparison supervision could extend beyond segmentation to detection, registration, or image enhancement, wherever a global quality judgment is easier to give than a spatial correction.
- Editorial inference: the decisive next experiment is a human study in which clinicians supply the better/worse judgments on the same datasets, to see whether the round-over-round gains survive real-world variability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a human-AI collaborative framework for medical image segmentation that aims to train a segmentation network using only binary "better or worse" expert feedback, thereby eliminating manual pixel-level or region-level annotation. The framework combines a DINOv2+LoRA feature extractor, feature-similarity-based label propagation, a REINFORCE-trained clicking agent, and multi-round fine-tuning of a PVT-based segmentation network (HSNet) on pseudo-labels. Experiments on BUSI, ISIC 2018, and combined polyp datasets report consistent improvement in Dice and HD95 over five annotation rounds. However, the feedback is simulated by an oracle that computes Dice improvement from ground-truth masks, the evaluation protocol is not clearly described, no baseline comparisons are provided, and the paper's own Section 4 acknowledges that the oracle oversimplifies real-world expert judgments.
Significance. If validated with real human feedback and a rigorous evaluation protocol, the core idea of training a clicking agent from binary preference feedback would be a valuable contribution to reducing annotation burden in medical imaging. The integration of a foundation model with preference-driven pseudo-labeling is timely and the multi-round bootstrapping concept is interesting. The paper is also transparent about the oracle limitation, which is a strength. However, the current experimental evidence is not yet sufficient to support the central claims of achieving "competitive segmentation performance" or eliminating the need for manual annotation. The absence of held-out evaluation, baselines, and ablations, together with the reliance on a ground-truth-based oracle, make the quantitative results hard to interpret.
major comments (4)
- [§3.1, Algorithm 1 line 12; §4] The simulated oracle that generates the reward is computed directly from ground-truth Dice: the reward is +1 if the new mask's Dice against ground truth increases, and −1 otherwise. This is the sole learning signal for the clicking agent and the same metric is used for evaluation. The paper acknowledges in Section 4 that this "oversimplifies real-world expert judgments," but that acknowledgment actually highlights the load-bearing weakness: the central claim is that binary "better or worse" feedback from a human expert suffices, yet the only evidence is with an omniscient, dense, pixel-level evaluator. A real clinician's binary judgment may be noisy, locally inconsistent, or correlated with clinical relevance rather than global Dice. The authors should provide a real human study or a carefully justified noise model for human feedback that preserves the qualitative improvement over rounds. Without this, the claim of "eliminating the need for any manual pixel-level and/or region-level annotation" is not demonstrated.
- [§3.2, Fig. 2 and Fig. 3] The evaluation protocol is not described. For BUSI the text says "we utilize the complete dataset for our experiments" and for ISIC and polyp datasets no train/test split is mentioned. If the reported Dice and HD95 values are computed on the same images that were used to generate pseudo-labels and fine-tune the segmentation network, then the results reflect training-set fit, not generalization, and the multi-round improvement could be a trivial consequence of fitting the training data. The authors must specify the exact train/validation/test protocol (e.g., a held-out test set or cross-validation) and evaluate the final model on images not used for pseudo-label generation or model fine-tuning.
- [§3.2, Fig. 2] The claim of "competitive segmentation performance" is unsupported because no baseline comparisons are reported. There is no comparison to a fully supervised model trained on the original ground-truth masks, to an interactive segmentation method such as SAM with clicks, to a random-click policy, or to existing weakly supervised approaches. Without such baselines, the reader cannot judge whether the proposed framework actually reaches competitive accuracy or whether the improvement over rounds is meaningful. Additionally, REINFORCE is stochastic, but no standard deviations over multiple runs or statistical significance tests are provided; the "consistent improvement" in Fig. 2 could be within one standard deviation and would be more persuasive with error bars.
- [Algorithm 1] Several key hyperparameters are fixed without sensitivity analysis or ablation: the number of rounds R=5, interaction steps per image T=5, similarity threshold τ=0.8, top-K filtering percentage, temperature parameter, and REINFORCE learning rate. In particular, the label-propagation threshold τ and the number of clicks T directly determine the quality of the pseudo-labels and the difficulty of the clicking agent's task. Ablation studies are needed to show that the framework's performance is not an artifact of a particular hyperparameter choice, and to isolate the contribution of each component (adaptable foundation model, clicking agent, label propagation, multi-round training). Without these, it is unclear how robust the method is across datasets and settings.
minor comments (5)
- [§1] The phrase "especially for complex 3D anatomies or subtle pathologies" is duplicated in the second paragraph of the introduction.
- [Fig. 2 and Fig. 3 captions] The captions of Figures 2 and 3 list the dataset order as "polyp (left), skin lesion (middle), and ultrasound (right)", while the text in Section 3.2 states the order as "ultrasound (left), skin lesion (middle), and polyp (right)". Please align the text and figure captions.
- [§2.2, Algorithm 1] The algorithm uses "DINO-based label propagation" while the text discusses DINOv2; please unify the terminology to avoid confusion.
- [§3.1] The temperature parameter for action sampling and the top-K filtering percentage are mentioned but never specified. Provide the exact values or state that they are set to defaults and list those defaults.
- [§3.1] Please report the training details for the main segmentation network HSNet, including optimizer, learning rate, number of epochs, and the LoRA rank for the adaptable foundation model, to enable reproducibility.
Circularity Check
The clicking agent is trained on rewards computed from ground-truth Dice and evaluated on the same Dice, so the reported improvement is forced by the oracle rather than being evidence that human binary feedback suffices.
-
fitted input called prediction
[Section 3.1 Experimental Setup; Section 2.2 Reward; Section 3.2 Results]
"We use the provided ground truth masks in these datasets only as an oracle to simulate the expert's "better or worse" feedback, not as supervised training masks for the segmentation model. To simulate expert feedback, we use the ground truth mask as an oracle. We compute the Dice Similarity Coefficient (DSC) between the oracle and the segmentation mask both before and after the agent's click. If the DSC increases, the outcome is labeled "better" (reward = +1)."
The reward that trains the clicking agent is the sign of the change in Dice computed against the ground-truth mask. The paper's main result is then reported as increasing Dice and decreasing HD95 against the same ground-truth masks (Fig. 2). Therefore the improvement across rounds is, by construction, the result of rewarding the agent for improving the exact metric used for evaluation. It does not demonstrate that a human expert's binary 'better or worse' judgment, which is not defined by Dice, would provide the same learning signal. The claimed elimination of manual pixel-level annotation is not supported because the oracle itself consumes the manual annotations.
full rationale
The method's internal derivation chain is not mathematically circular: pseudo-labels are produced by feature-similarity propagation from clicks, and the segmentation network is trained on those pseudo-labels rather than directly on ground truth. The circularity is in the experimental validation. The sole learning signal for the clicking agent is a binary reward defined as the sign of the change in Dice against the ground-truth mask (Section 3.1), and the evaluation metric is Dice/HD95 against the same ground-truth masks (Section 3.2). Hence the agent is trained to optimize the same quantity used to measure success, so the reported improvement is a direct consequence of the reward definition. The paper itself concedes in Section 4 that the simulated oracle 'oversimplifies real-world expert judgments.' Without a demonstration that a real clinician's better-or-worse comparisons carry the same information as a ground-truth Dice oracle, the central claim of training competitive segmentation models from only human binary feedback is not independently established. There are no significant self-citation or uniqueness-importation issues.
Assumptions & free parameters
free parameters (6)
- similarity threshold tau =
0.8
- number of rounds R =
5
- interaction steps per image T =
5
- temperature parameter =
not specified
- top-K filtering percentage =
not specified
- REINFORCE learning rate =
not specified
assumptions (4)
- domain assumption DICE improvement approximates human 'better or worse' preference
- domain assumption DINOv2 features generalise to medical imaging
- domain assumption Cosine similarity in feature space with threshold tau yields correct label propagation
- domain assumption Pseudo-labels from propagated clicks are sufficient to train a segmentation model
Cite this review
Pith. "Pith review of Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only "Better or Worse" Expert Feedback." pith.science (2026). https://pith.science/paper/PFGGSZIR
@misc{pith2026250705815,
author = {Pith},
title = {Pith review of: Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only "Better or Worse" Expert Feedback},
year = {2026},
howpublished = {\url{https://pith.science/paper/PFGGSZIR}},
note = {Machine review of arXiv:2507.05815}
}
read the original abstract
Manual annotation of medical images is a labor-intensive and time-consuming process, posing a significant bottleneck in the development and deployment of robust medical imaging AI systems. This paper introduces a novel hands-free Human-AI collaborative framework for medical image segmentation that substantially reduces the annotation burden by eliminating the need for explicit manual pixel-level labeling. The core innovation lies in a preference learning paradigm, where human experts provide minimal, intuitive feedback -- simply indicating whether an AI-generated segmentation is better or worse than a previous version. The framework comprises four key components: (1) an adaptable foundation model (FM) for feature extraction, (2) label propagation based on feature similarity, (3) a clicking agent that learns from human better-or-worse feedback to decide where to click and with which label, and (4) a multi-round segmentation learning procedure that trains a state-of-the-art segmentation network using pseudo-labels generated by the clicking agent and FM-based label propagation. Experiments on three public datasets demonstrate that the proposed approach achieves competitive segmentation performance using only binary preference feedback, without requiring experts to directly manually annotate the images.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Data in brief28, 104863 (2020)
Al-Dhabyani, W., Gomaa, M., Khaled, H., Fahmy, A.: Dataset of breast ultrasound images. Data in brief28, 104863 (2020)
2020
-
[2]
saliency maps from physicians
Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilar- iño, F.: Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics 43, 99–111 (2015)
2015
-
[3]
Can, Y.B., Chaitanya, K., Mustafa, B., Koch, L.M., Konukoglu, E., Baumgartner, C.F.: Learning to segment medical images with scribble-supervision alone. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Work- shop, ML-CDS 2018, Held in Conjunctio...
work page 2018
-
[4]
arXiv preprint arXiv:2308.16184 (2023)
Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., et al.: Sam-med2d. arXiv preprint arXiv:2308.16184 (2023)
arXiv 2023
-
[5]
Cheplygina, V., de Bruijne, M., Pluim, J.P.: Not-so-supervised: a survey of semi- supervised,multi-instance,andtransferlearninginmedicalimageanalysis.Medical image analysis 54, 280–296 (2019)
work page 2019
-
[6]
In: International Con- ference on Learning Representations (2021)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Con- ference on Learning Representations (2021)
2021
-
[7]
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022) 10 Y. Zhang
work page 2022
-
[8]
Nature methods 18(2), 203–211 (2021)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
2021
Show all 21 references
-
[9]
In: International con- ference on multimedia modeling
Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., De Lange, T., Johansen, D., Johansen, H.D.: Kvasir-seg: A segmented polyp dataset. In: International con- ference on multimedia modeling. pp. 451–462. Springer (2019)
2019
-
[10]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3879–3893 (2023)
2023
-
[11]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Konwer, A., Yang, Z., Bas, E., Xiao, C., Prasanna, P., Bhatia, P., Kass-Hout, T.: Enhancing sam with efficient prompting and preference optimization for semi- supervised medical image segmentation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. ...
2025
-
[12]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Liao, X., Li, W., Xu, Q., Wang, X., Jin, B., Zhang, X., Wang, Y., Zhang, Y.: Iteratively-refined interactive 3d medical image segmentation with multi-agent re- inforcement learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9394...
2020
-
[13]
Medical Image Analysis92, 103028 (2024)
Liao, Z., Hu, S., Xie, Y., Xia, Y.: Modeling annotator preference and stochastic an- notation error for medical image segmentation. Medical Image Analysis92, 103028 (2024)
2024
-
[14]
Medical image analysis42, 60–88 (2017)
Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Medical image analysis42, 60–88 (2017)
2017
-
[15]
Nature Communications15(1), 654 (2024)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)
2024
-
[16]
In: International Conference on Medical image computing and computer-assisted intervention
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
2015
-
[17]
Scientific data 5(1), 1–9 (2018)
Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1), 1–9 (2018)
2018
-
[18]
IEEE transactions on medical imaging 37(7), 1562–1573 (2018)
Wang, G., Li, W., Zuluaga, M.A., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al.: Interactive medical image segmen- tation using deep learning with image-specific fine tuning. IEEE transactions on medical imaging 37(7), 1562–1573 (2018)
2018
-
[19]
Computational visual media 8(3), 415–424 (2022)
Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pvt v2: Improved baselines with pyramid vision transformer. Computational visual media 8(3), 415–424 (2022)
2022
-
[20]
Machine learning8, 229–256 (1992)
Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning8, 229–256 (1992)
1992
-
[21]
Computers in biology and medicine150, 106173 (2022)
Zhang, W., Fu, C., Zheng, Y., Zhang, F., Zhao, Y., Sham, C.W.: Hsnet: A hybrid semantic network for polyp segmentation. Computers in biology and medicine150, 106173 (2022)
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.