Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only "Better or Worse" Expert Feedback

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A medical image segmentation network can be trained to competitive accuracy using only binary 'better or worse' expert feedback on AI-generated masks, with no manual pixel-level or region-level annotation.

desk verdict Promising preference-based segmentation framework, but the Dice oracle and missing baselines leave the central claim undemonstrated. read the letter →

arxiv 2507.05815 v2 pith:PFGGSZIR submitted 2025-07-08 eess.IV cs.LG

classification eess.IVcs.LG
keywords medicalimagesegmentationpreferencelearningbetterorworsefeedbackclickingagentpseudo-labelingfoundationmodelreinforcementlabelpropagation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Medical-image segmentation normally depends on experts tracing pixel-level ground truth, which is slow and costly. This paper claims that a segmentation network can instead be trained from a much weaker signal: an expert simply states whether a new AI-generated mask is better or worse than the previous one. The authors build a loop in which a clicking agent, rewarded by these binary comparisons, learns where to place corrective clicks, a foundation model propagates each click into a dense pseudo-mask, and a standard segmentation network is fine-tuned on the resulting pseudo-labels over several rounds. On three public datasets—polyp, skin lesion, and breast ultrasound—the reported performance improves round over round and reaches competitive Dice scores. If the claim holds, the human's role in annotation shifts from careful pixel drawing to high-level comparison.

What carries the argument

The load-bearing machinery is a preference-driven clicking loop. A lightweight U-Net policy, trained by the REINFORCE policy-gradient algorithm, observes a state made of the image plus the current predicted mask and outputs a softmax over click locations; the binary better/worse reward is the only learning signal. Each accepted click is converted from a single point to a dense pseudo-label by comparing patch features from a DINOv2-based foundation model (adapted with LoRA) under a cosine-similarity threshold, and only masks judged better replace the current one. Pseudo-labels from all images are filtered by quality and used to fine-tune the main segmentation network, which then seeds the next round's clicks, creating a bootstrapping loop.

What would settle it

Run the same five-round protocol with clinicians giving real better/worse feedback on the same three datasets and compare final Dice and HD95 with the simulated-oracle results; the central claim is falsified if real feedback fails to produce round-over-round improvement or falls well short of the oracle-guided performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that binary preference feedback—'better' or 'worse' relative to the previous mask—is sufficient supervision to train a medical image segmentation model without any manual pixel-level or region-level annotation. The mechanism works by treating each corrective click as a reinforcement-learning action: the clicking agent samples a coordinate, the click is propagated into a dense mask through cosine similarity in a LoRA-adapted DINOv2 feature space, and the expert's binary judgment is returned as a +1/−1 reward. The updated mask is kept only when the feedback is positive, and the accumulated pseudo-masks are used to fine-tune the segmentation network, whose improved outputs initialize the next round. Across the BUSI ultrasound, ISIC 2018 skin lesion, and combined Kvasir-SEG/CVC-ClinicDB polyp datasets, the authors report consistent gains in Dice and HD95 over five annotation rounds, with final distributions concentrated around high Dice scores.

Load-bearing premise

The load-bearing premise is that the simulated oracle—'better' when ground-truth Dice rises and 'worse' when it falls—behaves like a real expert's judgment; if human better/worse decisions diverge from Dice changes, the clicking agent's reward signal may be misleading.

Editorial extensions

If this is right

  • An expert can train or refine a segmenter just by comparing masks, avoiding pixel tracing and reducing cognitive load.
  • The clicking agent's policy improves across rounds, so later rounds need fewer corrective interactions for the same quality.
  • Because pseudo-labels are filtered by quality before training, a fraction of bad interactions does not derail the final model.
  • The framework's structure is not tied to one imaging modality; with a suitable foundation model it should transfer to other medical segmentation tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: real clinician feedback will be noisier than a Dice oracle, so practical deployment may need reward smoothing, repeated comparisons, or confidence thresholds to keep the loop stable.
  • Editorial inference: the same binary-comparison supervision could extend beyond segmentation to detection, registration, or image enhancement, wherever a global quality judgment is easier to give than a spatial correction.
  • Editorial inference: the decisive next experiment is a human study in which clinicians supply the better/worse judgments on the same datasets, to see whether the round-over-round gains survive real-world variability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a human-AI collaborative framework for medical image segmentation that aims to train a segmentation network using only binary "better or worse" expert feedback, thereby eliminating manual pixel-level or region-level annotation. The framework combines a DINOv2+LoRA feature extractor, feature-similarity-based label propagation, a REINFORCE-trained clicking agent, and multi-round fine-tuning of a PVT-based segmentation network (HSNet) on pseudo-labels. Experiments on BUSI, ISIC 2018, and combined polyp datasets report consistent improvement in Dice and HD95 over five annotation rounds. However, the feedback is simulated by an oracle that computes Dice improvement from ground-truth masks, the evaluation protocol is not clearly described, no baseline comparisons are provided, and the paper's own Section 4 acknowledges that the oracle oversimplifies real-world expert judgments.

Significance. If validated with real human feedback and a rigorous evaluation protocol, the core idea of training a clicking agent from binary preference feedback would be a valuable contribution to reducing annotation burden in medical imaging. The integration of a foundation model with preference-driven pseudo-labeling is timely and the multi-round bootstrapping concept is interesting. The paper is also transparent about the oracle limitation, which is a strength. However, the current experimental evidence is not yet sufficient to support the central claims of achieving "competitive segmentation performance" or eliminating the need for manual annotation. The absence of held-out evaluation, baselines, and ablations, together with the reliance on a ground-truth-based oracle, make the quantitative results hard to interpret.

major comments (4)
  1. [§3.1, Algorithm 1 line 12; §4] The simulated oracle that generates the reward is computed directly from ground-truth Dice: the reward is +1 if the new mask's Dice against ground truth increases, and −1 otherwise. This is the sole learning signal for the clicking agent and the same metric is used for evaluation. The paper acknowledges in Section 4 that this "oversimplifies real-world expert judgments," but that acknowledgment actually highlights the load-bearing weakness: the central claim is that binary "better or worse" feedback from a human expert suffices, yet the only evidence is with an omniscient, dense, pixel-level evaluator. A real clinician's binary judgment may be noisy, locally inconsistent, or correlated with clinical relevance rather than global Dice. The authors should provide a real human study or a carefully justified noise model for human feedback that preserves the qualitative improvement over rounds. Without this, the claim of "eliminating the need for any manual pixel-level and/or region-level annotation" is not demonstrated.
  2. [§3.2, Fig. 2 and Fig. 3] The evaluation protocol is not described. For BUSI the text says "we utilize the complete dataset for our experiments" and for ISIC and polyp datasets no train/test split is mentioned. If the reported Dice and HD95 values are computed on the same images that were used to generate pseudo-labels and fine-tune the segmentation network, then the results reflect training-set fit, not generalization, and the multi-round improvement could be a trivial consequence of fitting the training data. The authors must specify the exact train/validation/test protocol (e.g., a held-out test set or cross-validation) and evaluate the final model on images not used for pseudo-label generation or model fine-tuning.
  3. [§3.2, Fig. 2] The claim of "competitive segmentation performance" is unsupported because no baseline comparisons are reported. There is no comparison to a fully supervised model trained on the original ground-truth masks, to an interactive segmentation method such as SAM with clicks, to a random-click policy, or to existing weakly supervised approaches. Without such baselines, the reader cannot judge whether the proposed framework actually reaches competitive accuracy or whether the improvement over rounds is meaningful. Additionally, REINFORCE is stochastic, but no standard deviations over multiple runs or statistical significance tests are provided; the "consistent improvement" in Fig. 2 could be within one standard deviation and would be more persuasive with error bars.
  4. [Algorithm 1] Several key hyperparameters are fixed without sensitivity analysis or ablation: the number of rounds R=5, interaction steps per image T=5, similarity threshold τ=0.8, top-K filtering percentage, temperature parameter, and REINFORCE learning rate. In particular, the label-propagation threshold τ and the number of clicks T directly determine the quality of the pseudo-labels and the difficulty of the clicking agent's task. Ablation studies are needed to show that the framework's performance is not an artifact of a particular hyperparameter choice, and to isolate the contribution of each component (adaptable foundation model, clicking agent, label propagation, multi-round training). Without these, it is unclear how robust the method is across datasets and settings.
minor comments (5)
  1. [§1] The phrase "especially for complex 3D anatomies or subtle pathologies" is duplicated in the second paragraph of the introduction.
  2. [Fig. 2 and Fig. 3 captions] The captions of Figures 2 and 3 list the dataset order as "polyp (left), skin lesion (middle), and ultrasound (right)", while the text in Section 3.2 states the order as "ultrasound (left), skin lesion (middle), and polyp (right)". Please align the text and figure captions.
  3. [§2.2, Algorithm 1] The algorithm uses "DINO-based label propagation" while the text discusses DINOv2; please unify the terminology to avoid confusion.
  4. [§3.1] The temperature parameter for action sampling and the top-K filtering percentage are mentioned but never specified. Provide the exact values or state that they are set to defaults and list those defaults.
  5. [§3.1] Please report the training details for the main segmentation network HSNet, including optimizer, learning rate, number of epochs, and the LoRA rank for the adaptable foundation model, to enable reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

The clicking agent is trained on rewards computed from ground-truth Dice and evaluated on the same Dice, so the reported improvement is forced by the oracle rather than being evidence that human binary feedback suffices.

  1. fitted input called prediction [Section 3.1 Experimental Setup; Section 2.2 Reward; Section 3.2 Results]
    "We use the provided ground truth masks in these datasets only as an oracle to simulate the expert's "better or worse" feedback, not as supervised training masks for the segmentation model. To simulate expert feedback, we use the ground truth mask as an oracle. We compute the Dice Similarity Coefficient (DSC) between the oracle and the segmentation mask both before and after the agent's click. If the DSC increases, the outcome is labeled "better" (reward = +1)."

    The reward that trains the clicking agent is the sign of the change in Dice computed against the ground-truth mask. The paper's main result is then reported as increasing Dice and decreasing HD95 against the same ground-truth masks (Fig. 2). Therefore the improvement across rounds is, by construction, the result of rewarding the agent for improving the exact metric used for evaluation. It does not demonstrate that a human expert's binary 'better or worse' judgment, which is not defined by Dice, would provide the same learning signal. The claimed elimination of manual pixel-level annotation is not supported because the oracle itself consumes the manual annotations.

full rationale

The method's internal derivation chain is not mathematically circular: pseudo-labels are produced by feature-similarity propagation from clicks, and the segmentation network is trained on those pseudo-labels rather than directly on ground truth. The circularity is in the experimental validation. The sole learning signal for the clicking agent is a binary reward defined as the sign of the change in Dice against the ground-truth mask (Section 3.1), and the evaluation metric is Dice/HD95 against the same ground-truth masks (Section 3.2). Hence the agent is trained to optimize the same quantity used to measure success, so the reported improvement is a direct consequence of the reward definition. The paper itself concedes in Section 4 that the simulated oracle 'oversimplifies real-world expert judgments.' Without a demonstration that a real clinician's better-or-worse comparisons carry the same information as a ground-truth Dice oracle, the central claim of training competitive segmentation models from only human binary feedback is not independently established. There are no significant self-citation or uniqueness-importation issues.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The system relies on a simulated oracle for rewards, a threshold-based propagation rule, and assumptions about foundation model transferability. These are explicit design choices, not derived from first principles.

free parameters (6)
  • similarity threshold tau = 0.8
    Used in Algorithm 1 for label propagation; set to 0.8 without sensitivity analysis.
  • number of rounds R = 5
    Set to 5 across experiments; no ablation on this.
  • interaction steps per image T = 5
    Set to 5; no study of impact.
  • temperature parameter = not specified
    Used for action sampling in the clicking agent; value not given.
  • top-K filtering percentage = not specified
    Optional filtering of pseudo-labels mentioned but not quantified.
  • REINFORCE learning rate = not specified
    Training details missing.
assumptions (4)
  • domain assumption DICE improvement approximates human 'better or worse' preference
    Section 3.1 uses ground-truth Dice as oracle reward; human judgments may differ.
  • domain assumption DINOv2 features generalise to medical imaging
    Section 2.1 assumes pre-trained DINOv2, adapted with LoRA, provides useful features for medical images.
  • domain assumption Cosine similarity in feature space with threshold tau yields correct label propagation
    Section 2.2 assigns labels to all patches with similarity >0.8; assumption that this captures semantic regions.
  • domain assumption Pseudo-labels from propagated clicks are sufficient to train a segmentation model
    Section 2.3 trains HSNet on generated pseudo-masks; no theoretical guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only "Better or Worse" Expert Feedback." pith.science (2026). https://pith.science/paper/PFGGSZIR

@misc{pith2026250705815,
  author       = {Pith},
  title        = {Pith review of: Beyond Manual Annotation: A Human-AI Collaborative Framework for Medical Image Segmentation Using Only "Better or Worse" Expert Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFGGSZIR}},
  note         = {Machine review of arXiv:2507.05815}
}
read the original abstract

Manual annotation of medical images is a labor-intensive and time-consuming process, posing a significant bottleneck in the development and deployment of robust medical imaging AI systems. This paper introduces a novel hands-free Human-AI collaborative framework for medical image segmentation that substantially reduces the annotation burden by eliminating the need for explicit manual pixel-level labeling. The core innovation lies in a preference learning paradigm, where human experts provide minimal, intuitive feedback -- simply indicating whether an AI-generated segmentation is better or worse than a previous version. The framework comprises four key components: (1) an adaptable foundation model (FM) for feature extraction, (2) label propagation based on feature similarity, (3) a clicking agent that learns from human better-or-worse feedback to decide where to click and with which label, and (4) a multi-round segmentation learning procedure that trains a state-of-the-art segmentation network using pseudo-labels generated by the clicking agent and FM-based label propagation. Experiments on three public datasets demonstrate that the proposed approach achieves competitive segmentation performance using only binary preference feedback, without requiring experts to directly manually annotate the images.

Figures

Figures reproduced from arXiv: 2507.05815 by the authors.

Figure 1
Figure 1. Overview of the proposed framework. scalability and rapid deployment of AI-powered segmentation tools in diverse clinical settings. To mitigate this annotation burden, the community has explored various weakly supervised and interactive segmentation paradigms. These methods aim to reduce the annotation effort by leveraging simpler forms of human guidance, such as bounding boxes, scribbles, or points [18, 3]. More re… view at source ↗
Figure 2
Figure 2. Segmentation performance across annotation rounds, showing progressive im￾provement with increasing Dice scores and decreasing HD95 on polyp (left), skin lesion (middle), and ultrasound (right) datasets [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Violin plots showing the distribution of final interactive Dice scores across annotation rounds, highlighting improved annotation quality and consistency over time. Results are shown for polyp (left), skin lesion (middle), and ultrasound (right) datasets. foundation model primarily trained on natural scene images, which may limit its effectiveness in capturing ultrasound-specific characteristics [PITH_FULL_IMAGE:fi… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Segmentation examples on the BUSI dataset. Top: input ultrasound image; Middle: ground truth; Bottom: prediction by our method [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Segmentation examples on the skin lesion dataset. Top: input dermoscopy im￾age; Middle: ground truth mask; Bottom: segmentation predicted by our method. judgments, which are often subjective and prioritize clinical relevance over global metrics. Human feedback also int…
Figure 6
Figure 6. Figure 6: Segmentation examples on the polyp dataset. Top: input endoscopic image; Middle: ground truth mask; Bottom: segmentation predicted by our method. without requiring any direct manual annotation from users. Our experiments across polyp, skin lesion, and breast ultrasound…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages

  1. [1]

    Data in brief28, 104863 (2020)

    Al-Dhabyani, W., Gomaa, M., Khaled, H., Fahmy, A.: Dataset of breast ultrasound images. Data in brief28, 104863 (2020)

  2. [2]

    saliency maps from physicians

    Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilar- iño, F.: Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics 43, 99–111 (2015)

  3. [3]

    Can, Y.B., Chaitanya, K., Mustafa, B., Koch, L.M., Konukoglu, E., Baumgartner, C.F.: Learning to segment medical images with scribble-supervision alone. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Work- shop, ML-CDS 2018, Held in Conjunctio...

  4. [4]

    arXiv preprint arXiv:2308.16184 (2023)

    Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., et al.: Sam-med2d. arXiv preprint arXiv:2308.16184 (2023)

  5. [5]

    Cheplygina, V., de Bruijne, M., Pluim, J.P.: Not-so-supervised: a survey of semi- supervised,multi-instance,andtransferlearninginmedicalimageanalysis.Medical image analysis 54, 280–296 (2019)

  6. [6]

    In: International Con- ference on Learning Representations (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Con- ference on Learning Representations (2021)

  7. [7]

    ICLR1(2), 3 (2022) 10 Y

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022) 10 Y. Zhang

  8. [8]

    Nature methods 18(2), 203–211 (2021)

    Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)

Show all 21 references
  1. [9]

    In: International con- ference on multimedia modeling

    Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., De Lange, T., Johansen, D., Johansen, H.D.: Kvasir-seg: A segmented polyp dataset. In: International con- ference on multimedia modeling. pp. 451–462. Springer (2019)

  2. [10]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3879–3893 (2023)

  3. [11]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Konwer, A., Yang, Z., Bas, E., Xiao, C., Prasanna, P., Bhatia, P., Kass-Hout, T.: Enhancing sam with efficient prompting and preference optimization for semi- supervised medical image segmentation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. ...

  4. [12]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liao, X., Li, W., Xu, Q., Wang, X., Jin, B., Zhang, X., Wang, Y., Zhang, Y.: Iteratively-refined interactive 3d medical image segmentation with multi-agent re- inforcement learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9394...

  5. [13]

    Medical Image Analysis92, 103028 (2024)

    Liao, Z., Hu, S., Xie, Y., Xia, Y.: Modeling annotator preference and stochastic an- notation error for medical image segmentation. Medical Image Analysis92, 103028 (2024)

  6. [14]

    Medical image analysis42, 60–88 (2017)

    Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Medical image analysis42, 60–88 (2017)

  7. [15]

    Nature Communications15(1), 654 (2024)

    Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)

  8. [16]

    In: International Conference on Medical image computing and computer-assisted intervention

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)

  9. [17]

    Scientific data 5(1), 1–9 (2018)

    Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1), 1–9 (2018)

  10. [18]

    IEEE transactions on medical imaging 37(7), 1562–1573 (2018)

    Wang, G., Li, W., Zuluaga, M.A., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al.: Interactive medical image segmen- tation using deep learning with image-specific fine tuning. IEEE transactions on medical imaging 37(7), 1562–1573 (2018)

  11. [19]

    Computational visual media 8(3), 415–424 (2022)

    Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pvt v2: Improved baselines with pyramid vision transformer. Computational visual media 8(3), 415–424 (2022)

  12. [20]

    Machine learning8, 229–256 (1992)

    Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning8, 229–256 (1992)

  13. [21]

    Computers in biology and medicine150, 106173 (2022)

    Zhang, W., Fu, C., Zheng, Y., Zhang, F., Zhao, Y., Sham, C.W.: Hsnet: A hybrid semantic network for polyp segmentation. Computers in biology and medicine150, 106173 (2022)

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.