REVIEW 2 major objections 5 minor 31 references
A multi-decoder segmentation network pre-trained on cholecystectomy videos transfers to rectal-surgery images, reaching 62.4% dice after full fine-tuning and matching a from-scratch model (60.0%) with no fine-tuning.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 05:35 UTC pith:3LBKO52U
load-bearing objection Useful, incremental empirical work on decoder-specific transfer for laparoscopic segmentation, but the zero-shot transfer claim is under-specified to the point of being unverifiable as written. the 2 major comments →
Leveraging Transfer Learning with Class-Specific Decoders for Laparoscopic Segmentation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that a Common Encoder–Multiple Decoder (CEMD) architecture, pre-trained on CholecSeg8K and fully fine-tuned on the Dresden Surgical Anatomy (DSA) dataset, achieves the best segmentation performance at 62.4% dice, with faster convergence than training from scratch. It further claims that the same pre-trained model evaluated on DSA with no fine-tuning reaches about 60% dice, comparable to a model trained on DSA from scratch, demonstrating effective cross-surgical knowledge transfer despite only partial class overlap. On the source dataset, CEMD outperforms a shared-decoder baseline by about 4% overall dice, with a 27-point gain on the underrepresented Gallbladder class. Fine-t
What carries the argument
The central object is the Common Encoder–Multiple Decoder (CEMD) architecture: a shared encoder (an Attention U-Net, i.e., a U-Net with attention gates) with a dedicated decoder head for each organ class, contrasted with a Common Encoder–Common Decoder (CECD) baseline. The class-specific decoders are the mechanism that lets the model separate anatomical features per organ; the transfer protocol (CholecSeg8K pre-training followed by DSA fine-tuning with the encoder either frozen or trainable) is what exposes where transferable knowledge lives.
Load-bearing premise
The load-bearing premise is that the eleven DSA decoder heads used in the no-fine-tuning evaluation have meaningful initializations from CholecSeg8K pre-training; the paper reports dice for DSA-only organs without stating how those decoder heads are initialized or which CholecSeg8K class maps to them.
What would settle it
Reset the decoder heads for DSA-only organs (Spleen, Ureter, etc.) to random weights and run the same zero-shot evaluation on DSA; if the overall dice remain near 60%, the transfer result is not coming from learned organ-specific decoders. Inspecting the pre-trained checkpoint to confirm each of the eleven decoder heads was actually updated on CholecSeg8K would settle the same question.
If this is right
- Pre-training on one laparoscopic procedure type can substitute for training from scratch on another, at equal or better accuracy and with substantially faster convergence.
- Full-network fine-tuning is necessary to realize transfer gains; freezing the encoder and updating only decoders is worse than doing no fine-tuning at all.
- Organ-specific decoder heads transfer better than a shared decoder under domain shift, losing 4.9 versus 5.6 dice points when the encoder is frozen.
- Class-specific decoders concentrate their benefit on rare classes (Gallbladder +27% dice on CholecSeg8K), but transfer learning does not fix class imbalance for the smallest organs on DSA.
- The zero-shot transfer result, if reliable, means target-domain labels are not strictly required to obtain a usable segmentation model for a new surgical domain.
Where Pith is reading between the lines
- The zero-shot 60% dice figure is only interpretable if all eleven DSA decoder heads have defined initializations from CholecSeg8K; the paper's 6-class source evaluation leaves open how decoders for DSA-only organs are initialized, so a repeatable experiment should spell that mapping out.
- A natural extension the authors do not test: pre-train on multiple source procedures jointly and measure whether the zero-shot floor rises above 60% dice before any fine-tuning.
- The DFT degradation suggests a testable mechanism — shared decoders entangle class representations with the encoder's feature distribution, so a frozen encoder plus retrained decoder misaligns; a feature-distribution shift analysis before and after DFT could confirm this.
- For deployment, the 0FT result hints at a label-efficient workflow: generate initial segmentations with the zero-shot model, have surgeons correct only the low-confidence regions, then fine-tune; this is an editorial extrapolation, not reported in the paper.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies transfer learning between two laparoscopic datasets (CholecSeg8K and DSA) for multi-organ segmentation under two architectures: a shared-decoder baseline (CECD) and a class-specific multi-decoder architecture (CEMD). It reports that CEMD outperforms CECD on both datasets, that a CholecSeg8K-pretrained CEMD evaluated on DSA without fine-tuning reaches roughly 60% Dice (comparable to training from scratch), that full fine-tuning improves this to 62.4% Dice, and that decoder-only fine-tuning degrades performance relative to no fine-tuning. The paper interprets these results as evidence of effective cross-surgical knowledge transfer and argues that class-specific decoders retain transferred knowledge better. It also emphasizes that class imbalance remains a persistent bottleneck.
Significance. If the experimental protocol is made precise, the paper would provide a useful empirical data point: supervised transfer between two different laparoscopic procedure domains under a class-specific decoder architecture, with a clean separation of encoder-only versus decoder-only adaptation. The paper is transparent about building on the authors' earlier CEMD design [17] and re-baselines it, so circularity concerns are low. It reports held-out test Dice/IoU, includes convergence curves, and candidly discusses limitations. The main risk is not circularity but incomplete specification of the zero-shot evaluation protocol: the CholecSeg8K model is trained on a 6-class subset, yet Tables 2-4 report zero-shot Dice for all 11 DSA classes without stating a decoder mapping or initialization. Because the zero-shot result is the paper's second headline contribution, this gap is substantive.
major comments (2)
- [§4.1.2, Tables 2-4] The central zero-shot transfer claim is not reproducible as written. CholecSeg8K pre-training is described for a 6-class subset (Background, Abdominal Wall, Liver, Fat, Gallbladder, L-hook), but Tables 2-4 report 0FT Dice/IoU for all 11 DSA classes. Section 3.1 notes that the Cholec gastrointestinal class combines stomach and small intestine, but it does not specify how Colon, Pancreas, Spleen, Ureter, Vesicular Glands, Inferior Mesenteric Artery, or Intestinal Veins are represented by the pre-trained CEMD. The paper must state: (i) how many decoders the CholecSeg8K CEMD has; (ii) how the decoders for DSA classes absent from CholecSeg8K are initialized for 0FT (random, copied, or mapped from a related class); and (iii) the exact class mapping used to produce the per-class 0FT numbers. Without this, the conclusion that CEMD 0FT is comparable to training from scratch (Table 2: 60.0 vs. 60.
- [§3.2, §4.2.1, Tables 3-4] The decoder fine-tuning (DFT) comparison has the same ambiguity and affects the encoder/decoder adaptation claim. If CEMD is instantiated with 11 decoders during CholecSeg8K training, it is unclear which decoders receive gradients from the 6 training classes. If it is instantiated with 6 decoders, it is unclear how the 11 DSA decoders are created for DFT/FFT. The claim that CEMD is more resilient to DFT degradation than CECD (−4.9% vs. −5.6%) could conflate freshly initialized decoders for unseen DSA organs with transferred decoders. The paper should specify the number of decoders in each phase and report which decoders were pre-trained, so the resilience comparison is interpretable.
minor comments (5)
- [§4.1.1, Table 5, Conclusion] The text says CEMD improves overall Dice by about 3.5% over CECD when trained from scratch on DSA, but Table 5 reports 60.0 vs. 59.3, a 0.7-point gain. If the 3.5% refers to per-class average gain or to a different baseline from [17], state that explicitly. As written, the numbers conflict.
- [Table 2] The Best column is not the maximum of the three reported epoch columns: CEMD 0FT shows 60.7 at epoch 150 but Best = 60.0. If Best means the checkpoint with lowest validation loss rather than highest test Dice, say so; the current presentation is confusing.
- [Tables 3-4] The notation uses a leading plus/minus both for the DFT/FFT values and for the change relative to 0FT (e.g., −85.52−75.13). This makes the tables hard to parse. Please reformat, e.g., give 0FT, DFT, FFT as separate columns with delta columns clearly labeled.
- [§3.3, §4.2.1] The phrase No Fine-Tuning (0FT) is used both for training from scratch on the same dataset and for cross-dataset zero-shot evaluation. This dual meaning makes sentences like the scores are in a similar range to the in-dataset evaluation on the DSA dataset ambiguous. Use distinct labels, e.g., scratch vs. zero-shot transfer.
- [§3.3, Limitations] All results are from a single seed with no confidence intervals or significance tests. Given that several headline differences are small (e.g., CECD FFT vs. CECD 0FT: 59.4 vs. 59.3), the paper should at least acknowledge the lack of variance quantification.
Circularity Check
No significant circularity: the paper's claims are measured benchmark results, and the self-citation to the authors' prior CEMD work is corroborative rather than load-bearing.
full rationale
The central claims are empirical: the 0FT, DFT, and FFT dice scores in Tables 2-4 are measured on a held-out DSA test set, not derived from parameters fitted to DSA and renamed as predictions. No equation in the paper identifies a derived quantity with an input. The CEMD-vs-CECD advantage is independently re-established in this paper on DSA (§4.1.1) and CholecSeg8K (§4.1.2, Table 1), so the authors' prior work [17] is used for architectural motivation and consistency, not as an unverified premise that forces the new results. The 'without any fine-tuning... comparable to training from scratch' claim is a direct evaluation of a CholecSeg8K-trained model on DSA, so it is not a fit-to-target wrapped as a prediction. A genuine reproducibility gap exists: §4.1.2 says CholecSeg8K pre-training used six selected classes, while §3.2 and Tables 3-4 report 0FT dice for eleven DSA classes without specifying how the additional CEMD decoders are initialized or mapped from Cholec classes. That is a validity/reporting concern for the 0FT transfer claim, but it does not make the result equivalent to its own inputs by construction. There is no uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result; therefore the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (6)
- initial learning rate =
1e-4
- learning-rate decay schedule =
0.5 every 10 epochs
- max epochs / early stopping =
150 epochs, patience 10
- DSA test-split class proportions =
kept high and close to training proportions
- CholecSeg8K class subset =
6 of 13 classes (Background, Abdominal Wall, Liver, Fat, Gallbladder, L-hook)
- random seed =
fixed but not disclosed
axioms (4)
- domain assumption DSA and CholecSeg8K ground-truth annotations are reliable and their class labels can be related for transfer (Liver, Abdominal Wall, and gastrointestinal structures).
- ad hoc to paper A model trained on a 6-class CholecSeg8K subset can be evaluated zero-shot on 11 DSA classes without an explicit decoder alignment.
- domain assumption Attention U-Net with dice loss and Adam, trained with the stated hyperparameters, is representative of surgical segmentation models.
- domain assumption The selected test splits are representative of each dataset's class distribution aside from class imbalance.
read the original abstract
Effective multi-organ segmentation in surgical data requires learning the intricate anatomical features and alleviating the challenge of class imbalance, which results from relatively lower proportions of small and limitedly exposed structures. Recent works on laparoscopic multi-organ segmentation focus on learning structure-specific features through class-specific decoder architectures and report favorable results. This work extends the decoder-focused architectures to investigate knowledge sharing in the cross-surgical domain. We utilize two datasets representing different surgical domains, rectal and cholecystectomy surgeries, to explore how surgical conceptual knowledge transfers under partially common anatomical representations. Additionally, we compare the feature adaptation for the encoder and decoder at different training stages to analyse the knowledge adaptation and retention in the network. Our results corroborate previous findings on decoder-specific architectures and demonstrate that the organ-specific decoder model (CEMD), fully fine-tuned after cross-domain pre-training, achieves the highest segmentation performance (62.4\% dice) while converging substantially faster than training from scratch. However, we also find that class imbalance in surgical data remains a persistent challenge that transfer learning does not fully resolve for underrepresented anatomical structures.
Figures
Reference graph
Works this paper leans on
-
[1]
Vision techniques for anatomical structures in laparoscopic surgery: a comprehen- sive review.Frontiers in Surgery, 12:1557153, 2025
Ru Zhou, Dan Wang, Hanwei Zhang, Ying Zhu, Lijun Zhang, Tianxiang Chen, Wenqiang Liao, and Zi Ye. Vision techniques for anatomical structures in laparoscopic surgery: a comprehen- sive review.Frontiers in Surgery, 12:1557153, 2025
2025
-
[2]
Tobias Rueckert, Daniel Rueckert, and Christoph Palm. Methods and datasets for segmenta- tion of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art.Computers in Biology and Medicine, 169:107929, 2024
2024
-
[3]
Deep learning for surgical instrument recognition and segmentation in robotic- assisted surgeries: a systematic review.Artificial Intelligence Review, 58(1):1, 2024
Fatimaelzahraa Ali Ahmed, Mahmoud Yousef, Mariam Ali Ahmed, Hasan Omar Ali, Anns Mahboob, Hazrat Ali, Zubair Shah, Omar Aboumarzouk, Abdulla Al Ansari, and Shidin 15 Balakrishnan. Deep learning for surgical instrument recognition and segmentation in robotic- assisted surgeries: a systematic review.Artificial Intelligence Review, 58(1):1, 2024
2024
-
[4]
Augmenting efficient real-time surgical instrument segmentation in video with point tracking and segment anything
Zijian Wu, Adam Schmidt, Peter Kazanzides, and Septimiu E Salcudean. Augmenting efficient real-time surgical instrument segmentation in video with point tracking and segment anything. Healthcare Technology Letters, 12(1):e12111, 2025
2025
-
[5]
Segmatch: semi-supervised surgical instrument segmentation.Scientific Reports, 15(1):14042, 2025
Meng Wei, Charlie Budd, Luis C Garcia-Peraza-Herrera, Reuben Dorent, Miaojing Shi, and Tom Vercauteren. Segmatch: semi-supervised surgical instrument segmentation.Scientific Reports, 15(1):14042, 2025
2025
-
[6]
D. K. Venkatesh, D. Rivoir, M. Pfeiffer, F. Kolbinger, and S. Speidel. Data augmentation for surgical scene segmentation with anatomy-aware diffusion models. In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2280–2290, 2025. doi: 10.1109/W ACV61041.2025.00228
arXiv 2025
-
[7]
Towards more precise automatic analysis: a comprehensive survey of deep learning-based multi-organ segmentation, 2023
Xiaoyu Liu, Linhao Qu, Ziyue Xie, Jiayue Zhao, Yonghong Shi, and Zhijian Song. Towards more precise automatic analysis: a comprehensive survey of deep learning-based multi-organ segmentation, 2023
2023
-
[8]
Mosmos: Multi-organ segmentation facilitated by medical report supervision.Biomedical Signal Processing and Control, 106:107743, 2025
Weiwei Tian, Xinyu Huang, Junlin Hou, Caiyue Ren, Longquan Jiang, Rui-Wei Zhao, Gang Jin, Yuejie Zhang, and Daoying Geng. Mosmos: Multi-organ segmentation facilitated by medical report supervision.Biomedical Signal Processing and Control, 106:107743, 2025. ISSN 1746-8094
2025
-
[9]
M¨ uller-Stich, Martin Wagner, and Franziska Mathis-Ullrich
Paul Maria Scheikl, Stefan Laschewski, Anna Kisilenko, Tornike Davitashvili, Benjamin M¨ uller, Manuela Capek, Beat P. M¨ uller-Stich, Martin Wagner, and Franziska Mathis-Ullrich. Deep learning for semantic segmentation of organs and tissues in laparoscopic surgery.Current Directions in Biomedical Engineering, 6(1):20200016, 2020
2020
-
[10]
Harinder Kaur, Navjot Anttal, and Nirvair Neeru. Evolution of multiorgan segmentation techniques from traditional to deep learning in abdominal ct images – a systematic review. Displays, 73:102223, 04 2022. doi: 10.1016/j.displa.2022.102223
arXiv 2022
-
[11]
A review of deep learning based methods for medical image multi-organ segmentation.Physica Medica, 85:107–122, 2021
Yabo Fu, Yang Lei, Tonghe Wang, Walter J Curran, Tian Liu, and Xiaofeng Yang. A review of deep learning based methods for medical image multi-organ segmentation.Physica Medica, 85:107–122, 2021
2021
-
[12]
W-Y Hong, C-L Kao, Y-H Kuo, J-R Wang, W-L Chang, and C-S Shih. Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80.arXiv preprint arXiv:2012.12453, 2020
Pith/arXiv arXiv 2012
-
[13]
Matthias Carstens, Franziska Rinner, Sebastian Bodenstedt, Alexander Jenke, J¨ urgen Weitz, et al. The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science.Scientific Data, 10, 01 2023. doi: 10.1038/s41597-022-01719-2. 16
-
[14]
Strategies to improve real-world applicability of laparoscopic anatomy segmentation models
Fiona R Kolbinger, Jiangpeng He, Jinge Ma, and Fengqing Zhu. Strategies to improve real-world applicability of laparoscopic anatomy segmentation models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2275–2284, 2024
2024
-
[15]
Fiona R. Kolbinger, Franziska M. Rinner, Alexander C. Jenke, Matthias Carstens, Stefan Leger, et al. Anatomy segmentation in laparoscopic surgery: Comparison of machine learning and human expertise.medRxiv, 2023. doi: 10.1101/2022.11.11.22282215
-
[16]
One model to use them all: training a segmentation model with com- plementary datasets.International journal of computer assisted radiology and surgery, 19(6): 1233–1241, 2024
Alexander C Jenke, Sebastian Bodenstedt, Fiona R Kolbinger, Marius Distler, J¨ urgen Weitz, and Stefanie Speidel. One model to use them all: training a segmentation model with com- plementary datasets.International journal of computer assisted radiology and surgery, 19(6): 1233–1241, 2024
2024
-
[17]
Effective disjoint representational learning for anatomical segmentation
Priya Tomar, Aditya Parikh, Philipp Feodorovici, Jan Arensmeyer, Hanno Matthaei, Christian Bauckhage, Helen Schneider, and Rafet Sifa. Effective disjoint representational learning for anatomical segmentation. InMedical Imaging with Deep Learning, 2025
2025
-
[18]
Efficient anatomy segmentation in laparoscopic surgery using multi-teacher knowledge distillation
Lennart Maack, Finn Behrendt, Debayan Bhattacharya, Sarah Latus, and Alexander Schlae- fer. Efficient anatomy segmentation in laparoscopic surgery using multi-teacher knowledge distillation. InMedical Imaging with Deep Learning, 2024
2024
-
[19]
Towards more precise automatic analysis: a systematic review of deep learning-based multi-organ seg- mentation.BioMedical Engineering OnLine, 23(1):52, 2024
Xiaoyu Liu, Linhao Qu, Ziyue Xie, Jiayue Zhao, Yonghong Shi, and Zhijian Song. Towards more precise automatic analysis: a systematic review of deep learning-based multi-organ seg- mentation.BioMedical Engineering OnLine, 23(1):52, 2024
2024
-
[20]
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation.arXiv preprint arXiv:2102.04306, 2021
Pith/arXiv arXiv 2021
-
[21]
Advancements and challenges in medical image segmentation: A comprehensive survey.Sensors and AI, pages 3–29, 2025
Guanqiu Qi, Zhiqin Zhu, Ke Li, and Han Xiao. Advancements and challenges in medical image segmentation: A comprehensive survey.Sensors and AI, pages 3–29, 2025
2025
-
[22]
Improving surgical scene seman- tic segmentation through a deep learning architecture with attention to class imbalance
Claudio Urrea, Yainet Garcia-Garcia, and John Kern. Improving surgical scene seman- tic segmentation through a deep learning architecture with attention to class imbalance. Biomedicines, 12(6):1309, 2024
2024
-
[23]
Warfield, and Ali Gholipour
Davood Karimi, Simon K. Warfield, and Ali Gholipour. Transfer learning in medical image segmentation: New insights from analysis of the dynamics of model parameters and learned representations.Artificial Intelligence in Medicine, 116:102078, 2021. ISSN 0933-3657
2021
-
[24]
Curran Associates Inc., Red Hook, NY, USA, 2019
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio.Transfusion: understanding transfer learning for medical imaging. Curran Associates Inc., Red Hook, NY, USA, 2019
2019
-
[25]
Critical assessment of transfer learning for medical image segmentation with fully convolutional neural networks, 05 2020
Davood Karimi, Simon Warfield, and Ali Gholipour. Critical assessment of transfer learning for medical image segmentation with fully convolutional neural networks, 05 2020. 17
2020
-
[26]
Jumpstarting surgical computer vision
Deepak Alapatt, Aditya Murali, Vinkle Srivastav, AI4SafeChole Consortium, Pietro Mascagni, and Nicolas Padoy. Jumpstarting surgical computer vision. InMedical Image Computing and Computer Assisted Intervention – MICCAI 2024, pages 328–338. Springer, 2024
2024
-
[27]
Jaspers, Ronald L.P.D
Tim J.M. Jaspers, Ronald L.P.D. de Jong, Yiping Li, Carolus H.J. Kusters, Franciscus H.A. Bakker, Romy C. van Jaarsveld, Gino M. Kuiper, Richard van Hillegersberg, Jelle P. Ruurda, Willem M. Brinkman, Josien P.W. Pluim, Peter H.N. de With, Marcel Breeuwer, Yasmina Al Khalil, and Fons van der Sommen. Scaling up self-supervised learning for improved surgica...
2026
-
[28]
Efficient generative-adversarial u-net for multi-organ medical image segmentation.Journal of Imaging, 11(1):19, 2025
Haoran Wang, Gengshen Wu, and Yi Liu. Efficient generative-adversarial u-net for multi-organ medical image segmentation.Journal of Imaging, 11(1):19, 2025
2025
-
[29]
A unified loss for handling inter-class and intra-class imbalance in medical image segmentation
Fei Xu, Fan Yang, Xinghui Li, and Xiaofeng Zhang. A unified loss for handling inter-class and intra-class imbalance in medical image segmentation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8842–8850, 2025
2025
-
[30]
Endonet: a deep architecture for recognition tasks on laparoscopic videos
Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging, 36(1):86–97, 2016
2016
-
[31]
Ozan Oktay, Jo Schlemper, Lo ¨ ıc Le Folgoc, Matthew C. H. Lee, Mattias P. Heinrich, et al. Attention u-net: Learning where to look for the pancreas.CoRR, abs/1804.03999, 2018. 18
Pith/arXiv arXiv 2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.