REVIEW 4 major objections 5 minor 24 references
Sim2Real in endoscopy segmentation with a novel structure aware image translation
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Adding depth consistency to CycleGAN preserves colon fold structure and lifts real fold segmentation by 11.66 mean-IoU points.
desk verdict A useful new benchmark for colonoscopy fold segmentation and a plausible sim2real pipeline, but the depth-consistency mechanism is under-validated and the novelty claim overlaps with SPIGAN. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the depth-consistency loss of Eq. (1), a scale-invariant log-depth error between the depth predicted by a fixed pretrained estimator on the translated image and the ground-truth depth of the simulated input. Added to the CycleGAN objective, this loss penalizes the generator whenever texture transfer moves surfaces closer or farther in depth, so geometric ridges such as colon folds are preserved. A second component is the paired training structure, where the downstream segmentation model sees batches containing both original and translated simulated images, which compensates for residual distortion introduced by translation.
What would settle it
On the released benchmark, compare the depth predicted from a translated image against the ground-truth depth of its simulated source; if the predicted depth is systematically wrong in fold regions, or if replacing the depth estimator with a deliberately poor one still yields the same 11.66-point IoU gain, then the depth-consistency term is not the mechanism responsible for the improvement.
Extended reading notes
Core claim
The central claim is that a CycleGAN-based translation model supervised with a scale-invariant depth-consistency loss can add realistic texture to simulated endoscopy images while preserving the shape and location of anatomical structures, and that the resulting images are sufficient to train a fold-segmentation model that generalizes to real colonoscopy data without any real labels. In the authors' reported experiments, their full pipeline reaches 44.3 mean IoU on the EM test set, an 11.66-point improvement over the best FoldIt variant, with the depth-consistency translation and paired-batch training each contributing measurable gains.
Load-bearing premise
The method assumes the fixed pretrained depth estimator is trustworthy on CycleGAN-translated images; if its depth predictions on translated images are biased, the depth-consistency loss enforces the wrong geometry and the claimed fold preservation may not hold.
Editorial extensions
If this is right
- Fold segmentation on real colonoscopy can be trained with zero real annotations, using only automatically labeled simulated frames processed by the proposed translation approach.
- The released simulated benchmark with ground-truth fold masks and depth maps provides a public testbed for fold segmentation, which previously lacked one.
- The same depth-consistency translation recipe transfers to other endoscopy styles, such as NBI lighting and laparoscopy, where it also reduces CycleGAN structure distortion.
- Because the downstream model trains only on simulated data, the approach can be extended to other anatomical landmarks with automatic labels, such as polyps or lesions.
- In the paper's ablations, adding the depth-consistency translation and the paired-batch structure each improve IoU beyond the already stronger EndoFM-TransUNet baseline.
Reading between the lines
- The depth-consistency mechanism is only as good as the fixed depth estimator, so the obvious next check is to validate depth accuracy on translated images; a biased estimator could silently enforce wrong geometry while downstream IoU still improves for unrelated reasons.
- Because the method only needs a simulator that can render depth, it likely generalizes as a structure-preserving sim-to-real recipe beyond colonoscopy, for example to laparoscopy or capsule endoscopy.
- A direct measurement of fold-shape preservation, such as mask IoU between the original simulated label and the label re-projected on the translated image, would isolate how much of the downstream gain comes from translation fidelity versus the stronger segmentation backbone.
- The paired-batch training suggests some residual distortion remains even with the depth loss, so adapting the loss over training or adding a local shape-consistency term could close the remaining gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a CycleGAN-based image translation method that adds a depth-consistency loss (Eq. 1) to preserve scene structure (specifically the shape and location of haustral folds) when transferring realistic texture from real colonoscopy images (EndoMapper) to simulated images (VR-CAPS). The translated images, optionally mixed with the original simulated images, are then used to train an EndoFM-TransUNet model for fold segmentation. The authors also release a new simulated benchmark with ground-truth fold labels and depth maps, plus manual annotations on 100 real EndoMapper test frames. In the main comparison, the best configuration reaches 44.3 mean IoU on the EM test set versus 32.64 for the best FoldIt baseline, an absolute difference of 11.66 points. Additional qualitative results are shown for NBI colonoscopy and laparoscopy style transfer.
Significance. If the mechanism is substantiated, the paper would provide a practical and data-efficient recipe for sim2real in endoscopy: a lightweight depth-consistency term that prevents the structural distortion typical of CycleGAN, plus a strong segmentation backbone, allowing training on synthetic data alone to transfer to real images. The strongest evidence is the within-model comparison under EndoFM-TransUNet: the proposed translation (42.5 IoU) clearly outperforms CycleGAN (26.9) and I2I (37.6) when training only on translated images, indicating that the depth loss brings a real benefit. The release of the simulated benchmark, ground-truth depth maps, and EM metadata is a valuable community contribution. However, the paper does not yet directly validate the depth estimator on the translated distribution, does not compare against the closest prior work (SPIGAN), and does not provide significance testing, so the central claim that depth consistency is what preserves structure remains partially unsupported.
major comments (4)
- [Sec. 3, Eq. (1)] The depth-consistency loss relies on a fixed pretrained monocular depth estimator (SCSfM-Learner pretrained on VR-CAPS) to judge whether the translated image y preserves the geometry of the simulated input x. The paper provides no evidence that this estimator produces reliable relative depth on the translated-image distribution, which contains realistic textures, lighting, and artifacts that the estimator may not have seen. A direct validation is required: for example, compute a scale-invariant depth error between the predicted depth of the translated image and the ground-truth depth of the corresponding simulated image, and report it for each translation baseline (CycleGAN, I2I, Ours). Without such a check, the central claim that Eq. (1) is what preserves fold shape and location remains unsupported.
- [Sec. 2 and Table 2] The most closely related prior work, SPIGAN [12], also uses depth estimation to maintain scene layout during CycleGAN-based sim2real translation and trains a segmentation model jointly. It is discussed in Related Work but never compared experimentally. Since the paper's novelty centers on the depth-consistency loss, a comparison to SPIGAN, or at least a principled explanation of why it is not applicable to this setting, is necessary to position the contribution and rule out that the proposed loss is a re-implementation of an existing idea.
- [Table 2 and Sec. 4.2] The reported means and standard deviations are over test frames, not over multiple training runs, and no significance testing is performed. The differences between, e.g., Sim-Aug. Ours (42.5 ± 11.59) and Sim-Aug. & Sim. Ours (44.3 ± 11.47) are within one standard deviation; it is unclear whether the 1.8-point gain from the paired-batch strategy is meaningful. The authors should provide confidence intervals, paired tests across multiple seeds, or per-run scatter plots to support the claim of significant improvement over baselines.
- [Sec. 4.2 and Table 2] The headline comparison between the proposed pipeline and FoldIt conflates two change factors: the segmentation architecture (EndoFM-TransUNet vs. FoldIt) and the translation method. The stated 11.66% improvement is an absolute difference on the EM test set (44.3 vs. 32.64) between different models trained on different data. The paper should clearly separate the within-model gain produced by the translation (4.6 points over the no-translation EndoFM-TransUNet baseline) from the cross-model gain, and should also isolate the paired-batch effect by reporting a single ablation that varies only whether the Sim. set is included alongside Sim-Aug.
minor comments (5)
- [Tables 1 and 2] The table captions contain a formatting error: 'T able' should be 'Table'.
- [Sec. 4.2] The sentence 'We show an overall 11.66% improvement ... of which 4.6% is the result of our proposed image translation' should specify that these are absolute percentage points, not relative improvements.
- [Sec. 3] The weight given to the depth-consistency loss (lambda_depth) in the total CycleGAN objective is never reported. Since this is the key free parameter of the method, its value and ideally a sensitivity analysis should be provided for reproducibility.
- [Fig. 5(b)] The comparison with I2I from [14] uses different training data sizes (the caption says I2I was trained on 20,000 simulated and 74,000 real images, while Ours uses 2,000 simulated and 2,840 real). The text should explicitly state that this is a data-efficiency comparison, as the qualitative parity in the figure could otherwise be misread.
- [Abstract and Sec. 4.1] The claim that the method 'produces realistic images in different endoscopy scenarios' is supported only qualitatively for NBI colonoscopy and laparoscopy; the quantitative evaluation is limited to fold segmentation in white-light colonoscopy. The authors should either soften this claim or provide a quantitative metric for the additional scenarios.
Circularity Check
No significant circularity: the depth-consistency loss uses a fixed external depth estimator plus simulated ground-truth depth, and the downstream segmentation is evaluated on manually labeled held-out real frames.
full rationale
The paper's central claim is that a depth-consistency loss added to CycleGAN preserves scene structure during simulated-to-real image translation, and that training a segmentation model on the translated images improves fold segmentation on real colonoscopy data. The derivation chain is not circular. Equation (1) computes a scale-invariant log-depth loss between the depth predicted by a fixed pretrained network (SCSfM-Learner or DepthAnything) on the translated image y and the ground-truth simulated depth d* of the source image x; no parameter is fitted to the downstream test set, and the depth estimator is external, not derived from the paper's own claims. The downstream segmentation model is trained only on simulated images (original and translated) and evaluated on 100 manually annotated real EndoMapper frames that are held out from training, so the reported IoU gains are independent evidence. The only self-citation is the EndoMapper dataset [1], for which the authors overlap, but that citation is data provenance rather than a load-bearing unverified result, and the new test annotations are produced by the paper. The skeptic's concern that the pretrained depth estimator may be unreliable on translated images is a legitimate correctness risk, not a circularity: it questions whether the loss is effective, not whether it reduces by construction to its inputs.
Assumptions & free parameters
free parameters (2)
- depth loss weight (lambda_depth) =
not reported
- paired-batch ratio =
1:1 implied
assumptions (3)
- domain assumption Folds correspond to areas where the principal curvature direction changes in the VR-CAPS mesh, and this automatic labeling is correct.
- domain assumption The pretrained depth estimator (SCSFM-Learner pretrained on VR-CAPS, or DepthAnything) produces reliable depth on CycleGAN-translated images.
- domain assumption EndoFM pretrained features transfer to fold segmentation.
Cite this review
Pith. "Pith review of Sim2Real in endoscopy segmentation with a novel structure aware image translation." pith.science (2026). https://pith.science/paper/POW3NS7X
@misc{pith2026250502654,
author = {Pith},
title = {Pith review of: Sim2Real in endoscopy segmentation with a novel structure aware image translation},
year = {2026},
howpublished = {\url{https://pith.science/paper/POW3NS7X}},
note = {Machine review of arXiv:2505.02654}
}
read the original abstract
Automatic segmentation of anatomical landmarks in endoscopic images can provide assistance to doctors and surgeons for diagnosis, treatments or medical training. However, obtaining the annotations required to train commonly used supervised learning methods is a tedious and difficult task, in particular for real images. While ground truth annotations are easier to obtain for synthetic data, models trained on such data often do not generalize well to real data. Generative approaches can add realistic texture to it, but face difficulties to maintain the structure of the original scene. The main contribution in this work is a novel image translation model that adds realistic texture to simulated endoscopic images while keeping the key scene layout information. Our approach produces realistic images in different endoscopy scenarios. We demonstrate these images can effectively be used to successfully train a model for a challenging end task without any real labeled data. In particular, we demonstrate our approach for the task of fold segmentation in colonoscopy images. Folds are key anatomical landmarks that can occlude parts of the colon mucosa and possible polyps. Our approach generates realistic images maintaining the shape and location of the original folds, after the image-style-translation, better than existing methods. We run experiments both on a novel simulated dataset for fold segmentation, and real data from the EndoMapper (EM) dataset. All our new generated data and new EM metadata is being released to facilitate further research, as no public benchmark is currently available for the task of fold segmentation.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[12]
arXiv preprint arXiv:1810.03756 (2018)
Lee, K.H., Ros, G., Li, J., Gaidon, A.: Spigan: Privileged adversarial learning from simulation. arXiv preprint arXiv:1810.03756 (2018)
arXiv 2018
-
[1]
Azagra, P., Sostres, C., Ferr´ andez, ´A., Riazuelo, L., Tomasini, C., Barbed, O.L., Morlana, J., Recasens, D., Batlle, V.M., G´ omez-Rodr´ ıguez, J.J., Elvira, R., L´ opez, 10 Clara Tomasini, Luis Riazuelo, and Ana C. Murillo S T S T S T Fig. 7. Examples of simulated colonoscopy images processed with our image translation approach to have realistic textu...
work page 2023
-
[2]
Bian, J.W., Zhan, H., Wang, N., Li, Z., Zhang, L., Shen, C., Cheng, M.M., Reid, I.: Unsupervised scale-consistent depth learning from video. Int. Journal of Computer Vision 129(9), 2548–2564 (2021)
work page 2021
-
[3]
Scientific data 7(1), 1–14 (2020)
Borgli, H., Thambawita, V., Smedsrud, P.H., Hicks, S., Jha, D., Eskeland, S.L., Randel, K.R., Pogorelov, K., Lux, M., Nguyen, D.T.D., Johansen, D., Griwodz, C., Stensland, H.K., Garcia-Ceja, E., Schmidt, P.T., Hammer, H.L., Riegler, M.A., Halvorsen, P., de Lange, T.: Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal end...
work page 2020
-
[4]
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.: Transunet: Transformers make strong encoders for medical image segmentation. arXiv:2102.04306 (2021)
arXiv 2021
-
[5]
In: Proceedings of the IEEE/CVF Conf
Deng, Y., Tang, F., Dong, W., Ma, C., Pan, X., Wang, L., Xu, C.: Stytr2: Im- age style transfer with transformers. In: Proceedings of the IEEE/CVF Conf. on computer vision and pattern recognition. pp. 11326–11336 (2022)
work page 2022
-
[6]
In: Proceedings of the IEEE Int
Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE Int. Conf. on computer vision. pp. 2650–2658 (2015)
work page 2015
-
[7]
In: Proceedings of the IEEE Int
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE Int. Conf. on computer vision. pp. 1501–1510 (2017)
work page 2017
Show all 24 references
-
[8]
In: Proceedings of the European Conf
Huang, X., Liu, M.Y., Belongie, S., Kautz, J.: Multimodal unsupervised image- to-image translation. In: Proceedings of the European Conf. on computer vision (ECCV). pp. 172–189 (2018)
2018
-
[9]
Incetan, K., Celik, I.O., Obeid, A., Gokceler, G.I., Ozyoruk, K.B., Almalioglu, Y., Chen, R.J., Mahmood, F., Gilbert, H., Durr, N.J., Turan, M.: Vr-caps: A virtual environment for capsule endoscopy (2020)
2020
-
[10]
In: MICCAI Workshop on Data Engineering in Medical Imaging
Jin, W., Daher, R., Stoyanov, D., Vasconcelos, F.: A self-supervised approach for detecting the edges of haustral folds in colonoscopy video. In: MICCAI Workshop on Data Engineering in Medical Imaging. pp. 56–66 (2023)
2023
-
[11]
Proceedings of the IEEE/CVF Int
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Doll´ ar, P., Girshick, R.: Segment anything. Proceedings of the IEEE/CVF Int. Conf. on Computer Vision pp. 4015–4026 (2023)
2023
-
[13]
Mathew, S., Nadeem, S., Kaufman, A.: Foldit: Haustral folds detection and seg- mentation in colonoscopy videos. In: Int. Conf. on Medical Image Computing and Computer-Assisted Intervention. pp. 221–230 (2021)
2021
-
[14]
Pfeiffer, M., Funke, I., Robu, M.R., Bodenstedt, S., Strenger, L., Engelhardt, S., Roß, T., Clarkson, M.J., Gurusamy, K., Davidson, B.R., Maier-Hein, L., Riediger, C., Welsch, T., Weitz, J., Speidel, S.: Generating large labeled data sets for la- paroscopic image processing ta...
2019
-
[15]
Annals of Internal Medicine 141, 352– 359 (2004) Sim2Real in endoscopy segmentation 13
Pickhardt, P.J., Nugent, P.A., Mysliwiec, P.A., Choi, J., Schindler, W.R.: Location of adenomas missed by optical colonoscopy. Annals of Internal Medicine 141, 352– 359 (2004) Sim2Real in endoscopy segmentation 13
2004
-
[16]
Sahu, M., Str¨ omsd¨ orfer, R., Mukhopadhyay, A., Zachow, S.: Endo-sim2real: Con- sistency learning-based domain adaptation for instrument segmentation. In: Int. Conf. on Medical Image Computing and Computer-Assisted Intervention. pp. 784– 794 (2020)
2020
-
[17]
the cancer imaging archive (2015)
Smith, K., Clark, K., Bennett, W., Nolan, T., Kirby, J., Wolfsberger, M., Moulton, J., Vendt, B., Freymann, J.: Data from ct colonography. the cancer imaging archive (2015)
2015
-
[18]
IEEE transactions on medical imaging 36(1), 86–97 (2016)
Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging 36(1), 86–97 (2016)
2016
-
[19]
Wang, Z., Liu, C., Zhang, S., Dou, Q.: Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. In: Int. Conf. on Medical Image Computing and Computer-Assisted Intervention. pp. 101–111 (2023)
2023
-
[20]
In: CVPR (2024)
Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., Zhao, H.: Depth anything: Un- leashing the power of large-scale unlabeled data. In: CVPR (2024)
2024
-
[21]
In: Proceedings of the IEEE/CVF Conf
Zhang, Y., Huang, N., Tang, F., Huang, H., Ma, C., Dong, W., Xu, C.: Inversion- based style transfer with diffusion models. In: Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. pp. 10146–10156 (2023)
2023
-
[22]
IEEE Transactions on Biomedical Engineering 60(2), 321–331 (2012)
Zhu, H., Barish, M., Pickhardt, P., Liang, Z.: Haustral fold segmentation with curvature-guided level set evolution. IEEE Transactions on Biomedical Engineering 60(2), 321–331 (2012)
2012
-
[23]
In: Virtual Colonoscopy and Abdominal Imaging
Zhu, H., Li, L., Fan, Y., Liang, Z.: Haustral fold segmentation of ct colonogra- phy using ridge line detection. In: Virtual Colonoscopy and Abdominal Imaging. Computational Challenges and Clinical Opportunities. pp. 24–31 (2011)
2011
-
[24]
In: Proceedings of the IEEE Int
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE Int. Conf. on computer vision. pp. 2223–2232 (2017)
2017
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.