REVIEW 4 major objections 4 minor 23 references
Benchmarking and Enhancing Surgical Phase Recognition Models for Robotic-Assisted Esophagectomy
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A causal transformer with hierarchical attention outperforms four prior models on a new surgical phase recognition benchmark for robotic esophagectomy.
desk verdict A useful new RAMIE phase-recognition dataset and a plausible causal-transformer adaptation, but the AutoLaparo comparison rests on literature baselines, so the unqualified superiority claim is not yet earned. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a causal transformer with one encoder and three decoders, each with ten blocks built from causal dilated convolutions and masked self-attention. At layer $l$, the query tensor $Q_l \in \mathbb{R}^{T_l \times d \times h_l}$ and the key tensor $K_l \in \mathbb{R}^{T_l \times d \times 2h_l}$ operate on a halved sequence length with doubled head dimension, and a causal mask forces each position to attend only to earlier frames. The mechanism is adapted from ASformer, an action-segmentation transformer, and it is coupled with a loss that combines per-frame cross-entropy with a clamped temporal smoothness penalty to reduce fragmented phase predictions.
What would settle it
Re-run SV-RCNet, TeCNO, TMRNet, and Trans-SVNet with the same training/validation/test splits and the same evaluation code on both RAMIE and AutoLaparo; if the reported Edit Score gaps shrink or reverse, the claimed superiority is an artifact of protocol mismatch.
Extended reading notes
Core claim
The central claim is that adding a causal mask to hierarchical self-attention in an encoder-decoder transformer substantially improves phase recognition on procedures with repetitive phases and non-sequential transitions. On the new RAMIE dataset, the model reaches 78.28 accuracy and an Edit Score of 59.50, far above the compared baselines; on AutoLaparo it reports 83.18 accuracy versus 78.29 for Trans-SVNet. The paper attributes the gains to causal modeling of temporal order, multi-layer decoder refinement, and a smoothing loss that suppresses over-segmentation.
Load-bearing premise
The AutoLaparo comparison assumes the baseline numbers reproduced from the original paper were computed under the same metric implementation and data split, since those baselines were not re-run with the authors' evaluation code.
Editorial extensions
If this is right
- If the causal architecture is the reason for the gains, temporal causality should become a standard design choice for phase-recognition models on non-sequential surgical workflows.
- The RAMIE dataset and its 13-phase annotation scheme give the community a benchmark for procedures with phase-order variability, where existing benchmarks are largely sequential.
- Further improvements should target phase boundaries and short phases such as right pleural and aortopulmonary lymph node dissection, where most classification errors concentrate.
- Segment-level metrics like Edit Score and F1@tau should be reported alongside frame accuracy, because frame accuracy alone did not reflect the model's large improvement.
Reading between the lines
- Editorial inference: If the AutoLaparo results hold under a fully re-run protocol, the same causal design may transfer to other multi-phase laparoscopic procedures with non-canonical phase orders, such as complex hysterectomy variants.
- Editorial inference: The reported sensitivity to phase boundaries suggests an extension that predicts transition points explicitly, or that weights the loss by phase risk, could push clinically useful accuracy further.
- Editorial inference: Because the RAMIE labels come from a single annotator, a natural next step is a multi-annotator study to measure how much of the performance gap is annotation noise rather than model behavior.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a new surgical phase recognition dataset for the thoracic phase of robotic-assisted minimally invasive esophagectomy (RAMIE), comprising 27 videos with 13 annotated phases and a 14/4/9 train/validation/test split. Using this dataset, the authors benchmark four existing surgical phase recognition models (SV-RCNet, TMRNet, TeCNO, Trans-SVNet) and propose a new temporal model: an encoder-decoder transformer with causal hierarchical attention, adapted from ASFormer, trained with a cross-entropy plus smoothing loss. On RAMIE, the proposed model achieves large improvements in Edit score and F1@25/50/75 over the baselines, but its accuracy is comparable to or slightly lower than TeCNO. The paper also reports results on the public AutoLaparo dataset, where the proposed model's metrics exceed numbers taken from the original AutoLaparo paper. The authors discuss remaining challenges such as over-segmentation near phase transitions, class imbalance, and limitations of current evaluation metrics.
Significance. If the empirical results hold, the paper makes a useful contribution by providing a new dataset for a surgical procedure with complex, non-sequential phase transitions, and by showing that a causal transformer variant can substantially improve temporal segmentation metrics on that dataset. Strengths include the use of the Funke et al. evaluation codebase for all RAMIE experiments, the comparison of several baseline models run from original author code, the inclusion of Edit score and segment-level F1 metrics in addition to frame-level accuracy, and the qualitative analysis of error patterns. However, the significance is currently limited by the lack of ablations supporting the attribution of gains to the causal mechanism, by an uncontrolled AutoLaparo baseline comparison that relies on literature numbers, and by the small, single-annotator RAMIE test set. These issues are fixable and do not undermine the dataset contribution, but they must be addressed before the central claim of superiority can be accepted.
major comments (4)
- [§3.5, Table 3] The AutoLaparo superiority claim is not supported by controlled evidence. Table 3 states that baseline results are taken from [12] rather than re-run under the evaluation protocol used for the proposed model (Funke et al. codebase, §3.4). Because metric definitions, train/validation splits, preprocessing, and feature extractors differ across surgical-phase papers, the baseline numbers in [12] are not directly comparable to the proposed model's scores. Moreover, §3.1.2 says the AutoLaparo split is 10/4/9 videos, which sums to 23 rather than the stated 21, and gives the same frame count (12,056) for validation and test; this internal inconsistency must be resolved. Re-running SV-RCNet, TMRNet, TeCNO, and Trans-SVNet under the paper's protocol, or at minimum reporting the exact split and metric implementations with baseline numbers, is necessary before claiming superiority on this dataset.
- [§4; §3.3.2] The Discussion attributes the observed gains to the causal hierarchical attention mechanism and the smoothing loss, but no ablation isolates these components. The paper does not compare the proposed causal encoder-decoder against its non-causal ASFormer counterpart, nor against variants with λ=0 in Eq. (1), nor against configurations with different numbers of decoder layers. Without such ablations, the central claim that the causal design is responsible for the large Edit/F1 improvements on RAMIE is an unsupported attribution. Adding at least a causal-vs-non-causal comparison and a smoothing-loss on/off comparison would strengthen the paper's main conclusion.
- [§3.1.1; Tables 1 and 2] The RAMIE benchmark has limited statistical grounding. The ground truth was produced by a single annotator (a PhD student guided by a medical PhD student and surgeon), with no inter-annotator agreement reported, which is important for a dataset intended as a community benchmark. The test set contains only 9 videos, and several metrics in Table 1 overlap within one standard deviation (e.g., Accuracy: Ours 78.28 ± 4.42 vs. TeCNO 78.46 ± 3.97; Jaccard: Ours 61.94 ± 7.24 vs. TeCNO 58.34 ± 4.75). The paper reports no per-video significance tests. The large Edit/F1 margins in Table 2 are encouraging, but the broad claim of superiority should be restricted to metrics where the gap is clear, or supported by significance testing.
- [Abstract; §3.4] The abstract's unqualified statement that the proposed model 'demonstrates superior performance compared to existing models' is not consistent with all reported metrics. On the RAMIE dataset, the proposed model's Accuracy (78.28) is numerically lower than TeCNO's (78.46) in Table 1. The claim should be qualified to temporal segmentation metrics (Edit score, F1@τ), or the accuracy result should be discussed explicitly; otherwise the headline claim overreaches the evidence.
minor comments (4)
- [§3.3.2, Eq. (1)-(2)] The loss notation is confusing: Eq. (1) uses H(S(p_{y_t,t}), y_t), but cross-entropy is written as if between a probability vector and a label without a one-hot encoding, and Eq. (2) defines Δ_t per class c while Eq. (1) applies clamp(Δ_t², 0, 16) without a class subscript. Please clarify the dimensions and define the softmax subscript.
- [§3.2, §3.4] The text says baselines were implemented from original author code 'maintaining all original settings', but §3.4 says all models were evaluated with the Funke et al. codebase. Please clarify how the original settings interact with the Funke evaluation harness, especially regarding input preprocessing and feature extraction.
- [Table 3] The baseline rows in Table 3 report only means, while the proposed model reports mean ± standard deviation. If the baseline numbers must remain from [12], at least state this explicitly in the text and note that standard deviations are unavailable for the baselines.
- [Throughout] There are minor consistency issues in notation and naming: 'ASformer' and 'ASFormer' are used interchangeably, 'Trans-SVnet' appears in Tables 1-2 while 'Trans-SVNet' is used elsewhere, and Figure 3's l, T_l, h_l notation is defined only in the caption. A consistent notation pass would improve readability.
Circularity Check
No significant circularity: the paper's empirical claims rest on external baselines, a new dataset, and a transparently cited architecture adaptation.
full rationale
The manuscript makes an empirical claim: a causal hierarchical attention encoder-decoder outperforms four external baselines on a new RAMIE dataset and on the public AutoLaparo dataset. The derivation chain is not definitional. The proposed model is a neural architecture adapted transparently from ASFormer [17], an external prior work with no author overlap; the loss (cross-entropy plus a smoothness penalty) is standard and its hyperparameter is fixed before evaluation. No fitted parameter is renamed as a prediction, and no equation defines the reported metric in terms of the model's own training objective. On the RAMIE dataset, the comparison is plausibly controlled: Section 3.2 states the baselines were implemented from original author code, and Section 3.4 states all models were evaluated with the Funke et al. codebase, so the large Edit/F1 gaps in Tables 1 and 2 are internally meaningful. The AutoLaparo comparison in Table 3 does rely on baseline numbers taken from [12] rather than re-runs under the paper's own protocol, which is a legitimate validity concern about comparability of metric implementations, splits, and preprocessing; however, that is an external-baseline provenance issue, not circularity. There are no load-bearing self-citations, no uniqueness theorem imported from the authors, and no renamed known result. The central claim therefore has independent empirical content, and no circular step can be exhibited from the paper's text.
Assumptions & free parameters
free parameters (1)
- smooth loss weight lambda =
0.15
assumptions (4)
- domain assumption Ground truth phase annotations are accurate and consistent
- domain assumption The AutoLaparo baseline results from [12] are directly comparable to the authors' own evaluation
- domain assumption Two-stage training with frozen ResNet features is sufficient
- ad hoc to paper The causal transformer is a faithful adaptation of ASFormer's hierarchical attention
Cite this review
Pith. "Pith review of Benchmarking and Enhancing Surgical Phase Recognition Models for Robotic-Assisted Esophagectomy." pith.science (2026). https://pith.science/paper/IY6M4IV4
@misc{pith2026241204039,
author = {Pith},
title = {Pith review of: Benchmarking and Enhancing Surgical Phase Recognition Models for Robotic-Assisted Esophagectomy},
year = {2026},
howpublished = {\url{https://pith.science/paper/IY6M4IV4}},
note = {Machine review of arXiv:2412.04039}
}
read the original abstract
Robotic-assisted minimally invasive esophagectomy (RAMIE) is a recognized treatment for esophageal cancer, offering better patient outcomes compared to open surgery and traditional minimally invasive surgery. RAMIE is highly complex, spanning multiple anatomical areas and involving repetitive phases and non-sequential phase transitions. Our goal is to leverage deep learning for surgical phase recognition in RAMIE to provide intraoperative support to surgeons. To achieve this, we have developed a new surgical phase recognition dataset comprising 27 videos. Using this dataset, we conducted a comparative analysis of state-of-the-art surgical phase recognition models. To more effectively capture the temporal dynamics of this complex procedure, we developed a novel deep learning model featuring an encoder-decoder structure with causal hierarchical attention, which demonstrates superior performance compared to existing models.
Figures
Reference graph
Works this paper leans on
-
[12]
Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, and Yunhui Liu. Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 486–
-
[1]
Global cancer statistics 2022: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries
Freddie Bray, Mathieu Laversanne, Hyuna Sung, Jacques Ferlay, Rebecca L Siegel, Isabelle Soerjomataram, and Ahmedin Jemal. Global cancer statistics 2022: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians, 74(3):229–263, 2024
2022
-
[2]
Hans F Fuchs, Justin W Collins, Benjamin Babic, Christopher DuCoin, Ozanan R Meireles, Peter P Grim- minger, Matthew Read, Abbas Abbas, Rubens Sallum, Beat P M¨ uller-Stich, et al. Robotic-assisted mini- mally invasive esophagectomy (ramie) for esophageal cancer training curriculum—a worldwide delphi con- sensus study. Diseases of the Esophagus, 35(6):doa...
work page 2022
-
[3]
Pieter C van der Sluis, Jelle P Ruurda, Sylvia van der Horst, Lucas Goense, and Richard van Hillegersberg. Learning curve for robot-assisted minimally invasive thoracoscopic esophagectomy: results from 312 cases. The Annals of Thoracic Surgery, 106(1):264–271, 2018
work page 2018
-
[4]
RB Den Boer, TJM Jaspers, C De Jongh, JPW Pluim, F Van Der Sommen, T Boers, R van Hillegersberg, MAJM Van Eijnatten, and JP Ruurda. Deep learning-based recognition of key anatomical structures during robot-assisted minimally invasive esophagectomy. Surgical endoscopy, 37(7):5164–5175, 2023
work page 2023
-
[5]
Kazuma Sato, Takeo Fujita, Hiroki Matsuzaki, Nobuyoshi Takeshita, Hisashi Fujiwara, Shuichi Mitsunaga, Takashi Kojima, Kensaku Mori, and Hiroyuki Daiko. Real-time detection of the recurrent laryngeal nerve in thoracoscopic esophagectomy using artificial intelligence. Surgical Endoscopy, 36(7):5531–5539, 2022
work page 2022
-
[6]
Masashi Takeuchi, Hirofumi Kawakubo, Kosuke Saito, Yusuke Maeda, Satoru Matsuda, Kazumasa Fukuda, Rieko Nakamura, and Yuko Kitagawa. Automated surgical-phase recognition for robot-assisted minimally invasive esophagectomy using artificial intelligence. Annals of Surgical Oncology, 29(11):6847–6855, 2022
work page 2022
-
[7]
Tecno: Surgical phase recognition with multi-stage temporal convolutional net- works
Tobias Czempiel, Magdalini Paschali, Matthias Keicher, Walter Simson, Hubertus Feussner, Seong Tae Kim, and Nassir Navab. Tecno: Surgical phase recognition with multi-stage temporal convolutional net- works. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings,...
work page 2020
Show all 23 references
-
[8]
Active learning for extracting surgomic features in robot-assisted minimally invasive esophagectomy: a prospective annotation study
Johanna M Brandenburg, Alexander C Jenke, Antonia Stern, Marie TJ Daum, Andr´ e Schulze, Rayan Younis, Philipp Petrynowski, Tornike Davitashvili, Vincent Vanat, Nithya Bhasker, et al. Active learning for extracting surgomic features in robot-assisted minimally invasive esophag...
2023
-
[9]
Surgical data science-from concepts to clinical translation
Lena Maier-Hein, Matthias Eisenmann, Duygu Sarikaya, Keno M¨ arz, Toby Collins, Anand Malpani, Jo- hannes Fallert, Hubertus Feussner, Stamatia Giannarou, Pietro Mascagni, et al. Surgical data science-from concepts to clinical translation. arXiv preprint arXiv:2011.02284, 2, 2020
2011 arXiv
-
[10]
Outcomes and complications after robot- assisted minimally invasive esophagectomy
Gijsbert van Boxel, Richard van Hillegersberg, and Jelle Ruurda. Outcomes and complications after robot- assisted minimally invasive esophagectomy. Journal of Visualized Surgery, 5, 2019
2019
-
[11]
A standardized approach for the thoracic dissection in robotic-assisted minimally invasive esophagectomy (ramie).Diseases of the Esophagus, 33(Supplement 2):doaa066, 2020
BF Kingma, M Read, R Van Hillegersberg, YK Chao, and JP Ruurda. A standardized approach for the thoracic dissection in robotic-assisted minimally invasive esophagectomy (ramie).Diseases of the Esophagus, 33(Supplement 2):doaa066, 2020
2020
-
[13]
Sv-rcnet: workflow recognition from surgical videos using recurrent convolutional network
Yueming Jin, Qi Dou, Hao Chen, Lequan Yu, Jing Qin, Chi-Wing Fu, and Pheng-Ann Heng. Sv-rcnet: workflow recognition from surgical videos using recurrent convolutional network. IEEE transactions on medical imaging, 37(5):1114–1126, 2017
2017
-
[14]
Temporal memory relation network for workflow recognition from surgical video
Yueming Jin, Yonghao Long, Cheng Chen, Zixu Zhao, Qi Dou, and Pheng-Ann Heng. Temporal memory relation network for workflow recognition from surgical video. IEEE Transactions on Medical Imaging, 40 (7):1911–1923, 2021
1911
-
[15]
Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer
Xiaojie Gao, Yueming Jin, Yonghao Long, Qi Dou, and Pheng-Ann Heng. Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer. InMedical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conferenc...
2021
-
[16]
Endonet: a deep architecture for recognition tasks on laparoscopic videos
Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging, 36(1):86–97, 2016
2016
-
[17]
Asformer: Transformer for action segmentation
Fangqiu Yi, Hongyu Wen, and Tingting Jiang. Asformer: Transformer for action segmentation. arXiv preprint arXiv:2110.08568, 2021
2021 arXiv
-
[18]
Kuehne, A
H. Kuehne, A. B. Arslan, and T. Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of Computer Vision and Pattern Recognition Conference (CVPR), 2014
2014
-
[19]
Combining embedded accelerometers with computer vision for recognizing food preparation activities
Sebastian Stein and Stephen J McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 729–738, 2013
2013
-
[20]
Ms-tcn: Multi-stage temporal convolutional network for action seg- mentation
Yazan Abu Farha and Jurgen Gall. Ms-tcn: Multi-stage temporal convolutional network for action seg- mentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3575–3584, 2019
2019
-
[21]
Metrics matter in surgical phase recognition
Isabel Funke, Dominik Rivoir, and Stefanie Speidel. Metrics matter in surgical phase recognition. arXiv preprint arXiv:2305.13961, 2023
2023 arXiv
-
[22]
Segmental spatiotemporal cnns for fine- grained action segmentation
Colin Lea, Austin Reiter, Ren´ e Vidal, and Gregory D Hager. Segmental spatiotemporal cnns for fine- grained action segmentation. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 36–52. Spr...
2016
-
[23]
Temporal convolutional networks for action segmentation and detection
Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 156–165, 2017
2017
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.