REVIEW 4 major objections 5 minor 1 cited by
SemiVT-Surge: Semi-Supervised Video Transformer for Surgical Phase Recognition
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Semi-supervised surgical phase recognition can match fully supervised accuracy with a quarter of the annotations by using temporal consistency and class prototypes to leverage unlabeled video.
desk verdict A sensible FixMatch-style extension to surgical video that likely works, but the headline Cholec80 SOTA comparison rests on uncontrolled random splits and a private dataset, so I'd want a revision before trusting the numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a teacher-student video transformer pair with temporal sampling as the augmentation. A weak view (current frame plus the preceding 15 consecutive frames) is passed to an EMA-updated teacher, which produces pseudo-labels; a strong view (current frame plus 15 randomly chosen past frames) is passed to the student, and cross-entropy between the two predictions is counted only when the teacher's maximum probability exceeds a confidence threshold $\delta$. In parallel, a triplet margin loss pulls normalized features toward their class prototype and pushes them away from the three nearest negative prototypes, with prototypes updated by exponential moving average. The temporal sampling is what makes the regularization surgical-domain specific: consistency across short- and long-term context encodes the fact that the same phase can recur and that phase transitions are anchored in the operative workflow.
What would settle it
Re-run the Cholec80 comparison at 20, 10, and 5 labeled videos over at least ten random subset draws with fixed seeds and report mean and standard deviation; if the reported gaps over SurgSSL (90.4 vs 87.0, 88.8 vs 85.0, 85.3 vs 83.2) do not persist across most draws, the label-efficiency claim collapses to a dataset-luck artifact.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that self-training a video transformer on unlabeled surgical footage need not be limited to pretraining: a pseudo-labeling loop with temporal consistency and prototype-based contrastive learning can lift phase-recognition accuracy directly. With labels only, the model reaches 78.7% accuracy on RAMIE; adding 20 unlabeled videos with the full objective raises it to 83.4%, above the fully supervised TeCNO baseline (78.5%) and above two large self-supervised pretraining baselines. On Cholec80, 20 labeled videos yield 90.4% accuracy versus 87.0% for SurgSSL and 88.6% for fully supervised TeCNO, and the same pattern holds at 10 and 5 labeled videos. The paper therefore claims that its objective is label-efficient and that it sets a benchmark for semi-supervised surgical phase recognition.
Load-bearing premise
The load-bearing assumption is that the randomly sampled labeled subsets used in the Cholec80 experiments are representative of the subsets used for the comparison baselines; because no seed or repeated sampling is reported, a single lucky draw of 5 to 20 videos could explain the claimed gains.
Editorial extensions
If this is right
- Unlabeled surgical videos from the same procedure type can be used as a training signal during fine-tuning, not only for pretraining, which lowers the annotation cost of phase recognition systems.
- With only a quarter of the labeled Cholec80 videos, the method reports higher accuracy than the fully supervised baseline, so fully manual annotation of every training video may not be necessary for this task.
- Combining temporal consistency regularization with prototype-based contrastive learning gives larger gains than either component alone, according to the paper's ablation results.
- Adding a causal temporal convolutional head on top of the transformer further improves long-term phase modeling, showing that the two forms of temporal modeling are complementary.
Reading between the lines
- An implication the paper leaves implicit is a deployment path: hospitals with large archives of unlabeled surgical video could bootstrap phase-recognition models from just a few annotated cases of the same procedure.
- Because the confidence threshold is set separately per dataset (0.6 for RAMIE, 0.8 for Cholec80), a natural extension is to make the threshold per class or per video, which could improve recall on underrepresented phases without changing the objective.
- The same long-short temporal sampling scheme could transfer to other long-video workflow recognition tasks where phases recur and are defined by operative progress rather than by scene content.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SemiVT-Surge, a semi-supervised video transformer for surgical phase recognition. The method uses a TimeSformer encoder in a student-teacher EMA framework; unlabeled clips are weakly and strongly augmented through short-term consecutive versus long-term random frame sampling, and a confidence-gated temporal consistency loss is applied. A prototype-based triplet loss, using EMA-updated class prototypes and teacher pseudo-labels, is added to structure the embedding space. Experiments on the private RAMIE dataset and the public Cholec80 dataset report accuracy gains over TeCNO, FixMatch, SurgSSL, and self-supervised pretraining baselines. The headline claims are state-of-the-art accuracy on RAMIE (83.4%) and accuracy exceeding the fully supervised TeCNO baseline with only 20 of 40 Cholec80 labels (90.4% versus 88.6%).
Significance. If the results are reproducible and the comparisons are well controlled, the paper would be a useful contribution to the under-explored area of semi-supervised surgical phase recognition. It combines several known semi-supervised ideas, including temporal sampling as augmentation, EMA teacher-student consistency, and prototype-based contrastive learning, in a way that is tailored to long surgical videos. The ablations in Table 1 are directionally consistent on both datasets, and the authors release code, which supports reproducibility. However, the central claims currently rest on comparisons whose labeled splits are not matched, no significance testing or repeated-split variance is reported, and one of the two evaluation datasets is private with a placeholder citation. These issues are addressable but must be fixed before the stated conclusions can be considered supported.
major comments (4)
- [§3, Table 3] The comparison against SurgSSL on Cholec80 is not controlled. The paper states that for reduced-annotation experiments 'we randomly sample a subset of the training set as labeled data' but reports no random seed, no repeated sampling, and no split-level variance. With only 5, 10, or 20 labeled videos, the identity of the chosen videos can dominate performance, and the reported gaps over SurgSSL (2.1–3.4 accuracy points) may be within subset-selection noise. The standard deviations in Table 3 are computed across test videos, not across repeated labeled-subset draws, so they do not address this concern. In addition, the SurgSSL numbers appear to be quoted from the original paper, which likely uses a different split protocol. To support the state-of-the-art claim, the authors should either adopt the exact labeled splits used by SurgSSL or report mean and standard deviation over several random splits with fixed seeds, and should state whether the SurgSSL baselines were re-run under the same protocol.
- [§3, Table 2] The headline 4.9% accuracy improvement on RAMIE is not statistically supported. The test set contains only 9 videos, and no significance test or confidence interval is reported for the difference between the proposed method (83.4%) and the fully supervised TeCNO baseline (78.5%). The RAMIE dataset is also private, and reference [2] is a placeholder ('This paper is accepted by XXX but not yet available to the public'), so the result cannot be checked by reviewers or readers. Please provide a significance test (or a bootstrap confidence interval for the difference), and either make the dataset available or supply a detailed, de-identified dataset description and a complete citable reference.
- [Abstract and §3, Table 3] The 'comparable to full supervision' claim is supported only against TeCNO, which is a different architecture. Table 3 compares Ours with 20 labeled videos (90.4%) to fully supervised TeCNO (88.6%), but the authors do not report the accuracy of their own TimeSformer-based model trained with all 40 labels. The apparent advantage may come from the video transformer backbone rather than from the semi-supervised framework. Please add a fully supervised upper bound of the same model trained with all labeled data, and phrase the abstract's claim in terms of that comparison.
- [§3, Table 1] The ablations on Cholec80 are reported from a single run with a single randomly drawn labeled subset. The incremental gains between adjacent rows (e.g., 84.3±10.7 to 87.0±6.6 to 89.6±7.0 to 90.4±7.0) are small relative to the across-video standard deviations, and the text describes each addition as an improvement without repeated-trial or statistical support. At minimum, report the mean and standard deviation over multiple labeled-subset draws for the Cholec80 ablations.
minor comments (5)
- [§2.2 and Algorithm 1] The unsupervised consistency loss is called L_U in Equation (1) but LReg in Algorithm 1 and in the text of Section 2.2; please use a single symbol throughout.
- [§2.4] The word 'emperically' should be 'empirically'.
- [§2.4] Weak augmentation is described as 'random cropping, normalization, random rotation, and random crop', with 'random crop' appearing twice; also, the strong augmentation configuration 'rand-m9-n5-mstd0.8-inc1' is a RandAugment policy, not an AutoAugment policy. Please clarify the exact augmentation library and policy.
- [§2.4 and Algorithm 1] The strong augmentation randomly selects T-1 frames from the full video history; since frames are extracted at 1 fps, this can span the entire surgery. Please state whether the selection is uniform over the whole history or constrained to a local temporal window, as this directly affects the interpretation of 'temporal consistency'.
- [Figure 1] The symbols L_Tri-L and L_Tri-U are used in the figure without being defined there; please refer explicitly to Equation (2) and the corresponding text in Section 2.3.
Circularity Check
No circularity: the paper's claims are empirical comparisons against external baselines; the noted split-matching and placeholder-citation issues are correctness risks, not circular reasoning.
full rationale
This is an empirical systems paper, not a derivation. The central claims (SOTA accuracy on RAMIE and comparable-to-full-supervision accuracy on Cholec80 with 1/4 labeled data) are supported by ablations in Table 1 and comparisons to external baselines (TeCNO, FixMatch, SurgSSL, EndoFM, SurgeNetXL) in Tables 2 and 3. None of the loss terms (L_Sup, L_U, L_tri) is defined in terms of the reported test accuracy, and no fitted parameter is renamed as a prediction: the confidence threshold and prototype margin are fixed hyperparameters, not calibrated on the test metric. The self-citations [12,13] appear only as comparison baselines or contextual prior work, not as load-bearing premises that force the conclusion. The RAMIE dataset reference [2] is a placeholder citation, and the Cholec80 labeled subsets are 'randomly sample[d]' without seeds or repeated sampling, so the SOTA comparison carries statistical and comparability risk; these are correctness/verifiability concerns, not circularity. No equation in the paper reduces to a fitted value or to a self-citation chain, so the derivation chain is self-contained with respect to circularity.
Assumptions & free parameters
free parameters (5)
- Pseudo-label confidence threshold delta =
0.6 on RAMIE, 0.8 on Cholec80
- Triplet margin m =
0.3
- Number of nearest negative prototypes k =
3
- EMA decay alpha and prototype momentum eta =
0.9
- Number of unlabeled RAMIE videos =
20
assumptions (3)
- domain assumption Teacher pseudo-labels above the confidence threshold delta are accurate enough to train the student on unlabeled data.
- domain assumption A surgical phase is determined by the current frame and is robust to which past frames are sampled, so random long-range sampling is a valid strong augmentation.
- domain assumption Unlabeled RAMIE videos come from the same phase distribution and are compatible with the labeled videos.
Cite this review
Pith. "Pith review of SemiVT-Surge: Semi-Supervised Video Transformer for Surgical Phase Recognition." pith.science (2026). https://pith.science/paper/OZQFRYU5
@misc{pith2026250601471,
author = {Pith},
title = {Pith review of: SemiVT-Surge: Semi-Supervised Video Transformer for Surgical Phase Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/OZQFRYU5}},
note = {Machine review of arXiv:2506.01471}
}
read the original abstract
Accurate surgical phase recognition is crucial for computer-assisted interventions and surgical video analysis. Annotating long surgical videos is labor-intensive, driving research toward leveraging unlabeled data for strong performance with minimal annotations. Although self-supervised learning has gained popularity by enabling large-scale pretraining followed by fine-tuning on small labeled subsets, semi-supervised approaches remain largely underexplored in the surgical domain. In this work, we propose a video transformer-based model with a robust pseudo-labeling framework. Our method incorporates temporal consistency regularization for unlabeled data and contrastive learning with class prototypes, which leverages both labeled data and pseudo-labels to refine the feature space. Through extensive experiments on the private RAMIE (Robot-Assisted Minimally Invasive Esophagectomy) dataset and the public Cholec80 dataset, we demonstrate the effectiveness of our approach. By incorporating unlabeled data, we achieve state-of-the-art performance on RAMIE with a 4.9% accuracy increase and obtain comparable results to full supervision while using only 1/4 of the labeled data on Cholec80. Our findings establish a strong benchmark for semi-supervised surgical phase recognition, paving the way for future research in this domain.
Figures
Forward citations
Cited by 1 Pith paper
-
BleedOrigin: Dynamic Bleeding Source Localization in Endoscopic Submucosal Dissection via Dual-Stage Detection and Tracking
A new ESD bleeding-source dataset and a dual-stage detection-tracking framework report 96.85% onset, 70.24% source, and 96.11% tracking accuracy within defined tolerances.
Reference graph
Works this paper leans on
-
[2]
In: This paper is accepted by XXX but not yet available to the public
Author, A.: Paper title. In: This paper is accepted by XXX but not yet available to the public. More details on the dataset are provided in this work. (2025)
work page 2025
-
[1]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Alapatt, D., Murali, A., Srivastav, V., Consortium, A., Mascagni, P., Padoy, N.: Jumpstarting surgical computer vision. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 328–338. Springer (2024)
2024
-
[3]
In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition
Basak, H., Yin, Z.: Pseudo-label guided contrastive learning for semi-supervised medical image segmentation. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. pp. 19786–19797 (2023)
work page 2023
-
[4]
International Journal of Computer Assisted Radiology and Surgery19(6), 1085–1091 (2024)
Batić, D., Holm, F., Özsoy, E., Czempiel, T., Navab, N.: Endovit: pretraining vision transformers on a large collection of endoscopic images. International Journal of Computer Assisted Radiology and Surgery19(6), 1085–1091 (2024)
work page 2024
-
[5]
Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML. vol. 2, p. 4 (2021)
2021
-
[6]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Cubuk,E.D.,Zoph,B.,Mane,D.,Vasudevan,V.,Le,Q.V.:Autoaugment:Learning augmentation strategies from data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 113–123 (2019)
2019
-
[7]
Czempiel, T., Paschali, M., Keicher, M., Simson, W., Feussner, H., Kim, S.T., Navab, N.: Tecno: Surgical phase recognition with multi-stage temporal con- volutional networks. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23. pp. 343–352. Sprin...
work page 2020
-
[8]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Dave, I.R., Rizve, M.N., Chen, C., Shah, M.: Timebalance: Temporally-invariant and temporally-distinctive video representations for semi-supervised action recog- nition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2341–2352 (2023)
work page 2023
Show all 26 references
-
[9]
arXiv preprint arXiv:2305.13961 (2023)
Funke, I., Rivoir, D., Speidel, S.: Metrics matter in surgical phase recognition. arXiv preprint arXiv:2305.13961 (2023)
2023 arXiv
-
[10]
In: International Conference on Medical Im- age Computing and Computer-Assisted Intervention
Guan, J., Zou, X., Tao, R., Zheng, G.: Label-guided teacher for surgical phase recognition via knowledge distillation. In: International Conference on Medical Im- age Computing and Computer-Assisted Intervention. pp. 349–358. Springer (2024)
2024
-
[11]
In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention
He, A., Li, T., Zhao, Y., Zhao, J., Fu, H.: Open-set semi-supervised medical image classification with learnable prototypes and outlier filter. In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention. pp. 492–501. Springer (2024)
2024
-
[12]
In: MICCAI Workshop on Data Engineering in Medical Imaging
Jaspers, T.J., de Jong, R.L., Al Khalil, Y., Zeelenberg, T., Kusters, C.H., Li, Y., vanJaarsveld,R.C.,Bakker, F.H.,Ruurda,J.P.,Brinkman,W.M.,etal.: Exploring the effect of dataset diversity in self-supervised learning for surgical computer vision. In: MICCAI Workshop on Data E...
2024
-
[13]
arXiv preprint arXiv:2501.09436 (2025)
Jaspers, T.J., de Jong, R.L., Li, Y., Kusters, C.H., Bakker, F.H., van Jaarsveld, R.C., Kuiper, G.M., van Hillegersberg, R., Ruurda, J.P., Brinkman, W.M., et al.: Scaling up self-supervised learning for improved surgical foundation models. arXiv preprint arXiv:2501.09436 (2025)
2025
-
[14]
IEEE transactions on medical imaging42(7), 1920–1931 (2022)
Kassem,H.,Alapatt,D.,Mascagni,P.,Karargyris,A.,Padoy,N.:Federatedcycling (fedcy): Semi-supervised federated learning of surgical phases. IEEE transactions on medical imaging42(7), 1920–1931 (2022)
2022
-
[15]
arXiv preprint arXiv:1705.06950 (2017)
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al.: The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017)
2017 arXiv
-
[16]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Liu, Y., Huo, J., Peng, J., Sparks, R., Dasgupta, P., Granados, A., Ourselin, S.: Skit: a fast key information video transformer for online surgical phase recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21074–21084 (2023)
2023
-
[17]
Maier-Hein, L., Eisenmann, M., Sarikaya, D., März, K., Collins, T., Malpani, A., Fallert, J., Feussner, H., Giannarou, S., Mascagni, P., et al.: Surgical data science– fromconceptstowardclinicaltranslation.Medicalimageanalysis 76,102306(2022)
2022
-
[18]
In: International Confer- ence on Medical Image Computing and Computer-Assisted Intervention
Pérez, A., Rodríguez, S., Ayobi, N., Aparicio, N., Dessevres, E., Arbeláez, P.: Must: Multi-scale t ransformers for surgical phase recognition. In: International Confer- ence on Medical Image Computing and Computer-Assisted Intervention. pp. 422–
-
[19]
Medical Image Analysis 73, 102158 (2021)
Shi, X., Jin, Y., Dou, Q., Heng, P.A.: Semi-supervised learning with progressive unlabeled data excavation for label-efficient surgical workflow recognition. Medical Image Analysis 73, 102158 (2021)
2021
-
[20]
Advances in neural information processing systems33, 596–608 (2020)
Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C.A., Cubuk, E.D., Kurakin, A., Li, C.L.: Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems33, 596–608 (2020)
2020
-
[21]
Advances in neural information processing systems30 (2017) Semi-Supervised Video Transformer for Surgical Phase Recognition 11
Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems30 (2017) Semi-Supervised Video Transformer for Surgical Phase Recognition 11
2017
-
[22]
IEEE transactions on medical imaging36(1), 86–97 (2016)
Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging36(1), 86–97 (2016)
2016
-
[23]
In: International Conference on Med- ical Image Computing and Computer-Assisted Intervention
Wang, Z., Liu, C., Zhang, S., Dou, Q.: Foundation model for endoscopy video anal- ysis via large-scale self-supervised pre-train. In: International Conference on Med- ical Image Computing and Computer-Assisted Intervention. pp. 101–111. Springer (2023)
2023
-
[24]
In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition
Xing, Z., Dai, Q., Hu, H., Chen, J., Wu, Z., Jiang, Y.G.: Svformer: Semi-supervised video transformer for action recognition. In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 18816–18826 (2023)
2023
-
[25]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Xu, Y., Wei, F., Sun, X., Yang, C., Shen, Y., Dai, B., Zhou, B., Lin, S.: Cross- model pseudo-labeling for semi-supervised action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2959–2968 (2022)
2022
-
[26]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Yang, S., Luo, L., Wang, Q., Chen, H.: Surgformer: Surgical transformer with hierarchical temporal attention for surgical phase recognition. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 606–616. Springer (2024)
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.