REVIEW 5 major objections 5 minor 33 references
StepAL: Step-aware Active Learning for Cataract Surgical Videos
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read StepAL, an active learning method for surgical step recognition, selects entire videos for annotation by clustering a step-aware representation built from pseudo-labels, and reports that it outperforms existing active learning methods on…
desk verdict StepAL is a reasonable new combination for video-level active learning in surgical step recognition, but the single-run evidence is too thin to back the 'consistently outperforms' claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the step-aware feature representation (SFR), a vector per video formed by concatenating $\ell_2$-normalized per-step average clip features, where the per-step grouping comes from pseudo-labels (Eq. 2-3). The selection mechanism is entropy-weighted clustering (EWC): weighted KMeans on these SFR vectors with each video's average clip-level entropy as its weight, then picking the videos closest to the cluster centers (Eq. 6). SFR preserves the step-composition structure that global averaging destroys; EWC biases selection toward uncertain videos while clustering keeps the chosen set diverse.
What would settle it
A direct test is to compare StepAL's round-1 selection against a variant with randomly shuffled pseudo-labels (preserving each video's marginal pseudo-label distribution) on Cataract-1k; if the shuffled variant matches StepAL's accuracy, the step-aware representation is not doing the claimed work. Conversely, an oracle variant that builds SFR from ground-truth step labels should not beat StepAL by much if pseudo-labels are already informative.
Extended reading notes
Core claim
The central claim is that pseudo-labels, even when imperfect, carry enough step-composition information that averaging clip features within each predicted step yields a representation that makes active learning select more useful videos. The paper argues this is why StepAL outperforms methods that average over all clips or only exploit uncertainty. On Cataract-1k, StepAL reaches 71.69% frame-wise accuracy after the first active learning round against 66.79% for the best baseline, CoreGCN; on Cataract-101 the corresponding numbers are 80.16% and 78.93%. The ablation shows the step-aware representation is the critical piece: replacing it with global clip averaging (the EWC variant) drops accuracy from 71.69% to 64.08%.
Load-bearing premise
The method's gains rest on the assumption that pseudo-labels from a model trained on only 10% of the data are accurate enough that averaging clip features per predicted step reveals the true step composition; the paper itself notes pseudo-labels may be imperfect early in the active learning cycle.
Editorial extensions
If this is right
- If StepAL's reported gains hold, full-video selection with step-aware pseudo-label representations becomes a stronger default than clip-level active learning for multi-step surgical videos.
- Annotation budgets for surgical step recognition could be cut substantially, since the method posts large gains with only 10% labeled data at the first active learning round.
- The method should transfer to other long, untrimmed procedural videos such as laparoscopy or endoscopy, where step dependencies are similar.
- The ablation suggests that the feature representation matters more than the clustering trick, pointing future active learning work toward step-aware embeddings.
Reading between the lines
- The step-aware representation could be reused beyond active learning, for example as a compact signature for video retrieval or dataset summarization in surgical video libraries.
- A natural extension is to refine pseudo-labels during early cycles, e.g., with consistency regularization, which could strengthen the representation and further reduce annotation cost.
- The entropy weight and the number of clusters are hyperparameters; tuning the balance between uncertainty and diversity may yield additional gains on datasets with more class imbalance.
- The method's benefit likely depends on how cleanly surgical steps are separated in the video; applying it to less structured long videos may require adapting the pseudo-label grouping.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes StepAL, an active learning framework for surgical step recognition that selects entire videos rather than frames or clips. StepAL combines a step-aware feature representation (SFR), built by averaging clip-level features according to pseudo-labels per predicted step (Eqs. 1-3), with an entropy-weighted KMeans clustering step (Eq. 6) that balances video-level uncertainty and diversity. Experiments on Cataract-1k and Cataract-101 using a VideoViT base model compare StepAL with Random, Margin, Entropy, Coreset, and CoreGCN, reporting accuracy gains at the first active learning round and across cycles, together with ablations of the proposed components.
Significance. If the reported results hold, StepAL addresses a real practical gap: annotators of long surgical videos need whole-video context, whereas most active learning methods select frames or clips. The method is clearly formulated, uses two public datasets, and includes ablations that separate the contributions of the step-aware representation and the entropy-weighted clustering. However, the central claim of consistent superiority rests on single-run numbers with an incompletely specified experimental protocol, so the current evidence is encouraging but not yet conclusive.
major comments (5)
- [Section 3, Implementation Details] The active learning budget is not well-defined for Cataract-1k. With 25 training videos, 10% of the training set is 2.5 videos, which is not an integer. The paper does not state whether the initial labeled set and each per-cycle addition contain 2 or 3 videos, and Algorithm 1 does not resolve this. This makes the exact protocol irreproducible and can change all downstream results, especially at the first cycle. Please specify the rounding rule, justify it, and state how it is applied consistently across all baselines.
- [Table 1 and Figure 2] All reported metrics come from a single run, with no error bars, no seed protocol, and no significance tests. The abstract's claim that StepAL 'consistently outperforms' existing methods is therefore not supported by the evidence. For example, the 4.66% accuracy advantage over CoreGCN at R=1 on Cataract-1k is measured on only 24 test videos, and could easily lie within run-to-run variation. I request multiple seeds with mean and standard deviation reported, and a paired statistical test (e.g., bootstrap or McNemar) at each active learning round.
- [Section 2.1, Eq. (2)] The step-aware representation is constructed entirely from pseudo-labels produced by a model trained on a very small labeled set in early cycles. The paper itself concedes in Section 1 that pseudo-labels may be imperfect in early AL cycles. Since Eqs. (2)-(3) are the core of the proposed representation, the method's behavior under noisy pseudo-labels needs direct investigation. I suggest an ablation that uses oracle labels during selection, or a sensitivity study in which pseudo-labels are artificially corrupted, to show that the mechanism, rather than an artifact of label noise, drives the gains.
- [Section 3, Results] The adaptation of the baseline methods to whole-video selection is not described. Coreset and CoreGCN are originally defined for sets of images or clips, and it is unclear how their features are aggregated to video level, how the graph for CoreGCN is constructed, or whether the same pseudo-label and entropy computations are used. Without an exact description of the video-level protocol for each baseline, the comparison may not be apples-to-apples. Please provide implementation details for all baselines, including how clip features are pooled and how any hyperparameters (e.g., number of clusters or graph edges) are set.
- [Algorithm 1 and Figure 2] There is an inconsistency in the number of active learning cycles. Algorithm 1 runs cycles r = 1 to R, while the text says 'Each cycle (R = 4 total)' and 'This evaluates model performance up to 50% labeled data utilization,' which matches an initial 10% plus four 10% additions. However, Figure 2's caption refers to '5 Active Learning Cycles (R = 0 to 4)'. Please clarify whether R=0 denotes the model trained on the initial labeled set and R=1 to R=4 are the active learning rounds, and make the numbering consistent between Algorithm 1, the text, and the figure.
minor comments (5)
- [Section 1] There are typographical issues, including 'pseudo-labels may be imperfect, particularly inearly AL cycles' and 'Itdirectlyaddresses', which should be corrected.
- [Eqs. (3)-(5)] The value of epsilon used in the ℓ2 normalization and in the entropy computation is never specified. Please state the value or explain how it is chosen.
- [Table 2] The ME-KMeans baseline is described only in the text, with no algorithmic description or equation. Since it is an important ablation point, please define it precisely in Section 2 or in the experiment section.
- [Section 1] The claim that StepAL is 'the first AL framework specifically designed for video selection' in surgical step recognition is a strong statement that is not supported by a thorough literature search. I suggest tempering the claim to 'to our knowledge' and citing any related surgical video active learning work.
- [Figure 2] The figure appears to report mean values, but there is no indication of variability. Please add error bars or shaded regions if the curves are averaged over runs; otherwise, state that each curve is a single run.
Circularity Check
No significant circularity: StepAL's reported gains are empirical comparisons on held-out data, not consequences of its own definitions.
full rationale
StepAL is an empirical active-learning selection framework, not a derivation from first principles, so none of the circularity patterns apply. The step-aware representation zV (Eq. 3) is built from pseudo-labels produced by the current model, and the video-level entropy E(V) (Eq. 5) is computed from the same model's softmax outputs; these are inputs used to rank videos, and the central claim (higher recognition accuracy with fewer labels) is then tested on held-out test videos from Cataract-1k and Cataract-101 against Random, Margin, Entropy, Coreset, and CoreGCN baselines. No parameter is fitted to the test set, and no reported quantity is defined as the thing it is supposed to predict. The article contains several self-citations by the authors (refs. 17-20, 23-26), but they are used to point to prior active-learning strategies and to motivate annotation cost; they do not supply the selection rule or the experimental numbers. The non-integer 10% budget and the absence of repeated-seed variance are statistical-robustness concerns, not circularity. The central comparison is therefore self-contained with respect to circularity.
Assumptions & free parameters
free parameters (3)
- Number of KMeans clusters =
Unstated; implied to equal budget b
- Initial labeled fraction and per-cycle budget =
10% of training videos per step; 4 cycles total
- Entropy epsilon =
Unstated small constant
assumptions (5)
- domain assumption Annotators must label complete videos rather than individual clips because surgical steps have sequential dependencies and clips lack context.
- domain assumption Pseudo-labels from the current model approximate the true step distribution well enough to guide selection.
- domain assumption Features extracted from a Kinetics-400-pretrained VideoViT provide a meaningful distance metric for surgical step composition.
- domain assumption Weighted KMeans with entropy weights selects videos whose annotation improves model accuracy more than other heuristic selections.
- standard math The standard weighted KMeans objective is appropriate for clustering concatenated step prototypes.
Cite this review
Pith. "Pith review of StepAL: Step-aware Active Learning for Cataract Surgical Videos." pith.science (2026). https://pith.science/paper/KOUVFNO5
@misc{pith2026250722059,
author = {Pith},
title = {Pith review of: StepAL: Step-aware Active Learning for Cataract Surgical Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/KOUVFNO5}},
note = {Machine review of arXiv:2507.22059}
}
read the original abstract
Active learning (AL) can reduce annotation costs in surgical video analysis while maintaining model performance. However, traditional AL methods, developed for images or short video clips, are suboptimal for surgical step recognition due to inter-step dependencies within long, untrimmed surgical videos. These methods typically select individual frames or clips for labeling, which is ineffective for surgical videos where annotators require the context of the entire video for annotation. To address this, we propose StepAL, an active learning framework designed for full video selection in surgical step recognition. StepAL integrates a step-aware feature representation, which leverages pseudo-labels to capture the distribution of predicted steps within each video, with an entropy-weighted clustering strategy. This combination prioritizes videos that are both uncertain and exhibit diverse step compositions for annotation. Experiments on two cataract surgery datasets (Cataract-1k and Cataract-101) demonstrate that StepAL consistently outperforms existing active learning approaches, achieving higher accuracy in step recognition with fewer labeled videos. StepAL offers an effective approach for efficient surgical video analysis, reducing the annotation burden in developing computer-assisted surgical systems.
Figures
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the IEEE/CVF international confer- ence on computer vision
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: Proceedings of the IEEE/CVF international confer- ence on computer vision. pp. 6836–6846 (2021)
2021
-
[2]
arXiv preprint arXiv:1906.03671 (2019)
Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, A.: Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671 (2019)
arXiv 2019
-
[3]
In: Interna- tional Conference on Computational Learning Theory
Balcan, M.F., Broder, A., Zhang, T.: Margin based active learning. In: Interna- tional Conference on Computational Learning Theory. pp. 35–50. Springer (2007)
work page 2007
-
[4]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Caramalau, R., Bhattarai, B., Kim, T.K.: Sequential graph convolutional network for active learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9583–9592 (2021)
work page 2021
-
[5]
Czempiel, T., Paschali, M., Keicher, M., Simson, W., Feussner, H., Kim, S.T., Navab, N.: Tecno: Surgical phase recognition with multi-stage temporal convolu- tional networks. In: MICCAI 2020. pp. 343–352. Springer (2020)
work page 2020
-
[6]
Machine learning28, 133–168 (1997)
Freund, Y., Seung, H.S., Shamir, E., Tishby, N.: Selective sampling using the query by committee algorithm. Machine learning28, 133–168 (1997)
work page 1997
-
[7]
Funke, I., Mees, S.T., Weitz, J., Speidel, S.: Video-based surgical skill assessment using 3d convolutional neural networks. IJCARS (2019)
work page 2019
-
[8]
Gao, X., Jin, Y., Long, Y., Dou, Q., Heng, P.A.: Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer. In: MICCAI 2021. pp. 593–603. Springer (2021)
work page 2021
Show all 33 references
-
[9]
Scientific data 11(1), 373 (2024)
Ghamsarian, N., El-Shabrawi, Y., Nasirihaghighi, S., Putzgruber-Adamitsch, D., Zinkernagel, M., Wolf, S., Schoeffmann, K., Sznitman, R.: Cataract-1k dataset for deep-learning-assisted analysis of cataract surgery videos. Scientific data 11(1), 373 (2024)
2024
-
[10]
Information Geometry6(1), 81–106 (2023)
Hino, H., Eguchi, S.: Active learning by query by committee with robust diver- gences. Information Geometry6(1), 81–106 (2023)
2023
-
[11]
arXiv preprint arXiv:1705.06950 (2017)
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al.: The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017)
2017 arXiv
-
[12]
arXiv preprint arXiv:1412.6980 (2014)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
2014 arXiv
-
[13]
In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention
Ma, S., Du, H., Curran, K.M., Lawlor, A., Dong, R.: Adaptive curriculum query strategy for active learning in medical image classification. In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention. pp. 48–57. Springer (2024) 10 N. Shah et al
2024
-
[14]
Nature Biomedical Engineering 1(9), 691–696 (2017)
Maier-Hein, L., Vedula, S.S., Speidel, S., Navab, N., Kikinis, R., Park, A., Eisen- mann, M., Feussner, H., Forestier, G., Giannarou, S., et al.: Surgical data science for next-generation interventions. Nature Biomedical Engineering 1(9), 691–696 (2017)
2017
-
[15]
Minimally Invasive Therapy & Allied Technologies28(2), 82–90 (2019)
Padoy, N.: Machine and deep learning for workflow recognition during surgery. Minimally Invasive Therapy & Allied Technologies28(2), 82–90 (2019)
2019
-
[16]
ACM computing surveys (CSUR)54(9), 1–40 (2021)
Ren, P., Xiao, Y., Chang, X., Huang, P.Y., Li, Z., Gupta, B.B., Chen, X., Wang, X.: A survey of deep active learning. ACM computing surveys (CSUR)54(9), 1–40 (2021)
2021
-
[17]
In: Proceedings of the Winter Conference on Applications of Computer Vision (WACV)
Safaei, B., Patel, V.M.: Active learning for vision language models. In: Proceedings of the Winter Conference on Applications of Computer Vision (WACV). pp. 4902– 4912 (February 2025)
2025
-
[18]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Safaei, B., Siddiqui, F., Xu, J., Patel, V.M., Lo, S.Y.: Filter images first, gener- ate instructions later: Pre-instruction data selection for visual instruction tuning. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 14247–14256 (2025)
2025
-
[19]
Proceedings of the AAAI Conference on Artificial Intelligence 38(5), 4686–4694 (2024)
Safaei, B., Vibashan, V., de Melo, C.M., Patel, V.M.: Entropic open-set active learning. Proceedings of the AAAI Conference on Artificial Intelligence 38(5), 4686–4694 (2024)
2024
-
[20]
arXiv preprint arXiv:2505.19421 (2025)
Safaei, B., VS, V., Patel, V.M.: Certainty and uncertainty guided active domain adaptation. arXiv preprint arXiv:2505.19421 (2025)
2025 arXiv
-
[21]
In: Proceedings of the 9th ACM Multimedia Systems Conference
Schoeffmann, K., Taschwer, M., Sarny, S., Münzer, B., Primus, M.J., Putzgruber, D.: Cataract-101: video dataset of 101 cataract surgeries. In: Proceedings of the 9th ACM Multimedia Systems Conference. p. 421–425. MMSys ’18, Association for Computing Machinery, New York, NY, US...
2018
-
[22]
arXiv preprint arXiv:1708.00489 (2017)
Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core- set approach. arXiv preprint arXiv:1708.00489 (2017)
2017 arXiv
-
[23]
In: Proceedings of the International Symposium on Biomedical Imaging (ISBI) (2025)
Shah, N.A., Bandara, C., Skider, S., Vedula, S.S., Patel, V.M.: CSMAE: Cataract surgical masked autoencoder (MAE) based pre-training. In: Proceedings of the International Symposium on Biomedical Imaging (ISBI) (2025)
2025
-
[24]
In: MICCAI (2023)
Shah, N.A., Sikder, S., Vedula, S.S., Patel, V.M.: Glsformer: Gated-long, short sequence transformer for step recognition in surgical videos. In: MICCAI (2023)
2023
-
[25]
In: 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI)
Shah, N.A., Sikder, S., Vedula, S.S., Patel, V.M.: Step detection in cataract surgery videos. In: 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2025)
2025
-
[26]
In: Medical Imaging with Deep Learning (2025)
Shah, N.A., Xia, M., Vijay, S., Sikder, S., Vedula, S.S., Patel, V.M.: A vision foun- dation model for cataract surgery using joint-embedding predictive architecture. In: Medical Imaging with Deep Learning (2025)
2025
-
[27]
In: Proceedings of the IEEE/CVF international conference on computer vision
Sinha, S., Ebrahimi, S., Darrell, T.: Variational adversarial active learning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5972–5981 (2019)
2019
-
[28]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
Taketsugu, H., Ukita, N.: Active transfer learning for efficient video-specific hu- man pose estimation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1880–1890 (2024)
2024
-
[29]
IEEE transactions on medical imaging36(1), 86–97 (2016)
Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging36(1), 86–97 (2016)
2016
-
[30]
In: 2014 International joint conference on neural networks (IJCNN)
Wang, D., Shang, Y.: A new active labeling method for deep learning. In: 2014 International joint conference on neural networks (IJCNN). pp. 112–119. IEEE (2014) StepAL: Step-aware Active Learning for Cataract Surgical Videos 11
2014
-
[31]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Wu, J., Chen, J., Huang, D.: Entropy-based active learning for object detection with progressive diversity constraint. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9397–9406 (2022)
2022
-
[32]
JAMA network open2(4), e191860–e191860 (2019)
Yu, F., Croso, G.S., Kim, T.S., Song, Z., Parker, F., Hager, G.D., Reiter, A., Vedula, S.S., Ali, H., Sikder, S.: Assessment of automated identification of phases in videos of cataract surgery using machine learning and deep learning techniques. JAMA network open2(4), e191860–...
2019
-
[33]
In: MICCAI 2018
Zisimopoulos, O., Flouty, E., Stoyanov, D., et al.: Deepphase: surgical phase recog- nition in cataracts videos. In: MICCAI 2018. pp. 265–272. Springer (2018)
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.