Pith. sign in

REVIEW 4 major objections 6 minor 30 references

For few-shot surgical skill assessment, domain-relevant pre-training data beats scale: small procedure-specific videos outperform large generic and even large surgical datasets on OSATS-based classification.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 19:15 UTC pith:FWINQSWQ

load-bearing objection First systematic pre-training study for few-shot SSA with a genuinely useful annotated subset, but the headline claim rests on small, untested differences. the 4 major comments →

arxiv 2509.09327 v1 pith:FWINQSWQ submitted 2025-09-11 cs.CV cs.LG

Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment

Dimitrios Anastasiou , Razvan Caramalau , Nazir Sirajudeen , Matthew Boal , Philip Edwards , Justin Collins , John Kelly , Ashwin Sridhar
show 6 more authors
Maxine Tran Faiz Mumtaz Nevil Pavithran Nader Francis Danail Stoyanov Evangelos B. Mazomenos
This is my paper
classification cs.CV cs.LG
keywords few-shot learningsurgical skill assessmentOSATSself-supervised pre-trainingdomain gaptransfer learningSAR-RARP50video masked autoencoder
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that when surgical skill must be assessed from only one, two, or five labeled videos, the choice of pre-training data matters more than its size. On a newly OSATS-annotated subset of robot-assisted prostatectomy videos, a small private dataset of nephrectomy suturing (RALPN) outperforms SurgToolLoc and Something-Something-v2 in 1- and 2-shot settings and stays competitive at 5-shot. Combining procedure-specific unlabeled prostatectomy videos with RALPN raises accuracy by 1.22% and F1 by 2.28% on average, while combining them with less similar large datasets degrades performance. If true, this means domain alignment, quantified by an Earth Mover's Distance on ImageNet features, is a first-order design factor for few-shot surgical video models.

Core claim

The central claim is that for few-shot surgical skill assessment, self-supervised pre-training on small, domain-aligned datasets yields better transfer than pre-training on much larger but less aligned datasets, and that mixing procedure-specific data with an external dataset only helps when the domain gap is small. The evidence is an accuracy of 60.16% (1-shot), 66.03% (2-shot), and 73.65% (5-shot) using the small RALPN dataset, compared with lower or comparable numbers from the 206-hour SurgToolLoc and 232-hour Something-Something-v2; and consistent average gains of +1.22% accuracy and +2.28% F1 when RALPN is combined with procedure-specific SAR-RARP50U, versus negative or mixed gains when

What carries the argument

The central mechanism is a domain-gap estimator: the Earth Mover's Distance between feature distributions of pre-training and downstream videos, computed from an ImageNet-pretrained Vision Transformer. This ranks datasets by alignment (SAR-RARP50U < RALPN < SurgToolLoc < Something-Something-v2) and is used to explain why small aligned data transfers better. The learning pipeline is VideoMAEv2 self-supervised pre-training, followed by frozen-encoder fine-tuning with a linear head or a temporal convolutional network, evaluated in episodic 1/2/5-shot classification.

Load-bearing premise

The load-bearing premise is that the binary split of OSATS scores (proficient 19–24 vs expert 25–30) on 33 annotated videos is a reliable, representative measure of surgical skill; the paper reports no inter-rater reliability for the two expert annotators and no clinical validation of the threshold.

What would settle it

Re-annotate the 33 SAR-RARP50 videos with a different expert panel and check whether class assignments change; or shift the GRS threshold by one point and re-run all pre-training comparisons. If the ranking of pre-training datasets (e.g., RALPN vs Something-Something-v2) flips or the reported accuracy gaps vanish, the central conclusion would be measuring label noise rather than transferability. Additionally, an unrelated dataset engineered to have a tiny EMD gap (e.g., synthetic suturing renderings) that fails to transfer would challenge the sufficiency of the domain-gap metric.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Dataset selection for few-shot SSA should prioritize domain alignment over raw hours of footage; a few hours of related surgical video can beat hundreds of hours of less relevant data.
  • Adding procedure-specific unlabeled data to pre-training is beneficial only when the external source is closely aligned; combining it with large but dissimilar sources can hurt downstream accuracy and F1.
  • The EMD-based gap computed from ImageNet features can act as a cheap pre-screening tool for pre-training data before expensive video SSL runs.
  • At higher shot counts (5-shot), large generic datasets close the gap in temporal evaluation, so the advantage of domain alignment is strongest in the extreme low-data regime.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to perturb the OSATS threshold (e.g., 19–23 vs 24–30) and check whether the ranking of pre-training datasets is stable; if not, the headline result may be an artifact of label construction.
  • The EMD-on-ImageNet-features ranking could be reused as a cheap prior to select pre-training data for other video-level surgical tasks that share the structure of whole-procedure reasoning, such as phase recognition or complication prediction.
  • The divergence between linear and temporal evaluation suggests that large-scale pre-training may contribute mostly motion/temporal dynamics, while domain-aligned pre-training contributes task-specific spatial features; a controlled ablation of reconstruction vs. temporal objectives could separate these contributions.
  • Combining several small aligned datasets may approximate the benefit of procedure-specific data without needing an exact match, since the paper's gain comes from shared domain characteristics rather than dataset identity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper studies how self-supervised pre-training data selection affects few-shot surgical skill assessment (SSA). The authors annotate 33 videos from SAR-RARP50 with binary OSATS labels (proficient vs expert), pre-train VideoMAEv2 on several sources (SAR-RARP50U, RALPN, SurgToolLoc, Something-Something-v2, plus combinations with SAR-RARP50U), and evaluate 1-, 2-, and 5-shot accuracy/F1 using linear and temporal (TCN) classifiers. They also propose an EMD-based domain gap between pre-training and downstream feature distributions. The central claims are that small domain-relevant datasets (RALPN) outperform larger, less aligned sources, and that combining RALPN with procedure-specific SAR-RARP50U improves performance by +1.22% accuracy and +2.28% F1 on average.

Significance. If the claims hold, the paper provides useful evidence for dataset selection in SSL pre-training for few-shot SSA, a task that is indeed underexplored. The strengths include the new OSATS annotation of SAR-RARP50 videos, the systematic comparison across pre-training sources, the public code and models, and the attempt to quantify domain gap. However, the empirical evidence is constrained by the small downstream set (33 videos), the absence of statistical significance testing, and heterogeneous results across settings. The contribution is currently a preliminary study rather than an established ranking of pre-training strategies.

major comments (4)
  1. [§3.1 and Table 1] The central quantitative claims rest on average gains of +1.22% accuracy and +2.28% F1 computed over 100 episodes that resample a fixed pool of 33 videos. These episodes are not independent, and no confidence intervals, significance tests, or effect-size intervals are provided; the abstract's 'significantly boosts' is therefore not supported by the reported statistics. Please report paired tests across episodes (e.g., Wilcoxon signed-rank or paired bootstrap) and confidence intervals for the differences, and report the per-episode variance explicitly.
  2. [§2.1] The binary GRS threshold (19–24 vs 25–30) is the target used in all evaluations, but the manuscript reports no inter-rater reliability for the two expert annotators and no analysis of threshold stability. If the labels are noisy or the cutoff is unstable, the comparisons in Fig. 3 measure the constructed variable, not surgical skill. Please report kappa/ICC, the label distribution, and sensitivity of the main ranking to moving the threshold by ±1 point, and reconcile the 'exploratory' caveat with the strong wording in the abstract.
  3. [§4, Fig. 3] The abstract's unconditional claim that small domain-relevant datasets outperform large-scale less aligned ones is contradicted by the 5-shot temporal setting, where Something-Something-v2 reaches 74.09% accuracy vs. RALPN's 73.65%; the text's claim that Something-Something-v2 achieves the 'highest accuracy and F1-score' is also inaccurate because RALPN's F1 is 71.2% vs. 70.9%. The support for the claim is confined mainly to linear evaluation and low-shot temporal evaluation. Please qualify the claim to the settings where it holds and test the pre-training-source × shot/evaluation interaction.
  4. [§2.3, Fig. 1] The proposed EMD domain gap ranks SurgToolLoc closer to SAR-RARP50L than Something-Something-v2, yet Something-Something-v2 outperforms SurgToolLoc in most settings (§4). This rank mismatch means the domain-gap metric does not predict the observed transfer ordering. To support the claim that domain gap 'influences' downstream performance, report quantitative correlations between EMD and accuracy/F1 per setting, and discuss why the metric fails for this pair.
minor comments (6)
  1. [§4] First paragraph contains a typo: 'the the pre-training configurations.'
  2. [Fig. 3] Caption and in-text references: clarify that 'temporal' and 'linear' rows correspond to TCN and linear classifier; currently 'temporal (top)' is ambiguous.
  3. [References] Several references have incomplete bibliographic entries (e.g., refs. 1, 24, 28 lack titles/venues), and the GitHub link appears as 'ssa-fsl' without a URL.
  4. [§2.3] Specify K (number of sampled snippets per video) and which ViT layer/feature is used for the EMD calculation, to allow reproduction.
  5. [§3.1] The fine-tuning batch size of 1 should be clarified: one clip or one video per step.
  6. [§4, EndoViT] Because EndoViT uses a different SSL objective (spatial MAE) and possibly backbone, its underperformance is not a clean dataset-comparison result; the paper acknowledges this, but the figure should mark it as a reference point rather than a comparable pre-training source.

Circularity Check

0 steps flagged

No significant circularity; the pre-training comparison is empirical and the domain-gap measure is independent of the downstream task.

full rationale

The paper's central claim—that small but domain-relevant pre-training sets can outperform larger, less aligned ones, and that adding procedure-specific data helps only when the domain gap is small—is an empirical result rather than a derivation from its own definitions. The pre-training datasets (SAR-RARP50U, RALPN, SurgToolLoc, Something-Something-v2) are external to the downstream OSATS labels; VideoMAEv2 pre-training is self-supervised and uses no skill annotations. The domain-gap metric (EMD) is computed using a separate ImageNet-pretrained ViT encoder, not the downstream labels or the VideoMAEv2 models being compared, so the dataset ranking is not fitted to the reported accuracies. The only notable self-citation is [26], prior work by overlapping authors, used in §2.1 to describe OSATS annotation practice; it is not load-bearing for the transferability conclusion. The paper explicitly acknowledges limitations—single dataset, 33 annotated videos, fewer than 32 samples per class, binary thresholding—in §2.1 and §5, which are statistical-power concerns rather than circular reasoning. No equation, fitted parameter, or self-citation chain makes any reported gain equal to an input by construction. The score of 1 reflects a minor self-citation that does not affect the independent empirical content of the study.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

Empirical transfer-learning study; no derivation. The main hand-chosen quantity is the GRS threshold. Axioms are domain assumptions about label validity and the domain-gap proxy.

free parameters (1)
  • GRS binary threshold = 19-24 vs 25-30
    Hand-chosen threshold to balance class distribution; defines the task and affects all reported accuracies.
axioms (3)
  • domain assumption OSATS scores provide a valid measure of surgical skill
    Underpins the binary label; no inter-rater reliability reported in Section 2.1.
  • domain assumption Unlabeled videos from the same dataset (SAR-RARP50U) can be used for pre-training without leaking labels into downstream evaluation
    Assumes the 21 unlabeled and 33 labeled videos are independent; video-level separation is claimed but not explicitly verified.
  • domain assumption Earth Mover's Distance on ImageNet-pretrained ViT features approximates task-relevant domain similarity
    Section 2.3 uses this to rank datasets; ranking only partially matches performance, so the measure's validity is uncertain.

pith-pipeline@v1.3.0-alltime-deepseek · 7742 in / 11768 out tokens · 114354 ms · 2026-08-04T19:15:56.267809+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment." pith.science (2026). https://pith.science/paper/FWINQSWQ

@misc{pith2026250909327,
  author       = {Pith},
  title        = {Pith review of: Exploring Pre-training Across Domains for Few-Shot Surgical Skill Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FWINQSWQ}},
  note         = {Machine review of arXiv:2509.09327}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Automated surgical skill assessment (SSA) is a central task in surgical computer vision. Developing robust SSA models is challenging due to the scarcity of skill annotations, which are time-consuming to produce and require expert consensus. Few-shot learning (FSL) offers a scalable alternative enabling model development with minimal supervision, though its success critically depends on effective pre-training. While widely studied for several surgical downstream tasks, pre-training has remained largely unexplored in SSA. In this work, we formulate SSA as a few-shot task and investigate how self-supervised pre-training strategies affect downstream few-shot SSA performance. We annotate a publicly available robotic surgery dataset with Objective Structured Assessment of Technical Skill (OSATS) scores, and evaluate various pre-training sources across three few-shot settings. We quantify domain similarity and analyze how domain gap and the inclusion of procedure-specific data into pre-training influence transferability. Our results show that small but domain-relevant datasets can outperform large scale, less aligned ones, achieving accuracies of 60.16%, 66.03%, and 73.65% in the 1-, 2-, and 5-shot settings, respectively. Moreover, incorporating procedure-specific data into pre-training with a domain-relevant external dataset significantly boosts downstream performance, with an average gain of +1.22% in accuracy and +2.28% in F1-score; however, applying the same strategy with less similar but large-scale sources can instead lead to performance degradation. Code and models are available at https://github.com/anastadimi/ssa-fsl.

Figures

Figures reproduced from arXiv: 2509.09327 by Ashwin Sridhar, Danail Stoyanov, Dimitrios Anastasiou, Evangelos B. Mazomenos, Faiz Mumtaz, John Kelly, Justin Collins, Matthew Boal, Maxine Tran, Nader Francis, Nazir Sirajudeen, Nevil Pavithran, Philip Edwards, Razvan Caramalau.

Figure 1
Figure 1. Figure 1: Left: EMD-based domain gap of each dataset with respect to SAR [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our investigation framework: Pre-training is performed using [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Classification Accuracy and F1-score for each pre-training configuration [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 6 linked inside Pith

  1. [1]

    Alapatt, D., Murali, A., Srivastav, V., Consortium, A., Mascagni, P., Padoy, N.: Jumpstarting surgical computer vision15006, 568–578 (2024) 10 Dimitrios Anastasiou et al

    2024

  2. [2]

    IEEE Robotics and Automation Letters8(3), 1755–1762 (2023)

    Anastasiou, D., Jin, Y., Stoyanov, D., Mazomenos, E.: Keep your eye on the best: Contrastive regression transformer for skill assessment in robotic surgery. IEEE Robotics and Automation Letters8(3), 1755–1762 (2023)

    2023

  3. [3]

    Batić, D., Holm, F., Özsoy, E., Czempiel, T., Navab, N.: Whether and when does endoscopy domain pretraining make sense? (2023), arXiv:2303.17636

    Pith/arXiv arXiv 2023

  4. [4]

    International Journal of Computer Assisted Radiology and Surgery19, 1085–1091 (2024)

    Batić,D.,Holm,F.,Özsoy,E.,Czempiel,T.,Navab,N.:Endovit:Pretrainingvision transformers on a large collection of endoscopic images. International Journal of Computer Assisted Radiology and Surgery19, 1085–1091 (2024)

    2024

  5. [5]

    Wiley Series in Probability and Statis- tics, John Wiley & Sons, Inc

    van Belle, G.: Statistical Rules of Thumb. Wiley Series in Probability and Statis- tics, John Wiley & Sons, Inc. (2008)

    2008

  6. [6]

    arXiv preprint arXiv:2104.14294 (2021)

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.14294 (2021)

    Pith/arXiv arXiv 2021

  7. [7]

    CVPR (2009)

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. CVPR (2009)

    2009

  8. [8]

    ICLR (2020)

    Dhillon, G.S., Chaudhari, P., Ravichandran, A., Soatto, S.: A baseline for few-shot image classification. ICLR (2020)

    2020

  9. [9]

    ICLR 2021 - 9th International Conference on Learning Representations (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021 - 9th International Conference on Learning Representations (2021)

    2021

  10. [10]

    Funke, I., Mees, S.T., Weitz, J., Speidel, S.: Video-based surgical skill assessment using 3dconvolutional neuralnetworks. Int. J.Comput. Assist.Radiol. Surg.7(14), 1217–1225 (2019)

    2019

  11. [11]

    ICCV (2017)

    Goyal, R., Kahou, S.E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I., Memisevic, R.: The ’something something’ video database for learning and evaluating visual common sense. ICCV (2017)

    2017

  12. [12]

    CVPR (2022)

    He, K., Chen, X., Xie, S., Li, Y., Dollar, P., Girshick, R.: Masked autoencoders are scalable vision learners. CVPR (2022)

    2022

  13. [13]

    Hirsch, R., et al.: Self-supervised learning for endoscopic video analysis14224, 571–581 (2023)

    2023

  14. [14]

    In: Bhattarai, B., Ali, S., Rau, A., Caramalau, R., Nguyen, A., Gyawali, P., Nam- burete, A., Stoyanov, D

    Jaspers, T.J.M., de Jong, R.L.P.D., Al Khalil, Y., et al.: Exploring the effect of dataset diversity in self-supervised learning for surgical computer vision. In: Bhattarai, B., Ali, S., Rau, A., Caramalau, R., Nguyen, A., Gyawali, P., Nam- burete, A., Stoyanov, D. (eds.) Data Engineering in Medical Imaging. pp. 43–53 (2025)

    2025

  15. [15]

    Li, Z., Gu, L., Wang, W., Nakamura, R., Sato, Y.: Surgical skill assessment via video semantic aggregation. Med. Image Comput. Comput. Assist. Interv.13437, 410–420 (2022)

    2022

  16. [16]

    Liu, D., Li, Q., Jiang, T., Wang, Y., Miao, R., Shan, F., Li, Z.: Towards unified surgical skill assessment. CVPR pp. 9517–9526 (2021)

    2021

  17. [17]

    Medical Image Analysis76, 102306 (2022)

    Maier-Hein, L., Eisenmann, M., Sarikaya, D., et al.: Surgical data science – from concepts toward clinical translation. Medical Image Analysis76, 102306 (2022)

    2022

  18. [18]

    Nature Biomedical Engineering1(9), 691–696 (2017)

    Maier-Hein, L., Vedula, S.S., Speidel, S., Navab, N., et al.: Surgical data science for next-generation interventions. Nature Biomedical Engineering1(9), 691–696 (2017)

    2017

  19. [19]

    British Journal of Surgery84(1997) Title Suppressed Due to Excessive Length 11

    Martin, J.A., Regehr, G., Reznick, R., Macrae, H., Murnaghan, J., Hutchison, C., Brown, M.: Objective structured assessment of technical skill (osats) for surgical residents. British Journal of Surgery84(1997) Title Suppressed Due to Excessive Length 11

    1997

  20. [20]

    NeurIPS (2022)

    Oh, J., Kim, S., Ho, N., Kim, J.H., Song, H., Yun, S.Y.: Understanding cross- domain few-shot learning based on domain similarity and few-shot difficulty. NeurIPS (2022)

    2022

  21. [21]

    arXiv:2203.04291 (2022)

    Parnami, A., Lee, M.: Learning from few examples: A summary of approaches to few-shot learning. arXiv:2203.04291 (2022)

    Pith/arXiv arXiv 2022

  22. [22]

    arXiv:2401.00496 (2023)

    Psychogyios, D., Colleoni, E., Amsterdam, B.V., Li, C.Y., Huang, S.Y., Li, Y., Jia, F.: Sar-rarp50: Segmentation of surgical instrumentation and action recognition on robot-assisted radical prostatectomy challenge. arXiv:2401.00496 (2023)

    Pith/arXiv arXiv 2023

  23. [23]

    arXiv preprint arXiv:2103.00020 (2021)

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021)

    Pith/arXiv arXiv 2021

  24. [24]

    Medical Image Analysis88, 102844 (2023)

    Ramesh, S., Srivastav, V., Alapatt, D., et al.: Dissecting self-supervised learning methods for surgical computer vision. Medical Image Analysis88, 102844 (2023)

    2023

  25. [25]

    ICCV (1998)

    Rubner, Y., Tomasi, C., Guibas, L.J.: A metric for distributions with applications to image databases. ICCV (1998)

    1998

  26. [26]

    Surgical Endoscopy38(12), 7663–7671 (Dec 2024)

    Sirajudeen, N., Boal, M., Anastasiou, D., Xu, J., Stoyanov, D., Kelly, J., Collins, J.W., Sridhar, A., Mazomenos, E., Francis, N.K.: Deep learning prediction of error and skill in robotic prostatectomy suturing. Surgical Endoscopy38(12), 7663–7671 (Dec 2024)

    2024

  27. [27]

    Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., Qiao, Y.: Videomaev2:Scalingvideomaskedautoencoderswithdualmasking.CVPR(2023)

    2023

  28. [28]

    Wang, Z., Liu, C., Zhang, S., Dou, Q.: Foundation model for endoscopy video analysis via large-scale self-supervised pre-train14228, 94–104 (2023)

    2023

  29. [29]

    Computers in Biology and Medicine174, 108470 (2024)

    Yanik, E., Schwaitzberg, S., Yang, G., Intes, X., Norfleet, J., Hackett, M., De, S.: One-shot skill assessment in high-stakes domains with limited data via meta learning. Computers in Biology and Medicine174, 108470 (2024)

    2024

  30. [30]

    arXiv:2305.07152 (2023)

    Zia, A., Bhattacharyya, K., Liu, X., Berniker, M., Wang, Z., Nespolo, R.: Surgi- cal tool classification and localization: results and methods from the miccai 2022 surgtoolloc challenge. arXiv:2305.07152 (2023)

    Pith/arXiv arXiv 2022