Pith. sign in

REVIEW 3 major objections 5 minor 17 references

A generic, publicly available video foundation model—trained only on everyday videos—matches the top phase-recognition entries of the EndoVis2017 surgical challenge, and self-supervised finetuning on unlabeled operating-room footage pushes

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A generic video foundation model, finetuned on unlabeled surgical footage and optionally fused with OR sensor streams, reaches competitive phase recognition on HeiCo and improves in-house outcome prediction.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Solid, honest empirical recipe for adapting V-JEPA to surgical video, but the 'on par with EndoVis2017' claim leans on a favorable metric and an approximated protocol. the 3 major comments →

arxiv 2509.06831 v1 pith:G5OR3IEW submitted 2025-09-08 cs.CV

Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis

Simon Pezold , J\'er\^ome A. Kurylec , Jan S. Liechti , Beat P. M\"uller , Jo\"el L. Lavanchy This is my paper
classification cs.CV
keywords foundation modelssurgical phase recognitionV-JEPAdomain adaptationmultimodal fusionoperating room data streamsself-supervised learningsurgical data science
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a generic video foundation model—pretrained on ordinary real-world videos, not on medical footage—can form the backbone of surgical video analysis, and whether additional non-video streams from the operating room improve it. Using V-JEPA as a frozen encoder, the authors reach surgical phase recognition accuracy on the public HeiCo dataset that is on par with the best-performing EndoVis2017 challenge submissions. Continued self-supervised training on unlabeled laparoscopic videos improves results further on most tasks, including length-of-stay prediction, and integrating vital-sign streams as a second modality improves in-house outcome predictions. The paper concludes that public generic foundation models are a viable, low-cost starting point for surgical data science, with domain adaptation and informative complementary data streams as further levers.

Core claim

On its own terms, the paper claims to demonstrate three things. First, that V-JEPA—a video transformer pretrained self-supervised on two million generic videos—can be used out of the box as a frozen embedding source for surgical downstream tasks: an attentive classifier on its embeddings reaches 62% accuracy on HeiCo phase recognition, comparable to the top three EndoVis2017 challenge entries. Second, that continued JEPA training on unlabeled laparoscopic video raises performance, improving HeiCo accuracy to 64% and strengthening in-house outcome predictions. Third, that a separate encoder trained to align OR data streams with V-JEPA embeddings adds value only when the stream is task-informa

What carries the argument

V-JEPA: a self-supervised video transformer that learns by predicting the teacher's embeddings of masked video tokens from a student's embeddings of unmasked tokens, with the teacher an exponential moving average of the student. The paper freezes this encoder and builds multimodal fusion on top: a HEALNet-inspired cross-attention encoder updates a state vector (initialized as the V-JEPA embedding) with each additional OR data stream, using layer-shared key/query/value projections and a state-change penalty that keeps the new embeddings aligned with the video embedding. A four-step training recipe—V-JEPA pretraining/finetuning, decoder training on the downstream task, stream-encoder training

Load-bearing premise

The parity claim against EndoVis2017 relies on an approximation of the original challenge's evaluation protocol—the paper fills gaps with reasonable assumptions, skips out-of-body segments, and samples clips at one-second intervals—so if the official protocol scored frames or handled the exception class differently, the headline comparison could shift.

What would settle it

Run the same pretrained and finetuned V-JEPA under the original EndoVis2017 scoring procedure (all frames, including out-of-body segments and the exception class, using the challenge's own sampling) and check whether the accuracy still matches the top-three leaderboard entries; also, repeat HeiCo phase recognition without finetuning on the challenge's validation split to isolate the finetuning effect.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A public, generic video encoder can be used directly for surgical phase recognition, reaching challenge-winner-level accuracy without any surgery-specific pretraining.
  • Self-supervised finetuning on unlabeled surgical video is a cheap performance lever: it raised HeiCo accuracy from 62% to 64% and improved length-of-stay prediction.
  • The value of adding non-visual OR data is conditional on the stream's information content; the paper finds vital signs helpful and device status not.
  • The released code and finetuned weights let other groups reproduce or extend the recipe on other surgical video tasks.
  • The modular design lets the model run on video alone or with any subset of the extra streams, since the multimodal encoder can be skipped.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same four-step recipe should transfer to other procedural videos and other surgical questions (tool state, skill assessment), since nothing in the method is specific to liver or colorectal surgery.
  • The modest compute budget for finetuning—seven JEPA epochs in twelve hours on four accelerators—suggests domain adaptation of large video encoders is within reach of clinical research groups.
  • The modality-dominance result hints that before collecting extra OR streams, one should test whether the stream is causally related to the outcome; the same framework could serve as that test.
  • The frozen-encoder design means future, stronger generic video models could slot into the same pipeline without retraining the stream encoders, so the recipe is model-agnostic in principle.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether a generic video foundation model (V-JEPA) can be adapted for surgical video analysis and whether additional time-resolved operating-room data streams can improve downstream task performance. Using an in-house dataset of liver surgery videos for length-of-stay and complication prediction, and the public HeiCo dataset for surgical phase recognition, the authors compare pretrained vs. self-supervised finetuned V-JEPA and video-only vs. video-plus-streams setups. They report that finetuning improves results overall, that the pretrained video-only baseline achieves accuracy on par with top EndoVis2017 challenge submissions, and that adding OR streams helps on the in-house data. Code and model weights are released.

Significance. If the claims hold, the paper demonstrates a practical recipe: a publicly available, generic video foundation model can be adapted with modest compute to surgical phase recognition and outcome prediction, and OR sensor streams can add value when they carry task-relevant signal. The study is clearly described, uses public datasets, and the release of code and weights is a concrete contribution to reproducibility. The main weakness is that the headline external comparison to EndoVis2017 relies on an approximated evaluation protocol and on one of four metrics, while several finetuning and multimodal gains are small and mixed across tasks. These issues do not invalidate the direction of the work but require careful reframing and additional evidence.

major comments (3)
  1. [§3, 'HeiCo data handling' and Table 2] The claim of being 'on par' with top EndoVis2017 submissions rests on a non-identical evaluation protocol. The authors state they 'followed the EndoVis2017 evaluation as closely as possible,' skipping out-of-body segments and densely sampling at one-second intervals, and they report results both including and excluding class 13 without specifying which matches the leaderboard protocol. The original challenge may have scored all frames, used a different sampling rate, or handled class 13 differently. Moreover, on the other three metrics the pretrained video-only model trails the #1 entry (aIoU 37 vs. 40, mIoU 32 vs. 38, qIoU 25 vs. 37); only accuracy is higher (62 vs. 61). Calling this 'on par' is therefore metric-dependent and protocol-dependent. I recommend either obtaining official re-scoring on the challenge test set or clearly qualifying the headline as 'under our evaluation protocol
  2. [§3, Table 3 and §4] The claim that finetuning 'increases model performance' is stronger than the evidence. In the in-house LoS task, video-only accuracy is unchanged at 50% after finetuning (IoU1 only 11→13). In the CCI task, finetuning raises accuracy from 59% to 70% but lowers IoU1 from 19% to 13%, and the authors themselves note the decline 'could be random.' No confidence intervals, standard deviations, or multiple-seed results are reported, and evaluation is performed on densely sampled clips with per-patient labels repeated across frames, so clips from the same patient are not independent. At minimum, the abstract and discussion should be tempered, and the authors should report patient-level aggregated metrics or statistical intervals for the observed differences.
  3. [§3, Tables 2 and 3] The multimodal benefit is not consistent across tasks. On HeiCo, adding device-status streams lowers accuracy in both pretrained (62→59) and finetuned (64→62) settings, although mIoU improves. On the in-house data, the benefit is clear for LoS (50→58→63) but mixed for CCI: pretrained improves 59→75, while finetuned + streams (72) is lower than pretrained + streams (75) and only slightly above finetuned video-only (70). The paper acknowledges this nuance for HeiCo, but the abstract's statement that 'integrating additional time-resolved data likewise benefits the model' needs to be restricted to the LoS task or supported with an analysis of why the streams are informative there and not in CCI/HeiCo. This is central to hypothesis 3.
minor comments (5)
  1. [§2 heading] Typo: 'V-JEP A' should be 'V-JEPA'. Also 'T raining' at the start of the training subsection.
  2. [§3, 'HeiCo data handling'] Please specify exactly how class 13 is handled in the numbers compared with the EndoVis2017 leaderboard. The current text reports results both including and excluding class 13 but does not state which row/column corresponds to the challenge protocol.
  3. [Table 1] The learning-rate schedule differs between HeiCo and in-house step-2 training, but no rationale is given. A sentence explaining this choice would improve reproducibility.
  4. [§2, 'Multimodal extension' and Table 1] The state-change penalty weight (1e-3) and stream-encoder depth (4) are fixed without sensitivity analysis. Since the multimodal conclusions are mixed, a brief justification or a small ablation would help.
  5. [Figure 1] The use of emoji symbols in the figure may be distracting in a formal venue; consider replacing with standard vector-graphic symbols or text labels.

Circularity Check

0 steps flagged

No circularity: the claims rest on external benchmarks, held-out labels, and standard domain adaptation, not on self-referential definitions.

full rationale

The paper is an empirical study with no derivation chain that reduces to its own inputs. V-JEPA is a published external foundation model; finetuning is standard self-supervised continued training on unlabeled surgical videos, and all downstream evaluations use held-out labels from external or in-house datasets. The stream encoder is trained with downstream-task gradients plus a small state-change penalty that encourages similarity to V-JEPA embeddings; this is a regularizer, not a construction that forces the reported accuracy. The HeiCo comparison is against an external EndoVis2017 leaderboard, not against any quantity fitted in this paper. The only self-citations are to a dataset and related work by overlapping authors, and they are not load-bearing for the central claims. The paper explicitly notes a limitation at Section 2, step 3: 'since the V-JEPA embedding is an input to the encoder, we did not explore whether meaningful outputs could indeed be produced this way' — this is an acknowledged design limitation, not a circular step. Approximations in the EndoVis2017 protocol (skipping out-of-body frames, 1-second dense sampling) affect comparability but are a validation-methodology concern, not circularity. No fitted parameter is renamed a prediction, no uniqueness theorem is imported, and no known result is merely renamed. Score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper's contribution is an empirical evaluation, so the ledger records the external models and data assumptions it depends on rather than invented entities. No new particles, forces, or theoretical objects are introduced.

free parameters (3)
  • state change penalty weight = 10^-3
    Weight on the L2 state-change loss in step 3; chosen by hand, no ablation reported.
  • stream encoder depth = 4 layers
    Cross-attention fusion depth; set heuristically, no sensitivity analysis.
  • finetuning duration = 7 epochs (12h wall-clock)
    Compute-limited stop, not convergence-based; could affect the magnitude of finetuning benefit.
axioms (3)
  • domain assumption V-JEPA's pretrained video embeddings are a suitable initial state for surgical downstream tasks
    Central to the approach; supported by prior JEPA results on natural video, but not established for the surgical domain.
  • domain assumption Unlabeled laparoscopic videos from other procedures (Cholec80, MultiBypass140, HeiCo splits) provide sufficient domain-specific adaptation for in-house liver surgery tasks
    Finetuning uses these sources; if cross-procedure transfer fails, the finetuning benefit on liver data would not hold.
  • domain assumption The EndoVis2017 challenge evaluation protocol can be approximated with the stated assumptions (skipping out-of-body segments, one-second dense sampling)
    Supports the 'on par with top performers' claim.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis." pith.science (2026). https://pith.science/paper/G5OR3IEW

@misc{pith2026250906831,
  author       = {Pith},
  title        = {Pith review of: Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G5OR3IEW}},
  note         = {Machine review of arXiv:2509.06831}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We investigate how both the adaptation of a generic foundation model via transfer learning and the integration of complementary modalities from the operating room (OR) can support surgical data science. To this end, we use V-JEPA as the single-modality foundation of a multimodal model for minimally invasive surgery support. We analyze how the model's downstream performance can benefit (a) from finetuning on unlabeled surgical video data and (b) from providing additional time-resolved data streams from the OR in a multimodal setup. In an in-house dataset of liver surgery videos, we analyze the tasks of predicting hospital length of stay and postoperative complications. In videos of the public HeiCo dataset, we analyze the task of surgical phase recognition. As a baseline, we apply pretrained V-JEPA to all tasks. We then finetune it on unlabeled, held-out videos to investigate its change in performance after domain adaptation. Following the idea of modular decision support networks, we integrate additional data streams from the OR by training a separate encoder to form a shared representation space with V-JEPA's embeddings. Our experiments show that finetuning on domain-specific data increases model performance. On the in-house data, integrating additional time-resolved data likewise benefits the model. On the HeiCo data, accuracy of the pretrained video-only, single-modality baseline setup is on par with the top-performing submissions of the EndoVis2017 challenge, while finetuning on domain-specific data increases accuracy further. Our results thus demonstrate how surgical data science can leverage public, generic foundation models. Likewise, they indicate the potential of domain adaptation and of integrating suitable complementary data streams from the OR. To support further research, we release our code and model weights at https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025.

Figures

Figures reproduced from arXiv: 2509.06831 by Beat P. M\"uller, Jan S. Liechti, J\'er\^ome A. Kurylec, Jo\"el L. Lavanchy, Simon Pezold.

Figure 1
Figure 1. Figure 1: Architecture and training steps. V-JEPA training step (top left): Video encoder EV,θ is trained by producing an embedding s (green) of the unmasked tokens from par￾tially masked, tokenized video data M(t) (red), which enables auxiliary predictor P to produce an estimate sˆ of the masked video tokens’ embedding s¯, provided by expo￾nential moving average (ema) teacher EV,θ¯ (see main text for details). Used… view at source ↗
Figure 2
Figure 2. Figure 2: Employed public data. Videos by dataset for V-JEPA finetuning (left); videos and splits used for the surgical phase recognition experiment on HeiCo (right). 00:00h 24:00h 48:00h 72:00h training validation testing LoS 00:00h 24:00h 48:00h 72:00h training validation testing LoS 00:00h 24:00h 48:00h 72:00h training validation testing CCI 00:00h 24:00h 48:00h 72:00h training validation testing CCI [PITH_FULL_… view at source ↗
Figure 3
Figure 3. Figure 3: Employed in-house data. Videos and splits used for the experiments of predict￾ing length of stay (LoS) in terms of postoperative hospital days (left) and complications in terms of CCI (right) [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages · 1 internal anchor

  1. [1]

    https://doi.org/10.48550/arXiv.2404.08471

    Bardes, A., Garrido, Q., Ponce, J., Chen, X., Rabbat, M., LeCun, Y., Assran, M., Ballas, N.: Revisiting Feature Prediction for Learning Visual Representations from Video (Feb 2024). https://doi.org/10.48550/arXiv.2404.08471

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.08471 2024

  2. [2]

    https://doi.org/10.48550/arXiv.2010.11929

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Jun 2021). https://doi.org/10.48550/arXiv.2010.11929

    Pith/arXiv arXiv doi:10.48550/arxiv.2010.11929 2021

  3. [3]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked Autoencoders Are Scalable Vision Learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16000–16009 (2022)

    work page 2022

  4. [4]

    Hemker, K., Simidjievski, N., Jamnik, M.: HEALNet: Multimodal Fusion for Het- erogeneous Biomedical Data. Advances in Neural Information Processing Systems 37, 64479–64498 (Dec 2024), https://proceedings.neurips.cc/paper_files/paper/ 2024/hash/765871e77d2ca65126d3d64d31aa6908-Abstract-Conference.html

    work page 2024

  5. [5]

    IEEE Reviews in Biomedical Engineer- ing pp

    Khan, W., Leem, S., See, K.B., Wong, J.K., Zhang, S., Fang, R.: A Comprehensive Survey of Foundation Models in Medicine. IEEE Reviews in Biomedical Engineer- ing pp. 1–22 (2025). https://doi.org/10.1109/RBME.2025.3531360 12 S. Pezold et al

    work page arXiv 2025

  6. [6]

    International Journal of Computer Assisted Radiology and Surgery (May 2024)

    Lavanchy, J.L., Ramesh, S., Dall’Alba, D., Gonzalez, C., Fiorini, P., Müller-Stich, B.P., Nett, P.C., Marescaux, J., Mutter, D., Padoy, N.: Challenges in multi-centric generalization: phase and step recognition in Roux-en-Y gastric bypass surgery. International Journal of Computer Assisted Radiology and Surgery (May 2024). https://doi.org/10.1007/s11548-0...

    doi:10.1007/s11548-024-03166-3 2024

  7. [7]

    Scientific Data8, 101 (Apr 2021)

    Maier-Hein, L., Wagner, M., Ross, T., Reinke, A., Bodenstedt, S., Full, P.M., Hempe, H., Mindroc-Filimon, D., Scholz, P., Tran, T.N., Bruno, P., Kisilenko, A., Müller, B., Davitashvili, T., Capek, M., Tizabi, M.D., Eisenmann, M., Adler, T.J., Gröhl, J., Schellenberg, M., Seidlitz, S., Lai, T.Y.E., Pekdemir, B., Roeth- lingshoefer, V., Both, F., Bittel, S....

    work page 2021

  8. [8]

    https://doi.org/10.48550/arXiv

    Schmidgall, S., Cho, J., Zakka, C., Hiesinger, W.: GP-VLS: A general-purpose vision language model for surgery (Aug 2024). https://doi.org/10.48550/arXiv. 2407.19305

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.19305 2024

  9. [9]

    Annals of Surgery258(1), 1 (Jul 2013)

    Slankamenac, K., Graf, R., Barkun, J., Puhan, M.A., Clavien, P.A.: The Com- prehensive Complication Index: A Novel Continuous Scale to Measure Surgical Morbidity. Annals of Surgery258(1), 1 (Jul 2013). https://doi.org/10.1097/SLA. 0b013e318296c732

    doi:10.1097/sla 2013

  10. [10]

    MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks

    Swamy, V., Satayeva, M., Frej, J., Bossy, T., Vogels, T., Jaggi, M., Käser, T., Hart- ley, M.A.: MultiModN—Multimodal, Multi-Task, Interpretable Modular Networks (Nov 2023). https://doi.org/10.48550/arXiv.2309.14118

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2309.14118 2023

  11. [11]

    PLOS Digital Health2(7), e0000108 (Jul 2023)

    Trottet, C., Vogels, T., Keitel, K., Kulinkina, A.V., Tan, R., Cobuccio, L., Jaggi, M., Hartley, M.A.: Modular Clinical Decision Support Networks (MoDN)—Updatable, interpretable, and portable predictions for evolving clinical environments. PLOS Digital Health2(7), e0000108 (Jul 2023). https://doi.org/10. 1371/journal.pdig.0000108

    work page 2023

  12. [12]

    IEEE Transactions on Medical Imaging36(1), 86–97 (Jan 2017)

    Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., de Mathelin, M., Padoy, N.: EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. IEEE Transactions on Medical Imaging36(1), 86–97 (Jan 2017). https://doi.org/ 10.1109/TMI.2016.2593957

    arXiv 2017

  13. [13]

    In: Advances in Neural Information Processing Systems

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is All you Need. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017)

    work page 2017

  14. [14]

    Yann LeCun: A Path Towards Autonomous Machine Intelligence (Jun 2022), https: //openreview.net/forum?id=BZ5a1r-kVsf

    work page 2022

  15. [15]

    International Journal of Computer Assisted Radiology and Surgery19(7), 1409–1417 (Jul 2024)

    Yuan, K., Kattel, M., Lavanchy, J.L., Navab, N., Srivastav, V., Padoy, N.: Advanc- ing surgical VQA with scene graph knowledge. International Journal of Computer Assisted Radiology and Surgery19(7), 1409–1417 (Jul 2024). https://doi.org/10. 1007/s11548-024-03141-y

    work page 2024

  16. [16]

    Medical Image Analysis105, 103644 (Oct 2025)

    Yuan, K., Srivastav, V., Yu, T., Lavanchy, J.L., Marescaux, J., Mascagni, P., Navab, N., Padoy, N.: Learning multi-modal representations by watching hun- dreds of surgical video lectures. Medical Image Analysis105, 103644 (Oct 2025). https://doi.org/10.1016/j.media.2025.103644

    work page arXiv 2025

  17. [17]

    Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: LiT: Zero-Shot Transfer With Locked-Image Text Tuning. pp. 18123–18133 (2022), https://openaccess.thecvf.com/content/CVPR2022/html/Zhai_LiT_Zero-Shot_ Transfer_With_Locked-Image_Text_Tuning_CVPR_2022_paper.html Leveraging generic foundation models for multimodal surgical...

    work page 2022

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.