Pith. sign in

REVIEW 3 major objections 5 minor 80 references

Fine-tuning a vision encoder with latent-action and forward-prediction losses, plus a patch-level distribution regularizer, improves surgical interaction recognition and grounds change features in instrument-tissue regions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:24 UTC pith:ZHMMNLWF

load-bearing objection A well-engineered fine-tuning recipe with consistent but modest gains; the latent-action mechanism story is weaker than the abstract implies, and the grounding gains are carried by the patch-SIG regularizer, not the IDM/FWM. the 3 major comments →

arxiv 2607.19889 v1 pith:ZHMMNLWF submitted 2026-07-22 cs.CV

LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition

Jiajun Cheng , Subarna Tripathi , Sainan Liu , Xiaofan Yu , Shan Lin This is my paper
classification cs.CV
keywords surgical action-triplet recognitionlatent-action modelinginverse dynamics modelforward world modelpatch-level SIG regularizervision-language fine-tuninginstrument-tissue interactionspatial grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a vision encoder can be fine-tuned for surgical instrument-tissue interaction recognition by training it jointly with a latent-action module, rather than by freezing the encoder or fine-tuning on classification alone. The authors argue that standard fine-tuning aligns global semantics but leaves the encoder weak at localizing where an action happens, and they propose a recipe: an inverse dynamics model compresses the difference between consecutive frame states into a low-dimensional action token, a forward world model must use that token to predict the next state, and a patch-level distribution regularizer keeps local features from collapsing. On two surgical action-triplet benchmarks and across three pretrained encoders, the method improves triplet mean average precision over each encoder's standard fine-tuned baseline, with the best configuration reaching 18.50 and 27.51. The authors further report that the compressed transition subspace becomes more concentrated in instrument-tissue regions, measured by higher spatial grounding IoU, and that the patch regularizer is what prevents overly aggressive compression from killing that grounding. A sympathetic reader would care because the method offers a way to adapt pretrained visual representations to fine-grained, spatially grounded video understanding without bounding boxes or interaction-region annotations.

Core claim

On the paper's own terms, the central claim is that adding latent-action supervision to vision-language fine-tuning reshapes a pretrained vision encoder toward interaction-relevant spatial and transition features, and that this yields better performance than standard fine-tuning of the same encoder. The authors' mechanism: an inverse dynamics model that operates on the difference between consecutive pooled states sends that change through a low-dimensional bottleneck to produce a compact latent action; the forward world model is conditioned on this action via adaptive layer normalization and must reconstruct the next state from a stop-gradient target, so the action must carry exactly the tra

What carries the argument

The load-bearing object is a cheap latent-action bottleneck: a difference-based inverse dynamics model maps the difference between consecutive frame states to a 128-dimensional action token, and a DiT-style forward world model uses that token as the only transition path to predict the next state from the current one. Because the action dimension is far smaller than the state dimension, the forward model cannot copy the next state through the action, so the encoder must compress frame-to-frame change into few informative directions. The second mechanistic piece is the patch-level SIG regularizer, a sliced Cramer-Wold Epps-Pulley test that drives randomly sampled per-patch token distributions

Load-bearing premise

The paper's central grounding claim rests on the untested premise that supervising the forward model with a single average feature per frame, together with a Gaussianity regularizer on random patches, is enough to push patch-level features toward the instrument-tissue interaction region.

What would settle it

Train the same framework but supervise the forward model with patch-level rather than pooled-state targets, or recompute the grounding IoU using manually annotated interaction masks instead of propagated ones; if the reported IoU gains vanish or reverse under either change, the paper's attribution of grounding to the state-level IDM/FWM loss and its evaluation both come into question.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Across a vanilla transformer encoder, a self-distillation image encoder, and a surgically pretrained variant of the same architecture, latent-action guidance improves triplet mean average precision over each encoder's standard fine-tuned baseline on both benchmarks, with the largest gains on the weakest encoder.
  • An effective-dimension measure of the transition covariance drops substantially under IDM/FWM training (for example, from about 157 to 34 for the vanilla encoder), showing that inter-state change is compressed into fewer dominant directions.
  • Without the patch-level regularizer, compression collapses further (effective dimensions around 13-18 and the leading-component count capturing 90 percent of variance falls as low as 19), while grounding IoU stagnates or drops; with it, grounding IoU rises to 0.074, 0.137, and 0.181 across the three encoders.
  • Dropping action tokens at inference lowers triplet mAP (for example, from 17.22 to 15.79 on one benchmark and 23.58 to 21.84 on the other), so the latent action carries information beyond the training-time objective.
  • Both too small and too large action dimensions weaken subspace-grounding IoU and cluster quality, suggesting an intermediate bottleneck size is needed to keep action-relevant transitions concentrated.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the proposed mechanism transfers, the same inverse-dynamics plus forward-model plus patch-regularizer recipe could improve other fine-grained video-recognition tasks that need spatially grounded change features, such as hand-object interaction or robot manipulation, without region annotations.
  • The paper supervises the forward model with mean-pooled frame states; a natural stress-test is to replace the pooled target with patch- or block-level targets. If grounding improves further, the current attribution understates the role of target granularity; if it does not, the pooled-signal premise is corroborated.
  • Because the reported grounding numbers inherit the accuracy of the propagated segmentation masks used as ground truth, an evaluation on a small set of manually annotated frames would establish how much of the IoU gain survives mask-estimation error.
  • The results show CH separability and grounding IoU can move in opposite directions (the patch regularizer sometimes lowers CH while raising mAP), so representation studies of fine-tuning should report cluster-quality and grounding metrics jointly rather than in isolation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LAViFiT, a fine-tuning framework for surgical action-triplet recognition that couples a pretrained vision encoder with an inverse dynamics model (IDM), a forward world model (FWM), and a patch-level SIG regularizer. The IDM maps pooled-frame-state differences into low-dimensional latent actions; the FWM predicts the next pooled state from the current state and action; and SIG regularizers are applied to states, actions, and sampled patch tokens. The method is evaluated on CholecT50 and ProstaTD across ViT-L, DINOv2-L, and SurgeNet-L backbones, reporting gains in triplet mAP over standard fine-tuning and over a CLIP4Clip-style states-only baseline. The paper also presents mechanism analyses: PCA visualizations, Calinski-Harabasz separability, transition-covariance effective rank, and IoU of dominant change directions with an instrument-tissue interaction mask.

Significance. If the recognition claims hold, LAViFiT offers a practical fine-tuning recipe for surgical vision encoders that needs no region-level supervision and is specified with enough detail (App. A.1) to reproduce. The paper ships a public code link, uses external benchmarks and standard metrics, and its recognition ablations are internally consistent. The mechanism story, however, goes beyond what Table 2 supports: the IDM/FWM module alone does not improve the reported grounding IoU, and the patch-level regularizer is spatially agnostic, so the causal chain from latent-action compression to interaction-region grounding is not established. The paper's central contribution is defensible, but the overclaimed mechanism and the missing control condition require a major revision.

major comments (3)
  1. [§6, §5.4, Table 2] The conclusion states that 'compact IDM-FWM training ... improves their spatial alignment with action-relevant instrument-tissue regions.' Table 2 contradicts this: comparing the No-IDM&FWM row with the No-Patch-SIG row (which retains IDM/FWM), IoU_norm changes only 0.057→0.060 for ViT-L, 0.081→0.080 for DINOv2, and 0.111→0.096 for SurgeNet. The IDM/FWM module compresses transitions (PR/d90 drop) but does not by itself improve grounding; the large IoU gains appear only when patch-SIG is added. The supported claims are (a) IDM/FWM compresses transition covariance and (b) patch-SIG improves IoU; the claim that IDM/FWM improves spatial alignment must be re-stated or supported with additional evidence.
  2. [§3.2, Eqs. (7)-(8), Table 2] No ablation isolates the patch-SIG regularizer without IDM/FWM, so the paper cannot attribute the final grounding improvement to the combination. Ordinary patch-SIG is spatially agnostic: it samples 32 random patches per frame and matches them to an isotropic Gaussian (Eq. 8), with no exposure to the interaction-region mask M_act (Eq. 11) or any spatial conditioning. A control with 'No IDM/FWM + patch-SIG' would determine whether patch-SIG alone yields the IoU increases reported in Table 2. Without such a control, the causal narrative in §5.4 ('IDM/FWM compresses -> patch-SIG restores diversity -> grounding improves') is underdetermined and should be labeled a hypothesis.
  3. [§3.1, Eq. (1); §3.2, Eq. (5)] The IDM/FWM supervision is routed through frame-level pooled states s_t = pool(p_t), and the forward-prediction loss (Eq. 5) supervises only these pooled states. The latent action therefore carries no direct patch-level spatial information. Yet the paper claims that this pooled-state bottleneck shapes patch-level feature geometry and grounds transitions in the instrument-tissue region. This is a load-bearing gap: the mechanism by which a spatially impoverished state signal transfers to patch-level grounding is neither derived nor tested. The CH and IoU analyses are post-hoc evaluations; no training signal connects them to the interaction mask. Please either provide evidence for this transfer (e.g., show action-token dependence on per-patch motion) or explicitly acknowledge it as an untested premise.
minor comments (5)
  1. [Table 1, §4.1] The main table compares single-frame image-encoder baselines with an 8-frame LAViFiT pipeline. Although a states-only CLIP4Clip-style baseline is provided in the ablations, the main-table caption should clearly state that the standard fine-tuning baselines for image encoders use a single frame while the method uses an 8-frame clip, to avoid misleading readers about the source of temporal gain.
  2. [§1, §4, Table 1] Text in §4.1 refers to 'fine-tuned CLIP ViT-L/16 baseline' while the table and method use ViT-L/14. Please correct the mismatched notation and specify the exact patch size used for each encoder.
  3. [Eq. (10) vs. Eq. (2)] The symbol v is used both for the aggregated clip embedding (Eq. 2) and for the verb component (Eq. 10, y_t = (i_t, v_t, r_t)). This is confusing in a paper where verb AP is a reported metric; please disambiguate.
  4. [§5, Table 1] CholecT50 results are from a single run, and the SurgeNet-L improvement over standard fine-tuning is small (18.04→18.50). ProstaTD results include means and std across folds, but no paired significance or confidence intervals are reported. Please add per-fold numbers or a statistical test to support the claim of consistent gains, particularly for the smallest effect.
  5. [Table 1, No action tokens at inference] Removing action tokens only at inference is an input-corruption test, not a training-ablation: the model was optimized with action tokens and then evaluated without them. The interpretation should be framed accordingly, not as evidence that action tokens are the sole driver of training.

Circularity Check

0 steps flagged

No significant circularity: recognition and grounding claims rest on external benchmarks, evaluation masks, and ablations; self-citations are contextual only.

full rationale

LAViFiT's central recognition result is an empirical comparison against external benchmarks (CholecT50 RDV split, ProstaTD 5-fold) using the standard ivtmetrics mAP; no fitted parameter is defined in terms of the reported mAP. The grounding analysis uses SASVI-propagated CholecSeg8k masks and ground-truth triplet instrument/target annotations only as an evaluation region (Eq. 11), not as supervision in the training objective (Eq. 9); the patch-level SIG regularizer (Eq. 8) samples random patches and matches them to an isotropic Gaussian with no access to M_act, so the reported IoU_norm improvements are not forced by construction. The same-author citations [3] and [49] appear in related-work or motivation contexts (explainability concerns; prior trajectory-supervised methods) and are not load-bearing for any derivation. The paper's own Table 2 does raise a correctness concern about attributing grounding gains to the IDM/FWM module, but that is an underdetermination/mechanism-interpretation issue, not a circular reduction: no equation makes the grounding metric equal to an optimized loss or to a self-citation. Thus the derivation chain is self-contained with respect to circularity; independent support comes from external benchmarks, fixed evaluation masks, and standard held-out test protocols.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The ledger is dominated by hand-chosen hyperparameters (action dimension, loss weights, SIGReg projection counts, per-encoder learning rates) and by domain assumptions inherited from the latent-action literature: that a small bottleneck recovers action-induced transitions [47], that mean-pooled states carry enough spatial signal for the FWM loss to reshape patch features, and that Gaussianity-matching prevents collapse without destroying structure. No new physical or empirical entities are introduced; the latent action is an internal code with no independent falsifiable handle.

free parameters (5)
  • latent action dimension d_a = 128
    Hand-chosen bottleneck width for all recognition results; §A.2 sweep (Fig. 5-6) shows different encoders and component metrics peak at different d_a (some near 256), so 128 is a compromise point of a hyperparameter sweep, not a derived value.
  • loss weights λ_pred, λ_s, λ_p = 1.0, 0.1, 0.3
    Set by hand (App. A.1); §6 concedes that end-to-end world-model training with SIG-style regularization 'is highly sensitive to hyperparameter choices'.
  • SIGReg projection counts M = 1024 (state/action), 256 (patch)
    Hand-chosen Monte-Carlo approximations of the sliced Cramér-Wold Epps-Pulley statistic (Eq. 6, §3.2).
  • per-component learning rates = 3e-5 to 1e-4 depending on encoder (Table 3)
    Hand-chosen per backbone; different encoders get different learning rates, which softens the 'fair comparison' claim even though the paper's baselines share the same rates.
  • patch subset size |K_t| = 32 patches per frame
    Random subsample per frame to keep the O(N^2) SIGReg cost tractable (Eq. 8); the randomness affects which patches are regularized in each step.
axioms (5)
  • domain assumption A compact latent-action bottleneck forces the encoder to represent action-induced transition change and suppresses task-irrelevant nuisance variation.
    Motivation for IDM/FWM (§1, §3.2, §5.4); the paper cites [47] which shows IDMs can recover action-induced transitions but that 'structured nuisance variation may also be retained' — the caveat is acknowledged but assumed away in the surgical setting.
  • domain assumption Mean-pooled patch states retain enough information for the IDM/FWM objective to steer patch-level features toward the interaction region.
    Eq. 1 defines states as the mean over patch tokens; L_pred (Eq. 5) supervises only these pooled states, yet the grounding claims are at patch level. No argument is given for why pooled-state supervision transfers to patch features.
  • domain assumption SIGReg Gaussianization prevents representational collapse without destroying task-relevant feature structure.
    Adopted from LeWorldModel/LeJEPA [80,33]; the paper's own §5.4 shows it lowers CH on already-well-separated frames, so the trade-off is real; the weights 0.1/0.3 are asserted to hit the right balance.
  • domain assumption SASVI-propagated CholecSeg8k masks are a valid ground truth for the instrument-tissue interaction region in evaluation.
    All grounding and CH measurements (§4.2, §4.3, Table 2) use masks from an external segmentation-estimation method; errors in those masks would directly bias the IoU numbers.
  • domain assumption Eight-frame clips at 1 fps capture the action-relevant transitions between consecutive states.
    States are 1-fps frames and the FWM predicts next-frame states; fast instrument motion may alias across frames, and the paper does not vary clip length or frame rate.
invented entities (1)
  • latent action a_t (d_a=128) no independent evidence
    purpose: Low-dimensional code for the inter-frame transition, used by the forward world model and by the clip aggregator for triplet prediction.
    An internal latent variable of this method family (LAPO [71], Delta-JEPA [55], LeWorldModel [80]); its meaning is inferred from post-hoc analysis (§5.3) rather than from any external measurement, so it has no falsifiable handle outside the framework itself.

pith-pipeline@v1.3.0-alltime-deepseek · 20186 in / 18963 out tokens · 195162 ms · 2026-08-01T11:24:28.058575+00:00 · methodology

0 comments
read the original abstract

Understanding instrument-tissue interactions is essential for context-aware surgical AI and autonomous robotic surgery. Pretrained vision-language models (VLMs) and vision encoders offer an alternative to conventional interaction classifiers by transferring broad visual and semantic knowledge. However, adapting them to fine-grained surgical interactions remains challenging: (1) freezing the vision encoder depends entirely on pretrained representations that may retain noise and provide weak spatial localization, while (2) full fine-tuning can improve global semantic alignment without ensuring that the encoder learns meaningful features in the correct action region. We address these limitations by introducing LAViFiT, an end-to-end latent-action-guided framework for vision-language fine-tuning. An inverse dynamics model captures the visual changes induced by each action, while a forward world model drives the encoder to represent action-relevant regions. A patch-level SIG Regularizer further prevents local feature collapse without additional supervision, such as bounding boxes or pseudo-labels. Experiments across multiple encoders and datasets improve recognition and image-text alignment, while representation analyses show stronger grounding over the complete instrument-tissue interaction region and more spatially coherent features.

Figures

Figures reproduced from arXiv: 2607.19889 by Jiajun Cheng, Sainan Liu, Shan Lin, Subarna Tripathi, Xiaofan Yu.

Figure 1
Figure 1. Figure 1: Visualization of vision encoder behavior under standard fine-tuning and our method, and the changes [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of LAViFiT. The vision encoder maps a surgical clip to state tokens, while the IDM compresses consecutive-state changes into latent actions and the FWM predicts the next state using a stop-gradient target. Frame￾and patch-level SIGReg prevent collapse, and the state-action sequence is aggregated and the aggregation token aligned with instrument-verb-target text embeddings. 2.3 Inverse Dynamics and… view at source ↗
Figure 3
Figure 3. Figure 3: PCA-RGB visualizations of encoders trained with and without our latent action framework on (a) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Per-bin CH→ mAP trajectories (da = 128, triplet mAP). Test frames are split into five CH quintile bins (z-scored, pooled per pair); each arrow joins the mean CH (x) and triplet mAPIVT (y) of the two models (start hollow, end filled), bins colored dark → bright for low → high CH. CH is over the Instrument-tissues interaction region object classes plus one background class, so it rewards objects separating f… view at source ↗
Figure 5
Figure 5. Figure 5: Effect of the action-token dimension da on CholecT50. Each panel plots mAP versus da ∈ {64,...,1024} for the three encoders (ViT-L, DINOv2, SurgeNet), for the triplet metric and the instrument/verb/target components [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of the action dimension da on representation quality for ViT-L, DINOv2-L, and SurgeNet-L. (a) Subspace-grounding IoU and (b) CH score versus da. The grounding IoU is computed from the top-r eigenvectors of the transition covariance with r=128 fixed for all models. Each panel uses its own y-scale to make per-encoder trends visible. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 23 linked inside Pith

  1. [1]

    & Speidel, S

    Kati ´c, D., Wekerle, A., Gärtner, F., Kenngott, H., Müller-Stich, B., Dillmann, R. & Speidel, S. A System for Context-Aware Intra-Operative Augmented Reality and Surgical Action Triplet Modeling.Information Processing In Computer-Assisted Interventions (IPCAI). (2014)

    2014

  2. [2]

    & Pinto, L

    Cui, Z., Pan, H., Iyer, A., Haldar, S. & Pinto, L. Dynamo: In-domain dynamics pretraining for visuo-motor control.Advances In Neural Information Processing Systems.37pp. 33933-33961 (2024)

    2024

  3. [3]

    & Lin, S

    Cheng, J., Zhao, X., Liu, S., Yu, X., Prakash, R., Codd, P., Katz, J. & Lin, S. Surgxbench: Explainable vision- language model benchmark for surgery.Proceedings Of The IEEE/CVF Winter Conference On Applications Of Computer Vision. pp. 8188-8198 (2026)

    2026

  4. [4]

    & Padoy, N

    Nwoye, C., Gonzalez, C., Yu, T., Mascagni, P., Mutter, D., Marescaux, J. & Padoy, N. Recognition of Instrument- Tissue Interactions in Endoscopic Videos via Action Triplets.Medical Image Computing And Computer Assisted Intervention (MICCAI). pp. 364-374 (2020)

    2020

  5. [5]

    & Padoy, N

    Nwoye, C., Yu, T., Gonzalez, C., Seeliger, B., Mascagni, P., Mutter, D., Marescaux, J. & Padoy, N. Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos.Medical Image Analysis.78pp. 102433 (2022)

    2022

  6. [6]

    Luo, H., Ji, L., Zhong, M., Chen, Y ., Lei, W., Duan, N. & Li, T. Clip4clip: An empirical study of clip for end to end video clip retrieval.ArXiv Preprint ArXiv:2104.08860. (2021)

    Pith/arXiv arXiv 2021

  7. [7]

    & Others UniSurg: A Video-Native Foundation Model for Universal Understanding of Surgical Videos.ArXiv Preprint ArXiv:2602.05638

    Wu, J., Holm, F., Chen, C., Wang, A., Hu, Y ., Ye, X., Zang, Z., Xu, M., Zhou, L., Liao, H. & Others UniSurg: A Video-Native Foundation Model for Universal Understanding of Surgical Videos.ArXiv Preprint ArXiv:2602.05638. (2026)

    Pith/arXiv arXiv 2026

  8. [8]

    & Padoy, N

    Sharma, S., Mutter, D. & Padoy, N. fine-clip: Enhancing zero-shot fine-grained surgical action recognition with vision-language models.ArXiv Preprint ArXiv:2503.19670. (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    & Torresani, L

    Bertasius, G., Wang, H. & Torresani, L. Is space-time attention all you need for video understanding?.Icml.2pp. 4 (2021)

    2021

  10. [10]

    & Qin, J

    Chen, Y ., Li, Z., Xu, C., Liu, A., Cui, R., Xu, X., Teoh, J., He, S. & Qin, J. ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection.The Fourteenth International Conference On Learning Representations. (2026), https://openreview.net/forum?id=0NkXZ98BjJ

    2026

  11. [11]

    & Others Embeddinggemma: Powerful and lightweight text representations.ArXiv Preprint ArXiv:2509.20354

    Vera, H., Dua, S., Zhang, B., Salz, D., Mullins, R., Panyam, S., Smoot, S., Naim, I., Zou, J., Chen, F. & Others Embeddinggemma: Powerful and lightweight text representations.ArXiv Preprint ArXiv:2509.20354. (2025)

    Pith/arXiv arXiv 2025

  12. [12]

    & Sommen, F

    Jaspers, T., Jong, R., Li, Y ., Kusters, C., Bakker, F., Jaarsveld, R., Kuiper, G., Hillegersberg, R., Ruurda, J., Brinkman, W., Pluim, J., With, P., Breeuwer, M., Khalil, Y . & Sommen, F. Scaling up self-supervised learning for improved surgical foundation models.Medical Image Analysis.108pp. 103873 (2026), https://www.sciencedirect.com/science/article/p...

    2026

  13. [13]

    & Others Learning transferable visual models from natural language supervision.International Conference On Machine Learning

    Radford, A., Kim, J., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. & Others Learning transferable visual models from natural language supervision.International Conference On Machine Learning. pp. 8748-8763 (2021)

    2021

  14. [14]

    & Others Dinov2: Learning robust visual features without supervision.ArXiv Preprint ArXiv:2304.07193

    Oquab, M., Darcet, T., Moutakanni, T., V o, H., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A. & Others Dinov2: Learning robust visual features without supervision.ArXiv Preprint ArXiv:2304.07193. (2023)

    Pith/arXiv arXiv 2023

  15. [15]

    & Bardes, A

    Mur-Labadia, L., Muckley, M., Bar, A., Assran, M., Sinha, K., Rabbat, M., LeCun, Y ., Ballas, N. & Bardes, A. V-jepa 2.1: Unlocking dense features in video self-supervised learning.ArXiv Preprint ArXiv:2603.14482. (2026)

    Pith/arXiv arXiv 2026

  16. [17]

    & Liang, P

    Kumar, A., Raghunathan, A., Jones, R., Ma, T. & Liang, P. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution.International Conference On Learning Representations. (2022)

    2022

  17. [18]

    & Bach, S

    Menghini, C., Delworth, A. & Bach, S. Enhancing clip with clip: Exploring pseudolabeling for limited-label prompt tuning.Advances In Neural Information Processing Systems.36pp. 60984-61007 (2023)

    2023

  18. [19]

    & Bojanowski, P

    Darcet, T., Oquab, M., Mairal, J. & Bojanowski, P. Vision transformers need registers.International Conference On Learning Representations.2024pp. 2632-2652 (2024)

    2024

  19. [20]

    & Yuille, A

    Wang, F., Mei, J. & Yuille, A. Sclip: Rethinking self-attention for dense vision-language inference.European Conference On Computer Vision. pp. 315-332 (2024)

    2024

  20. [21]

    Bu, Q., Yang, Y ., Cai, J., Gao, S., Ren, G., Yao, M., Luo, P. & Li, H. Univla: Learning to act anywhere with task-centric latent actions.ArXiv Preprint ArXiv:2505.06111. (2025)

    Pith/arXiv arXiv 2025

  21. [22]

    & Chen, Z

    Sun, J., Zhang, W., Qi, Z., Ren, S., Liu, Z., Zhu, H., Sun, G., Jin, X. & Chen, Z. Vla-jepa: Enhancing vision-language-action model with latent world model.ArXiv Preprint ArXiv:2602.10098. (2026)

    arXiv 2026

  22. [23]

    & Zhang, W

    Lan, M., Chen, C., Ke, Y ., Wang, X., Feng, L. & Zhang, W. Clearclip: Decomposing clip representations for dense vision-language inference.European Conference On Computer Vision. pp. 143-160 (2024)

    2024

  23. [24]

    & Jampani, V

    El Banani, M., Raj, A., Maninis, K., Kar, A., Li, Y ., Rubinstein, M., Sun, D., Guibas, L., Johnson, J. & Jampani, V . Probing the 3d awareness of visual foundation models.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 21795-21806 (2024)

    2024

  24. [25]

    & Ghanem, B

    Zohra, F., Zhao, C., Itani, H. & Ghanem, B. b-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 680-689 (2026)

    2026

  25. [26]

    Cali ´nski, T. & JA, H. A Dendrite Method for Cluster Analysis.Communications In Statistics - Theory And Methods.3pp. 1-27 (1974,1)

    1974

  26. [27]

    & Eom, C

    Choi, H., Jang, Y . & Eom, C. Goal: Global-local object alignment learning.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 4070-4079 (2025)

    2025

  27. [28]

    & Xiong, H

    Ding, S., Li, M., Yang, T., Qian, R., Xu, H., Chen, Q., Wang, J. & Xiong, H. Motion-aware contrastive video representation learning via foreground-background merging.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 9716-9726 (2022)

    2022

  28. [29]

    & Shih, C

    Hong, W., Kao, C., Kuo, Y ., Wang, J., Chang, W. & Shih, C. Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80.ArXiv Preprint ArXiv:2012.12453. (2020)

    Pith/arXiv arXiv 2012

  29. [30]

    & Mukhopadhyay, A

    Sivakumar, S., Frisch, Y ., Ranem, A. & Mukhopadhyay, A. SASVi: segment any surgical video.International Journal Of Computer Assisted Radiology And Surgery.20, 1409-1419 (2025)

    2025

  30. [31]

    & Khan, F

    Khattak, M., Rasheed, H., Maaz, M., Khan, S. & Khan, F. Maple: Multi-modal prompt learning.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 19113-19122 (2023)

    2023

  31. [32]

    & Yuan, J

    Xi, N., Meng, J. & Yuan, J. Chain-of-Look Prompting for Verb-Centric Surgical Triplet Recognition in Endoscopic Videos.Proceedings Of The 31st ACM International Conference On Multimedia. pp. 5007-5016 (2023)

    2023

  32. [33]

    & LeCun, Y

    Balestriero, R. & LeCun, Y . Lejepa: Provable and scalable self-supervised learning without the heuristics.ArXiv Preprint ArXiv:2511.08544. (2025)

    Pith/arXiv arXiv 2025

  33. [34]

    & Tang, J

    Li, P., Shu, X., Feng, Y ., Feng, Y ., Zuo, W. & Tang, J. Surgical Video Workflow Analysis via Visual-Language Learning.Npj Health Systems.2, 5 (2025)

    2025

  34. [35]

    & Mohareri, O

    Jamal, M. & Mohareri, O. SurgMAE: Masked autoencoders for long surgical video analysis.ArXiv Preprint ArXiv:2305.11451. (2023) 14 LA VIFiT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction RecognitionA PREPRINT

    Pith/arXiv arXiv 2023

  35. [36]

    & Navab, N

    Bati ´c, D., Holm, F., Özsoy, E., Czempiel, T. & Navab, N. Whether and When does Endoscopy Domain Pretraining Make Sense?.ArXiv Preprint ArXiv:2303.17636. (2023)

    Pith/arXiv arXiv 2023

  36. [38]

    & Garcia-Peraza-Herrera, L

    Che, C., Wang, C., Vercauteren, T., Tsoka, S. & Garcia-Peraza-Herrera, L. Lemon: A large endoscopic monocular dataset and foundation model for perception in surgical settings.Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition. pp. 42659-42669 (2026)

    2026

  37. [39]

    & Padoy, N

    Yuan, K., Srivastav, V ., Yu, T., Lavanchy, J., Marescaux, J., Mascagni, P., Navab, N. & Padoy, N. Learning multi- modal representations by watching hundreds of surgical video lectures.Medical Image Analysis. pp. 103644 (2025)

    2025

  38. [40]

    & Padoy, N

    Yuan, K., Srivastav, V ., Navab, N. & Padoy, N. HecVL: hierarchical video-language pretraining for zero-shot surgical phase recognition.International Conference On Medical Image Computing And Computer-Assisted Intervention. pp. 306-316 (2024)

    2024

  39. [41]

    & Padoy, N

    Yuan, K., Srivastav, V ., Navab, N. & Padoy, N. Procedure-aware surgical video-language pretraining with hierarchi- cal knowledge augmentation.Advances In Neural Information Processing Systems.37pp. 122952-122983 (2024)

    2024

  40. [42]

    The robot will see you now: Foundation models are the path forward for autonomous robotic surgery

    Yip, M. The robot will see you now: Foundation models are the path forward for autonomous robotic surgery. Science Robotics.10, eadt0684 (2025)

    2025

  41. [44]

    & Padoy, N

    Nwoye, C., Yu, T., Gonzalez, C., Seeliger, B., Mascagni, P., Mutter, D., Marescaux, J. & Padoy, N. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos.Medical Image Analysis. 78pp. 102433 (2022)

    2022

  42. [45]

    & Wang, Z

    Gui, S. & Wang, Z. Tail-enhanced representation learning for surgical triplet recognition.International Conference On Medical Image Computing And Computer-Assisted Intervention. pp. 689-699 (2024)

    2024

  43. [46]

    Liu, D., Hu, A., Shah, M. & Xu, C. Surgical triplet recognition via diffusion model.ArXiv Preprint ArXiv:2406.13210. (2024)

    Pith/arXiv arXiv 2024

  44. [47]

    What do latent action models actually learn?

    C. Zhang, T. Pearce, P. Zhang, K. Wang, X. Chen, W. Shen, L. Zhao, and J. Bian, “What do latent action models actually learn?”Advances in Neural Information Processing Systems, vol. 38, pp. 146676-146697, 2026

    2026

  45. [48]

    & Padoy, N

    Nwoye, C. & Padoy, N. Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets. ArXiv Preprint ArXiv:2204.05235. (2022)

    Pith/arXiv arXiv 2022

  46. [49]

    & Lin, S

    Cheng, J., Yu, X., Tripathi, S., Liu, S. & Lin, S. TrajPred: Trajectory-Conditioned Joint Embedding Pre- diction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models.ArXiv Preprint ArXiv:2603.06999. (2026)

    arXiv 2026

  47. [50]

    & Xie, S

    Peebles, W. & Xie, S. Scalable diffusion models with transformers.Proceedings Of The IEEE/CVF International Conference On Computer Vision. pp. 4195-4205 (2023)

    2023

  48. [51]

    & Heng, P

    Pei, J., Zhang, J., Qin, G., Wang, K., Jin, Y . & Heng, P. Instrument-Tissue-Guided Surgical Action Triplet Detection via Textual-Temporal Trail Exploration.IEEE Transactions On Medical Imaging.44, 5278-5289 (2025)

    2025

  49. [52]

    & Others CholecTriplet2021: A Benchmark Challenge for Surgical Action Triplet Recognition

    Nwoye, C. & Others CholecTriplet2021: A Benchmark Challenge for Surgical Action Triplet Recognition. Medical Image Analysis.86pp. 102803 (2023)

    2023

  50. [53]

    & Padoy, N

    Sharma, S., Nwoye, C., Mutter, D. & Padoy, N. Surgical action triplet detection by mixed supervised learning of instrument-tissue interactions.International Conference On Medical Image Computing And Computer-Assisted Intervention. pp. 505-514 (2023)

    2023

  51. [54]

    & Others CholecTriplet2022: Show Me a Tool and Tell Me the Triplet — An Endoscopic Vision Challenge for Surgical Action Triplet Detection.Medical Image Analysis

    Nwoye, C. & Others CholecTriplet2022: Show Me a Tool and Tell Me the Triplet — An Endoscopic Vision Challenge for Surgical Action Triplet Detection.Medical Image Analysis. (2023)

    2023

  52. [55]

    Delta-JEPA: Learn- ing Action-Sensitive World Models via Latent Difference Decoding,

    Z. Zhang, Y . Wang, Z. Guan, Y . Yang, B. Shi, T. Zong, H. Yi, G. Chao, X. Chen, T. Yang,et al., “Delta-JEPA: Learn- ing Action-Sensitive World Models via Latent Difference Decoding,”arXiv preprint arXiv:2606.31232, 2026

    Pith/arXiv arXiv 2026

  53. [56]

    & Padoy, N

    Sharma, S., Nwoye, C., Mutter, D. & Padoy, N. Rendezvous in Time: An Attention-Based Temporal Fusion Approach for Surgical Triplet Recognition.International Journal Of Computer Assisted Radiology And Surgery. (2023)

    2023

  54. [57]

    & Jia, F

    Li, Y ., Xia, T., Luo, H., He, B. & Jia, F. MT-FiST: A Multi-Task Fine-Grained Spatial-Temporal Framework for Surgical Action Triplet Recognition.IEEE Journal Of Biomedical And Health Informatics.27, 4983-4994 (2023) 15 LA VIFiT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction RecognitionA PREPRINT

    2023

  55. [58]

    & Cao, Y

    Gui, S., Wang, Z., Chen, J., Zhou, X., Zhang, C. & Cao, Y . MT4MTL-KD: A Multi-Teacher Knowledge Distillation Framework for Triplet Recognition.IEEE Transactions On Medical Imaging.43, 1628-1639 (2024)

    2024

  56. [59]

    & Others Capturing Action Triplet Correlations for Accurate Surgical Activity Recognition.Computerized Medical Imaging And Graphics

    Li, Y . & Others Capturing Action Triplet Correlations for Accurate Surgical Activity Recognition.Computerized Medical Imaging And Graphics. (2025)

    2025

  57. [60]

    & Jia, F

    Li, Y ., Bai, B. & Jia, F. Parameter-Efficient Framework for Surgical Action Triplet Recognition.International Journal Of Computer Assisted Radiology And Surgery.19, 1291-1299 (2024)

    2024

  58. [61]

    & Others ITG-Trip: Integrating Temporal and Visual-Linguistic Cues for Surgical Action Triplet Recognition.ArXiv Preprint

    Pei, J. & Others ITG-Trip: Integrating Temporal and Visual-Linguistic Cues for Surgical Action Triplet Recognition.ArXiv Preprint. (2025), verify authors/venue before camera-ready

    2025

  59. [62]

    & Xiong, J

    Li, J., Quaranto, B., Xu, C., Mishra, I., Qin, R., Liu, D., Kim, P. & Xiong, J. Recognize any surgical object: unleashing the power of weakly-supervised data.International Conference On Learning Representations.2025 pp. 33811-33823 (2025)

    2025

  60. [63]

    Others SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence.ArXiv Preprint ArXiv:2506.02555. (2025)

    Pith/arXiv arXiv 2025

  61. [64]

    & Padoy, N

    Yuan, K., Kattel, M., Lavanchy, J., Navab, N., Srivastav, V . & Padoy, N. Advancing Surgical VQA with Scene Graph Knowledge.International Journal Of Computer Assisted Radiology And Surgery (IPCAI). (2024)

    2024

  62. [65]

    & Others Egocentric Video-Language Pretraining.Advances In Neural Information Processing Systems (NeurIPS)

    Lin, K., Wang, A., Soldan, M. & Others Egocentric Video-Language Pretraining.Advances In Neural Information Processing Systems (NeurIPS). (2022)

    2022

  63. [66]

    & Others EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone.International Conference On Computer Vision (ICCV)

    Pramanick, S., Song, Y ., Nag, S. & Others EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone.International Conference On Computer Vision (ICCV). (2023)

    2023

  64. [67]

    & Others InternVideo2: Scaling Foundation Models for Multimodal Video Understanding.ArXiv Preprint

    Wang, Y . & Others InternVideo2: Scaling Foundation Models for Multimodal Video Understanding.ArXiv Preprint. (2024)

    2024

  65. [68]

    & Others Ego4D: Around the World in 3,000 Hours of Egocentric Video.IEEE/CVF Conference On Computer Vision And Pattern Recognition (CVPR)

    Grauman, K. & Others Ego4D: Around the World in 3,000 Hours of Egocentric Video.IEEE/CVF Conference On Computer Vision And Pattern Recognition (CVPR). (2022)

    2022

  66. [69]

    Others EgoHOIBench: Benchmarking Egocentric Hand-Object Interaction Understanding.ArXiv Preprint ArXiv:2405.17719. (2024)

    Pith/arXiv arXiv 2024

  67. [70]

    & Isbell, C

    Edwards, A., Sahni, H., Schroecker, Y . & Isbell, C. Imitating Latent Policies from Observation.International Conference On Machine Learning (ICML). (2019)

    2019

  68. [71]

    & Jiang, M

    Schmidt, D. & Jiang, M. Learning to Act without Actions.International Conference On Learning Representations (ICLR). (2024)

    2024

  69. [72]

    & Others Genie: Generative Interactive Environments.International Conference On Machine Learning (ICML)

    Bruce, J., Dennis, M., Edwards, A. & Others Genie: Generative Interactive Environments.International Conference On Machine Learning (ICML). (2024)

    2024

  70. [73]

    & Others Latent Action Learning Requires Supervision in the Presence of Distractors.ArXiv Preprint ArXiv:2502.00379

    Nikulin, A. & Others Latent Action Learning Requires Supervision in the Presence of Distractors.ArXiv Preprint ArXiv:2502.00379. (2025)

    Pith/arXiv arXiv 2025

  71. [74]

    Liang, A., Czempin, P., Hong, M., Zhou, Y ., Biyik, E. & Tu, S. Clam: Continuous latent action models for robot learning from unlabeled demonstrations.ArXiv Preprint ArXiv:2505.04999. (2025)

    Pith/arXiv arXiv 2025

  72. [75]

    & Others Latent Action Pretraining from Videos.International Conference On Learning Representations (ICLR)

    Ye, S. & Others Latent Action Pretraining from Videos.International Conference On Learning Representations (ICLR). (2024)

    2024

  73. [76]

    A Path Towards Autonomous Machine Intelligence

    LeCun, Y . A Path Towards Autonomous Machine Intelligence. (2022), OpenReview

    2022

  74. [77]

    & Others Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

    Assran, M. & Others Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. IEEE/CVF Conference On Computer Vision And Pattern Recognition (CVPR). (2023)

    2023

  75. [78]

    & Others V-JEPA: Latent Video Prediction for Visual Representation Learning

    Bardes, A., Garrido, Q., Ponce, J. & Others V-JEPA: Latent Video Prediction for Visual Representation Learning. International Conference On Learning Representations (ICLR). (2024)

    2024

  76. [79]

    & Others V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

    Assran, M. & Others V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. ArXiv Preprint ArXiv:2506.09985. (2025)

    Pith/arXiv arXiv 2025

  77. [80]

    & Balestriero, R

    Maes, L., Le Lidec, Q., Scieur, D., LeCun, Y . & Balestriero, R. LeWorldModel: Stable End-to-End Joint- Embedding Predictive Architecture from Pixels.ArXiv Preprint ArXiv:2603.19312. (2026)

    Pith/arXiv arXiv 2026

  78. [81]

    & Others Deep Delta Learning.ArXiv Preprint ArXiv:2601.00417

    Zhang, Y . & Others Deep Delta Learning.ArXiv Preprint ArXiv:2601.00417. (2026)

    Pith/arXiv arXiv 2026

  79. [82]

    Contrastive localized language-image pre-training,

    H.-Y . Chen, Z. Lai, H. Zhang, X. Wang, M. Eichner, K. You, M. Cao, B. Zhang, Y . Yang, and Z. Gan, “Contrastive localized language-image pre-training,”arXiv preprint arXiv:2410.02746, 2024

    Pith/arXiv arXiv 2024

  80. [83]

    Multi-grained vision-language pre-training: Aligning texts with visual concepts,

    Y . Zeng, X. Zhang, and H. Li, “Multi-grained vision-language pre-training: Aligning texts with visual concepts,” arXiv preprint arXiv:2111.08276, 2021. 16 LA VIFiT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction RecognitionA PREPRINT A Appendix A.1 Implementation details. All three encoders share the same latent-action components and di...

    Pith/arXiv arXiv 2021