Pith. sign in

REVIEW 4 major objections 4 minor 136 references

SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Surgical action triplet recognition transfers across hospitals when the model reasons explicitly about instrument–target and instrument–verb relations instead of treating triplets as flat labels.

desk verdict Solid four-center surgical triplet benchmark plus a relational model that wins on cross-center transfer, but the paper's central causal claim is undercut by missing ablation on the transfer split and an AP aggregation inconsistency. read the letter →

arxiv 2608.02188 v1 pith:WVGQSTES submitted 2026-08-03 cs.CV

Saurav Sharma , Lorenzo Arboit , Nabani Banik , Sarah Meuli , Julia Alekseenko , Jan Liechti , Franziska Heitzinger , Michela Orsi
show 6 more authors
Didier Mutter Daniel Gero Philipp C. Nett Beat P. Muller Joel L. Lavanchy Nicolas Padoy
This is my paper · ORCID
classification cs.CV
keywords surgicalactiontripletrecognitioninstrument-tissueinteractioncross-centergeneralizationrelationalmodelingmulti-centricdatasetRoux-en-Ygastricbypassspatio-temporalvideounderstandingknowledgedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that fine-grained surgical action understanding — recognizing which instrument acts on which anatomical target with which verb — fails to transfer across hospitals when models treat the resulting triplets as flat labels, because what shifts between centers is not raw appearance but the structure of instrument–tissue interactions. To make this concrete, the authors introduce a four-center Roux-en-Y gastric bypass dataset with 420,000+ frame-level triplet labels plus a hidden-test protocol that separates a familiar center from two fully unseen centers. Their SPIRIT framework first learns spatio-temporal representations of instruments, verbs, and targets, then explicitly models the two pairwise relation spaces (instrument–target interaction and instrument–verb affordance), and finally composes valid triplets through a restrictable graph. On the unseen-center test, SPIRIT reaches 42.3 triplet AP versus 33.2 for the strongest non-relational baseline built on the same vision-language backbone, with the largest gains at the pairwise level. The paper's central claim is that hidden-test transfer in triplet recognition is fundamentally a relational generalization problem rather than a feature-quality problem.

What carries the argument

The load-bearing object is the decomposition of a flat triplet label into a three-stage pipeline. A text-conditioned unary feature module (TUF) turns class-name prompts into queries that attend to an 8-frame spatio-temporal memory, producing instrument, verb, and target node features. A pairwise interaction coupling module (PIC) builds two dense bipartite graphs — instrument–target (interaction semantics) and instrument–verb (affordance) — whose edge classifiers produce relation heatmaps and whose message passing refines node features. A triplet graph reasoning module (TGR) then reinterprets the learned pairwise edge embeddings as nodes of a higher-order graph, connecting only structurally v

What would settle it

Re-annotate a random sample of frames from each of the four centers with a second blinded annotator and compute per-center inter-annotator agreement (e.g., Cohen's kappa) on triplet labels; alternatively, inject label noise into the training frames at the noise level measured at the unseen centers and check whether SPIRIT's hidden-test advantage over the plain backbone persists. If agreement is markedly lower at the two fully unseen centers, or if the advantage collapses under matched label noise, the relational-generalization claim would be undermined.

Watch

Extended reading notes

Core claim

The core discovery is that explicitly modeling intermediate pairwise interaction structure is what makes triplet representations survive center shift. Element-wise decomposition — instrument, verb, target — plus the two pairwise relation spaces (where the instrument acts, and what the instrument can plausibly do) are the units that generalize, not the 85-class triplet surface. On the challenge split's hidden test, where two centers are entirely absent from training, the relational framework (SPIRIT) improves triplet AP from 33.2 to 42.3 over the same-backbone temporal baseline; per-center analysis shows the margin is largest precisely on the unseen centers and specifically in the pairwise an

Load-bearing premise

The result stands on the reliability and cross-center consistency of the new apex-based triplet annotations: if label noise or ambiguity differs systematically among the four centers, the measured center shift and SPIRIT's transfer advantage could reflect annotation artifacts rather than genuine relational generalization.

Editorial extensions

If this is right

  • If the relational hypothesis is correct, any triplet-recognition model that decomposes into unary plus pairwise structure will transfer to unseen hospitals better than a flat classifier with the same backbone; the backbone choice does not close the gap.
  • Center-wise hidden-test evaluation, separating familiar from fully unseen centers, should be part of standard benchmarks, because single-center or all-center splits systematically overstate deployment readiness.
  • The pairwise and triplet gains (rather than unary gains) indicate that the bottleneck is association, not detection; investment should go into interaction modeling and affordability constraints rather than only stronger features.
  • Distilling intermediate relation heads transfers across centers without extra inference cost, since the distilled student matches the teacher's parameter count and latency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the relational invariance is real, a natural next step is domain adaptation on the relation spaces themselves — aligning the instrument–target and instrument–verb heatmap distributions across centers — which may transfer better than aligning raw image features.
  • The apex-based temporal annotation (roughly a 3-second window around the clearest contact moment) means labels are temporally smoothed; a testable extension is whether point-annotated frames or shorter windows change the observed center-shift gap, which would clarify how much of the effect is temporal smoothing versus relational reasoning.
  • The same compositional argument may apply beyond surgery: for any activity-recognition task with a role–action–object structure (e.g., assembly, cooking, instrument use), explicit intermediate relation modeling could be the invariant that survives domain shift in acquisition and style.
  • Per-verb analysis shows at least one verb where appearance cues dominate (the paper notes 'pass' is best handled by the plain backbone), suggesting a hybrid model that dynamically weights unary versus relational evidence per class could push transfer further; this is not tested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SPIRIT, a structured framework for surgical action triplet recognition that represents a triplet as a composition of unary instruments/verbs/targets, pairwise interaction relations (instrument-target and instrument-verb), and a final triplet graph reasoning module, trained with multi-head distillation. To study cross-center generalization, the authors introduce MultiBypass-4C-T40, a new multi-centric RYGB dataset with dense triplet annotations and auxiliary phase/step labels, and define all-centers, cross-validation, and challenge (public/hidden, including unseen centers) evaluation protocols. Across these protocols, SPIRIT variants consistently outperform strong triplet-specific and foundation-model baselines, with the largest gains at pairwise and triplet levels. The paper claims that explicit relational modeling, rather than stronger visual features alone, is what drives reliable hidden-test transfer.

Significance. If the result holds, the paper makes a valuable contribution in two respects: (i) it provides a much-needed multi-centric benchmark for fine-grained surgical triplet recognition, extending prior workflow-level multi-centric analyses; and (ii) it offers a concrete architectural hypothesis — explicit intermediate relational structure improves cross-institution transfer — supported by a large body of experiments on three split protocols, multiple baselines, ablations, per-class analyses, and a paired permutation test. The authors are also to be credited for committing to release code and for transparently reporting computational costs. The key caveat is that the central causal claim about relational modeling and transfer is not yet supported by a transfer-split ablation.

major comments (4)
  1. [§6.4, §6.6, Table 9] The central claim — that explicit pairwise/triplet relational modeling is what makes hidden-test transfer reliable — is not tested on the challenge split. The component ablation (Table 9) is run only on the all-centers split, where removing PIC/TGR lowers AP_IVT from 47.9 to 39.1, below the DinoV3-L baseline (42.5). On the challenge hidden test, SPIRIT-TS gains large margins over DinoV3-L-T (42.3 vs 33.2), but no equivalent ablation is reported for that split. SPIRIT-S/TS differ from DinoV3-L-T in several non-relational ways (text-conditioned unary queries, multi-head losses, and distillation), so the hidden-test gains cannot yet be causally attributed to the relational modules. A challenge-split ablation (e.g., full SPIRIT vs. a relational-less SPIRIT with TUF and all losses, and SPIRIT with/without distillation) is needed to support the paper's main conclusion.
  2. [§6.5, Tables 7 and 8] The reported hidden-test AP_IVT is internally inconsistent with the per-center numbers. Table 7 gives SPIRIT-TS hidden-test AP_IVT as 42.3, while Table 8 gives per-center AP_IVT for C1, C3, C4 as 46.7, 39.0, and 36.4. With 5 videos per center and the evaluation protocol described in §5.1 (per-video AP, then averaged over videos), the hidden-test average should be the equal-weighted mean, 40.7, not 42.3. This discrepancy suggests different aggregation or metric usage between the two tables. Please clarify the exact computation and report a single consistent protocol; as written, the main hidden-test numbers cannot be cross-checked against per-center results.
  3. [§6.14] The statistical significance analysis does not cover the unseen-center transfer setting. The paired permutation test is run on the 9-video public challenge test, whose videos come from the training centers (C1 and C2), while the central claim concerns the hidden test containing unseen centers C3 and C4. Moreover, the hidden test in Table 8 is only 5 videos per center, with no error bars or significance testing for C3/C4. To support the cross-center claim, the authors should provide significance/confidence analysis on the hidden test (e.g., paired bootstrap over hidden-test videos, or per-center paired tests).
  4. [§3.3, §3.4] The new dataset is the foundation of the transfer measurements, but the paper provides no inter-annotator agreement statistics or label-noise analysis. Algorithm 1 defines a 3-second-before/after-apex interval, and §3.4 describes expert mediation, but if annotation noise or interval-boundary conventions vary systematically across centers, the measured 'center shift' and SPIRIT's transfer gains could partly reflect annotation artifacts rather than true generalization. Reporting IAA (e.g., frame-level agreement on overlapping videos) or a label-noise robustness check would materially strengthen the benchmark's validity.
minor comments (4)
  1. [§5.3] Typo: 'dino.txt large vision-language backbone' should presumably be 'DINOv3-L text-conditioned backbone' or similar.
  2. [Appendix, Table A3] Table A3 caption says 'Part 1: IDs 0–53' but the table rows continue to ID 84; the caption is likely meant to say Part 2 (or the table should be split consistently).
  3. [§6.5] The explanation for excluding C2 from per-center analysis is that its videos are already in the public MultiBypass140 release, but the challenge split uses C2 videos for training/validation. Please state explicitly how the visible-center C2 set overlaps with training and why C2 is not included in the hidden test, to avoid reader confusion.
  4. [§4.7 / Eq. (31)] The loss weights λ_i ... λ_ivt are listed in the equation but their exact values are not reported in §5.3 (only distillation weights are). Please report the supervised loss weights for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model's intermediate targets are label-space factorizations, not fitted inputs, and the main transfer claim rests on external and internal experiments rather than on self-referential derivations.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs by construction. The pairwise and unary supervision are obtained by projecting the triplet labels onto their components (Sec. 5.2: 'each triplet annotation is projected onto its associated instrument, verb, and target components'), which is standard label factorization, not a fitted parameter renamed as a prediction. The triplet head is independently supervised (Eqs. 25-26) and is not a deterministic function of the pairwise heads, so the final AP_IVT is not forced by the derived pairwise labels. The dataset ontology is used only as a validity mask and graph-construction constraint (Eq. 23), exactly analogous to a classifier's label set, not as a source of test information. Model selection is based on validation mAP, and no parameters are fitted to the hidden test set. The central transfer claim is supported by experiments on the new benchmark and additionally by external evaluation on CholecT45 (Table A4), where SPIRIT-TS achieves the best AP_IVT, making the results externally falsifiable. The lack of a challenge-split ablation for PIC/TGR is an evidentiary gap for causal attribution, not a circularity. Self-citations (CholecT50, MultiBypass140, Rendezvous-in-Time) are used for protocol and baselines and are not load-bearing uniqueness arguments. No circular step can be exhibited from the paper's own equations.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard supervised learning assumptions and the new dataset. No new physical entities are introduced. The main free parameters are architectural hyperparameters and loss weights; their values are reported for some and tuned on validation for others. The most consequential assumption is that the derived frame-level triplet/pairwise labels are reliable enough to support the multi-center transfer claim.

free parameters (8)
  • Causal clip length T = 8 frames
    Temporal window chosen for short-term context; ablated in §6.8 but not swept.
  • Pooled spatial grid S = 8×8
    Spatial pooling resolution of visual features; set by design.
  • Latent dimension d = 128
    Shared projection dimension for visual and text tokens.
  • Relational output dim d_o = 256
    Node feature dimension after graph reasoning.
  • Loss weights λ_i...λ_ivt = not specified numerically; tuned on validation
    Weighted sum of six BCE losses (§4.7). Exact values not reported.
  • Distillation weights ω_i=0.2, ω_pair=0.4, ω_ivt=1.6 = 0.2/0.4/1.6
    Set to emphasize triplet head during distillation (§5.3).
  • Reliability scaling β = 1.0
    Controls suppression of distillation weight from teacher-ground-truth disagreement (§4.6).
  • Quantile thresholds = 0.1 and 0.9
    Mid-confidence filtering in distillation.
assumptions (4)
  • domain assumption Each triplet label factorizes into exactly one instrument, verb, and target (CholecT50 protocol).
    Used in §5.2 to derive unary and pairwise supervision; if a triplet can have multiple simultaneous verbs, this factorization under-specifies.
  • domain assumption The dataset ontology defines the complete set of valid triplets; only these are used for supervision and graph edges.
    The validity mask in §4.5 excludes invalid compositions, which may make the task easier than unconstrained classification.
  • domain assumption Text prompts 'a photo of {c}' provide useful semantic priors for instrument/verb/target classes.
    Used in §4.2 to initialize class queries; if class names are ambiguous (e.g., 'SBS-A'), the prior may be weak.
  • domain assumption Frame-level labels at 1 FPS derived from annotated intervals are correct.
    Annotations converted to frame labels following CholecT50; interval boundaries from Algorithm 1 may introduce noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition." pith.science (2026). https://pith.science/paper/WVGQSTES

@misc{pith2026260802188,
  author       = {Pith},
  title        = {Pith review of: SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WVGQSTES}},
  note         = {Machine review of arXiv:2608.02188}
}
read the original abstract

Fine-grained understanding of surgical activity is essential for context-aware assistance in the operating room, including safety monitoring, adverse event identification, and skill assessment. Surgical action triplets, defined as tuples of the form <instrument, verb, target>, provide a structured description of instrument-tissue interactions. A key open problem, however, is how to learn triplet representations that remain reliable across institutions, where surgical video varies in acquisition conditions, surgeon style, tool usage, and tissue handling, while existing triplet datasets do not support explicit evaluation of center-wise transfer. To address this problem, we propose \textbf{SPIRIT}, a structured framework for surgical action triplet recognition designed to learn interaction representations that transfer more reliably across centers. Instead of treating each triplet as a flat class label, SPIRIT first learns spatio-temporal representations for instruments, verbs, and targets, then models their pairwise relations, and finally composes them into coherent triplet predictions, with multi-head distillation used to stabilize learning. To evaluate this setting, we establish \textbf{MultiBypass-4C-T40}, a multi-centric dataset for dense surgical action triplet recognition in Roux-en-Y gastric bypass across four geographically distinct centers, with auxiliary phase and step annotations. Across multiple evaluation protocols, SPIRIT consistently outperforms strong recent baselines, highlighting the value of explicit relational reasoning for multi-centric triplet recognition. Code will be available at https://github.com/CAMMA-public/multibypass-4c-t40.

Figures

Figures reproduced from arXiv: 2608.02188 by the authors.

Figure 1
Figure 1. Illustrative examples from the MultiBypass-4C-T40 dataset showing surgical action triplets together with accompanying phase and step-level [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the temporal evolution of hierarchical workflow annotations, from phases and steps to fine-grained surgical action triplets, in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of the MultiBypass-4C-T40 dataset, illustrating the distribution of fine-grained surgical action triplets and their constituent instrument, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Overview of the three evaluation settings used in MultiBypass [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Overview of SPIRIT, which consists of three modules: text-conditioned unary feature learning (TUF), pairwise interaction coupling (PIC), and [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Boxplot of per-class APIVT across models on the all-centers split of MultiBypass-4C-T40 dataset. correct triplet with high reliability, but SPIRIT-TS remains the best in every case. These results reinforce the findings from the AP-based eval￾uation. The advantage of SP…
Figure 7
Figure 7. Figure 7: Per-class performance comparison across models on verb component on the [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Visualization of component-wise spatio-temporal cross-attention maps in the TUF block of SPIRIT on the MultiBypass-4C-T40 dataset. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

136 extracted references · 1 canonical work pages

  1. [1]

    energy_device,dissect,mesentery 35 404 45 123

    grasper,grasp,stomach 17137 10347 4282 4626 46. energy_device,dissect,mesentery 35 404 45 123

  2. [2]

    energy_device,cut,small_bowel 0 360 0 192

    grasper,grasp,thread 24512 9591 7928 2211 47. energy_device,cut,small_bowel 0 360 0 192

  3. [3]

    energy_device,cut,stomach 136 177 0 188

    grasper,grasp,omentum 8913 4763 1835 4772 48. energy_device,cut,stomach 136 177 0 188

  4. [4]

    energy_device,cut,omentum 2467 2120 811 840

    grasper,grasp,liver 61 117 0 0 49. energy_device,cut,omentum 2467 2120 811 840

  5. [5]

    energy_device,cut,mesentery 163 429 112 233

    grasper,grasp,needle 3161 3277 1064 266 50. energy_device,cut,mesentery 163 429 112 233

  6. [6]

    energy_device,cut,adhesion 152 391 0 0

    grasper,grasp,mesentery 4296 812 609 103 51. energy_device,cut,adhesion 152 391 0 0

  7. [7]

    energy_device,dissect,omentum 764 803 305 662

    grasper,grasp,small_bowel 16369 13338 3895 8301 45. energy_device,dissect,omentum 764 803 305 662

  8. [8]

    energy_device,coagulate,omentum 197 73 81 186

    grasper,retract,small_bowel 1576 1372 696 597 53. energy_device,coagulate,omentum 197 73 81 186

Show all 136 references
  1. [9]

    IA,retract,small_bowel 0 42 33 136

    grasper,retract,stomach 1920 2256 798 2105 54. IA,retract,small_bowel 0 42 33 136

    1920

  2. [10]

    IA,retract,stomach 74 27 19 57

    grasper,retract,omentum 1281 1629 680 330 55. IA,retract,stomach 74 27 19 57

  3. [11]

    IA,aspirate,fluid 1894 2593 932 1410

    grasper,retract,liver 1432 1480 18 0 56. IA,aspirate,fluid 1894 2593 932 1410

  4. [12]

    IA,irrigate,fluid 6 27 176 174

    grasper,retract,mesentery 577 252 302 45 57. IA,irrigate,fluid 6 27 176 174

  5. [13]

    needle_driver,grasp,thread 638 733 315 96

    grasper,retract,sponge 121 0 0 0 58. needle_driver,grasp,thread 638 733 315 96

  6. [14]

    energy_device,coagulate,stomach 422 147 35 33

    grasper,grasp,colon 75 55 60 39 52. energy_device,coagulate,stomach 422 147 35 33

  7. [15]

    needle_driver,retract,small_bowel 114 50 31 23

    grasper,dissect,small_bowel 532 35 0 0 60. needle_driver,retract,small_bowel 114 50 31 23

  8. [16]

    needle_driver,suture,small_bowel 268 873 20 37

    grasper,dissect,stomach 2691 566 102 246 61. needle_driver,suture,small_bowel 268 873 20 37

  9. [17]

    needle_driver,suture,mesentery 8897 0 908 0

    grasper,dissect,omentum 562 400 42 830 62. needle_driver,suture,mesentery 8897 0 908 0

  10. [18]

    needle_driver,suture,SBS-A 12828 11348 3821 0

    grasper,dissect,mesentery 413 190 26 167 63. needle_driver,suture,SBS-A 12828 11348 3821 0

  11. [19]

    needle_driver,suture,SB-A 9057 7506 1655 2034

    grasper,suture,small_bowel 86 498 6 528 64. needle_driver,suture,SB-A 9057 7506 1655 2034

  12. [20]

    clipper,clip,small_bowel 0 0 0 110

    grasper,suture,mesentery 5285 0 375 0 65. clipper,clip,small_bowel 0 0 0 110

  13. [21]

    needle_driver,grasp,needle 1025 814 388 63

    grasper,retract,colon 57 27 10 4 59. needle_driver,grasp,needle 1025 814 388 63

  14. [22]

    bipolar_forceps,grasp,sponge 107 0 0 0

    grasper,suture,SB-A 5723 4416 801 1069 67. bipolar_forceps,grasp,sponge 107 0 0 0

  15. [23]

    bipolar_forceps,coagulate,stomach 1192 0 0 0

    stapler,grasp,small_bowel 2130 2441 287 209 68. bipolar_forceps,coagulate,stomach 1192 0 0 0

  16. [24]

    suture_passer,pass,thread 630 0 119 0

    stapler,grasp,stomach 1780 1144 632 727 69. suture_passer,pass,thread 630 0 119 0

  17. [25]

    swab_forceps,retract,small_bowel 0 169 0 0

    stapler,grasp,SBS-A 636 143 0 0 70. swab_forceps,retract,small_bowel 0 169 0 0

  18. [26]

    swab_forceps,retract,liver 0 343 0 0

    stapler,grasp,SB-A 625 319 216 211 71. swab_forceps,retract,liver 0 343 0 0

  19. [27]

    swab_forceps,clean,small_bowel 0 472 23 0

    stapler,retract,stomach 0 97 0 0 72. swab_forceps,clean,small_bowel 0 472 23 0

  20. [28]

    clipper,clip,stomach 45 41 39 312

    grasper,suture,SBS-A 9151 7176 1432 0 66. clipper,clip,stomach 45 41 39 312

  21. [29]

    gauze,clean,stomach 3425 0 604 0

    stapler,staple,stomach 732 958 347 624 74. gauze,clean,stomach 3425 0 604 0

  22. [30]

    grasper,null_verb,null_target 13825 9022 2551 1998

    stapler,staple,SBS-A 197 171 0 0 75. grasper,null_verb,null_target 13825 9022 2551 1998

    1998

  23. [31]

    stapler,null_verb,null_target 909 930 267 103

    stapler,staple,SB-A 245 270 77 99 76. stapler,null_verb,null_target 909 930 267 103

  24. [32]

    electric_hook,null_verb,null_target 466 140 102 0

    electric_hook,dissect,small_bowel 788 0 133 0 77. electric_hook,null_verb,null_target 466 140 102 0

  25. [33]

    RAD,null_verb,null_target 0 126 39 3

    electric_hook,dissect,stomach 1080 0 137 0 78. RAD,null_verb,null_target 0 126 39 3

  26. [34]

    energy_device,null_verb,null_target 586 1222 249 137

    electric_hook,dissect,omentum 1283 0 0 0 79. energy_device,null_verb,null_target 586 1222 249 137

  27. [35]

    swab_forceps,clean,stomach 0 299 111 71

    stapler,staple,small_bowel 167 211 42 102 73. swab_forceps,clean,stomach 0 299 111 71

  28. [36]

    needle_driver,null_verb,null_target 421 95 159 2

    RAD,grasp,stomach 0 141 0 122 81. needle_driver,null_verb,null_target 421 95 159 2

  29. [37]

    clipper,null_verb,null_target 29 66 21 19

    RAD,dissect,small_bowel 21 398 18 103 82. clipper,null_verb,null_target 29 66 21 19

  30. [38]

    bipolar_forceps,null_verb,null_target 339 0 0 0

    RAD,dissect,stomach 6 193 21 51 83. bipolar_forceps,null_verb,null_target 339 0 0 0

  31. [39]

    swab_forceps,null_verb,null_target 0 305 0 0

    energy_device,grasp,omentum 64 51 17 0 84. swab_forceps,null_verb,null_target 0 305 0 0

  32. [40]

    energy_device,retract,small_bowel 7 92 6 0

  33. [41]

    energy_device,retract,stomach 107 367 57 51

  34. [42]

    IA,null_verb,null_target 44 245 68 75

    electric_hook,coagulate,spleen 0 272 0 0 80. IA,null_verb,null_target 44 245 68 75

  35. [43]

    energy_device,dissect,small_bowel 0 415 51 199

  36. [44]

    Class-wise triplet mAP across 12 models on the hidden test set of thechallengesplit (Part 1: IDs 0–53)

    energy_device,dissect,stomach 810 1191 351 521 Saurav Sharmaet al.30 Table A2. Class-wise triplet mAP across 12 models on the hidden test set of thechallengesplit (Part 1: IDs 0–53). Green, violet, and orange cells indicate the best, second-best, and third-best values in each ...

  37. [45]

    energy_device,dissect,omentum 11070.034 0.109 0.235 0.241 0.261 0.221 0.244 0.240 0.182 0.192 0.310 0.350

  38. [46]

    energy_device,dissect,mesentery 1680.024 0.049 0.228 0.336 0.324 0.221 0.278 0.353 0.166 0.250 0.576 0.479

  39. [47]

    energy_device,cut,small_bowel 1920.024 0.053 0.328 0.566 0.572 0.141 0.484 0.274 0.189 0.351 0.525 0.785

    1920

  40. [48]

    energy_device,cut,stomach 1880.045 0.047 0.306 0.600 0.6810.333 0.339 0.276 0.121 0.256 0.427 0.584

  41. [49]

    energy_device,retract,omentum 295 216 14 62

  42. [50]

    energy_device,cut,mesentery 4390.067 0.072 0.412 0.728 0.548 0.615 0.582 0.495 0.362 0.563 0.607 0.735

  43. [51]

    energy_device,cut,adhesion 920.047 0.039 0.108 0.421 0.365 0.377 0.123 0.594 0.533 0.465 0.181 0.357

  44. [52]

    grasper,grasp,small_bowel 250640.383 0.452 0.809 0.844 0.825 0.821 0.824 0.837 0.805 0.831 0.862 0.890

  45. [53]

    grasper,grasp,stomach 163680.433 0.539 0.788 0.839 0.807 0.844 0.815 0.842 0.811 0.839 0.857 0.863

  46. [54]

    grasper,grasp,thread 195040.360 0.446 0.669 0.701 0.699 0.749 0.749 0.781 0.705 0.756 0.786 0.842

  47. [55]

    grasper,grasp,omentum 98750.320 0.392 0.631 0.650 0.636 0.637 0.603 0.646 0.623 0.647 0.688 0.692

  48. [56]

    grasper,grasp,liver 610.010 0.027 0.015 0.044 0.032 0.015 0.035 0.010 0.013 0.013 0.155 0.085

  49. [57]

    grasper,grasp,needle 21440.075 0.091 0.300 0.364 0.165 0.422 0.312 0.362 0.375 0.383 0.426 0.482

  50. [58]

    grasper,grasp,mesentery 26890.129 0.158 0.187 0.232 0.217 0.250 0.226 0.276 0.217 0.263 0.263 0.289

  51. [59]

    grasper,grasp,colon 1220.010 0.011 0.050 0.055 0.034 0.008 0.016 0.039 0.030 0.051 0.1160.011

  52. [60]

    grasper,retract,small_bowel 20260.051 0.053 0.100 0.107 0.085 0.118 0.119 0.140 0.089 0.123 0.159 0.150

  53. [61]

    grasper,retract,stomach 33610.130 0.117 0.341 0.4300.341 0.357 0.336 0.421 0.255 0.398 0.314 0.411

  54. [62]

    grasper,retract,omentum 13360.045 0.069 0.108 0.069 0.082 0.126 0.117 0.083 0.094 0.086 0.069 0.097

  55. [63]

    grasper,retract,liver 1910.032 0.022 0.198 0.221 0.265 0.225 0.184 0.274 0.169 0.246 0.347 0.314

    1910

  56. [64]

    grasper,retract,mesentery 4960.089 0.069 0.110 0.105 0.105 0.117 0.067 0.088 0.1330.092 0.099 0.099

  57. [65]

    grasper,retract,sponge 0 0 0 0 0 0 0 0 0 0 0 0 0

  58. [66]

    grasper,retract,colon 140.044 0.010 0.006 0.029 0.042 0.045 0.0960.043 0.041 0.034 0.003 0.052

  59. [67]

    grasper,dissect,small_bowel 1000.028 0.074 0.434 0.388 0.262 0.411 0.302 0.464 0.399 0.476 0.388 0.557

  60. [68]

    grasper,dissect,stomach 8290.051 0.065 0.365 0.362 0.369 0.381 0.4010.359 0.320 0.370 0.383 0.387

  61. [69]

    grasper,dissect,omentum 15370.069 0.056 0.167 0.283 0.227 0.232 0.220 0.218 0.228 0.260 0.357 0.398

  62. [70]

    grasper,dissect,mesentery 2830.084 0.040 0.296 0.296 0.349 0.175 0.448 0.304 0.311 0.348 0.559 0.592

  63. [71]

    grasper,suture,small_bowel 6200.040 0.050 0.056 0.194 0.066 0.044 0.072 0.040 0.031 0.028 0.392 0.312

  64. [72]

    grasper,suture,mesentery 17350.222 0.486 0.668 0.638 0.697 0.7940.662 0.652 0.782 0.636 0.757 0.740

  65. [73]

    grasper,suture,SBS-A 30730.450 0.439 0.609 0.657 0.661 0.700 0.699 0.619 0.682 0.583 0.762 0.791

  66. [74]

    grasper,suture,SB-A 38390.310 0.380 0.711 0.771 0.764 0.726 0.734 0.780 0.670 0.782 0.841 0.858

  67. [75]

    stapler,grasp,small_bowel 11150.051 0.073 0.381 0.467 0.476 0.394 0.5060.474 0.435 0.475 0.408 0.486

  68. [76]

    stapler,grasp,stomach 19910.104 0.176 0.591 0.698 0.660 0.649 0.675 0.656 0.666 0.680 0.678 0.703

  69. [77]

    stapler,grasp,SBS-A 2290.057 0.170 0.752 0.787 0.751 0.717 0.718 0.830 0.703 0.838 0.804 0.803

  70. [78]

    stapler,grasp,SB-A 6840.026 0.036 0.284 0.475 0.377 0.406 0.373 0.389 0.398 0.400 0.4950.392

  71. [79]

    stapler,retract,stomach 0 0 0 0 0 0 0 0 0 0 0 0 0

  72. [80]

    stapler,staple,small_bowel 2100.028 0.028 0.326 0.338 0.413 0.265 0.342 0.311 0.152 0.309 0.403 0.481

  73. [81]

    stapler,staple,stomach 10810.049 0.118 0.332 0.431 0.530 0.464 0.532 0.446 0.344 0.422 0.524 0.539

  74. [82]

    stapler,staple,SBS-A 380.007 0.013 0.392 0.263 0.566 0.412 0.331 0.574 0.442 0.523 0.6690.504

  75. [83]

    stapler,staple,SB-A 2390.017 0.023 0.357 0.358 0.404 0.325 0.5620.376 0.236 0.403 0.429 0.562

  76. [84]

    electric_hook,dissect,small_bowel 3830.050 0.127 0.517 0.7140.547 0.632 0.535 0.602 0.508 0.501 0.553 0.580

  77. [85]

    electric_hook,dissect,stomach 2980.072 0.123 0.433 0.492 0.372 0.480 0.350 0.5080.405 0.507 0.384 0.352

  78. [86]

    electric_hook,dissect,omentum 5730.169 0.242 0.748 0.802 0.809 0.743 0.812 0.800 0.779 0.798 0.789 0.834

  79. [87]

    electric_hook,coagulate,spleen 0 0 0 0 0 0 0 0 0 0 0 0 0

  80. [88]

    RAD,grasp,stomach 1220.024 0.039 0.082 0.301 0.589 0.076 0.343 0.036 0.021 0.199 0.544 0.716

  81. [89]

    RAD,dissect,small_bowel 1210.007 0.026 0.191 0.329 0.113 0.235 0.098 0.176 0.097 0.110 0.572 0.549

  82. [90]

    RAD,dissect,stomach 720.061 0.023 0.054 0.349 0.183 0.248 0.354 0.065 0.030 0.038 0.388 0.581

  83. [91]

    energy_device,grasp,omentum 300.100 0.028 0.028 0.249 0.140 0.097 0.134 0.2640.143 0.204 0.195 0.241

  84. [92]

    energy_device,retract,small_bowel 130.006 0.014 0.037 0.029 0.069 0.068 0.013 0.044 0.018 0.039 0.061 0.091

  85. [93]

    energy_device,retract,stomach 1130.008 0.012 0.010 0.014 0.011 0.010 0.010 0.017 0.018 0.019 0.030 0.032

  86. [94]

    energy_device,retract,omentum 970.006 0.009 0.030 0.056 0.103 0.057 0.1620.031 0.017 0.047 0.052 0.135

  87. [95]

    energy_device,dissect,small_bowel 2500.012 0.020 0.216 0.340 0.322 0.280 0.293 0.138 0.079 0.171 0.227 0.272

  88. [96]

    energy_device,dissect,stomach 9190.070 0.094 0.328 0.333 0.360 0.345 0.4070.346 0.313 0.312 0.342 0.393

  89. [101]

    energy_device,cut,omentum 20450.229 0.264 0.502 0.572 0.580 0.510 0.504 0.477 0.451 0.496 0.556 0.614

  90. [104]

    energy_device,coagulate,stomach 910.007 0.015 0.102 0.114 0.158 0.071 0.1660.074 0.045 0.049 0.129 0.064

  91. [105]

    Class-wise triplet mAP across 12 models on the hidden test set of thechallengesplit (Part 1: IDs 0–53)

    energy_device,coagulate,omentum 2670.017 0.023 0.184 0.146 0.2170.119 0.157 0.189 0.190 0.148 0.074 0.145 Saurav Sharmaet al.31 Table A3. Class-wise triplet mAP across 12 models on the hidden test set of thechallengesplit (Part 1: IDs 0–53). Green, violet, and orange cells ind...

  92. [106]

    IA,retract,small_bowel 1690.013 0.023 0.041 0.051 0.082 0.029 0.1570.068 0.030 0.095 0.046 0.058

  93. [107]

    IA,retract,stomach 760.010 0.009 0.031 0.039 0.118 0.023 0.030 0.016 0.014 0.014 0.159 0.127

  94. [108]

    IA,aspirate,fluid 29440.089 0.145 0.432 0.520 0.467 0.392 0.523 0.409 0.355 0.414 0.558 0.660

  95. [109]

    IA,irrigate,fluid 3560.020 0.014 0.054 0.085 0.1240.024 0.050 0.039 0.042 0.057 0.115 0.104

  96. [110]

    needle_driver,grasp,thread 5960.023 0.025 0.037 0.066 0.058 0.068 0.103 0.081 0.042 0.077 0.136 0.142

  97. [111]

    needle_driver,grasp,needle 6640.023 0.023 0.050 0.064 0.045 0.080 0.097 0.164 0.110 0.147 0.116 0.190

  98. [112]

    needle_driver,retract,small_bowel 1160.005 0.004 0.010 0.018 0.015 0.010 0.017 0.028 0.006 0.011 0.0290.014

  99. [113]

    needle_driver,suture,small_bowel 2360.039 0.022 0.050 0.073 0.106 0.153 0.074 0.038 0.073 0.042 0.220 0.154

  100. [114]

    needle_driver,suture,mesentery 32020.300 0.452 0.786 0.782 0.877 0.842 0.844 0.811 0.813 0.787 0.890 0.921

  101. [115]

    needle_driver,suture,SBS-A 66200.416 0.617 0.786 0.824 0.8670.833 0.849 0.810 0.779 0.797 0.852 0.860

  102. [116]

    needle_driver,suture,SB-A 67700.262 0.316 0.795 0.858 0.861 0.855 0.880 0.843 0.801 0.823 0.871 0.912

  103. [117]

    clipper,clip,small_bowel 1100.006 0.025 0.015 0.037 0.013 0.005 0.030 0.025 0.009 0.004 0.210 0.350

  104. [118]

    clipper,clip,stomach 3510.060 0.039 0.019 0.048 0.047 0.048 0.051 0.053 0.118 0.164 0.211 0.234

  105. [119]

    bipolar_forceps,grasp,sponge 110.003 0.002 0.003 0.003 0.003 0.003 0.014 0.003 0.002 0.002 0.156 0.033

  106. [120]

    bipolar_forceps,coagulate,stomach 2790.071 0.449 0.569 0.683 0.677 0.729 0.636 0.701 0.689 0.708 0.719 0.781

  107. [121]

    suture_passer,pass,thread 3100.141 0.309 0.607 0.736 0.591 0.645 0.620 0.678 0.604 0.551 0.646 0.739

  108. [122]

    swab_forceps,retract,small_bowel 0 0 0 0 0 0 0 0 0 0 0 0 0

  109. [123]

    swab_forceps,retract,liver 0 0 0 0 0 0 0 0 0 0 0 0 0

  110. [124]

    swab_forceps,clean,small_bowel 230.006 0.040 0.555 0.831 0.505 0.815 0.333 0.324 0.435 0.478 0.868 0.911

  111. [125]

    swab_forceps,clean,stomach 1820.016 0.012 0.315 0.546 0.550 0.544 0.642 0.340 0.301 0.471 0.558 0.715

  112. [126]

    gauze,clean,stomach 27320.273 0.306 0.304 0.333 0.3330.303 0.306 0.260 0.290 0.269 0.282 0.269

  113. [127]

    grasper,null_verb,null_target 83990.132 0.130 0.269 0.296 0.284 0.315 0.326 0.368 0.325 0.361 0.366 0.445

  114. [128]

    stapler,null_verb,null_target 6410.052 0.071 0.319 0.275 0.179 0.432 0.311 0.396 0.352 0.418 0.493 0.526

  115. [129]

    electric_hook,null_verb,null_target 1690.009 0.015 0.071 0.071 0.054 0.119 0.099 0.123 0.060 0.091 0.148 0.134

  116. [130]

    RAD,null_verb,null_target 420.003 0.003 0.015 0.040 0.008 0.148 0.2900.006 0.009 0.020 0.018 0.038

  117. [131]

    energy_device,null_verb,null_target 4770.027 0.031 0.172 0.282 0.140 0.221 0.237 0.254 0.189 0.257 0.367 0.367

  118. [132]

    IA,null_verb,null_target 1590.004 0.004 0.025 0.079 0.061 0.022 0.075 0.037 0.034 0.029 0.162 0.139

  119. [133]

    needle_driver,null_verb,null_target 3920.014 0.009 0.035 0.049 0.029 0.027 0.073 0.031 0.036 0.042 0.058 0.089

  120. [134]

    clipper,null_verb,null_target 580.003 0.002 0.003 0.008 0.003 0.003 0.005 0.004 0.003 0.037 0.054 0.042

  121. [135]

    bipolar_forceps,null_verb,null_target 1160.008 0.011 0.143 0.099 0.074 0.108 0.097 0.162 0.076 0.176 0.202 0.170

  122. [136]

    Cross-validation results for surgical action triplet recognition on CholecT45

    swab_forceps,null_verb,null_target 0 0 0 0 0 0 0 0 0 0 0 0 0 Table A4. Cross-validation results for surgical action triplet recognition on CholecT45. Values are reported as mean±standard deviation AP (%). SPIRIT-TS achieves the highest triplet-level performance (AP IVT ), indi...

    2022

  123. [1089]

    Funke, I., Jenke, A., Mees, S.T., Weitz, J., Speidel, S., Bodenstedt, S.,

    URL:https://doi.org/10.1007/s11548-016-1371-x, doi:10.1007/s11548-016-1371-x. Funke, I., Jenke, A., Mees, S.T., Weitz, J., Speidel, S., Bodenstedt, S.,

    doi

  124. [2012]

    Medical image analysis 16, 632–641

    Statistical modeling and recognition of surgical workflow. Medical image analysis 16, 632–641. Ramesh, S., Dall’Alba, D., Gonzalez, C., Yu, T., Mascagni, P., Mutter, D., Marescaux, J., Fiorini, P., Padoy, N., 2021. Multi-task temporal convolu- tional networks for joint recogni...

    2021

  125. [2016]

    Automatic data-driven real-time segmentation and recognition of surgical workflow. Int. J. Comput. Assist. Radiol. Surg. 11, 1081–

  126. [2018]

    Temporal coherence-based self-supervised learning for laparoscopic workflow analysis, in: OR 2.0 Context-Aware Operating Theaters, Com- puter Assisted Robotic Endoscopy, Clinical Image-Based Procedures, - and - Skin Image Analysis - First International Workshop, OR 2.0 2018, 5...

    2018

  127. [2022]

    Instrument-tissue interaction quintuple detection in surgery videos, in: International Conference on Medical Image Computing and Computer- Assisted Intervention, Springer. pp. 399–409. Maier-Hein, L., Vedula, S., Speidel, S., Navab, N., Kikinis, R., Park, A., Eisenmann, M., Fe...

    2017 arXiv

  128. [2024]

    International Journal of Computer Assisted Radiology and Surgery 19, 1243–1250

    Optimizing latent graph representations of surgical scenes for un- seen domain generalization. International Journal of Computer Assisted Radiology and Surgery 19, 1243–1250. Sharma, S., Nwoye, C.I., Mutter, D., Padoy, N., 2023a. Rendezvous in time: an attention-based temporal...

    2023 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.