Pith. sign in

REVIEW 3 major objections 5 minor 85 references

HTT-Net shows that structured, hand-written phase and transition descriptions, injected during segment construction and calibration, improve surgical video phase recognition—reaching 96.4% relaxed accuracy on Cholec80 and a 5-point gain on

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:55 UTC pith:TFOOIA3O

load-bearing objection Novel text-guided transition architecture with credible Cholec80 results, but the LCRS-100 SOTA claim is undercut by their own baseline beating every reproduced competitor; needs code and fair baselines. the 3 major comments →

arxiv 2607.16787 v1 pith:TFOOIA3O submitted 2026-07-18 cs.CV

HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition

Kunjie Deng , Jinghui Zhang , Weidong Chen , Ganbin Li , Xiangjun Lyu , Zhendong Mao , Yingchi Yang This is my paper
classification cs.CV
keywords surgical video phase recognitionhierarchical text-guided modelingtransition-aware segment constructionsemantic calibrationvision-language learningsurgical workflow analysisCholec80LCRS-100
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that surgical phase recognition benefits from explicit procedural semantics rather than relying solely on visual-temporal learning. It builds a hierarchical memory of textual descriptions—what each phase looks like, how adjacent phases transition, and fine-grained cues such as instruments and anatomical landmarks—and uses those at two moments: first to construct phase-consistent segments, then to calibrate those segments against semantic references. On the public Cholec80 benchmark, the method sets a state-of-the-art clip-based relaxed accuracy of 96.4% and the best Jaccard among clip-based methods; on the in-house LCRS-100 dataset with 23 phases, it outscores reproduced baselines by roughly 5 accuracy points. A sympathetic reader would care because the design targets the actual failure modes of such systems—ambiguous boundaries and visually similar phases—without requiring dense, frame-by-frame vision-language fusion, keeping inference efficient.

Core claim

The central claim is that hierarchical text-guided transition modeling adds a genuine signal beyond raw visual-temporal models. Concretely, the paper argues that intra-phase descriptions stabilize preliminary frame predictions, inter-phase transition descriptions verify whether a candidate change point is a real procedural boundary rather than transient noise, and fine-grained semantic units help separate phases that share similar tools or anatomical views. Evidence includes consistently higher Jaccard indices in boundary windows, gains on hard phases, and ablations showing each semantic level contributes. On Cholec80, the clip-based model achieves 96.4% relaxed accuracy and 88.3 Jaccard; on

What carries the argument

The central machinery is the hierarchical surgical semantic memory combined with two transition-aware modules. The memory comprises intra-phase descriptions, inter-phase transition descriptions, and fine-grained semantic units, encoded by a frozen biomedical text encoder. TAS-Con (Transition-Aware Segment Construction) fuses intra-phase semantics with preliminary frame evidence to estimate stable phase responses, detects candidate change points from visual contrast and phase consistency, and uses inter-phase transition embeddings to verify plausibility, thereby producing coherent segments with pseudo-labels. TAS-Calib (Transition-Aware Segment Calibration) then cross-attends segment represen

Load-bearing premise

The load-bearing premise is that the LCRS-100 comparison is fair—the dataset is private, the split is chosen by the authors, and every baseline was reproduced by the authors without released training configurations—so if those reproductions were undertrained, the reported 5-point margin would not reflect the true advantage of HTT-Net.

What would settle it

Replace the hand-written phase and transition descriptions with scrambled or randomly permuted text of the same length and re-run the LCRS-100 experiment: if accuracy and Jaccard stay at the same 51.1%/17.0 levels, the semantic content is not driving the gain. Alternatively, have an independent team train Surgformer, B2Q-Net, or DiffAct on the same private split using the authors' released code and configurations; if those baselines achieve accuracy above roughly 47% or Jaccard above 15%, the claimed margin is not credible.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, adding structured procedural text descriptions can push surgical phase recognition past the plateau of purely visual-temporal models, especially near gradual boundaries and between visually confusable phases.
  • Segment-level semantic calibration keeps the computational overhead modest: the paper reports 321 FPS inference with 3.7 TFLOPs, suggesting the technique can be deployed in near-real-time intraoperative guidance.
  • The online frame-based variant (HTT-Net-F), using only past and present information, still outperforms several offline models under relaxed evaluation, so the benefit is not contingent on future-frame context.
  • Ablations indicate that each memory level matters: removing inter-phase transition descriptions or fine-grained units degrades recall and Jaccard, implying the hierarchy, not just any text, is what helps.
  • On the more challenging private dataset with 23 phases and diverse workflows, the model's margin over reproduced baselines suggests the approach scales to procedures beyond cholecystectomy, though that scale-up is the part most dependent on the fairness of the comparison.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gain is real, it suggests a general recipe for fine-grained medical video understanding: encode procedural knowledge as structured text and use it at the segment level, rather than only at the label level—an idea that could extend to action triplet recognition, tool tracking, and other surgical workflow tasks.
  • A testable extension is whether the hand-crafted descriptions can be generated automatically—from large language models or surgical atlases—without losing the benefit; the paper's descriptions are clinician-authored, so sensitivity to wording and coverage remains an open question.
  • The LCRS-100 margins are likely to shrink under independent evaluation, because the dataset is private, the split is author-chosen, and all baselines were reproduced by the authors without released configs; a public benchmark or a released baseline harness would clarify the true margin.
  • The approach could be combined with long-video state-space models to jointly provide procedural semantics and long-range temporal coherence, potentially yielding further gains on very long procedures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HTT-Net, a surgical-video phase-recognition method built on two new modules: Transition-Aware Segment Construction (TAS-Con), which groups frame-level evidence into phase-aware segments using intra-phase and inter-phase transition text descriptions, and Transition-Aware Segment Calibration (TAS-Calib), which refines segment representations by cross-attending to hierarchical semantic memory. A frozen PubMedBERT encoder produces the semantic embeddings, so text encoding is offline. The method is evaluated on Cholec80 and on a private in-house dataset, LCRS-100. On Cholec80 the clip-based variant reports state-of-the-art Jaccard among clip-based methods (88.3 relaxed / 84.0 unrelaxed) with accuracy comparable to B2Q-Net; on LCRS-100 it reports 51.1% accuracy versus 45.9% for the best reproduced baseline. Ablations on LCRS-100 attribute gains to both modules and to the three-level semantic memory.

Significance. The core idea is reasonable and potentially useful: using structured textual phase/transition descriptions to guide segment construction rather than dense frame-level vision-language fusion is a sensible design, and the reported 321 FPS inference with frozen, precomputed text embeddings is a practical strength. The Cholec80 results use the standard split and are internally consistent; the ablations show monotonic contributions from the two proposed modules. However, the headline SOTA claim rests heavily on LCRS-100, a private dataset with author-defined splits and author-reproduced baselines, and the reproduced baselines look implausibly weak relative to the paper's own backbone. The paper ships no code, no released text descriptions, and no error-bar or significance analysis for LCRS-100. If the LCRS-100 comparison is fair, the method is a solid empirical contribution; if not, the claimed advantage over prior art is unsupported. The contribution is, therefore, conditionally significant, pending a reproducible and fair evaluation.

major comments (3)
  1. [§IV-B, Table II vs Table IV] The LCRS-100 comparison is not externally verifiable and is internally suspicious. LCRS-100 is private, the 45/10/45 split is defined solely by the authors, and all baselines in Table II are reproduced by the authors without released configuration files or checkpoints. More concretely, Table IV shows that the authors' own plain visual-temporal backbone (no TAS-Con, no TAS-Calib) reaches 47.3% accuracy on LCRS-100, yet every reproduced baseline in Table II is at or below 45.9% (DiffAct 45.9%, B2Q-Net 44.5%, Surgformer 37.0%). If the authors' own baseline already outperforms the strongest published method, the reproduced baselines are almost certainly undertrained or evaluated under a mismatched protocol. The claimed 5.2-point margin over the best baseline therefore cannot currently be attributed to the proposed modules. The paper should provide a reproducible evaluation: released code/con
  2. [§III-B, §IV-A, and not-released artifacts] The hierarchical semantic memory is the central source of the method's claimed improvement, but the hand-authored descriptions are not released. Because the descriptions are used on the same private LCRS-100 test distribution and the text could have been iteratively tuned against the dev/test set, there is a real risk of unseen test-set fitting. The paper should release the full text prompts/descriptions for both datasets and demonstrate that the same generic descriptions are used across all videos and are not per-video or per-case tailored. Without this, the LCRS-100 gain could be an artifact of the textual priors rather than of the proposed architecture.
  3. [§III-C and §IV-A] TAS-Con is the core module, but its implementation is underspecified. The paper mentions a candidate change-point selection threshold, a transition plausibility threshold, a locally smooth phase path decoder, and short-segment merging, but none of these are defined with concrete values or algorithms. The equations only describe feature fusion and aggregation; the decision procedure that actually determines segment boundaries is left to prose. Since TAS-Con is a main contribution and the ablations show a 2.7-point accuracy drop when it is removed, this is not a minor omission. The authors should provide a precise algorithmic description (or code) so the method can be replicated and so the effect of the thresholds on the reported numbers can be assessed.
minor comments (5)
  1. [Fig. 2 and Fig. 3 captions] Figure 2 contains literal '??' placeholder tokens in the caption and in the figure text ('semantic-enhanced segment features?? ???? ? ?'), and the overview is titled 'Overview of SemanticTransitionNet' while the paper is HTT-Net. These artifacts must be cleaned before publication.
  2. [Eq. (11)] The boundary auxiliary loss uses labels g_t, but the text only says 'derived boundary label.' Please specify how g_t is generated from frame-level phase changes (e.g., smoothing, tolerance window, class imbalance handling), since the loss directly supervises the change scorer in TAS-Con.
  3. [Table I] The unrelaxed frame-based HTT-Net-F is lower than B2Q-Net-F on accuracy (94.2 vs 94.8), precision (88.4 vs 90.3), recall (90.4 vs 90.8), and Jaccard (81.4 vs 82.8), yet the text claims HTT-Net-F is 'competitive.' Please make the claim precise and acknowledge this trade-off without overstatement.
  4. [General] The reference list includes a long block of self-citations ([56]–[78]) that are unrelated to surgical phase recognition and are explicitly included 'for completeness' in §II. In a journal submission this is inappropriate and should be removed or replaced with a focused related-work discussion.
  5. [§IV-D, Table VI] The input-sequence-length study reports a drop at 512 frames but offers only a hand-wavy explanation ('redundant frames'). Please include a quantitative analysis (e.g., number of segments, boundary errors) or at least a narrative grounded in the model's behavior.

Circularity Check

0 steps flagged

No significant circularity: the method is an empirical architecture evaluated on the public Cholec80 benchmark; the LCRS-100 self-evaluation is a fairness risk, not a circular derivation.

full rationale

Walking the derivation chain, HTT-Net's components are defined from visual features, a frozen text encoder, and hand-authored semantic descriptions, not from the target predictions. The core equations are standard supervised losses: frame-level cross-entropy (Eq. 8), segment-level cross-entropy with pseudo-labels derived from ground-truth frames (Eqs. 9-10), and a boundary auxiliary loss supervised by ground-truth phase changes (Eq. 11). These are training objectives, not predictions re-derived from fitted constants. The transition-aware construction uses inter-phase transition embeddings as an independent semantic prior: Eq. (1) encodes hand-authored descriptions with a frozen PubMedBERT encoder, and TAS-Con compares local visual-phase contexts against these embeddings. There is no equation in which the claimed output (phase labels) is defined in terms of the same labels, and no fitted parameter is renamed as a prediction. The paper invokes no uniqueness theorem and does not rest on a self-citation chain. The only self-citations are explicitly bracketed as unrelated: "For completeness, additional publications from the same research group on multimodal generation, structured semantics, visual reasoning, and related learning tasks are also cited in this arXiv version [56]-[78]" (Related Work, Section II-B). These are not load-bearing. The LCRS-100 evaluation is described as "an in-house laparoscopic colorectal resection surgery dataset" with "representative baselines ... reproduced under the same data split and training protocol" (Section IV-B.1), which raises legitimate concerns about baseline tuning and external checkability, but that is a correctness/empirical-fairness risk, not a circularity of derivation: the method's outputs are not constructed to equal its inputs. On the public Cholec80 benchmark, HTT-Net is compared against published numbers under a standard split, providing independent ground for the claims. Therefore, no circular step can be exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The model contributes a new architecture but rests on hand-authored text knowledge that is not shipped and on a private evaluation set. Numeric hyperparameters (alpha, loss weights, thresholds) are unreported; the input length is selected via a validation scan. The main untested premise is that frozen biomedical text embeddings align with visual features sufficiently for transition verification, and that the in-house LCRS-100 comparison is fair. No new physical entities are introduced; the hierarchical semantic memory is best understood as a dataset-specific artifact.

free parameters (4)
  • alpha (semantic prior fusion weight) = not reported
    Eq. (2) blends visual phase evidence with the intra-phase text prior; the value is presumably tuned per dataset but is not listed in the implementation details.
  • lambda_seg and lambda_bnd (loss weights) = not reported
    Eq. (12) combines frame, segment, and boundary losses; the weighting coefficients are not reported.
  • input sequence length = 256 frames
    Chosen on LCRS-100 by scanning {64, 128, 256, 512} (Table VI); this is a data-dependent design choice.
  • candidate change-point selection threshold / transition plausibility threshold = not reported
    TAS-Con retains or suppresses candidate boundaries using a plausibility score; the criterion is described verbally in Section III-C but no threshold or scoring function parameters are given.
axioms (4)
  • domain assumption Hand-authored intra-phase, inter-phase transition, and fine-grained text descriptions are accurate and complete for both datasets.
    Section III-B builds the entire semantic memory from these descriptions; incomplete or inaccurate descriptions would misguide TAS-Con and TAS-Calib.
  • domain assumption Frozen PubMedBERT embeddings are meaningfully aligned with visual features for cross-attention despite no vision-language pretraining of the text encoder.
    Eqs. (1) and (5) assume text vectors can be used as keys/values for visual segment tokens; the paper does not independently validate this alignment.
  • domain assumption The private LCRS-100 annotations, split, and evaluation protocol are correct and unbiased.
    Section IV-A.1 defines the in-house dataset; all LCRS-100 claims depend on this premise.
  • domain assumption Derived boundary labels from frame-level phase changes provide useful supervision for candidate change scoring.
    Eq. (11) uses frame-label changes to supervise transition scores; if the frame annotations are noisy near boundaries, the boundary supervision is noisy as well.
invented entities (1)
  • Hierarchical surgical semantic memory (intra-phase / transition / fine-grained descriptions) no independent evidence
    purpose: Provides hand-written text priors for segment construction and calibration.
    This is a hand-constructed component and is not released (only one example is shown in Fig. 1). There is no external falsifiable handle beyond the paper's own in-house experiments, so the descriptions could be tuned per dataset in ways that inflate results.

pith-pipeline@v1.3.0-alltime-deepseek · 17645 in / 13654 out tokens · 126153 ms · 2026-08-01T19:55:33.650860+00:00 · methodology

0 comments
read the original abstract

Surgical video phase recognition is a fundamental task in computer-assisted intervention, supporting workflow understanding, intraoperative guidance, and surgical quality assessment. Although recent visual-temporal models have achieved promising progress, accurate and temporally coherent phase recognition remains challenging due to local visual ambiguity, transient prediction noise, and insufficient use of procedural semantics. To address these challenges, we propose HTT-Net, a Hierarchical Text-guided Transition modeling Network for surgical video phase recognition. The key idea is to introduce structured surgical semantic knowledge into phase-aware segment construction and semantic refinement. Specifically, we construct a hierarchical surgical semantic memory with intra-phase descriptions, inter-phase transition descriptions, and fine-grained semantic units. Based on this memory, the proposed Transition-Aware Segment Construction (TAS-Con) organizes frame-level evidence into coherent segment representations and handles boundary clips with inter-phase transition descriptions. Furthermore, we introduce Transition-Aware Segment Calibration (TAS-Calib), which calibrates phase-aware segment representations through hierarchical surgical semantics and improves discrimination under visual ambiguity without dense frame-level vision-language fusion. Experiments on Cholec80 and LCRS-100 demonstrate the effectiveness of HTT-Net for robust surgical video phase recognition.

Figures

Figures reproduced from arXiv: 2607.16787 by Ganbin Li, Jinghui Zhang, Kunjie Deng, Weidong Chen, Xiangjun Lyu, Yingchi Yang, Zhendong Mao.

Figure 1
Figure 1. Figure 1: Conceptual motivation of text-guided transition-aware surgical video [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed HTT-Net. The framework first extracts frame-level visual features and preliminary phase evidence from the input surgical [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of Transition-Aware Segment Construction (TAS-Con), which constructs phase-aware segments using intra-phase semantics and inter-phase [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of Transition-Aware Segment Calibration (TAS-Calib), which refines phase-aware segment representations with hierarchical surgical [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison within temporal windows around phase transitions on Cholec80. HTT-Net achieves consistently better performance than [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative visualization of prediction timelines on Cholec80. The proposed method produces phase sequences that are highly consistent with ground [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Semantic attention maps over hierarchical surgical semantic memory. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

85 extracted references · 9 linked inside Pith

  1. [1]

    Statistical modeling and recognition of surgical workflow,

    N. Padoy, T. Blum, S.-A. Ahmadi, H. Feussner, M.-O. Berger, and N. Navab, “Statistical modeling and recognition of surgical workflow,” Medical Image Analysis, vol. 16, no. 3, pp. 632–641, 2012

    2012

  2. [2]

    Surgical process modeling,

    T. Neumuth, “Surgical process modeling,”Innovative Surgical Sciences, vol. 2, no. 3, pp. 123–137, 2017. IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, 2026 10

    2017

  3. [3]

    Machine learning for surgical phase recognition: a systematic review,

    C. R. Garrow, K.-F. Kowalewski, L. Li, M. Wagner, M. W. Schmidt, S. Engelhardt, D. A. Hashimoto, H. G. Kenngott, S. Bodenstedt, S. Speidelet al., “Machine learning for surgical phase recognition: a systematic review,”Annals of Surgery, pp. 684–693, 2021

    2021

  4. [4]

    Deep learning in surgical workflow analysis: a review of phase and step recognition,

    K. C. Demir, H. Schieber, T. Weise, D. Roth, M. May, A. Maier, and S. H. Yang, “Deep learning in surgical workflow analysis: a review of phase and step recognition,”IEEE Journal of Biomedical and Health Informatics, pp. 5405–5417, 2023

    2023

  5. [5]

    Recovery of surgical workflow without explicit models,

    S.-A. Ahmadi, T. Sielhorst, R. Stauder, M. Horn, H. Feussner, and N. Navab, “Recovery of surgical workflow without explicit models,” in International Conference on Medical Image Computing and Computer- Assisted Intervention, 2006, pp. 420–428

    2006

  6. [6]

    Workflow mining for visualization and analysis of surgeries,

    T. Blum, N. Padoy, H. Feussner, and N. Navab, “Workflow mining for visualization and analysis of surgeries,”International Journal of Computer Assisted Radiology and Surgery, pp. 379–386, 2008

    2008

  7. [7]

    Knowledge-driven formalization of la- paroscopic surgeries for rule-based intraoperative context-aware assis- tance,

    D. Katic, A.-L. Wekerle, F. Gartner, H. Kenngott, B. P. Muller-Stich, R. Dillmann, and S. Speidel, “Knowledge-driven formalization of la- paroscopic surgeries for rule-based intraoperative context-aware assis- tance,” inInformation Processing in Computer-Assisted Interventions, 2014, pp. 158–167

    2014

  8. [8]

    Endonet: a deep architecture for recognition tasks on laparoscopic videos,

    A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. De Mathelin, and N. Padoy, “Endonet: a deep architecture for recognition tasks on laparoscopic videos,”IEEE Transactions on Medical Imaging, vol. 36, no. 1, pp. 86–97, 2016

    2016

  9. [9]

    Sv-rcnet: workflow recognition from surgical videos using recurrent convolutional network,

    Y . Jin, Q. Dou, H. Chen, L. Yu, J. Qin, C.-W. Fu, and P.-A. Heng, “Sv-rcnet: workflow recognition from surgical videos using recurrent convolutional network,”IEEE Transactions on Medical Imaging, vol. 37, no. 5, pp. 1114–1126, 2017

    2017

  10. [10]

    Tecno: Surgical phase recognition with multi- stage temporal convolutional networks,

    T. Czempiel, M. Paschali, M. Keicher, W. Simson, H. Feussner, S. T. Kim, and N. Navab, “Tecno: Surgical phase recognition with multi- stage temporal convolutional networks,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 2020, pp. 343–352

    2020

  11. [11]

    Temporal memory relation network for workflow recognition from surgical video,

    Y . Jin, Y . Long, C. Chen, Z. Zhao, Q. Dou, and P.-A. Heng, “Temporal memory relation network for workflow recognition from surgical video,” IEEE Transactions on Medical Imaging, vol. 40, no. 7, pp. 1911–1923, 2021

    1911

  12. [12]

    Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer,

    X. Gao, Y . Jin, Y . Long, Q. Dou, and P.-A. Heng, “Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 2021, pp. 593–603

    2021

  13. [13]

    Cascade multi-level transformer network for surgical workflow analysis,

    W. Yue, H. Liao, Y . Xia, V . Lam, J. Luo, and Z. Wang, “Cascade multi-level transformer network for surgical workflow analysis,”IEEE Transactions on Medical Imaging, pp. 2817–2831, 2023

    2023

  14. [14]

    Skit: a fast key information video transformer for online surgical phase recognition,

    Y . Liu, J. Huo, J. Peng, R. Sparks, P. Dasgupta, A. Granados, and S. Ourselin, “Skit: a fast key information video transformer for online surgical phase recognition,” inIEEE/CVF International Conference on Computer Vision, 2023, pp. 21 074–21 084

    2023

  15. [15]

    Surgformer: Surgical transformer with hierarchical temporal attention for surgical phase recognition,

    S. Yang, L. Luo, Q. Wang, and H. Chen, “Surgformer: Surgical transformer with hierarchical temporal attention for surgical phase recognition,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 2024, pp. 606–616

    2024

  16. [16]

    Lovit: Long video transformer for surgical phase recognition,

    Y . Liu, M. Boels, L. C. Garcia-Peraza-Herrera, T. Vercauteren, P. Das- gupta, A. Granados, and S. Ourselin, “Lovit: Long video transformer for surgical phase recognition,”Medical Image Analysis, vol. 99, p. 103366, 2025

    2025

  17. [17]

    Sr-mamba: Effective surgical phase recognition with state space model,

    R. Cao, J. Wang, and Y .-H. Liu, “Sr-mamba: Effective surgical phase recognition with state space model,” arXiv preprint arXiv:2407.08333, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Surgpetl: Parameter-efficient image-to-surgical-video transfer learning for surgical phase recognition,

    S. Yang, Z. Cai, L. Luo, N. Ma, S. Xu, and H. Chen, “Surgpetl: Parameter-efficient image-to-surgical-video transfer learning for surgical phase recognition,”IEEE Transactions on Medical Imaging, vol. 45, no. 3, pp. 1013–1023, 2026

    2026

  19. [19]

    Dacat: Dual-stream adaptive clip-aware time modeling for robust online surgical phase recognition,

    K. Yang, Q. Li, and Z. Wang, “Dacat: Dual-stream adaptive clip-aware time modeling for robust online surgical phase recognition,” inIEEE International Conference on Acoustics, Speech and Signal Processing, 2025

    2025

  20. [20]

    B2q- net: Bidirectional branch query network for surgical phase recognition,

    W. Zhang, Z. Li, Y . Bi, X. Jia, R. Song, Y . Zhang, and W. Zhang, “B2q- net: Bidirectional branch query network for surgical phase recognition,” IEEE Transactions on Medical Imaging, 2026

    2026

  21. [21]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,”International Confer- ence on Machine Learning, pp. 8748–8763, 2021

    2021

  22. [22]

    Clip-adapter: Better vision-language models with feature adapters,

    P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y . Zhang, H. Li, and Y . Qiao, “Clip-adapter: Better vision-language models with feature adapters,” International Journal of Computer Vision, pp. 1–15, 2023

    2023

  23. [23]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inInternational Conference on Machine Learning, 2023, pp. 19 730–19 742

    2023

  24. [24]

    Self-supervised learning for endoscopic video analysis,

    R. Hirsch, M. Caron, R. Cohen, A. Livne, T. Shapiro, T. Golany, R. Goldenberg, D. Freedman, and E. Rivlin, “Self-supervised learning for endoscopic video analysis,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 2023, pp. 569– 578

    2023

  25. [25]

    Endovit: Pretraining vision transformers on a large collection of endoscopic images,

    D. Batic, F. Holm, E. Ozsoy, T. Czempiel, and N. Navab, “Endovit: Pretraining vision transformers on a large collection of endoscopic images,”International Journal of Computer Assisted Radiology and Surgery, vol. 19, no. 6, pp. 1085–1091, 2024

    2024

  26. [26]

    Dissecting self- supervised learning methods for surgical computer vision,

    S. Ramesh, V . Srivastav, D. Alapatt, T. Yu, A. Murali, L. Sestini, C. I. Nwoye, I. Hamoud, S. Sharma, A. Fleurentinet al., “Dissecting self- supervised learning methods for surgical computer vision,”Medical Image Analysis, vol. 88, p. 102844, 2023

    2023

  27. [27]

    Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos,

    C. I. Nwoye, T. Yu, C. Gonzalez, B. Seeliger, P. Mascagni, D. Mutter, J. Marescaux, and N. Padoy, “Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos,”Medical Image Analysis, vol. 78, p. 102433, 2022

    2022

  28. [28]

    Multi-attention network for compressed video referring object segmentation,

    W. Chen, D. Hong, Y . Qi, Z. Han, S. Wang, L. Qing, Q. Huang, and G. Li, “Multi-attention network for compressed video referring object segmentation,” inProceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 4416–4425

    2022

  29. [29]

    Combatting data imbalance and noise in micro-action recognition,

    C. Wang, W. Chen, X. Cui, Y . Zhao, Z. Qi, P. Huang, X. Liu, and W. Zhang, “Combatting data imbalance and noise in micro-action recognition,” inProceedings of the 33rd ACM International Conference on Multimedia, 2025, pp. 14 229–14 235

    2025

  30. [30]

    Hard frame detection and online mapping for surgical phase recognition,

    F. Yi and T. Jiang, “Hard frame detection and online mapping for surgical phase recognition,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 2019, pp. 449– 457

    2019

  31. [31]

    Multi- task recurrent convolutional network with correlation loss for surgical video analysis,

    Y . Jin, H. Li, Q. Dou, H. Chen, J. Qin, C.-W. Fu, and P.-A. Heng, “Multi- task recurrent convolutional network with correlation loss for surgical video analysis,”Medical Image Analysis, vol. 59, p. 101572, 2020

    2020

  32. [32]

    Opera: Attention-regularized transformers for surgical phase recognition,

    T. Czempiel, M. Paschali, D. Ostler, S. T. Kim, B. Busam, and N. Navab, “Opera: Attention-regularized transformers for surgical phase recognition,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 2021, pp. 604–614

    2021

  33. [33]

    A vision transformer for decoding surgeon activity from surgical videos,

    D. Kiyasseh, R. Ma, T. F. Haque, B. J. Miles, C. Wagner, D. A. Donoho, A. Anandkumar, and A. J. Hung, “A vision transformer for decoding surgeon activity from surgical videos,”Nature Biomedical Engineering, pp. 780–796, 2023

    2023

  34. [34]

    On the pitfalls of batch normal- ization for end-to-end video learning: a study on surgical workflow analysis,

    D. Rivoir, I. Funke, and S. Speidel, “On the pitfalls of batch normal- ization for end-to-end video learning: a study on surgical workflow analysis,”Medical Image Analysis, p. 103126, 2024

    2024

  35. [35]

    Graph mixture of experts and memory-augmented routers for multivariate time series anomaly detec- tion,

    X. Huang, W. Chen, B. Hu, and Z. Mao, “Graph mixture of experts and memory-augmented routers for multivariate time series anomaly detec- tion,” inProceedings of the AAAI Conference on Artificial Intelligence, 2025

    2025

  36. [36]

    Towards efficient partially relevant video retrieval with active moment discovering,

    P. Song, L. Zhang, L. Lan, W. Chen, D. Guo, X. Yang, and M. Wang, “Towards efficient partially relevant video retrieval with active moment discovering,”IEEE Transactions on Multimedia, vol. 27, pp. 6740–6751, 2025

    2025

  37. [37]

    Difference-aware itera- tive reasoning network for key relation detection,

    B. Zhao, W. Chen, B. Hu, H. Xie, and Z. Mao, “Difference-aware itera- tive reasoning network for key relation detection,” inIEEE International Conference on Multimedia and Expo, 2023, pp. 276–281

    2023

  38. [38]

    Vivit: A video vision transformer,

    A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lucic, and C. Schmid, “Vivit: A video vision transformer,” inIEEE/CVF International Confer- ence on Computer Vision, 2021, pp. 6836–6846

    2021

  39. [39]

    Is space-time attention all you need for video understanding?

    G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” inInternational Conference on Machine Learning, 2021

    2021

  40. [40]

    Sur- gical workflow recognition: from analysis of challenges to architectural study,

    T. Czempiel, A. Sharghi, M. Paschali, N. Navab, and O. Mohareri, “Sur- gical workflow recognition: from analysis of challenges to architectural study,” inEuropean Conference on Computer Vision, 2022, pp. 556–568

    2022

  41. [41]

    Metrics matter in surgical phase recognition,

    I. Funke, D. Rivoir, and S. Speidel, “Metrics matter in surgical phase recognition,” arXiv preprint arXiv:2305.13961, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Ms-tcn: Multi-stage temporal convolutional network for action segmentation,

    Y . A. Farha and J. Gall, “Ms-tcn: Multi-stage temporal convolutional network for action segmentation,” inIEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2019, pp. 3575–3584

    2019

  43. [43]

    How much temporal long-term context is needed for action segmentation?

    E. Bahrami, G. Francesca, and J. Gall, “How much temporal long-term context is needed for action segmentation?” inIEEE/CVF International Conference on Computer Vision, 2023. IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, 2026 11

    2023

  44. [44]

    Fact: Frame-action cross-attention temporal modeling for efficient supervised action segmentation,

    Z. Lu and E. Elhamifar, “Fact: Frame-action cross-attention temporal modeling for efficient supervised action segmentation,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  45. [45]

    Diffusion ac- tion segmentation,

    D. Liu, Q. Li, A.-D. Dinh, T. Jiang, M. Shah, and C. Xu, “Diffusion ac- tion segmentation,” inIEEE/CVF International Conference on Computer Vision, 2023

    2023

  46. [46]

    Cascade cross-modal attention network for video actor and action segmentation from a sentence,

    W. Chen, G. Li, X. Zhang, H. Yu, S. Wang, and Q. Huang, “Cascade cross-modal attention network for video actor and action segmentation from a sentence,” inProceedings of the 29th ACM International Con- ference on Multimedia, 2021, pp. 4053–4062

    2021

  47. [47]

    Weakly supervised text-based actor-action video segmentation by clip-level multi-instance learning,

    W. Chen, G. Li, X. Zhang, S. Wang, L. Li, and Q. Huang, “Weakly supervised text-based actor-action video segmentation by clip-level multi-instance learning,”ACM Transactions on Multimedia Computing, Communications and Applications, 2023

    2023

  48. [48]

    Improving image captioning via predicting structured concepts,

    T. Wang, W. Chen, Y . Tian, Y . Song, and Z. Mao, “Improving image captioning via predicting structured concepts,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, 2023

    2023

  49. [49]

    Exploring visual rela- tionships via transformer-based graphs for enhanced image captioning,

    J. Li, Z. Mao, H. Li, W. Chen, and Y . Zhang, “Exploring visual rela- tionships via transformer-based graphs for enhanced image captioning,” ACM Transactions on Multimedia Computing, Communications and Applications, 2024

    2024

  50. [50]

    Improving video summarization by exploring the coherence between corresponding captions,

    C. Ye, W. Chen, B. Hu, L. Zhang, Y . Zhang, and Z. Mao, “Improving video summarization by exploring the coherence between corresponding captions,”IEEE Transactions on Image Processing, vol. 34, pp. 5369– 5384, 2025

    2025

  51. [51]

    Dual-path collaborative generation network for emotional video captioning,

    C. Ye, W. Chen, J. Li, L. Zhang, and Z. Mao, “Dual-path collaborative generation network for emotional video captioning,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 496– 505

    2024

  52. [52]

    Multi- round mutual emotion-cause pair extraction for emotion-attributed video captioning,

    C. Ye, W. Chen, P. Song, X. Liu, L. Zhang, and Z. Mao, “Multi- round mutual emotion-cause pair extraction for emotion-attributed video captioning,” inProceedings of the 33rd ACM International Conference on Multimedia, 2025, pp. 3320–3329

    2025

  53. [53]

    Subjective- objective emotion correlated generation network for subjective video captioning,

    W. Chen, C. Ye, P. Song, L. Zhang, Y . Zhang, and Z. Mao, “Subjective- objective emotion correlated generation network for subjective video captioning,”IEEE Transactions on Image Processing, vol. 35, pp. 540– 555, 2026

    2026

  54. [54]

    Towards accurate emotion-attributed video captioning via fine-grained emotion- cause pair extraction,

    W. Chen, C. Ye, Z. Mao, L. Wang, X. Liu, and Y . Zhang, “Towards accurate emotion-attributed video captioning via fine-grained emotion- cause pair extraction,”arXiv preprint arXiv:2606.08566, 2026

    Pith/arXiv arXiv 2026

  55. [55]

    Audio-visual exchange-aware token pruning for efficient audio-visual captioning,

    Z. Meng, D. Hong, W. Chen, Z. Zhou, B. Hu, and Z. Mao, “Audio-visual exchange-aware token pruning for efficient audio-visual captioning,” in CAAI International Conference on Artificial Intelligence, 2026

    2026

  56. [56]

    Bootstrapping large language models for radiology report generation,

    C. Liu, Y . Tian, W. Chen, Y . Song, and Y . Zhang, “Bootstrapping large language models for radiology report generation,” inProceedings of the AAAI Conference on Artificial Intelligence, 2024

    2024

  57. [57]

    Creatidesign: A unified multi-conditional diffusion transformer for creative graphic design,

    H. Zhang, D. Hong, M. Yang, Y . Cheng, Z. Zhang, W. Chen, J. Shao, X. Wuet al., “Creatidesign: A unified multi-conditional diffusion transformer for creative graphic design,” inInternational Conference on Learning Representations, 2025

    2025

  58. [58]

    Prompting few-shot multi- hop question generation via comprehending type-aware semantics,

    Z. Lin, W. Chen, Y . Song, and Y . Zhang, “Prompting few-shot multi- hop question generation via comprehending type-aware semantics,” in Findings of the Association for Computational Linguistics: NAACL, 2024, pp. 3730–3740

    2024

  59. [59]

    Improving radiology report generation with multi-grained abnormality prediction,

    Y . Jin, W. Chen, Y . Tian, Y . Song, and C. Yan, “Improving radiology report generation with multi-grained abnormality prediction,”Neurocom- puting, vol. 600, p. 128122, 2024

    2024

  60. [60]

    Text style transfer with contrastive transfer pattern mining,

    J. Han, Q. Wang, L. Zhang, W. Chen, Y . Song, and Z. Mao, “Text style transfer with contrastive transfer pattern mining,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023

    2023

  61. [61]

    End-to-end aspect-based sentiment analysis with combinatory categorial grammar,

    Y . Tian, W. Chen, B. Hu, Y . Song, and F. Xia, “End-to-end aspect-based sentiment analysis with combinatory categorial grammar,” inFindings of the Association for Computational Linguistics: ACL, 2023, pp. 13 597– 13 609

    2023

  62. [62]

    Improving radiology report generation with d2-net: When diffusion meets discrim- inator,

    Y . Jin, W. Chen, Y . Tian, Y . Song, C. Yan, and Z. Mao, “Improving radiology report generation with d2-net: When diffusion meets discrim- inator,” inIEEE International Conference on Acoustics, Speech and Signal Processing, 2024

    2024

  63. [63]

    Sentiment-oriented transformer- based variational autoencoder network for live video commenting,

    F. Fu, S. Fang, W. Chen, and Z. Mao, “Sentiment-oriented transformer- based variational autoencoder network for live video commenting,” ACM Transactions on Multimedia Computing, Communications and Applications, 2024

    2024

  64. [64]

    Rethinking pseudo word learning in zero-shot composed image retrieval: From an object-aware perspective,

    Z. Li, L. Zhang, K. Zhang, W. Chen, Y . Zhang, and Z. Mao, “Rethinking pseudo word learning in zero-shot composed image retrieval: From an object-aware perspective,” inProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2025

    2025

  65. [65]

    Creatiposter: Towards editable and controllable multi- layer graphic design generation,

    D. Hong, Z. Zhang, W. Chen, Y . Cheng, M. Yang, G. Shi, H. Zhang, and Z. Mao, “Creatiposter: Towards editable and controllable multi- layer graphic design generation,” inCAAI International Conference on Artificial Intelligence, 2025

    2025

  66. [66]

    Stimuli-aware emotion adaptor for enhancing llm in affective explanation captioning,

    Z. Zhang, P. Song, J. Hu, W. Chen, L. Ni, and X. Yang, “Stimuli-aware emotion adaptor for enhancing llm in affective explanation captioning,” inIEEE International Conference on Acoustics, Speech and Signal Processing, 2026

    2026

  67. [67]

    Contour-augmented concept prediction network for image captioning,

    T. Wang, W. Chen, J. Li, Y . Peng, and Z. Mao, “Contour-augmented concept prediction network for image captioning,” inInternational Conference on Artificial Neural Networks, 2023, pp. 180–191

    2023

  68. [68]

    A multi-agent framework with structured reasoning and reflective refine- ment for multimodal empathetic response generation,

    L. Wang, C. Ye, W. Chen, P. Song, B. Hu, and Z. Mao, “A multi-agent framework with structured reasoning and reflective refine- ment for multimodal empathetic response generation,”arXiv preprint arXiv:2604.18988, 2026

    Pith/arXiv arXiv 2026

  69. [69]

    Creatiparser: Generative image parsing of raster graphic designs into editable layers,

    W. Chen, D. Hong, Z. Mao, Y . Cheng, X. Liu, L. Zhang, and Y . Zhang, “Creatiparser: Generative image parsing of raster graphic designs into editable layers,”arXiv preprint arXiv:2604.19632, 2026

    Pith/arXiv arXiv 2026

  70. [70]

    Face-net: Factual calibration and emotion augmenta- tion for retrieval-enhanced emotional video captioning,

    W. Chen, C. Ye, Z. Mao, P. Song, X. Liu, L. Zhang, X. Chang, and Y . Zhang, “Face-net: Factual calibration and emotion augmenta- tion for retrieval-enhanced emotional video captioning,”arXiv preprint arXiv:2603.17455, 2026

    arXiv 2026

  71. [71]

    Query-based collaborative multimodal token pruning for audio-visual question answering,

    X. Qin, D. Hong, W. Chen, C. Ye, X. Liu, P. Song, and L. Zhang, “Query-based collaborative multimodal token pruning for audio-visual question answering,” inInternational Conference on Artificial Intelli- gence, Human-Computer Interaction and Smart Agriculture, 2025

    2025

  72. [72]

    Hier- archical knowledge distillation for cross-lingual stance detection,

    Q. Zhou, J. Yao, S. Tang, W. Chen, L. Cheng, and J. Tang, “Hier- archical knowledge distillation for cross-lingual stance detection,” in International Conference on Artificial Intelligence, Human-Computer Interaction and Smart Agriculture, 2025

    2025

  73. [73]

    Emo- verse: A mllms-driven emotion representation dataset for interpretable visual emotion analysis,

    Y . Guo, D. Hong, W. Chen, Z. She, C. Ye, X. Chang, and Z. Mao, “Emo- verse: A mllms-driven emotion representation dataset for interpretable visual emotion analysis,”arXiv preprint arXiv:2511.12554, 2025

    Pith/arXiv arXiv 2025

  74. [74]

    Matching street view and satellite images via drone imagery and semantic descriptions,

    X. Liu, W. Chen, Z. Qi, B. Zhang, and W. Zhang, “Matching street view and satellite images via drone imagery and semantic descriptions,” in International Workshop on UAVs in Multimedia, 2025

    2025

  75. [75]

    Bridging subjectivity in affective explanation captioning via consensus-prompted emotion reasoning,

    P. Song, Z. Zhang, W. Chen, J. Hu, X. Yang, and X. Chang, “Bridging subjectivity in affective explanation captioning via consensus-prompted emotion reasoning,”IEEE Transactions on Image Processing, 2026

    2026

  76. [76]

    Histllm: A unified framework for llm-based multimodal recommendation with user history encoding and compression,

    C. Zhang, B. Hu, W. Chen, and Z. Mao, “Histllm: A unified framework for llm-based multimodal recommendation with user history encoding and compression,”arXiv preprint arXiv:2504.10150, 2025

    Pith/arXiv arXiv 2025

  77. [77]

    Emostyle: Affective conditioning of style-specialist experts for emo- tional image generation,

    D. Hong, Y . Guo, W. Chen, X. Liu, Z. Zou, Z. Mao, and Y . Zhang, “Emostyle: Affective conditioning of style-specialist experts for emo- tional image generation,”arXiv preprint arXiv:2607.10165, 2026

    Pith/arXiv arXiv 2026

  78. [78]

    Geometry-aware gaussian prior and axial attention for cervical cytology image classification,

    Y . Li, C. Ye, N. Lyu, W. Chen, and Z. Mao, “Geometry-aware gaussian prior and axial attention for cervical cytology image classification,” arXiv preprint arXiv:2607.10278, 2026

    Pith/arXiv arXiv 2026

  79. [79]

    Workshop and challenges on modeling and monitoring of computer assisted interventions,

    A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. De Mathelin, and N. Padoy, “Workshop and challenges on modeling and monitoring of computer assisted interventions,” http://camma.u-strasbg.fr/m2cai2016/, 2016

    2016

  80. [80]

    Convnext v2: Co-designing and scaling convnets with masked au- toencoders,

    S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “Convnext v2: Co-designing and scaling convnets with masked au- toencoders,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

Showing first 80 references.