Pith. sign in

REVIEW 5 major objections 4 minor 74 references

EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EndoARSS argues that a frozen DINOv2 encoder with task-specific low-rank adapters and spatially aware attention outperforms single-task and prior multi-task models on endoscopic activity recognition and semantic segmentation.

desk verdict Solid engineering with genuinely useful new datasets, but the SOTA claim hinges on a possible frame-level train/test leak and the two headline modules are known parts reassembled; worth a serious referee, not worth taking on faith. read the letter →

arxiv 2506.06830 v1 pith:EW76M5PR submitted 2025-06-07 cs.CV cs.AI

Guankun Wang , Rui Tang , Mengya Xu , Long Bai , Huxin Gao , Hongliang Ren This is my paper
classification cs.CVcs.AI
keywords multi-tasklearningendoscopicsurgerysurgicalactivityrecognitionsemanticsegmentationlow-rankadaptationfoundationmodelDINOv2spatiallyawareattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EndoARSS is a multi-task learning framework that tries to show that a frozen DINOv2 image encoder, adapted with low-rank parameter updates, can perform surgical activity recognition and semantic segmentation together better than either task alone or than prior multi-task architectures. The paper reports that this combination reaches 92.16% mIoU and 95.79% Dice on its new MTLESD dataset, 72.95% mIoU and 83.47% Dice on MTLEndovis, and 49.10% mIoU on the held-out MTLEndovis-Gen dataset, while keeping training on a single GPU practical. The central claim is that two mechanisms carry the gains: task-specific low-rank adapters (TESLA) that separate gradients between tasks, and a Spatially-Aware Multi-Scale Attention (SMA) module that gives the model a global receptive field while preserving convolutional spatial bias. If true, this would make foundation-model-based multi-task perception a realistic route for real-time surgical assistance.

What carries the argument

The load-bearing machinery is a frozen DINOv2 Vision Transformer encoder with three small trainable additions. LoRA layers, applied to the query and value projections, express weight updates as products of two low-rank matrices so the backbone stays frozen. TESLA, the Task Efficient Shared Low-rank Adapters, attaches two task-specific low-rank adapters to the shared backbone and combines them by task identifier, giving each task its own gradient path and reducing training conflicts. SMA, the Spatially-Aware Multi-Scale Attention, runs before patchification: it convolves the input, unfolds it into non-overlapping patches, applies a multi-scale attention module that pools along two spatial directions, and folds the result back, so every output pixel can receive information from the whole $H\times W$ image while retaining convolutional spatial bias. These components feed two decoder heads, one for activity classification and one for segmentation.

What would settle it

Re-run the MTLESD and MTLEndovis evaluations with the 4:1 split made at the level of entire surgical videos and compare mIoU and accuracy; if the margins over baselines shrink or the absolute numbers fall, the results depended on frame-level overlap rather than on the TESLA adapters and SMA attention.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a frozen general-purpose vision foundation model can be turned into a strong surgical multi-task system by adding only small trainable adapters. The authors report that EndoARSS reaches 92.16% mIoU and 95.79% Dice on MTLESD, 72.95% mIoU and 83.47% Dice on MTLEndovis, and 49.10% mIoU on the MTLEndovis-Gen domain-shift set, with the best robustness under 19 corruption types (mean mIoU 79.57%). They attribute the gains to TESLA, which isolates each task's gradients, and to SMA, which gives a global receptive field with convolutional spatial bias, and they interpret the consistent margins over both single-task and prior multi-task baselines as evidence that cross-task feature interaction, not a bigger backbone, is doing the work.

Load-bearing premise

The reported gains assume the 4:1 train/test split separates entire surgical videos, so no near-duplicate frames from the same procedure appear in both training and test sets; the paper does not state that this is the split rule.

Editorial extensions

If this is right

  • On MTLESD, adding the activity-recognition task to a frozen DINOv2 encoder lifts segmentation mIoU from 88.19% (segmentation only) to 92.16%, so the two tasks reinforce rather than compete.
  • Explicit gradient separation through task-specific low-rank adapters lets one backbone serve heterogeneous surgical data without the conflict that limits shared-parameter multi-task learning.
  • SMA's global receptive field resolves visual confusion between instruments and tissue, which is why robustness under corruption also improves, reaching a mean mIoU of 79.57% against 72.06-72.88% for the compared baselines.
  • Training requires about 9.9 GB of GPU memory and inference runs at 35.66 images per second, so the approach fits near-real-time surgical assistive systems on modest hardware.
  • The MTLEndovis-Gen evaluation, where a model trained on MTLEndovis is tested on a different challenge dataset, indicates that the adapted representations transfer across endoscopic domain shifts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not state whether the train/test splits for MTLESD and MTLEndovis are video-level; if frames from the same recording cross the split, the reported margins over baselines could shrink under a video-level evaluation (our inference from Section IV-A).
  • Because the same cross-activity feature ambiguity appears in other endoscopic tasks, the frozen-backbone-plus-TESLA recipe could plausibly extend to depth estimation, tool-tissue interaction detection, or phase recognition, which the authors do not test.
  • The current model is frame-wise; adding a temporal head over the SMA features is a direct extension that could exploit the sequential structure of surgical activities.
  • The robustness results use synthetic corruptions, so an unstated next test is whether the model stays robust under real endoscope artifacts such as lens fog, specular highlights, and motion blur during live video.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. EndoARSS proposes a multi-task learning framework for simultaneous surgical activity recognition and semantic segmentation, built on a frozen DINOv2 encoder with low-rank adapters, task-specific shared low-rank adapters (TESLA), and a spatially-aware multi-scale attention (SMA) module. The authors introduce three datasets—MTLESD, MTLEndovis, and MTLEndovis-Gen—derived from porcine ESD procedures, EndoVis-18, and EndoVis-17 respectively, and report improvements over internal MTL baselines on accuracy, mIoU, Dice, HD, and SSIM, plus robustness to image corruptions. The paper also releases code and data.

Significance. The datasets and code release are potentially valuable community contributions, and the held-out MTLEndovis-Gen evaluation is a genuine generalization test that strengthens the work. If the reported performance differences survive properly separated splits and significance testing, the parameter-efficient adapter design combined with the SMA module would be a useful contribution to surgical scene understanding. However, as presented, the empirical evidence is not yet strong enough to support the state-of-the-art and robustness claims, primarily because of split-leakage risk, missing error bars on the main comparisons, and very large ablation standard deviations.

major comments (5)
  1. [IV-A Datasets] The paper does not state whether the train/test splits are performed at the video level. MTLESD is 'partitioned into training and test sets in a 4:1 ratio' per activity, and MTLEndovis is 'randomly partitioned in the same ratio', but both datasets are sequences extracted from surgical videos, so consecutive frames are highly correlated in illumination, tissue deformation, and instrument pose. If frames from the same procedure appear in both training and test, the reported numbers in Tables II, III, IV, and VII are inflated and the state-of-the-art claim would not transfer to unseen procedures. Please clarify the split granularity and, if the current splits are frame-level, provide video-level or clip-level splits and report the corresponding results.
  2. [Tables II-V] The main results tables (II, III, IV) report single runs without error bars, while the ablation study in Table V reports large standard deviations: for example, mIoU is 92.16±5.06 on MTLESD and 72.95±4.42 on MTLEndovis, and activity accuracy on MTLEndovis is 72.18±21.07. With these magnitudes of variance, the small differences between configurations (e.g., 91.99 vs 92.16 mIoU in Table II, or 70.64 vs 72.95 in Table III) are not statistically meaningful. Please report multiple-seed means and standard deviations for all main comparisons, and ideally paired significance tests, before claiming that the proposed components provide consistent gains.
  3. [Table VI and Section IV-C4] The rank size of the TESLA adapters is selected using test-set performance: ranks 1, 4, 8, and 16 are evaluated on the test sets, and rank 4 is chosen because it gives the best mIoU. This is a form of selection on the test data, which can bias the reported gains. Please use a validation split for hyperparameter selection, or perform nested cross-validation, and report results for the selected rank on truly held-out test data.
  4. [Tables II-IV and Section IV-C] The paper claims to 'outperform existing models' and achieve 'SOTA performance', but the comparisons in Tables II-IV are limited to internal combinations of the selected backbones (ResNet101, DINOv2, EndoARSS backbone), architectures (MoCo, Aligned-MTL, STCH, DB-MTL), and optimization strategies (MMoE, DSelect_k, TESLA). No comparison is made with published state-of-the-art surgical segmentation or surgical activity recognition methods on the same data. The state-of-the-art claim is therefore not supported by the presented experimental design; at minimum, the wording should be tempered to 'competitive with the internal MTL baselines', or external published methods should be included.
  5. [Table VII and Section IV-C5] The robustness claim is based on a comparison of only three configurations (ResNet101 with DSelect_k, ResNet101 with MMoE, and EndoARSS), all evaluated on MTLESD, and no standard deviations are reported. Since Table V shows that mIoU on MTLESD has a standard deviation above 5 points, the robustness margin in Table VII (e.g., mean mIoU 79.57 vs 72.88) cannot be interpreted without repeated runs. In addition, the robustness evaluation is not performed on MTLEndovis or MTLEndovis-Gen, so the general claim of 'best robustness' is broader than the evidence. Please add error bars and, ideally, robustness results on the other datasets, or restrict the claim.
minor comments (4)
  1. [Section V] The ethical statement says 'all endoscopic videos used in this study were sourced from publicly available datasets', but MTLESD is derived from the DREAMS system, which is not described as a public dataset. This apparent inconsistency should be clarified.
  2. [Section III-D, Eq. (6)] The sentence describing the reshaping of W'' is garbled: 'we reshape the integrated weight W'' ∈ R^{b×Cout×Cin×k×k} to R^{bCout×Cin×k×k}, r to R^{1×bCin×H×W}'. This needs a clean mathematical restatement.
  3. [Table II] All models achieve 99.99% activity recognition accuracy on MTLESD, so the activity-recognition comparison on this dataset is at ceiling and does not discriminate between methods. This should be acknowledged, and the discriminative results should be the focus.
  4. [Table VII] The baseline accuracy collapses to 8.71% under smoke and contrast corruption, but EndoARSS also drops to 50.53% and 29.85% respectively; the discussion should acknowledge that the proposed method still degrades substantially in these conditions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EndoARSS is an empirical multi-task learning study whose claims rest on held-out evaluations, not on a derivation that reduces to its inputs.

full rationale

EndoARSS makes no first-principles derivation; its claims are empirical benchmark results. The central comparisons (Tables II-IV and VII) are obtained by training on MTLESD/MTLEndovis training splits and measuring mIoU and accuracy on the corresponding test splits, and the MTLEndovis-Gen evaluation trains solely on MTLEndovis and tests on EndoVis-17 frames, which is a genuinely held-out domain and provides independent support. The use of self-cited data sources ([64],[65] DREAMS; [66] VQLA-Extend) is provenance for the datasets and annotations, not a load-bearing argument: the activity and segmentation labels are external ground truth used to define the tasks, and the model does not fit and then re-predict those labels as a derived quantity. There is no uniqueness theorem, no ansatz smuggled by self-citation, and no renamed empirical pattern presented as organization. The concern that frame-level random splits may leak temporally correlated frames is a test-validity and correctness risk, not circularity, because the reported numbers would still be measurements, not identities; it does not reduce any equation to its inputs. Accordingly, the paper is self-contained as an empirical study against external and held-out benchmarks, and no circular step can be quoted from the text.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the transferability of DINOv2 features, the validity of the auxiliary-task signal, the statistical independence of the data splits, and the reliability of the annotations. No new physical or mathematical entities are introduced. The main free parameters are the LoRA rank (chosen on test data), the learning rates, the adapter weights in Eq. 5, and the SMA patch size, which is not specified numerically.

free parameters (4)
  • LoRA rank r = 4
    Rank 4 is chosen as the best value in Table VI based on test-set mIoU for ranks 1, 4, 8, 16; no separate validation split is described.
  • Learning rates = 1e-4 (MTLESD), 1e-3 (MTLEndovis)
    Set per dataset in Section IV-B and not swept; they affect final performance.
  • Adapter weights alpha_i in Eq. 5 = Not reported (assumed 1)
    Equation 5 weights task adapters by alpha_i, but the paper never reports how these are set; defaulting them to 1 is a hand choice.
  • SMA patch size n = Not specified
    Section III-B refers to an n by n convolution and patch height/width h,w <= n, but no numerical value is given, leaving a key architectural choice unspecified.
assumptions (5)
  • domain assumption DINOv2 features transfer to the endoscopic domain.
    Section III-A assumes the frozen DINOv2 encoder provides useful visual features for surgical scenes; this is supported mainly by their own experiments, not external evidence.
  • domain assumption Activity recognition provides a useful cross-task signal for segmentation.
    Section I and Figure 2 motivate multi-task learning as reducing cross-activity interference, but no control with a dummy auxiliary task is tested; on MTLESD activity recognition is saturated at 99.99%, so the cross-task signal is not measured.
  • domain assumption Frames in training and test splits are independent.
    Section IV-A describes 4:1 random splits but does not state that videos were split at the video level; if frames from the same video appear in both train and test, temporal correlation inflates scores.
  • domain assumption The annotations are reliable.
    Section IV-A relies on expert annotations from Qilu Hospital and activity labels from EndoVis-18-VQLA-Extend [66] without inter-annotator agreement or label-quality metrics.
  • domain assumption The MTLEndovis-Gen evaluation is a fair generalization test.
    MTLEndovis-Gen contains only 97 frames, so the generalization metric has high variance; the paper reports no confidence intervals for this set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery." pith.science (2026). https://pith.science/paper/EW76M5PR

@misc{pith2026250606830,
  author       = {Pith},
  title        = {Pith review of: EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EW76M5PR}},
  note         = {Machine review of arXiv:2506.06830}
}
read the original abstract

Endoscopic surgery is the gold standard for robotic-assisted minimally invasive surgery, offering significant advantages in early disease detection and precise interventions. However, the complexity of surgical scenes, characterized by high variability in different surgical activity scenarios and confused image features between targets and the background, presents challenges for surgical environment understanding. Traditional deep learning models often struggle with cross-activity interference, leading to suboptimal performance in each downstream task. To address this limitation, we explore multi-task learning, which utilizes the interrelated features between tasks to enhance overall task performance. In this paper, we propose EndoARSS, a novel multi-task learning framework specifically designed for endoscopy surgery activity recognition and semantic segmentation. Built upon the DINOv2 foundation model, our approach integrates Low-Rank Adaptation to facilitate efficient fine-tuning while incorporating Task Efficient Shared Low-Rank Adapters to mitigate gradient conflicts across diverse tasks. Additionally, we introduce the Spatially-Aware Multi-Scale Attention that enhances feature representation discrimination by enabling cross-spatial learning of global information. In order to evaluate the effectiveness of our framework, we present three novel datasets, MTLESD, MTLEndovis and MTLEndovis-Gen, tailored for endoscopic surgery scenarios with detailed annotations for both activity recognition and semantic segmentation tasks. Extensive experiments demonstrate that EndoARSS achieves remarkable performance across multiple benchmarks, significantly improving both accuracy and robustness in comparison to existing models. These results underscore the potential of EndoARSS to advance AI-driven endoscopic surgical systems, offering valuable insights for enhancing surgical safety and efficiency.

Figures

Figures reproduced from arXiv: 2506.06830 by the authors.

Figure 1
Figure 1. Endoscopic submucosal dissection activity information can be utilized to improve performance in cross-activity segmentation. (a) Examples of ESD [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Segmentation results of DINOv2 [19] and MTL model (DINOv2 [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of our proposed EndoARSS framework. EndoARSS comprises spatial-aware multi-scale attention, freezed DINOv2 image encoder, trainable [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The overview of TESLA. Blue rectangles represent shared weights [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 1
Figure 1. Figure 1: In total, we extracted and annotated 2,020 images. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png]
Figure 5
Figure 5. Figure 5: Segmentation visualization on the MTLESD dataset. All MTL configurations incorporate DB-MTL as an optimization strategy due to its superior [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison of robustness across severity levels. We evaluate the [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 40 canonical work pages

  1. [63]

    Exploring training on heterogeneous data with mixture of low-rank adapters,

    Y . Zhou, Z. Zhao, H. Li, S. Du, J. Yao, Y . Zhang, and Y . Wang, “Exploring training on heterogeneous data with mixture of low-rank adapters,”arXiv preprint arXiv:2406.09679, 2024

    arXiv 2024

  2. [61]

    Efficient multi-scale attention module with cross-spatial learning,

    D. Ouyang, S. He, G. Zhang, M. Luo, H. Guo, J. Zhan, and Z. Huang, “Efficient multi-scale attention module with cross-spatial learning,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  3. [1]

    Rethinking exemplars for continual semantic segmentation in endoscopy scenes: Entropy-based mini-batch pseudo-replay,

    G. Wang, L. Bai, Y . Wu, T. Chen, and H. Ren, “Rethinking exemplars for continual semantic segmentation in endoscopy scenes: Entropy-based mini-batch pseudo-replay,”Computers in Biology and Medicine, vol. 165, p. 107412, 2023. 13

    work page 2023

  4. [2]

    Endoscopy disease detection challenge 2020

    S. Ali, N. Ghatwary, B. Braden, D. Lamarque, A. Bailey, S. Realdon, R. Cannizzaro, J. Rittscher, C. Daul, and J. East, “Endoscopy disease detection challenge 2020,”arXiv preprint arXiv:2003.03376, 2020

    work page Pith review arXiv 2020

  5. [3]

    Complications following endoscopic submucosal dissection for gastric, esophageal, and colorectal cancer: a review of studies based on nationwide large-scale databases,

    H. Odagiri and H. Yasunaga, “Complications following endoscopic submucosal dissection for gastric, esophageal, and colorectal cancer: a review of studies based on nationwide large-scale databases,”Annals of Translational Medicine, vol. 5, no. 8, 2017

    work page 2017

  6. [4]

    Endoscopic submucosal dissection for early gastric cancer performed by supervised residents: assessment of feasibility and learning curve,

    S. Yamamoto, N. Uedo, R. Ishihara, N. Kajimoto, H. Ogiyama, Y . Fukushima, S. Yamamoto, Y . Takeuchi, K. Higashino, H. Iishiet al., “Endoscopic submucosal dissection for early gastric cancer performed by supervised residents: assessment of feasibility and learning curve,” Endoscopy, pp. 923–928, 2009

    work page 2009

  7. [5]

    Savanet: Surgical action-driven visual attention network for autonomous endoscope control,

    H. Gao, W. Fan, L. Qiu, X. Yang, Z. Li, X. Zuo, Y . Li, M. Q.-H. Meng, and H. Ren, “Savanet: Surgical action-driven visual attention network for autonomous endoscope control,”IEEE Transactions on Automation Science and Engineering, vol. 20, no. 4, pp. 2655–2667, 2022

    work page 2022

  8. [6]

    2018 robotic scene segmentation challenge,

    M. Allan, S. Kondo, S. Bodenstedt, S. Leger, R. Kadkhodamoham- madi, I. Luengo, F. Fuentes, E. Flouty, A. Mohammed, M. Pedersen et al., “2018 robotic scene segmentation challenge,”arXiv preprint arXiv:2001.11190, 2020

    arXiv 2018

Show all 74 references
  1. [7]

    Learning and reasoning with the graph structure representation in robotic surgery,

    M. Islam, L. Seenivasan, L. C. Ming, and H. Ren, “Learning and reasoning with the graph structure representation in robotic surgery,” in International Conference on Medical Image Computing and Computer- Assisted Intervention. Springer, 2020, pp. 627–636

    2020

  2. [8]

    Domain adaptive sim- to-real segmentation of oropharyngeal organs,

    G. Wang, T.-A. Ren, J. Lai, L. Bai, and H. Ren, “Domain adaptive sim- to-real segmentation of oropharyngeal organs,”Medical & Biological Engineering & Computing, vol. 61, no. 10, pp. 2745–2755, 2023

    2023

  3. [9]

    Surgical-vqla: Transformer with gated vision-language embedding for visual question localized-answering in robotic surgery,

    L. Bai, M. Islam, L. Seenivasan, and H. Ren, “Surgical-vqla: Transformer with gated vision-language embedding for visual question localized-answering in robotic surgery,”arXiv preprint arXiv:2305.11692, 2023

    2023 arXiv

  4. [10]

    Global-reasoned multi-task learning model for surgical scene understanding,

    L. Seenivasan, S. Mitheran, M. Islam, and H. Ren, “Global-reasoned multi-task learning model for surgical scene understanding,”IEEE Robotics and Automation Letters, 2022

    2022

  5. [11]

    Sar-rarp50: Segmentation of surgical instrumentation and action recognition on robot-assisted radical prostatectomy challenge,

    D. Psychogyios, E. Colleoni, B. Van Amsterdam, C.-Y . Li, S.-Y . Huang, Y . Li, F. Jia, B. Zou, G. Wang, Y . Liuet al., “Sar-rarp50: Segmentation of surgical instrumentation and action recognition on robot-assisted radical prostatectomy challenge,”arXiv preprint arXiv:2401.00496, 2023

    2023 arXiv

  6. [12]

    Copesd: A multi-level surgical motion dataset for training large vision-language models to co-pilot endoscopic submucosal dissection,

    G. Wang, H. Xiao, H. Gao, R. Zhang, L. Bai, X. Yang, Z. Li, H. Li, and H. Ren, “Copesd: A multi-level surgical motion dataset for training large vision-language models to co-pilot endoscopic submucosal dissection,” arXiv preprint arXiv:2410.07540, 2024

    arXiv

  7. [13]

    Endoood: Uncertainty- aware out-of-distribution detection in capsule endoscopy diagnosis,

    Q. Tan, L. Bai, G. Wang, M. Islam, and H. Ren, “Endoood: Uncertainty- aware out-of-distribution detection in capsule endoscopy diagnosis,” arXiv preprint arXiv:2402.11476, 2024

    2024 arXiv

  8. [14]

    Ossar: Towards open-set surgical activity recognition in robot-assisted surgery,

    L. Bai, G. Wang, J. Wang, X. Yang, H. Gao, X. Liang, A. Wang, M. Islam, and H. Ren, “Ossar: Towards open-set surgical activity recognition in robot-assisted surgery,”arXiv preprint arXiv:2402.06985, 2024

    2024 arXiv

  9. [15]

    Artificial intelligence: A powerful paradigm for scientific research,

    Y . Xu, X. Liu, X. Cao, C. Huang, E. Liu, S. Qian, X. Liu, Y . Wu, F. Dong, C.-W. Qiuet al., “Artificial intelligence: A powerful paradigm for scientific research,”The Innovation, vol. 2, no. 4, 2021

    2021

  10. [16]

    Multimix: sparingly- supervised, extreme multitask learning from medical images,

    A. Haque, A. Wang, D. Terzopouloset al., “Multimix: sparingly- supervised, extreme multitask learning from medical images,” in2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021, pp. 693–696

    2021

  11. [17]

    Sam meets robotic surgery: an empirical study on generalization, robustness and adaptation,

    A. Wang, M. Islam, M. Xu, Y . Zhang, and H. Ren, “Sam meets robotic surgery: an empirical study on generalization, robustness and adaptation,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 234–244

    2023

  12. [18]

    Surgical-dino: adapter learning of foundation models for depth estimation in endoscopic surgery,

    B. Cui, M. Islam, L. Bai, and H. Ren, “Surgical-dino: adapter learning of foundation models for depth estimation in endoscopic surgery,” International Journal of Computer Assisted Radiology and Surgery, pp. 1–8, 2024

    2024

  13. [19]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Noubyet al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  14. [20]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  15. [21]

    Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,

    J. Ma, Z. Zhao, X. Yi, J. Chen, L. Hong, and E. H. Chi, “Modeling task relationships in multi-task learning with multi-gate mixture-of-experts,” inProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 1930–1939

    2018

  16. [22]

    Smooth tchebycheff scalarization for multi-objective optimization,

    X. Lin, X. Zhang, Z. Yang, F. Liu, Z. Wang, and Q. Zhang, “Smooth tchebycheff scalarization for multi-objective optimization,” arXiv preprint arXiv:2402.19078, 2024

    2024 arXiv

  17. [23]

    Multitask learning: A knowledge-based source of inductive bias1,

    R. Caruana, “Multitask learning: A knowledge-based source of inductive bias1,” inProceedings of the Tenth International Conference on Machine Learning. Citeseer, 1993, pp. 41–48

    1993

  18. [24]

    Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser,

    L. Duong, T. Cohn, S. Bird, and P. Cook, “Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser,” inProceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natur...

    2015

  19. [25]

    Learning multiple tasks with multilinear relationship networks,

    M. Long, Z. Cao, J. Wang, and P. S. Yu, “Learning multiple tasks with multilinear relationship networks,”Advances in neural information processing systems, vol. 30, 2017

    2017

  20. [26]

    Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification,

    Y . Lu, A. Kumar, S. Zhai, Y . Cheng, T. Javidi, and R. Feris, “Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5334–5343

    2017

  21. [27]

    Multi-task learning using uncer- tainty to weigh losses for scene geometry and semantics,

    A. Kendall, Y . Gal, and R. Cipolla, “Multi-task learning using uncer- tainty to weigh losses for scene geometry and semantics,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7482–7491

    2018

  22. [28]

    Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,

    Z. Chen, V . Badrinarayanan, C.-Y . Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” inInternational conference on machine learning. PMLR, 2018, pp. 794–803

    2018

  23. [29]

    Multi-task learning as multi-objective opti- mization,

    O. Sener and V . Koltun, “Multi-task learning as multi-objective opti- mization,”Advances in neural information processing systems, vol. 31, 2018

    2018

  24. [30]

    Cross-stitch net- works for multi-task learning,

    I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch net- works for multi-task learning,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3994–4003

    2016

  25. [31]

    End-to-end multi-task learning with attention,

    S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 1871–1880

    2019

  26. [32]

    Task adaptive parameter sharing for multi- task learning,

    M. Wallingford, H. Li, A. Achille, A. Ravichandran, C. Fowlkes, R. Bhotika, and S. Soatto, “Task adaptive parameter sharing for multi- task learning,” inProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2022, pp. 7561–7570

    2022

  27. [33]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,”arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  28. [34]

    M 3vit: Mixture-of-experts vision transformer for efficient multi- task learning with model-accelerator co-design,

    Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y . Cheng, C. Hao, Z. Wang et al., “M 3vit: Mixture-of-experts vision transformer for efficient multi- task learning with model-accelerator co-design,”Advances in Neural Information Processing Systems, vol. 35, pp. 28 441–28 457, 2022

    2022

  29. [35]

    Heterogeneous multi-task learn- ing with expert diversity,

    R. Aoki, F. Tung, and G. L. Oliveira, “Heterogeneous multi-task learn- ing with expert diversity,”IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 19, no. 6, pp. 3093–3102, 2022

    2022

  30. [36]

    Unit: Multimodal multitask learning with a unified transformer,

    R. Hu and A. Singh, “Unit: Multimodal multitask learning with a unified transformer,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 1439–1449

    2021

  31. [37]

    Taskprompter: Spatial-channel multi-task prompting for dense scene understanding,

    H. Ye and D. Xu, “Taskprompter: Spatial-channel multi-task prompting for dense scene understanding,” inThe Eleventh International Confer- ence on Learning Representations, 2022

    2022

  32. [38]

    Orthogonal transfer for multitask optimization,

    S.-H. Wu, Z.-H. Zhan, K. C. Tan, and J. Zhang, “Orthogonal transfer for multitask optimization,”IEEE Transactions on Evolutionary Com- putation, vol. 27, no. 1, pp. 185–200, 2022

    2022

  33. [39]

    Lymph node gross tumor volume detection and segmentation via distance-based gating using 3d ct/pet imaging in radiotherapy,

    Z. Zhu, D. Jin, K. Yan, T.-Y . Ho, X. Ye, D. Guo, C.-H. Chao, J. Xiao, A. Yuille, and L. Lu, “Lymph node gross tumor volume detection and segmentation via distance-based gating using 3d ct/pet imaging in radiotherapy,” inInternational Conference on Medical Image Computing and ...

    2020

  34. [40]

    Assisted probe positioning for ultrasound guided radiotherapy using image sequence classification,

    A. Grimwood, H. McNair, Y . Hu, E. Bonmati, D. Barratt, and E. J. Har- ris, “Assisted probe positioning for ultrasound guided radiotherapy using image sequence classification,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, ...

    2020

  35. [41]

    Multi-task temporal convo- lutional networks for joint recognition of surgical phases and steps in gastric bypass procedures,

    S. Ramesh, D. Dall’Alba, C. Gonzalez, T. Yu, P. Mascagni, D. Mutter, J. Marescaux, P. Fiorini, and N. Padoy, “Multi-task temporal convo- lutional networks for joint recognition of surgical phases and steps in gastric bypass procedures,”International journal of computer assiste...

    2021

  36. [42]

    Multi-task learning for registering images with large deformation,

    B. Du, J. Liao, B. Turkbey, and P. Yan, “Multi-task learning for registering images with large deformation,”IEEE journal of biomedical and health informatics, vol. 25, no. 5, pp. 1624–1633, 2020. 14

    2020

  37. [43]

    Less is more: Simultaneous view classification and land- mark detection for abdominal ultrasound images,

    Z. Xu, Y . Huo, J. Park, B. Landman, A. Milkowski, S. Grbic, and S. Zhou, “Less is more: Simultaneous view classification and land- mark detection for abdominal ultrasound images,” inMedical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Con...

    2018

  38. [44]

    Prediction of prognostic risk factors in hepatocellular carcinoma with transarterial chemoembolization using multi-modal multi-task deep learning,

    Q.-P. Liu, X. Xu, F.-P. Zhu, Y .-D. Zhang, and X.-S. Liu, “Prediction of prognostic risk factors in hepatocellular carcinoma with transarterial chemoembolization using multi-modal multi-task deep learning,”EClin- icalMedicine, vol. 23, 2020

    2020

  39. [45]

    J. Yao, Y . Shi, K. Cao, L. Lu, J. Lu, Q. Song, G. Jin, J. Xiao, Y . Hou, and L. Zhang, “Deepprognosis: Preoperative prediction of pancreatic cancer survival and surgical margin via comprehensive understanding of dynamic contrast-enhanced ct imaging and tumor-vascular contact ...

    2021

  40. [46]

    Task- aware asynchronous multi-task model with class incremental contrastive learning for surgical scene understanding,

    L. Seenivasan, M. Islam, M. Xu, C. M. Lim, and H. Ren, “Task- aware asynchronous multi-task model with class incremental contrastive learning for surgical scene understanding,”International Journal of Computer Assisted Radiology and Surgery, vol. 18, no. 5, pp. 921–928, 2023

    2023

  41. [47]

    3d multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification,

    Y . Zhang, H. Li, J. Du, J. Qin, T. Wang, Y . Chen, B. Liu, W. Gao, G. Ma, and B. Lei, “3d multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification,” IEEE transactions on medical imaging, vol. 40, no. 6, pp. 1618...

    2021

  42. [48]

    An end-to-end multi- task deep learning framework for skin lesion analysis,

    L. Song, J. Lin, Z. J. Wang, and H. Wang, “An end-to-end multi- task deep learning framework for skin lesion analysis,”IEEE journal of biomedical and health informatics, vol. 24, no. 10, pp. 2912–2921, 2020

    2020

  43. [49]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inProceedings of naacL-HLT, vol. 1. Minneapolis, Minnesota, 2019, p. 2

    2019

  44. [50]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  45. [51]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026

    2023

  46. [52]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafsonet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  47. [53]

    Fedfmsl: Federated learning of foundations models with sparsely activated lora,

    P. Wu, K. Li, T. Wang, Y . Dong, V . C. Leung, and F. Wang, “Fedfmsl: Federated learning of foundations models with sparsely activated lora,” IEEE Transactions on Mobile Computing, 2024

    2024

  48. [54]

    Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery,

    G. Wang, L. Bai, W. J. Nah, J. Wang, Z. Zhang, Z. Chen, J. Wu, M. Islam, H. Liu, and H. Ren, “Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery,”arXiv preprint arXiv:2405.10948, 2024

    2024 arXiv

  49. [55]

    Robust federated finetuning of foundation models via alternating minimization of lora,

    S. Chen, Y . Ju, H. Dalal, Z. Zhu, and A. Khisti, “Robust federated finetuning of foundation models via alternating minimization of lora,” arXiv preprint arXiv:2409.02346, 2024

    2024 arXiv

  50. [56]

    Rapid adaptation of earth observation foundation models for segmentation,

    K. P. Selvam, R. Ramos-Pollan, and F. Kalaitzis, “Rapid adaptation of earth observation foundation models for segmentation,”arXiv preprint arXiv:2409.09907, 2024

    2024 arXiv

  51. [57]

    One initialization to rule them all: Fine-tuning via explained variance adaptation,

    F. Paischer, L. Hauzenberger, T. Schmied, B. Alkin, M. P. Deisenroth, and S. Hochreiter, “One initialization to rule them all: Fine-tuning via explained variance adaptation,”arXiv preprint arXiv:2410.07170, 2024

    2024

  52. [58]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,”IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834–848, 2017

    2017

  53. [59]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  54. [60]

    Mobilevit: light-weight, general- purpose, and mobile-friendly vision transformer,

    S. Mehta and M. Rastegari, “Mobilevit: light-weight, general- purpose, and mobile-friendly vision transformer,”arXiv preprint arXiv:2110.02178, 2021

    2021 arXiv

  55. [62]

    Customized segment anything model for medical image segmentation,

    K. Zhang and D. Liu, “Customized segment anything model for medical image segmentation,”arXiv preprint arXiv:2304.13785, 2023

    2023 arXiv

  56. [64]

    Transendoscopic flexible parallel continuum robotic mechanism for bimanual endoscopic sub- mucosal dissection,

    H. Gao, X. Yang, X. Xiao, X. Zhu, T. Zhang, C. Hou, H. Liu, M. Q.-H. Meng, L. Sun, X. Zuoet al., “Transendoscopic flexible parallel continuum robotic mechanism for bimanual endoscopic sub- mucosal dissection,”The International Journal of Robotics Research, p. 02783649231209338, 2023

    2023

  57. [65]

    A novel miniature transendoscopic telerobotic system for endoscopic submucosal dissection,

    X. Yang, H. Gao, S. Fu, R. Ji, C. Hou, H. Liu, N. Luan, H. Ren, L. Sun, J. Yanget al., “A novel miniature transendoscopic telerobotic system for endoscopic submucosal dissection,”Gastrointestinal Endoscopy, 2023

    2023

  58. [66]

    Surgical-vqla++: Adversarial contrastive learning for calibrated robust visual question-localized answering in robotic surgery,

    L. Bai, G. Wang, M. Islam, L. Seenivasan, A. Wang, and H. Ren, “Surgical-vqla++: Adversarial contrastive learning for calibrated robust visual question-localized answering in robotic surgery,”Information Fusion, vol. 113, p. 102602, 2025

    2025

  59. [67]

    2017 robotic instrument segmentation challenge,

    M. Allan, A. Shvets, T. Kurmann, Z. Zhang, R. Duggal, Y .-H. Su, N. Rieke, I. Laina, N. Kalavakondaet al., “2017 robotic instrument segmentation challenge,”arXiv preprint arXiv:1902.06426, 2019

    2017 arXiv

  60. [68]

    Libmtl: A python library for multi-task learning,

    B. Lin and Y . Zhang, “Libmtl: A python library for multi-task learning,” arXiv preprint arXiv:2203.14338, 2022

    2022 arXiv

  61. [69]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  62. [70]

    Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning,

    H. Hazimeh, Z. Zhao, A. Chowdhery, M. Sathiamoorthy, Y . Chen, R. Mazumder, L. Hong, and E. Chi, “Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning,”Ad- vances in Neural Information Processing Systems, vol. 34, pp. 29 335– 2...

    2021

  63. [71]

    Mitigating gradient bias in multi-objective learning: A prov- ably convergent approach

    H. Fernando, H. Shen, M. Liu, S. Chaudhury, K. Murugesan, and T. Chen, “Mitigating gradient bias in multi-objective learning: A prov- ably convergent approach.” International Conference on Learning Representations, 2023

    2023

  64. [72]

    Independent component alignment for multi-task learning,

    D. Senushkin, N. Patakin, A. Kuznetsov, and A. Konushin, “Independent component alignment for multi-task learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 083–20 093

    2023

  65. [73]

    Dual-balancing for multi-task learning,

    B. Lin, W. Jiang, F. Ye, Y . Zhang, P. Chen, Y .-C. Chen, S. Liu, and J. Kwok, “Dual-balancing for multi-task learning,” 2023

    2023

  66. [74]

    Rethinking atrous convolution for semantic image segmentation,

    L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,”arXiv preprint arXiv:1706.05587, 2017

    2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.