Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation

Binod Bhattarai; Krit Duangprom; Tryphon Lambrou

REVIEW 4 major objections 5 minor 25 references

Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read With two epochs of LoRA fine-tuning, Qwen2.5-VL-3B predicts surgical-tool keypoints at MPJPE 0.0627 and PCK@0.10 0.8908, beating YOLOv8-Pose and SurgeoNet.

desk verdict A plausible transfer of the LoRA-VLM keypoint recipe to surgical tools, but the headline 'outperforms' claim is built on a single test-set-selected run and a 0.0024 MPJPE gap. read the letter →

arxiv 2508.20830 v1 pith:7A6FLWGL submitted 2025-08-28 cs.CV

Krit Duangprom , Tryphon Lambrou , Binod Bhattarai This is my paper

classification cs.CV

keywords 2Dkeypointestimationvision-languagemodelslow-rankadaptation(LoRA)surgicaltoolposevisualquestionansweringSurgeoNetprompt-basedfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper turns 2D keypoint detection for surgical tools into a visual question-answering task: given an image and a fixed prompt, a vision-language model outputs the tool name and the coordinates of 12 keypoints as text. It shows that after only two epochs of low-rank (LoRA) fine-tuning, Qwen2.5-VL-3B reaches a mean per-joint position error of 0.0627 and a PCK@0.10 of 0.8908 on the SurgeoNet benchmark, slightly better than fully trained YOLOv8-Pose and SurgeoNet baselines while updating only about 20.7M parameters. The result matters because it suggests that a frozen pretrained VLM, rather than a task-specific architecture trained from scratch, can carry most of the localization burden in a small medical-image dataset. The paper frames this as evidence that VLM-based pipelines are a viable low-resource route to surgical tool analysis and a stepping stone toward 3D hand-tool pose estimation.

What carries the argument

The load-bearing mechanism is the pairing of a frozen pretrained vision-language model with Low-Rank Adaptation (LoRA): only two small matrices per transformer attention projection are trained, so the model keeps its pretrained visual-semantic knowledge while learning the surgical keypoint vocabulary. On top of this sits the prompt-based visual question-answering formulation, which converts 12 numeric (x,y) coordinates into a token sequence; the standard causal language-modeling loss then teaches the model to produce tool names and coordinates jointly. The LoRA rank controls how much task-specific capacity is injected, and the paper identifies rank 8 (with scaling factor 16) as the point whe

What would settle it

Fine-tune the same model with random visual weights (same architecture, no pretraining) under the identical LoRA recipe; if MPJPE stays near 0.0627, the localization is coming from the training loss rather than the pretrained representation, and if it jumps toward the no-fine-tune level (~0.43), pretraining is essential. A second check: evaluate on an instrument type held out from the 14 SurgeoNet classes—if PCK collapses, the adapters are memorizing tools instead of generalizing keypoint structure.

Watch

Extended reading notes

Core claim

The central claim is that vision-language models, lightly adapted with LoRA, can perform structured spatial localization—estimating 12 semantic 2D keypoints across 14 surgical instrument types—at a level comparable to or better than dedicated CNN and transformer detectors. The paper demonstrates this by reformulating keypoint detection as prompt-conditioned text generation: the model receives an RGB image plus a fixed prompt asking for the tool name and 12 keypoints, and is trained with a standard causal language-modeling loss to output the coordinates token by token. On the SurgeoNet test set, the LoRA fine-tuned Qwen2.5-VL-3B achieves MPJPE 0.0627 and PCK@0.10 0.8908, improving on YOLOv8-P

Load-bearing premise

The frozen VLM already holds enough visual and spatial knowledge that a rank-8 LoRA update, trained for just two epochs on text-formatted coordinates, can convert that knowledge into accurate, consistent surgical-tool keypoints.

Editorial extensions

If this is right

If the result holds, small annotated medical datasets no longer require training a large detector from scratch; a frozen VLM plus a two-epoch, low-rank update can reach keypoint accuracy competitive with specialized models.
Because the same prompt-and-coordinate format is used for every tool type, extending the pipeline to new instruments or to hand-tool joint keypoints is largely a matter of enlarging the instruction dataset.
The rank ablation implies that under a fixed short schedule, the adapter capacity—not the base model alone—controls whether the localization task is learned; rank 8 is a stable operating point for similar dense spatial prediction tasks.
Fine-tuned VLMs remain competitive even when trained for far fewer epochs than the CNN/transformer baselines, suggesting the main cost shifts from training compute to prompt and data construction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not isolate where the localization signal comes from; a natural test would be to freeze the visual encoder and train only the language-side LoRA head, or to swap in a randomly initialized visual backbone, to see whether the pretrained visual features are actually the source of the gain.
Because the un-fine-tuned models score near zero PCK, the two epochs of LoRA are evidently creating the keypoint mapping rather than merely 'unlocking' an existing one; the paper's framing that pretraining 'harnesses generalization' is therefore an interpretation, not yet demonstrated by ablation.
The text-coordinate output may impose a precision ceiling tied to the tokenizer's numeric representation; a head-to-head against a same-backbone regression or heatmap head would show whether VLM tokenization is the limiting factor or a free lunch.
A sharper practical test is cross-domain evaluation on the SurgeoNet split: fine-tune on synthetic images and evaluate on real captures (or held-out instrument types) to see whether the VLM's pretraining actually narrows the synthetic-to-real gap, which the paper's aggregate numbers do not reveal.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a clean, straightforward empirical study showing that LoRA-finetuning a VLM (Qwen2.5-VL-3B) for two epochs gets MPJPE 0.0627 on SurgeoNet, close to YOLOv8-Pose and SurgeoNet. The new bit is the domain transfer to surgical tools and the comparison of two VLMs (Qwen and DeepSeek-VL2), plus a small rank ablation. The experiment is honest in design: public benchmark, standard metrics, sensible baselines, trainable parameter counts. They also clearly show that without fine-tuning the VLMs do essentially nothing (MPJPE ~0.43, PCK near zero), which is a useful sanity check.

The soft spots are real and they matter. The abstract says the adapted VLM 'outperforms the baseline models,' but Table 1's best number comes from a single run at LoRA rank 8, and Table 2 shows rank 8 was chosen by an ablation on the test set. No seed variation, no error bars, no repeated runs. The gap over SurgeoNet is 0.0024 MPJPE; over YOLOv8-Pose it's 0.0045. For a 3B-parameter model trained with a stochastic LM objective for two epochs, that margin is well inside plausible run-to-run variance. So the 'outperforms' wording is overclaiming; the defensible claim is 'comparable to task-specific detectors.'

Two more omissions. The closely related KPT-LLM and CapE-LLM are cited but never compared, even though they use the same recipe on animal joints. And the paper doesn't specify the coordinate normalization used to serialize keypoints into text or the parsing rule for images with more than one tool, which makes the exact numbers non-reproducible from the text. No code or data either.

None of that sinks the core result. The study is small but coherent, and the research question is fair: can a frozen VLM with a low-rank adapter do structured spatial output on a low-resource medical domain? The answer here is plausibly yes, with a modest caveat. Who benefits: surgical-vision researchers who want a quick baseline for keypoint tasks without training a detector, and anyone thinking about VLM-based structured prediction. It deserves a serious referee; the review should push for repeated runs, a validation split for rank selection, and code, but not a desk reject.

My recommendation: engage with it as a tentative empirical result, and don't cite the 'outperforms' claim without verifying the margin holds under repeated runs.

Referee Report

4 major / 5 minor

Summary. The paper proposes a VLM-based pipeline for 2D keypoint estimation of surgical tools. The task is reformulated as visual question answering: given an image and the prompt “What is/are this/these tool(s) and find 12 keypoints?”, a VLM fine-tuned with LoRA outputs the tool name and 12 numeric (x,y) keypoints. Experiments on the SurgeoNet dataset compare Qwen2.5-VL-3B and DeepSeek-VL2-tiny with LoRA against YOLOv8-Pose and SurgeoNet. The main empirical claim is that after only two epochs of LoRA fine-tuning, Qwen2.5-VL-3B achieves MPJPE 0.0627 and PCK@0.10 0.8908, outperforming YOLOv8-Pose (0.0672, 0.8466) and SurgeoNet (0.0651, 0.8519). A rank ablation on the same test set selects rank 8 as the best configuration. The discussion concludes that VLMs with lightweight adaptation can be competitive for structured localization tasks.

Significance. If the empirical claim is robust, the result is noteworthy: it would show that a frozen pretrained VLM can be adapted with a small number of trainable parameters (20.7M of 3.78B) to perform dense, structured 2D keypoint localization at a level comparable to dedicated CNN/Transformer pose estimators, using only a token-level language modeling loss and no heatmap or regression head. The paper also provides a useful comparison of two VLM families and a LoRA-rank ablation. The manuscript’s strengths include evaluation on a public dataset, comparison to strong baselines, and a simple, conceptually clear pipeline. However, the central “outperforms” claim currently rests on a single unseeded run with a hyperparameter chosen on the test set, so the stated margins cannot be distinguished from run-to-run noise.

major comments (4)

[Table 2 and Table 1] The headline improvement of Qwen2.5-VL-3B over SurgeoNet is 0.0024 MPJPE (0.0627 vs 0.0651) and over YOLOv8-Pose is 0.0045. Rank 8 was selected by an ablation on the same test set used for the final evaluation, and no validation split or repeated runs are reported. With a 3B-parameter model trained for two epochs with a stochastic objective, these margins are within plausible run-to-run variance. The paper should either report mean±std over multiple seeds, or choose LoRA rank on a validation split and evaluate only once on the test set. Without this, the specific “outperforms” claim in the abstract is not secure.
[Section 4 (Evaluation Metrics and Dataset)] The manuscript does not specify how keypoint coordinates are normalized or serialized into text, nor how images containing more than one tool are handled. The prompt asks about “this/these tool(s)” and the dataset contains “one or more surgical tools,” but the evaluation pipeline is not described for multi-tool images. The normalization factor L in Eq. (4) and the image resolution/coordinate scaling used in Eq. (3) are also unspecified. These details are needed for independent reproduction of the reported numbers; without them the exact experimental protocol is ambiguous.
[Section 4 (Experimental Setup) and Section 3 (Eq. 2)] The paper reports only “two epochs” for the VLM and “more than 100 epochs” for baselines, but omits essential training details: optimizer, learning rate, batch size, sequence length, LoRA target modules (beyond “all attention projections and optionally feed-forward”), and whether the language modeling loss is the only objective. Since the central empirical result depends on the LoRA-fine-tuned model’s convergence behavior, the lack of these details, together with the absence of any seed or variance analysis, makes the result difficult to evaluate or reproduce.
[Abstract and Section 5 (Discussion)] The abstract states that “the adapted VLM outperforms the baseline models,” but Table 1 shows that DeepSeek-VL2 (fine-tuned) does not outperform YOLOv8-Pose or SurgeoNet on PCK and has higher MPJPE than SurgeoNet. Only the Qwen model outperforms both baselines on all metrics. The conclusion in Section 5 more cautiously says the method “reached a comparable level,” which is inconsistent with the abstract. The claim should be scoped to the specific model and metrics, or the abstract should be revised.

minor comments (5)

[Title/Keywords] Typos and formatting issues: “T able” in Table 1, “perforamnce” in Section 4, and inconsistent whitespace around equations. Please proofread.
[Conclusions] The conclusion states “We designed tailored prompts and reconstructed a synthetic dataset,” but the experiments section says the authors used the SurgeoNet dataset. Clarify whether a synthetic dataset was reconstructed and, if so, how it differs from the original SurgeoNet data.
[Figure 1 and Figure 2] The figures are referenced in the text but not described in enough detail. Figure 2 especially is a generic LoRA diagram; consider removing or making it concrete to the architecture used.
[References] Some references have formatting issues (e.g., Ref. 16 “arXiv preprint arXiv 2411.06869” is missing a colon/dot). Also, since the paper compares against SurgeoNet, more detail on the original SurgeoNet protocol (stereo vs monocular) would help contextualize the “monocular camera” qualification in Table 1.
[Section 4 (Ablation Study)] The ablation claims rank 16 “may lead to suboptimal convergence or mild overfitting,” but no training curves or convergence analysis are provided. Either support this claim or soften it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: predictions are supervised outputs trained and tested on the public SurgeoNet data against external baselines.

full rationale

The paper's derivation chain is straightforward supervised learning: the VLM is fine-tuned with LoRA on SurgeoNet training images using a causal language-modeling loss over serialized keypoint coordinates, and is then evaluated on the SurgeoNet test set against external baselines (YOLOv8-Pose, SurgeoNet). No equation in the paper defines a predicted quantity in terms of a fitted constant, no claimed 'prediction' is derived from the same data used to fit it in a way that forces the result, and no load-bearing argument reduces to a self-citation. The only notable methodological weakness is that the LoRA rank ablation in Table 2 uses the test set to select rank 8, which is evaluation bias and may make the reported 'outperforms' claim less secure, but it is not circularity: the final MPJPE/PCK values are still genuine outputs of a trained model on held-out test images. The self-citation in related work ([25], a survey co-authored by one of the present authors) is not load-bearing. Therefore no circular step can be quoted, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical transfer-learning study with no derivation. The central claim depends on SurgeoNet annotation quality, the assumption that a frozen VLM's spatial prior can be unlocked by a rank-8 LoRA update, and hyperparameter choices (rank, alpha, epochs) that were partly selected via test-set ablation.

free parameters (4)

LoRA rank r = 8
Selected from {4, 8, 16} based on best test-set MPJPE/PCK in Table 2; this same rank is used for the main results in Table 1.
LoRA scaling factor alpha = 16
Fixed by hand in Section 3; no ablation reported.
LoRA dropout = 0.05
Fixed by hand in Section 3; no ablation reported.
Fine-tuning epochs = 2
Chosen as part of the low-resource recipe; no convergence analysis or sensitivity study is reported.

assumptions (4)

domain assumption SurgeoNet ground-truth 12 keypoints correctly label the same semantic tool parts across all 14 instrument types and train/test splits.
The entire evaluation depends on these annotations; the dataset description is in Section 4.
domain assumption Causal language modeling loss over generated coordinate tokens provides a valid training signal for spatial localization.
Equation (1) in Section 3 is the only learning objective; the paper gives no analysis of coordinate tokenization, parsing, or sensitivity to text format.
domain assumption Pretrained VLM representations transfer to surgical instruments after low-rank adaptation.
This is the central hypothesis; Table 1 shows without fine-tuning performance is near zero, so all capability is assumed to be unlocked by LoRA in two epochs.
domain assumption Baseline comparisons use fair, fixed evaluation code and comparable data preprocessing.
The paper reports baselines trained for more than 100 epochs but gives no details on how they were trained or evaluated for this table.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation." pith.science (2026). https://pith.science/paper/7A6FLWGL

@misc{pith2026250820830,
  author       = {Pith},
  title        = {Pith review of: Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7A6FLWGL}},
  note         = {Machine review of arXiv:2508.20830}
}

read the original abstract

This paper presents a novel pipeline for 2D keypoint estima- tion of surgical tools by leveraging Vision Language Models (VLMs) fine- tuned using a low rank adjusting (LoRA) technique. Unlike traditional Convolutional Neural Network (CNN) or Transformer-based approaches, which often suffer from overfitting in small-scale medical datasets, our method harnesses the generalization capabilities of pre-trained VLMs. We carefully design prompts to create an instruction-tuning dataset and use them to align visual features with semantic keypoint descriptions. Experimental results show that with only two epochs of fine tuning, the adapted VLM outperforms the baseline models, demonstrating the ef- fectiveness of LoRA in low-resource scenarios. This approach not only improves keypoint detection performance, but also paves the way for future work in 3D surgical hands and tools pose estimation.

Figures

Figures reproduced from arXiv: 2508.20830 by the authors.

**Figure 1.** Overview of the proposed VLM-based keypoint estimation pipeline. The model receives an image and a prompt, and outputs the tool name along with its 12 semantic keypoints. After generating the output, the predicted keypoints are parsed and compared with the corresponding ground-truth annotations from the training dataset. The learning objective is to minimize the difference between the predicted and true output sequ… view at source ↗

**Figure 2.** Visualization of Low-Rank Adaptation (LoRA): A frozen weight matrix is augmented with a low-rank update via two trainable matrices A and B. 4 Experiments Dataset: We performed our experiments SurgeoNet dataset [8] which has 13,649 training and 1,763 test RGB images covering 14 instrument types. Each image contains one or more surgical tools annotated with 12 semantic 2D keypoints. Example images are shown in [PITH… view at source ↗

**Figure 3.** Examples from the SurgeoNet dataset: synthetic images (left) and real-world captures (right). Experimental Results We compare our proposed pipeline with strong baselines for 2D keypoint estimation, including YOLOv8-Pose [19], a CNN-based with 26.4M parameters, and SurgeoNet [8], which adds a ViT backbone (1M extra parameters) on top of YOLOv8m. The baselines were fully trained for more than 100 epochs. In contrast… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Keypoint predictions before (left) and after (right) LoRA fine-tuning. Example shows real surgical images with annotated tool keypoints [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Comparison keypoints prediction of models in synthetic test data 5 Discussion Our results demonstrate that Vision-Language Models (VLMs) can perform 2D keypoint estimation at a level comparable to CNN or Transformer-based vision models. As shown in [PITH_FULL_IMAGE:fi…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 19 canonical work pages

[1]

arXiv preprint arXiv:2502.11534 (2025)

Wu, Z., Schmidt, A., Moore, R., Zhou, H., Banks, A., Kazanzides, P., Salcudean, S.E.: SurgPose: A Dataset for Articulated Robotic Surgical Tool Pose Estimation and Tracking. arXiv preprint arXiv:2502.11534 (2025)

arXiv 2025
[2]

Engineering Applications of Artificial Intelligence, 153, 110893

Hamza,H.,Shabir,D.,Aboumarzouk,O.,Al-Ansari,A.,Shaban,K.,Navkar,N.V.: Automated Skills Assessment in Open Surgery: A Scoping Review. Engineering Applications of Artificial Intelligence, 153, 110893. Elsevier (2025)

work page 2025
[3]

International Journal of Computer Assisted Radiology and Surgery, 18(7), 1279–1285

Bkheet, E., D’Angelo, A.-L., Goldbraikh, A., Laufer, S.: Using Hand Pose Esti- mation to Automate Open Surgery Training Feedback. International Journal of Computer Assisted Radiology and Surgery, 18(7), 1279–1285. Springer (2023)

work page 2023
[4]

Medical Image Analysis, p

Hein, J., Cavalcanti, N., Suter, D., Zingg, L., Carrillo, F., Calvet, L., Farshad, M., Navab, N., Pollefeys, M., Fürnstahl, P.: Next-Generation Surgical Navigation: Marker-less Multi-View 6DoF Pose Estimation of Surgical Instruments. Medical Image Analysis, p. 103613. Elsevier (2025)

work page 2025
[5]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Sun, K., Xiao, B., Liu, D., Wang, J.: Deep High-Resolution Representation Learn- ing for Human Pose Estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019) 10 K. Duangprom et al

work page 2019
[6]

arXiv preprint arXiv:2407.12138 (2024)

Spektor, R., Friedman, T., Or, I., Bolotin, G., Laufer, S.: Monocular Pose Es- timation of Articulated Surgical Instruments in Open Surgery. arXiv preprint arXiv:2407.12138 (2024)

work page arXiv 2024
[7]

In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Maji, D., Nagori, S., Mathew, M., Poddar, D.: YOLO-Pose: Enhancing YOLO for Multi-Person Pose Estimation Using Object Keypoint Similarity Loss. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2637–2646 (2022)

work page 2022
[8]

In: DAGM German Con- ference on Pattern Recognition, pp

Aboukhadra, A.T., Robertini, N., Malik, J., Elhayek, A., Reis, G., Stricker, D.: SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images Using a Synthetically-Trained Network. In: DAGM German Con- ference on Pattern Recognition, pp. 199–211. Springer (2024)

work page 2024

Show all 25 references

[9]

In: International Conference on Machine Learning, pp

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning Transferable Visual Models from Natural Language Supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)

2021
[10]

arXiv preprint arXiv:2502.13923 (2025)

Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923 (2025)

2025 arXiv
[11]

In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

Subramanian, S., Ng, E., Müller, L., Klein, D., Ginosar, S., Darrell, T.: Pose Pri- ors from Language Models. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7125–7135 (2025)

2025
[12]

In: 2024 WRC Symposium on Advanced Robotics and Automation (WRC SARA), pp

Huang, J., Limberg, C., Arshad, S.M.N., Zhang, Q., Li, Q.: Combining VLM and LLM for Enhanced Semantic Object Perception in Robotic Handover Tasks. In: 2024 WRC Symposium on Advanced Robotics and Automation (WRC SARA), pp. 135–140. IEEE (2024)

2024
[13]

arXiv preprint arXiv:2412.06292 (2024)

Gong, B., Gomez, D., Hamdi, A., Eldesokey, A., Abdelreheem, A., Wonka, P., Ovsjanikov, M.: ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint De- tection from Large Language Models. arXiv preprint arXiv:2412.06292 (2024)

2024 arXiv
[14]

IEEE Transactions on Circuits and Systems for Video Technology (2024)

Lin, X., Zhu, M., Dang, R., Zhou, G., Shu, S., Lin, F., Liu, C., Chen, Q.: CLIPose: Category-Level Object Pose Estimation with Pre-Trained Vision-Language Knowl- edge. IEEE Transactions on Circuits and Systems for Video Technology (2024)

2024
[15]

arXiv preprint arXiv:2411.01846 (2024)

Yang, J., Zeng, W., Jin, S., Xu, L., Liu, W., Qian, C., Zhang, R.: KPT-LLM: Unveiling the Power of Large Language Model for Keypoint Comprehension. arXiv preprint arXiv:2411.01846 (2024)

2024 arXiv
[16]

CapE-LLM Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Models

Kim, J., Chung, H., Kim, B.H. CapE-LLM Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Models. arXiv preprint arXiv 2411.06869 (2024)

2024 arXiv
[17]

In: International Conference on Learning Representations (ICLR) (2022)

Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: LoRA: Low-Rank Adaptation of Large Language Models. In: International Conference on Learning Representations (ICLR) (2022)

2022
[18]

arXiv preprint arXiv:2412.10302 (2024)

Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al.: DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding. arXiv preprint arXiv:2412.10302 (2024)

2024 arXiv
[19]

Jocher, G., Chaurasia, A., Qiu, J.Ultralytics YOLOv8, Version 8.0.0. (2023)

2023
[20]

In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

Zhao, Y., Huang, J., Hu, J., Wang, X., Mao, Y., Zhang, D., Jiang, Z., Wu, Z., Ai, B., Wang, A., et al.: SWIFT: A Scalable Lightweight Infrastructure for Fine- Tuning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 28, pp. 29733–29735 (2025)

2025
[21]

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.:An Image is Estimating 2D Keypoints of Surgical Tools Using VLM with LoRA 11 Worth 16x16 Words: Transformers for Image Recognition...

2010 arXiv
[22]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp

Wang, R., Ktistakis, S., Zhang, S., Meboldt, M., Lohmeyer, Q.: POV-Surgery: A Dataset for Egocentric Hand and Tool Pose Estimation During Surgical Activities. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 440–450. Springer (2023)

2023
[23]

In: Advances in Neural Information Processing Systems, vol

Dong, H., Chharia, A., Gou, W., Vicente Carrasco, F., De la Torre, F.D.: HAMBA: Single-View 3D Hand Reconstruction with Graph-Guided Bi-Scanning Mamba. In: Advances in Neural Information Processing Systems, vol. 37, pp. 2127–2160 (2024)

2024
[24]

In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

Qi, H., Zhao, C., Salzmann, M., Mathis, A.: HOISDF: Constraining 3D Hand- Object Pose Estimation with Global Signed Distance Fields. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10392– 10402. IEEE (2024)

2024
[25]

A., Bhattarai, B

Shrestha, P., Amgain, S., Khanal, B., Linte, C. A., Bhattarai, B. (2023). Medical vision language pretraining: A survey. arXiv preprint arXiv:2312.06224

2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:2502.11534 (2025)

Wu, Z., Schmidt, A., Moore, R., Zhou, H., Banks, A., Kazanzides, P., Salcudean, S.E.: SurgPose: A Dataset for Articulated Robotic Surgical Tool Pose Estimation and Tracking. arXiv preprint arXiv:2502.11534 (2025)

arXiv 2025

[2] [2]

Engineering Applications of Artificial Intelligence, 153, 110893

Hamza,H.,Shabir,D.,Aboumarzouk,O.,Al-Ansari,A.,Shaban,K.,Navkar,N.V.: Automated Skills Assessment in Open Surgery: A Scoping Review. Engineering Applications of Artificial Intelligence, 153, 110893. Elsevier (2025)

work page 2025

[3] [3]

International Journal of Computer Assisted Radiology and Surgery, 18(7), 1279–1285

Bkheet, E., D’Angelo, A.-L., Goldbraikh, A., Laufer, S.: Using Hand Pose Esti- mation to Automate Open Surgery Training Feedback. International Journal of Computer Assisted Radiology and Surgery, 18(7), 1279–1285. Springer (2023)

work page 2023

[4] [4]

Medical Image Analysis, p

Hein, J., Cavalcanti, N., Suter, D., Zingg, L., Carrillo, F., Calvet, L., Farshad, M., Navab, N., Pollefeys, M., Fürnstahl, P.: Next-Generation Surgical Navigation: Marker-less Multi-View 6DoF Pose Estimation of Surgical Instruments. Medical Image Analysis, p. 103613. Elsevier (2025)

work page 2025

[5] [5]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Sun, K., Xiao, B., Liu, D., Wang, J.: Deep High-Resolution Representation Learn- ing for Human Pose Estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019) 10 K. Duangprom et al

work page 2019

[6] [6]

arXiv preprint arXiv:2407.12138 (2024)

Spektor, R., Friedman, T., Or, I., Bolotin, G., Laufer, S.: Monocular Pose Es- timation of Articulated Surgical Instruments in Open Surgery. arXiv preprint arXiv:2407.12138 (2024)

work page arXiv 2024

[7] [7]

In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Maji, D., Nagori, S., Mathew, M., Poddar, D.: YOLO-Pose: Enhancing YOLO for Multi-Person Pose Estimation Using Object Keypoint Similarity Loss. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2637–2646 (2022)

work page 2022

[8] [8]

In: DAGM German Con- ference on Pattern Recognition, pp

Aboukhadra, A.T., Robertini, N., Malik, J., Elhayek, A., Reis, G., Stricker, D.: SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images Using a Synthetically-Trained Network. In: DAGM German Con- ference on Pattern Recognition, pp. 199–211. Springer (2024)

work page 2024