Pith. sign in

REVIEW 9 cited by

LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.07981 v1 pith:WVHAAXEG submitted 2024-08-15 cs.CV cs.AI

Jiajie Li , Garrett Skinner , Gene Yang , Brian R Quaranto , Steven D Schwaitzberg , Peter C W Kim , Jinjun Xiong This is my paper · ORCID
classification cs.CVcs.AI
keywords surgicalvideosdatasetmultimodalgenerationllava-surgmodelsanswering
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Multimodal large language models (LLMs) have achieved notable success across various domains, while research in the medical field has largely focused on unimodal images. Meanwhile, current general-domain multimodal models for videos still lack the capabilities to understand and engage in conversations about surgical videos. One major contributing factor is the absence of datasets in the surgical field. In this paper, we create a new dataset, Surg-QA, consisting of 102,000 surgical video-instruction pairs, the largest of its kind so far. To build such a dataset, we propose a novel two-stage question-answer generation pipeline with LLM to learn surgical knowledge in a structured manner from the publicly available surgical lecture videos. The pipeline breaks down the generation process into two stages to significantly reduce the task complexity, allowing us to use a more affordable, locally deployed open-source LLM than the premium paid LLM services. It also mitigates the risk of LLM hallucinations during question-answer generation, thereby enhancing the overall quality of the generated data. We further train LLaVA-Surg, a novel vision-language conversational assistant capable of answering open-ended questions about surgical videos, on this Surg-QA dataset, and conduct comprehensive evaluations on zero-shot surgical video question-answering tasks. We show that LLaVA-Surg significantly outperforms all previous general-domain models, demonstrating exceptional multimodal conversational skills in answering open-ended questions about surgical videos. We will release our code, model, and the instruction-tuning dataset.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A retrieval-based surgical video model that searches a surgery-specific concept vocabulary achieves state-of-the-art zero-shot results on most benchmarks at a fraction of generative latency.

  2. Specialized Foundation Models for Intelligent Operating Rooms

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ORQA is a multimodal question-answering benchmark and specialized model family that beats generalist vision-language models on 23 surgical scene understanding tasks.

  3. Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data

    cs.CV 2025-01 conditional novelty 6.0 of 10

    RASO recognizes surgical instruments and anatomy in images and video using a weakly supervised training pipeline built from automatically generated tag-image-text pairs from surgical lecture videos.

  4. DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splatting

    cs.CV 2026-07 conditional novelty 5.5 of 10

    Decoupling geometry and semantics in 4DGS via HexPlane kinematic latents and rasterization-native extraction raises surgical semantic mIoU from 53.46% to 68.20% on CholecSeg8k.

  5. SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splatting

    eess.IV 2025-06 conditional novelty 5.0 of 10

    SurgTPGS is a text-promptable 3D Gaussian Splatting pipeline that segments surgical instruments and anatomy from natural-language queries at interactive frame rates.

  6. SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysis

    cs.CV 2025-06 conditional novelty 5.0 of 10

    SurgBench assembles a 53-million-frame surgical video pretraining corpus from 16 sources plus a 72-task evaluation benchmark, and shows continual pretraining with VideoMAE improves accuracy by 7.9% top-3 over Kinetics...

  7. EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery

    cs.CV 2025-01 reject novelty 5.0 of 10

    EndoChat is a grounded multimodal LLM for endoscopic surgery, trained on the new Surg-396K dataset and reported to outperform prior MLLMs, though its evaluation is confounded by training-data overlap.

  8. OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining

    cs.CV 2024-11 conditional novelty 5.0 of 10

    A hierarchical retrieval-augmented CLIP-style model pretrained on 375K ophthalmic surgery video-text pairs substantially outperforms CLIP on zero-shot phase and instrument recognition.

  9. Multimodal Large Language Models for Medicine: A Comprehensive Survey

    cs.LG 2025-04 conditional novelty 2.0 of 10

    A comprehensive review cataloging medical MLLMs, their uses in report generation, diagnosis, and treatment, and the challenges of accuracy, hallucination, fairness, privacy, and deployment.

Pith tools