Pith. sign in

REVIEW 7 cited by

LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.07981 v1 pith:WVHAAXEG submitted 2024-08-15 cs.CV cs.AI

Jiajie Li , Garrett Skinner , Gene Yang , Brian R Quaranto , Steven D Schwaitzberg , Peter C W Kim , Jinjun Xiong This is my paper
classification cs.CVcs.AI
keywords surgicalvideosdatasetmultimodalgenerationllava-surgmodelsanswering
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal large language models (LLMs) have achieved notable success across various domains, while research in the medical field has largely focused on unimodal images. Meanwhile, current general-domain multimodal models for videos still lack the capabilities to understand and engage in conversations about surgical videos. One major contributing factor is the absence of datasets in the surgical field. In this paper, we create a new dataset, Surg-QA, consisting of 102,000 surgical video-instruction pairs, the largest of its kind so far. To build such a dataset, we propose a novel two-stage question-answer generation pipeline with LLM to learn surgical knowledge in a structured manner from the publicly available surgical lecture videos. The pipeline breaks down the generation process into two stages to significantly reduce the task complexity, allowing us to use a more affordable, locally deployed open-source LLM than the premium paid LLM services. It also mitigates the risk of LLM hallucinations during question-answer generation, thereby enhancing the overall quality of the generated data. We further train LLaVA-Surg, a novel vision-language conversational assistant capable of answering open-ended questions about surgical videos, on this Surg-QA dataset, and conduct comprehensive evaluations on zero-shot surgical video question-answering tasks. We show that LLaVA-Surg significantly outperforms all previous general-domain models, demonstrating exceptional multimodal conversational skills in answering open-ended questions about surgical videos. We will release our code, model, and the instruction-tuning dataset.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    A kinematic-to-visual lifting paradigm combined with hierarchically routed control generates action-conditioned surgical videos with better faithfulness, fidelity, and efficiency.

  2. MedStreamBench: A Time-Aware Benchmark for Streaming and Proactive Medical Video Understanding

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    MedStreamBench integrates 22 medical datasets into 5,419 QA instances across retrospective, present, future, and proactive temporal settings to evaluate streaming and proactive medical video understanding.

  3. Fine-tuning a multimodal large language model for clinician-grade autism behavioral scoring from short home videos

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Fine-tuning Gemini 2.5 Pro with LoRA on 400 home videos improves per-feature agreement with clinicians by 40% and zero-shot ASD diagnosis F1 by 53% on held-out data, with classifier pipelines reaching 77% accuracy.

  4. SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    SurgOnAir introduces a streaming vision-language model trained on a hierarchical surgical dataset to generate real-time, multi-level narrations with explicit transition tokens.

  5. Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery

    cs.CV 2026-03 unverdicted novelty 6.0 of 10

    A video-only speech-guided system for skull-base surgery segments and tracks instruments to deliver 2.32 mm tool-tip accuracy and rapid 3D model registration.

  6. DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splatting

    cs.CV 2026-07 conditional novelty 5.5 of 10

    Decoupling geometry and semantics in 4DGS via HexPlane kinematic latents and rasterization-native extraction raises surgical semantic mIoU from 53.46% to 68.20% on CholecSeg8k.

  7. Watch, Remember, Reason: Human-View Video Understanding with MLLMs

    cs.CV 2026-06 unverdicted novelty 4.0 of 10

    This is a survey that frames video MLLM research via a human-view formulation of perceptual representations, memory states, reasoning traces, and predictions, then reviews methods, datasets, benchmarks, and open problems.

Pith tools