Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By generating direct and indirect memory from the image itself, S2CAN raises surgical VQA accuracy to 69.6%, 46.6%, and 94.7% on three benchmark datasets.

desk verdict The memory-augmentation idea is clean and the benchmark numbers are strong, but the training-time hint sets always contain the ground-truth answer, so the central claim that memory improves scene understanding is not yet supported. read the letter →

arxiv 2411.10937 v1 pith:Y7EXQLRS submitted 2024-11-17 cs.CV cs.CL

Wenjun Hou , Yi Cheng , Kaishuai Xu , Yan Hu , Wenjie Li , Jiang Liu This is my paper · ORCID
classification cs.CVcs.CL
keywords surgicalvisualquestionansweringmultimodallargelanguagemodelsself-containedinquirydirectmemoryindirecthintgenerationsceneunderstandingLoRAfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a multimodal large language model can answer questions about surgical scenes more accurately when it first generates its own memory of the scene instead of relying on external object features or scene graphs. The proposed system, S2CAN, creates two kinds of memory from the image and question alone: Direct Memory, a short list of candidate answers (hints) for the posed question, and Indirect Memory, a set of self-generated question-hint pairs that describe the broader scene. On three public surgical VQA benchmarks the method reports 69.6% accuracy on EndoVis-18-VQA, 46.6% on EndoVis-17-VQLA, and 94.7% on Cholec80-VQA, topping both classification-based specialists and larger medical MLLMs. If the claim holds, surgical VQA can be improved without external resources, because the context needed for reasoning is produced by the model itself.

What carries the argument

The load-bearing mechanism is the self-contained inquiry loop that builds the memory set $M$. Direct Memory is $\{(Q, H)\}$ where $H$ is a list of $K$ candidate answers generated from the image and question; Indirect Memory is a list of question-hint pairs generated from the prompt "Generate a list of relevant questions with hints based on the provided surgical image," with the most relevant entries selected by TF-IDF cosine similarity. The same MLLM then reads $I$, $M$, and $Q$ and produces the final answer, so all contextual information is elicited from the model's own perception of the image rather than imported from outside.

What would settle it

Retrain S2CAN with direct-memory hint sets that never include the ground-truth answer (only frequent distractors) and compare accuracy on EndoVis-18-VQA; a drop from 69.6% toward the 65.7% no-memory baseline would indicate answer copying, while sustained accuracy would confirm that the memory itself supports reasoning.

Watch

Extended reading notes

Core claim

The central claim is that conditioning the final answer on memory generated by the same model, rather than on externally pre-extracted context, improves both scene understanding and question comprehension in surgical VQA. Given image $I$ and question $Q$, S2CAN first prompts itself to produce Direct Memory $DM$ as $K$ hints for $Q$, then prompts itself to generate Indirect Memory $IM$ as question-hint pairs about the image, selects the top $M$ entries by TF-IDF cosine similarity to $Q$, and finally answers as $A = \mathrm{S2CAN}(I, [M; Q])$ with $M = \{DM, IM_Q\}$. The paper reports that this pipeline reaches state-of-the-art accuracy on three datasets and, in ablations, that removing memory costs 3.9 points on EndoVis-18-VQA, 10.8 points on EndoVis-17-VQLA, and 0.3 points on Cholec80-VQA, with indirect memory contributing more than direct memory.

Load-bearing premise

The training-time hint sets always append the ground-truth answer, so the reported accuracy gain is assumed to reflect better scene understanding; it could instead reflect the model copying the correct answer from the hints whenever the self-generated candidates contain it.

Editorial extensions

If this is right

  • No external object detectors, scene graphs, or retrieval databases are required, so the method can be dropped into any surgical video setting where image-question pairs exist.
  • Indirect memory is the larger contributor: removing it costs more accuracy than removing direct memory, implying that broader scene-level context matters more than question-specific hints.
  • The memory mechanism transfers across datasets, since a model tuned on EndoVis-18-VQA answers the unseen EndoVis-17-VQLA set better than all compared baselines.
  • A 3.8B-parameter backbone tuned with LoRA beats 7B medical MLLMs, suggesting the gain comes from the memory-augmented reasoning structure rather than model scale.
  • Accuracy saturates with roughly two to three indirect memory entries, so the inference cost of generating memory stays modest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct way to test the mechanism is to swap the labels inside the hints while keeping the image fixed; if predictions follow the swapped labels, S2CAN is exploiting prompt statistics rather than visual reasoning.
  • Because Indirect Memory is produced by the same model, memory-generation errors propagate into the answer; filtering low-confidence memory or generating several candidate memories and voting could yield larger gains than adding more entries.
  • The recipe is not surgery-specific and should transfer to other dense-scene VQA settings, such as robotic manipulation or autonomous driving, where multiple objects and their states must be understood together.
  • Replacing TF-IDF cosine selection with a learned relevance scorer could let Indirect Memory pick up semantically related questions that share few words with the user's question, reducing vocabulary-overlap leakage.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes S2CAN, a memory-augmented multimodal LLM framework for Surgical VQA. S2CAN generates two types of memory from the surgical image and the user question: Direct Memory (DM), which is a list of candidate hints for the answer, and Indirect Memory (IM), which is a set of self-contained question-hint pairs about the scene. The model is trained with three losses: direct-memory generation, indirect-memory generation, and memory-augmented VQA; at inference, the generated DM and IM are concatenated with the question and fed to the VQA head. Using BLIP-3 (Phi-3 3.8B) as the backbone with LoRA tuning, the authors report state-of-the-art accuracy on EndoVis-18-VQA (69.6%), EndoVis-17-VQLA (46.6%), and Cholec80-VQA (94.7%), outperforming the no-memory baseline by +3.9, +2.5, and +0.3 points respectively, along with ablations and case studies. The central claim is that self-generated memory improves surgical scene understanding and question comprehension, rather than relying on external resources.

Significance. If the reported improvements truly come from reasoning over self-generated memory, the work would be a useful step toward autonomous surgical VQA, since it avoids external object detectors or scene graphs and leverages an MLLM's own capabilities. The paper has concrete strengths: it evaluates on three public datasets, provides implementation details (LoRA rank, number of hints, memory sizes, prompt templates) sufficient for reimplementation, and includes ablations over memory components and memory-entry counts. However, the central mechanistic claim is currently not supported because the training-time memory construction gives the VQA head a direct route to the ground-truth answer: Section 4.1 appends the ground-truth answer to the hint set, Section 4.2 applies the same recipe to indirect memory, and the ablations in Table 3 remove the memory and the leakage simultaneously. The reported accuracy gains may therefore reflect answer-copying rather than memory-based scene understanding. The paper needs additional controlled experiments to separate these effects.

major comments (4)
  1. [§4.1, Eq. (7)] The Hint Annotation procedure in Section 4.1 always appends the ground-truth answer to the top K−1 frequent candidates, so the Direct Memory used during memory-augmented VQA training always contains the correct answer to the target question. This gives the model a trivial copy route: it can minimize LMVQA by selecting the answer token already present in DM. The paper's own Error Analysis in Figure 4(b) states that 'the model directly copies an answer from memory,' confirming that this shortcut is used. No experiment quantifies how often the final prediction is a verbatim copy of a hint, nor whether accuracy survives when the ground-truth answer is removed from the hint set. This is load-bearing because the central claim is that memory improves scene understanding rather than enabling answer copying.
  2. [Table 3] The ablations in Table 3 cannot separate the contextual signal from the leakage signal. Removing memory entirely ('S2CAN w/o M') removes both the added context and the copy route; removing only DM or only IM still leaves a context that contains the ground-truth answer for the target question. The +3.9/+2.5/+0.3 accuracy differences between 'S2CAN w/o M' and the full model are therefore consistent with a model that learns to rely on the presence of the correct answer in the memory. To support the paper's interpretation, the authors should train and evaluate with hint sets that exclude the ground-truth answer (e.g., top K frequent candidates excluding the correct one, or randomly sampled distractors), and also report oracle-memory versus generated-memory accuracy to isolate memory quality from VQA-head behavior.
  3. [§4.2] The Indirect Memory annotation uses the same ground-truth-appended hint recipe, and the memory-augmented VQA prompt in Table 7 (Supplementary) includes question-hint pairs whose hints contain the ground-truth answers of those pairs. Although the given question is excluded from the indirect memory to avoid leakage, the Direct Memory for the target question still contains its ground-truth answer. Consequently, the combined memory M = {IM, DM} used in Eq. (6) is not leakage-free at training time. The claim in Section 4.2 that converting answers into hints 'prevents the model from memorizing answers' is not substantiated, because the hint conversion preserves the answer tokens rather than removing them.
  4. [§6.2, Figure 4] The error analysis identifies 'wrong indirect memory' as a failure mode where 'the model directly copies an answer from memory.' This admission directly contradicts the interpretation that memory reasoning is the mechanism of improvement. A necessary control is to measure the copy rate on the test sets (e.g., the fraction of predictions exactly equal to one of the generated hints) and to compare accuracy between the full model and a variant trained with ground-truth answers removed from all memory annotations. Without such measurements, the reported gains cannot be attributed to contextual reasoning rather than to exploiting the training-time presence of the answer.
minor comments (5)
  1. [Abstract / Introduction] The abstract uses 'SCAN' while the rest of the paper uses 'S2CAN'; please make the notation consistent throughout.
  2. [§6.1] The text says 'The results, shown in Table 3, indicate...' when discussing question-type accuracy; this should refer to Figure 3, since Table 3 is the ablation table.
  3. [§6.1] The sentence 'S2CAN achieves a 3.8% gain on Action questions' does not specify the dataset; please state that this refers to Cholec80-VQA (or whichever dataset is intended).
  4. [§4.2] The indirect-memory selection in Eq. (5) uses TF-IDF features f*, but it is not stated whether these features are computed on the training corpus alone or on the full dataset, and whether the TF-IDF vocabulary is frozen for the EndoVis-17-VQLA transfer test. Please clarify.
  5. [Table 5] The caption says 'Experimental results of S2CAN on three datasets with different numbers of entries in the indirect memory,' but the M=0 row corresponds to no indirect memory; please state this explicitly in the caption or in the text.

Circularity Check

2 steps flagged · score 6.0 of 10

Direct and Indirect Memory hint sets are constructed by appending ground-truth answers, so the reported gains may reduce to copying the answer from the prompt; no ablation isolates the copy route.

  1. self definitional [Section 4.1 (Hint Annotation), Section 4.3 (Memory-Augmented VQA), Eq. (6)-(7), Table 7 prompt]
    "To annotate hints for each question, we first collect all answer candidates from the training data. Next, we select the top K − 1 candidates based on their frequency. Finally, the ground-truth answer is appended to this list of candidates to form the complete hint set."

    Direct Memory is defined so that the ground-truth answer is always one of the hints. The Memory-Augmented VQA prompt in Table 7 feeds the model 'Memory: ... Question: {Question} {Hints}', and Eq. (6)-(7) train the final answer conditioned on that memory. Thus the training-time input M contains the target answer by construction, and the learned mapping from (I, M, Q) to A can reduce to copying the answer token already present in M. The ablations in Table 3 compare memory vs. no memory, but removing memory removes both the contextual signal and the copy route; the reported +3.9/+2.5/+0.3 gains therefore do not establish that self-generated memory improves scene understanding. No experiment removes the ground-truth answer from the hint set while keeping memory.

  2. fitted input called prediction [Section 4.2 (Indirect Memory Annotation), Section 6.2 (Error Analysis, Figure 4b)]
    "To prevent the model from memorizing answers during training, we convert the answers into hints using the same method described in §4.1. ... In case (b), where wrong indirect memory is the primary cause, the model directly copies an answer from memory."

    The paper claims to prevent memorization, but converting answers into hints with the same recipe still places the ground-truth answer inside the question-hint pairs used as Indirect Memory. The memory-augmented VQA model is trained to produce answers from prompts that contain those answers, so the Indirect Memory component, described as providing broader scene context, is in part a fitted container for the very answers it is later used to predict. The paper's own Error Analysis admits that the model 'directly copies an answer from memory,' confirming the shortcut exists, yet no experiment quantifies how often the final answer is a verbatim copy of a generated hint or whether accuracy survives when the ground-truth answer is removed from the hint set.

full rationale

The paper is not built on a self-citation chain: the SOTA claims are evaluated against external baselines, and the transfer to EndoVis-17-VQLA provides independent benchmark evidence. The circularity is internal to the S2CAN recipe. Section 4.1 defines Direct Memory by appending the ground-truth answer to the top K-1 frequent candidates, and Section 4.2 applies the same recipe to Indirect Memory. The memory-augmented VQA stage (Eq. 6-7, Table 7) is therefore trained on prompts that contain the target answer whenever the hint is correct. The model can learn to copy from the memory rather than reason about the scene, and the ablation study cannot separate contextual benefit from the copy shortcut because removing memory removes both. The paper explicitly documents copying behavior in its Error Analysis. The stated conclusion that self-generated memory improves scene understanding is thus not established, although the reported accuracy numbers themselves are legitimate train/test results. Score 6 reflects a partial, construction-level confound rather than a fully forced equivalence between the prediction and the input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

S2CAN introduces no new physical entities; Direct Memory and Indirect Memory are procedural constructs within the method, not unexplained postulates.

free parameters (4)
  • K (number of hints per question) = 2 for all datasets
    Chosen from preliminary experiments with K in {2,3} and dataset statistics (#A/Q); affects both hint generation and DM conditioning.
  • M (number of indirect memory entries) = 3 for EndoVis-18-VQA and EndoVis-17-VQLA, 1 for Cholec80-VQA
    Selected per dataset; Table 5 shows M=2 yields 70.0 on EndoVis-18, so the final M=3 is not the best on that dataset.
  • N (frequency threshold for keeping a QA pair as indirect memory) = 500
    Set for all datasets; determines which training QA pairs become indirect memory annotations.
  • LoRA rank r and alpha = r=8, alpha=32, dropout=0.1
    Standard LoRA hyperparameters for tuning the LLM; chosen without stated search.
assumptions (4)
  • domain assumption BLIP-3 can generate accurate hints and self-contained question-hint pairs from surgical images without external object features
    The whole method depends on the quality of self-generated memory; no independent measure of hint recall or question quality is reported.
  • domain assumption TF-IDF cosine similarity selects the most useful indirect memory entries for a given question
    Used in Eq. 5; no ablation compares TF-IDF with other retrieval or learned selectors.
  • ad hoc to paper Training with the ground-truth answer inside the hint set does not reduce the method to a copying shortcut
    Section 4.1 appends the GT answer to the hint list; the paper's interpretation of memory benefit requires that the model reasons over images rather than copies, but this is not tested.
  • standard math Autoregressive negative log-likelihood is a valid training objective for the three generation tasks
    Equation 7 combines L_DM, L_IM, L_MVQA; standard practice for generative MLLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry." pith.science (2026). https://pith.science/paper/Y7EXQLRS

@misc{pith2026241110937,
  author       = {Pith},
  title        = {Pith review of: Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquiry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y7EXQLRS}},
  note         = {Machine review of arXiv:2411.10937}
}
read the original abstract

Comprehensively understanding surgical scenes in Surgical Visual Question Answering (Surgical VQA) requires reasoning over multiple objects. Previous approaches address this task using cross-modal fusion strategies to enhance reasoning ability. However, these methods often struggle with limited scene understanding and question comprehension, and some rely on external resources (e.g., pre-extracted object features), which can introduce errors and generalize poorly across diverse surgical environments. To address these challenges, we propose SCAN, a simple yet effective memory-augmented framework that leverages Multimodal LLMs to improve surgical context comprehension via Self-Contained Inquiry. SCAN operates autonomously, generating two types of memory for context augmentation: Direct Memory (DM), which provides multiple candidates (or hints) to the final answer, and Indirect Memory (IM), which consists of self-contained question-hint pairs to capture broader scene context. DM directly assists in answering the question, while IM enhances understanding of the surgical scene beyond the immediate query. Reasoning over these object-aware memories enables the model to accurately interpret images and respond to questions. Extensive experiments on three publicly available Surgical VQA datasets demonstrate that SCAN achieves state-of-the-art performance, offering improved accuracy and robustness across various surgical scenarios.

Figures

Figures reproduced from arXiv: 2411.10937 by the authors.

Figure 1
Figure 1. Given an image and a question, S 2CAN generates memory via self-contained inquiry (→) and answers the question (→). these, visual question answering (VQA) [7, 13, 14, 52], has emerged as a unified approach that can frame these tasks as a comprehensive problem of image- and video-grounded text generation. This paradigm not only streamlines the ana￾lytical process but also reduces the complexity of designing custom wo… view at source ↗
Figure 2
Figure 2. Illustration of our proposed S 2CAN framework, which first generates memory and then utilizes it for VQA. Red spans provide highly-related information for the answers. Note that Indirect Memory can be used to enhance any relevant questions of the given image. ical models have shown significant potential across various modalities, such as radiology and pathology. Li et al. [27] introduced a medical version of LLaVA [… view at source ↗
Figure 3
Figure 3. Performance of LLaVA-Med-v1.5, BLIP-3, and our [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Two major causes of errors produced by S 2CAN, i.e., wrong direct memory and wrong indirect memory. Case (a) and (b) are selected from the EndoVis-18-VQA and Cholec80-VQA datasets. than DM. Furthermore, since both types of memory share overlapping objectives, their com…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A retrieval-based surgical video model that searches a surgery-specific concept vocabulary achieves state-of-the-art zero-shot results on most benchmarks at a fraction of generative latency.

  2. EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery

    cs.CV 2025-01 reject novelty 5.0 of 10

    EndoChat is a grounded multimodal LLM for endoscopic surgery, trained on the new Surg-396K dataset and reported to outperform prior MLLMs, though its evaluation is confounded by training-data overlap.

Reference graph

Works this paper leans on

58 extracted references · 52 canonical work pages · cited by 2 Pith papers

  1. [1]

    Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, S ´ebastien Bubeck, Qin Cai, Martin Cai, Caio C´esar Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Yen-Chun Chen, Yi-Ling C...

    work page 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Se- bastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj B...

    work page 2022

  3. [3]

    Fusion of detected objects in text for visual ques- tion answering

    Chris Alberti, Jeffrey Ling, Michael Collins, and David Re- itter. Fusion of detected objects in text for visual ques- tion answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2131–2140, Hong Kong, China, 2019. A...

    work page 2019

  4. [4]

    2017 robotic instrument segmentation chal- lenge, 2019

    Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, Luis Her- rera, Wenqi Li, Vladimir Iglovikov, Huoling Luo, Jian Yang, Danail Stoyanov, Lena Maier-Hein, Stefanie Speidel, and Mahdi Azizian. 2017 robotic instrument segmentation chal- lenge, 2019. 4

    work page 2017

  5. [5]

    2018 robotic scene segmentation challenge, 2020

    Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Fe- lix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, Avinash Kori, Varghese Alex, Ganapathy Krish- namurthi, David Rauber, Robert Mendel, Christoph Palm, Sophia Bano, Guinther Saibro, Chi-Sheng Shih, Hsun-An Chiang, Juntang Zhuang, Junlin Y...

    work page 2018

  6. [6]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku

  7. [7]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015. 1

    work page 2015

  8. [8]

    Ehrxqa: A multi-modal question answer- ing dataset for electronic health records with chest x-ray im- ages

    Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang, Tackeun Kim, et al. Ehrxqa: A multi-modal question answer- ing dataset for electronic health records with chest x-ray im- ages. Advances in Neural Information Processing Systems, 36, 2024. 2

    work page 2024

Show all 58 references
  1. [9]

    Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond, 2023. 2

    2023

  2. [10]

    Surgical-vqla:transformer with gated vision- language embedding for visual question localized-answering in robotic surgery

    Long Bai, Mobarakol Islam, Lalithkumar Seenivasan, and Hongliang Ren. Surgical-vqla:transformer with gated vision- language embedding for visual question localized-answering in robotic surgery. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 6859–...

    2023

  3. [11]

    Surgical-vqla++: Adversarial contrastive learning for calibrated robust visual question-localized answering in robotic surgery, 2024

    Long Bai, Guankun Wang, Mobarakol Islam, Lalithkumar Seenivasan, An Wang, and Hongliang Ren. Surgical-vqla++: Adversarial contrastive learning for calibrated robust visual question-localized answering in robotic surgery, 2024. 2

    2024

  4. [12]

    Hasan, Vivek V

    Asma Ben Abacha, Sadid A. Hasan, Vivek V . Datla, Joey Liu, Dina Demner-Fushman, and Henning M¨uller. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Working Notes of CLEF 2019, Lugano, Switzerland, 2019. CEUR-WS.org. 2

    2019

  5. [13]

    Mutan: Multimodal tucker fusion for visual ques- tion answering

    Hedi Ben-younes, Remi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual ques- tion answering. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2631–2639, 2017. 1

    2017

  6. [14]

    Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection

    Hedi Ben-younes, R ´emi Cadene, Nicolas Thome, and Matthieu Cord. Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection

  7. [15]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024

    Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou 9 Wang. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024. 3, 5

    2024

  8. [16]

    Multi-modal masked autoencoders for medical vision-and-language pre-training

    Zhihong Chen, Yuhao Du, Jinpeng Hu, Yang Liu, Guanbin Li, Xiang Wan, and Tsung-Hui Chang. Multi-modal masked autoencoders for medical vision-and-language pre-training. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022. 2

    2022

  9. [17]

    Generative bias for robust visual question an- swering

    Jae Won Cho, Dong-Jin Kim, Hyeonggon Ryu, and In So Kweon. Generative bias for robust visual question an- swering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11681–11690, 2023. 2

    2023

  10. [18]

    Caption- aware medical vqa via semantic focusing and progres- sive cross-modality comprehension

    Fuze Cong, Shibiao Xu, Li Guo, and Yinbing Tian. Caption- aware medical vqa via semantic focusing and progres- sive cross-modality comprehension. In Proceedings of the 30th ACM International Conference on Multimedia, page 3569–3577, New York, NY , USA, 2022. Association for Co...

    2022

  11. [19]

    BERT: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    2019

  12. [20]

    Sedigheh Eslami, Christoph Meinel, and Gerard de Melo. PubMedCLIP: How much does CLIP benefit visual ques- tion answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1181–1193, Dubrovnik, Croatia, 2023. Association for ...

    2023

  13. [21]

    Data filtering networks, 2023

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks, 2023. 6

    2023

  14. [22]

    Pathvqa: 30000+ questions for medical visual question answering, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering, 2020. 1, 2

    2020

  15. [23]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 6

    2022

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L ´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

    2023

  17. [25]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ...

    2017

  18. [26]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018. 1, 2

    2018

  19. [27]

    LLaV A-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. LLaV A-med: Training a large language-and-vision assistant for biomedicine in one day. In Thirty-seventh Conference on Neural Information Processing...

    2023

  20. [28]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 2

    2023

  21. [29]

    Llava-surg: Towards multimodal surgical assistant via struc- tured surgical video learning, 2024

    Jiajie Li, Garrett Skinner, Gene Yang, Brian R Quaranto, Steven D Schwaitzberg, Peter C W Kim, and Jinjun Xiong. Llava-surg: Towards multimodal surgical assistant via struc- tured surgical video learning, 2024. 3

    2024

  22. [30]

    Visualbert: A simple and performant baseline for vision and language, 2019

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language, 2019. 2, 5

    2019

  23. [31]

    Masked vision and language pre-training with uni- modal and multimodal contrastive losses for medical visual question answering

    Pengfei Li, Gang Liu, Jinlong He, Zixu Zhao, and Shenjun Zhong. Masked vision and language pre-training with uni- modal and multimodal contrastive losses for medical visual question answering. pages 374–383, 2023. 2

    2023

  24. [32]

    Instrument-tissue interaction detection framework for surgical video understanding

    Wenjun Lin, Yan Hu, Huazhu Fu, Mingming Yang, Chin- Boon Chng, Ryo Kawasaki, Cheekong Chui, and Jiang Liu. Instrument-tissue interaction detection framework for surgical video understanding. IEEE Transactions on Medical Imaging, pages 1–1, 2024. 1

    2024

  25. [33]

    Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654, 2021. 1, 2

    2021

  26. [34]

    Bootstrapping large language models for radiology report generation

    Chang Liu, Yuanhe Tian, Weidong Chen, Yan Song, and Yongdong Zhang. Bootstrapping large language models for radiology report generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18635–18643,

  27. [35]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 2, 3

    2023

  28. [36]

    Med-flamingo: a multimodal medical few-shot learner, 2023

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Ra- jpurkar, and Jure Leskovec. Med-flamingo: a multimodal medical few-shot learner, 2023. 3, 5

    2023

  29. [37]

    Nguyen, Thanh-Toan Do, Binh X

    Binh D. Nguyen, Thanh-Toan Do, Binh X. Nguyen, Tuong Do, Erman Tjiputra, and Quang D. Tran. Overcoming data limitation in medical visual question answering. In MICCAI,

  30. [38]

    Rendezvous: Attention mech- anisms for the recognition of surgical action triplets in en- doscopic videos

    Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mech- anisms for the recognition of surgical action triplets in en- doscopic videos. Medical Image Analysis, 78:10...

    2022

  31. [39]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. 2

    2023

  32. [40]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. 2024. 7 10

    2024

  33. [41]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 2, 5

    2019

  34. [42]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 7

    2021

  35. [43]

    Kr- ishna, and Hongliang Ren

    Lalithkumar Seenivasan, Mobarakol Islam, Adithya K. Kr- ishna, and Hongliang Ren. Surgical-vqa: Visual question answering in surgical scenes using transformer. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2022, pages 33–43, Cham, 2022. Springer Nature...

    2022

  36. [44]

    Surgicalgpt: End-to-end language-vision gpt for visual question answering in surgery

    Lalithkumar Seenivasan, Mobarakol Islam, Gokul Kannan, and Hongliang Ren. Surgicalgpt: End-to-end language-vision gpt for visual question answering in surgery. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pages 281–290, Cham, 2023. Springer Natu...

    2023

  37. [45]

    A statistical interpretation of term specificity and its application in retrieval, page 132–142

    Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval, page 132–142. Tay- lor Graham Publishing, GBR, 1988. 4

    1988

  38. [46]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lan...

    2023

  39. [47]

    Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy

    Andru P. Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy. Endonet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Transactions on Medical Imaging, 36(1):86– 97, 2017. 1, 2, 5

    2017

  40. [48]

    Towards holistic surgical scene understanding

    Natalia Valderrama, Paola Ruiz, Isabela Hern´andez, Nicol´as Ayobi, Mathilde Verlyck, Jessica Santander, Juan Caicedo, Nicol´as Fern ´andez, and Pablo Arbel ´aez. Towards holistic surgical scene understanding. In Medical Image Computing and Computer Assisted Intervention – MIC...

    2022

  41. [49]

    Rule: Reliable mul- timodal rag for factuality in medical vision language models

    Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable mul- timodal rag for factuality in medical vision language models. arXiv preprint arXiv:2407.05131, 2024. 3

    2024 arXiv

  42. [50]

    xgen-mm (blip-3): A family of open large multimodal models, 2024

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin C...

    2024

  43. [51]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024

  44. [52]

    Stacked attention networks for image ques- tion answering

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image ques- tion answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1

    2016

  45. [53]

    Multi- modal factorized bilinear pooling with co-attention learning for visual question answering

    Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi- modal factorized bilinear pooling with co-attention learning for visual question answering. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1839–1848,

    2017

  46. [54]

    Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering

    Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. IEEE TNNLS, 29(12):5947–5959, 2018. 5

    2018

  47. [55]

    Lavanchy, Nassir Navab, Vinkle Srivastav, and Nicolas Padoy

    Kun Yuan, Manasi Kattel, Joel L. Lavanchy, Nassir Navab, Vinkle Srivastav, and Nicolas Padoy. Advancing surgical vqa with scene graph knowledge, 2024. 2, 5

    2024

  48. [56]

    Medical visual question answering via conditional reasoning

    Li-Ming Zhan, Bo Liu, Lu Fan, Jiaxin Chen, and Xiao- Ming Wu. Medical visual question answering via conditional reasoning. In Proceedings of the 28th ACM International Conference on Multimedia, page 2345–2354, New York, NY , USA, 2020. Association for Computing Machinery. 2

    2020

  49. [57]

    Pmc-vqa: Visual in- struction tuning for medical visual question answering, 2024

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual in- struction tuning for medical visual question answering, 2024. 2

    2024

  50. [58]

    MiniGPT-4: Enhancing vision- language understanding with advanced large language mod- els

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision- language understanding with advanced large language mod- els. In The Twelfth International Conference on Learning Representations, 2024. 2 11 Memory-Augmented Multimodal LLMs for ...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.