{"id":"80a40efd-ebcc-4123-8035-f042e5fe2c22","arxiv_id":"2411.10937","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"S2CAN generates self-contained direct and indirect memories (hints and question-hint pairs) from surgical images, then reasons over them to answer surgical VQA questions, reporting SOTA results on EndoVis-18, EndoVis-17, and Cholec80 benchmarks.","lead":"What if a surgical AI could quiz itself about what it sees before answering? This paper proposes S2CAN, a memory-augmented multimodal LLM that generates its own question-answer hints to improve surgical visual question answering, and reports state-of-the-art accuracy on three surgical video datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-time hint sets always contain the ground-truth answer, so the reported gains may reflect copying the answer out of the prompt rather than memory-based scene understanding; this confound is unquantified.","rationale":"The reader's weakest assumption and my own stress-test converge on the same load-bearing issue: the training-time hint annotations always append the ground-truth answer, so the memory-augmented VQA model can learn to copy the answer from the prompt instead of grounding its prediction in the image. This directly threatens the paper's central claim that the accuracy gain comes from self-contained contextual memory and scene understanding. The paper's Figure 4(b) even admits that 'the model directly copies an answer from memory,' but no experiment separates the copy route from genuine memory-based reasoning. I do not think this concern invalidates the reported SOTA numbers outright: the empirical comparisons could still hold, and the method could be re-described as candidate generation plus reranking rather than scene-comprehension memory. However, without the proposed no-leak control or a copy-rate statistic, the mechanism claim remains unverified. Since the reader's verdict was already CONDITIONAL with medium correctness risk, and my concern matches that assessment, I do not move the verdict. Secondary issues (no released code, no error bars, and the numeric inconsistency between Table 3 'w/o IM' at 67.2 and Table 5 M=0 at 65.7) further support the conditional stance but are not the primary objection.","tokens_in":17804,"tokens_out":4826,"duration_ms":59736,"concrete_test":"Train a no-leak control on EndoVis-18-VQA: in DM and IM annotations, replace the appended ground-truth hint with a randomly sampled non-GT candidate (or with only the top-K frequent candidates), and keep all other settings identical. If the accuracy gap between full S2CAN and the no-memory baseline collapses (e.g., from +3.9 to less than 1 point), the reported gains come from copying the GT out of the hints; if the gap is preserved, the memory mechanism is real. Additionally, at inference on the standard test split, compute the verbatim-copy rate (final answer appears verbatim in DM) for correct answers; if more than 90% of correct answers are verbatim copies, the shortcut dominates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism claim is that S2CAN improves surgical VQA by reasoning over self-generated memory (DM/IM) rather than by exploiting answer leakage. That claim is not yet supported. In §4.1 (Hint Annotation), training-time Direct Memory is formed by taking the top K-1 frequent answer candidates and appending the ground-truth answer, so DM always contains the target answer during memory-augmented VQA training (Eq. 7, LMVQA). §4.2 applies the same recipe to Indirect Memory, so IM question-hint pairs also contain the GT answer of those pairs during training. The model therefore has a direct copy route: output the GT token(s) present in DM on training examples, and at inference output whatever candidate the DM generator (trained on the same GT-appended pattern) proposes. The ablations in Table 3 compare memory vs no-memory, but removing memory removes both the contextual signal and the copy route; they cannot identify which one drives the +3.9/+2.5/+0.3 accuracy differences. The paper's own Error Analysis (Figure 4b) states 'the model directly copies an answer from memory,' confirming the shortcut exists, yet no experiment quantifies how often the final answer is a verbatim copy of a generated hint, nor whether accuracy survives when the GT is removed from the hint set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes S2CAN, a memory-augmented multimodal LLM framework for Surgical VQA. S2CAN generates two types of memory from the surgical image and the user question: Direct Memory (DM), which is a list of candidate hints for the answer, and Indirect Memory (IM), which is a set of self-contained question-hint pairs about the scene. The model is trained with three losses: direct-memory generation, indirect-memory generation, and memory-augmented VQA; at inference, the generated DM and IM are concatenated with the question and fed to the VQA head. Using BLIP-3 (Phi-3 3.8B) as the backbone with LoRA tuning, the authors report state-of-the-art accuracy on EndoVis-18-VQA (69.6%), EndoVis-17-VQLA (46.6%), and Cholec80-VQA (94.7%), outperforming the no-memory baseline by +3.9, +2.5, and +0.3 points respectively, along with ablations and case studies. The central claim is that self-generated memory improves surgical scene understanding and question comprehension, rather than relying on external resources.","tokens_in":18126,"tokens_out":3486,"duration_ms":37139,"significance":"If the reported improvements truly come from reasoning over self-generated memory, the work would be a useful step toward autonomous surgical VQA, since it avoids external object detectors or scene graphs and leverages an MLLM's own capabilities. The paper has concrete strengths: it evaluates on three public datasets, provides implementation details (LoRA rank, number of hints, memory sizes, prompt templates) sufficient for reimplementation, and includes ablations over memory components and memory-entry counts. However, the central mechanistic claim is currently not supported because the training-time memory construction gives the VQA head a direct route to the ground-truth answer: Section 4.1 appends the ground-truth answer to the hint set, Section 4.2 applies the same recipe to indirect memory, and the ablations in Table 3 remove the memory and the leakage simultaneously. The reported accuracy gains may therefore reflect answer-copying rather than memory-based scene understanding. The paper needs additional controlled experiments to separate these effects.","major_comments":[{"comment":"The Hint Annotation procedure in Section 4.1 always appends the ground-truth answer to the top K−1 frequent candidates, so the Direct Memory used during memory-augmented VQA training always contains the correct answer to the target question. This gives the model a trivial copy route: it can minimize LMVQA by selecting the answer token already present in DM. The paper's own Error Analysis in Figure 4(b) states that 'the model directly copies an answer from memory,' confirming that this shortcut is used. No experiment quantifies how often the final prediction is a verbatim copy of a hint, nor whether accuracy survives when the ground-truth answer is removed from the hint set. This is load-bearing because the central claim is that memory improves scene understanding rather than enabling answer copying.","section":"§4.1, Eq. (7)"},{"comment":"The ablations in Table 3 cannot separate the contextual signal from the leakage signal. Removing memory entirely ('S2CAN w/o M') removes both the added context and the copy route; removing only DM or only IM still leaves a context that contains the ground-truth answer for the target question. The +3.9/+2.5/+0.3 accuracy differences between 'S2CAN w/o M' and the full model are therefore consistent with a model that learns to rely on the presence of the correct answer in the memory. To support the paper's interpretation, the authors should train and evaluate with hint sets that exclude the ground-truth answer (e.g., top K frequent candidates excluding the correct one, or randomly sampled distractors), and also report oracle-memory versus generated-memory accuracy to isolate memory quality from VQA-head behavior.","section":"Table 3"},{"comment":"The Indirect Memory annotation uses the same ground-truth-appended hint recipe, and the memory-augmented VQA prompt in Table 7 (Supplementary) includes question-hint pairs whose hints contain the ground-truth answers of those pairs. Although the given question is excluded from the indirect memory to avoid leakage, the Direct Memory for the target question still contains its ground-truth answer. Consequently, the combined memory M = {IM, DM} used in Eq. (6) is not leakage-free at training time. The claim in Section 4.2 that converting answers into hints 'prevents the model from memorizing answers' is not substantiated, because the hint conversion preserves the answer tokens rather than removing them.","section":"§4.2"},{"comment":"The error analysis identifies 'wrong indirect memory' as a failure mode where 'the model directly copies an answer from memory.' This admission directly contradicts the interpretation that memory reasoning is the mechanism of improvement. A necessary control is to measure the copy rate on the test sets (e.g., the fraction of predictions exactly equal to one of the generated hints) and to compare accuracy between the full model and a variant trained with ground-truth answers removed from all memory annotations. Without such measurements, the reported gains cannot be attributed to contextual reasoning rather than to exploiting the training-time presence of the answer.","section":"§6.2, Figure 4"}],"minor_comments":[{"comment":"The abstract uses 'SCAN' while the rest of the paper uses 'S2CAN'; please make the notation consistent throughout.","section":"Abstract / Introduction"},{"comment":"The text says 'The results, shown in Table 3, indicate...' when discussing question-type accuracy; this should refer to Figure 3, since Table 3 is the ablation table.","section":"§6.1"},{"comment":"The sentence 'S2CAN achieves a 3.8% gain on Action questions' does not specify the dataset; please state that this refers to Cholec80-VQA (or whichever dataset is intended).","section":"§6.1"},{"comment":"The indirect-memory selection in Eq. (5) uses TF-IDF features f*, but it is not stated whether these features are computed on the training corpus alone or on the full dataset, and whether the TF-IDF vocabulary is frozen for the EndoVis-17-VQLA transfer test. Please clarify.","section":"§4.2"},{"comment":"The caption says 'Experimental results of S2CAN on three datasets with different numbers of entries in the indirect memory,' but the M=0 row corresponds to no indirect memory; please state this explicitly in the caption or in the text.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the answer-leakage confound in the training-time memory construction. The paper is otherwise clearly written and the experiments are reproducible in principle. If the authors add the proposed controls (ground-truth-free hint training, copy-rate measurement, and oracle-versus-generated memory comparison) and the gains persist, the paper could become suitable for publication. The small +0.3 gain on Cholec80-VQA and the lack of statistical significance testing should also be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a useful empirical paper with a clean two-stage memory idea, and the SOTA numbers on three surgical VQA benchmarks are interesting. But the central interpretation—that memory improves scene understanding—is not yet supported, because the training-time hint sets always contain the ground-truth answer. The gain could simply be the model copying the answer out of the prompt.\n\nWhat's genuinely new: S2CAN generates two levels of self-contained memory—direct hints for the question and TF-IDF-selected question-hint pairs for the image—and uses them to augment a multimodal LLM. That pipeline is simple, well described, and matches the stated goal of avoiding external resources. The experiments are thorough for a paper of this type: three public datasets, several MLLM baselines, ablations, case studies, and an error analysis. Implementation details are sufficient to reimplement. That is real work, and the numbers are plausible.\n\nThe soft spots are load-bearing, not cosmetic. Section 4.1 says the ground-truth answer is appended to the top K−1 frequent candidates to build the hint set; Section 4.2 applies the same recipe to indirect memory. So during training, the model always sees the correct answer inside the memory. At inference, the memory generator is the same model, trained on that pattern, so it can produce hints that contain the answer, and the final answer stage can copy. The ablation in Table 3 removes memory entirely, which removes both the contextual signal and the copy route; it cannot tell which one is driving the +3.9/+2.5/+0.3 accuracy differences. The paper's own Figure 4b states 'the model directly copies an answer from memory,' confirming the shortcut exists, yet no experiment quantifies how often that happens or checks accuracy when the GT is removed from the hint set. There is also a numeric inconsistency: EndoVis-17-VQLA Recall is 42.8 in Table 1 but 40.1 in Table 3 for what looks like the same configuration. No code or artifacts are released.\n\nNone of this makes the paper worthless. The method is a plausible way to get strong accuracy, and the confound is addressable: train a version where hints are only top-K frequent answers, no GT; report copy rates and error bars; add a fixed-distractor control. If those controls show the gain persists, the paper would be much stronger. As it stands, the empirical claim of SOTA may hold, but the mechanism claim does not.\n\nWho is it for? Anyone working on surgical VQA or memory-augmented MLLMs will find the method and the benchmark results worth knowing. It deserves a serious referee—I'd send it to review with heavy revision requests, not desk reject it. The experiments are too substantial to dismiss.","headline":"The memory-augmentation idea is clean and the benchmark numbers are strong, but the training-time hint sets always contain the ground-truth answer, so the central claim that memory improves scene understanding is not yet supported.","tokens_in":18611,"tokens_out":3721,"would_cite":false,"duration_ms":37432,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By generating direct and indirect memory from the image itself, S2CAN raises surgical VQA accuracy to 69.6%, 46.6%, and 94.7% on three benchmark datasets.","keywords":["surgical visual question answering","multimodal large language models","self-contained inquiry","direct memory","indirect memory","hint generation","surgical scene understanding","LoRA fine-tuning"],"falsifier":"Retrain S2CAN with direct-memory hint sets that never include the ground-truth answer (only frequent distractors) and compare accuracy on EndoVis-18-VQA; a drop from 69.6% toward the 65.7% no-memory baseline would indicate answer copying, while sustained accuracy would confirm that the memory itself supports reasoning.","tokens_in":17637,"feed_emoji":"🏥","tokens_out":8323,"duration_ms":78857,"temperature":0.7,"pith_summary":"This paper sets out to show that a multimodal large language model can answer questions about surgical scenes more accurately when it first generates its own memory of the scene instead of relying on external object features or scene graphs. The proposed system, S2CAN, creates two kinds of memory from the image and question alone: Direct Memory, a short list of candidate answers (hints) for the posed question, and Indirect Memory, a set of self-generated question-hint pairs that describe the broader scene. On three public surgical VQA benchmarks the method reports 69.6% accuracy on EndoVis-18-VQA, 46.6% on EndoVis-17-VQLA, and 94.7% on Cholec80-VQA, topping both classification-based specialists and larger medical MLLMs. If the claim holds, surgical VQA can be improved without external resources, because the context needed for reasoning is produced by the model itself.","feed_headline":"Hints the model writes itself lift surgical VQA accuracy","feed_subtitle":"A self-sufficient memory of question-hint pairs beats external-feature baselines on three surgical benchmarks.","key_machinery":"The load-bearing mechanism is the self-contained inquiry loop that builds the memory set $M$. Direct Memory is $\\{(Q, H)\\}$ where $H$ is a list of $K$ candidate answers generated from the image and question; Indirect Memory is a list of question-hint pairs generated from the prompt \"Generate a list of relevant questions with hints based on the provided surgical image,\" with the most relevant entries selected by TF-IDF cosine similarity. The same MLLM then reads $I$, $M$, and $Q$ and produces the final answer, so all contextual information is elicited from the model's own perception of the image rather than imported from outside.","core_discovery":"The central claim is that conditioning the final answer on memory generated by the same model, rather than on externally pre-extracted context, improves both scene understanding and question comprehension in surgical VQA. Given image $I$ and question $Q$, S2CAN first prompts itself to produce Direct Memory $DM$ as $K$ hints for $Q$, then prompts itself to generate Indirect Memory $IM$ as question-hint pairs about the image, selects the top $M$ entries by TF-IDF cosine similarity to $Q$, and finally answers as $A = \\mathrm{S2CAN}(I, [M; Q])$ with $M = \\{DM, IM_Q\\}$. The paper reports that this pipeline reaches state-of-the-art accuracy on three datasets and, in ablations, that removing memory costs 3.9 points on EndoVis-18-VQA, 10.8 points on EndoVis-17-VQLA, and 0.3 points on Cholec80-VQA, with indirect memory contributing more than direct memory.","pith_inferences":["A direct way to test the mechanism is to swap the labels inside the hints while keeping the image fixed; if predictions follow the swapped labels, S2CAN is exploiting prompt statistics rather than visual reasoning.","Because Indirect Memory is produced by the same model, memory-generation errors propagate into the answer; filtering low-confidence memory or generating several candidate memories and voting could yield larger gains than adding more entries.","The recipe is not surgery-specific and should transfer to other dense-scene VQA settings, such as robotic manipulation or autonomous driving, where multiple objects and their states must be understood together.","Replacing TF-IDF cosine selection with a learned relevance scorer could let Indirect Memory pick up semantically related questions that share few words with the user's question, reducing vocabulary-overlap leakage."],"forward_implications":["No external object detectors, scene graphs, or retrieval databases are required, so the method can be dropped into any surgical video setting where image-question pairs exist.","Indirect memory is the larger contributor: removing it costs more accuracy than removing direct memory, implying that broader scene-level context matters more than question-specific hints.","The memory mechanism transfers across datasets, since a model tuned on EndoVis-18-VQA answers the unseen EndoVis-17-VQLA set better than all compared baselines.","A 3.8B-parameter backbone tuned with LoRA beats 7B medical MLLMs, suggesting the gain comes from the memory-augmented reasoning structure rather than model scale.","Accuracy saturates with roughly two to three indirect memory entries, so the inference cost of generating memory stays modest."],"supporting_citations":[{"why":"Introduces the EndoVis-18-VQA and Cholec80-VQA datasets and the VisualBERT ResMLP baseline that S2CAN must exceed.","marker":"[43]"},{"why":"Supplies the SurgicalGPT language-prior baseline, the strongest classification-based comparison on Cholec80-VQA.","marker":"[44]"},{"why":"Contributes the EndoVis-17-VQLA dataset and the gated vision-language baseline used to test generalization.","marker":"[10]"},{"why":"Represents the external scene-graph method whose reliance on outside resources the self-contained design is meant to avoid.","marker":"[55]"},{"why":"Provides the BLIP-3 general-purpose multimodal LLM that serves as S2CAN's backbone and strongest MLLM baseline.","marker":"[50]"},{"why":"Gives the LLaVA-Med medical MLLM baseline, which S2CAN surpasses under LoRA tuning.","marker":"[27]"},{"why":"Gives the HuatuoGPT-Vision medical MLLM baseline used for zero-shot and LoRA comparisons.","marker":"[15]"},{"why":"Annotated as source of the Cholec80 data, grounding the largest dataset's statistics and phase labels.","marker":"[47]"},{"why":"Supplies the TF-IDF features used to select which indirect memory entries are relevant to each question.","marker":"[45]"}],"fun_headline_variants":["Self-contained memory outperforms external features in surgical VQA","Model-written hints beat external features in surgery QA","Internal memory outperforms external context for surgical VQA","Self-crafted memory lifts surgical VQA accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training-time hint sets always append the ground-truth answer, so the reported accuracy gain is assumed to reflect better scene understanding; it could instead reflect the model copying the correct answer from the hints whenever the self-generated candidates contain it.","fun_headline_variants_meta":{"raw":{"variants":["Self-contained memory outperforms external features in surgical VQA","Model-written hints beat external features in surgery QA","Internal memory outperforms external context for surgical VQA","Self-crafted memory lifts surgical VQA accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4073,"prompt_tokens":971,"completion_tokens":3102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":3040}},"tokens_in":587,"tokens_out":3102,"duration_ms":23842,"temperature":1.0,"reasoning_tokens":3040,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:08:04.440547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain S2CAN with direct-memory hint sets that never include the ground-truth answer (only frequent distractors) and compare accuracy on EndoVis-18-VQA; a drop from 69.6% toward the 65.7% no-memory baseline would indicate answer copying, while sustained accuracy would confirm that the memory itself supports reasoning.","supporting_citations":[{"cited_title":"Kr- ishna, and Hongliang Ren","cited_arxiv_id":null,"evidence_quote":"Introduces the EndoVis-18-VQA and Cholec80-VQA datasets and the VisualBERT ResMLP baseline that S2CAN must exceed."},{"cited_title":"Surgicalgpt: End-to-end language-vision gpt for visual question answering in surgery","cited_arxiv_id":null,"evidence_quote":"Supplies the SurgicalGPT language-prior baseline, the strongest classification-based comparison on Cholec80-VQA."},{"cited_title":"Surgical-vqla:transformer with gated vision- language embedding for visual question localized-answering in robotic surgery","cited_arxiv_id":null,"evidence_quote":"Contributes the EndoVis-17-VQLA dataset and the gated vision-language baseline used to test generalization."},{"cited_title":"Lavanchy, Nassir Navab, Vinkle Srivastav, and Nicolas Padoy","cited_arxiv_id":null,"evidence_quote":"Represents the external scene-graph method whose reliance on outside resources the self-contained design is meant to avoid."},{"cited_title":"xgen-mm (blip-3): A family of open large multimodal models, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the BLIP-3 general-purpose multimodal LLM that serves as S2CAN's backbone and strongest MLLM baseline."},{"cited_title":"LLaV A-med: Training a large language-and-vision assistant for biomedicine in one day","cited_arxiv_id":null,"evidence_quote":"Gives the LLaVA-Med medical MLLM baseline, which S2CAN surpasses under LoRA tuning."},{"cited_title":"Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024","cited_arxiv_id":null,"evidence_quote":"Gives the HuatuoGPT-Vision medical MLLM baseline used for zero-shot and LoRA comparisons."},{"cited_title":"Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy","cited_arxiv_id":null,"evidence_quote":"Annotated as source of the Cholec80 data, grounding the largest dataset's statistics and phase labels."},{"cited_title":"A statistical interpretation of term specificity and its application in retrieval, page 132–142","cited_arxiv_id":null,"evidence_quote":"Supplies the TF-IDF features used to select which indirect memory entries are relevant to each question."}],"review_version":1}