{"id":"86cc0034-cc5f-4f68-a07d-3ccb22f13229","arxiv_id":"2607.27122","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Multi-task LoRA fine-tuning with Grad-CAM grounding and terminology-free descriptions raises small-VLM GI VQA accuracy and implicit answer-to-region alignment on in- and out-of-distribution data.","lead":"Small vision-language models answer GI endoscopy questions more accurately when also trained to localize and describe findings, using reused polyp masks and Grad-CAM weak labels. The recipe is cheap to build and improves how answer tokens align with the right image region, including on unseen data.","discovery_kind":"extension","skeptic_critique":null,"referee_report":{"model":"moonshotai/kimi-k3","summary":"The manuscript proposes a multi-task QLoRA fine-tuning recipe for GI endoscopy VQA on small (<2B) VLMs. Auxiliary tasks are constructed with minimal new annotation: expert Kvasir-SEG polyp masks are reused directly, Grad-CAM masks from a frozen GastroNet-5M ResNet-50 classifier provide weak localization supervision for abnormalities and landmarks, and Gemma-27B generates terminology-free visual descriptions as a third task. Three backbones (Florence-2 0.2B, Qwen3.5 0.8B, InternVL3.5 1.1B) are trained under matched VQA-only and multi-task recipes on a 50k subset of Kvasir-VQA-x1 and evaluated on a held-out 5k set with a DeepSeek-V4 rubric judge. The paper reports consistent overall accuracy gains (+1.37 to +2.74pp), concentrated in appearance/spatial question types and increasing with question complexity for two of three backbones, plus an implicit grounding analysis (decoder answer-token to encoder patch cosine similarity; pointing-game and concentration-ratio metrics) showing large improvements over VQA-only training on in-distribution Kvasir-SEG polyps and out-of-distribution CVC-ClinicDB, with a Wilcoxon signed-rank test (p=1.47e-36) on the OOD concentration ratio.","tokens_in":8988,"tokens_out":4036,"duration_ms":262847,"significance":"If the results hold, the work is a useful, honest contribution to medical VQA: it targets small, deployable VLMs rather than 4B+ models; the recipe is cheap (single RTX 3090 Ti, one epoch, no new pixel annotation); the matched VQA-only vs MT comparison across three backbones with per-complexity and per-question-type breakdowns is methodologically cleaner than much of the medical VQA literature; and the inclusion of an out-of-distribution grounding evaluation (CVC-ClinicDB) with a paired significance test is a genuinely falsifiable check that the grounding improvements are not merely in-distribution memorization. The Vis/Des ablation, showing complementary behavior of segmentation vs description supervision on discrete vs diffuse findings, is an informative result in its own right. The scope is workshop/conference-paper scale rather than a definitive clinical study, but within that scope the experimental design is reasonable and the claims are mostly calibrated.","major_comments":[{"comment":"All accuracy numbers appear to come from a single training run per backbone/recipe, with no seeds, confidence intervals, or per-class sample sizes. The headline gains are small (+1.37pp for Qwen3.5, +1.53pp for InternVL3.5), Qwen3.5 C1 actually regresses (87.0→86.6, Table 1), and the largest single claim (polyp size +12.1pp for Florence-2, Table 3) is on a class whose n is not reported (landmark color with n=11 was excluded, implying other classes may also be small). LoRA fine-tuning variance across seeds can easily exceed 1–2pp. The 'consistent gains' claim needs either multi-seed runs (≥3) or, at minimum, paired bootstrap confidence intervals on the 5k eval set and per-class n in Table 3.","section":"§4.1, Tables 1–3"},{"comment":"All reported accuracy depends on DeepSeek-V4 as a text-only rubric judge, with no validation of the judge itself. Two specific risks: (i) the partial-credit rubric (0.5 for partial multi-item coverage, 0.5 for off-by-one counts) is exactly where judge reliability matters most; (ii) MT training teaches the model to mimic Gemma-27B description phrasing, which could systematically shift answer style in ways a text-only judge rewards or penalizes independently of correctness. A small human-vs-judge agreement audit on a few hundred scored answers (reporting Cohen's kappa or accuracy of the judge against human labels) would substantially de-risk the paper's primary quantitative claim.","section":"§3 (evaluation protocol)"},{"comment":"The grounding metric's representations are under-specified. It is not stated whether e_ti and e_vj are static input embeddings or contextual hidden states, nor from which layer. If e_ti is a static vocabulary embedding, cosine similarity to encoder patch embeddings is only meaningful if the two modalities share a representation space, which is architecture-dependent (plausible for Florence-2, less obvious generally); if they are hidden states, the layer choice matters and should be justified. Relatedly, the in-distribution PG/CR evaluation in Table 4 uses the same Kvasir-SEG polyp masks that supply the MT segmentation supervision, so the in-distribution grounding improvement is partly by construction; the OOD CVC-ClinicDB result is the load-bearing evidence and the text should say so explicitly.","section":"§4.2, Eq. (2)"},{"comment":"The recipe's novelty rests on Grad-CAM masks from a frozen-backbone classifier being faithful localization targets for non-polyp findings, yet mask quality is explicitly deferred to future work (Conclusion). A cheap, in-scope sanity check exists: the classifier also predicts polyp, so the authors can compute IoU of the polyp Grad-CAM masks (thresholded by the same pipeline) against Kvasir-SEG expert masks and report it, even though those masks are unused in training. Without some such audit, the attribution of MT gains on abnormality/landmark classes to 'grounding supervision' rather than noisy class-correlated heatmaps remains an assumption.","section":"§2.1 (Grad-CAM mask pipeline)"}],"minor_comments":[{"comment":"The claim that 'Full MT achieves the best or near-best result in every condition' is strained by the OOD pointing-game numbers: MT scores 0.645 vs 0.727 for Vis-only, an 8.2pp gap. Please soften the wording and discuss why adding the description task degrades OOD PG relative to Vis alone.","section":"§4.2, Table 4"},{"comment":"Only one significance test is reported (Wilcoxon on CVC CR). PG differences and the ablation comparisons are untested, and no multiple-comparison consideration is given across the four conditions × two datasets × two metrics. At minimum, report paired tests for the headline PG contrast.","section":"§4.2"},{"comment":"State the per-class sample counts in Table 3 and the exclusion threshold used (landmark color, n=11). Without n, readers cannot judge which per-class deltas are meaningful.","section":"§3"},{"comment":"Hyperparameters are given, which is commendable, but no code or mask-release statement appears. Given the single-epoch, 50k-subset, single-GPU setup, releasing the training/eval code and the generated Grad-CAM masks would materially strengthen the paper.","section":"§2.3 / reproducibility"},{"comment":"The inflammation rows are assessed purely by visual inspection with no ground truth, which the authors acknowledge; please also note explicitly that the 'less noisy' judgment is qualitative and consider reporting a quantitative proxy (e.g., CR against the Grad-CAM masks used in training, with the caveat that this is partially circular).","section":"§4.3, Fig. 2"},{"comment":"There are spacing/typographical artifacts in the PDF (e.g., 'Grad-CAM-basedweaklocalizationandaterminology-free' in the contributions list, 'T able 1/2/3/4' captions). Please proofread the final camera copy.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a follow-up to the authors' own Medico 2025 challenge submission (ref [7]); the overlap is disclosed and the delta (Grad-CAM replacing ClipSeg pseudo-masks, three backbones instead of one, and the implicit-alignment analysis) is legitimate, though the editor may wish to verify the venue's novelty bar given the shared motivation and dataset. The work reads as a solid workshop/short-conference paper; it would not meet the bar of a top-tier journal without the seed/judge/mask-quality additions requested. No concerns about citation practice beyond the expected self-citation."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a clean, limited-scope engineering paper that shows matched multi-task QLoRA (phrase grounding + terminology-free description) beats VQA-only on three sub-2B backbones for Kvasir-VQA-x1, and that the gains show up in implicit token-to-patch alignment on both Kvasir polyps and OOD CVC-ClinicDB. It does not invent grounded medical VQA; it simplifies their own prior ClipSeg stack and measures more carefully.\n\nWhat is actually new is the weak-supervision path (GastroNet Grad-CAM instead of prompt-engineered ClipSeg), Gemma descriptions stripped of clinical jargon, three small backbones under matched recipes, and the Vis/Des ablations on pointing-game and concentration ratio. That last bit is the most useful part. Des drives in-distribution concentration; Vis generalizes better OOD; full MT is least noisy. Florence-2 and InternVL pick up appearance/spatial and higher-complexity questions; Qwen is flatter. The design is honest about inference using only VQA.\n\nSoft spots, in proportion: (1) Grad-CAM masks for non-polyp findings are unvalidated—the authors themselves park a spot-check in future work, and that is load-bearing for the “grounding” story outside polyps. (2) Accuracy is scored by DeepSeek-V4 with no image access and no multi-seed or error bars on ~1–3 pp gains. (3) One epoch on a 50k subset, free thresholds in the mask pipeline, and no released code. None of that overturns the directional tables; it just caps how hard you should push the claim.\n\nCitations look fine—12-in-1, Molmo, Surgical-VQLA, EndoChat, their Medico note—without pretending this is a new paradigm. Math is standard Grad-CAM and cosine probes; nothing fancy, nothing broken.\n\nWho it is for: people fine-tuning small medical VLMs on a single GPU who care about whether answer tokens actually sit on the finding. Not for clinical deployment arguments. I would send it to peer review; a serious referee can demand mask QC, uncertainty, and a better judge protocol. Worth a skim if you work this area; not a must-read for a general vision reading group unless you are deep in MedVQA tooling.","headline":"Solid applied recipe paper: multi-task grounding on small GI VLMs works directionally, with real ablations and OOD probes, but mask quality and judge protocol still need tightening before you lean on the numbers.","tokens_in":10030,"tokens_out":601,"would_cite":false,"duration_ms":28749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Multi-task fine-tuning with cheap grounding and description tasks makes small GI endoscopy VLMs both more accurate and more visually aligned.","keywords":["Vision Language Models","Weak Supervision","Multi-Task Learning","Medical VQA","Visual Grounding","Parameter-Efficient Fine-Tuning","GI endoscopy","Grad-CAM"],"falsifier":"A human spot-check of the Grad-CAM masks against expert outlines for non-polyp findings, or a controlled re-run that replaces those weak masks with true expert masks or random masks and checks whether VQA gains and token-to-patch alignment collapse.","tokens_in":10158,"feed_emoji":"🔬","tokens_out":878,"duration_ms":17294,"temperature":0.7,"pith_summary":"Clinical GI image tools are moving from single-label classifiers to visual question answering, but a correct answer is not enough if the model is not looking at the right tissue. This paper argues that small vision-language models can be made both more accurate and more trustworthy by a simple multi-task fine-tuning recipe that needs almost no new expert annotation. It reuses existing polyp masks, builds weak region masks for other findings with a GI-pretrained classifier and Grad-CAM, and adds terminology-free visual descriptions of the relevant region. Across three sub-2B backbones trained with low-rank adapters, the multi-task recipe beats matched VQA-only training, with the largest gains on appearance and spatial questions and on harder multi-aspect questions. A separate analysis of internal token-to-patch similarity shows stronger alignment between answer words and the true image region, including on an out-of-distribution polyp dataset the models never trained on.","feed_headline":"Small GI VLMs get more accurate by learning to look","feed_subtitle":"Cheap masks and visual descriptions lift answer quality and token-to-region alignment on endoscopy VQA.","key_machinery":"The multi-task recipe: expert polyp masks where available, Grad-CAM weak masks from a GastroNet-initialized classifier for other findings, and Gemma-generated terminology-free visual descriptions, all used as auxiliary targets under matched QLoRA fine-tuning against a VQA-only baseline.","core_discovery":"Jointly training small VLMs on GI VQA plus phrase-grounded localization and terminology-free visual description yields consistent answer-accuracy gains over VQA-only fine-tuning and improves implicit cross-modal alignment between answer tokens and the clinically relevant image region, on both in-distribution and out-of-distribution data.","pith_inferences":["If weak-mask quality is the main risk, the same recipe should transfer cleanly to any finding class that already has a reliable classifier but no pixel labels.","Architecture-sensitive gains (stronger on Florence-2 and InternVL than Qwen) suggest spatial token design, not just task mix, may gate how much grounding supervision sticks.","The same cheap auxiliary construction could be a practical regularizer for other medical VQA settings where full grounded conversation datasets are too expensive to collect."],"forward_implications":["Small, single-GPU VLMs can be preferred for grounded GI VQA without sacrificing answer quality relative to VQA-only fine-tuning.","Appearance and spatial clinical questions, and multi-aspect questions, should improve most when localization and description auxiliaries are added.","Implicit answer-token to image-patch alignment can serve as an audit signal for trustworthiness even when the model is not asked to output a box or mask at inference.","Segmentation and description auxiliaries play complementary roles: description drives in-distribution concentration; localization generalizes better to unseen image sources."],"fun_headline_variants":["Multi-task fine-tuning lifts small GI VQA accuracy and token-region alignment","Cheap masks plus descriptions beat VQA-only tuning on small endoscopy VLMs","Grounding and description tasks improve GI VQA answers and implicit alignment","Small VLMs gain VQA accuracy from joint localization and visual description","Multi-task recipe improves answer quality and cross-modal alignment in GI VQA"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The weak Grad-CAM masks for non-polyp findings are good enough localization targets that the measured gains really come from teaching the model to look at the right tissue, not from noisy or merely class-correlated heatmaps.","fun_headline_variants_meta":{"raw":{"variants":["Multi-task fine-tuning lifts small GI VQA accuracy and token-region alignment","Cheap masks plus descriptions beat VQA-only tuning on small endoscopy VLMs","Grounding and description tasks improve GI VQA answers and implicit alignment","Small VLMs gain VQA accuracy from joint localization and visual description","Multi-task recipe improves answer quality and cross-modal alignment in GI VQA"]},"model":"grok-4.5","effort":"low","cost_usd":0.003367,"raw_usage":{"total_tokens":1115,"prompt_tokens":722,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":33668000,"prompt_tokens_details":{"text_tokens":722,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":310,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":722,"tokens_out":83,"duration_ms":6529,"temperature":1.0,"reasoning_tokens":310,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T11:29:27.414855+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A human spot-check of the Grad-CAM masks against expert outlines for non-polyp findings, or a controlled re-run that replaces those weak masks with true expert masks or random masks and checks whether VQA gains and token-to-patch alignment collapse.","supporting_citations":[],"review_version":1}