{"id":"3f5f6198-16d1-438b-91e0-07b4a8005416","arxiv_id":"2507.14544","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Fine-tuning Florence-2 on a 1% subset of Kvasir-VQA with medical image augmentations yields moderate VQA performance on gastrointestinal endoscopy questions.","lead":"This paper describes a system that answers questions about gastrointestinal endoscopy images by fine-tuning a large multimodal AI model (Florence-2) on a small subset of a medical dataset. The authors report moderate accuracy on the challenge's test sets and argue that even limited data can produce clinically relevant answers when augmented carefully.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central accuracy claim rests on an undocumented evaluation protocol: Section 3.1's split description leaves no test set, yet Table 1 reports 'Public' and 'Private' test scores, and early stopping may have selected on those splits.","rationale":"The reader's weakest assumption correctly identifies the ambiguity of the test splits and the missing official challenge metric. My analysis confirms this is the most load-bearing concern: the entire empirical contribution is a set of Table 1 scores, and if those scores come from an ill-defined or test-contaminated protocol, the central claim 'state-of-the-art performance on Kvasir-VQA' collapses. The split description in Section 3.1 is internally inconsistent, and the training protocol's early stopping creates a clear selection channel. I do not find a more fundamental flaw in the modeling approach itself; freezing the vision encoder and fine-tuning the decoder is a reasonable and reproducible strategy. The missing baselines and the unsubstantiated 'state-of-the-art' phrasing are secondary to the evaluation problem, since even a modest non-SOTA result would be acceptable as a challenge-participation report once the evaluation is clearly specified. Therefore the reader's CONDITIONAL verdict remains appropriate: the paper needs a corrected split definition, the official evaluation metric, and an unbiased test evaluation before its numbers can be trusted.","tokens_in":8984,"tokens_out":2397,"duration_ms":29878,"concrete_test":"Reproduce the exact 1% stratified subset and split using the public code repository and the paper's described procedure, train once with the stated hyperparameters (batch size 5, LR 7.8e-6, FP16), and evaluate on a genuinely held-out test split using the official ImageCLEFmed MEDVQA 2025 evaluation script. If the reproduced Public/Private METEOR and BLEU scores differ materially from Table 1, or if the official metric is not BLEU/ROUGE/METEOR, the central claim is unsupported. Additionally, check the training logs for early-stopping decisions to see whether any reported test split influenced checkpoint selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that fine-tuned Florence-2 achieves state-of-the-art performance on Kvasir-VQA with a 1% training subset. Every quantitative result in Table 1 depends on the validity of the 'Public' and 'Private' test sets, but those sets are never defined. Section 3.1 states: 'This subset was initially divided into 90% training and 10% testing. The testing portion was further split into 90% training and 10% validation' — this arithmetic yields a 99/1/0 train/validation/test split, leaving no test set at all. Table 1 nevertheless reports Public and Private test scores, so either the split description is wrong or the test sets were constructed unreported. Additionally, Section 4.5 says evaluation was performed after each epoch with early stopping after 3 epochs without improvement, and the 'Clarification' in Section 5 confirms evaluations at each epoch. If early stopping used the same 'Public' or 'Private' partitions that are later reported, then the reported METEOR 0.49 is a test-selected maximum, not an unbiased estimate. The paper also never identifies the official ImageCLEFmed MEDVQA 2025 metric, instead using BLEU, ROUGE, and METEOR via HuggingFace evaluate. If the challenge metric differs, the headline 'state-of-the-art on Kvasir-VQA' and 'accurate responses on official challenge metrics' are unsupported. No baseline comparison to prior Kvasir-VQA methods appears anywhere, so 'state-of-the-art' is asserted without evidence. These issues do not refute the plausibility of the core fine-tuning finding, but they make the reported numbers untrustworthy as a benchmark result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a fine-tuning pipeline for Subtask 1 of the ImageCLEFmed MEDVQA 2025 challenge, using Florence-2 as the backbone and a 1% stratified subset of Kvasir-VQA. The authors freeze the vision encoder, fine-tune the language decoder, and apply domain-specific augmentations. They report BLEU, ROUGE, and METEOR scores on splits they call 'Validation', 'Public', and 'Private', with the private-test METEOR of 0.49 as the headline result. The paper claims state-of-the-art performance on Kvasir-VQA and accurate responses on official challenge metrics, and it makes the code publicly available.","tokens_in":9228,"tokens_out":6014,"duration_ms":58436,"significance":"If the evaluation were properly documented, this paper would provide a useful, reproducible baseline for low-resource medical VQA: it uses a standard fine-tuning recipe, a frozen vision encoder, publicly available code, and fixed seeds. The augmentation ablation, though currently confounded, addresses a relevant practical question. However, the paper's central claims of state-of-the-art performance and accuracy on 'official challenge metrics' are not supported by the evidence as presented, because the test splits are undefined, the official metric is never identified, and no comparison to prior Kvasir-VQA methods is given. With revised evaluation and more modest claims, the work could serve as a legitimate challenge write-up, but in its current form the load-bearing results cannot be verified.","major_comments":[{"comment":"The split description is internally contradictory and leaves no test set. The text says the 1% subset was 'initially divided into 90% training and 10% testing. The testing portion was further split into 90% training and 10% validation, resulting in a final train/validation/test split.' Under this arithmetic, the original 10% testing portion becomes 9% additional training and 1% validation, with zero samples remaining for testing. Yet Table 1 reports scores on 'Public' and 'Private' test sets, which are never defined in terms of sample counts or construction. Because every reported accuracy number, including the headline METEOR 0.49 on 'Private', depends on these undefined partitions, the authors must specify exactly how the Public and Private test sets were formed, how many samples each contains, and that no model selection (including early stopping) used these partitions.","section":"3.1 (Dataset Description)"},{"comment":"The early stopping procedure creates a risk of optimistic bias. Section 4.5 states 'Evaluation was done after each epoch, with early stopping after 3 epochs without improvement,' and the 'Clarification' in Section 5 confirms that evaluations were performed at each epoch. However, the paper never states which split drove early stopping. If the Public or Private test partitions were used for model selection, the reported numbers are the maximum of a noisy selection process rather than unbiased test estimates. Please report which split was used for early stopping, the epoch at which the Table 1 numbers were obtained, and per-epoch evaluation curves for the validation and test splits.","section":"4.5 and 5 (Training Protocol and Results)"},{"comment":"The claim of 'state-of-the-art performance on the Kvasir-VQA benchmark' is unsupported. No comparison to any prior Kvasir-VQA method appears anywhere in the manuscript; the only comparisons are the paper's own augmentation variants (Table 2). Likewise, the Abstract's 'accurate responses on the official challenge metrics' cannot be checked because the official ImageCLEFmed MEDVQA 2025 scoring metric is never identified. The authors should either (a) add a quantitative comparison to published Kvasir-VQA results on identical splits and metrics, or (b) remove the state-of-the-art and official-metric claims and instead state that the paper reports an evaluation of the proposed pipeline on the authors' own splits.","section":"2 (Related Work) and Abstract"},{"comment":"The augmentation ablation appears confounded with evaluation split. The 'Standard Augmentation' row (BLEU 0.12, ROUGE-L 0.77, METEOR 0.42) exactly matches the 'Validation' row of Table 1, and the 'Fine-Tuned Augmentation' row (BLEU 0.15, ROUGE-L 0.80, METEOR 0.44) exactly matches the 'Public' row of Table 1. The table does not state which split is used for each row. If the augmentation strategies were evaluated on different splits, the improvement cannot be attributed to the augmentation. All augmentation strategies should be evaluated on the same held-out split, and that split should be explicitly identified.","section":"6 (Ablation Studies), Table 2"},{"comment":"The architecture description is inconsistent with the stated base model. Section 4.4 says the authors fine-tuned 'microsoft/Florence-2-base-ft', but Section 4.2 describes a '2.7B parameter causal language model', and Section 4.3 mentions parameter counts of 0.23B (base) and 0.77B (large). Florence-2-base has 0.23B parameters, so the 2.7B figure appears to describe a different model. The authors should correct the architecture description to match the checkpoint actually used and state precisely which components were fine-tuned versus frozen.","section":"4.2 and 4.4 (Architecture and Fine-Tuning Setup)"}],"minor_comments":[{"comment":"This section says 'Kvasir-VQA consists of over 6,500 annotated image-question-answer (IQA) triplets,' but Section 3.1 says the dataset comprises 58,849 triplets associated with 6,500 images. Please clarify whether 6,500 refers to images and 58,849 to triplets, and use the terminology consistently.","section":"2 (Related Work)"},{"comment":"Section 4.5 reports a cosine decay over 20 epochs, while Section 4.6 says the training used a '72-hour time budget over 10 epochs.' These are inconsistent; please align the stated number of epochs and the learning-rate schedule.","section":"4.5 and 4.6 (Training Protocol and Implementation Details)"},{"comment":"The 'Clarification' paragraph within the Results section repeats methodology already presented in Section 4. Consider moving or removing it to improve the paper's flow.","section":"5 (Results and Evaluation), 'Clarification'"},{"comment":"The dataset name is spelled 'KASVIR' in the Abstract and Acknowledgments; it should be 'Kvasir'.","section":"Abstract and Acknowledgments"},{"comment":"Table 2 is introduced by two consecutive 'Table 2:' captions, which is a formatting error. Also, the table caption should state the evaluation split used for each row.","section":"6 (Ablation Studies)"},{"comment":"The sentence 'Baselines were compared using paired t-tests (p < 0.05)' is not accompanied by any reported test statistic, p-value, or description of what baselines were compared. Either report the results or delete the sentence.","section":"4.5 (Training Protocol)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an early challenge write-up. The core fine-tuning experiment is plausible and reproducible, but the unspecified test splits and unsupported state-of-the-art claim are serious issues that prevent verification. I would encourage the authors to run the official ImageCLEFmed evaluation pipeline (if available), report the challenge metric, and add at least one external baseline. Once the evaluation protocol is made transparent and the claims are scaled to the evidence, the paper may become suitable for publication. There is no indication of misuse of sources; the reference list is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a quick look if you work on medical VQA baselines. The paper fine-tunes Florence-2 on a 1% stratified subset of Kvasir-VQA for the MEDVQA 2025 challenge, freezes the ViT backbone, and reports BLEU/ROUGE/METEOR. The core experiment is reasonable and reproducible: code link given, fixed seeds, standard HuggingFace pipeline. The augmentation ablation is the most useful part—no augmentation gives METEOR 0.31, heavy augmentation 0.25, standard 0.42, fine-tuned 0.44 on the validation split. That tells a clean story: simple augmentations help, aggressive ones hurt. The question-type breakdown is also informative (where/have strong, how weak). So there is real value here as a low-resource baseline.\n\nThe soft spots are the usual challenge-report ones, plus one genuinely load-bearing problem. The 'state-of-the-art' claim in Section 2 is asserted with no comparison to any prior Kvasir-VQA method. The official ImageCLEFmed metric is never identified; the paper uses BLEU, ROUGE, METEOR via HuggingFace evaluate, which may or may not match the challenge. More seriously, Section 3.1's split description is internally inconsistent: 90/10 then 90/10 on the test portion leaves no test set, yet Table 1 reports 'Public' and 'Private' test scores. If the intended split was 90/10 train/test and then train split into 90/10, the paper should say so. As written, the reported test numbers are unverifiable. The early stopping rule (evaluate after each epoch, stop after 3 without improvement) also risks selecting on the reported test partitions if those were used for model selection. And there are small inconsistencies: 20 epochs in 4.5 vs 10 epochs in 4.6.\n\nNone of this refutes the plausible finding that a frozen-encoder Florence-2 can produce passable VQA answers from a tiny training set. But the evaluation protocol has to be fixed before Table 1 can be trusted as a benchmark. The authors are transparent about their resource constraints and the limitations of a 1% subset, which I appreciate.\n\nFor peer review: worth sending to a venue that takes challenge reports, with a request for a major revision clarifying splits, adding at least one baseline, and adjusting the claims. For your own work, I'd cite it only as an existence proof of a low-data Florence-2 baseline, not for the absolute numbers.","headline":"A modest, honest low-data fine-tuning report whose main empirical numbers are undercut by an unclear evaluation protocol and an unsupported state-of-the-art claim.","tokens_in":9897,"tokens_out":2999,"would_cite":false,"duration_ms":32972,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned Florence-2 vision-language model answers gastrointestinal endoscopy questions from a 1% training slice, with a private-test METEOR of 0.49.","keywords":["Medical VQA","ImageCLEFmed 2025","Multimodal AI","Clinical Question Answering","Gastrointestinal endoscopy","Florence-2","Kvasir-VQA","Domain-specific data augmentation"],"falsifier":"Run the same fine-tuned checkpoint on the official MEDVQA-GI 2025 Subtask 1 test inputs and score the outputs with the challenge's own evaluation script; if the official metric is not n-gram overlap or the reported public/private BLEU and METEOR values do not reproduce, the central claim fails.","tokens_in":8689,"feed_emoji":"🩺","tokens_out":10710,"duration_ms":111978,"temperature":0.7,"pith_summary":"This paper tries to establish that a large multimodal foundation model can answer clinically relevant questions about gastrointestinal endoscopy images using very little training data. Its route is to freeze Florence-2's pretrained vision encoder, fine-tune its language decoder on a 1% stratified slice of the Kvasir-VQA GI-endoscopy dataset, and add domain-preserving image augmentations. Using standard text-overlap metrics, it reports METEOR scores of 0.42, 0.44, and 0.49 on its validation, public, and private splits, and it claims this recipe reaches state-of-the-art performance on the Kvasir-VQA benchmark. The practical interest, if true, is that a small, reproducible pipeline can serve as a baseline for medical visual question answering under limited data and compute.","feed_headline":"Fine-tuned Florence-2 answers endoscopy questions from 1% of data","feed_subtitle":"With the vision encoder frozen, the model reaches 0.49 METEOR on the MEDVQA-GI 2025 private test split.","key_machinery":"The carrier is Florence-2, a unified vision foundation model that casts every vision-language task as sequence-to-sequence generation from an image plus a prompt. A frozen DaViT/ViT-L/14 vision encoder produces 196 visual tokens per image, and a multi-modal encoder-decoder with a 2.7B-parameter causal language decoder generates the answer, trained with cross-entropy loss on tokenized answer sequences. The paper's added machinery is a small set of domain-aware augmentations, namely random crop, flip, and color jitter selected to keep endoscopy-relevant cues visible, plus temperature sampling at inference with $T = 0.7$. The mechanism works by letting the frozen visual tower supply general representations while the fine-tuned decoder adapts to the 20-question-template, 502-answer linguistic landscape of Kvasir-VQA.","core_discovery":"The central claim is that Florence-2's unified sequence-to-sequence design, which treats a question as a prompt and generates free-text answers, can be adapted to GI-endoscopy VQA by keeping the ViT-L/14 vision tower frozen and fine-tuning the multimodal encoder and 2.7B-parameter causal decoder. The paper reports that fine-tuned augmentation, meaning random crop, flip, and color jitter adjusted to preserve endoscopic features such as mucosal texture and bleeding points, raises METEOR from 0.31 with no augmentation to 0.44 on the public split and 0.49 on the private split, with BLEU 0.16 and ROUGE-L 0.88 on the private split. It also reports that heavy, unrealistic augmentations degrade performance to METEOR 0.25, and that performance is stronger on spatial and binary question types such as 'where' and 'have' than on procedural questions such as 'how'. On these grounds the paper states that its approach yields state-of-the-art performance on the Kvasir-VQA benchmark and provides a strong baseline for the MEDVQA-GI 2025 Subtask 1 challenge.","pith_inferences":["If the reported numbers reproduce under the official challenge metric, the same freeze-the-encoder, fine-tune-the-decoder recipe could transfer to other low-annotation medical imaging domains such as dermoscopy or pathology VQA, though the paper does not test this.","Because Kvasir-VQA has only 20 question templates, the 1% subset may overrepresent common templates; a testable extension is to measure per-template coverage and see whether the reported gains concentrate in high-frequency questions.","A clinician-rated correctness sample of private-test answers would clarify whether the METEOR gains reflect clinically usable answers, since n-gram overlap can reward paraphrases that change medical meaning; the paper reports no such evaluation.","The paper's discussion of unanswerable questions suggests a next step of measuring abstention behavior on out-of-distribution or unanswerable queries, which the current n-gram metrics do not capture."],"forward_implications":["A 1% stratified slice of Kvasir-VQA moves a frozen-encoder Florence-2 from BLEU 0.00 to a private-test METEOR of 0.49, suggesting data-efficient medical VQA is feasible without full dataset access.","Augmentation choice is decisive: domain-tuned standard augmentation improves METEOR from 0.31 to 0.44, while heavy unrealistic distortions drop it to 0.25, so augmentation design is a core component of the pipeline.","The generative decoding approach can handle the dataset's class imbalance and multi-word answers, escaping the fixed-answer-set restriction of classification-based Med-VQA methods.","Performance is uneven by question type: strong on spatial and binary questions such as 'where' (METEOR 0.58) and 'have' (0.77), weak on procedural 'how' (0.37), pointing future work toward reasoning-heavy templates.","These scores provide a reproducible baseline for future MEDVQA-GI submissions, particularly for teams with limited computational resources."],"supporting_citations":[{"why":"Supplies Florence-2, the unified vision-language backbone whose frozen vision encoder and fine-tuned decoder carry the VQA pipeline.","marker":"[20]"},{"why":"Provides the Kvasir-VQA GI endoscopy dataset, including the question templates and answer set the model is trained and evaluated on.","marker":"[9]"},{"why":"Defines the ImageCLEFmed MEDVQA 2025 Subtask 1 challenge whose test conditions the approach targets.","marker":"[13]"},{"why":"Represents the prior state-of-the-art concept-alignment Med-VQA line that the paper claims to surpass on the Kvasir-VQA benchmark.","marker":"[22]"}],"fun_headline_variants":["Florence-2 with frozen vision hits 0.49 METEOR on GI VQA","Domain-specific augmentations boost endoscopy VQA to 0.49 METEOR","Fine-tuned Florence-2 sets Kvasir-VQA baseline at 0.49 METEOR","Frozen encoder, tuned decoder: 0.49 METEOR on medical VQA","Augmentation strategy lifts GI VQA score to 0.49 METEOR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the BLEU, ROUGE, and METEOR numbers computed on the paper's own public and private splits are what the MEDVQA-GI 2025 challenge actually grades, since the paper never names the official scoring metric and gives an ambiguous description of how the test splits were formed.","fun_headline_variants_meta":{"raw":{"variants":["Florence-2 with frozen vision hits 0.49 METEOR on GI VQA","Domain-specific augmentations boost endoscopy VQA to 0.49 METEOR","Fine-tuned Florence-2 sets Kvasir-VQA baseline at 0.49 METEOR","Frozen encoder, tuned decoder: 0.49 METEOR on medical VQA","Augmentation strategy lifts GI VQA score to 0.49 METEOR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1887,"prompt_tokens":949,"completion_tokens":938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":820}},"tokens_in":565,"tokens_out":938,"duration_ms":8666,"temperature":1.0,"reasoning_tokens":820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:53:09.945916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuned checkpoint on the official MEDVQA-GI 2025 Subtask 1 test inputs and score the outputs with the challenge's own evaluation script; if the official metric is not n-gram overlap or the reported public/private BLEU and METEOR values do not reproduce, the central claim fails.","supporting_citations":[{"cited_title":"Hicks, Vajira Thambawita, P ˚ al Halvorsen, and Michael A","cited_arxiv_id":null,"evidence_quote":"Provides the Kvasir-VQA GI endoscopy dataset, including the question templates and answer set the model is trained and evaluated on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ImageCLEFmed MEDVQA 2025 Subtask 1 challenge whose test conditions the approach targets."},{"cited_title":"Multi-modal concept align- ment pre-training for generative medical visual question answering","cited_arxiv_id":null,"evidence_quote":"Represents the prior state-of-the-art concept-alignment Med-VQA line that the paper claims to surpass on the Kvasir-VQA benchmark."}],"review_version":1}