{"id":"14cab243-0dfc-4910-9e14-72cf19a4832e","arxiv_id":"2506.09958","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Kvasir-VQA-x1 expands Kvasir-VQA with 159,549 LLM-generated question-answer pairs stratified into three complexity levels, plus a robustness track using weakly augmented images.","lead":"This paper introduces Kvasir-VQA-x1, a large dataset of 159,549 question-answer pairs for medical visual question answering in gastrointestinal endoscopy, built on the existing Kvasir-VQA resource. It adds complex, LLM-generated reasoning questions, visual augmentations for robustness testing, and fine-tuned vision-language model baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The new QA pairs' correctness is the load-bearing premise; Section 2.3's 'expert validation' is undocumented (who, how many, agreement), so the benchmark's ground truth—and every accuracy computed on it—is unverified.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the correctness of LLM-generated and supposedly expert-validated QA pairs. I agree that this is the deepest issue because the dataset is the paper's central contribution, and all reported model accuracies inherit whatever noise is in the labels. The proposed clinician audit would settle it directly: low error rates and acceptable inter-annotator agreement would materially support the 'meticulously curated' claim, while high error rates would invalidate the benchmark's ground truth. I do not see a separate concern that changes the verdict. The open release of code, data, splits, and augmentation scripts is real evidence of reproducibility and a strength of the paper. The LLM-as-judge evaluation is acknowledged by the authors as potentially biased (Section 4.2.5), and that remains a secondary caveat on the empirical headline rather than the primary load-bearing premise. The reported 87% and 90% mean accuracies in Section 4.2.1 are not obviously reproduced from Table 7's per-category values, but that is an aggregation-reporting issue that could be clarified by the authors and does not itself change the conditional accept. The paper is a useful community resource if the validation audit passes; until then, the central claim about meticulous curation is unverified and the verdict should remain CONDITIONAL.","tokens_in":19506,"tokens_out":9284,"duration_ms":100978,"concrete_test":"Select 400 QA pairs stratified by complexity and question class; have two GI clinicians independently mark each against the source image and original atomic QA pair(s) as (1) answerable from the image, (2) clinically correct, and (3) entailed by the original QA. Pre-register an error-rate threshold (e.g., more than 5% non-entailed or clinically incorrect answers invalidates the validation claim), and report counts and Cohen's kappa. This directly tests whether the undocumented 'expert validation' in Section 2.3 actually secured the ground truth.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's primary contribution is a benchmark, so the central claim stands or falls on whether the 159,549 LLM-generated QA pairs are clinically correct and answerable from the image. Section 2.3 describes a Qwen3-30B-A3B pipeline for question merging and answer naturalization and says the output was 'followed by expert validation,' but provides no validation protocol: no information on who validated, how many pairs were reviewed, what exclusion or inclusion criteria were used, whether disagreements were resolved, or any agreement metric. Section 7 merely thanks medical experts. Because naturalized answers are explicitly required not to copy the raw atomic answers, the LLM could introduce clinical content that is not entailed by the original expert QA pairs (e.g., a specific polyp size or location), and a shallow review would not catch systematic errors. If ground-truth labels contain such errors, the complexity stratification and the fine-tuning gains in Section 4 are all measured against noisy targets, and the 'meticulously curated' claim is not established. This is not a disagreement with consensus; it is an unsupported empirical premise that a concrete audit can settle. The paper's own limitation on 'Homogeneity bias in LLM-as-a-Judge' (Section 4.2.5) reinforces that the evaluation layer is also vulnerable, but the dataset labels are the deeper issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Kvasir-VQA-x1, an expanded MedVQA dataset for gastrointestinal endoscopy built on Kvasir-VQA. It adds 159,549 new question-answer pairs generated with the Qwen3-30B-A3B language model by merging one, two, or three atomic QA pairs and naturalizing answers; each pair is assigned a complexity score (1–3) equal to the number of merged atomic pairs. The dataset also defines a robustness track using weak image augmentations (cropping, rotation, affine transforms, color jitter) and is released with code and data on GitHub and Hugging Face. The authors fine-tune MedGemma and Qwen2.5-VL with LoRA and evaluate them on original and transformed images using n-gram metrics and an LLM-based adjudicator, reporting that fine-tuning raises accuracy from roughly 30–45% to 87–90%.","tokens_in":19745,"tokens_out":5512,"duration_ms":61968,"significance":"If the dataset construction is sound, Kvasir-VQA-x1 would be a substantial community resource: it is large, publicly released, FAIR-aligned, stratified by a complexity score, and includes a robustness track with reproducible augmentation scripts. The paper is transparent about its pipeline and releases code, which supports reproducibility. The main claims—that the new QA pairs are clinically correct and that the complexity score reflects reasoning difficulty—are not yet established because the expert-validation step is undocumented and the complexity score is defined only as the number of merged atomic questions. The empirical evaluation is also weakened by an LLM adjudicator from the same family as the generator and one of the evaluated models, and by the absence of error bars and external comparisons. These issues are fixable with additional validation and analysis, so the work is a plausible candidate for a valuable benchmark, but the current manuscript does not fully support its central contribution.","major_comments":[{"comment":"The manuscript states that LLM-generated merged questions and naturalized answers were 'followed by expert validation' (Section 2.3) and thanks medical experts (Section 7), but it gives no details on the validation protocol: who the experts were, how many participated, how many QA pairs were reviewed, what inclusion/exclusion criteria were used, how disagreements were resolved, or any inter-annotator agreement metric. Since the 159,549 new QA pairs are the core dataset contribution and their correctness is the ground truth for all accuracy numbers in Section 4, this missing support is load-bearing. Please provide a detailed validation protocol and, ideally, a quantitative audit (e.g., number of pairs reviewed per expert, agreement rate, examples of corrected outputs), or at least a precise description of what 'expert validation' actually covered.","section":"Section 2.3"},{"comment":"Section 1.4 introduces 'quantitative complexity scores, capturing both visual and linguistic difficulty,' but Section 2.3 defines the complexity score as the number of atomic QA pairs merged (1, 2, or 3), and Table 4's per-class distributions follow exactly from that combinatorial sampling. The paper's own finding in Section 4.2.3 that Level 2 questions are sometimes answered more accurately than Level 1 (e.g., abnormality color: 67.16% vs. 56.69%) indicates that the score does not behave as a monotonic difficulty measure. Either revise the claim about what the score captures or provide evidence—such as human difficulty ratings or a separate visual- and linguistic-difficulty model—that the number of merged pairs tracks reasoning difficulty.","section":"Section 2.3 vs. Section 1.4"},{"comment":"The evaluation uses Qwen/Qwen3-30B-A3B as the LLM adjudicator, which is the same model family used in Section 2.3 to generate the QA pairs, and one of the evaluated models (Qwen2.5-VL) shares the Qwen lineage. The paper acknowledges this 'homogeneity bias' in Section 4.2.5 but still reports Qwen2.5-VL-ft accuracies of up to 90% as evidence of dataset effectiveness. Because the adjudicator's binary per-aspect scores underlie all categorical accuracies in Table 7, and no human-agreement study or independent-judge check is reported, the absolute accuracy numbers for Qwen-family models are not yet trustworthy. Please add at least one of: a human-annotated agreement study on a sample, an ensemble of heterogeneous adjudicators, or a sensitivity analysis showing that the relative model rankings are stable across judges.","section":"Section 2.5 and Section 4.2.5"},{"comment":"Section 4.2.1 states that MedGemma-ft and Qwen2.5-VL-ft achieve 'mean accuracies of 87% and 90%', but Table 7 reports per-category accuracies without an overall mean row, and Table 5 reports eval accuracies of 84.97%, 85.91%, and 85.78% on a 1% held-out subset, which do not match the stated numbers. Please state exactly how the 87%/90% means are computed, including the aggregation formula over aspects and complexity levels, and reconcile these numbers with Table 5. Without a traceable computation, the headline result cannot be verified.","section":"Section 4.2.1 vs. Tables 5 and 7"},{"comment":"Section 2.2 says 'Each entry in the dataset stores the original or augmented image, the newly formulated question, a naturalized answer, a complexity score, and the original question-answer pair(s),' but Section 2.3 says 'The released dataset only includes the original images, associated QA pairs, and metadata' and that augmented images are generated via scripts. This ambiguity affects the Transformed Setting: are the QA pairs in the transformed track identical to those in the original track with the image swapped, or were some QA pairs generated from augmented images so that their content depends on the augmentation? Please clarify the relationship between the two tracks and ensure that the release format and scripts make this unambiguous.","section":"Section 2.2 vs. Section 2.3"}],"minor_comments":[{"comment":"References [18] and [19] are the same CLEF paper, and references [34] and [35] are both Med-Flamingo; please deduplicate and cite each work once.","section":"References"},{"comment":"The text contains spacing artifacts such as 'T ransformed' and 'F AIR' (also in Section 3); these should be corrected for a polished final version.","section":"Abstract and Section 1"},{"comment":"The 'finding presence' category appears only at Level 1, with no Level 2 or 3 rows; the accompanying discussion explains this, but it would help to add a footnote in the table itself so readers can see the reason without searching the text.","section":"Table 4"},{"comment":"The text says 'standardized hyperparameters' but Table 3 shows different effective batch sizes (36 for MedGemma, 32 for Qwen2.5); please rephrase or clarify which hyperparameters were held constant.","section":"Section 2.6"},{"comment":"The dataset structure lists 'img id' but does not specify whether this ID matches the original Kvasir-VQA image IDs or uses a new mapping; please clarify to avoid ambiguity for users who want to cross-reference the original dataset.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The dataset is potentially valuable and the authors have done substantial work in releasing code and data. My main concern is the undocumented expert-validation step, which is the linchpin of the dataset's ground truth. The evaluation also has a self-referential flavor (Qwen-based generation, Qwen-based judging, Qwen-family model evaluated), and the claimed 87%/90% means are not directly traceable to the tables. These are fixable with additional validation details and a small human or independent-judge study. The complexity-score definition also needs to be aligned with the paper's claims. I recommend major revision rather than rejection because the core resource appears useful and the issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a real dataset contribution, not a method paper. It roughly triples Kvasir-VQA with 159,549 LLM-generated QA pairs, adds a complexity stratification (1–3) and a weak-augmentation robustness track, and ships code, data, and training/evaluation scripts. That is a useful community resource, and the FAIR release is genuinely good practice.\n\nThe construction pipeline is described in enough detail to be reproducible: group atomic QA pairs, sample 1–3, merge and naturalize with Qwen3-30B-A3B, then assign a complexity score equal to the number of merged pairs. Fine-tuning MedGemma and Qwen2.5-VL with LoRA is standard, and the reported gains (roughly 30–45% to 87–90% on the LLM-adjudicated accuracy) are dramatic but not implausible for in-domain fine-tuning. The paper also honestly flags its own homogeneity bias in using a Qwen-based judge for Qwen-lineage models.\n\nThe soft spot is exactly where the stress-test says it is: the correctness of the generated ground truth. Section 2.3 says the output was followed by \"expert validation,\" but there is no protocol: who validated, how many pairs, what criteria, what exclusion rate, or any agreement metric. Section 7 just thanks the experts. Because the naturalized answers are explicitly required not to copy atomic answers, the LLM could introduce content not entailed by the original expert QA pairs, and a shallow review would not catch systematic errors. This is not a manufactured flaw—it is a missing audit trail on the central artifact. Complexity scores that merely count merged pairs may also not capture reasoning difficulty, which the authors partially concede in the \"synthesis sweet spot\" discussion.\n\nOther issues are minor in comparison: no error bars, no external benchmark comparison, and the robustness track uses only weak augmentations where gains versus clean training are marginal (0.001–0.002 differences). None of that sinks the dataset's value, but it caps what the empirical section can claim.\n\nWho this is for: anyone building or evaluating MedVQA models in endoscopy, and anyone studying LLM-assisted dataset generation. It deserves a serious referee, but the referee should push hard for a validation audit—ideally a sample of the new QA pairs reviewed by independent clinicians with reported agreement, plus a clear exclusion policy. If that audit is supplied, the dataset is a solid contribution; without it, the benchmark's numbers rest on trust.\n\nRecommendation: engage with it, but require the validation protocol before accepting the empirical claims.","headline":"A genuinely useful GI-endoscopy VQA dataset whose ground truth is the load-bearing assumption: the 'expert validation' is undocumented, so the benchmark's correctness is unverified, but the resource merits a serious referee if that audit is provided.","tokens_in":20274,"tokens_out":1193,"would_cite":true,"duration_ms":17799,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that Kvasir-VQA-x1 turns GI-endoscopy VQA from simple recall into a reasoning benchmark, and that fine-tuning on its 159,549 complexity-scored QA pairs lifts MedGemma and Qwen2.5-VL from 30–45% to 87–90% accuracy.","keywords":["medical visual question answering","gastrointestinal endoscopy","multimodal dataset","vision-language models","visual question answering","visual perturbations","robustness benchmark","complexity scoring"],"falsifier":"Sample, say, 200 Level 2 and Level 3 question-answer pairs that were not used in any training run, have two or more independent gastroenterologists answer each from the image alone, and measure agreement with the dataset's labels; if a substantial fraction are judged unanswerable or answered differently across experts, the claim that the new pairs are clinically valid and image-grounded fails.","tokens_in":19279,"feed_emoji":"🩺","tokens_out":5311,"duration_ms":55296,"temperature":0.7,"pith_summary":"The paper is trying to establish that the main barrier to reasoning-capable medical VQA is not the models but the datasets, and it builds a dataset that targets that barrier for gastrointestinal endoscopy. It expands the Kvasir-VQA corpus from 58,849 to 159,549 QA pairs by combining one, two, or three atomic questions into single prompts answered in naturalized clinical language, and it scores each new prompt by reasoning complexity from 1 to 3. It also supplies a robustness track of weakly perturbed images meant to mimic real endoscopy artifacts, with separate evaluation tracks for standard accuracy and robustness. The empirical case is that domain fine-tuning on this resource transforms off-the-shelf vision-language models: MedGemma reaches 87% and Qwen2.5-VL reaches 90% mean accuracy, against 30–45% for their base checkpoints. A sympathetic reader should care because the paper is aiming to give the field a harder, clinically oriented benchmark on which future MedVQA progress can be measured.","feed_headline":"159,549 new GI endoscopy QA pairs push MedVQA past simple recall","feed_subtitle":"Fine-tuning on the expanded benchmark lifts accuracy from 30–45 percent to 87–90 percent, with robustness tests included.","key_machinery":"The load-bearing mechanism is a three-level complexity scaffold built by combinatorial question merging. Starting from atomic QA pairs, sets of 1, 2, or 3 pairs are sampled; a local Qwen3-30B-A3B server naturalizes answers and merges questions into a coherent prompt, and the number of merged pairs becomes the integer complexity score from 1 to 3. This scaffold is what makes the dataset more than a larger pile of examples: it creates a graded sequence from factual recall to multi-hop synthesis, and it also powers the paper's LLM-as-adjudicator evaluation, which scores each answer aspect, such as polyp type or instrument presence, as correct or incorrect rather than relying on n-gram overlap alone.","core_discovery":"On its own terms, the central discovery is that a curated expansion of an existing GI-endoscopy VQA dataset can turn a simple pattern-recognition benchmark into a reasoning benchmark without abandoning image-answerable ground truth. The dataset is built from 6,500 images from HyperKvasir and Kvasir-Instrument and the 58,849 atomic QA pairs of Kvasir-VQA; a local Qwen3-30B-A3B backbone merges up to three atomic pairs into one complex question and naturalizes the short answers into fluent medical prose, and the paper states that the pipeline's output was expert-validated. Each new pair carries a complexity score from 1 to 3 and references its original atomic pairs, which allows evaluation to be stratified by reasoning depth. The same images are weakly augmented with random resized cropping, rotation, affine shifts, and color jitter to produce a transformed robustness track. Fine-tuned MedGemma and Qwen2.5-VL checkpoints score 87% and 90% mean accuracy on LLM-adjudicated aspects, versus 30–45% for the base models, and the augmented-trained Qwen variant loses almost nothing on clean images while remaining stable on perturbed ones.","pith_inferences":["A testable extension is to use the complexity scores for curriculum learning, starting with Level 2 questions as the paper's own 'synthesis sweet spot' observation suggests, and comparing against curricula that begin with Level 1.","The LLM-as-adjudicator protocol inherits a homogeneity bias because the judge and the best-performing model share architectural lineage; an independent human-scored sample would strengthen any cross-model comparison.","Because augmented images are released only as regeneration scripts, a reproducibility test is to regenerate the exact transformed train/test splits and verify that the reported scores replicate.","The heavy class imbalance, with rare classes like landmark color having only hundreds of pairs, will push future work toward targeted augmentation or resampling rather than simply scaling total QA count."],"forward_implications":["Fine-tuning on Kvasir-VQA-x1 raises MedVQA accuracy from 30–45% to 87% for MedGemma and 90% for Qwen2.5-VL, showing that in-domain data, rather than scale alone, drives clinical competency.","Training on weakly augmented images keeps performance stable on both clean and perturbed inputs, with no measured trade-off on original images.","Complexity scoring enables stratified evaluation, and Level 2 questions often score above Level 1, suggesting that merged prompts can be less ambiguous than isolated atomic questions.","Level 3 questions remain the hardest, with errors compounding under all-or-nothing scoring, so the benchmark sets a clear target for multi-hop reasoning.","The dual-track normal/transformed evaluation provides a repeatable protocol for testing robustness to endoscopy-like imaging artifacts."],"supporting_citations":[{"why":"Supplies the original Kvasir-VQA dataset with 6,500 images and 58,849 atomic QA pairs that Kvasir-VQA-x1 expands.","marker":"[12]"},{"why":"Supplies the HyperKvasir source images and findings that underpin the endoscopic content.","marker":"[6]"},{"why":"Supplies Kvasir-Instrument source images of endoscopic tools used as additional visual material.","marker":"[23]"},{"why":"The Qwen3-30B-A3B model is the backbone used for question merging and answer naturalization.","marker":"[44]"},{"why":"Defines Qwen2.5-VL, one of the two vision-language models fine-tuned and evaluated on the dataset.","marker":"[4]"},{"why":"Defines MedGemma, the other fine-tuned model whose base accuracy is compared against its fine-tuned version.","marker":"[13]"},{"why":"Represents the small-scale radiology VQA benchmark that motivates the need for a larger, GI-specific resource.","marker":"[24]"},{"why":"Represents the large-scale but radiology-heavy benchmark whose QA-generation noise motivates expert validation.","marker":"[53]"},{"why":"Supplies BLEURT, a learned metric used to assess answer quality beyond n-gram overlap.","marker":"[42]"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning method used to adapt both models.","marker":"[20]"}],"fun_headline_variants":["159k new GI endoscopy QA pairs target clinical reasoning","GI endoscopy VQA dataset adds 159k reasoning QA pairs","159k new GI QA pairs push MedVQA from recall to reasoning","GI endoscopy MedVQA benchmark gains 159k complex QA pairs","Robust MedVQA: 159k new GI QA pairs test reasoning under artifacts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands or falls on whether the new merged questions and naturalized answers are genuinely clinically correct and answerable from the image, and the paper reports that 'expert validation' happened without saying who did it, how many pairs were checked, or what the agreement was.","fun_headline_variants_meta":{"raw":{"variants":["159k new GI endoscopy QA pairs target clinical reasoning","GI endoscopy VQA dataset adds 159k reasoning QA pairs","159k new GI QA pairs push MedVQA from recall to reasoning","GI endoscopy MedVQA benchmark gains 159k complex QA pairs","Robust MedVQA: 159k new GI QA pairs test reasoning under artifacts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000861,"raw_usage":{"total_tokens":3798,"prompt_tokens":1072,"completion_tokens":2726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":2631}},"tokens_in":688,"tokens_out":2726,"duration_ms":18993,"temperature":1.0,"reasoning_tokens":2631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:36:19.684934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample, say, 200 Level 2 and Level 3 question-answer pairs that were not used in any training run, have two or more independent gastroenterologists answer each from the image alone, and measure agreement with the dataset's labels; if a substantial fraction are judged unanswerable or answered differently across experts, the claim that the new pairs are clinically valid and image-grounded fails.","supporting_citations":[{"cited_title":"Hicks, Vajira Thambawita, Enrique Garcia-Ceja, Michael A","cited_arxiv_id":null,"evidence_quote":"Supplies Kvasir-Instrument source images of endoscopic tools used as additional visual material."},{"cited_title":"Medgemma hugging face, May 2025","cited_arxiv_id":null,"evidence_quote":"Defines MedGemma, the other fine-tuned model whose base accuracy is compared against its fine-tuned version."}],"review_version":1}