{"id":"5b9b4704-0590-47e9-ab9d-282e33443614","arxiv_id":"2508.18687","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RoMed and CCL: a 144k-question perturbation benchmark for medical VQA and a consistency-plus-contrastive training method that improves LLaVA-Med's accuracy and reduces answer variation.","lead":"A new dataset and training method aim to make medical visual question answering systems give the same answer when the same question is rephrased. The method reports large gains on the new benchmark, but the evaluation is in-distribution and the headline 50% consistency improvement is not clearly supported by the reported numbers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RoMed's GPT-4o-only validation and same-pipeline train/test make the claimed robustness gains potentially distribution-specific rather than general.","rationale":"The reader's weakest assumption correctly identifies the over-reliance on GPT-4o validation and the in-distribution nature of RoMed. This is the most load-bearing concern because the paper's central claim is about robust consistency, and the only evidence for that robustness is measured on a dataset constructed and validated by the same automated pipeline used to generate both training and test perturbations. If the perturbation distribution does not match natural clinical rephrasing variability, or if GPT-4o's validation is flawed, the observed gains could be artifacts of the generation process rather than genuine semantic robustness. The paper presents no human evaluation of the perturbations, no OOD evaluation, and no analysis of how the perturbations differ from natural language variation. Additionally, the abstract's '50% improvement in answer consistency' is not directly supported by the reported MAD/CV numbers, which show ~15-20% reductions; the 50% figure seems to refer to a different quantity (reduction in the accuracy drop), which is less directly about answer consistency. This ambiguity, together with the lack of OOD testing, makes the central claim insufficiently supported. However, the method does show consistent gains on standard benchmarks and the in-distribution comparison to SFT is informative; therefore, the appropriate verdict is conditional, requiring additional OOD and human-validation evidence before the robustness claim can be fully accepted.","tokens_in":8481,"tokens_out":5660,"duration_ms":61059,"concrete_test":"Create an out-of-distribution paraphrase test set: have three human clinicians independently paraphrase a random sample of 200 questions from Rad-VQA, SLAKE, and PathVQA (or use a different LLM-based paraphraser not involved in RoMed). Evaluate LLaVA-Med (both backbones, with and without CCL) on this OOD set, measuring accuracy and per-cluster answer consistency metrics (e.g., MAD/CV or answer agreement). If CCL's improvement over vanilla SFT on OOD rephrasings is not significant (e.g., less than half the effect seen on RoMed), the 'enhanced robustness' claim is distribution-specific and does not support real-world generalization.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that CCL improves robustness rests on RoMed's perturbations being semantically equivalent to the originals with identical answers, yet Section 2.1 states this is validated solely by GPT-4o. No human evaluation, no inter-annotator agreement, and no external verification is reported. If GPT-4o is over-permissive, the dataset may contain rephrasings that are not truly equivalent or that have answer drift, so training the consistency loss to predict the original answer for these variants could artificially inflate both consistency and accuracy on RoMed. Moreover, RoMed train and test are generated by the same pipeline, so the evaluation is in-distribution with respect to the perturbation process. The comparison to vanilla SFT (Fig. 5) shows CCL helps on this specific distribution, but it does not demonstrate transfer to real clinical rephrasings. Table 2 shows CV reductions of roughly 15-20%, not the '50%' claimed in the abstract; the claimed 50% appears to refer instead to the reduction in the accuracy drop, which is a different and less direct measure of consistency. Thus the headline consistency improvement is ambiguous and may not generalize.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the fragility of medical vision-language models (Med-VLMs) under semantically equivalent question rephrasings. The authors construct RoMed, a dataset of 144k perturbed questions derived from Rad-VQA, SLAKE, and PathVQA using a multi-agent pipeline (HuatuoGPT-Vision, HuatuoGPT-o1, GPT-4o) with word-, sentence-, and semantic-level perturbations. They report accuracy drops of existing Med-VLMs on RoMed and propose Consistency and Contrastive Learning (CCL), which adds a consistency loss (standard autoregressive LM loss on perturbed inputs) and a contrastive loss on hidden representations. Experiments compare LLaVA-Med backbones with and without CCL on the three base benchmarks and on RoMed, reporting state-of-the-art accuracy and improved robustness as measured by MAD and CV. The paper also includes ablations, a comparison to vanilla SFT, representation visualizations, and a data-scaling study.","tokens_in":8790,"tokens_out":4666,"duration_ms":57513,"significance":"The paper targets a real and important limitation of Med-VQA systems: inconsistent answers across equivalent phrasings. If the RoMed construction and the consistency metrics were properly validated, the dataset and the CCL training strategy would be useful contributions to trustworthy medical AI. The paper gives explicit credit to the multi-agent perturbation design, the inclusion of both accuracy and consistency metrics, and the ablation separating consistency and contrastive components. The main experiments show consistent improvements over the LLaVA-Med baseline. However, the central robustness claim depends on the validity of the perturbation pipeline and on the numerical treatment of free-form answers; both currently have gaps that prevent the evidence from being fully load-bearing.","major_comments":[{"comment":"The semantic equivalence of the perturbed questions is validated only by GPT-4o, i.e., the same model family that generated them. No human evaluation, inter-annotator agreement, or independent verification is reported. Moreover, the RoMed train and test sets are generated by the same multi-agent perturbation process, so the test distribution is in-distribution with respect to the perturbation style. Consequently, the consistency gains on RoMed could reflect fitting that particular perturbation distribution rather than general robustness to real clinical rephrasings. Please add human or independent-model validation, report agreement, and evaluate on a held-out perturbation source (e.g., human-written rephrasings or a different LLM generator).","section":"§2.1 (RoMed construction)"},{"comment":"MAD and CV are defined as mean absolute deviation and coefficient of variation over model answers x_i, but the paper does not specify how free-form text answers and binary (yes/no) answers are converted to numeric values. Without a defined numeric encoding and a rule for handling non-numeric or unanswerable outputs, these metrics are not well-defined, and the reported reductions in CV/MAD in Table 2 are difficult to interpret. Please specify the encoding, the treatment of open-ended vs. closed-ended answers, and whether the metrics are computed per cluster or across all questions.","section":"§2.1 (MAD and CV definitions)"},{"comment":"The abstract claims 'improves answer consistency by 50%' on RoMed, but Table 2 shows CV reductions of roughly 9–30% (e.g., LLaVA-Med StableLM: RAD CV 83.9→68.3, SLAKE 65.3→57.6, Path 96.0→67.7). The 50% figure appears to refer instead to the reduction in the accuracy drop between the original benchmark and RoMed, which is a different, less direct measure. The headline is therefore ambiguous and should be re-stated with the exact metric, and both the accuracy-drop reduction and the CV/MAD changes should be reported consistently.","section":"Abstract and Table 2"},{"comment":"All tables report a single run per configuration, with no error bars, multiple seeds, or statistical significance tests. Given the batch size of 2 and the joint loss with new hyperparameters (temperature τ and loss weighting), training stochasticity is non-trivial. The claim of 'notably improves' consistency and accuracy would be stronger with at least 3–5 seeds and reporting of mean±std, especially for the small gains in Table 1 and the ablation in Table 3.","section":"§3 (Experiments)"}],"minor_comments":[{"comment":"The 'knowledge-anchored' consistency loss is simply the standard autoregressive LM loss applied to the original and perturbed inputs. The term overstates the mechanism; no explicit medical knowledge source is injected. Similarly, 'bias-aware' contrastive learning is not supported by any direct measurement of bias.","section":"§2.2"},{"comment":"The total loss L = (L_ctr + L_consistency)/2 weights L_consistency, which already contains four LM terms, more heavily than L_ctr. This weighting choice is not motivated or ablated; please clarify the intended contribution of each term.","section":"Eq. for total loss"},{"comment":"The comparison between SFT and CCL is informative, but the caption of Table 4 says 'CCL++ indicates training with doubled dataset size' while the text says 'generating two additional variations per level'. Please align the description with the actual data construction.","section":"Figure 5 / Table 4"},{"comment":"There are typographical and consistency issues: 'Sinapore' (affiliation), 'RAD-VQA' vs 'Rad-VQA' inconsistent usage, and references [20] and [21] appear to be the same work. Also, Fig. 4(e,f) shows t-SNE-like visualizations without a quantitative separation measure; please report a numeric embedding-distance metric.","section":"General presentation"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is promising, but the evidence for the headline 50% consistency improvement is not yet solid. The authors should be asked to provide a human-validated or independent perturbation test set, a clear numeric encoding for MAD/CV, and multi-seed results. If the consistency gains remain only on in-distribution perturbations, the robustness claim should be tempered accordingly. The dataset release and validation details will be essential before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper earns its keep. It builds a large perturbed-question benchmark for medical VQA (RoMed, 144k questions with word/sentence/semantic variations) and a training method (CCL) that combines consistency-style training on paraphrases with supervised contrastive learning. On three standard benchmarks and on RoMed itself, the method moves accuracy up noticeably across two different LLaVA-Med backbones, and the ablation shows both loss terms matter. The SFT comparison is the most convincing piece: training on the same perturbed data with the ordinary objective gives almost nothing, while CCL gives a big jump. That supports the claim that the objective, not just the extra data, is doing the work.\n\nWhere I'd push back: the abstract says \"improves answer consistency by 50%,\" but Table 2 shows CV reductions of roughly 15–20%. The 50% refers to the drop in recall on the original benchmark, which is a different and weaker statement. That is a wording problem, not necessarily a hidden failure, but it should be fixed. More substantively, MAD and CV are computed on text answers, and the paper never says how answers are mapped to numbers. Without that encoding, those metrics are undefined as written. The dataset validation also rests entirely on GPT-4o, with no human check on a sample; combined with the fact that train and test perturbations come from the same multi-agent pipeline, the evaluation is measuring robustness to this particular perturbation style rather than robustness to real clinical rephrasings. The gains on the standard benchmarks are reassuring, but the RoMed test is in-distribution by construction.\n\nMinor points: no error bars or seed variance, though the gains are large enough that this is probably not masking a null result. The contrastive learning uses in-batch negatives; the paper reports batch size 2 per GPU, but the effective batch (with 8 GPUs) is never stated, so it is hard to tell how strong that supervision is. The \"knowledge-anchored\" description oversells what is actually just a language-model loss on knowledge-generated paraphrases.\n\nWho should read it: anyone building or evaluating medical VQA systems, and people who care about robustness benchmarks for multimodal LLMs. It deserves a serious referee: the empirical contribution is real, the flaws are addressable, and the dataset could be reused. Send it to review with requests for the answer encoding, error bars, and a small human validation of the perturbation pipeline.","headline":"Useful robustness benchmark and training recipe for Med-VQA, with real gains, but the headline 50% consistency claim is overstated and the evaluation is in-distribution with the perturbation pipeline.","tokens_in":9195,"tokens_out":1832,"would_cite":true,"duration_ms":23707,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Medical VQA models answer the same question differently when it is rephrased, and the paper's joint consistency-and-contrastive training objective largely closes that gap.","keywords":["medical visual question answering","vision-language models","answer consistency","perturbation robustness","consistency learning","contrastive learning","RoMed dataset","medical image analysis"],"falsifier":"Take a sample of RoMed clusters and have human clinicians label whether each perturbed question means the same thing and has the same answer as the original; then generate new rephrasings with a different pipeline, or have humans write them, and test whether CCL-trained models stay consistent. If consistency collapses on human-verified or independently generated paraphrases, the reported robustness is an artifact of the original perturbation distribution rather than a general property of the model.","tokens_in":8437,"feed_emoji":"🩺","tokens_out":10754,"duration_ms":79320,"temperature":0.7,"pith_summary":"Medical visual question answering (VQA)—models that answer questions about clinical images—should give the same answer when a question is merely reworded. The paper shows they often do not: word-, sentence-, and semantic-level rephrasings can flip the answer, and adding more training data barely helps. The proposed explanation is that standard fine-tuning rewards surface phrasing rather than shared medical meaning, and that the training data contain syntactic shortcuts. To make the problem measurable, the paper builds RoMed, a dataset of 144k perturbed questions with consistency metrics (mean absolute deviation and coefficient of variation) alongside accuracy. It then proposes Consistency and Contrastive Learning (CCL), which makes a model answer all variants of a question consistently while contrasting unrelated questions, and reports state-of-the-art accuracy on three medical VQA benchmarks plus a roughly 50% improvement in answer consistency on RoMed—the difference between a model that knows and one that guesses.","feed_headline":"New training objective stabilizes medical VQA under rephrasing","feed_subtitle":"Medical VQA models flip answers when a question is rephrased; the paper's training recipe cuts inconsistency by 50%.","key_machinery":"The load-bearing object is RoMed, a perturbation dataset in which each original medical VQA question is expanded into clusters of word-, sentence-, and semantic-level rephrasings generated by a multi-agent pipeline and validated for semantic equivalence. Consistency is quantified by Mean Absolute Deviation (MAD) and Coefficient of Variation (CV) of the model's answers within each cluster. The corrective mechanism is the CCL training objective: a knowledge-anchored consistency loss that applies the standard next-token prediction loss to original plus perturbed questions, and a bias-aware contrastive loss that treats original and perturbed versions as positive pairs and other questions in the","core_discovery":"The paper's central claim is that fragility under rephrasing is an objective problem, not a data-quantity problem: the standard supervised fine-tuning loss teaches a model to predict the next token for one surface form, so it never learns that different wordings of the same clinical question should converge to the same answer. CCL replaces that single objective with a joint one. A knowledge-anchored consistency loss applies the same next-token answer-generation loss to the original question and to word-, sentence-, and semantic-level perturbations, forcing the model to treat them as one task. A bias-aware contrastive loss pulls the hidden representations of original and perturbed questions t","pith_inferences":["If consistency across rephrasings is a valid proxy for understanding, then conventional accuracy benchmarks systematically overstate medical VQA reliability; the same perturbation-and-consistency evaluation could be applied to other high-stakes multimodal settings such as clinical note QA or surgical video QA.","Because one proprietary general-purpose model both generates and validates the perturbations, the evaluation is only as trustworthy as that validator; sampling clusters for human review, or generating rephrasings with several independent models, would test whether the consistency gains transfer to the way clinicians actually rephrase questions.","CCL's contrastive component reshapes hidden representations rather than simply expanding data, so it could combine with other robustness techniques such as preference optimization or chain-of-thought medical reasoning; the paper does not test those combinations.","The observed diminishing returns at two variations per level suggest an economical annotation budget for building robustness datasets in other medical domains: one well-designed perturbation per level captures most of the benefit."],"forward_implications":["Accuracy on original benchmark questions is not enough to certify a medical VQA model; robustness reporting should include consistency metrics such as MAD and CV over rephrased question clusters.","Adding perturbation diversity to the training set alone does not robustify a model; the training objective itself must reward invariance to rephrasing.","CCL is a plug-and-play objective that can be added on top of existing medical VLM fine-tuning and improves both open-ended and closed-ended accuracy on Rad-VQA, SLAKE, and PathVQA.","The benefit of adding more perturbations per question levels off after one variation per level, so the consistency gains are not simply a matter of more training data.","Under CCL, representations of different question variants cluster together, which is the mechanism behind the improved robustness."],"supporting_citations":[{"why":"Supplies the radiology VQA questions and answers that RoMed perturbs and on which CCL is evaluated.","marker":"[16]"},{"why":"Supplies the semantically labeled medical VQA dataset used to build RoMed and to evaluate CCL.","marker":"[19]"},{"why":"Supplies the pathology VQA dataset used to build RoMed and to evaluate CCL.","marker":"[8]"},{"why":"The medical vision-language model baseline onto which CCL is added and against which robustness is measured.","marker":"[17]"},{"why":"The medical multimodal agent that supplies image captions used in generating the perturbation levels.","marker":"[5]"},{"why":"The medical reasoning agent that produces intermediate reasoning steps for perturbation generation.","marker":"[4]"},{"why":"The general meta-agent that generates the three levels of perturbed questions and validates that they preserve meaning and answer.","marker":"[9]"},{"why":"The consistency-learning idea that the knowledge-anchored consistency loss adapts.","marker":"[6]"},{"why":"The supervised contrastive objective that the bias-aware contrastive loss adapts.","marker":"[15]"}],"fun_headline_variants":["Training tweak makes medical VQA models 50% more consistent on rephrased questions","New loss keeps medical VQA answers stable across rephrased questions","Medical VQA models flip answers on rephrased questions; new training fixes it","CCL: a training objective that makes medical VQA consistent under rephrasing"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that RoMed's perturbed questions are semantically equivalent to the originals and share the same answer—a property verified only by a single general-purpose validation model—and separately, the introduction's data-diversity argument contains an unresolved citation placeholder that makes part of the motivation untraceable.","fun_headline_variants_meta":{"raw":{"variants":["Training tweak makes medical VQA models 50% more consistent on rephrased questions","New loss keeps medical VQA answers stable across rephrased questions","Medical VQA models flip answers on rephrased questions; new training fixes it","CCL: a training objective that makes medical VQA consistent under rephrasing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":2882,"prompt_tokens":799,"completion_tokens":2083,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1993}},"tokens_in":543,"tokens_out":2083,"duration_ms":15124,"temperature":1.0,"reasoning_tokens":1993,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:16:51.597825+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of RoMed clusters and have human clinicians label whether each perturbed question means the same thing and has the same answer as the original; then generate new rephrasings with a different pipeline, or have humans write them, and test whether CCL-trained models stay consistent. If consistency collapses on human-verified or independently generated paraphrases, the reported robustness is an artifact of the original perturbation distribution rather than a general property of the model.","supporting_citations":[{"cited_title":"IEEE (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the semantically labeled medical VQA dataset used to build RoMed and to evaluate CCL."},{"cited_title":"Advances in Neural Information Processing Systems36 (2024)","cited_arxiv_id":null,"evidence_quote":"The medical vision-language model baseline onto which CCL is added and against which robustness is measured."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)","cited_arxiv_id":null,"evidence_quote":"The consistency-learning idea that the knowledge-anchored consistency loss adapts."}],"review_version":1}