{"id":"c3ea11a3-4663-4d10-8c50-d495d0160211","arxiv_id":"2607.17834","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Vision-language models can answer complex endoscopic questions correctly while failing the same-image atomic sub-questions; EndoCA measures this gap and ASR reduces it.","lead":"This paper introduces EndoCA, a benchmark that pairs complex endoscopy questions with the simple atomic questions they are built from, and shows that vision-language models often answer complex questions correctly while failing the underlying atomic checks. It also proposes a training-free revision-and-abstention method, ASR, that improves the paired consistency score.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EndoCA's scorer (Eq. 3) defines complex-answer correctness as exact match to atomic labels; with no human validation, the headline 'hidden inconsistency' and ASR gains could be artifacts of this label-conformance rule rather than genuine unreliability or improvement.","rationale":"The paper's central claim has two halves: complex-answer accuracy can hide same-image complex-atomic inconsistency, and ASR can reduce that inconsistency. Both halves depend on Eq. 3's operationalization of a 'correct complex answer.' If that score is only label-conformance, then the measured gap between complex and atomic accuracy is built into the metric rather than discovered, and ASR's gains are at least partly metric alignment. This is exactly the reader's weakest assumption, and I agree with it. I considered an alternative concern—train/test contamination for MedGemma-FT and Qwen2.5-VL-FT, since they are LoRA-adapted on Kvasir-VQA-x1 and no split is reported—but that would affect only two model rows and part of the ASR table, whereas scorer validity undermines the whole benchmark and the method's target. The proposed human-kappa test is the one check that would settle whether the concern lands: if clinicians agree with the scorer and ASR gains survive human scoring, the conditional objection is resolved; otherwise the headline numbers are scoring artifacts. Since the reader's verdict is already CONDITIONAL, I am not moving it—I am strengthening the condition already placed on it.","tokens_in":10832,"tokens_out":10759,"duration_ms":122126,"concrete_test":"Human-validation of the EndoCA scorer on ASR outputs: draw 400 random complex outputs per model from Table V's four models (1,600 total), stratified by EndoCA's correct/incorrect score. Two gastroenterologists, blinded to atomic annotations and scorer decisions, judge whether each output answers the complex question, treating clinically valid paraphrases as correct. Compute Cohen's kappa between human judgments and Ci (Eq. 3), then recompute Table V's complex accuracy, joint accuracy, and inconsistency deltas using human-validated Ci (atomic scores unchanged). Pre-register: if kappa < 0.80 or ASR-Revise's average joint-accuracy gain drops by more than 3 points under human scoring, the headline metrics are scorer artifacts; if kappa >= 0.80 and gains persist, the conditional objection is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C defines Ci via SC(Âi, xi) (Eq. 2–3): a complex answer is correct only if the normalized output matches every associated atomic annotation. This is a label-conformance rule, not a judgment of whether Âi actually answers Qi. The scorer normalizes surface variants but not semantic paraphrase, so a clinically correct complex answer that expresses the same components in different words can be marked wrong—and a string containing the canonical labels can be marked right without being a coherent answer to Qi. The central inconsistency metric (Eq. 6) is conditioned on Ci, and ASR-Revise feeds the model its own atomic answers—the exact labels the scorer checks—so Table V's joint-accuracy gains can partly reflect the model learning to emit the scorer's canonical label format rather than improving answer reliability. The manuscript reports no human-agreement study for the scorer; the only internal support is construction-time agreement between the Kvasir-VQA-x1 complex annotation and atomic annotations, which does not validate scoring of arbitrary model outputs. The conclusion defers free-form clinical questions to future work, conceding that the current protocol is tied to atomic decomposability. If the scorer misaligns with human judgment, both the benchmark's headline and ASR's measured improvement are artifacts of the scoring rule, not evidence about endoscopic VQA reliability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EndoCA, a paired complex-atomic answer consistency benchmark for endoscopic VQA, built from the two-level QA structure of Kvasir-VQA and Kvasir-VQA-x1. Each test sample pairs a complex question with its associated atomic questions; complex-answer correctness is defined by the EndoCA scorer as normalized exact agreement with all associated atomic annotations. The paper evaluates 11 VLMs and reports complex accuracy, atomic accuracy, joint accuracy, and complex-atomic inconsistency. It finds that several models achieve high complex-answer accuracy while showing materially lower joint accuracy and non-negligible complex-atomic inconsistency. It then proposes Atomic-Support Reconciliation (ASR), a training-free method that uses the model's own atomic answers as contextual premises for revising complex answers and for consistency-guided selective answering. On four models, ASR-Revise improves joint accuracy by +3.1 to +8.1 percentage points and reduces complex-atomic inconsistency by 10.4 points on average; ASR-Selective trades coverage for answered-subset accuracy.","tokens_in":11106,"tokens_out":6293,"duration_ms":65693,"significance":"If the EndoCA scorer is accepted as valid, this is a useful benchmark contribution: it provides a reproducible, fixed-protocol metric for a clinically relevant reliability gap, evaluates a broad and representative set of VLMs, includes a controlled complexity axis, and proposes a training-free mitigation with a clean ablation that separates atomic-premise revision from generic self-checking. The paper is careful to report four complementary metrics and to distinguish complex-answer accuracy from paired correctness. However, the central claim is contingent on the validity of the EndoCA scoring rule, and the current manuscript provides no human-agreement or paraphrase-sensitivity validation for arbitrary model outputs. Since the ASR method is also evaluated with the same scoring rule, the headline 'hidden inconsistency' numbers and the ASR gains could partly reflect label-conformance rather than genuine unreliability or improvement. These concerns are addressable with targeted experiments, and the strengths of the benchmark design justify a revision rather than rejection.","major_comments":[{"comment":"Section III-C and Eq. (2)-(3) define complex-answer correctness as normalized exact agreement with every associated atomic annotation, but Table II's worked example describes the scorer as 'extract[ing] component labels' from a free-form sentence ('visible text is present' becomes 'yes'). These two descriptions conflict. If the scorer performs semantic component extraction, the exact-match wording and the claim that no additional LLM judge is used are incomplete; if it only applies surface normalization, the example overstates what is checked. This matters because the paper's headline phenomenon—complex accuracy hiding inconsistency—and the ASR gains in Table V are both measured with this scorer. No human-agreement study is reported; the source-dataset construction-time agreement does not validate scoring of arbitrary model outputs. I request a precise specification of the matching/extra","section":"Section III-C, Eq. (2)-(3), Table II"},{"comment":"ASR-Revise conditions the revision on Pi = {(qij, a_hat_ij)}, the model's own atomic answers, while the EndoCA scorer checks complex answers against the associated atomic annotations. The reported +3.1 to +8.1 joint-accuracy gains and −10.4 average inconsistency reduction may therefore reflect the revised answer conforming to the premise-label format rather than a genuine improvement in evidence use. The no-atomic ablation (Table VI) isolates the presence of atomic premises, but it does not test whether the benefit comes from independent evidence reconciliation or label-copying. Please add controls in which the atomic premises are (i) another model's atomic answers and (ii) deliberately corrupted atomic labels; report whether the revised answers follow the premise labels, and whether the joint-accuracy gain persists under human or semantic scoring.","section":"Section IV-B, Tables V-VI"},{"comment":"ASR-Selective is evaluated only by coverage and accuracy on the answered subset, without a comparison at matched coverage. Direct has 100% coverage, so the 71.0% versus 63.3% comparison does not separate the benefit of abstaining from the benefit of selecting easy cases. The claim that ASR-Selective 'improves accuracy on answered cases' should be supported by a risk-coverage curve, by direct accuracy at the same coverage (for example, random or confidence-based selection), or by a calibrated selective-prediction metric. Without this, the selective-answering contribution is not yet fully established.","section":"Section V-C, Table V"}],"minor_comments":[{"comment":"There are LaTeX rendering issues: 'atomicquestion' should be 'atomic question', and 'LLaV A-Med' should be 'LLaVA-Med'. Please check all model names for consistent typography.","section":"Throughout"},{"comment":"No confidence intervals or significance tests are reported. Given deterministic decoding and large fixed sample sizes this may be acceptable, but the authors should state explicitly that all metrics are point estimates over a fixed benchmark and that no repeated trials were run.","section":"Tables III-V"},{"comment":"Table V reports only deltas for Joint Acc. and Complex-Atomic Inconsistency. It would help the reader to also show the absolute Direct and Revised values for these two metrics, or to include a note that the Direct values are those in Table III.","section":"Table V"},{"comment":"The conclusion concedes that the current protocol is tied to atomic decomposability and defers free-form clinical questions to future work. This limitation should appear earlier, in the benchmark construction or scoring section, so readers interpret the metric accordingly.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is a plausible benchmark contribution with a useful training-free mitigation, and the model evaluation is broad. The principal obstacle is that the EndoCA scoring rule is simultaneously the benchmark's criterion and the target of the ASR method; without human or semantic validation, the headline numbers remain conditional. I would be supportive if the revision adds a human-agreement study for the scorer, a precise specification of the scoring/extraction procedure, and an ASR control with external or corrupted premises. No integrity concerns were apparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The headline claim survives scrutiny: models really do score well on complex answers while contradicting themselves on the same-image atomic questions, and in EndoCA that gap is not a scoring artifact. The inconsistency metric (Eq. 6) is computed among correct complex answers and compares the model's complex output against its own separately generated atomic answers under shared labels, so MedGemma-FT's 74.5% complex vs. 55.8% joint with 25.1% inconsistency reflects genuine internal disagreement, not paraphrase penalty.\n\nThe second thing is the cost of that design: EndoCA defines complex-answer correctness as exact match to every associated atomic label. That makes the benchmark a label-conformance test by construction, and the authors admit in the conclusion that free-form clinical questions are out of scope. A clinically right paraphrase that doesn't reproduce the canonical labels is scored wrong. That bias applies to both complex and atomic scoring, so the inconsistency finding isn't an artifact — but any absolute accuracy number from EndoCA should be read as \"conforms to the inherited atomic decomposition,\" not \"answers the question well.\"\n\nWhat's genuinely new: a paired complex-atomic evaluation protocol for endoscopic VQA with four companion metrics, built from Kvasir-VQA/Kvasir-VQA-x1 at no new annotation cost, plus ASR, a training-free reconciliation method. The ablation is the best part: generic self-revision without atomic premises does not improve joint accuracy or inconsistency, while ASR does (+3.1 to +8.1 joint across four models, -10.4 inconsistency on average). That is an honest test of the mechanism, and the atomic premises are doing real work.\n\nSoft spots, in proportion. Biggest: no release link for the benchmark, scorer, or prompts; for a benchmark paper that is a substantive reproducibility gap. Second: no human-agreement study for the scorer, so the paraphrase-mismatch rate is unknown. Moderate, not fatal. Third: ASR runs on four models with no error bars or significance tests; minor, since the gains are consistent across all four.\n\nWho benefits: people building or evaluating endoscopic and medical VQA models. This deserves a serious referee. My recommendation: send it to review, and require that the benchmark and scorer be released plus a small human-validated sample be reported; the ASR story can stay conditional until then.","headline":"The inconsistency finding is real and the ASR ablation is honest; EndoCA is a useful benchmark for endoscopic VQA, but it needs a public release and human validation of its label-conformance scorer before the reliability claims fully convince.","tokens_in":11599,"tokens_out":6654,"would_cite":true,"duration_ms":67499,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Endoscopic VQA models can score high on complex answers while failing the component questions on the same image, and a training-free method that reconciles the two narrows the gap.","keywords":["endoscopic VQA","complex-atomic consistency","answer consistency","vision-language models","joint accuracy","selective answering","medical visual question answering","answer revision"],"falsifier":"Collect or construct a set of complex answers that clinicians judge clinically correct but that do not exactly match the inherited atomic label sets; recomputing the metrics with a lenient semantic-match rule would show whether the benchmark's gap and ASR's 10.4-point inconsistency reduction are genuine or partly artifacts of the exact-match scorer.","tokens_in":10679,"feed_emoji":"🩺","tokens_out":7251,"duration_ms":67013,"temperature":0.7,"pith_summary":"The paper's central claim is that in endoscopic visual question answering, a model's accuracy on complex multi-part questions can be high while its answers to the component atomic questions on the same image are often wrong, so complex-answer scores overstate reliability. To make this visible, the authors build a paired benchmark in which each complex question is evaluated together with its associated same-image atomic questions, and report that one strong endoscopy-adapted model reaches 74.5% complex accuracy but only 55.8% joint accuracy, with a quarter of its accepted complex answers contradicting its own atomic predictions. They then propose a training-free method, Atomic-Support Reconciliation, that feeds the model's own atomic answers back as premises for revising the complex answer or for abstaining. Across four models it raises joint accuracy by 3.1 to 8.1 percentage points and cuts complex-atomic inconsistency by an average of 10.4 points. A sympathetic reader would care because the result implies that current final-answer benchmarks in this medical imaging domain can hide exactly the kind of unstable evidence-grounded reasoning that matters clinically.","feed_headline":"Endoscopic VQA's complex-answer scores hide a 25-point consistency gap","feed_subtitle":"A paired benchmark shows accepted answers can fail same-image atomic checks; training-free revision narrows the gap.","key_machinery":"The paired evaluation unit—an image, a complex question, its complex answer annotation, and an associated list of atomic question-answer pairs—with a fixed question-type-aware scorer and four metrics: complex-answer accuracy, atomic-answer accuracy, joint accuracy, and complex-atomic inconsistency. Joint accuracy and complex-atomic inconsistency are the load-bearing quantities: they convert the question 'is the final answer right' into 'is every component judgment on the same image right, and do the two agree.' The reconciliation mechanism ASR reuses this structure, treating model-generated atomic answers as premises for a revision prompt and for a consistency-judged answer/abstain decision.","core_discovery":"The paper introduces EndoCA, a paired complex-atomic consistency benchmark built from an existing two-level endoscopic VQA resource, and uses it to argue that complex-answer accuracy alone is an incomplete signal. On 11 vision-language models, the benchmark's joint accuracy—requiring the complex answer and every separately generated atomic answer to be correct—runs well below complex-answer accuracy, and its complex-atomic inconsistency metric shows that a substantial share of accepted complex answers are accompanied by at least one wrong atomic prediction on the same image. The same paired signal motivates ASR, a training-free reconciliation mechanism: the model first answers the atomic que","pith_inferences":["The benchmark's inconsistency score is defined under exact normalized matching to inherited atomic labels; a lenient semantic-match scorer might shrink the reported gap, so the gap's magnitude is partly a function of scoring strictness, not only model behavior.","The ASR pattern—answer components, then reconcile or abstain—could transfer to other multi-component medical VQA settings such as radiology or pathology wherever atomic subquestions exist; the paper does not test this.","The abstention signal could double as a human-review trigger in clinical deployment: cases where atomic premises conflict are exactly the ones a clinician should re-check.","Because ASR uses only the model's own outputs, its gains could in principle generalize across architectures; a natural extension is to test whether fine-tuning on consistency pairs shrinks the gap more durably than inference-time reconciliation."],"forward_implications":["Endoscopic VQA evaluation should report joint accuracy and complex-atomic inconsistency alongside complex-answer accuracy whenever paired atomic questions exist; otherwise rankings and reported reliability can mislead.","Models with nearly identical complex-answer accuracy can differ by more than 14 percentage points in joint accuracy, so consistency-aware scores reorder model comparisons.","As question complexity rises—more atomic components per complex question—all-atomic accuracy falls faster than complex accuracy, making multi-part questions a stress test of paired correctness.","Model-generated atomic answers can serve as inference-time premises for revision without retraining, improving paired correctness on top models.","Consistency-guided abstention provides a reliability–coverage trade-off, letting models answer more often when atomic premises agree and abstain when they conflict."],"fun_headline_variants":["EndoCA: complex-answer accuracy hides atomic errors","VQA models often pass complex tests but fail atomic ones","Training-free reconciliation fixes complex-atomic answer gaps","Same-image atomic checks catch hidden misses in endoscopic VQA"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole inconsistency metric assumes that every correct complex answer can be exactly decomposed into the atomic labels inherited from the source dataset, so a semantically correct paraphrase that does not align with those labels is counted as inconsistent.","fun_headline_variants_meta":{"raw":{"variants":["EndoCA: complex-answer accuracy hides atomic errors","VQA models often pass complex tests but fail atomic ones","Training-free reconciliation fixes complex-atomic answer gaps","Same-image atomic checks catch hidden misses in endoscopic VQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000414,"raw_usage":{"total_tokens":1991,"prompt_tokens":773,"completion_tokens":1218,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1153}},"tokens_in":517,"tokens_out":1218,"duration_ms":14359,"temperature":1.0,"reasoning_tokens":1153,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:50:47.513910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect or construct a set of complex answers that clinicians judge clinically correct but that do not exactly match the inherited atomic label sets; recomputing the metrics with a lenient semantic-match rule would show whether the benchmark's gap and ASR's 10.4-point inconsistency reduction are genuine or partly artifacts of the exact-match scorer.","supporting_citations":[],"review_version":1}