{"id":"372012ba-b430-47f3-952e-ec8582be3550","arxiv_id":"2507.06959","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A preference optimization strategy using confidence-based hard example mining, similarity retrieval, and synthetic counterfactual rationales improves chest X-ray VQA accuracy by 8.93% relative over supervised fine-tuning.","lead":"CheXPO is a preference optimization pipeline that trains a chest X-ray vision-language model to avoid wrong or uncertain answers by mining hard examples and building synthetic counterfactual rejections without extra radiologist labels. It reports an 8.93% relative accuracy gain over its supervised fine-tuning baseline on chest X-ray VQA benchmarks, suggesting a cheaper path to more reliable medical image question answering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Counterfactual rejections may not be false for their paired image; a side-effect of the template-based construction and retrieval step is never checked, and the human evaluation in Table 9 does not test the rejection pairs used in DPO.","rationale":"The reader's weakest assumption is precisely the unverified falseness of counterfactual rejections, and I agree. The paper's own quality evaluation in Table 9 is on the chosen rationales against reports, not on the rejected-side substitutions used in DPO; the 100-sample human check is ambiguous but does not explicitly audit the paired image-conditioned falseness of rejections. The entire performance contribution of the counterfactual component (1.6 points in the 20k ablation, Table 10) rests on this. A concrete audit of rejected-side correctness is feasible and would settle the matter. Other concerns (missing uniform baseline at 30k, strict matching evaluation, narrow baselines) are real but secondary; the counterfactual-correctness issue is the one that, if wrong, would invalidate the central mechanism rather than just the benchmark reporting. The method itself is well-structured and the other ablations are internally consistent, so a conditional verdict with this one test is appropriate rather than rejection or full acceptance.","tokens_in":24089,"tokens_out":1761,"duration_ms":22629,"concrete_test":"Audit 500 constructed preference pairs from the 30k DPO set (stratified by question type) with radiologist labels for whether the rejected answer is false for the paired image; compute the accidentally-correct rejection rate. If >5%, rerun DPO with filtered pairs and compare to Table 10. Alternatively, test the counterfactual construction by ablating it entirely: train DPO on hard examples only but with the same number of pairs (30k) to see if the 8.93% relative gain persists; if it does, the counterfactual mechanism is not load-bearing.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central improvement of CheXPO over 'hard-example mining without CRej' comes from training on preference pairs whose rejected response is a constructed 'counterfactual rationale' (Sec. 4.3, Eq. 5). The construction replaces the short answer yA with a token from a domain-specific rejection pool (e.g., 'pneumonia' vs. 'pulmonary edema', 'left lung' vs. 'right lung') and then retrieves the Top-1 rationale from D_rest by BioMedCLIP similarity to the (replaced answer, original explanation) pair. This pipeline never verifies that the replacement answer is actually false for the specific image and question. For example, for a question about the right lung, swapping 'right lung' for 'left lung' is only a valid counterfactual if the left lung truly does not show the finding; because chest X-rays frequently show bilateral or multi-lobar disease, the swapped answer can be accidentally correct for the paired image. Similarly, 'pneumonia' vs. 'pulmonary edema' are confusable and co-occurring; the false answer may match the ground truth for a substantial fraction of cases. When the 'rejected' response is actually correct, DPO is given a noisy preference pair that penalizes a valid answer, directly undermining the mechanism claimed to produce the 8.93% relative gain over SFT (Table 1). The paper's quality evaluation (Table 9, Appendix F) evaluates the chosen rationales against reports (disease correctness, semantic similarity, LLM score, and a 100-sample human check) but does not evaluate whether the constructed rejections are incorrect for the paired image. The 100-sample human evaluation in Table 9 is explicitly on 'counterfactual rationales' as sampled outputs of the generation pipeline; however, the description says 'factual correctness of each rationale,' which checks answers against reports, not against the image-conditioned truth of the paired replacement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CheXPO, a preference-optimization pipeline for chest X-ray vision-language models. The authors first construct a large multi-task visual instruction dataset by synthesizing GPT-4o-generated rationales over MIMIC-CXR, Chest ImaGenome, MIMIC-CXR-VQA, and Medical-Diff-VQA, and use it to supervised fine-tune Phi-3.5V with LoRA. They then apply DPO using preference pairs selected by token-level confidence analysis and BioMedCLIP-based similarity retrieval, with 'counterfactual rationales' constructed by replacing the short-answer component with tokens from anatomy, abnormality, and severity rejection pools. The paper reports an 8.93% relative accuracy improvement over the SFT baseline using 30k preference pairs (about 5% of the SFT data), and reports strong results on MIMIC-CXR-VQA and Medical-Diff-VQA.","tokens_in":24455,"tokens_out":4200,"duration_ms":52356,"significance":"If the claims hold, the paper makes a useful contribution: it demonstrates a data-efficient, expert-annotation-free way to apply preference optimization to medical VLMs, and the proposed hard-example mining plus counterfactual rationale recipe is portable to other medical imaging domains. The paper also contributes a large public instruction dataset and reports ablations over data size, retrieval modality, and alternative preference objectives. The main experimental gains are consistent across many settings, and the work includes a human evaluation of the generated chosen rationales. However, the central mechanism of the counterfactual rejection construction is not validated per image, and the headline 30k result is not covered by the ablation curves; these gaps need to be closed before the claims can be accepted at face value.","major_comments":[{"comment":"The counterfactual rejection construction never verifies that the replaced short answer is actually incorrect for the specific image. The rejection pools contain anatomically or diagnostically confusable pairs, such as 'left lung' vs. 'right lung' and 'pneumonia' vs. 'pulmonary edema'; in chest X-rays these conditions are frequently bilateral or co-occurring, so a swapped answer can be accidentally correct for a given study. When this happens, the DPO pair in Eq. (6) penalizes a valid response, directly undermining the mechanism that Table 10 and Figure 6 attribute to the counterfactual rationale component. The quality evaluation in Table 9 and Appendix F addresses the generated chosen rationales, not the constructed rejected responses. Please add an automatic per-image verification or filtering step (for example, using the MIMIC-CXR structured labels or the original report), report the estimated rate of accidentally correct rejections, and show that the gains persist after removing or correcting such pairs.","section":"Section 4.3, Eq. (5)"},{"comment":"The headline result, the 8.93% relative gain over the SFT baseline, is reported for 30k DPO pairs in Table 1, but the ablation in Figure 6 stops at 20k pairs and does not include a uniform-sampling or w/o-CRej baseline at the 30k scale. As a result, the paper does not establish whether the 30k improvement comes from the proposed mining and counterfactual components or simply from using a larger preference set with a different selection strategy. Please include the 30k points for the uniform, w/o-CRej, and full CheXPO variants, or explicitly restrict the central claim to the 20k setting and present the 30k result as an additional scaling observation.","section":"Figure 6 and Table 1"},{"comment":"The presentation conflates two distinct objects: the quality evaluation in Table 9 is performed on generated dataset rationales (the chosen responses), while the counterfactual-rejected responses used in Eq. (5) are never evaluated on criteria such as factual incorrectness, semantic coherence with the image, or consistency between the replaced answer and the retrieved explanation. Because the paper's stated contribution is specifically the construction of counterfactual rejections, this asymmetry should be addressed directly rather than through the current Table 9 caption, which calls the evaluated items 'counterfactual rationales'.","section":"Section 6.2 / Appendix F"}],"minor_comments":[{"comment":"The values of gamma, sigma, and Top-K are described as 'adjusted based on the target preference dataset size,' but the paper does not report how these parameters were selected or whether they were tuned on a validation set; please clarify the selection procedure to avoid test-set contamination concerns.","section":"Section 5.2 / Table 8"},{"comment":"The notation for the constructed rejection response is unclear: the same symbol y is used for the model output, the replaced answer, and the retrieved Top-1 candidate. Please introduce distinct symbols for the original output, the modified answer, and the final rejected response.","section":"Equation (5)"},{"comment":"The rejection-pool replacement is described as 'randomly replace'; please state the random seed or the number of random trials, and whether the reported results are averaged over multiple seeds, because the preference data construction is stochastic.","section":"Section 4.3"},{"comment":"The metric 'Error Sample Hit Rate' in Figure 5 and the 'Forwarded' column in Table 2 are not defined in the main text; please define them precisely and report the size of the query and gallery sets used for the evaluation.","section":"Figure 5 and Table 2"},{"comment":"The win-rate metric is mentioned in the table footnote as 'the PO model's short answer over that of SFT,' but the exact evaluation protocol (ties, randomness, number of judges) is not given; please specify it.","section":"Appendix G, Table 10"}],"recommendation":"major_revision","confidential_remarks":"The counterfactual rejection validity issue is the most important technical concern: it is load-bearing for the claimed mechanism, and the current human evaluation does not cover the rejection pairs actually used in DPO. I would ask the authors for per-image validation or a noisy-pair ablation before considering acceptance. The missing 30k ablation for uniform/w-o-CRej is secondary but should also be fixed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you're working on preference optimization for medical VLMs. CheXPO is a clean data-construction recipe: mine hard examples from SFT failures using token-level confidence, expand via BioMedCLIP retrieval, and synthesize counterfactual rationales as DPO rejections. No expert annotation, and the gains look real: +8.93% relative over the SFT baseline on their unified test set, with consistent gains on MIMIC-CXR-VQA and Medical-Diff-VQA. The ablations are honest—multiple PO objectives, uniform sampling plateaus, and hard mining plus counterfactual rationales help across data sizes. The dataset and code are promised.\n\nThe soft spot is the one I'd push on: the counterfactual rejections are not verified to be false for the paired image. Swapping 'left lung' for 'right lung' or 'pneumonia' for 'pulmonary edema' can accidentally match the ground truth in bilateral or co-occurring disease. Table 9 evaluates the chosen rationales, not the rejections, so the preference pairs may be noisier than claimed. This doesn't sink the paper—the w/o-CRej ablation still improves over SFT, and DPO tolerates some label noise—but the mechanism claims need per-image verification. I'd want to see a sampling check or a filtering step before trusting the contribution of CRej.\n\nTwo smaller gaps: the headline 30k result lacks a uniform-sampling baseline at the same scale (Figure 6 stops at 20k), and the SOTA claim is against a thin baseline set—CheXalign and MMedPO are cited but not compared. Evaluation is strict string-matching on short answers, which is coarse for open-ended questions.\n\nOverall: a solid, reproducible contribution that deserves a serious referee. The core idea is useful, the experiments are mostly careful, and the flaws are addressable. I'd accept with major revisions and require the rejection verification.","headline":"Solid, reproducible preference-optimization recipe for chest X-ray VLMs with real gains, but the unverified counterfactual rejections and missing 30k uniform baseline need attention before I'd trust the headline numbers.","tokens_in":25036,"tokens_out":2966,"would_cite":true,"duration_ms":30046,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that preference tuning on 30k automatically mined and counterfactually constructed preference pairs raises a chest X-ray VLM's overall accuracy from 73.20% to 79.74% (an 8.93% relative gain) and improves results on the…","keywords":["chest X-ray VQA","medical vision-language model","direct preference optimization","counterfactual rationale","hard example mining","hallucination mitigation","long-tailed distribution","DPO"],"falsifier":"A reader could take a random sample of the constructed preference pairs, treat each counterfactual rejected answer as a hypothesis, and ask a radiologist whether that swapped answer is actually false for that image; if a substantial fraction turn out to be true findings, the preference supervision is partly inconsistent. This audit is concrete because the rejection pools are deterministic and the paper already performs a 100-sample human evaluation of chosen rationales, so the same protocol can be applied to rejected rationales.","tokens_in":1847,"feed_emoji":"🩻","tokens_out":4017,"duration_ms":96439,"temperature":0.7,"pith_summary":"The paper tries to establish that preference optimization can make a chest X-ray vision-language model substantially more reliable without expert annotation, provided the preference pairs are chosen for clinical value rather than sampled at random. It introduces CheXPO, a pipeline that finds hard examples where an SFT model fails or is uncertain, expands them by semantic retrieval to counter the long-tailed distribution of findings, and builds rejected responses by replacing correct answers with clinically opposite or confusable counterfactuals. The resulting DPO tuning, trained on 30k pairs from roughly 5% of the SFT samples, lifts CheX-Phi3.5V's unified-test accuracy from 73.20% to 79.74% and improves benchmark QA on MIMIC-CXR-VQA and Medical-Diff-VQA. If correct, this means hallucination-prone medical VLMs can be aligned to radiologist-style reasoning with data generated automatically from existing reports, rather than with expensive expert feedback.","feed_headline":"Wrong-answer training lifts chest X-ray AI by 8.9 percent","feed_subtitle":"Automatically generated counterfactual examples teach radiology models to avoid hallucinations with no expert labels.","key_machinery":"The load-bearing mechanism is the counterfactual rejection rationale: for a low-confidence but correct prediction, the short answer is replaced by a clinically opposite or confusable alternative from a domain-specific rejection pool (left versus right lung, pneumonia versus pulmonary edema, mild versus severe), and a semantically coherent candidate rationale is retrieved with BioMedCLIP to serve as the rejected response in a DPO pair. This is coupled with confidence-similarity joint mining: a length-normalized log-probability threshold flags hard examples, and multi-modal cosine-similarity Top-K retrieval balances the preference set across the long-tailed distribution of findings. Together these components convert an SFT model's own failures and uncertainties into preference signals without expert labels.","core_discovery":"The central claim is that DPO's value for medical VLMs is gated by data selection and negative construction, not by the preference objective itself. On a unified chest X-ray QA dataset of 640k expert-style rationales, SFT brings Phi-3.5V to 73.20% overall accuracy. CheXPO then identifies hard examples using length-normalized token-level log-probabilities of the short answer: wrong answers become rejected responses, and correct-but-low-confidence answers are paired with counterfactual rationales assembled from domain-specific rejection pools covering anatomy, abnormality, and severity, plus opposite answers such as left versus right lung. BioMedCLIP retrieval expands the hard set with similar questions, rationales, and images. DPO on 30k such pairs yields 79.74% accuracy, an 8.93% relative improvement over SFT, and better results than earlier medical VLMs on MIMIC-CXR-VQA (81.8% accuracy) and Medical-Diff-VQA (BLEU-4 of 0.564). The paper also reports that the preference-tuned model shows lower DPO training loss, higher confidence on previously uncertain Anatomy and Severity cases, and corrected confidently wrong Abnormality predictions.","pith_inferences":["The same counterfactual-pool recipe should transfer to other imaging modalities whenever a set of clinically opposite or confusable labels can be enumerated; a testable extension is to swap in modality-specific rejection pools.","The reported probability shifts for previously uncertain cases suggest the method may improve calibration, not just accuracy; measuring expected calibration error on hard question types would test this directly.","Because hard examples are defined by one SFT model's failures, applying the mining loop again on the preference-tuned model could yield a second round of gains, which the paper does not test.","The paper audits the correctness of chosen rationales but not the falsity of rejected counterfactuals; an independent audit of rejected pairs would directly test the method's central assumption."],"forward_implications":["A radiology VLM can be made more reliable with 30k preference pairs drawn from only 5% of the SFT samples, so preference alignment need not depend on expert-written feedback.","Gains concentrate in the three error families that dominate failures (Abnormality, Anatomy, Severity), so the method targets clinically risky cases rather than easy ones.","Counterfactual construction breaks the plateau that uniform and hard-only DPO hit near 10k pairs, allowing larger preference sets to keep improving performance.","The mining and counterfactual steps transfer across preference objectives, since DPO, SimPO, CPO, ORPO, and IPO all surpass SFT in the paper's ablations.","The GPT-4o rationales in the instruction dataset are grounded in MIMIC-CXR radiologist reports, so the preference signal traces back to expert-written clinical text."],"supporting_citations":[{"why":"Provides the DPO objective used to align the VLM with chosen and rejected preference pairs.","marker":"[37]"},{"why":"Provides the BioMedCLIP multimodal features used for similarity-based hard example retrieval and counterfactual candidate selection.","marker":"[47]"},{"why":"Supplies the MIMIC-CXR chest X-rays and radiologist reports that ground the synthesized rationales and the unified test set.","marker":"[17]"},{"why":"Provides the MIMIC-CXR-VQA question-answer pairs and the benchmark against which CheXPO is evaluated.","marker":"[2]"},{"why":"Provides the Medical-Diff-VQA comparative QA pairs and the benchmark for difference questions.","marker":"[14]"},{"why":"Provides the Chest ImaGenome bounding-box annotations used to build the Region-QA visual prompts.","marker":"[43]"},{"why":"Defines the Phi-3.5V base model that is SFT-tuned and then preference-tuned.","marker":"[1]"},{"why":"Provides the GPT-4o engine used to synthesize expert-style rationales from radiology reports for the instruction dataset.","marker":"[32]"}],"fun_headline_variants":["CheXPO: 8.9% gain in chest X-ray VLM via counterfactual DPO","Counterfactual rationale boosts chest X-ray VLM by 8.9%","Preference optimization with counterfactuals lifts X-ray VLM 8.9%","Hard-example mining plus counterfactuals improve X-ray VLM 8.9%","8.9% relative gain: CheXPO trains chest X-ray VLM on 5% of data"],"cache_read_input_tokens":27008,"weakest_assumption_plain":"The counterfactual rejected answers are assumed to be genuinely false for the specific image, but they are constructed by swapping in opposite or confusable terms from hand-built rejection pools without checking the actual image, so an accidentally correct swapped answer would turn the DPO pair into self-contradictory supervision.","fun_headline_variants_meta":{"raw":{"variants":["CheXPO: 8.9% gain in chest X-ray VLM via counterfactual DPO","Counterfactual rationale boosts chest X-ray VLM by 8.9%","Preference optimization with counterfactuals lifts X-ray VLM 8.9%","Hard-example mining plus counterfactuals improve X-ray VLM 8.9%","8.9% relative gain: CheXPO trains chest X-ray VLM on 5% of data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000359,"raw_usage":{"total_tokens":1964,"prompt_tokens":987,"completion_tokens":977,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":856}},"tokens_in":603,"tokens_out":977,"duration_ms":9711,"temperature":1.0,"reasoning_tokens":856,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:50:48.208028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could take a random sample of the constructed preference pairs, treat each counterfactual rejected answer as a hypothesis, and ask a radiologist whether that swapped answer is actually false for that image; if a substantial fraction turn out to be true findings, the preference supervision is partly inconsistent. This audit is concrete because the rejection pools are deterministic and the paper already performs a 100-sample human evaluation of chosen rationales, so the same protocol can be applied to rejected rationales.","supporting_citations":[{"cited_title":"Rao, Mu-Hsin Wei, Naveen Valluri, Cliff Wong, Matthew P","cited_arxiv_id":null,"evidence_quote":"Provides the BioMedCLIP multimodal features used for similarity-based hard example retrieval and counterfactual candidate selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MIMIC-CXR chest X-rays and radiologist reports that ground the synthesized rationales and the unified test set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-CXR-VQA question-answer pairs and the benchmark against which CheXPO is evaluated."},{"cited_title":"An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M","cited_arxiv_id":null,"evidence_quote":"Provides the Medical-Diff-VQA comparative QA pairs and the benchmark for difference questions."},{"cited_title":"Wu, Nkechinyere N","cited_arxiv_id":null,"evidence_quote":"Provides the Chest ImaGenome bounding-box annotations used to build the Region-QA visual prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GPT-4o engine used to synthesize expert-style rationales from radiology reports for the instruction dataset."}],"review_version":1}