{"id":"a8ee2cc5-3b5d-4661-9f0b-7d7e5c7f6d13","arxiv_id":"2504.20468","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Antidote trains LVLMs to reject false presuppositions in questions and reduce object hallucination using synthetic images, self-corrected preferences, and a new CP-Bench benchmark.","lead":"This paper introduces Antidote, a training method that reduces hallucination in vision-language AI by teaching models to recognize when a question assumes something that is not actually in the image. It also introduces CP-Bench, a benchmark for testing this failure mode, and reports large improvements on LLaVA models without hurting general performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Factual Assessor is the load-bearing link: Grounding-DINO false negatives on hallucination candidates are retained as 'verified absent' labels, corrupting DPO pairs; no precision/recall audit on synthetic images is reported.","rationale":"The reader's weakest assumption identifies the Factual Assessor as the load-bearing link, and I agree: the DPO preference pairs in Eq. (1) inherit their correctness from Grounding-DINO's detections, and the paper supplies no validation of that detector on its own synthetic images. This is the single most load-bearing concern because it threatens the training signal itself, not just an evaluation detail. A biased GPT-4o judge could inflate CP-Bench numbers, but POPE and CHAIR already provide independent evidence that the method reduces object-perception hallucinations; by contrast, if the assessor labels are noisy, the CPQ improvements could be driven by a learned tendency to refuse object questions rather than by genuine discrimination of counterfactual presuppositions. The concern is concrete and addressable: a human audit of retained triplets plus a retraining experiment on cleaned labels would settle it. The paper has real strengths—it releases code, reports gains on established benchmarks, and its test set is manually curated—so the appropriate response is to keep the reader's CONDITIONAL verdict and require the missing validation rather than to reject. The concern is not a demonstrated refutation; it is an unverified assumption at the center of the method.","tokens_in":16054,"tokens_out":7307,"duration_ms":83838,"concrete_test":"Sample 200 retained synthetic triplets uniformly from the reported 10k training set; have human annotators label, for each image, whether each Opre object is present and each Ohallu object is absent. Compare these labels against Grounding-DINO's outputs, computing per-object precision and recall on the generated images. If the false-negative rate for Ohallu (or false-positive rate for Opre) exceeds, say, 5%, retrain Antidote on the subset whose assessor labels pass human audit and re-measure CP-Bench, POPE, CHAIR, and SHR. If the headline gains persist on the cleaned subset, the concern is resolved; if they shrink substantially, the reported improvements are partly an artifact of label noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the preference pairs used in Eq. (1) being correctly labeled: the positive response says an object is absent/present, and the negative response says otherwise. Section 3.2 Step 3 assigns these labels with Grounding-DINO as the Factual Assessor, checking presence of Opre and absence of Ohallu in each Stable Diffusion 3 image. The failure mode that matters is asymmetric: a false negative on Ohallu is not filtered out; it is retained as a 'verified absent' sample, even though the object is actually visible. Similarly, a false positive on Opre retains a sample in which an object labeled as present is actually absent. The paper reports only that 'approximately 4k' of 14k triplets were filtered, which is not evidence of assessor correctness. No precision/recall statistics on the generated images, no per-object detection thresholds, and no human audit are provided. Since CP-Bench dev set is built by the same pipeline, assessor errors can also inflate apparent benchmark alignment. If even a few percent of the 10k retained triplets are mislabeled, the DPO signal teaches the model to refuse questions about objects that are actually present or to affirm objects that are absent; the large CP-Bench gains could then partly reflect a learned refusal bias rather than grounded factual perception. POPE and CHAIR provide some independent support, but the CPQ contribution, which is the paper's headline claim, rests directly on this unvalidated link.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Antidote, a synthetic-data-driven post-training framework that uses Stable Diffusion 3 generated images with decoupled object co-occurrences, a Grounding-DINO-based Factual Assessor, and DPO to reduce hallucinations in counterfactual presupposition questions (CPQs) and object perception. The authors also introduce CP-Bench, a benchmark with a dev set generated by the same pipeline and a manually curated test set from CC3M, evaluated by GPT-4o. Experiments on LLaVA-1.5-7B/13B and LLaVA-Next-Mistral-7B report substantial gains on CP-Bench, POPE, CHAIR, and SHR, with minimal loss on general benchmarks.","tokens_in":16293,"tokens_out":6962,"duration_ms":62779,"significance":"If the results are reproducible, the paper makes a useful contribution: it identifies a neglected failure mode (CPQs), proposes a fully automated training-data pipeline that does not require stronger LVLMs for preference-pair construction, and reports large improvements on both the new benchmark and existing hallucination benchmarks. The manually curated test set is a strength, and the release of code will aid reproducibility. However, the central mechanism depends on the correctness of the Factual Assessor's grounding labels, which is not validated, and the evaluation protocol has gaps (single runs, sole GPT-4o judge) that need to be addressed before the claims can be fully accepted.","major_comments":[{"comment":"The correctness of the DPO preference pairs in Eq. (1) depends on the Factual Assessor's ability to verify the presence of Opre and the absence of Ohallu in every synthetic image. The paper reports only that approximately 4k of 14k triplets were filtered, with no precision/recall audit of Grounding-DINO on the generated images. A false negative on an Ohallu object leaves a sample labeled 'verified absent' even though the object is visible, and a false positive on an Opre object retains a sample where the object is actually absent; either error teaches the model the wrong preference. Because the CP-Bench dev set is built by the same pipeline, assessor errors can also inflate the dev results. Please report per-object precision/recall on a human-annotated sample of the generated images, and describe precisely how the filtering handles individual objects (e.g., whether a missing Opre object invalidates the triplet).","section":"Sec. 3.2, Step 3, Fig. 4"},{"comment":"The dev set of CP-Bench is generated by the same synthetic pipeline used to create Antidote's training data. This shared distribution can make dev-set gains reflect overfitting to the pipeline rather than genuine improvement in presupposition discrimination. While the manually curated test set provides some independence, the paper often reports dev F1 (Table 1) without noting this caveat. Please either report the main results on the test set only, or provide an analysis (e.g., human evaluation of dev-set questions, or a comparison of model behavior on dev vs. test) that supports the dev set as a valid held-out benchmark.","section":"Sec. 4, CP-Bench dev set"},{"comment":"All experimental results appear to be single runs, with no standard deviations, confidence intervals, or significance tests. This is especially problematic for POPE, where the claimed improvements are 1.8-3.3 F1 points, and for the differences among baseline methods within the same table. Please report results over multiple seeds (or equivalent variance estimates) and perform paired significance tests where appropriate.","section":"Sec. 5.1, Tables 1-3"},{"comment":"CP-Bench relies exclusively on GPT-4o to convert open-ended responses into binary presupposition-correctness judgments, with no human agreement or alternative judge. Since the headline claim is a >50% F1 improvement on this benchmark, the reliability of the judge is load-bearing. Please provide a human-annotated subset (e.g., 200 samples) with inter-annotator agreement between GPT-4o and humans, or at minimum agreement with a second strong judge, and report the correlation.","section":"Sec. 4, Evaluation"}],"minor_comments":[{"comment":"The text mentions 'VPF-Bench' in the paragraph on model size; this appears to be a typo for 'CP-Bench'.","section":"Sec. 5.2"},{"comment":"The reward margin defined in Eq. (2) includes only the preferred-response log-ratio; the standard DPO reward margin is the difference between the preferred and rejected log-ratios. Please correct the definition or clarify.","section":"Eq. (2)"},{"comment":"The description of the filtering rule is ambiguous: 'If an object in Opre is not detected, it will be removed' could mean the object is removed from Opre or the triplet is discarded. Please clarify precisely how the object-level filtering affects the triplet and the downstream QA generation.","section":"Sec. 3.2, Step 3"},{"comment":"The citations for VDD and VCD appear inconsistent with the reference list; please verify the numbering.","section":"Tables 2-3"},{"comment":"The phrase 'all without relying on external supervision from stronger LVLMs or human feedback and introducing noticeable catastrophic forgetting issues' is ambiguous; add 'without' before 'introducing' for clarity.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision journal, and the CP-Bench benchmark is a timely contribution. The main technical risk is the unvalidated Factual Assessor; I would encourage the editor to require the suggested validation before publication. The related-work section is adequate, but the citation numbering errors and the ambiguous filtering description should be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know about this paper: it targets a genuine blind spot in LVLM hallucination work—counterfactual presupposition questions, where the question itself assumes an object that isn't there. The authors build a synthetic-data pipeline that generates scenes where co-occurring objects are deliberately removed, then use the known ground truth to have the model self-correct its answers and train with DPO. They also release CP-Bench, with a manually curated test set sampled from CC3M.\n\nThe paper does well on multiple fronts. The CP-Bench test set is independent of the training pipeline, and the reported gains are consistent across three families of benchmarks (CP-Bench, POPE, CHAIR/SHR). That transfer is the strongest evidence that the method isn't just overfitting to synthetic data. The unified treatment of CPQs and object existence is a real step beyond prior work that only handled one or the other.\n\nThe soft spots are real but mostly addressable. The load-bearing assumption is the 'Factual Assessor' (Grounding-DINO) correctly labeling which objects are present and absent in the generated images. The paper reports that 4k of 14k triplets were filtered, but no precision/recall on the assessor itself. A systematic false-negative rate on hallucination candidates would corrupt the preference pairs and could teach the model to refuse rather than perceive. Given the large CP-Bench gains, I'd want that audited. Also: single-run results with no error bars, and GPT-4o is the only CP-Bench judge without human agreement checks. The 'without external supervision' claim is overstated—DeepSeek-V2, Stable Diffusion, and Grounding-DINO are all external systems doing the labeling.\n\nNone of this is a demonstrated refutation. The central mechanism—using factual priors to create self-corrected responses and then doing DPO—is sound, and the independent test set plus POPE/CHAIR results give me confidence the effect is real.\n\nWho's this for: anyone working on LVLM hallucination mitigation or benchmarking. It's a solid paper for a workshop or main conference. I'd send it to review, with the ask that the authors provide assessor validation, multiple seeds, and a human evaluation on CP-Bench. My own verdict would be accept with revisions rather than reject.","headline":"A practical, well-evaluated post-training recipe for a real failure mode; the main caveat—unvalidated grounding labels—is worth an extra audit, not a rejection.","tokens_in":16904,"tokens_out":2441,"would_cite":true,"duration_ms":23106,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single synthetic-data post-training framework, Antidote, simultaneously reduces hallucinations from counterfactual presuppositions and object perception in LLaVA models, without external supervision or…","keywords":["large vision-language models","hallucination mitigation","counterfactual presupposition questions","preference optimization","direct preference optimization","synthetic data pipeline","object perception hallucination","CP-Bench"],"falsifier":"Take a random sample of the generated training images, have human annotators label whether each $O_{pre}$ and $O_{hallu}$ object is truly present, and measure the Factual Assessor's precision and recall. If recall on $O_{pre}$ or precision on $O_{hallu}$ is materially below 100%, the preference pairs contain systematically mislabeled examples; a second check would be to train Antidote with those mislabeled pairs excluded and see whether the CP-Bench gains shrink.","tokens_in":15793,"feed_emoji":"💉","tokens_out":5383,"duration_ms":50493,"temperature":0.7,"pith_summary":"The paper argues that current hallucination fixes miss a blind spot: a model that correctly says \"no car\" will still answer \"BMW\" when asked what brand the absent car is. To fix this, the authors build Antidote, a fully synthetic pipeline that generates images in which statistically co-occurring objects are stripped out (a speedboat with no bridge, a street with no car), uses a grounding detector to verify what is and is not present, and then lets the model itself correct its answers using that factual information. Those corrected answers become preferred training samples and the original hallucinated answers become rejected samples in a DPO-style preference optimization. Applied to LLaVA-1.5 and LLaVA-Next, the recipe raises CP-Bench F1 by over 50 points, improves POPE by 1.8–3.3%, and cuts CHAIR/SHR hallucinations by roughly 30–50%, without degrading general benchmarks. The paper also contributes CP-Bench, a benchmark of 1,000 curated counterfactual-presupposition and true-presupposition questions.","feed_headline":"One post-training recipe cuts LVLM hallucination rates in half","feed_subtitle":"Synthetic scenes and self-corrected answers teach LLaVA to reject false presuppositions—no stronger model needed.","key_machinery":"The load-bearing object is the synthetic data pipeline plus the preference pair it produces. For each caption, a language model lists objects that are present ($O_{pre}$) and objects that commonly co-occur but will be absent ($O_{hallu}$); Stable Diffusion 3 renders the scene using $O_{hallu}$ as a negative prompt; and Grounding-DINO, an open-set grounding detector, confirms that $O_{pre}$ objects appear and $O_{hallu}$ objects do not, discarding triplets where verification fails. The verified facts are then prepended to the question (\"Given the fact that there is no car in the image, what is the brand of the car?\"), and the model's self-corrected answer becomes the DPO-preferred response over its original hallucinated answer. This transforms hallucination mitigation into a preference-optimization problem without any external expert model or human preference labels.","core_discovery":"The central claim is that counterfactual-presupposition hallucination—a model accepting a false premise embedded in a question and answering about an object that is not in the image—is a distinct and widespread failure mode, and that it can be mitigated together with classic object-perception hallucination by a single preference-alignment post-training method. The mechanism: synthesize scenes where common co-occurrences are decoupled, verify the resulting object sets with an open-set grounding detector, inject the verified facts into the question prompt, collect the model's self-corrected answer as the preferred response and the original hallucinated answer as the rejected response, and optimize with DPO. On LLaVA-1.5-7B/13B and LLaVA-Next, the method is reported to raise CPQ correctness dramatically, reduce object-existence and description hallucinations, and preserve general ability.","pith_inferences":["The paper's own observation that post-training makes models cautious on LLaVA-Wild suggests a tunable trade-off between refusal and informativeness; an obvious extension is to weight CPQ samples against description samples or mix in a small amount of general instruction data to control this.","Because the CP-Bench dev set is generated by the same pipeline used for training, part of the reported CP-Bench gain may reflect distribution overlap; the strongest test of the paper's claim would be a human-written CPQ set drawn from scenes not seen in any training caption.","The same decoupling idea—remove a statistically expected object and ask about it—could be applied to attribute and relation hallucinations by extending the factual prior to attributes rather than existence alone."],"forward_implications":["A model trained with Antidote should refuse to answer questions whose presupposed object is absent, stating the absence rather than inventing an attribute or brand.","Object-existence hallucinations on adversarial POPE sets—where objects are chosen by co-occurrence frequency—should shrink, since the training explicitly decouples frequent co-occurrences.","The gains should transfer across LLaVA-scale architectures and, the paper argues, should not come at the cost of general benchmarks such as MMBench or ScienceQA.","Because the recipe uses only synthetic data and the model's own self-correction, it should be reproducible for other LVLMs without access to proprietary expert models."],"supporting_citations":[{"why":"Supplies the open-set grounding detector used as the Factual Assessor to verify object presence and absence in generated images.","marker":"[24]"},{"why":"Supplies the direct preference optimization objective that turns corrected versus original responses into the training signal.","marker":"[31]"},{"why":"Stable Diffusion 3 is the image generator that renders the decoupled synthetic scenes.","marker":"[7]"},{"why":"DeepSeek-V2 performs caption rewriting, visual scene understanding, and generation of hallucination candidates and task queries.","marker":"[19]"},{"why":"POPE is the benchmark used to measure object-existence hallucination before and after post-training.","marker":"[18]"},{"why":"CHAIR is the benchmark used to measure hallucinated objects in image descriptions.","marker":"[32]"},{"why":"Supplies the SHR metric for sentence-level description hallucination and the HA-DPO baseline that Antidote is compared against.","marker":"[50]"},{"why":"LLaVA-1.5 is the baseline model family that Antidote post-trains and evaluates.","marker":"[21]"},{"why":"CC3M provides the caption pool for synthetic training data and the source images for the CP-Bench test set.","marker":"[35]"}],"fun_headline_variants":["Antidote: one recipe to cut LVLM hallucinations by half","One DPO step: LVLMs reject false premises and stop hallucinating","Self-correction beats hallucination: DPO on synthetic data","CP-Bench exposes false presuppositions; Antidote fixes them","LVLMs hallucinate when questions lie; Antidote teaches truth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training signal depends on the grounding detector being correct: if a supposedly absent hallucination-candidate object is actually present in a generated image, or a supposedly present object is missed, then the factual prior and the DPO preference labels are wrong, and the model is trained to deny or assert things that are false.","fun_headline_variants_meta":{"raw":{"variants":["Antidote: one recipe to cut LVLM hallucinations by half","One DPO step: LVLMs reject false premises and stop hallucinating","Self-correction beats hallucination: DPO on synthetic data","CP-Bench exposes false presuppositions; Antidote fixes them","LVLMs hallucinate when questions lie; Antidote teaches truth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3349,"prompt_tokens":972,"completion_tokens":2377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2283}},"tokens_in":588,"tokens_out":2377,"duration_ms":17391,"temperature":1.0,"reasoning_tokens":2283,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:28:00.044150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the generated training images, have human annotators label whether each $O_{pre}$ and $O_{hallu}$ object is truly present, and measure the Factual Assessor's precision and recall. If recall on $O_{pre}$ or precision on $O_{hallu}$ is materially below 100%, the preference pairs contain systematically mislabeled examples; a second check would be to train Antidote with those mislabeled pairs excluded and see whether the CP-Bench gains shrink.","supporting_citations":[{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Supplies the direct preference optimization objective that turns corrected versus original responses into the training signal."},{"cited_title":"Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning","cited_arxiv_id":null,"evidence_quote":"CC3M provides the caption pool for synthetic training data and the source images for the CP-Bench test set."}],"review_version":1}