{"id":"a39cf4e9-04fa-4f65-8c22-30341d586375","arxiv_id":"2508.20181","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CHAIR-DPO labels DPO preference pairs with CHAIR hallucinations computed from detector outputs, reducing object hallucinations in LLaVA models without proprietary judges.","lead":"The authors cut object hallucinations in multimodal language models by labeling preference pairs with the CHAIR metric, computed with an off-the-shelf object detector, and fine-tuning with DPO. The method lowers hallucination rates on three benchmarks while keeping general performance close to baseline, and it avoids proprietary judge models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Detector oracle miss rate could distort both training labels and CHAIR evaluation; need error analysis on the 70k/77k preference set.","rationale":"The reader's verdict and my stress-test agree that the detector-as-oracle assumption is the load-bearing weak point. The paper's central claim is empirical: CHAIR-DPO reduces hallucinations on three benchmarks using DETR as ground truth. Eq. 2 makes every preference label rely on detector completeness and vocabulary coverage. A false negative (missed object) produces a reversed label: correct object mention becomes the loser. DPO then actively suppresses that object. Because CHAIR-MSCOCO and Object HalBench are computed with the same COCO-vocabulary matching, the evaluation can also over-count hallucinations for objects outside the vocabulary; however, AMBER's human-annotated object labels are a genuinely independent check, so the shared-metric criticism alone would be weaker. The strongest additional evidence supporting the paper is the consistency across LLaVA-1.5-7B and LLaVA-MORE-8B (Table 1) and the AMBER improvement, which cannot be explained purely by DETR false negatives on the training set if AMBER labels are trusted. Yet the direction of the failure mode is exactly what a hallucination-reduction method would produce: mention fewer objects, lower CHAIRi, and (in Table 1) Coverage drops by ~5 points on AMBER. The concern is not that the numbers are fabricated; it is that the method may be optimizing CHAIRi's blind spots rather than true visual grounding. This is testable with an oracle error analysis and a corrected-label retraining. The reader identified the same weakest assumption; I agree, and CONDITIONAL remains the appropriate verdict, with the concrete audit as a necessary condition for full acceptance.","tokens_in":17636,"tokens_out":2067,"duration_ms":20058,"concrete_test":"Build a gold-standard audit set by sampling ~300 images from LLaVA-Instruct-665k (plus ~100 from the AMBER split), running LLaVA-1.5-7B to produce y1,y2, then having human annotators or a stronger detector (e.g., DETR with lower confidence threshold or open-vocabulary detector like Grounding DINO) mark which mentioned objects are truly present. Compute the CHAIRi mislabel rate in Eq. 2 (preferred response actually has higher human-judged hallucination, or winner/loser tie flip). Then retrain CHAIR-DPO on the filtered subset with labels corrected for the audited images, and compare AMBER CHAIRi/HalRate/Cover and CHAIR-MSCOCO CHAIRs against Table 1. If the corrected model's hallucination reduction shrinks by more than ~20% relative to Table 1, the detector-oracle blind spot is load-bearing; if not, the concern is bounded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CHAIR-DPO reduces measured object hallucination. In Eq. 2, winner/loser labels are chosen by CHAIRi computed against DETR-DC5-R101 detections restricted to MSCOCO classes. Two failure modes are load-bearing: (a) if DETR misses a visible object of an MSCOCO class, a response mentioning that object is labeled hallucinated and DPO learns to suppress valid object mention; (b) if the image contains an object outside the 80 MSCOCO classes (or the synonyms list), any mention is labeled hallucinated regardless of correctness. The paper reports no precision/recall analysis of this oracle on LLaVA-Instruct-665k or on the evaluation sets, and no sensitivity check (e.g., varying detector, confidence threshold, or vocabulary). The concern is structurally distinct from 'training and evaluation share CHAIR': AMBER's human-annotated objects provide partial independence, but CHAIR-MSCOCO and Object HalBench use the same COCO-vocabulary object-matching scheme, and the DPO model is specifically trained to avoid CHAIR-defined hallucinations. The most direct version of the concern is measurable: on the preference dataset, compute how often y1/y2 contain objects that human annotators or a different detector (e.g., grounded SAM, DETR with higher threshold) judge present but DETR missed, and re-run training with labels corrected for those cases. If the corrected pipeline yields substantially smaller hallucination reductions, the headline result is partly an artifact of the oracle's blind spots. Additionally, because the manuscript identifies the filtering step as removing ~90% of pairs and retaining 70k/77k, the retained pairs are exactly those with nonzero CHAIRi difference; noisy labels there directly drive every DPO gradient. This is a correctness risk, not a mere benchmark-choice issue, and it is the weakest load-bearing assumption in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CHAIR-DPO, a preference-optimization method to reduce object hallucination in multimodal LLMs. It constructs preference pairs by sampling two responses from a reference MLLM for each visual instruction in LLaVA-Instruct-665k, detecting objects with an off-the-shelf DETR detector, and scoring each response with CHAIR_i (Eq. 1). The lower-scoring response is treated as the winner and the higher-scoring one as the loser (Eq. 2), and DPO is applied with LoRA. Pairs with equal CHAIR_i are discarded, leaving 70k/77k training samples. The method is evaluated on AMBER, CHAIR-MSCOCO, and Object HalBench, where it reports large decreases in CHAIR_i/HalRate (e.g., LLaVA-1.5-7B AMBER CHAIR_i drops from 7.6 to 3.0 at beta=0.2), with moderate Coverage reduction, plus general-benchmark results showing limited degradation. The main claimed contribution is an open-source, non-proprietary pipeline for hallucination-alignment data.","tokens_in":17973,"tokens_out":8683,"duration_ms":99327,"significance":"If the reported effects are robust, the contribution is practically significant: it shows that DPO with a simple, open-source reward (CHAIR_i plus a detector) can match or exceed methods that rely on GPT-4/Gemini judges on several hallucination benchmarks. The paper ships code and models, uses two backbones, includes ablations (data filtering, beta), and provides qualitative examples. The main caveat is that the preference signal and the headline evaluation metrics are closely related, so the interpretation requires additional validation before the state-of-the-art claim can be fully accepted.","major_comments":[{"comment":"The entire preference dataset is built from DETR-DC5-R101 detections restricted to the MSCOCO vocabulary. If the detector misses an object that is actually present, a correct response mentioning that object is labeled as hallucinated, and DPO is trained to suppress it. The paper reports no precision/recall analysis of this oracle on LLaVA-Instruct-665k or on the evaluation sets, and no sensitivity check (e.g., different detector, confidence threshold, or vocabulary). Since all 70k/77k training labels depend on this detector, this missing analysis is load-bearing. I recommend (i) quantifying detector recall/precision on a human-annotated sample of the training data, (ii) retraining with corrected labels or with a second detector, and (iii) reporting whether the CHAIR reductions survive.","section":"Sec. 3.2, Eq. (2); Sec. 4.1"},{"comment":"The DPO objective directly optimizes CHAIR_i, and the headline benchmarks CHAIR-MSCOCO and Object HalBench compute the same CHAIR_i/CHAIR_s with the same object-matching scheme. Therefore part of the reported gains is expected from direct reward optimization. AMBER is a partial exception because its object labels are human-annotated, but it still uses the CHAIR_i formula, and the observed Coverage drop (e.g., 51.7 to 46.6 for LLaVA-1.5-7B at beta=0.2) shows a shift toward more conservative answers. To support the claim of 'effectively diminishing hallucinations', the paper should report precision/recall jointly (e.g., F1 or a Pareto plot), and evaluate on a benchmark whose ground truth is not CHAIR/COCO-based (e.g., POPE) or provide a human study on a sample.","section":"Eq. (1), Eq. (2), Table 1"},{"comment":"All experiments are single-run, and the final checkpoint is selected based on the lowest CHAIR_i on a 500-sample validation set. This selection criterion is identical to the main evaluation criterion and can inflate differences. Without multiple seeds and error bars, it is not possible to assess the variability of the reported state-of-the-art improvements. Please report mean ± std over at least 3 runs (or otherwise justify single-run reporting), and select checkpoints using a criterion not directly optimizing the headline metric.","section":"Appendix A.1; Tables 1-3"}],"minor_comments":[{"comment":"The text says 'ground-truth object annotations come from COCO ground-truth sentences,' which is inconsistent with the standard CHAIR implementation, where COCO instance annotations define ground-truth objects. If a different protocol was used, this should be stated explicitly.","section":"Appendix A.2"},{"comment":"'We impute it' should be 'We attribute it'.","section":"Section 4.2"},{"comment":"The Object HalBench row for LLaVA-1.5-7B baseline contains '..' placeholders; these should be filled or explicitly marked as not reported.","section":"Table 1"},{"comment":"The qualitative examples mention objects like 'glasses' and 'contrails' that are not in the COCO vocabulary. The paper should clarify that the method only addresses COCO-class object hallucinations, and that hallucinations outside this vocabulary are out of scope.","section":"Figure 3 and Section 4.2"},{"comment":"The exact prompts used for evaluation benchmarks (e.g., the generative task on AMBER, CHAIR-MSCOCO, and Object HalBench) are not stated. This information is needed for reproducibility.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The most urgent issue is the detector-oracle analysis. The authors should be encouraged to perform the suggested experiments; if the main results are robust to detector choice, the paper would be a solid contribution. The evaluation circularity is a real but solvable concern. I do not see evidence of intentional misuse; the omissions appear to be standard for a paper that needs additional empirical rigor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper replaces the proprietary LLM judges used in most DPO-based hallucination pipelines with CHAIR—the old captioning hallucination metric—run against an open DETR detector, and it works. LLaVA-1.5-7B AMBER CHAIRi goes from 7.6 to 3.0, and the reductions hold on CHAIR-MSCOCO and Object HalBench across two backbones (LLaVA-1.5-7B and LLaVA-MORE-8B). General benchmarks hold up for LLaVA-1.5-7B, with a modest regression on LLaVA-MORE that the authors acknowledge. The practical takeaway is real: you can get most of the hallucination-mitigation benefit without paying for GPT-4/Gemini labels.\n\nThe new bit is the pipeline: CHAIRi computed against DETR detections selects winner/loser pairs for DPO, and the paper shows filtering out pairs with zero CHAIRi difference helps. It is a simple, honest engineering contribution, not a conceptual breakthrough, and the open-source components make it easy to reproduce and build on. The ablations and the performance-preservation table are well done.\n\nThe soft spot is the detector oracle. Eq. 2 uses DETR detections as ground truth for which objects exist. If DETR misses small or occluded objects, responses that correctly mention them get labeled hallucinated, and DPO pushes the model to suppress valid mentions. The retained 70k/77k pairs are exactly the nonzero-CHAIRi-difference pairs, so every gradient inherits the detector's blind spots. The paper reports no precision/recall of the oracle and no sensitivity check across detectors or confidence thresholds. On AMBER, the human-annotated objects provide a partial check: Coverage drops about 5 points, which is consistent with some suppression of valid objects, but CHAIRi still improves, so the headline is not purely artifact. On CHAIR-MSCOCO and Object HalBench you are largely reporting the metric you optimized, so those numbers are expected to look good. A detector error analysis, or re-running training with a second detector, would settle whether the magnitude is inflated.\n\nMinor issues: no seeds or error bars; checkpoints selected on validation CHAIRi; the state-of-the-art claim needs the REVERSE caveat—REVERSE gets a lower HalRate on AMBER at beta=0.2 but collapses Coverage to 26.9, which the paper does note, so the claim is fair with that context.\n\nBottom line: a solid, useful paper for MLLM alignment practitioners, and a serious referee should see it. I would want the detector noise analyzed before publication, but the method deserves proper review, not a desk reject.","headline":"CHAIR-DPO is a clean, cheap DPO recipe that cuts object hallucination substantially across two backbones; the detector-as-oracle noise is the one real soft spot and should be quantified, but the result is not just metric-chasing.","tokens_in":18590,"tokens_out":3749,"would_cite":true,"duration_ms":41222,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that object hallucinations in multimodal LLMs can be sharply reduced by using the CHAIR captioning metric as a preference labeler for DPO, with only an off-the-shelf object detector as the oracle.","keywords":["multimodal LLM","object hallucination","direct preference optimization","CHAIR metric","object detection oracle","preference data","alignment","LLaVA"],"falsifier":"Take a set of images containing objects that fall outside the MSCOCO class list or that are small, occluded, or otherwise likely to be missed by the detector. Run CHAIR-DPO fine-tuning and then measure whether the fine-tuned model mentions those genuinely present objects less frequently than the baseline model. A statistically significant drop in truthful mentions would confirm that the detector's blind spots are transferred into the model as learned suppression.","tokens_in":17533,"feed_emoji":"🖼️","tokens_out":3548,"duration_ms":38502,"temperature":0.7,"pith_summary":"The paper claims that object hallucinations in multimodal large language models can be substantially reduced by recasting the standard CHAIR captioning metric as a reward for Direct Preference Optimization (DPO). Instead of relying on proprietary judge models to decide which responses are hallucinated, the method scores a pair of model-generated answers with CHAIR, using detections from an off-the-shelf object detector as the ground truth, and then fine-tunes the model to prefer the lower-scoring answer. This matters because it offers a simple, fully open-source path to hallucination alignment. Experiments on AMBER, CHAIR-MSCOCO, and Object HalBench show large drops in hallucination rates (e.g., AMBER CHAIRi from 7.6 to 3.0 on LLaVA-1.5-7B) while largely preserving general capabilities.","feed_headline":"CHAIR-DPO cuts MLLM object hallucinations without proprietary judges","feed_subtitle":"Using the CHAIR metric as a DPO preference signal and a plain detector as oracle drops AMBER CHAIRi from 7.6 to 3.0.","key_machinery":"The central object is the CHAIRi score (Eq. 1), the fraction of mentioned objects that are not detected, which is used to label winner and loser completions in DPO (Eq. 3). The detector supplies the oracle set of real objects; a synonym list aligns the model's words to MSCOCO class names. A data-filtering step removes zero-difference pairs to avoid noisy supervision.","core_discovery":"CHAIR-DPO is a training pipeline that converts the CHAIR metric into a preference oracle. For each image and prompt, the model itself samples two candidate answers; an off-the-shelf detector (DETR with MSCOCO classes) provides the set of objects presumed present; CHAIRi is computed for each answer, with any mentioned object missing from the detection set counted as hallucinated. The answer with lower CHAIRi becomes the DPO winner and the other the loser. Fine-tuning with DPO then increases the probability of non-hallucinated completions. The paper demonstrates on three benchmarks that this procedure substantially reduces measured hallucinations on two different MLLMs, and that discarding pre","pith_inferences":["Because the detector vocabulary is fixed to MSCOCO classes, the method may silently penalize truthful mentions of objects outside that vocabulary; extending the oracle to larger or open-vocabulary detectors could be a natural next step.","The detector's miss rate is the hidden variable: if it misses objects that are actually present, CHAIR-DPO will train the model to suppress valid details, which the paper's coverage metric may not fully expose.","The approach is likely orthogonal to training-free decoding fixes such as contrastive decoding or over-trust penalties, so combining CHAIR-DPO with such methods could yield further gains.","The same preference-labeling schema could be applied with other preference losses (e.g., IPO, KTO) or other grounding signals, such as referring-expression grounding models, to move beyond pure object lists."],"forward_implications":["Object hallucination in MLLMs can be mitigated without proprietary judges, lowering the barrier for aligned multimodal models.","The method transfers across at least two open MLLMs (LLaVA-1.5-7B and LLaVA-MORE-8B), suggesting it is architecture-agnostic.","The beta regularizer in DPO provides a continuous trade-off between hallucination reduction and coverage of real objects.","Filtering out uninformative preference pairs can cut the training set by roughly 90% while improving final hallucination metrics.","The small coverage penalty observed on AMBER indicates hallucination reduction comes at a modest, controllable cost to object recall."],"supporting_citations":[{"why":"Defines the CHAIR metric and its implementation, which is the core reward signal for labeling preference pairs.","marker":"[41]"},{"why":"Introduces DPO, the optimization objective used to fine-tune the MLLM toward preferred answers.","marker":"[37]"},{"why":"Provides DETR-DC5-R101, the off-the-shelf detector that supplies ground-truth object sets for CHAIR computation.","marker":"[5]"},{"why":"Supplies the base LLaVA-1.5-7B model and its visual instruction tuning dataset, which is used to sample preference pairs.","marker":"[28]"},{"why":"Provides the AMBER benchmark and its hallucination metrics used for evaluation.","marker":"[51]"},{"why":"Provides the CHAIR-MSCOCO benchmark setup and the EOS baseline that CHAIR-DPO compares against.","marker":"[59]"},{"why":"Contributes the Object HalBench evaluation subset and protocol used in the comparison.","marker":"[57]"},{"why":"Baseline HA-DPO that CHAIR-DPO must exceed to demonstrate the advantage of its simpler preference-data construction.","marker":"[61]"},{"why":"Baseline HALVA which uses a proprietary judge (Gemini) for preference data, compared against CHAIR-DPO's open-source approach.","marker":"[43]"},{"why":"Baseline REVERSE that reaches lower hallucination rates at the cost of coverage, serving as the strongest comparison point on AMBER.","marker":"[53]"}],"fun_headline_variants":["CHAIR-DPO reduces MLLM hallucinations using CHAIR metric","CHAIR-based DPO reduces object hallucinations in MLLMs","Cut MLLM hallucinations with CHAIR-guided DPO","CHAIR-DPO: use CHAIR as reward to curb MLLM hallucinations","CHAIR-DPO trains MLLMs to avoid hallucinations"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The labels of which objects are 'hallucinated' come entirely from an off-the-shelf detector with a fixed MSCOCO vocabulary; if the detector misses objects that are actually in the image, the model is trained to suppress truthful mentions.","fun_headline_variants_meta":{"raw":{"variants":["CHAIR-DPO reduces MLLM hallucinations using CHAIR metric","CHAIR-based DPO reduces object hallucinations in MLLMs","Cut MLLM hallucinations with CHAIR-guided DPO","CHAIR-DPO: use CHAIR as reward to curb MLLM hallucinations","CHAIR-DPO trains MLLMs to avoid hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001417,"raw_usage":{"total_tokens":5581,"prompt_tokens":788,"completion_tokens":4793,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":4718}},"tokens_in":532,"tokens_out":4793,"duration_ms":33622,"temperature":1.0,"reasoning_tokens":4718,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:13:41.743134+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of images containing objects that fall outside the MSCOCO class list or that are small, occluded, or otherwise likely to be missed by the detector. Run CHAIR-DPO fine-tuning and then measure whether the fine-tuned model mentions those genuinely present objects less frequently than the baseline model. A statistically significant drop in truthful mentions would confirm that the detector's blind spots are transferred into the model as learned suppression.","supporting_citations":[{"cited_title":"Object Hallucination in Image Captioning","cited_arxiv_id":null,"evidence_quote":"Defines the CHAIR metric and its implementation, which is the core reward signal for labeling preference pairs."},{"cited_title":"Direct Preference Optimization: Your Language Model is Secretly a Reward Model","cited_arxiv_id":null,"evidence_quote":"Introduces DPO, the optimization objective used to fine-tune the MLLM toward preferred answers."},{"cited_title":"End-to-End Object Detection with Transformers","cited_arxiv_id":null,"evidence_quote":"Provides DETR-DC5-R101, the off-the-shelf detector that supplies ground-truth object sets for CHAIR computation."},{"cited_title":"Improved Baselines with Visual Instruction Tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the base LLaVA-1.5-7B model and its visual instruction tuning dataset, which is used to sample preference pairs."},{"cited_title":"Less is More: Mitigating Multimodal Hallucina- tion from an EOS Decision Perspective","cited_arxiv_id":null,"evidence_quote":"Provides the CHAIR-MSCOCO benchmark setup and the EOS baseline that CHAIR-DPO compares against."},{"cited_title":"RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback","cited_arxiv_id":null,"evidence_quote":"Contributes the Object HalBench evaluation subset and protocol used in the comparison."},{"cited_title":"Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment","cited_arxiv_id":null,"evidence_quote":"Baseline HALVA which uses a proprietary judge (Gemini) for preference data, compared against CHAIR-DPO's open-source approach."}],"review_version":1}