{"id":"ea16a378-a0e2-4700-8600-50370cbb6872","arxiv_id":"2412.04903","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A critic trained on filtered GPT-4V scoring data selects the best and worst self-generated responses, and DPO with extra SFT improves hallucination and reasoning benchmarks in multimodal LLMs.","lead":"EACO trains a critic model on 137,000 scoring instructions to judge an image model's own answers, then uses those judgments to pick preferred and non-preferred responses for preference fine-tuning. The method reports big gains on hallucination and reasoning benchmarks while using only 5,000 unlabeled images, offering a low-cost alignment recipe for open multimodal assistants.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates critic-guided DPO from the post-DPO SFT stage, so the central claim that critic-scored preferences drive the reported gains is not established.","rationale":"The reader identified the reliability of the critic model as the weakest assumption. My concern is related but distinct: even if the critic's scores are perfectly reliable, the paper does not demonstrate that the critic-guided DPO stage is what causes the reported gains. The post-DPO enhanced SFT stage is part of the method and could plausibly account for all of the measured improvement, especially since the paper cites STIC [10] as its inspiration and reuses 5k instruction-tuning data with model-generated captions. The absence of a DPO-only or SFT-only ablation is a concrete, checkable gap in the causal attribution, and it is more directly tied to the central novelty claim than the abstract's HallusionBench wording or the sign issue in Equation (2). I am not rejecting the paper: the consistent gains across three base models in Table 1 are real evidence that the full pipeline helps, and the self-rewarding comparison provides some signal that the trained critic matters. But that comparison still bundles critic-guided DPO with enhanced SFT, so it does not settle which component is responsible. I therefore keep the reader's CONDITIONAL verdict rather than strengthening or weakening it. A clean ablation experiment would either confirm the central mechanism or force the claim to be reframed as an SFT-based improvement with a critic that is not yet shown to matter.","tokens_in":22732,"tokens_out":4146,"duration_ms":44649,"concrete_test":"Using LLaVA-v1.6-Mistral-7B, 5k MSCOCO images, and the paper's hyperparameters, run four conditions: (A) full EACO: critic-scored DPO plus enhanced SFT; (B) DPO only: critic-scored DPO without enhanced SFT; (C) enhanced SFT only: the same SFT data and model-generated captions with no DPO; (D) random-pair DPO plus enhanced SFT: select y_w and y_l by random permutation of critic scores. Evaluate all conditions on the same seven-benchmark normalized average used in Section 5.3, with at least three seeds to estimate run-to-run noise. If (C) matches (A) within noise, the critic-guided DPO component is not load-bearing; if (D) matches (A), critic scoring quality is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that critic-guided DPO on self-generated preferences from 5k images produces the reported improvements. However, the full pipeline includes an \"enhanced supervised fine-tuning (SFT) stage after preference tuning\" described in Section 4.3 and modeled after STIC [10], which itself uses 5k instruction-tuning samples with model-generated captions. Section 5.3 lists ablations on preference-data scale, iterative alignment, critic prompt style, and self-rewarding critic, but it never removes the DPO component or the enhanced SFT component. There is no condition with enhanced SFT only, no DPO only, no random preference pairs, and no reversed critic scores. The reported iteration-0 to iteration-1 jump (61.106 to 66.327 in Section 5.3) is for the full pipeline, so it cannot attribute the gain to the critic model or to DPO specifically. The preference-data scaling results also show only tiny gains from 5k to 10k to 15k samples (0.204 and 0.071), suggesting the DPO component itself may contribute little. If enhanced SFT alone recovers most of the gain, the unique EACO mechanism is unsupported; if random preference pairs perform as well as critic-selected pairs, the critic model is unnecessary for the reported result. This attribution gap is load-bearing because the headline claim is specifically about critic-guided preference alignment, not about the additional SFT stage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EACO, a framework that trains a critic model on GPT-4V-annotated instruction data, uses that critic to score self-generated responses from a base MLLM, constructs preference pairs from the highest- and lowest-scoring responses, and then performs DPO with a modified objective (Equation 2) followed by an enhanced supervised fine-tuning stage. The method is evaluated on LLaVA-v1.6-Mistral-7B, Bunny-8B, and MiniCPM-V-8B, reporting consistent improvements across comprehensive, domain-specific, and hallucination benchmarks, with the largest gains on hallucination benchmarks.","tokens_in":22978,"tokens_out":6806,"duration_ms":65653,"significance":"If fully validated, EACO would offer an economical (5k images) and model-agnostic alignment recipe that reduces reliance on proprietary reward models while improving reasoning and hallucination metrics. The paper provides source code, and the reported gains are consistent across three different base architectures, which strengthens the empirical claim. The principal limitation is that the unique contribution of critic-guided DPO is not isolated from the post-DPO SFT stage, so the current evidence does not pin down the mechanism responsible for the gains.","major_comments":[{"comment":"The abstract states that EACO 'reduces the overall hallucinations by 65.6% on HallusionBench.' This is inconsistent with Table 1, where the HallusionBench score increases from 29.1 (LLaVA-v1.6-7B) to 48.2 (EACO). A relative increase of 65.6% in the accuracy metric is not a 65.6% reduction in hallucinations; the corresponding reduction in hallucination rate would be (70.9 - 51.8)/70.9 ≈ 26.9%. The current wording misstates the direction and magnitude of the effect and should be corrected in the abstract and in Section 5.2 where the same claim appears.","section":"Abstract and Section 5.2"},{"comment":"The paper attributes the reported improvements to critic-guided DPO, but the full pipeline also includes an enhanced SFT stage after DPO (Section 4.3). The ablation study in Section 5.3 varies data scale, iteration count, critic prompt style, and self-rewarding critic, but never removes the DPO component or the enhanced SFT component. Without conditions such as enhanced-SFT-only, DPO-only, random preference pairs, or reversed critic scores, the specific contribution of the critic-guided DPO to the observed gains is not established. This attribution gap is load-bearing because the central claim is about critic-guided preference alignment.","section":"Section 5.3 (ablation study)"},{"comment":"The modified DPO objective in Equation (2) includes the term -(α|yw| - α|yl|) inside the log-sigmoid, which is described as discouraging verbose responses. As written, this term penalizes only the length difference between the preferred and non-preferred responses; it does not penalize verbosity of the generated output in an absolute sense. If the intended regularization is to discourage long responses, the term should depend on absolute lengths (e.g., -α(|yw|+|yl|) or a length penalty appended outside the log-sigmoid). Please clarify the formulation and, if the equation is a typographical error, correct it, because the stated behavior of the objective does not match the equation.","section":"Section 4.3, Equation (2)"},{"comment":"The text states that 'the self-rewarding method shows some minor improvements in certain metrics, such as MME P improving from 1512.3 to 1506.2.' However, Table 1 reports MME P of 1506.2 for Self-Rewarding, which is lower than the baseline 1512.3, not an improvement. This is a factual contradiction that should be fixed.","section":"Section 5.3, Self-Rewarding Model paragraph"}],"minor_comments":[{"comment":"There are numerous typos and inconsistent abbreviations; examples include 'Multilingual Language Model' in Section 3.2 (should be 'Multimodal'), 'reasonging' in Figure 1, 'Perferred' in Figure 4, 'Imapct' in Section 5.3, 'Scaling up dateset' in Section 5.3, and 'Ohters' in Table 1's footnote. A careful proofreading pass is recommended.","section":"Throughout"},{"comment":"The notation for the reference model is inconsistent: the algorithm line shows 'pref (yw|x,v)' where a subscript or a clear reference-model notation (e.g., p_ref) is needed. Please align the notation between Algorithm 1 and Equation (2).","section":"Algorithm 1 and Equation (2)"},{"comment":"The text reports a gain of 'about 8.5%' for 5k samples, but Figure 5's left panel shows gains in the range 7–9% for data scales; please make sure the axis labels and numbers are consistent, and specify whether the percentages are relative or absolute.","section":"Section 5.3 and Figure 5"},{"comment":"The qualitative comparison of critics (Tables 3–6) shows that the LLaVA-v1.6 model, used as a critic, assigns perfect scores to flawed responses. This is an interesting observation, but the main text does not report any quantitative reliability measure of the trained critic (e.g., agreement with GPT-4V on a held-out set). Adding such a measure would support the assumption that the critic's rankings are meaningful.","section":"Supplementary Section 7"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a strong and consistent set of benchmark gains, and the code availability is a plus. The main concern is the missing ablation that isolates critic-guided DPO from the enhanced SFT stage. Given that the paper's core novelty is the critic-guided preference construction, this gap is important. I would ask the authors to add the required ablation conditions and to correct the misleading HallusionBench claim in the abstract before reconsideration. There is no indication of plagiarism or duplicate publication, but the authors' own 'Discussion with LLaVA-Critic' is somewhat defensive; the technical differences could be presented more briefly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: the reported pipeline is a genuinely economical alignment recipe—5k images, a critic distilled from filtered Silkie data, argmax/argmin self-scoring, DPO, and a post-DPO SFT—and Table 1 shows consistent gains across three different base models. If those numbers reproduce, it's a practical template for low-cost preference tuning in multimodal models. But the abstract says EACO “reduces hallucinations by 65.6%” when HallusionBench actually goes from 29.1 to 48.2, which is an improvement in the benchmark score, not a reduction in hallucinations. That direction error is not cosmetic; it undermines trust in the reporting.\n\nWhat's genuinely new here is the specific combination of components, not any single one. The score-gap filtering of Silkie data, the max/min selection rule, and the length-regularized DPO with post-DPO SFT are each borrowed from or adjacent to prior work, but the package and the three-model evaluation are not in the cited papers. The authors also deserve credit for openly discussing LLaVA-Critic and for stating a clear limitation: the critic is weak on chain-of-thought tasks.\n\nThe soft spots are real but mostly fixable. The most load-bearing one, which the stress-test note gets right, is the missing ablation: Section 5.3 never removes the enhanced SFT stage or uses random preference pairs, so the central claim that critic-guided DPO drives the gains is not established. The preference-data scaling results show only tiny gains from 5k to 15k samples, which weakens the case that the critic's rankings matter much. I also think the sign in Equation (2) contradicts the prose: the term −α log pθ(yw) would penalize the preferred response, not reinforce it. And the headline “8.5% average improvement” depends on an underspecified seven-benchmark normalization with no error bars. Finally, the critic is trained on GPT-4V-labeled Silkie data, so the word “self-generated” overstates the independence from proprietary labels.\n\nDespite these problems, the paper is not incoherent and the central idea is plausible. It deserves a serious referee, but only with major revisions: fix the abstract and equation, add the missing ablations, and release code. As written, I wouldn't cite it in my own work yet.\n\nFor a reading group, it's a decent example of how incremental MLLM alignment papers are structured, but the attribution gap makes it more useful as a caution than as a template.","headline":"A useful but sloppy recipe for critic-guided MLLM alignment; the headline claim is misreported and the key ablation is missing.","tokens_in":23593,"tokens_out":2048,"would_cite":false,"duration_ms":23616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EACO claims that 5,000 self-generated images, graded by a separate Critic model, are enough to cut hallucinations by 65.6% on HallusionBench and improve reasoning by 21.8% on MME-Cognition.","keywords":["multimodal large language models","hallucination mitigation","Direct Preference Optimization","critic model","self-generated preference data","visual reasoning","preference alignment"],"falsifier":"Run the full EACO pipeline on the same 5,000 images but replace the Critic's scores with random rankings of the same self-generated responses; if HallusionBench and MME-Cognition still improve by the reported margins, the critic-guided selection is not the cause of the gains.","tokens_in":22449,"feed_emoji":"👁️","tokens_out":13281,"duration_ms":118803,"temperature":0.7,"pith_summary":"EACO claims that a multimodal large language model can align itself using preference data it generates from just 5,000 unlabeled images, provided a separately trained Critic model grades the responses. The paper builds the Critic by fine-tuning an MLLM on over 137,000 scoring instructions refined from 51,000 images labeled by GPT-4V, then uses the Critic's scores to pick the best and worst of several self-generated answers. Direct Preference Optimization on those pairs, followed by an extra supervised fine-tuning stage, is reported to reduce hallucinations by 65.6% on HallusionBench and improve reasoning by 21.8% on MME-Cognition. The practical claim is that expensive human or proprietary preference labels can be replaced by a cheap open critic without sacrificing alignment quality.","feed_headline":"Critic-graded self-preference loop cuts vision-LLM hallucinations by 65.6%","feed_subtitle":"Reasoning climbs 21.8% on MME-Cognition, and the same 5k-image recipe lifts three model families.","key_machinery":"The load-bearing mechanism is the Critic model plus the selection rule it feeds. The Critic is an MLLM fine-tuned with LoRA on over 137,000 scoring instructions built from 51,000 images, producing scores across dimensions such as relevance, basic elements, and clarity. For each image-question pair, the target model self-generates $n$ responses; selection follows $y_w = \\arg\\max_j S_i^{y_j}$ and $y_l = \\arg\\min_j S_i^{y_j}$, where $S_i^{y_j}$ is the critic score. These pairs drive a modified DPO objective that adds a preferred-response reinforcement term and a verbosity penalty, followed by an enhanced SFT stage on DPO-improved captions. The Critic does the work of replacing expensive human or proprietary preference labels with a model trained on GPT-4V-labeled data.","core_discovery":"The central claim is that a critic-guided self-preference loop is enough to meaningfully align an MLLM. Starting from a target model, the method generates multiple responses per image-question pair, has a Critic model trained on refined GPT-4V-scored data assign quality scores, and selects the highest-scoring response as preferred and the lowest as non-preferred. These pairs are used in a modified DPO objective with two extra regularizers, one reinforcing the preferred response and one penalizing verbosity, and the resulting model is then given an enhanced SFT pass on its own DPO-improved captions. On LLaVA-v1.6-Mistral-7B this yields a 65.6% relative reduction on HallusionBench and a 21.8% gain on MME-Cognition, with consistent improvements also reported for Bunny-8B and MiniCPM-V. The paper's claim is therefore that open-source critic models can stand in for human and proprietary labelers, making preference alignment economical and transferable across architectures.","pith_inferences":["The paper does not ablate the two extra DPO regularizers separately, so a natural next test is to run EACO with only the verbosity penalty or only the preferred-response reinforcement; this would show how much of the gain is due to critic-ranked pairs rather than the modified objective.","The reported self-rewarding baseline is much weaker than the Critic version, which suggests the Critic's training data, not the self-generation loop, is what creates the improvement; swapping the Critic for GPT-4V scores on the same 5,000 images would quantify how close the open critic is to proprietary labeling.","The 65.6% HallusionBench reduction may be tied to the visual-faithfulness dimension in the Silkie-derived scoring rubric; a testable extension is to check whether the Critic's dimension scores individually predict gains on different hallucination categories, such as existence versus attribute errors."],"forward_implications":["Preference data for MLLM alignment no longer needs to be bought from human annotators or proprietary APIs: 5,000 unlabeled images and a fine-tuned critic are enough to reproduce large benchmark gains.","Hallucination control and reasoning improvement can come from the same tuning recipe, since the reported gains appear simultaneously on HallusionBench and MME-Cognition.","Because the same procedure improves LLaVA-v1.6, Bunny-8B, and MiniCPM-V, the alignment effect transfers across model families rather than being an artifact of one backbone.","The ablation's diminishing returns beyond 5,000 samples and after the first iteration imply that a single critic-guided preference pass captures most of the available benefit for this data budget."],"supporting_citations":[{"why":"Supplies the GPT-4V-annotated multimodal instructions that EACO samples and refines into a critic training set of 51,000 images and over 137,000 scoring instructions.","marker":"[20]"},{"why":"Defines Direct Preference Optimization, the base objective that EACO modifies with two extra regularization terms.","marker":"[34]"},{"why":"Provides the self-training recipe and the enhanced supervised fine-tuning stage applied after preference tuning.","marker":"[10]"},{"why":"Defines the LLaVA-v1.6-Mistral-7B backbone and baseline against which the headline improvements are measured.","marker":"[25]"},{"why":"Supplies the 5,000 unlabeled MSCOCO images used to self-generate the preference data.","marker":"[7]"},{"why":"Provides LoRA, the low-rank adaptation used to train the Critic and to run DPO and enhanced SFT efficiently.","marker":"[16]"}],"fun_headline_variants":["Self-critic loop cuts vision-LLM hallucinations by 65.6%","Open-source critic grades self-responses to slash hallucinations by 65.6%","Critic-guided preference tuning lifts reasoning 21.8% on MME-Cognition","Economical self-preference loop improves MLLM reasoning by 21.8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the separate evaluation model called the Critic, trained on GPT-4V-labeled data, ranks the target model's own self-generated responses reliably enough that the highest- and lowest-scored outputs really are the better and worse answers; if the scores are noisy or biased toward wording rather than visual facts, the preference pairs become near-random and the reported gains would come from the extra SFT stage instead.","fun_headline_variants_meta":{"raw":{"variants":["Self-critic loop cuts vision-LLM hallucinations by 65.6%","Open-source critic grades self-responses to slash hallucinations by 65.6%","Critic-guided preference tuning lifts reasoning 21.8% on MME-Cognition","Economical self-preference loop improves MLLM reasoning by 21.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3295,"prompt_tokens":1035,"completion_tokens":2260,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":2171}},"tokens_in":651,"tokens_out":2260,"duration_ms":16914,"temperature":1.0,"reasoning_tokens":2171,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:09:52.575396+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full EACO pipeline on the same 5,000 images but replace the Critic's scores with random rankings of the same self-generated responses; if HallusionBench and MME-Cognition still improve by the reported margins, the critic-guided selection is not the cause of the gains.","supporting_citations":[{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Defines Direct Preference Optimization, the base objective that EACO modifies with two extra regularization terms."}],"review_version":1}