{"id":"be69869e-162d-4f41-92fd-3d176a82059b","arxiv_id":"2411.18203","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A DPO-trained VLM critic that critiques and iteratively refines a reasoning VLM improves accuracy on several multimodal benchmarks, with large gains on MathVista and RealWorldQA.","lead":"The paper introduces Critic-V, a framework where a separate vision-language critic model critiques and iteratively refines answers from a reasoning model. It reports accuracy gains across several multimodal benchmarks, with the Qwen2-VL-7B plus Critic-V system beating GPT-4V on 5 of 8 benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unnamed training datasets and no decontamination analysis leave benchmark leakage a live confound for the headline gains; the claimed transferable critic improvement is unverified without an overlap check.","rationale":"The reader's conditional verdict already identifies the missing training-data provenance as the central weakness. My stress-test agrees and sharpens it: the same VQA-style datasets used for evaluation are plausible sources for the critique-VQA training pairs, and the reported gains on MathVista and RealWorldQA are large enough that even partial overlap could materially affect them. The paper gives no exclusion procedure, so the concern is not resolved by anything in the manuscript. I do not see a stronger objection: the theoretical RL/TextGrad equations are loose but not load-bearing, and the single-run nature of the experiments is secondary once temperature is near zero. The availability of the dataset and code means the overlap check is feasible and decisive, which is why the appropriate outcome remains conditional rather than rejection. I credit the authors for releasing links to the dataset, code, and paper, making this audit possible.","tokens_in":20431,"tokens_out":3380,"duration_ms":36401,"concrete_test":"Download the released critique-VQA dataset and compute exact and near-duplicate overlap against the evaluation subsets used in Table 1: MathVista testmini, MathVerse, RealWorldQA, ScienceQA test, MMBench, MMStar, SEEDBench, and MMT-Bench. Use normalized question-text hashing plus image hashing or CLIP embedding similarity to catch rephrased questions and reused images. If any non-zero overlap is found, rerun Critic-V only on the non-overlapping benchmark items and compare the deltas against the full-set results; if the gains shrink or disappear, the headline claim is not supported without dataset decontamination. A complementary check is to require the authors to disclose the exact source datasets and split construction for the 29,012 training pairs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the DPO-trained critic transfers general critique ability, so the reasoner improves because of better feedback rather than memorized benchmark answers. Section 2.2 constructs the 29,012 critique-VQA pairs by applying VEST to 'question-image pairs from VQA datasets', and Figure 3 says only 'Several VQA Datasets'. No source dataset or data split is named anywhere, and Table 7 reports only token statistics for questions, chosen critiques, and rejected critiques. This matters because the evaluation suite includes RealWorldQA, MathVista, MathVerse, ScienceQA, MMT-Bench, MMBench, MMStar, and SEEDBench: these are themselves VQA-style datasets. If any of their test questions, images, or near-duplicates appear in critique-VQA, the critic can memorize question-specific facts from the GPT-4o-generated fake answers and from the VLM critiques shown during training. At inference, the critic would then supply targeted hints on exactly those benchmark questions, inflating gains such as MathVista +11.8, RealWorldQA +4.8, and MathVerse +7.1 without improving general reasoning. The paper contains no decontamination statement, no list of contributing datasets, and no overlap analysis. Because the framework's practical value rests on the critic's transferable feedback rather than on memorization, this unaddressed leakage path is the most load-bearing weakness in the empirical argument.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Critic-V, an actor-critic style framework for vision-language models in which a separate Critic VLM provides natural-language feedback to a Reasoner VLM. The Critic is trained with DPO on a preference dataset of critiques built by VEST: GPT-4o inserts one to five fake details into ground-truth VQA answers, several VLMs produce critiques, and a rule-based reward combining a Jaccard index with a GPT-4o evaluation score ranks the critiques. At inference, the Reasoner generates an answer, the Critic critiques it, and the Reasoner revises its answer with the critique appended to the prompt, iterating until a stopping condition. Experiments on eight multimodal benchmarks show consistent improvements over the base models (Qwen2-VL-7B, DeepSeek-VL-7B, LLaVA-v1.5-7B), with Qwen2-VL-7B+Critic-V reported to outperform GPT-4V on five of the benchmarks.","tokens_in":20674,"tokens_out":4222,"duration_ms":39102,"significance":"If the empirical claims hold, Critic-V offers a practical, model-agnostic recipe for improving VLM reasoning by adding an external, preference-optimized critic, and the released 29,012-pair critique dataset is a useful resource. The paper's strengths include a clean framework, a large constructed dataset, consistent improvements across many benchmarks, and ablations separating the contribution of DPO training and of the evaluation prompt. The main significance caveat is that the transferability claim -- that the critic improves reasoning generally rather than memorizing question-specific hints -- depends on training-data provenance and on how well GPT-4o-inserted errors match real VLM errors; both are currently unverified.","major_comments":[{"comment":"The training data provenance is not disclosed. The paper states that VEST is applied to 'question-image pairs from VQA datasets' (Section 2.2) and Figure 3 says only 'Several VQA Datasets', while Table 7 reports only token statistics, not the source datasets or splits. The evaluation suite (RealWorldQA, MathVista, MathVerse, ScienceQA, MMT-Bench, MMBench, MMStar, SEEDBench) itself consists of VQA-style benchmark datasets. If any of these benchmark questions, images, or near-duplicates appear in the critique-VQA training set, the critic could memorize question-specific facts from the GPT-4o-generated fake answers and the VLM critiques, and then supply targeted hints on exactly those benchmark questions, inflating gains such as MathVista +11.8 and MathVerse +7.1 without improving general reasoning. The paper contains no decontamination statement and no overlap analysis. Because the framework's practical value rests on the critic's transferable feedback, the authors should name the contributing datasets and splits and report a formal overlap check against all eight evaluation benchmarks.","section":"Section 2.1, Section 6 (Algorithm 3), Appendix 10"},{"comment":"The theoretical framing does not match the implemented algorithm. Equations (2)-(5) describe a reinforcement-learning/TextGrad-style prompt update with gradients and a learning-rate parameter eta, and Equation (6) describes a policy-gradient update for the Critic. However, Algorithm 3 implements a simple loop where the critique is concatenated to the prompt (P_reasoner <- P_reasoner + delta P_reasoner), and Appendix 10 states eta=1.0, i.e., full concatenation. TextGrad is not used in the evaluation loop, the Critic is not updated by policy gradient, and the 'satisfactory' stopping condition in Algorithm 3 is not supported by any mechanism described in the Critic's training or inference prompt. The claim that the framework is 'theoretically driven by a reinforcement learning framework' (Abstract) is therefore not substantiated by the presented algorithm. Please either align the theoretical development with the actual mechanism or remove/replace the RL/TextGrad claims.","section":"Section 2.2, Eq. (8), Appendix 9"},{"comment":"The critique-quality signal is self-referential in a way that may limit transfer. The fake errors inserted into the ground-truth answers are generated by GPT-4o (Section 2.2, Algorithm 1), and the RBR score itself includes a GPT-4o-based evaluation term (Eq. (8), with alpha=0.1). The Critic is therefore trained to detect the particular error distribution that GPT-4o produces when asked to fabricate details, and the preference labels are partly assigned by the same model family. The paper assumes, but does not demonstrate, that this distribution simulates the errors made by Qwen2-VL-7B, DeepSeek-VL-7B, and LLaVA-v1.5-7B at inference time. Since the central claim is that the trained critic generalizes to real VLM errors, the authors should provide evidence for this match: for example, compare the distribution of GPT-4o-inserted bugs with actual VLM error patterns on a held-out sample, or evaluate the critic's error-detection accuracy against human-annotated VLM errors.","section":"Table 1 and Section 3.2"},{"comment":"The empirical comparison lacks variance or significance information. Table 1 reports single percentage numbers per benchmark, and the text describes improvements such as '+4.8', '+2.1', '+1.8', '+1.6', and '+0.4' as significant, but no standard deviations, confidence intervals, or repeated-evaluation results are provided. Even with temperature near zero, evaluation randomness can arise from benchmark subsets, decoding settings, or the Critic's stopping decisions. For the smaller deltas, the claim that Critic-V outperforms baselines on those benchmarks is not statistically supported. Please add multiple runs with variance estimates or a bootstrap analysis over the evaluation questions, particularly for the benchmarks with gains below about two points.","section":"Section 3.2"}],"minor_comments":[{"comment":"The heading 'Result ans Analysis' contains a typo; it should read 'Results and Analysis'.","section":"Figure 3 caption"},{"comment":"The caption uses 'VisualQA (critique-VQA)', but the dataset is called 'critique-VQA' elsewhere; unify the name.","section":"Section 2.1, Eq. (4)"},{"comment":"Equation (4) is mathematically unclear: it writes pi_theta_critic(delta P | P) = E[pi_theta_critic(delta P | P, s, a)] without specifying the distribution over which the expectation is taken. Please provide a precise definition or remove the equation.","section":"Table 1"},{"comment":"In Table 1, ScienceQA is marked as '(mm-only)' for some models but not for others; the text and benchmark description in Section 3.1 do not explain this difference. Clarify which splits are used for each model.","section":"References"},{"comment":"Reference [53] is written as 'X. Grok-1.5 vision preview'; this is not a proper citation for RealWorldQA. The dataset is from xAI and should be cited with its official title and authors.","section":"Appendix 9"},{"comment":"The DPO hyperparameter section states 'preference loss is set to sigmoid' and 'preference parameter beta of 1.0'; it is helpful to state explicitly that this is the standard DPO loss from Eq. (9) and whether the reference model is the frozen base Qwen2-VL-7B.","section":"Section 2.3 and Algorithm 3"},{"comment":"Algorithm 3's stopping condition 'If Critic determines that critique is satisfactory' is not operationalized. It is unclear whether the Critic is given a binary satisfaction task or whether the loop always runs to 'max iterations'; please specify the stopping decision used in the experiments.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The two load-bearing issues are the undisclosed training data sources and the mismatch between the RL/TextGrad theoretical framing and the implemented algorithm. Both are addressable in revision: the authors can name the VQA datasets and provide a decontamination/overlap report, and they can either implement the claimed TextGrad mechanism or reframe the method as a fixed iterative prompt-refinement loop. The reviewer also recommends that the editor ask for variance estimates or significance tests for the smaller numerical gains, since the abstract's 'significantly outperforms' claim is stronger than the presented evidence supports. The paper is likely to be a useful contribution if these empirical rigor points are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a plausible but unverified claim that a DPO-trained external critic improves frozen VLMs. The headline gains on math benchmarks are large, but the missing training-data provenance leaves a leakage channel open, and I would not trust the transferability claim until that is closed.\n\nWhat is actually new: the authors build a 29k critique dataset by injecting synthetic errors into VQA answers with GPT-4o, having several VLMs critique the fake answers, scoring those critiques with a Jaccard-plus-GPT score, and then DPO-training a Qwen2-VL-7B critic. That is a concrete, reproducible contribution, and the framework itself is genuinely plug-and-play: a frozen reasoner plus a separately trained critic. The ablations against Self-Refine and prompt-only are useful, and they include case studies showing the critic catching a factual error (Oregon capital) and a visual one. Code and dataset links are provided, which I take as good faith.\n\nWhere it is soft: the training data sources are never named. Section 2.2 says \"question-image pairs from VQA datasets\" and Figure 3 says \"Several VQA Datasets,\" but no list of datasets appears anywhere. Table 7 reports only token statistics. Since the evaluation suite is itself VQA-style (MathVista, RealWorldQA, SEEDBench, etc.), this is a live leakage path, and the stress-test note is right to call it the load-bearing weakness. The authors need a decontamination analysis and a full list of source datasets and splits. Second, the numbers have no variance estimates. The text says temperature 0.01, which helps determinism, but it is still a single run. Some improvements are small (SEED +1.8, MMT +1.6) and could easily be noise. Third, the RL/TextGrad equations in Section 2 are decorative; Algorithm 3 is simple iterative refinement, and the paper does not actually use those gradients. That is not a fatal flaw, but the theory adds little. Finally, the RBR scoring includes a GPT-4o score, and the fake answers also come from GPT-4o, so the critic is being trained to align with GPT-4o's judgment. That is a reasonable methodological choice, but it should be stated as such rather than implied to be model-agnostic.\n\nIf the leakage concern is resolved and the gains survive a decontamination check, this becomes a solid inference-time method. The reader's conditional verdict is about right; I am a bit more skeptical on the math, but not enough to recommend rejection. This paper deserves a serious referee. I would send it to review and ask for dataset transparency, overlap analysis, and ideally multiple seeds as a first revision.\n\nFor a reading group, it is a good example of a practical critic-based inference method, but I would pair it with a discussion of leakage in VQA-style benchmarks.","headline":"A likely-useful external-critic method for VLMs, but the unnamed training sources and missing decontamination leave the headline gains unverified.","tokens_in":21283,"tokens_out":2152,"would_cite":true,"duration_ms":22671,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An external critic model trained on synthetic corrupted answers can catch and fix VLM reasoning errors, pushing Qwen2-VL-7B past GPT-4V on five of eight multimodal benchmarks.","keywords":["multimodal reasoning","vision-language models","critic model","direct preference optimization","self-correction","hallucination detection","rule-based reward","VisualQA benchmark"],"falsifier":"A reader could settle the claim by taking the released critique-VQA dataset, checking its questions against the questions in MathVista, MathVerse, MMBench, SEEDBench, MMStar, MMT-Bench, ScienceQA, and RealWorldQA, and re-running the evaluation only on the non-overlapping subset; if the gains shrink to near the Self-Refine baseline, the conclusion that the trained critic generalizes would not hold.","tokens_in":20193,"feed_emoji":"🧠","tokens_out":4582,"duration_ms":39402,"temperature":0.7,"pith_summary":"This paper tries to establish that a separate, trained critic model can catch and correct errors made by a vision-language model, improving multimodal reasoning more than letting the model critique itself. It does this by generating 29,012 examples where a strong model inserts fake details into true answers, having several VLMs write critiques of the tampered answers, ranking those critiques, and training a critic with direct preference optimization. At inference, the critic's natural-language feedback is appended to the reasoner's prompt in a loop. The authors report that a 7B model with this critic beats the proprietary GPT-4V on five of eight benchmarks, with the largest gains on visual math tasks.","feed_headline":"A second VLM critic lifts reasoning scores past GPT-4V on 5 of 8 tests","feed_subtitle":"Qwen2-VL-7B plus a DPO-trained critic gains 11.8 points on MathVista by refining its own answers in a loop.","key_machinery":"The load-bearing mechanism is the Reasoner-Critic loop, framed as in-context reinforcement learning in which the critique itself acts as the gradient signal. The training data comes from the Vision Error Insertion Technique (VEST): GPT-4o adds one to five false statements to a correct VQA answer, three different VLMs write critiques identifying the false statements, and a rule-based reward scores each critique by $J(G,C)=|G\\cap C|/|G\\cup C|$ plus a GPT-based regularization term. The Critic is then trained with DPO to prefer high-scoring critiques, and at inference the Reasoner's prompt is updated by concatenating the Critic's feedback, so the text prompt itself plays the role of a policy that evolves.","core_discovery":"The claim is that VLM reasoning errors can be reduced by decoupling reasoning from evaluation: an independent Critic, trained to spot injected errors, gives natural-language feedback that the Reasoner folds into its text prompt, iterating until the Critic is satisfied. The Critic is a Qwen2-VL-7B fine-tuned with DPO on a dataset of 29,012 critique pairs, where the preferred critique is selected by a rule-based reward that combines a Jaccard overlap between injected and detected errors with a GPT-4o quality score. When the loop is applied to Qwen2-VL-7B and DeepSeek-VL-7B, the paper reports gains on 23 of 24 benchmark/model comparisons and top scores on five of eight benchmarks, including +11.8 on MathVista and +7.1 on MathVerse for Qwen2-VL-7B.","pith_inferences":["Beyond the paper: since the critic is trained only on injected textual errors, its benefit may concentrate on answer-level verification rather than deeper plan-level reasoning; a test that asks the critic to flag flawed intermediate reasoning steps, not just false facts, would separate the two.","Beyond the paper: the synthetic-corruption and preference-ranking recipe could be applied to train critics for video, audio, or embodied action sequences, where ground-truth answers are harder to obtain but injected errors are easy to define.","Beyond the paper: if the unnamed VQA sources behind the 29,012 training pairs are disclosed, a direct overlap check against the eight evaluation benchmarks would reveal whether part of the reported gain is memorization rather than generalized criticism."],"forward_implications":["A single DPO-trained critic can be bolted onto at least three different 7B reasoners, Qwen2-VL-7B, DeepSeek-VL-7B, and LLaVA-v1.5-7B, and improve accuracy across most benchmarks without retraining the reasoner.","The largest gains occur on math-heavy benchmarks, suggesting the loop is most valuable when reasoning errors, rather than pure perception errors, dominate the failure cases.","The critic adds only a few dozen extra tokens per question, so its accuracy benefit comes with modest additional inference cost.","DPO training of the critic is the key ingredient: a Self-Refine baseline without the trained critic improves far less and even hurts on one benchmark.","Because the framework formalizes critique as a text-prompt policy update, the same loop can inject constraints or knowledge into reasoning, not just corrections of factual errors."],"supporting_citations":[{"why":"Supplies the DPO loss used to train the Critic to prefer high-scoring critiques over low-scoring ones.","marker":"[42]"},{"why":"Provides the critic-model inspiration and motivates the design against nit-picky, overly long critiques.","marker":"[35]"},{"why":"GPT-4o is the model that inserts fake details into true answers to create the VEST training signal.","marker":"[39]"},{"why":"Qwen2-VL-7B is both the base Reasoner and the base model fine-tuned with DPO to become the Critic.","marker":"[49]"},{"why":"TextGrad supplies the text-based gradient update that connects the Critic's natural-language feedback to Reasoner prompt evolution.","marker":"[59]"},{"why":"Self-Refine is the main baseline used in ablations to show that a separately trained critic outperforms self-feedback.","marker":"[34]"},{"why":"GLM-4V-9B is one of the three VLMs that generate candidate critiques for the preference dataset.","marker":"[12]"},{"why":"GPT-4o mini is another VLM used to generate candidate critiques for the preference dataset.","marker":"[40]"},{"why":"MiniCPM-V is the third VLM used to generate candidate critiques for the preference dataset.","marker":"[57]"}],"fun_headline_variants":["VLM critic loop lifts reasoning past GPT-4V on 5 of 8 tests","Critic-V: decoupled critic boosts VLM reasoning, tops GPT-4V","Actor-critic VLM: reasoner plus critic outperforms GPT-4V","VLM critic and reasoner loop gains 11.8 on MathVista"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training pairs come from question-image sets that are not named, so the reported gains assume none of those questions overlap with the eight evaluation benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["VLM critic loop lifts reasoning past GPT-4V on 5 of 8 tests","Critic-V: decoupled critic boosts VLM reasoning, tops GPT-4V","Actor-critic VLM: reasoner plus critic outperforms GPT-4V","VLM critic and reasoner loop gains 11.8 on MathVista"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000486,"raw_usage":{"total_tokens":2445,"prompt_tokens":1045,"completion_tokens":1400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":1310}},"tokens_in":661,"tokens_out":1400,"duration_ms":9866,"temperature":1.0,"reasoning_tokens":1310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:24:32.823298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the claim by taking the released critique-VQA dataset, checking its questions against the questions in MathVista, MathVerse, MMBench, SEEDBench, MMStar, MMT-Bench, ScienceQA, and RealWorldQA, and re-running the evaluation only on the non-overlapping subset; if the gains shrink to near the Self-Refine baseline, the conclusion that the trained critic generalizes would not hold.","supporting_citations":[{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Supplies the DPO loss used to train the Critic to prefer high-scoring critiques over low-scoring ones."},{"cited_title":"Hello GPT-4o","cited_arxiv_id":null,"evidence_quote":"GPT-4o is the model that inserts fake details into true answers to create the VEST training signal."},{"cited_title":"Self-refine: It- erative refinement with self-feedback","cited_arxiv_id":null,"evidence_quote":"Self-Refine is the main baseline used in ablations to show that a separately trained critic outperforms self-feedback."},{"cited_title":"Chatglm: A family of large language mod- els from glm-130b to glm-4 all tools, 2024","cited_arxiv_id":null,"evidence_quote":"GLM-4V-9B is one of the three VLMs that generate candidate critiques for the preference dataset."}],"review_version":1}