{"id":"8cff44d2-86cd-4223-8b47-4a4d0fa4d567","arxiv_id":"2607.21722","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A consistency-focused benchmark and GRPO-based training that rewards agreeing answers to logically equivalent visual questions improve LVLM reasoning and generalization.","lead":"This paper introduces a new benchmark and a training method to make vision-language models answer logically equivalent questions about the same image consistently. It reports that the trained model beats larger open and closed models on that benchmark and also improves on an unrelated visual-search benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Consistency reward as implemented is correctness-parity, not logical agreement; pseudo-label error unmeasured, so the label-free consistency claim is unsupported.","rationale":"The reader's verdict, REJECT, is well-founded. The key issue is that the consistency reward and evaluation metric are not what the abstract and Eq. (8) claim. The implementation in Appendix E is correctness parity, requiring pseudo-labels, so the core mechanism is mis-specified. The paper acknowledges pseudo-answers are 'often noisy' (Sec 3.2) but never measures label error rate. Without this measurement, the parity reward could reward consistent wrong answers, and the claimed generalization to V*Bench could arise from GRPO's exploration or the accuracy reward alone rather than a consistency signal. Moreover, the 'without strict answer supervision' claim in the abstract is contradicted because the consistency function needs reference answers. The proposed concrete test—a label-free consistency arm and human pseudo-label audit—would settle whether the framework's success depends on labelled supervision. If the label-free arm fails, the paper's title and abstract overstate the contribution; if it succeeds, the contradiction remains only at the definitional level. Thus the current draft does not support its strongest claims.","tokens_in":26928,"tokens_out":4624,"duration_ms":44831,"concrete_test":"On a fixed 500-image training subset, train three GRPO variants: (1) current r = r_acc + 0.5·1[both pseudo-correct or both pseudo-incorrect]; (2) label-free r = 1[o1 ≡ o2] using exact-match or an entailment model, with no correctness check; (3) r = r_acc only. Also have human annotators verify the GPT-4.1 pseudo-answers on a random 200-sample subset. If (2) does not match (1) on ConVBench consistency/accuracy, or if pseudo-label error > ~20%, the paper's claim that consistency can be learned without answer supervision fails, and the reported gains are likely dependent on noisy labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that consistency training without strict answer supervision improves robust visual reasoning. This is undermined by the implementation: Eq. (8) defines r_con = c(o1_i, o2_i) with 'logically consistent' semantics, but Appendix E specifies c(o1,o2)=1 iff both answers are correct or both incorrect against GPT-4.1 pseudo-labels. Thus the reward is a correctness-parity signal, not an agreement signal: a pair of wrong-but-identical responses receives reward γ, and a pair with one correct and one incorrect is penalized even if semantically aligned. The same parity definition is used for evaluation metrics CI/AI (Sec 4.1), so the benchmark's 'logical consistency' actually measures neither logical equivalence nor answer agreement, but co-correctness under noisy labels. The abstract's 'without strict answer supervision' is contradicted: computing c requires reference answers, so training relies on pseudo-labels whose error rate is never measured. If pseudo-labels are noisy, the consistency term can reinforce systematic errors (both wrong → positive reward), and the reported gains on ConVBench/V*Bench may largely reflect the accuracy reward or other GRPO effects rather than consistency. This is the load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConVBench, a vision-centric benchmark of 686 images with paired logically equivalent questions across six reasoning categories, and two metrics: logical consistency (CI) and robust accuracy (AI). It also presents ConVLM, a GRPO-based reinforcement learning method that trains an LVLM with a dual reward combining an accuracy reward and a consistency reward over automatically generated question-answer pairs. The authors report state-of-the-art open-source results on ConVBench (73.36% consistency and 66.83% robust accuracy for the 7B model), strong generalization to V*Bench (84.90%), and ablations showing contributions from both reward terms. The central technical claim is that a consistency reward improves robust visual reasoning without strict answer supervision. However, the paper contains an internal contradiction between the formal definition of the consistency function and its implemented form: Eq. (8) defines consistency as logical agreement between paired responses, while Appendix E implements it as equality of correctness status against GPT-4.1 pseudo-labels. This discrepancy affects both the training reward and the evaluation metric and is load-bearing for the paper's main claims.","tokens_in":27207,"tokens_out":5799,"duration_ms":60288,"significance":"If the consistency mechanism were implemented as described, the paper would make a useful contribution: a human-validated paired-question benchmark, a scalable generation pipeline, and an RL objective that encourages cross-question agreement. The human verification statistics (Appendix O), the independent V*Bench evaluation, and the careful ablation structure are strengths. However, as it stands, the central claim is not supported. The implemented reward is co-correctness under pseudo-labels, not logical agreement; the pseudo-label error rate is never measured; and the primary evaluation metric CI has the same functional form as the training reward. These issues are not presentation-level but concern the core definition of consistency, so the reported gains on ConVBench cannot be interpreted as evidence of improved logical consistency.","major_comments":[{"comment":"Eq. (8) defines r_con = c(o1_i, o2_i) and the text states that c(·) 'returns 1 if the two outputs are logically consistent'. Appendix E specifies the implemented c(·) as: 'If both generated answers are correct or incorrectly, it is 1; otherwise, it is 0', judged against GPT-4.1 pseudo-answers. These are different functions. The implemented reward is XNOR of correctness statuses, not agreement between response contents. A wrong-wrong pair receives the consistency reward even when the answers are logically contradictory, while a correct-incorrect pair receives 0 even when semantically aligned. This also contradicts the abstract's claim of functioning 'without strict answer supervision': computing c requires reference answers, so the learning signal is supervised by pseudo-labels.","section":"§3.2, Eq. (8) vs. Appendix E"},{"comment":"The benchmark metric CI is defined as '1 if both questions are answered either correctly or incorrectly, and 0 otherwise' — the same XNOR-of-correctness function used as the training reward c. The primary reported consistency gains are therefore gains on a training objective with the same functional form, not on an independently defined logical-consistency measure. Human-validated ground truth and the V*Bench generalization result partially mitigate circularity, but the headline consistency numbers on ConVBench cannot be interpreted as measuring logical agreement. Table 1 illustrates the problem: DeepSeek-VL2-tiny scores 63.63 CI on Causal & Intent with 0.56 accuracy, showing that a model that is consistently wrong can receive high consistency. A metric that awards both-wrong responses is not a logical-consistency metric.","section":"§4.1, CI metric and Table 1"},{"comment":"The method's noise-tolerance argument rests on the assumption that GPT-4.1 pseudo-answers are accurate enough for the parity signal to be meaningful. The introduction states that pseudo-answers are 'often noisy and frequently incorrect', but the paper provides no error-rate estimate, no comparison of pseudo-label accuracy against human labels, and no analysis of how the reward behaves under label noise. If the pseudo-label error rate is high, the γ·r_con term systematically rewards pairs that are both wrong, and r_acc is also computed against the same noisy references. This omission is load-bearing because both reward terms depend on unverified labels; without it, the weakly supervised learning claim is unsupported.","section":"§3.2 and Appendix E: pseudo-label quality"},{"comment":"The distinction between 'weakly supervised' and 'without strict answer supervision' is not established. The w/o-Acc ablation still uses the consistency reward, which itself requires pseudo-reference answers; therefore no experiment trains without some form of answer supervision. As implemented, both rewards require a reference answer (pseudo-label) to compute, so the claim in the abstract that the framework 'functions effectively with or without strict answer supervision' is not demonstrated by any experiment in the paper.","section":"§3.2, 'without strict answer supervision' claim"}],"minor_comments":[{"comment":"The pseudocode refers to 'Eq. (eq:advantages)' but the label is not defined in the text; it should refer to Eq. (5).","section":"Algorithm 1"},{"comment":"The symbol r_i(θ) is used for the probability ratio π_θ(o_i|I,q)/π_{θ_old}(o_i|I,q), but r was previously used for reward. Using the same symbol for two different quantities is confusing; a distinct symbol such as ρ_i(θ) would be clearer.","section":"Eq. (6)"},{"comment":"Row names 'w/-GroupCon' and 'w/-SampleCon' are awkward and inconsistent with the rest of the paper; also the repeated phrase 'percentage points' in the paragraph should be cleaned.","section":"Table 3"},{"comment":"GPT-4.1 is cited as 'Brown, 2020', which is the GPT-3 paper. A correct citation for GPT-4.1 should be provided.","section":"References"},{"comment":"There are typos: 'GoundTruth' and 'GroundTruth' appear inconsistently; the figure caption should be proofread.","section":"Appendix I / Figure 6"}],"recommendation":"reject","confidential_remarks":"The paper contains a potentially useful benchmark and some positive independent generalization results, but the internal contradiction between the consistency reward's formal definition and its implementation affects the central claim. The evaluation metric shares the same functional form as the training reward, and pseudo-label quality is not measured. These are not local presentation issues; addressing them would require redefining the consistency signal and re-running the experiments. I would be open to a resubmission that uses a genuine semantic-agreement function and provides a pseudo-label error analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, ConVBench is a genuinely useful resource: 686 human-validated pairs of logically equivalent visual questions across six reasoning categories, with a clean separation from the training data. Second, the paper's central method claim does not match its own implementation. The consistency reward is described as encouraging agreement between paired responses, but Appendix E implements c(·) as correctness parity against GPT-4.1 pseudo-labels — both correct or both incorrect earns a reward. The evaluation metric CI is the same parity function. So what is being optimized and measured as 'consistency' is co-correctness under noisy labels, not logical agreement.\n\nThe benchmark construction is the real contribution. The human verification stats (57% accepted, 22% corrected, 21% discarded) are transparent, and the six categories cover a reasonable slice of vision-centric reasoning. The V*Bench gains (80.21% and 84.90% for the 3B and 7B ConVLM) are real and interesting, and the ablation shows the consistency reward adds a few points beyond the accuracy reward. Something in the RL recipe transfers, even if the mechanism is not what the abstract claims.\n\nThe soft spots are not minor. The contradiction between Eq. (8) and Appendix E is load-bearing: the claim of working 'without strict answer supervision' is false as implemented, because computing c requires reference answers. If GPT-4.1 pseudo-labels are noisy, the reward can systematically reinforce wrong-but-agreeing outputs. The paper reports no pseudo-label error rate. Also, the training reward and the evaluation metric are the same XNOR-of-correctness function, so the headline metric is partly circular; the independent V*Bench results are what keep the empirical claim alive. The benchmark itself is small (686 images / 1,372 questions), and the reported numbers have no error bars, so the 1–2 point ablations should be treated cautiously.\n\nWho this is for: people building or evaluating LVLMs on vision-centric reasoning, and anyone designing RL rewards that combine correctness and agreement. The benchmark deserves serious referee time; the method section does not, in its current form. If the authors re-implement the consistency reward as true response agreement, or honestly measure pseudo-label accuracy and report the parity-based version as a weak-supervision baseline, the paper could be solid. As written, I'd send it back for major revision, not desk reject.","headline":"Useful benchmark, but the label-free consistency claim collapses once you read Appendix E: the 'consistency' reward is correctness parity against pseudo-labels, not answer agreement.","tokens_in":27696,"tokens_out":4478,"would_cite":true,"duration_ms":40396,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConVLM claims that training a vision-language model to give consistent answers to logically equivalent questions improves visual reasoning, and backs it with a new benchmark, ConVBench.","keywords":["vision-language models","visual reasoning","logical consistency","reinforcement learning","GRPO","benchmark","automatically generated QA pairs","robustness"],"falsifier":"Take a random sample of the automatically generated training question pairs, have humans re-label the answers, and compare the model's ConVBench robust accuracy when trained on original versus corrected labels. If the pseudo-label error rate is high and the gains shrink or reverse under corrected labels, the central claim that consistency itself drives improvement would be weakened.","tokens_in":26816,"feed_emoji":"🧠","tokens_out":3741,"duration_ms":37740,"temperature":0.7,"pith_summary":"The paper argues that large vision-language models answer paraphrased questions about the same image inconsistently, and that this is a trainable weakness rather than a fixed limitation. It introduces ConVBench, a benchmark of 686 images each paired with two logically equivalent questions, along with metrics for logical consistency and robust accuracy. To improve models, it proposes ConVLM, which uses GRPO reinforcement learning with a dual reward: one term for answer accuracy and one for consistency between paired answers. On ConVBench, the 7B model reaches 73.36% consistency and 66.83% robust accuracy, surpassing the closed-source baselines reported in the paper, and it generalizes to V*Bench with 84.90% accuracy. The central claim is that consistency is a meaningful and exploitable dimension of robust visual reasoning, and that automatically generated question pairs provide enough supervision to train it.","feed_headline":"Consistency training lifts visual reasoning by 25 points","feed_subtitle":"Rewarding agreement across equivalent questions pushes a 7B model past closed-source baselines on ConVBench.","key_machinery":"The central object is the logically equivalent question pair: for each image, a pair (q1, a1), (q2, a2) is generated so that the answer to one can be inferred from the answer to the other together with the image. The training objective uses GRPO, a group-relative policy optimization method, with a dual reward r = racc + γ·rcon. The consistency reward rcon compares model outputs on the paired questions, with the implemented rule being 1 if both answers are correct or both are incorrect according to the auto-generated pseudo-labels, and 0 otherwise. This couples two rollouts from the same image and encourages the policy to treat logically equivalent prompts as carrying the same answer, while t","core_discovery":"The central claim is that logical consistency across equivalent questions is a distinct, trainable dimension of visual reasoning. The paper shows that baseline LVLMs often give contradictory answers to equivalent questions about the same image, and that current benchmarks do not measure this failure. To fix this, ConVLM optimizes a reward that combines accuracy on individual answers with agreement between answers to paired logically equivalent questions. The authors report that this consistency-oriented training improves both consistency and robust accuracy on ConVBench, and that the gains transfer to other visual reasoning benchmarks. They further claim that the consistency reward alone, wi","pith_inferences":["Extension: because the implemented consistency rule treats 'both wrong' as consistent, the method's effectiveness likely depends on the accuracy of the auto-generated pseudo-labels; if those labels are often wrong, the model could be rewarded for being confidently wrong.","Extension: the paper does not estimate the pseudo-label error rate on training data, so a concrete next step is to measure that error rate and test whether model gains survive label correction.","Extension: replacing the binary both-correct-or-both-wrong rule with a softer semantic-agreement score, which the paper lists as future work, would separate 'consistency as agreement' from 'consistency as agreement with the generator'.","Extension: applying the same consistency reward to video or text-only reasoning tasks would test whether the mechanism is general or specific to single-image visual QA."],"forward_implications":["If the claim holds, consistency can be used as a training signal without human-annotated ground truth, because the question pairs are automatically generated from image captions and object boxes.","Training with the consistency reward alone improves both consistency and accuracy, implying that logical agreement itself nudges the model toward better visual understanding.","Baselines that score high on consistency but near-zero on accuracy show that single-question benchmarks can miss systematic failures; ConVBench-style paired evaluation would make these visible.","The reported generalization to V*Bench (84.90% for ConVLM-7B) suggests the consistency training does not merely overfit to the benchmark's question format.","Scaling the automatically generated training data from 1,000 to 5,000 images gives consistent gains, with diminishing returns, indicating a cheap path to improving visual reasoning robustness."],"fun_headline_variants":["Consistency reward boosts LVLM reasoning on equivalent questions","Training for logical consistency lifts visual reasoning in LVLMs","Agreement-based RL improves visual reasoning consistency","ConVBench: measuring and fixing inconsistent visual reasoning","Rewarding answer agreement sharpens LVLM visual logic"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The training reward's consistency term relies on auto-generated pseudo-answers: an output pair counts as consistent only when both answers match the proposer's labels, so if those labels are frequently wrong the model can be rewarded for being consistently wrong, and the paper provides no estimate of how often they are wrong.","fun_headline_variants_meta":{"raw":{"variants":["Consistency reward boosts LVLM reasoning on equivalent questions","Training for logical consistency lifts visual reasoning in LVLMs","Agreement-based RL improves visual reasoning consistency","ConVBench: measuring and fixing inconsistent visual reasoning","Rewarding answer agreement sharpens LVLM visual logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000472,"raw_usage":{"total_tokens":2153,"prompt_tokens":685,"completion_tokens":1468,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":1403}},"tokens_in":429,"tokens_out":1468,"duration_ms":10438,"temperature":1.0,"reasoning_tokens":1403,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:53:13.613114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the automatically generated training question pairs, have humans re-label the answers, and compare the model's ConVBench robust accuracy when trained on original versus corrected labels. If the pseudo-label error rate is high and the gains shrink or reverse under corrected labels, the central claim that consistency itself drives improvement would be weakened.","supporting_citations":[],"review_version":1}