{"id":"77948841-f775-4168-9e2b-b101efafb351","arxiv_id":"2412.07801","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new GPT-4-generated dataset of distractors and corrective feedback for visual commonsense reasoning, plus a compact LMM (PEIFG) that produces explainable corrections and beats existing baselines in automatic and human evaluation.","lead":"The paper introduces a new benchmark and a fine-tuned multimodal model for generating explanatory feedback that corrects wrong answer choices in visual commonsense questions. It evaluates whether large multimodal models can identify why a distractor is wrong and explain the error, much as a teacher does.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on a closed GPT-4 loop: GPT-4 generates the distractors, feedback, DPO preferences, and evaluation references, while human checks are thin; so the benchmark's validity and PEIFG's superiority are jointly unverified.","rationale":"The paper's stated contribution has two parts: a new benchmark VCR-DF and a model PEIFG that significantly surpasses existing LMMs. Both parts ultimately depend on the quality and independence of the benchmark's ground truth. Section 3.1 shows that ground truth comes from GPT-4, with a light manual filter; Section 4.3.1 shows that even the fine-tuning preferences are GPT-4 self-assessments; Section 5.3 evaluates against GPT-4-written references with overlap metrics. This triple use of GPT-4 creates a correlation between the training objective and the test metric that can inflate PEIFG's scores without implying better error correction. The human evaluation is too small and too thinly reported to break the loop: 5 raters, 200 items, no agreement statistic, and the table actually shows GPT-4V beating PEIFG on two of five criteria. I therefore regard the closed GPT-4 loop as the load-bearing concern; if a human-authored validation set does not preserve PEIFG's advantage, the central empirical claim fails. A secondary reproducibility concern is Eq. 5: the argmin over cosine similarities contradicts the text's 'top-K' closest selection and Eq. 6's maximization objective; either the equation is a typo or the implemented selection differs from the paper. This does not change my verdict, but it should be corrected. The reader's weakest assumption identifies the same GPT-4 circularity, so my read is aligned; no verdict change is needed beyond the CONDITIONAL already assigned.","tokens_in":20026,"tokens_out":4937,"duration_ms":48275,"concrete_test":"Build a 200-item held-out set in which distractors and reference feedback are written by independent human annotators who see the images (not by GPT-4), then run PEIFG, CogAgent, and GPT-4V on it. Score outputs with human preference judgments and with the same automatic metrics against the human references, and report inter-annotator agreement (e.g., Krippendorff's alpha). If PEIFG's margin narrows or reverses, or if automatic metrics disagree with human preferences, the reported superiority is an artifact of the GPT-4 closed loop.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 constructs VCR-DF using language-only GPT-4: given textual event, place, object boxes, question, and answer, GPT-4 writes the distractors (Table 9) and then writes the misconception and explanation feedback (Table 10). Section 4.3.1 uses GPT-4's self-scored answers to five diagnostic questions as the DPO preference signal. Automatic metrics in Table 1 are n-gram and BERTScore overlap with those same GPT-4 references. Consequently the training target, the reward signal, and the evaluation reference all come from the same model family, and PEIFG is explicitly trained to imitate that text style. A stronger general LMM can produce feedback that is more useful but stylistically different, and it will still score lower on BLEU/CIDEr. The human evaluation (B.5) is limited to 200 samples and 5 raters, reports no inter-annotator agreement or significance tests, and in fact shows GPT-4V at or above PEIFG on Fluency (1.78 vs 1.74) and Relevance (0.93 vs 0.88). Because GPT-4 is language-only, any image detail absent from the human-written event/place annotations is invisible to the benchmark generator, so VCR-DF may not require true visual grounding. The paper does not report rater disagreement or annotator counts for the 'over 90% pass' filter in Section 3.1.2, and the dataset itself is not released, so the claimed new benchmark and the 'significantly outperforms' result cannot be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VCR-DF, a benchmark for explainable feedback generation on visual commonsense reasoning distractors, and proposes PEIFG, a 1.8B-parameter LMM with a visual marker perceiver, CLIP features, a learnable expert prompt selector, and a DPO refinement stage. The dataset is constructed by using language-only GPT-4 to generate distractors and feedback, followed by manual filtering. Experiments compare PEIFG against several open LMM baselines and GPT-4V using automatic metrics and a small human evaluation, reporting that PEIFG outperforms existing LMMs on both feedback and distractor generation.","tokens_in":20384,"tokens_out":5610,"duration_ms":55372,"significance":"If the benchmark and evaluation are valid, this is a useful new task: error-correction feedback for visual commonsense reasoning is underexplored, and the proposed PEIFG architecture with expert prompt selection and vision-grounded features is clearly described and thoughtfully ablated. The authors release code, and the ablations indicate that the visual branches, the expert prompt selector, and the refinement step all contribute. However, the main evidential value of the paper depends on breaking out of the GPT-4 loop: the training data, the DPO reward, and the automatic references all come from GPT-4, while the human evaluation is small and lacks agreement statistics. The claim of visual grounding is also not yet established because the benchmark generator receives only textual annotations and never the image. With stronger independent evaluation and statistical rigor, the contribution would be solid.","major_comments":[{"comment":"The evaluation is closed around GPT-4: GPT-4 writes the VCR-DF distractors and feedback (§3.1.1–3.1.2), GPT-4 scores the five diagnostic questions used for DPO refinement (§4.3.1 and Table 11), and the automatic metrics in Tables 1–2 are computed against those same GPT-4-written references (§5.3.1). Because PEIFG is trained to imitate this exact text style, high BLEU/CIDEr/BERTScore can indicate stylistic imitation rather than better error correction, and a stronger general LMM that produces useful but differently worded feedback would be penalized. The human evaluation in B.5 is the only independent check, but it covers only 200 samples with 5 raters, reports no inter-annotator agreement or significance tests, and Table 7 shows GPT-4V at or above PEIFG on Fluency (1.78 vs. 1.74) and Relevance (0.93 vs. 0.88). Please provide a larger independent human evaluation with agreement statistics and significance tests, and an analysis that separates imitation quality from correction quality (for example, by scoring whether the feedback identifies the actual visual contradiction rather than matching GPT-4 wording).","section":"§3.1, §4.3.1, §5.3"},{"comment":"The benchmark generator is language-only: the GPT-4 prompt in Table 9 provides Event, Object boxes, Place, Question, Answer, and Educational Level, but not the image itself. Consequently VCR-DF can only contain misconceptions that are detectable from the textual annotations, and any image detail not captured in the event/place text is invisible to the benchmark. This undermines the claim that VCR-DF measures visual commonsense reasoning and raises the question whether PEIFG's visual branches (VMP and CLIP, §4.1) are actually necessary; a text-only ablation or a sanity check that feeds a mismatched image should be reported to show that the generated feedback depends on visual content beyond the provided annotations.","section":"§3.1, Table 9"},{"comment":"No standard deviations, confidence intervals, or significance tests are reported for any automatic metric. Several differences are small relative to the likely sampling noise (e.g., BLEU-1 46.53 vs. 46.47 for CogAgent in Table 1; BLEU-4 17.69 vs. 17.32 for K=1 vs. K=3 in Table 4), so the phrase 'significantly outperforms' in the abstract and §6 is not supported by the reported evidence. Please report variances across runs or seeds and perform paired significance tests for the main comparisons, including the GPT-4V comparison in Table 2.","section":"Tables 1–2, §5.4"},{"comment":"The manual quality checks are not reported with enough detail to validate the ground truth: the paper states that 'over 90% of the preliminary samples' pass the Accuracy and Clarity criteria (§3.1.2), but gives no annotator count, no operational definition of the criteria, and no inter-annotator agreement; the same is true for the 200-sample human evaluation in B.5, which also lacks significance tests. Without this information, neither the claimed benchmark quality nor the human-evaluation advantage over baselines can be independently assessed.","section":"§3.1.2, B.5"}],"minor_comments":[{"comment":"There are several typos: 'distracors' in §3, 'percevier' in §5.2.2, 'assisted bv GPT-4' in §5.4.2, and 'Q-Fromer' in §4.2.1; these should be corrected to 'distractors', 'perceiver', 'assisted by GPT-4', and 'Q-Former'.","section":"§3, §5.4, §5.2.2"},{"comment":"In Table 10, the third generated distractor is labeled 'Distractor1' instead of 'Distractor3', which makes the prompt example confusing.","section":"Table 10"},{"comment":"The model name is written inconsistently as 'QWen1.5' and 'Qwen1.5'; the standard spelling is 'Qwen1.5'.","section":"§4.3, §5.1"},{"comment":"The claim that VisualGLM and LLaVA-v1.5 predict the 'understand' level for almost all samples would be easier to verify if the per-level confusion matrix or per-level accuracy were reported.","section":"§5.4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the camera-ready version of an ACM MM 2024 paper. If this is a journal submission, the incremental contribution over the published version should be clarified. Also, the VCR-DF dataset itself is not released, which is a notable limitation for a benchmark paper; I did not put this in the major comments because it is not strictly an internal technical flaw, but the editor may wish to ask the authors about availability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: they construct VCR-DF, a benchmark for a genuinely new task—explainable feedback on visual commonsense reasoning distractors. The original VCR distractors do have an entity-overlap bias, and their point that those distractors are easy to dismiss is correct. The new distractors and the misconception/explanation feedback make sense as an evaluation axis for LMMs, especially for tutoring-style applications. The model, PEIFG, is a reasonable engineering effort: SAM-based region features with marker perceiving, CLIP global features, Q-Former instruction-aware features, a learnable prompt pool with top-K selection, and a DPO refinement step. The ablations show each component contributes, and the gains over baselines in Table 1 are consistent. That is real work.\n\nThe soft spots are mostly about the evaluation loop. GPT-4 writes the distractors, writes the feedback, scores the DPO preferences, and supplies the references for the automatic metrics. The model is therefore trained to imitate the same teacher that defines the target. A stronger model that gives better but stylistically different feedback would score lower on BLEU/CIDEr. The stress-test note is right about this. The human evaluation is thin: 200 samples, 5 raters, no inter-annotator agreement, and it shows GPT-4V beating PEIFG on fluency and relevance, with PEIFG ahead on helpfulness and logical consistency. That does not support a blanket 'significantly outperforms' claim. The paper also doesn't release the dataset, so the benchmark can't be independently scrutinized.\n\nI don't think this is fatal. The authors are transparent about using GPT-4; this is not a case of hiding a circularity. The benchmark is still likely useful as a stress test for error-correction behavior, and the manual filtering, though lightly reported, is more than many GPT-4-based datasets do. But the claims should be scaled to match the evidence: 'outperforms on automatic metrics and on some human-judged dimensions' is accurate; 'pioneering' and 'significantly outperforms' are overclaiming.\n\nWho should read it: people working on LMM evaluation, visual commonsense, or AI feedback generation. It's also a good reading-group case study for the pitfalls of LLM-generated benchmarks. I'd send it to review with requests: release the data, add significance tests, report annotator agreement, and ideally validate on a small set of human-authored distractors to break the closed loop.","headline":"A solid new benchmark and model for explaining VCR distractors, but the closed GPT-4 loop means the central 'significantly outperforms' claim is only as strong as the teacher.","tokens_in":20907,"tokens_out":2911,"would_cite":false,"duration_ms":29267,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a 1.8B multimodal model can be trained to identify visual commonsense errors in distractor options and generate explanations that help correct them, and that a new GPT-4-built benchmark, VCR-DF, exposes a gap in existing…","keywords":["Large Multimodal Model","Visual Commonsense Reasoning","Error Correction","Feedback Generation","Distractor","Benchmark","Direct Preference Optimization","GPT-4"],"falsifier":"Take a random sample of VCR-DF test items and have independent human experts judge whether each GPT-4 'misconception' label actually identifies a genuine error relative to the image content; if human-expert agreement with GPT-4's labels falls well below the reported 90% internal check, the benchmark's validity collapses and PEIFG's advantage may be an artifact of training and evaluating on the same GPT-4 target.","tokens_in":19833,"feed_emoji":"🎓","tokens_out":7179,"duration_ms":60321,"temperature":0.7,"pith_summary":"This paper claims that large multimodal models (LMMs) can learn a capability current models lack: looking at a multiple-choice visual commonsense question, noticing that a distractor option conflicts with the image, naming the underlying misconception, and explaining it in a way that guides a learner toward the correct answer. To make this testable, the authors build VCR-DF, a benchmark of 22,401 samples in which GPT-4 acts as a teacher, generating image-grounded distractors along with feedback (educational level, misconception, explanation) for each. They then propose PEIFG, a 1.8B-parameter LMM that combines region-level visual markers with global CLIP features, a learnable expert prompt pool, and DPO refinement to produce such feedback. Experiments report that PEIFG outperforms LMMs with far more parameters—including GPT-4V on automatic metrics—and the authors argue this opens a new direction for evaluating LMMs beyond forward reasoning.","feed_headline":"A small model beats big LMMs at explaining wrong answers","feed_subtitle":"New VCR-DF benchmark and PEIFG model push LMMs beyond forward reasoning to error correction.","key_machinery":"The argument is carried by two artifacts. First, the VCR-DF dataset: GPT-4 receives the image's event description, objects, question, and correct answer, classifies the question by Bloom's taxonomy level, generates five image-relevant distractors (manually filtered to the top three), and annotates each distractor with a misconception and an explanation; over 90% of these preliminary samples pass manual checks for accuracy and clarity. Second, the PEIFG model: a visual feature extractor that fuses region-level features from a SAM-based visual marker perceiver (trained with OPT-350M to read object boxes) with global CLIP features; a learnable pool of ten expert prompts from which the top three are selected via Q-Former instruction-aware features under a cosine-similarity key-matching loss and a correlation loss; and a QWen1.5 text generator with LoRA that ingests image tokens, expert prompts, question, answer, and distractor to produce feedback. A final refinement stage samples multiple outputs, scores them with five GPT-4 diagnostic questions, and applies direct preference optimization. The machinery's work is to ground error analysis in fine-grained object location and global scene context, channeled through specialist prompts.","core_discovery":"The central claim is that error correction in visual commonsense reasoning is a distinct, learnable capability: given a distractor that conflicts with visual commonsense, a model can be trained to name the misconception and explain the error in a way that guides a learner toward the correct answer. The paper's evidence is that on the newly constructed VCR-DF benchmark, the proposed PEIFG model achieves the highest scores on BLEU, METEOR, ROUGE-L, CIDEr, and BERTScore for both feedback and distractor generation, surpassing much larger open LMMs (3B–18B parameters), and it scores higher than GPT-4V on automatic metrics while ranking highest on human-rated helpfulness and logical consistency. The paper further asserts that existing LMMs, trained for forward reasoning, cannot perform this correction without specialized prompts, and that the VCR-DF benchmark exposes this gap.","pith_inferences":["Because GPT-4 supplies the training labels, the DPO reward, and the automatic evaluation references, the reported gap between PEIFG and other LMMs may partly reflect distillation toward GPT-4's own annotation style; an independent human-authored reference set would test whether the gap survives.","A natural transfer test is to apply the VCR-DF pipeline to other multiple-choice domains (science QA, driving scenes, medical imaging) to see whether the expert-prompt architecture generalizes beyond VCR.","The paper's entity-overlap diagnostic suggests other vision-language benchmarks may overstate reasoning ability; re-scoring them with image-relevant distractors could reveal similar inflation."],"forward_implications":["VCR-DF offers a reusable evaluation that does not reward entity-overlap heuristics: on the original VCR distractors, maximum entity overlap already gives over 60% accuracy, whereas the new distractors force genuine visual commonsense understanding.","Error-correction ability becomes a reportable axis of LMM evaluation, distinct from answer accuracy, and the paper's ablations show that specialized expert-prompt guidance matters more than model scale for this axis.","A single trained model serves both directions: PEIFG's feedback-generation skill transfers to generating new image-grounded distractors, enabling a loop of distractor design followed by misconception diagnosis.","The DPO refinement with GPT-4 diagnostic scoring is a reusable recipe for improving the faithfulness of generated explanations in other explanation-generation tasks."],"supporting_citations":[{"why":"Source dataset supplying images, questions, answers, and object boxes from which VCR-DF is derived.","marker":"[49]"},{"why":"GPT-4 acts as the teacher that generates educational levels, distractors, misconceptions, explanations, and the diagnostic scores used in DPO refinement.","marker":"[32]"},{"why":"Provides the self-instruction paradigm and the prompting approach followed to generate image-relevant distractors with GPT-4; also serves as a baseline (LLaVA-v1.5).","marker":"[27]"},{"why":"InstructBLIP baseline whose Q-Former design is adopted for extracting instruction-aware visual features.","marker":"[8]"},{"why":"CLIP image encoder supplies global-level visual features for the model.","marker":"[34]"},{"why":"SAM backbone of the visual marker perceiver for region-level visual features.","marker":"[20]"},{"why":"LoRA adapters enable parameter-efficient fine-tuning of the frozen text generator.","marker":"[18]"},{"why":"DPO is the refinement algorithm used to optimize the model on GPT-4 diagnostic preferences.","marker":"[36]"},{"why":"CogAgent is the strongest open LMM baseline; outperforming it supports the claim of PEIFG's advantage.","marker":"[17]"}],"fun_headline_variants":["Small model tops GPT-4V in explaining visual mistakes","New benchmark teaches AI to correct visual commonsense errors","Explainable feedback: making LMMs correct visual reasoning","VCR-DF: a benchmark for learning to explain visual errors","Teaching LMMs to explain visual commonsense errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire benchmark and the measured superiority of PEIFG rest on the assumption that GPT-4's generated distractors and feedback are a valid ground truth for visual commonsense errors—that GPT-4's judgments about what counts as a misconception and what explanation is correct are themselves correct.","fun_headline_variants_meta":{"raw":{"variants":["Small model tops GPT-4V in explaining visual mistakes","New benchmark teaches AI to correct visual commonsense errors","Explainable feedback: making LMMs correct visual reasoning","VCR-DF: a benchmark for learning to explain visual errors","Teaching LMMs to explain visual commonsense errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001047,"raw_usage":{"total_tokens":4403,"prompt_tokens":948,"completion_tokens":3455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":3374}},"tokens_in":564,"tokens_out":3455,"duration_ms":24768,"temperature":1.0,"reasoning_tokens":3374,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:19:54.704217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of VCR-DF test items and have independent human experts judge whether each GPT-4 'misconception' label actually identifies a genuine error relative to the image content; if human-expert agreement with GPT-4's labels falls well below the reported 90% internal check, the benchmark's validity collapses and PEIFG's advantage may be an artifact of training and evaluating on the same GPT-4 target.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source dataset supplying images, questions, answers, and object boxes from which VCR-DF is derived."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"InstructBLIP baseline whose Q-Former design is adopted for extracting instruction-aware visual features."},{"cited_title":"Berg, Wan-Yen Lo, Piotr Dollár, and Ross B","cited_arxiv_id":null,"evidence_quote":"SAM backbone of the visual marker perceiver for region-level visual features."}],"review_version":1}