{"id":"8eee926f-3f2a-4bf8-983e-8513474c2f6a","arxiv_id":"2507.13405","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"COREVQA introduces a 5,608-pair true/false visual entailment benchmark for crowd images on which the strongest tested vision-language models reach only 77.57% accuracy.","lead":"COREVQA is a new benchmark of 5,608 true/false visual entailment questions built from crowded-scene images, and top vision-language models score below 80% accuracy on it. It is designed to expose weaknesses in how AI models verify fine-grained details in busy images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing majority-class baseline (always false = 72.1%) weakens the claim that below-80% accuracies reveal VLM limitations; Janus Pro's 72.31% accuracy with recall 1.85% matches a constant-false classifier.","rationale":"The reader's conditional verdict identifies the missing always-false baseline as a key issue; that is the strongest load-bearing concern because it directly undercuts the interpretation of the reported accuracies. The annotation-reliability issue is real but secondary, since even with perfect labels the baseline problem would remain. The paper's own data show Janus Pro's near-zero recall, making its 72.31% accuracy clearly consistent with a constant-false policy. GPT-4.1 and GPT-4o mini are only modestly above baseline, so the central claim that the benchmark reveals 'key limitations' is overstated without a comparative baseline. The recommended check (balanced accuracy and per-class accuracy) would settle the matter. The paper still contributes a novel dataset and a generation pipeline, so a conditional acceptance requiring these supplementary results is appropriate, rather than moving to reject or unverified. The reader's verdict already reflects this conditionality, so no change to the reader's verdict is needed.","tokens_in":10211,"tokens_out":3803,"duration_ms":41078,"concrete_test":"Recompute all model evaluation metrics on COREVQA with a majority-class baseline: report balanced accuracy = (TPR + TNR)/2 and the margin over the always-false baseline (72.1%) for each model. Also report per-class accuracy (true vs. false statements). If Janus Pro's balanced accuracy is near 50% and GPT-4.1's margin over baseline is less than 10 points, the conclusion that models 'struggle' should be reframed as 'only slightly above a trivial constant-false classifier', and the central claim about revealing key limitations is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that VLMs struggle on COREVQA rests on absolute accuracy numbers, but the dataset has 72.1% false statements, so always answering 'False' yields 72.1% accuracy. Among the five evaluated models, Janus Pro scores 72.31% with recall of 1.85% and F1 of 3.60, which is the signature of a model that almost always answers 'False' and thus is essentially at the trivial baseline. GPT-4.1 (77.57%) and GPT-4o mini (76.60%) exceed the baseline by only ~5 percentage points, and their balanced accuracy (mean of true-positive rate and true-negative rate) would be lower than the unweighted accuracy suggests. Without reporting per-class accuracy, balanced accuracy, or a majority-class baseline, the paper overstates the extent to which below-80% accuracies reveal visual-reasoning failures. The omission is particularly consequential because the paper's own analysis (A.7.2) shows Janus Pro has the highest accuracy on 'difficult' cases (64.0%) while GPT-4.1 and GPT-4o mini drop to 0.3% and 0.2% on those same cases, so the aggregate numbers can mislead about which models are actually reasoning. The claim that these results 'reveal key limitations' requires showing models do substantially better than exploiting the label prior; the current evidence does not establish this for Janus Pro and shows only a modest margin for the top models.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces COREVQA, a benchmark of 5,608 image-statement pairs for visual entailment in crowded scenes. Images are taken from CrowdHuman, and true/false statements are synthetically generated via LLM prompts; ground truth is hand-labeled. Five vision-language models are evaluated, and the paper reports that even the best model (GPT-4.1) achieves only 77.57% accuracy, with other models ranging from 39.98% to 72.31%, interpreting this as evidence of fundamental limitations in VLM visual reasoning.","tokens_in":10469,"tokens_out":5893,"duration_ms":62501,"significance":"If the benchmark is validated and the empirical claims are properly baselined, COREVQA would be a useful resource for studying fine-grained visual entailment under visual clutter, a relatively under-tested capability. The authors provide a public dataset, code, and detailed generation prompts, and they report evaluations of several current models. However, the significance of the main empirical finding is currently weakened by the absence of a majority-class baseline and the lack of label-reliability evidence, both of which are directly addressable.","major_comments":[{"comment":"The evaluation does not include a majority-class baseline or per-class accuracy. Because 72.1% of statements are false (Table 1), a trivial always-false classifier achieves 72.1% accuracy. Janus Pro's reported 72.31% accuracy with recall of 1.85% and F1 of 3.60 is behaviorally indistinguishable from such a classifier, so the central claim that models 'struggle' on COREVQA is not established for this model. For GPT-4.1 (77.57%) and GPT-4o mini (76.60%), the margin over the always-false baseline is only about 5 percentage points. The paper should explicitly compare every model against the always-false baseline and report balanced accuracy or per-class accuracy. This is load-bearing for the abstract's claim that results 'reveal key limitations'; the A.7.2 finding that Janus Pro scores 64.0% on 'difficult' cases while GPT-4.1 scores 0.3% further illustrates that aggregate accuracy can mislead when the label prior is ignored.","section":"Section 4.2, Table 3; Table 1"},{"comment":"The ground truth is hand-labeled, but the paper reports no number of annotators, no annotation disagreement-resolution protocol, and no inter-annotator agreement measure (e.g., Cohen's kappa) on a sample. Since every model score is computed against these labels, the benchmark's validity rests entirely on their correctness. The sentence 'To ensure complete accuracy of ground truths, the entire dataset was manually labeled' is an assertion, not a demonstration. The authors should provide inter-annotator agreement statistics on a subset or otherwise document a second-pass verification process so that readers can assess label reliability.","section":"Section 3.3"},{"comment":"The 'difficult cases' used for the failure-pattern analysis are defined as questions where at least two of the five evaluated models answer incorrectly. This makes the reported failure-pattern percentages (81.3% action recognition failures, 78.1% detail oversight, etc.) dependent on the particular model set and, if used to infer general VLM weaknesses, is circular. The percentages could change substantially with a different set of models or with an independent notion of difficulty (e.g., human response time or label agreement). The authors should either define difficulty using an independent criterion or present the failure analysis per model with appropriate caveats about the model-dependent subset.","section":"Section 4.3 and A.7.2"}],"minor_comments":[{"comment":"The paper does not describe how model outputs were parsed into 'True' or 'False' (e.g., exact-match, case sensitivity, handling of verbose responses). This detail is needed for reproducibility.","section":"Section 4.1"},{"comment":"No confidence intervals or significance tests are reported. For N=5,608, the 0.97-point accuracy difference between GPT-4.1 (77.57%) and GPT-4o mini (76.60%) is within sampling error, so the statement that 'GPT-4.1 achieves the highest overall accuracy' should be qualified or accompanied by statistical support.","section":"Section 4.2 and Table 3"},{"comment":"The paper does not report how many true and false statements were generated by ChatGPT-4.1 versus Claude Opus, making it impossible to assess whether the label imbalance (27.9% true) is an artifact of one generator or of the generation prompts.","section":"Section 3.2.2 and Table 1"},{"comment":"Table 2 contains a formatting issue ('Y es' appears as two words), and the comparison to SNLI-VE should clarify that SNLI-VE is also a visual entailment benchmark; the current row makes COREVQA appear unique in a way that the text later qualifies.","section":"Table 2"},{"comment":"The limitation that 'using a true or false format does not provide insight into key areas where a model went wrong' is contradicted by the failure analysis in A.7.2; either clarify the intended meaning or revise the sentence.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clearly written and the benchmark resource is potentially useful, but the central empirical claim needs to be re-baselined with a majority-class comparison and per-class metrics, and the ground-truth labeling needs reliability evidence. These are addressable within the manuscript's scope, so I recommend major revision rather than rejection. The paper may also benefit from a careful check of whether the 'first large-scale crowd-based visual entailment benchmark' claim is appropriately qualified given existing visual entailment datasets such as SNLI-VE."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: COREVQA is a real, niche dataset—crowd images paired with synthetic true/false entailment statements—and the authors did the right thing by hand-labeling ground truth when their automated solver only hit 89%. But the headline claim that below-80% accuracies reveal VLM limitations doesn't survive contact with the label distribution, and the appendix contains a result that quietly contradicts the main story.\n\nWhat's new: nobody else has combined dense crowd imagery with visual entailment verification. The generation pipeline (GPT-4.1 for true statements, Claude for false, with explicit prompt strategies like quantifier bait and occlusion traps) is documented in enough detail to reproduce. They report precision, recall, and F1, not just accuracy, and they acknowledge the true/false imbalance in the limitations section. Data and code are public. All of that is credit-worthy.\n\nThe soft spots are in the evaluation, and they're not minor. 72.1% of statements are false, so an always-false classifier gets 72.1% accuracy. Janus Pro's 72.31% with 1.85% recall is exactly that classifier; the paper even notices the bias ('strong bias toward answering false') but doesn't connect it to the baseline problem. GPT-4.1 (77.57%) and GPT-4o mini (76.60%) clear the trivial baseline by only about five points, and we're not given per-class accuracy or balanced accuracy to see whether that margin is real signal or mostly the prior. The absence of a majority-class baseline is the single biggest omission. Second, there's no inter-annotator agreement on the hand-labeled ground truth; with 5,608 pairs and crowded images, some labeling noise is likely, and the paper gives us no way to estimate it. Third—and this is the one that would make me stop as a referee—Appendix A.7.2 reports that on 'difficult' questions, GPT-4.1 and GPT-4o mini drop to 0.3% and 0.2% accuracy while Janus Pro gets 64.0%. That inverts the aggregate ranking and suggests the aggregate numbers are dominated by easy cases and label prior, not by reasoning ability. The paper doesn't reconcile this.\n\nWho this is for: people building or evaluating VLMs on dense-scene visual verification. As a dataset release, it deserves referee time; as a demonstration of VLM limitations, the current evidence is too weak. My recommendation: accept it as a benchmark resource, but require the authors to add the majority-class baseline, per-class metrics, annotation agreement, and to either explain or remove the confusing A.7.2 analysis.","headline":"A useful niche dataset whose evaluation section needs a majority-class baseline before its 'VLM struggle' claim can be taken seriously.","tokens_in":11007,"tokens_out":3773,"would_cite":false,"duration_ms":38209,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COREVQA is a 5,608-pair true/false visual entailment benchmark for crowded scenes, and the paper shows that current vision-language models, led by GPT-4.1 at 77.57%, fail to reach 80% accuracy.","keywords":["visual entailment","vision-language models","VQA benchmark","crowd scenes","synthetic question generation","true/false statements","CrowdHuman","model evaluation"],"falsifier":"Run an always-false classifier over COREVQA, which would score 72.1% by construction; if top models' advantage over that trivial baseline is small, or if a random sample of statements re-labeled by independent annotators disagrees with the authors' labels, the benchmark's difficulty claim weakens.","tokens_in":10011,"feed_emoji":"👥","tokens_out":4428,"duration_ms":48358,"temperature":0.7,"pith_summary":"COREVQA is a benchmark of 5,608 image–statement pairs built from CrowdHuman crowd photographs, asking vision-language models to decide whether a synthetically written true/false statement is entailed by the image. The paper's central claim is that even the best tested models, GPT-4.1 at 77.57% accuracy, stay below 80%, with other models ranging from 39.98% to 76.60%, and that this gap exposes systematic weaknesses in visual entailment under crowded, cluttered conditions. The authors attribute the failures to recurring patterns: action recognition, oversight of small details, counting errors, spatial reasoning, and negation. If correct, the benchmark provides a diagnostic tool that existing crowd datasets, which focus on detection and counting, do not offer.","feed_headline":"Vision models top out below 80% on crowd-reasoning test","feed_subtitle":"A 5,608-pair true/false benchmark from real crowd images exposes weak spots in action, count, and spatial reasoning.","key_machinery":"The central object is the COREVQA benchmark and its synthetic generation pipeline. The pipeline pairs CrowdHuman images with statements crafted by two LLMs under separate true/false prompts that include strategies like quantifier bait, occlusion traps, and hidden contradictions, plus a self-reflection step that forces the generator to justify how the statement might deceive. Statements average 30 words, 57.7% use spatial terms, 39.0% reference clothing, and 35.1% mention color, and the binary true/false format is a key design choice meant to remove multiple-choice artifacts. Hand-labeling of all 5,608 ground truths is what makes the reported accuracy, precision, recall, and F1 scores interpretable as measures of visual entailment ability.","core_discovery":"The paper introduces COREVQA, a benchmark pairing dense crowd images with synthetically generated true/false statements designed to require meticulous visual inspection and multi-step visual entailment. Statements are generated by prompting ChatGPT-4.1 for true claims and Claude 3 Opus for false claims, with prompt strategies such as quantifier bait, occlusion traps, and hidden contradictions, and every ground truth is hand-labeled because an automated solver reached only 89% agreement on a 250-pair sample. On the full set, evaluated models show that even the strongest performer, GPT-4.1, reaches only 77.57% accuracy, while GPT-4o mini scores 76.60%, Janus Pro 72.31%, Qwen2.5 vl 72b 69.95%, and LLaVa-NeXT 39.98%, with LLaVa-NeXT showing near-perfect recall but low precision and Janus Pro showing a strong bias toward answering false. These results support the claim that current vision-language models cannot reliably verify complex claims about dense crowd scenes, with failure patterns concentrated in action recognition, detail oversight, counting, spatial relations, and negation.","pith_inferences":["The paper reports no always-false baseline; since 72.1% of statements are false, that trivial rule already scores 72.1%, so the meaningful headroom above chance is narrower than the raw accuracy numbers suggest.","A natural follow-up the paper does not run is per-category scoring: computing accuracy separately on counting, spatial, negation, and action subsets could show whether model rankings change by skill.","The same pipeline could generate entailment benchmarks for other dense domains, such as animal groups or vehicle traffic, to test whether the observed weaknesses are crowd-specific or general to clutter.","Because the labels were hand-labeled by the authors without reported inter-annotator agreement, an independent re-labeling study would be the cheapest way to harden the benchmark and confirm that model rankings reflect visual entailment rather than label noise."],"forward_implications":["If COREVQA scores reflect capability, current vision-language models are not reliable enough for automated judgment in crowd monitoring or safety applications, where below-80% accuracy on verified statements implies frequent errors.","The dataset's failure categories give a concrete diagnostic checklist: action recognition, fine-detail verification, counting, spatial relations, and negation are the specific skills that models need to improve.","Because statements are synthetically generated from images, the pipeline can be retargeted to new image collections with relatively little human effort beyond label verification, making new entailment benchmarks cheaper to produce.","Other benchmarks can adopt the true/false format to reduce the multiple-choice shortcut documented in LLM evaluation, since models cannot infer the correct answer from the answer choices themselves."],"supporting_citations":[{"why":"Supplies all source images from the CrowdHuman dataset, giving the benchmark its dense, multi-person visual foundation.","marker":"(Shao et al., 2018)"},{"why":"Defines visual entailment as a task and provides SNLI-VE, the prior benchmark whose simpler imagery COREVQA claims to go beyond.","marker":"(Xie et al., 2019)"},{"why":"Provides HallusionBench, the adversarial diagnostic benchmark that COREVQA positions itself against while adding crowded-scene visual complexity.","marker":"(Guan et al., 2024)"},{"why":"Documents how models can answer multiple-choice questions without the question, motivating the true/false format used in COREVQA.","marker":"(Balepur et al., 2024)"},{"why":"Provides VQAv2, the standard general VQA benchmark used as a baseline comparison to show COREVQA's more targeted difficulty.","marker":"(Goyal et al., 2017)"},{"why":"Supplies the GPT-4 technical report used as a basis for the evaluated GPT-series models.","marker":"(Achiam et al., 2023)"}],"fun_headline_variants":["New benchmark shows VLMs fail crowd-reasoning tests","Crowd VQA benchmark stumps best vision models","VLMs under 80% on new visual entailment benchmark","COREVQA: Why vision models struggle with crowded scenes","Benchmark exposes blind spots in vision-language models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-labeled ground-truth answers are correct, even though no inter-annotator agreement is reported to show that the labels are reliable.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark shows VLMs fail crowd-reasoning tests","Crowd VQA benchmark stumps best vision models","VLMs under 80% on new visual entailment benchmark","COREVQA: Why vision models struggle with crowded scenes","Benchmark exposes blind spots in vision-language models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1392,"prompt_tokens":948,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":564,"tokens_out":444,"duration_ms":5085,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:37:44.626033+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an always-false classifier over COREVQA, which would score 72.1% by construction; if top models' advantage over that trivial baseline is small, or if a random sample of statements re-labeled by independent annotators disagrees with the authors' labels, the benchmark's difficulty claim weakens.","supporting_citations":[],"review_version":1}