{"id":"9377aa6a-c1a2-4c22-b783-fc29f7066fb8","arxiv_id":"2411.15201","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors introduce PARROT-360V, a benchmark of 2,487 visual word puzzles on which three leading VLMs score 28-56%, much lower than on common benchmarks.","lead":"PARROT-360V is a new benchmark of 2,487 image-based word puzzles that tests vision-language models on multi-step reasoning. On it, GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro score far lower than on standard benchmarks, exposing a gap in complex visual reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The performance gap is uninterpretable without a human/chance baseline and a validated scoring protocol; Eq. (4) defines hallucination as 1 minus the overall score, so the claimed reasoning failures are not established.","rationale":"The reader's weakest-assumption diagnosis focuses on data contamination. That is a genuine concern, but it is not the most load-bearing for the central claim: if the models had memorized these publicly scraped Jumble puzzles and still scored only 28–56%, the low scores would arguably be even stronger evidence of limited transferable reasoning. The more fundamental problem is that the benchmark's scoring protocol has not been validated as a measure of multi-step visual reasoning. The exact-match, cascading penalty structure, the lack of a human or chance baseline, and the explicit conflation of hallucination rate with overall error rate mean that the reported numbers do not yet support the interpretive leap from 'low scores on PARROT-360V' to 'VLMs lack complex visual reasoning.' This concern is fixable: a human-baseline study and a released, rubric-based scoring script would settle it. Since the reader already issued a CONDITIONAL verdict, my analysis does not change the disposition, but it does shift the primary condition from contamination to construct validity. I therefore mark agreement as partial and recommend UNCHANGED, with the condition understood to include scoring validation and human baselines.","tokens_in":7084,"tokens_out":3641,"duration_ms":39372,"concrete_test":"Sample 100 PARROT-360V puzzles. Have three independent human raters solve them from the same input images with the same instructions, and score both human and model outputs with the paper's exact-match protocol. Then have a blinded rater rescore all model outputs with a partial-credit rubric. Release the raw outputs and per-component scores. If human exact-match scores are near or above the VLM scores and partial-credit rescoring does not move VLM scores by more than 10 points, the reported gap is real. If human scores are also low, or partial-credit rescoring substantially changes scores or rankings, the gap is an artifact of the scoring protocol rather than evidence about VLM reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PARROT-360V scores of 28–56% reveal VLM limitations in multi-step visual reasoning. For this claim to hold, the benchmark score must measure the intended construct, and the drop relative to MMMU/ChartQA must be attributable to reasoning complexity rather than to task format, scoring brittleness, or annotation issues. That condition is not secured. In §4.2, Eq. (3) awards −5 for every incorrect or missing component and clips the total at zero, so a single unscrambling failure makes the bonus answer impossible and cascades into a near-zero score. The paper reports no per-component breakdown, no parsing protocol, no chance level, and no human ceiling. Section 8 concedes that task complexity may obscure whether failures come from reasoning difficulty or intricate formatting, and that perception and reasoning have not been separated. Moreover, Eq. (4) defines HallucinationRate = 1 − PARROT360VScore, which conflates every error with hallucination even though §4 defines hallucination as introducing information not present in the input; the reported 43–72% hallucination rates therefore do not measure hallucination. Without a human baseline or a rubric-validated rescoring, the low absolute scores could reflect exact-match brittleness, output-format mismatch, or benchmark artifacts rather than a genuine reasoning deficit. The cross-benchmark comparison is also confounded: Jumble puzzles are intentionally harder and more formatting-sensitive than multiple-choice QA, so a lower score does not by itself demonstrate that existing benchmarks overestimate capability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PARROT-360V, a benchmark of 2487 Jumble-style visual puzzles collected from public internet sources, and evaluates three state-of-the-art VLMs (GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-Pro) on them. The authors report large performance drops relative to popular benchmarks, with scores of 57%, 50%, and 28% respectively, and interpret the drop as evidence that current VLMs lack complex multi-step visual reasoning ability. The paper also defines a hallucination rate as 1 minus the benchmark score and reports hallucination rates of 43%, 50%, and 72% for the three models.","tokens_in":7394,"tokens_out":3793,"duration_ms":42829,"significance":"The benchmark resource is potentially valuable: it addresses a real gap by requiring step-by-step visual reasoning, the dataset is publicly released, and the task is genuinely multi-modal rather than a multiple-choice proxy. If the performance numbers were properly validated, the conclusion that standard benchmarks overestimate VLM reasoning would be an important contribution to the field. However, the current manuscript does not yet secure the validity of its own metrics: the hallucination rate is not measured as defined, the scoring protocol is brittle and unreported in its components, and there are no human or chance baselines. The significance is therefore conditional on substantial additional validation.","major_comments":[{"comment":"The hallucination rate does not measure hallucination as defined in the text. The paper defines hallucination as the frequency with which models introduce information not present in the input, but Eq. (4) sets HallucinationRate = 1 − PARROT360VScore, which counts every incorrect or missing answer component as a hallucination. Consequently, the claims in Section 5.2 that GPT-4o hallucinated 43%, Claude-3.5-Sonnet 50%, and Gemini-1.5-Pro 72% of the time are not supported by the reported computation. To draw conclusions about hallucination, the authors need a per-response annotation of whether the model actually generated content absent from the input, rather than a direct transform of the aggregate score.","section":"Section 4.2, Eq. (4)"},{"comment":"The scoring protocol is too brittle to support the reported absolute scores without additional diagnostic information. Each incorrect or missing component receives −5, and the total is clipped at zero; because the bonus clue and puzzle answer depend on the four unscrambled words, a single failure can make the remaining components impossible and drive the whole puzzle to zero. The paper reports no per-component breakdown, no parsing protocol for free-form model outputs, and no inter-annotator agreement on judging correctness. Section 8 itself concedes that task complexity may obscure whether failures come from reasoning difficulty or task intricacy. Without component-level results and a human ceiling or chance-level baseline, the 28–57% scores could reflect exact-match brittleness, parsing artifacts, or output-format mismatch rather than the intended reasoning deficit.","section":"Section 4.2, Eq. (3), and Section 5"},{"comment":"The cross-benchmark performance gap is not statistically grounded. Table 2 lists MMMU, MathVista, AI2D, and ChartQA scores without confidence intervals, number of runs, or evaluation-protocol details, and PARROT-360V scores in Figure 2 are shown without error bars. The comparison is also confounded by format: the reference benchmarks are largely multiple-choice or short-answer tasks, while PARROT-360V requires multi-step free-form answers with a rigid scoring rule. The claim that the performance drop is 'significant' therefore needs either matched statistical testing on the same models or a human baseline on the same PARROT-360V protocol.","section":"Section 5 and Table 2"},{"comment":"The data-contamination safeguard is not sufficient as stated. Section 3 reports that puzzles were scraped from public internet sources, and Section 4 claims the setup is 'entirely novel' because the prompting is novel. Novel prompting does not make the puzzle images unseen: the same Jumble puzzles could easily appear in web-scale training corpora. The paper provides no contamination check, no temporal split, and no evidence that the evaluated models were not exposed to these exact images. If a model had encountered a puzzle, high scores could reflect memorization, whereas if the model had not, low scores could reflect unfamiliarity with the puzzle format; either way the central performance-gap interpretation requires an explicit contamination analysis.","section":"Sections 3 and 4"}],"minor_comments":[{"comment":"There are several grammatical issues, for example 'Rather benchmarking for VLMs should evaluate perception' and 'not adequately capturing the abilities of the model'; these should be rewritten.","section":"Section 2.1"},{"comment":"The spelling of the benchmark name is inconsistent: 'PARROT360V' appears in Section 4.1 while 'PARROT-360V' is used elsewhere.","section":"Section 4"},{"comment":"The phrase 'with-in' should be 'within', and 'required higher-order detail to reasoning' is ungrammatical; the intended meaning should be stated clearly.","section":"Section 5.1"},{"comment":"The sentence 'And Hallucination rate is the error rate, i.e. the proportion of incorrect predictions given by an VLM' should be rewritten; 'an VLM' should be 'a VLM', and the sentence should not begin with 'And'.","section":"Section 4.2"},{"comment":"The paper cites Wei et al. 2022 and Wei et al. 2023 as though they were two separate papers, but the reference list entries are the same arXiv paper; this should be corrected.","section":"References"},{"comment":"The phrase 'scored between 28 to 56 percentage' should be 'scored between 28% and 56%' or 'between 28 and 56 percentage points'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is already accepted at COLING 2025, and my report is written in that context. The benchmark resource itself is potentially useful, and I do not see grounds for rejection. However, the validity of the headline claims is not yet established: the hallucination metric is definitionally circular, the scoring protocol lacks validation and diagnostics, and the contamination argument is weaker than claimed. These are fixable with additional analysis (component-level scores, human/chance baselines, and a contamination check), which is why I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing worth knowing about this paper is that the dataset is real and useful: 2,487 Jumble-style puzzles, scraped from a public source, with structured ground truth (clues, answers, circled letters, bonus solution) and a HuggingFace release. That is a concrete artifact the community can build on, and evaluating GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro on it is a reasonable first pass. The task genuinely requires multi-step integration of text, layout, and cartoon clues, which is more demanding than most multiple-choice VLM benchmarks. Credit where due: the data collection and annotation are straightforward but the resulting benchmark fills a real gap.\n\nThe soft spots are serious, though. The hallucination rate is defined in Eq. (4) as 1 minus the overall score, which does not measure hallucination as described in Section 4 (introducing information not present in the input). Calling that a hallucination rate conflates every error type — wrong unscrambling, wrong circled letter, format mismatch — with fabrication. The authors' own numbers therefore do not support the claim that models hallucinate at 43–72%.\n\nSecond, there are no baselines that make the low absolute scores interpretable. No human ceiling, no chance level, no per-component breakdown, no confidence intervals. The scoring system is brittle: a −5 penalty for each wrong component with clipping at zero means one early failure cascades into a near-zero score, so the raw scores could reflect exact-match brittleness or output-format mismatch rather than reasoning failure. The authors acknowledge part of this in the Limitations section — task complexity may obscure whether failures come from reasoning difficulty or intricate formatting, and perception and reasoning have not been separated. That concession is honest but it undercuts the central interpretation.\n\nThird, the contamination story is thin. The puzzles are scraped from the public internet, so they could easily be in training corpora. The claim that the setup is \"entirely novel\" (Section 4) is asserted, not demonstrated, and the planned mitigation (regular updates) is future work.\n\nThe cross-benchmark comparison to MMMU/ChartQA/AI2D is also confounded: Jumble puzzles are deliberately harder and more format-sensitive than multiple-choice QA, so a lower score alone does not show that existing benchmarks overestimate capability. The argument may be true, but this paper does not nail it.\n\nWho should read this: researchers working on VLM evaluation benchmarks, especially those interested in puzzle-style reasoning tasks. The dataset is worth citing and the paper deserves a serious referee, but only with substantial revisions — fix the hallucination metric, add human and chance baselines, report component-level scores, and release the evaluation code. I would not take the 28–56% numbers at face value yet.","headline":"A genuinely new puzzle-based VLM benchmark with a public dataset, but the headline claims about reasoning failure are undercut by a misdefined hallucination metric and missing baselines.","tokens_in":7955,"tokens_out":1447,"would_cite":true,"duration_ms":51853,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new visual-puzzle benchmark reports that top vision-language models score only 28% to 56%.","keywords":["PARROT-360V","vision-language models","visual reasoning benchmark","multi-step reasoning","hallucination","Jumble puzzles","chain-of-thought evaluation","data contamination"],"falsifier":"Build a fresh set of same-format Jumble puzzles that cannot be in the models' training data and compare scores with the published PARROT-360V set; if the fresh set closes the performance gap, the reported 28% to 56% scores are explained by memorization rather than by a genuine visual-reasoning limit.","tokens_in":6904,"feed_emoji":"🧩","tokens_out":14376,"duration_ms":137933,"temperature":0.7,"pith_summary":"PARROT-360V is a benchmark of 2,487 Jumble-style visual puzzles that asks a vision-language model to unscramble four words, read circled letters from the image, interpret a cartoon clue, and assemble a final bonus answer. The paper reports that GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro score roughly 56%, 50%, and 28% on this test, far below their averages of 72% to 80% on popular benchmarks such as MMMU, ChartQA, AI2D, and MathVista. The authors' intended point is that current benchmarks overstate real-world visual reasoning because their questions can often be answered with text knowledge or single-step image matching. If the benchmark holds, it identifies a concrete weakness in current VLMs: multi-step visual integration, fine-grained letter perception, and resistance to hallucination.","feed_headline":"Top vision models score 28-56% on new puzzle benchmark","feed_subtitle":"Leading models fall to 28-56% on Jumble-style puzzles, suggesting current benchmarks overstate visual reasoning.","key_machinery":"The central object is PARROT-360V itself: a dataset of 2,487 Jumble puzzles scraped from the internet, with ground-truth labels drawn from the solved puzzles. Each entry contains a question screenshot, four scrambled words, circled-letter positions, a visual cartoon clue, and the final bonus answer; models are prompted to work through the puzzle step by step using a chain-of-thought plan. The scoring system gives 10 points for each unscrambled word, 10 points for extracting the circled characters, and 20 points for the final bonus answer, for a maximum of 70 per puzzle; wrong or missing answers subtract 5 points, negative totals are clipped to zero, and the normalized score is reported. Hallucination rate is defined as one minus that score, so the benchmark's headline numbers bundle perception errors, reasoning errors, and invented characters into a single measure.","core_discovery":"On the paper's own terms, the discovery is a performance differential. State-of-the-art models that average 0.72 to 0.80 on MMMU, MathVista, AI2D, and ChartQA score 0.28 to 0.57 on PARROT-360V (GPT-4o at 0.57, Claude-3.5-Sonnet at 0.50, Gemini-1.5-Pro at 0.28; the abstract states the range as 28% to 56%). The benchmark's design attributes this drop to tasks that cannot be answered by recall: each puzzle requires sequential word unscrambling, extraction of circled letters from the image, interpretation of a cartoon clue, and synthesis of a final bonus answer. The paper reports hallucination rates of 43%, 50%, and 72%, defined as one minus the benchmark score, and interprets the gap as evidence that conventional leaderboards inflate what VLMs can actually do in complex visual tasks.","pith_inferences":["A natural decomposition experiment the paper does not run would feed the model the four unscrambled words and the circled letters as text, leaving only the cartoon clue to interpret; if scores stay low, the bottleneck is visual-clue integration, while a large jump would implicate letter-level perception.","Because all three evaluated models are proprietary, PARROT-360V is hard to audit externally; running the same benchmark on open-weight models would make contamination checks and score verification possible for independent researchers.","The daily publication cycle of Jumble puzzles means the benchmark could be re-generated continuously; a rolling version would keep the test unseen and automatically retire the contamination assumption the fixed 2,487-puzzle set depends on."],"forward_implications":["A high score on MMMU, ChartQA, AI2D, or MathVista should no longer be read as evidence that a model can handle multi-step visual reasoning; the same model can fall by more than half on PARROT-360V.","The benchmark's per-component scoring lets developers see exactly where a model fails, such as circled-letter recognition versus final bonus synthesis, making it a diagnostic tool rather than just a leaderboard.","The reported hallucination rates, however defined, point to a practical failure mode: these models introduce letters and answers not supported by the input image when forced to do fine-grained visual work.","If current state-of-the-art models sit at 28% to 56%, then the field's progress claims on complex visual reasoning should be re-examined, and evaluation suites should include stepwise visual puzzles of this kind."],"supporting_citations":[{"why":"Supplies the MMMU baseline whose high model scores create the contrast that PARROT-360V is designed against.","marker":"Yue et al. (2024)"},{"why":"Supplies the ChartQA benchmark that appears in the comparison table against PARROT-360V.","marker":"Masry et al. (2022)"},{"why":"Supplies the AI2D benchmark, cited as an example where models face straightforward visual questions and score much higher.","marker":"Kembhavi et al. (2016)"},{"why":"Provides the chain-of-thought prompting technique the benchmark uses to structure models' step-by-step puzzle solving.","marker":"Wei et al. (2022)"},{"why":"Supports the paper's concern that models can overfit or be contaminated on existing benchmarks, which motivates the new dataset.","marker":"Samuel et al. (2024)"}],"fun_headline_variants":["Vision models score 28-56% on new complex puzzle benchmark","New puzzle benchmark stumps top vision models, scores 28-56%","Vision language models fail complex puzzles, scoring 28-56%","Complex puzzles reveal vision models' reasoning gap: 28-56%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Jumble puzzles scraped from a public source were not already memorized by the evaluated models, so their low scores measure perception and reasoning rather than failure to recall a seen puzzle.","fun_headline_variants_meta":{"raw":{"variants":["Vision models score 28-56% on new complex puzzle benchmark","New puzzle benchmark stumps top vision models, scores 28-56%","Vision language models fail complex puzzles, scoring 28-56%","Complex puzzles reveal vision models' reasoning gap: 28-56%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2665,"prompt_tokens":939,"completion_tokens":1726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1660}},"tokens_in":555,"tokens_out":1726,"duration_ms":12636,"temperature":1.0,"reasoning_tokens":1660,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:59:57.750591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a fresh set of same-format Jumble puzzles that cannot be in the models' training data and compare scores with the published PARROT-360V set; if the fresh set closes the performance gap, the reported 28% to 56% scores are explained by memorization rather than by a genuine visual-reasoning limit.","supporting_citations":[],"review_version":1}