{"id":"f18c6978-dbe1-4bc0-9bd0-ebba021f6db1","arxiv_id":"2506.11595","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EasyARC is a new procedurally generated visual reasoning benchmark where state-of-the-art vision-language models score below 20%, despite tasks designed to be easy.","lead":"This paper introduces EasyARC, a set of simple visual puzzles for testing whether AI models can truly reason about images rather than just recognize them. Current top vision-language models mostly fail these easy tasks, so the benchmark offers a cheap, verifiable way to measure real visual thinking.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No text-only control undermines the claim that EasyARC measures true visual reasoning; if models solve from textual grid transcriptions, the benchmark reduces to extraction plus text reasoning.","rationale":"The reader's weakest assumption correctly identifies the lack of a text-only control as the central load-bearing issue. The paper's own observations about QVQ's extract-then-reason strategy and the text-formatted test input in Appendix B make this concern concrete rather than speculative. The proposed test directly settles whether EasyARC measures visual reasoning or merely multimodal extraction plus text reasoning. Since the reader already issued a CONDITIONAL verdict focused on this gap, my stress-test does not change that verdict; it reinforces the condition. I considered whether the single-sample evaluation or missing code links might be more load-bearing, but those are practical limitations that do not threaten the construct validity of the benchmark as directly as the text-only control does. Thus the verdict remains conditional pending this specific experimental check.","tokens_in":183,"tokens_out":1531,"duration_ms":24039,"concrete_test":"Run a text-only baseline on a random subset of at least 100 EasyARC test tasks: convert each input and output grid into a text format identical to the paper's prompt (rows as newline-separated strings of color names or single letters), provide the same demonstrations and test input in text, and query a strong text-capable model (e.g., GPT-4o with vision disabled, or a non-vision LLM) with the same evaluation protocol. If text-only accuracy is comparable to or higher than the VLM accuracy in Figure 3, then the benchmark does not require visual reasoning and the central claim fails; if text-only accuracy is near zero, the visual pathway is necessary and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EasyARC tests 'true visual reasoning' and that current VLMs fail because they cannot reason visually. The load-bearing premise is that solving the tasks requires reasoning over the image itself, not just extracting its contents into text. The paper provides internal evidence that this premise is insecure: Section 3.5 documents that QVQ-72B-Preview succeeds by extracting the grids into text and then reasoning textually, and Appendix B shows the prompt already includes the test input as a text grid. The tasks use grids of only 3–10 rows/columns, so a faithful textual transcription is small and unambiguous. Yet no text-only baseline is reported: no experiment feeds the same input-output examples as text (without images) to a strong language or multimodal model and measures accuracy. Without that control, the observed failures could be attributed to imperfect visual extraction rather than absence of 'true visual reasoning.' If a text-only model achieves high accuracy, then EasyARC primarily measures multimodal extraction plus text reasoning, and the paper's stronger claims about a new standard for evaluating visual reasoning are not supported. The absence of a human baseline (Section 7.3) further weakens the 'extremely easy' characterization, but the text-only control is the decisive missing piece.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EasyARC, a procedurally generated vision-language benchmark of grid-transformation puzzles inspired by the ARC challenge. It contains five task families (Cross/Star, Counting Cells, Double Grid, Dominant Side, Drop One Color) with three difficulty levels, 5000 training and 500 evaluation tasks, and exact-match evaluation against ground-truth grids. The authors evaluate five state-of-the-art VLMs at temperature 0.5 with one sample per model, reporting that all models except Claude 3.7 Sonnet score below 20% on tasks they describe as extremely easy. They analyze failure modes, including QVQ-72B-Preview's strategy of extracting grids into text before reasoning, and argue that EasyARC provides a scalable, verifiable testbed for true visual reasoning and test-time scaling in RL pipelines.","tokens_in":7157,"tokens_out":2736,"duration_ms":27846,"significance":"If validated, EasyARC would be a useful contribution: it is fully verifiable through exact-match scoring, procedurally expandable through configurable generators, and openly released with evaluation code. The task families and difficulty progression are sensible, and the initial finding that strong VLMs struggle on simple grid transformations is interesting and worth investigating. The caveat is that the paper's central construct---'true visual reasoning'---is not yet established, because the benchmark does not include the text-only or perceptual controls needed to distinguish reasoning over images from textual extraction followed by text-based reasoning.","major_comments":[{"comment":"The claim that EasyARC measures 'true visual reasoning' rests on the assumption that the tasks cannot be solved by transcribing the image contents into text and then reasoning textually. The paper itself provides evidence that this assumption is insecure: Section 3.5 shows QVQ-72B-Preview explicitly follows an 'extract then reason' strategy, and Appendix B shows the prompt already includes the test input grid in text form. There is no text-only baseline in which the same demonstrations and test input are provided without images. I request such a control: run a strong text-only LLM (or a VLM with images replaced by faithful textual grids) on the same prompts and report accuracy. If a text-only model scores high, the benchmark primarily measures multimodal extraction plus text reasoning, and the paper's stronger claims about a new standard for visual reasoning would need to be substantially revised.","section":"Section 3.5 and Appendix B"},{"comment":"The headline result---that all models except Claude 3.7 Sonnet score below 20%---is based on a single sample per model at temperature 0.5, with no error bars or repeated runs. This is load-bearing for the central empirical claim, because low temperature does not eliminate sampling variance, especially for reasoning models with long generations. I ask the authors to report results over multiple runs (or use deterministic decoding) and provide per-model success rates with confidence intervals or at least raw trial counts, so that the comparison across models is not driven by a single lucky or unlucky sample.","section":"Section 3"},{"comment":"The paper characterizes EasyARC as 'extremely easy' and expects human performance to be 'probably significantly above' the H-ARC range of 73.3--77.2%, but no human baseline is collected on EasyARC itself. H-ARC is a harder benchmark, so extrapolating its human scores to EasyARC is not a substitute for actual measurement, especially because EasyARC's counting and fine-grained grid tasks may be nontrivial for humans too. I recommend either running a small human evaluation on a sample of EasyARC tasks or softening the 'extremely easy' and human-expectation claims.","section":"Section 7.3"},{"comment":"The benchmark's verifiability depends on each task having a unique intended transformation rule, but the paper does not describe any automated or formal check that generated samples have unambiguous solutions. Appendix B states that 'to guarantee uniqueness of the solution we make sure to include diverse demonstrations,' yet no procedure or validation is reported. Since exact-match evaluation treats any alternative valid rule as a failure, I ask the authors to either add a uniqueness verification step in the generator or explicitly discuss how demonstration diversity is checked.","section":"Section 2.2"}],"minor_comments":[{"comment":"The opening sentence of Section 7 is incomplete: 'In the following section we give an example for a successful and unsuccessful reasoning trace to give some insight into the' ends mid-phrase and should be finished or removed.","section":"Section 7"},{"comment":"There is a typo in the heading: 'unsucessful' should be 'unsuccessful'.","section":"Section 7.2"},{"comment":"The prompt text says 'columns are separated by column'; this should read 'separated by spaces' or another concrete delimiter, since the current phrasing is not clear.","section":"Appendix B"},{"comment":"The success-rate bar chart would be easier to interpret if the number of samples per model and per category were stated in the caption or in the text; currently the reader cannot assess the statistical weight of each bar.","section":"Figure 3"},{"comment":"The phrase 'the model perceives a blurry representation of images' is an interpretive claim; the evidence presented (better performance on easy tasks, worse on noise tasks) supports a coarse-feature hypothesis but not specifically 'blurry' perception. Recasting this as one possible hypothesis would avoid overstatement.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a potentially useful benchmark, but the central construct-validity question---whether EasyARC requires true visual reasoning rather than extraction plus text reasoning---needs to be answered with a text-only control. The one-sample evaluation and missing human baseline also need attention. I do not see a fatal flaw that would require rejection; the missing controls are within the scope of a revision, and the authors should be encouraged to add them. The paper would also benefit from a clearer statement of what is new relative to existing ARC-inspired VLM benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper up front. First, the asset is real: EasyARC is a procedurally generated, exact-match-verifiable benchmark with five task categories, progressive difficulty, and a clean interface to RL pipelines. Second, the central claim outruns the evidence: the authors say EasyARC measures true visual reasoning, but they never test whether a text-only model can solve the tasks from a transcription of the grids. The paper even supplies the test input as text in the prompt (Appendix B) and documents QVQ-72B-Preview succeeding by extract-then-reason (Section 3.5). That is internal evidence the visual pathway may not be load-bearing.\n\nWhat the paper does well: the generators are simple but effective, the evaluation protocol is stated plainly (one sample per model, temperature 0.5), and the failure-mode analysis is genuinely instructive — Claude 3.7 Sonnet's struggles with Counting Cells versus Drop One Color is exactly the kind of diagnostic signal a benchmark should produce. The observation that QVQ resorts to text-based reasoning is a useful qualitative result, even if it undermines the stronger claim. The design choices (grid sizes 3–10, difficulty sampling 0.5/0.35/0.15) are transparent and easy to reproduce.\n\nThe soft spots, in order of severity. One: no text-only baseline. Without feeding the same examples as text and measuring accuracy, you cannot conclude the failures come from missing visual reasoning. This is the decisive gap. Two: single-sample evaluation with no error bars or multiple runs at temperature 0.5 — the reported scores could shift noticeably with more samples. Three: no human baseline on EasyARC itself; the paper extrapolates from H-ARC, which is not the same task. Four: no comparison with existing ARC-derived or VLM reasoning benchmarks, so the novelty claim is underspecified. Five: the promised open-source dataset and code are not linked anywhere in the manuscript.\n\nNone of these are fatal, and most are straightforward to fix. The benchmark itself is a useful contribution, and the paper is honest about its own limitations, which I respect. The reader's stress-test note is correct: the text-only control is the missing piece, and the paper's own evidence makes that clear.\n\nRecommendation: this deserves a serious peer review, not a desk reject. The editor should send it out, and the referee should ask for a text-only baseline, multi-run evaluation, and a human baseline on EasyARC proper before publication. I would cite the benchmark if I were working on VLM evaluation, though only after the controls are in place.","headline":"EasyARC is a useful, honest benchmark evaluation with a clear design, but its central claim that it measures true visual reasoning needs a text-only control before it can stand.","tokens_in":7640,"tokens_out":1184,"would_cite":true,"duration_ms":12563,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EasyARC, a procedurally generated benchmark of simple grid puzzles, claims that state-of-the-art vision-language models mostly fail, with all but one scoring below 20 percent, a sign that true visual reasoning is unsolved.","keywords":["EasyARC","visual reasoning benchmark","abstract reasoning","vision-language models","grid transformations","procedural generation","test-time scaling","multimodal reasoning"],"falsifier":"Run the EasyARC test set with inputs supplied as textual row-by-row grids to a strong language-only model, and compare its accuracy with the vision-language results; if the text-only model matches or exceeds the vision-language models, then the tasks do not require true visual reasoning and EasyARC measures extraction-plus-text instead.","tokens_in":6718,"feed_emoji":"🧩","tokens_out":8273,"duration_ms":76865,"temperature":0.7,"pith_summary":"This paper introduces EasyARC, a procedurally generated benchmark of grid-transformation puzzles modeled on the ARC challenge, and argues that it measures true visual reasoning rather than visual information extraction. On tasks designed to be extremely easy, every tested vision-language model except one scored below 20 percent, a result the authors present as evidence that current models lack genuine visual cognition. The benchmark is designed to be verifiable, since outputs are exact grids, and scalable, since generators can produce unlimited examples with controlled difficulty levels. For a sympathetic reader, the payoff would be a practical testbed for studying test-time scaling and reinforcement learning in visual reasoning, with a clear quantitative target for improvement.","feed_headline":"All but one AI model fail EasyARC's simple grid puzzles","feed_subtitle":"A new verifiable benchmark shows models cannot reason step-by-step over images, not just extract them.","key_machinery":"The central object is the EasyARC generator suite: procedural generators that produce input-output grid pairs for five task classes, each with easy, medium, and hard variants defined by the skills they require. The mechanism that carries the argument is exact-output verification, because a model's answer must match the ground-truth grid exactly, so evaluation is unambiguous and does not depend on free-form answer scoring. A second mechanism is the multi-image task format, which presents several input-output demonstration pairs followed by a single test input and thereby forces the model to form, check, and refine a transformation hypothesis before applying it to the test grid.","core_discovery":"The paper's central claim is that EasyARC is the first benchmark aimed squarely at true visual reasoning in vision-language models, and that state-of-the-art models fail on it even though the tasks are simple by design. The claim rests on an evaluation of five task families (cross/star pattern completion, counting connected components, grid duplication with flips, dominant-side detection, and color-drop mappings) in which all tested models except the best-performing one scored below 20 percent. The paper further argues that this failure is diagnostic: models can often state the transformation rule in words but cannot execute it on the grid, and the one reasoning model examined at length solves tasks by extracting images into text rather than by reasoning visually. In the paper's view, these patterns show that fine-grained spatial reasoning over images is a missing capability in current vision-language models.","pith_inferences":["The paper never runs a text-only baseline, so its own observation of extract-then-reason behavior implies a direct control: if a strong language model solves the puzzles from textual grid transcriptions, EasyARC would be measuring multimodal extraction plus text reasoning rather than visual reasoning alone.","The near-zero image-only performance on the original ARC noted in the paper suggests EasyARC's usefulness may be as a calibrated probe: its difficulty levels could isolate individual visual subskills, one generator at a time, in future studies.","A natural extension the paper does not explore is to give the model intermediate visual scaffolding, such as highlighted regions or partially constructed outputs, to test whether step-by-step visual interaction, rather than a single pass, is what unlocks grid reasoning.","Given the human baseline on the much harder ARC, EasyARC tasks are likely near-ceiling for humans, which means the benchmark measures a model capability gap rather than task ambiguity."],"forward_implications":["EasyARC supplies a verifiable reward signal for reinforcement-learning pipelines, with difficulty levels that can be tuned to shape curriculum learning.","Test-time scaling on images becomes measurable: because outputs are exact grids, researchers can ask whether additional inference compute improves true visual reasoning, and the paper's failure analysis suggests it does not yet.","The observed failure modes imply that fine-grained perception, such as counting connected components, is a binding constraint even when a model can verbalize the rule.","Because new generators can be added without re-annotation, the benchmark can grow beyond its initial five task families while maintaining a clean separation of generated train and test samples.","If the benchmark's difficulty calibration is correct, it provides a sharper instrument than existing multimodal benchmarks for isolating where visual reasoning breaks down."],"supporting_citations":[{"why":"Defines the ARC challenge and its memorization-robust reasoning goal, and supplies the input-output grid task format that EasyARC adapts.","marker":"[4]"},{"why":"Categorizes ARC training tasks into seed tasks with generators and shows induction and transduction are complementary, directly inspiring EasyARC's generator design.","marker":"[10]"},{"why":"Provides the geometry-problem dataset whose extract-then-solve style is used to explain the reasoning model's failure mode on EasyARC.","marker":"[11]"},{"why":"Supplies human performance estimates on ARC, which the appendix uses to argue that EasyARC should be near-ceiling for humans.","marker":"[8]"},{"why":"Describes test-time training for abstract reasoning, motivating EasyARC's use as a testbed for test-time scaling and reinforcement learning.","marker":"[1]"}],"fun_headline_variants":["EasyARC: Most AI models fail simple visual grid puzzles","New benchmark EasyARC exposes weak visual reasoning in AI","EasyARC: Only one AI model passes its simple grid tests","Vision models fail EasyARC's step-by-step grid reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that EasyARC tasks genuinely require reasoning over images and cannot be solved by converting the example grids into text and then applying text-based reasoning, yet the paper provides no text-only baseline or perceptual control to demonstrate that this is the case.","fun_headline_variants_meta":{"raw":{"variants":["EasyARC: Most AI models fail simple visual grid puzzles","New benchmark EasyARC exposes weak visual reasoning in AI","EasyARC: Only one AI model passes its simple grid tests","Vision models fail EasyARC's step-by-step grid reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000479,"raw_usage":{"total_tokens":2318,"prompt_tokens":839,"completion_tokens":1479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":1412}},"tokens_in":455,"tokens_out":1479,"duration_ms":11483,"temperature":1.0,"reasoning_tokens":1412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:56.050722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the EasyARC test set with inputs supplied as textual row-by-row grids to a strong language-only model, and compare its accuracy with the vision-language results; if the text-only model matches or exceeds the vision-language models, then the tasks do not require true visual reasoning and EasyARC measures extraction-plus-text instead.","supporting_citations":[],"review_version":1}