{"id":"9fbb9a8d-f165-41d3-91d9-e91c77750418","arxiv_id":"2608.07584","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new verifiable benchmark shows VLMs struggle to recover complete decision problems from images, with large sensitivity to presentation and only modest gains from agent scaffolds.","lead":"COMPLEXITYWORLD is a 390-task benchmark that renders hidden decision problems as images and scores complete solutions with an executable verifier. It shows that current VLMs often cannot turn visual evidence into constraints-satisfying decisions: most models score below 40%, and the same problem can differ by up to 38 points across visual presentations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Verifier correctness and image-evidence completeness are not independently validated; the central bottleneck claim inherits this assumption.","rationale":"The reader identified verifier correctness and image-quality assumptions as the weakest point, and I agree. My independent read of the full text found that the pipeline section explicitly acknowledges that generation and verification share code, so the sensitivity test is not an independent validation of verifier correctness. This is the load-bearing assumption because the novelty of the benchmark is precisely that decisions are scored by executable verifiers; if the verifiers are wrong, every downstream number and the bottleneck interpretation inherit that error. The paper's internal evidence is substantial but behavioral: the 301/301 format/identifier rejections and 288/295 decision rejections show the verifiers are not trivially broken, but the 7 accepted altered outputs demonstrate that acceptance can be broader than intended, and the blinded QC is only sampled. I do not see an internal inconsistency that would justify REJECT; the paper is honest about its limitation and the conditional acceptance recommended by the reader is appropriate. The concrete independence test I propose would settle whether the concern lands. If the audit passes, the central numbers gain external support; if it fails, the benchmark needs a re-verification pass before the quantitative claims are trusted.","tokens_in":9667,"tokens_out":1333,"duration_ms":11496,"concrete_test":"Run an independent audit on a random sample of at least 50 of the 390 frozen tasks: have a human expert reconstruct the intended constraint set from the rendered image alone, compare it against the verifier code, and then attempt to produce a solution that satisfies a human reading of the image but is rejected by the verifier, and vice versa. Report the rate of verifier-human disagreement; if it exceeds roughly 5% of sampled tasks, the headline VAR numbers and bottleneck conclusion need re-quantification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that visual-to-decision bottlenecks persist because VAR is low under images and high under explicit structure. This claim depends on two premises: (1) every executable verifier correctly implements the intended constraints for all 390 tasks, and (2) each rendered image contains complete, unambiguous decision evidence without leakage. The paper's own Section 3.3 concedes that because generation and verification share task-specific code, the sensitivity test provides behavioral evidence rather than independent correctness validation. The 7 accepted 'decision changes' among 295 altered outputs show that verifiers sometimes accept unintended outputs, and the blinded image-quality check is described only at a high level with four reproduced examples; no release link is provided so independent checks are impossible. If a verifier is too permissive or an image omits a needed constraint, a low VAR could reflect an invalid task rather than a model limitation. Conversely, a leaked task label or answer would inflate GPT-5.6-Sol's 75.6%. The bottleneck conclusion is robust to some verifier looseness, but the exact headline numbers and the claim that 'one missed relation can invalidate an otherwise reasonable solution' require the verifier to enforce every displayed relation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces ComplexityWorld, a benchmark of 390 visual decision tasks drawn from 39 template-world pairs across 29 decision categories. Each task is generated from a hidden structured specification, rendered as a 1600-by-1000 image, and scored by an executable verifier that accepts any feasible solution, yielding a verifier acceptance rate (VAR, Eq. 1). Under direct inference, GPT-5.6-Sol reaches 75.6% VAR while the other three evaluated models remain below 40% in the observed sample. The authors find that replacing images with answer-free explicit-structure records raises Qwen3.7-Plus by 40.6 points and GPT-5.6-Sol by 21.4 points, that matched visual presentations produce gaps up to 38.3 points, and that agent scaffolds give smaller, model-dependent gains. They conclude that current VLMs exhibit a persistent visual-to-decision bottleneck that additional inference alone does not remove.","tokens_in":9863,"tokens_out":7046,"duration_ms":70221,"significance":"If the benchmark's verifiers and images are valid, this is a valuable contribution: it provides exact constraint-based evaluation with multiple acceptable solutions, controlled representation diagnostics, and a clear bottleneck finding with practical implications for VLM design. The paper is notably honest about its limitations, including the behavioral-only verifier check, the confounded presentation study, and the lack of causal attribution, which strengthens trust in its claims. The main risk is that the central metric inherits unverified assumptions about verifier correctness and image-evidence completeness; addressing these would make the leaderboard and bottleneck conclusions solid.","major_comments":[{"comment":"The sole validation of verifier correctness is the shared-code sensitivity test, which the authors themselves describe as behavioral evidence rather than an independent correctness validation. Because VAR in Eq. (1) is the only scoring metric and the bottleneck claim in Sections 5.1 and 7 depends on every verifier enforcing exactly the constraints intended for all 390 tasks, this is load-bearing. Please add an independent check of verifier behavior for at least a random stratified sample of tasks (e.g., human adjudication of accepted/rejected outputs against the rendered images, or an independently reimplemented reference verifier), report agreement, and specify whether the seven accepted decision changes from the stress test are genuinely feasible alternative solutions under the stated task constraints.","section":"Section 3.3 (Construction Pipeline; verifier stress test)"},{"comment":"Image completeness and leakage are equally load-bearing: a low VAR on a task whose image omits or ambiguously presents a constraint would reflect a flawed task, not a model limitation. The blinded image-quality check is described only at a high level, with four example images reproduced in the Supplementary Material, and no repository link is provided. Please report the sample size, selection criteria, inter-annotator agreement, the full set of inspected images (or a stable link), and per-template pass/fail results for the automated leakage checks. This is necessary to rule out the alternative that some tasks do not present complete, unambiguous decision evidence.","section":"Section 3.3 (Rendering and QC; blinded check)"}],"minor_comments":[{"comment":"Report 95% confidence intervals for the main VARs; for Qwen3.7-Plus, 154/390 = 39.5% has an approximate interval of [34.6%, 44.4%], so the abstract's claim that all non-GPT models 'remain below 40%' is not statistically robust.","section":"Table 1 and Abstract"},{"comment":"The paper states that the frozen benchmark and construction pipeline will be released, but no repository URL, DOI, or other stable artifact link is given; without it, the reproducibility claims cannot be verified.","section":"Section 3.3 and Section 7"},{"comment":"Per-template results appear in Figure 3, but Table 3 aggregates them; add per-template confidence intervals and clarify that the Holm-adjusted McNemar correction applies to the two presentation comparisons within each template.","section":"Section 5.3 (Table 3)"},{"comment":"The metric name is rendered inconsistently as 'V AR' in the abstract and several places; use 'VAR' consistently.","section":"Throughout"},{"comment":"The phrase 'answer-free explicit-structure record' is potentially confusing because the record contains all decision-relevant constraints; consider renaming it to 'solution-free structured record' to clarify that only the reference solution and verifier are excluded.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is promising and the authors have disclosed their main limitations honestly. However, the central validity depends on verifier correctness and image-evidence completeness, both of which are currently validated only behaviorally or at a high level. I recommend asking for an independent verifier validation, a fuller QC report, and a stable artifact link before acceptance. The paper is within scope for a journal that publishes benchmark contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the takeaway: this is a useful benchmark with a real diagnostic finding, but the exact scores should be treated as provisional until the verifiers and images get independent validation. The authors combine hidden structured specifications, an executable verifier that accepts any feasible solution, matched explicit-structure records, and equivalent-presentation controls. That's a thoughtful design. The central result—models do much better when the same decision problem is given as a structured record than as an image—is robust enough to survive some verifier looseness, and the 0/234 no-evidence control plus the 596-output sensitivity test show good discipline.\n\nThe main soft spot is the load-bearing premise: each verifier correctly implements the intended constraints, and each rendered image contains complete, unambiguous evidence. The paper concedes in Section 3.3 that its validation is behavioral—generation and verification share task-specific code. Seven of 295 altered decision outputs were accepted by verifiers; those are described as feasible alternatives, but without the release we cannot check whether they are true alternatives or loopholes. The blinded image-quality check is described only at a high level. The headline 75.6% vs. below-40% gap is big, so the bottleneck conclusion likely stands, but the exact numbers and per-category comparisons need the release to be credible.\n\nTwo smaller issues: the main table lacks confidence intervals, and the equivalent-problem study changes several visual factors together, so it establishes presentation sensitivity rather than isolating a cause. Those limits are disclosed, but they should be remembered when citing.\n\nThe paper is well-scoped, cites relevant work, and does not overclaim: it is a benchmark plus diagnostics, not a training study. It deserves a serious referee. My recommendation: send it to review, but ask for the release and for an independent audit of a sample of verifiers before the exact scores are relied on.","headline":"A solid benchmark with a real diagnostic result; exact scores are provisional until verifiers and images get independent validation.","tokens_in":10342,"tokens_out":3629,"would_cite":true,"duration_ms":33090,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ComplexityWorld presents evidence that current vision-language models can recognize images but often cannot turn visual evidence into a complete, constraint-satisfying decision: on 390 verifier-checked tasks, only GPT-5.6-Sol exceeds 40%…","keywords":["vision-language models","visual decision making","verifiable benchmark","executable verifier","constraint satisfaction","verifier acceptance rate","visual-to-decision bottleneck","representation sensitivity"],"falsifier":"Present each model with the same task twice: once as the rendered image and once as a faithful textual transcription of the image's entities and constraints, then compare verifier acceptance rates; if the gap vanished, the bottleneck would be visual recovery, while a persistent gap would point to solution construction. In parallel, check the verifiers themselves by independently re-implementing a sample from the released task specifications and confirming agreement on the published 596 altered outputs, since the reported VAR is only meaningful if the verifiers enforce the intended constraints.","tokens_in":9492,"feed_emoji":"🧩","tokens_out":11890,"duration_ms":95362,"temperature":0.7,"pith_summary":"ComplexityWorld tests whether vision-language models can use an image as the sole evidence for a complete decision—a route, schedule, placement, or assignment whose parts must jointly satisfy global constraints—and return a solution a hidden verifier accepts. The paper builds 390 tasks spanning 39 visual worlds and 29 decision categories, each generated from a hidden structured specification and rendered as a scene. Under direct inference every evaluated model except GPT-5.6-Sol stays below 40% verifier acceptance rate, while GPT-5.6-Sol reaches 75.6%. Giving the same instances as explicit structured records lifts performance by 21 to 41 points, and equivalent problems in different visual presentations differ by up to 38.3 points. The paper concludes that a persistent visual-to-decision bottleneck remains: current models are much better at solving an explicit problem than at recovering the problem from pixels, and extra inference via agent scaffolds gives only smaller, model-dependent gains.","feed_headline":"New benchmark: 3 of 4 vision models fail complete visual decisions","feed_subtitle":"On 390 verifier-checked tasks, recognizing what an image shows is not enough to construct a feasible decision.","key_machinery":"The load-bearing machinery is a generate-render-verify pipeline. A seeded generator creates a solvable problem from a hidden structured specification; a visual-world adapter renders it as a 1600-by-1000 image using domain-specific names, icons, and layouts; and a task-specific verifier program checks every constraint, accepting any feasible solution rather than one serialized reference answer. The model sees only the image, instruction, and output schema, so the image is the only source of decision evidence. The same pipeline produces matched diagnostics: answer-free explicit-structure records hold the decision problem and verifier fixed while removing the image, and multiple adapters render identical hidden specifications in different visual forms. The primary metric is verifier acceptance rate (VAR), the fraction of returned decisions that parse, refer to valid entities, and satisfy the hidden verifier.","core_discovery":"The central claim is that reliable visual decision making requires two abilities that current VLMs do not have in equal measure: recovering the decision-relevant structure from visual evidence, and constructing a globally consistent solution from that structure. The paper's evidence is the gap between conditions: direct image inference yields low verifier acceptance rates (75.6% for GPT-5.6-Sol, 39.5% for Qwen3.7-Plus, 32.3% for Gemini-3.5-Flash, 24.4% for MiMo-v2.5), but replacing the image with an answer-free structured record of the same instances improves Qwen by 40.6 points and GPT by 21.4 points, and identical underlying problems presented in three matched visual forms produce gaps of up to 38.3 points. The same pattern appears across the eight constraint-signature families: strong performance on one family does not predict performance on another. The paper therefore concludes that strong solution construction in one representation does not imply reliable visual decision making across representations—success depends on how decision information is organized and presented, not just on model scale or reasoning effort.","pith_inferences":["A testable extension the paper does not run: train on the frozen main panel and evaluate on the other available templates from the released 113-template library, holding out whole visual-world adapters, to measure out-of-distribution generalization of any verifier-trained model.","If the bottleneck is as persistent as measured, interface design becomes a first-order lever: presenting pairwise relations explicitly (adjacency tables, cost tables, traversability graphs) may be a cheap deployment fix, though the paper does not isolate which visual factor causes the gain because several change together.","The benchmark's distinction between decision categories and constraint-signature families could serve as a diagnostic grid for future model releases: reporting VAR per family would show whether a model's improvement comes from better visual extraction or better constraint construction, since the two map to different failure profiles.","One could turn the presentation-sensitivity result into a robustness metric: a model that solves the same problem in several renderings is more reliable than one that peaks in a single format, and the paper's all-three-success statistic is a step in that direction."],"forward_implications":["Success on perception-heavy benchmarks should not be read as success at decision making; a model can answer visual questions while failing to produce a verifier-accepted complete solution.","Benchmark scores are presentation-sensitive: reporting one rendering per problem can overstate or understate capability, because reorganizing identical decision information moved acceptance by tens of points.","Scaling inference alone is unlikely to close the gap: the two agent scaffolds tested improved their base models by 7.4 and 3.8 points, with the smaller gain statistically indistinguishable from zero.","Explicit structured inputs are a strong intervention, so hybrid systems that transcribe images into structured problem records before solving may outperform end-to-end VLMs on constrained decisions.","The released generators, frozen tasks, and verifiers give a reproducible target for training on verified outcomes, with the paper's suggested discipline of withholding whole visual-world adapters when testing generalization."],"supporting_citations":[{"why":"Supplies the dynamic multimodal benchmark of NP-hard reasoning that motivates generated, verifiable tasks.","marker":"Fan et al. 2024"},{"why":"PuzzleBench's programmatic generation and dynamic evaluation are the methodological foundation for ComplexityWorld's fresh instances and global constraint checking.","marker":"Zhang et al. 2025"},{"why":"MM-OptBench grounds multimodal optimization modeling in solvers, the same executable-evaluation pattern the verifier uses.","marker":"Li et al. 2026b"},{"why":"ORAgentBench supplies the end-to-end operations-research agent setting that motivates the agent-scaffold comparison.","marker":"Li et al. 2026a"},{"why":"ReAct's reasoning-and-acting loop is the agent paradigm behind the two scaffolded configurations the paper evaluates.","marker":"Yao et al. 2023"},{"why":"Provides the NP-completeness framework the paper uses to justify the split between hard global construction and efficient verification.","marker":"Karp 1972"}],"fun_headline_variants":["New benchmark exposes VLMs' visual-decision bottleneck","Seeing isn't deciding: VLMs fail on constraint-heavy visual tasks","ComplexityWorld: Most VLMs can't solve visually grounded decisions","Visual reasoning? Most VLMs score under 40% on new test","New benchmark: VLMs see but can't make complete visual decisions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that the verifiers enforce exactly the constraints the tasks intend and that every image shows everything needed without revealing the answer; the paper's own checks of this are behavioral (altered outputs and visual inspection), not an independent correctness proof.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark exposes VLMs' visual-decision bottleneck","Seeing isn't deciding: VLMs fail on constraint-heavy visual tasks","ComplexityWorld: Most VLMs can't solve visually grounded decisions","Visual reasoning? Most VLMs score under 40% on new test","New benchmark: VLMs see but can't make complete visual decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001589,"raw_usage":{"total_tokens":6353,"prompt_tokens":980,"completion_tokens":5373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":5284}},"tokens_in":596,"tokens_out":5373,"duration_ms":36174,"temperature":1.0,"reasoning_tokens":5284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:31:42.183814+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present each model with the same task twice: once as the rendered image and once as a faithful textual transcription of the image's entities and constraints, then compare verifier acceptance rates; if the gap vanished, the bottleneck would be visual recovery, while a persistent gap would point to solution construction. In parallel, check the verifiers themselves by independently re-implementing a sample from the released task specifications and confirming agreement on the published 596 altered outputs, since the reported VAR is only meaningful if the verifiers enforce the intended constraints.","supporting_citations":[],"review_version":1}