{"id":"5c544dba-ca0a-4561-b015-43d881b507bf","arxiv_id":"2512.11995","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"V-REX shows that VLMs' multi-step visual exploration can be measured separately as planning (choosing sub-questions) and following (answering them), and that planning is the key bottleneck.","lead":"V-REX is a new benchmark that tests how well vision-language models explore images step by step: it separates the ability to pick useful next questions (planning) from the ability to answer those questions (following). Evaluated on 32 models, it finds that bigger models plan better and that guiding models with sub-questions improves final answers, though open-ended exploration remains a major weakness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Planning accuracy may measure agreement with GPT-5/Qwen3-VL-32B's notion of helpfulness rather than intrinsic planning ability; the cross-model comparisons and 'planning bottleneck' claim depend on distractor calibration being model-independent.","rationale":"Following the reader's conditional verdict, I focused on the weakest point in the argument for V-REX as a disentangled measure of Planning and Following. The Planning task's ground truth is a single human-authored chain, and its difficulty is calibrated by GPT-5-generated, Qwen3-VL-32B-filtered distractors. If 'helpfulness' is not model-invariant, Planning scores are not directly comparable across VLMs. This is load-bearing because the paper's most novel empirical findings (variance in Planning across model families, planning as bottleneck, recovery asymmetry) all depend on the Planning metric meaning the same thing for every model. The paper includes strong safeguards for the gold chains (human cross-verification, helpfulness/order checks) but none for distractor calibration. A cross-selector robustness check would resolve whether the benchmark measures an intrinsic ability or an agreement artifact. In the meantime, the correct disposition is conditional acceptance: the benchmark is valuable and the protocol is transparent, but the key empirical claims require this validation before they can be fully trusted. This matches the reader's verdict; I agree with the identified weakest assumption and sharpen it with a concrete test.","tokens_in":24753,"tokens_out":10049,"duration_ms":103427,"concrete_test":"Construct a second version of the Planning task for a random subset of ~50 samples (covering all 4 categories): use the same GPT-5 candidate distractors, but select the final distractor set using a different strong VLM, e.g., InternVL3-38B instead of Qwen3-VL-32B. Evaluate the same panel of six models spanning families/sizes (GPT-5, o3, InternVL3-38B, Qwen3-VL-8B, InternVL2.5-8B, LLaVA-OV-1B) on both versions under identical prompts. Compute per-model Planning accuracy and the Spearman rank correlation between the two versions. If any model's score shifts by more than 5 percentage points, or the rank correlation falls below 0.8, Planning scores depend on the selector model and are not model-independent.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.2 defines Planning as selecting the most helpful next question, and the Planning metric (Eq. 3) scores a model by how often it picks the gold question among LLM-generated distractors. Appendix 9.1 reveals that these distractors are generated by GPT-5 and filtered/selected by Qwen3-VL-32B-Instruct, which automatically identifies the chain that causes the highest confusion for that selector. Thus the difficulty of the Planning task is calibrated to the inductive biases of two specific LLMs. A question that GPT-5/Qwen3-VL find unhelpful need not be unhelpful to a differently-trained VLM: it may direct attention to a cue that the model otherwise ignores, making it genuinely useful in that model's reasoning economy. The paper's headline finding — that Planning varies more across models and is a primary differentiator (Figure 6, Findings 2 and 5) — would then be an artifact of how well each model's question-selection preferences align with those of the LLMs that built the distractors, rather than a model-independent measure of planning skill. No control is reported: there is no human rating of distractor helpfulness, no alternative distractor set, and no analysis of per-item choice distributions to check whether any distractor is commonly chosen by strong models. Without such validation, the construct validity of the Planning dimension is unestablished, which undermines the central claim of disentangling Planning and Following.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces V-REX, a benchmark of 702 samples and 2,504 questions spanning four reasoning categories (Deduction, Guessing, Navigation, Retrieval) and 15 scenarios. Each sample is annotated with a human-constructed Chain-of-Questions (CoQ) of 2–6 steps. The benchmark disentangles multi-step exploratory visual reasoning into two tasks: Planning, where the model selects the most helpful next sub-question among LLM-generated distractors (Eq. 3), and Following, where the model selects the correct answer among distractors at each step (Eq. 4). The authors evaluate 32 VLMs and report that CoQ hints generally improve final-answer accuracy, that performance scales with model size, that Planning and Following both correlate with overall performance, that small models favor Following over Planning, and that models recover better from failed Planning than from failed Following.","tokens_in":25158,"tokens_out":5043,"duration_ms":52896,"significance":"If the proposed metrics are construct-valid, V-REX is a useful contribution: it is one of the first benchmarks to evaluate exploration in question space separately from answer-following, it uses a controlled MCQ protocol, and it includes a broad model suite with both open and proprietary VLMs. The explicit disentanglement of Planning and Following, together with the detailed taxonomy and the planned public release, could support more diagnostic evaluations of multi-step visual reasoning. However, the core Planning measure depends on the quality and neutrality of the distractor questions, and the current paper does not yet provide sufficient evidence that those distractors are model-independent. The central findings, especially the planning/following asymmetry and the claim that Planning is the primary differentiator, would be stronger with explicit construct-validity checks.","major_comments":[{"comment":"The Planning task's distractor questions are generated by GPT-5 and then filtered by Qwen3-VL-32B-Instruct, which 'automatically identify[ies] the chain that causes the highest confusion.' GPT-5 is itself one of the evaluated models in Table 1. Without a human rating of distractor helpfulness, an alternative distractor set, or per-item choice distributions, the Planning metric risks measuring agreement with the two LLMs that constructed the distractors rather than a model-independent planning ability. This is load-bearing because Findings 2 and 5 and the claimed planning/following disentanglement rely on comparing Planning scores across models. Please add construct-validity evidence: e.g., expert labels of distractor helpfulness, choice distributions showing whether strong models systematically prefer any distractor, or a robustness run with a differently generated distractor set.","section":"§4.2.2 and Appendix 9.1, Eq. (3)"},{"comment":"The text says that 'the accuracy drops more sharply in Figure 11 than in Figure 10, which further verifies that models are generally more robust to wrong following steps than wrong planning steps.' This is inverted: Figure 11, the wrong-Following condition, drops to 0.000 at five errors, while Figure 10, the wrong-Planning condition, remains at 0.429; Table 2 also shows higher recovery from failed Planning than from failed Following. The appendix sentence contradicts the main text's Finding 5 and should be corrected, with the stepwise analysis re-interpreted accordingly.","section":"Appendix 12, Figures 10–11, and Table 2"}],"minor_comments":[{"comment":"The performance-changing ratio (AccCoQ − Acc)/Acc is reported without confidence intervals or per-category sample sizes. Since some categories are small (e.g., Retrieval), adding variability measures would help assess the strength of Finding 1.","section":"§5.2 and Table 3"},{"comment":"The number of MCQ options varies across steps, so reporting chance-level accuracy or a random-choice baseline would aid interpretation of absolute Planning and Following scores.","section":"§4.3 and Table 1"},{"comment":"The generation prompts include an 'unhelpfulness_score', but the paper does not state how this score is used in the filtering pipeline. Please clarify whether it influences the final distractor selection.","section":"Appendix 9.2"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about LLM-generated planning distractors is the main reason for major revision. The paper is otherwise well specified, but the construct validity of the Planning metric needs explicit evidence before the headline claims can be fully trusted. The Appendix 12 inversion is a clear error that should be fixed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — V-REX is a genuinely useful new benchmark. It formalizes multi-step visual reasoning as a Chain-of-Questions and cleanly separates Planning (choosing which question to ask next, with ground-truth answers provided) from Following (answering fixed questions in the chain). That separation is enforced by the MCQ protocol, not by an LLM judge, which is a real step up from prior intermediate-step evaluations. The dataset is substantive: 702 samples, 2,504 questions, 15 scenarios, human-built chains with two rounds of cross-verification, and 32 models evaluated. The main findings — CoQ hints mostly improve final accuracy, performance scales with size, and following is the more stable skill — are plausible and honestly presented.\n\nThe weak spot is the planning distractor construction. Distractors are generated by GPT-5 and then filtered by Qwen3-VL-32B to select the chain that causes the highest confusion for that selector. GPT-5 is also one of the evaluated models. So the planning difficulty is calibrated to two specific LLMs' notion of unhelpfulness. A question that is a distractor for them may actually engage a different model's reasoning, which makes the planning scores partly a measure of agreement with GPT-5/Qwen preferences, not a pure measure of planning ability. The paper reports no human ratings of distractor helpfulness and no alternative distractor set. That weakens the central claim that planning is the primary differentiator. The following task uses human-written distractors and is on much firmer ground.\n\nThere's also an internal contradiction on failure recovery: Finding 5 says models recover better from failed planning than failed following, and the data in Figures 10–11 support that. But the Appendix text says the opposite ('models are generally more robust to wrong following steps than wrong planning steps'), which is a typo that should be fixed.\n\nMinor: no dataset or code licensing details in the paper, just a GitHub link.\n\nWho this is for: anyone building or evaluating VLMs that need to reason over multiple steps. The benchmark could become a standard tool if the distractor calibration issue is addressed, at minimum by adding human validation and showing the planning results are stable across alternative distractor sets. It deserves a serious referee and, after revision, publication. I'd send it to review.","headline":"A solid new benchmark for exploratory visual reasoning that cleanly separates planning from following, but the planning metric is only as clean as the GPT-5/Qwen-generated distractors and needs extra validation before the headline claims are fully trusted.","tokens_in":25574,"tokens_out":2674,"would_cite":true,"duration_ms":26574,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Exploratory visual reasoning can be split into planning and following, and current VLMs are much better at following.","keywords":["exploratory visual reasoning","Chain-of-Questions","V-REX benchmark","vision-language models","planning vs. following","multi-step visual reasoning","intermediate-step evaluation","visual question answering"],"falsifier":"Regenerate the Planning distractor set using a different LLM generator (or human annotators) and drop the model-based filtering step; then compare model Planning rankings to the paper's. If rankings shift substantially, the scores reflect distractor provenance rather than planning skill.","tokens_in":24694,"feed_emoji":"🕵️","tokens_out":7633,"duration_ms":67677,"temperature":0.7,"pith_summary":"V-REX is a benchmark for multi-step visual reasoning: 702 images-and-questions tasks, 2,504 sub-questions, spanning deduction, guessing, navigation, and retrieval. It represents each solution as a Chain-of-Questions — a sequence of sub-questions and answers — and scores models separately on Planning (choosing the useful sub-question) and Following (answering it). The paper's central claim is that this decomposition is meaningful and measurable, and that across the 32 models tested, Planning is the weaker, more variable skill. The benchmark also shows that giving a model a good chain of hints usually improves its final answer, while errors in answering sub-questions are harder to recover from than errors in choosing them. The reader should care because most visual-reasoning benchmarks only grade the final answer, leaving the exploratory process that leads to it unevaluated.","feed_headline":"Planning, not answering, is the weak step for visual AI","feed_subtitle":"Guided by question chains, most vision-language models answer better—but choosing the next question trips them up.","key_machinery":"The central object is the Chain-of-Questions (CoQ), a manually constructed QA chain of 2–6 interconnected sub-questions leading to a final answer. The benchmark turns each chain into multiple-choice probes: at each step, Planning presents the model with the ground-truth question alongside unrelated 'distractor' questions and asks it to pick the helpful one (ground-truth answers are given, so only question choice is tested); Following presents the ground-truth question with plausible wrong answers and asks the model to pick the correct one (ground-truth questions are given, so only answer choice is tested). This decoupling is what allows the two skills to be scored independently.","core_discovery":"Exploratory visual reasoning is not one ability but two, and a finite multiple-choice Chain-of-Questions can measure them separately. Ground-truth answers are supplied during Planning and ground-truth questions during Following, isolating question-selection from answer-execution. Across 32 VLMs, both abilities scale with model size and correlate with end-to-end accuracy (Pearson 0.858 Planning, 0.948 Following), but Planning is the scarcer skill: same-size models vary more in Planning, smaller models are better at Following, and models recover better from bad plans than bad answers. The bottleneck in multi-step visual reasoning is asking the right next question.","pith_inferences":["If Planning is the genuine bottleneck, training objectives that reward choosing a useful next question (for example, preference learning over question choices) might improve open-ended, agent-like visual reasoning more than training on final-answer accuracy alone.","The benchmark's Planning difficulty is defined by GPT-5-generated distractors filtered by Qwen3-VL-32B; a replication with distractors from a different generator (or human-written distractors) would test whether planning scores measure a model-general skill or agreement with those generators' notion of helpfulness.","The same finite-option CoQ protocol could be applied to audio, video, or embodied navigation tasks, where exploration is naturally sequential; the key constraint is that each step must be reducible to a small menu of candidate questions/answers.","The asymmetry in recovery (models recover better from failed Planning than failed Following) suggests an actionable design: in deployed systems, let the user or an external planner pick sub-questions, and spend verification effort on the answers."],"forward_implications":["Providing a CoQ chain as a hint improves most models' final-answer accuracy, so structured intermediate questions can serve as test-time scaffolding for VLMs.","Planning remains the larger differentiator at every scale: models of the same size vary far more in Planning than in Following, so planning ability is not just a matter of parameter count.","Both intermediate skills contribute to end-to-end ability, with Following the stronger correlate; a model strong at both should beat a model strong at only one even when final answers alone look similar.","Errors in Following propagate sharply into the final answer, whereas wrong Planning steps are often recoverable, meaning verification effort on answers matters more than on question choices.","Final-answer-only benchmarks systematically miss where VLMs fail; V-REX shows the failure often sits in the question space, not the answer space."],"fun_headline_variants":["Question selection is the true bottleneck for visual AI","Planning beats answering as the hard part of visual reasoning","Why visual AI needs better question-asking, not answering","Benchmark reveals: planning questions is the weak link in VLMs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the questions labeled 'distractors' in the Planning task are genuinely unhelpful to every evaluated model; they are generated by GPT-5 and filtered by Qwen3-VL-32B (Appendix 9.1), so a model's Planning score partly measures how closely its notion of a useful next question matches those two LLMs, not just its intrinsic planning skill.","fun_headline_variants_meta":{"raw":{"variants":["Question selection is the true bottleneck for visual AI","Planning beats answering as the hard part of visual reasoning","Why visual AI needs better question-asking, not answering","Benchmark reveals: planning questions is the weak link in VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1151,"prompt_tokens":784,"completion_tokens":367,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":528,"tokens_out":367,"duration_ms":4099,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T16:42:20.274466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the Planning distractor set using a different LLM generator (or human annotators) and drop the model-based filtering step; then compare model Planning rankings to the paper's. If rankings shift substantially, the scores reflect distractor provenance rather than planning skill.","supporting_citations":[],"review_version":1}