{"id":"8794ca49-7bbf-4ecb-a7e1-d9f78510731d","arxiv_id":"2607.06522","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"VAORA aligns VLM chain-of-thought reasoning with visual scene observations and post-action outcomes via structured symbolic rewards, achieving cross-task and cross-environment generalization on physical reasoning benchmarks.","lead":"This paper introduces VAORA, a reward design that trains vision-language models to ground their chain-of-thought reasoning in visual outcomes of their actions, improving generalization on physical reasoning tasks. A smart generalist might read it to understand how aligning a model's stated reasoning with observable physical consequences can reduce hallucination and improve cross-task transfer in AI agents.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Cross-task generalization is validated only on 10/25 PHYRE task types where the first collision is the key interaction; the three evaluation splits also share overlapping task types, weakening the stability claim.","rationale":"The reader's CONDITIONAL verdict is appropriate. The first-collision restriction is a genuine scope limitation on the central generalization claim, disclosed only in the appendix. The split overlap further weakens the robustness argument. However, these issues do not invalidate the contribution: the ablation (Table 5) demonstrates each reward component contributes meaningfully, the reward breakdown analysis (Table 4) provides genuine mechanistic insight, and the cross-environment and CRAFT VQA results, while involving protocol modifications, still show consistent improvements. The paper's reward design is well-motivated and the experimental gains are real within the tested scope. The condition for full acceptance should be: (1) discuss the task-type restriction in the main text, (2) acknowledge split overlap, and (3) clarify the Virtual Tool protocol changes' impact on baseline comparisons. The concern is about scope of claims, not correctness of results within that scope.","tokens_in":27092,"tokens_out":4325,"duration_ms":320559,"concrete_test":"Construct a new PHYRE cross-task split where the 5 held-out task types are drawn from the 15 excluded multi-step collision task types (e.g., types 00004, 00005, 00007, 00008, 00010). Train VAORA on the remaining 20 types and evaluate Pass@1/3/5. If performance drops below the DQN expert baseline or below the levels seen on first-collision splits, the generalization claim should be explicitly scoped to first-collision-sufficient tasks in the main text, not just the appendix.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the most load-bearing concern. The collision reward r_C (Eq. 4) only models the first object the red ball collides with, as disclosed in Appendix B.2: 'the generalization capability we aim to evaluate is meaningful only for tasks in which the key interaction is itself the first collision event.' This restricts the held-out testing pool to 10 of 25 PHYRE task types. The central claim—that VAORA enables cross-task generalization surpassing the DQN expert—is therefore validated only on tasks matching this structural constraint. For the 15 excluded task types requiring multi-step collision chains, r_C provides no training signal and the generalization claim is untested.\n\nA compounding issue not noted by the reader: the three splits are not independent. Task types 00000, 00001, 00002, and 00003 each appear in two or more of the three testing sets (Appendix B.2). The paper frames this as verifying 'stability across different held-out task distributions,' but the substantial overlap means the three splits test largely the same task structures with different training partitions. This provides weaker evidence of robustness than three genuinely disjoint splits would. The DQN circularity (DQN as both reward source and baseline) is mitigated by the ablation showing +EG alone underperforms +EG+VAORA, so I do not consider it the primary concern. The first-collision restriction and split overlap together mean the headline generalization claim is narrower than presented in the main text.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"The paper introduces VAORA (Visual Action Outcome Reasoning Alignment), a dual-reward framework for training VLMs on interactive physical reasoning tasks. The approach combines a Visual Alignment Reward (grounding reasoning to the initial scene) with a Visual-Action Alignment Reward (grounding reasoning to post-action visual outcomes), using a pre-trained DQN expert to provide smooth, dense success-probability estimates that stabilize RL training. Experiments span three settings: cross-task generalization on PHYRE (held-out task types), cross-environment zero-shot transfer from PHYRE to Virtual Tool, and broader physical understanding on CRAFT VQA. The central claim is that an 8B model trained solely on PHYRE with VAORA surpasses the DQN expert on unseen tasks, matches frontier closed-source models on Virtual Tool, and improves causal reasoning on CRAFT VQA.","tokens_in":27937,"tokens_out":1433,"duration_ms":282417,"significance":"The paper addresses a genuine gap: connecting CoT reasoning in VLMs to physical outcomes via reward signals, rather than relying on SFT imitation or sparse task-success RL. The dual-reward decomposition (grounding, placement, collision) is well-motivated, and the reward breakdown analysis (Table 4) and component ablation (Table 5) provide useful mechanistic insight into why each reward matters. The cross-environment transfer result (Table 2) and the CRAFT VQA gains (Table 3) are notable demonstrations that the learned reasoning extends beyond the training simulator. The approach is falsifiable: the ablation cleanly isolates each reward component, and the three-split protocol provides replicability. The commitment to releasing datasets, checkpoints, and code is a strength.","major_comments":[{"comment":"Appendix B.2 discloses that the collision reward r_C (Eq. 4) models only the first object the red ball collides with, restricting the held-out testing pool to 10 of 25 PHYRE task types. The central cross-task generalization claim—surpassing the DQN expert on unseen PHYRE tasks—is therefore validated only on tasks where the key physical interaction is the first collision event. For the 15 excluded task types requiring multi-step collision chains, r_C provides no training signal and the generalization claim is untested. This is load-bearing for the headline claim in §1 ('surpassing the DQN expert on unseen PHYRE tasks') and should be explicitly qualified in the main text (not only in the appendix). The paper should state the restriction in §4.2 and frame the claim as applying to first-collision-dominated tasks.","section":null},{"comment":"The three testing splits (Appendix B.2) share substantial overlap: task types 00000, 00001, 00002, and 00003 each appear in two or more splits. The paper frames the three splits as verifying 'stability across different held-out task distributions,' but the overlap means the splits test largely the same task structures with different training partitions. This provides weaker evidence of robustness than three genuinely disjoint splits would. The paper should either (a) acknowledge this overlap and reframe the stability claim accordingly, or (b) construct at least one fully disjoint split from the 10 eligible task types to demonstrate that performance is not an artifact of particular task-type selection.","section":null},{"comment":"§3.2, Eq. (2): the symbolic extraction functions phi(r) and psi(o)/psi(v) are described as 'general' but are instantiated only for the PHYRE domain (3x3 spatial grid, first-collision extraction). The cross-environment transfer to Virtual Tool (Table 2) and the CRAFT VQA results (Table 3) presumably reuse the same phi/psi, but the paper does not describe how these functions are adapted for environments with different object types, action spaces, or physics. Since the cross-environment claim is a central contribution, the paper should clarify whether phi/psi are reused as-is or adapted, and what the adaptation entails.","section":null}],"minor_comments":[{"comment":"Table 1: the DQN-expert row shows Testing Set 2 Pass@1 = 0.260, while +EG+VAORA achieves 0.198 on the same split. The claim of 'surpassing the DQN expert across nearly all metrics' should be qualified to note that on Testing Set 2, VAORA does not surpass the DQN on Pass@1.","section":null},{"comment":"§4.3, Table 2: the Virtual Tool evaluation protocol is modified (tool selection removed, colors remapped). While these adaptations are reasonable, they change the benchmark's original difficulty. The paper should note this explicitly in the main text, not only in Appendix A.2.","section":null},{"comment":"Table 3: the CRAFT VQA gains over +EG are modest (Overall: 44.56 to 46.06, a 1.5-point improvement). The paper should discuss whether this is within noise bounds, given that only 1,000 questions per category are sampled.","section":null},{"comment":"§3.3: the reward composition r_DQN * (1 + r_P + r_C) is described, but the choice of the additive form (1 + r_P + r_C) versus a multiplicative or normalized form is not justified. A brief note on why this particular composition was chosen, or whether alternatives were tried, would strengthen the method section.","section":null},{"comment":"Appendix B.4, Eq. (8): the penalty constraint w_c * p_c = 0.1 is presented as ensuring uniform penalty across reward types. This is a reasonable design choice but is presented without justification for why 0.1 is the right scale. A brief sensitivity note would strengthen the method.","section":null},{"comment":"The paper uses future-dated references (e.g., Qwen3-VL, Gemini 3.1, GPT-5.4, Claude Sonnet 4.6). If these are real models available at submission time, this is fine; if any are hypothetical or projected, this should be disclosed.","section":null},{"comment":"Figure 2 caption references '(a)' and '(b)' components but the figure layout could be clearer. A brief label on each subfigure would improve readability.","section":null}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern about the first-collision restriction and split overlap is well-founded and is the primary reason for the major_revision recommendation. The DQN circularity concern (DQN as both reward source and baseline) is partially mitigated by the ablation showing +EG alone underperforms +EG+VAORA, so I do not consider it a blocking issue. The cross-environment and CRAFT VQA results are interesting but depend on the generality of phi/psi, which needs clarification. If the authors can (1) qualify the generalization claim in the main text, (2) address the split overlap, and (3) clarify the phi/psi adaptation for non-PHYRE environments, the paper should be publishable."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. The referee correctly identifies three issues: (1) the collision reward's first-collision limitation restricts the held-out testing pool to 10 of 25 PHYRE task types, which is not stated in the main text; (2) the three testing splits share substantial task-type overlap, weakening the robustness claim; and (3) the symbolic extraction functions phi/psi are described as general but instantiated only for PHYRE, with no explanation of how they are adapted for Virtual Tool and CRAFT VQA. We agree with all three points and will revise the manuscript accordingly. Below we address each comment in detail.","responses":[{"response":"The referee is correct on both the factual point and its implications for the headline claim. The restriction to 10 of 25 task types is currently disclosed only in Appendix B.2, while the main text (§1, §4.2) states the generalization claim without qualification. This is a genuine gap between what we claim and what we evaluate. We will revise the manuscript as follows: (1) In §4.2, we will explicitly state that the held-out testing pool is restricted to the 10 task types where the key physical interaction is the first collision event triggered by the red ball, and that the remaining 15 task types requiring multi-step collision chains are excluded because r_C provides no training signal for them. (2) In §1, we will qualify the headline claim from 'surpassing the DQN expert on unseen PHYRE tasks' to 'surpassing the DQN expert on unseen PHYRE tasks where the key physical interaction is the first collision event.' (3) We will add a sentence in the Limitation section noting that extending r_C to model multi-step collision chains is necessary to evaluate generalization on the remaining 15 task types. We agree this is load-bearing for the headline claim and should not be buried in the appendix.","revision_made":"yes","referee_comment":"Appendix B.2 discloses that the collision reward r_C models only the first object the red ball collides with, restricting held-out testing to 10 of 25 PHYRE task types. The central cross-task generalization claim is therefore validated only on first-collision-dominated tasks. This should be explicitly qualified in the main text, not only in the appendix."},{"response":"The referee's observation is accurate. Examining the three splits: Testing Set 1 = {00001, 00002, 00003, 00006, 00013}; Testing Set 2 = {00000, 00001, 00003, 00012, 00014}; Testing Set 3 = {00000, 00001, 00002, 00009, 00015}. Task types 00000, 00001, 00002, and 00003 each appear in two splits, meaning the splits share substantial task-type overlap and do not constitute three genuinely independent evaluations of robustness. We will take option (a): acknowledge the overlap and reframe the stability claim. Specifically, in §4.2 and Appendix B.2, we will reframe the three-split protocol as verifying 'consistency across different train/test partitions of the eligible task types' rather than 'stability across different held-out task distributions.' We will explicitly note the overlap in task types across splits. Regarding option (b), constructing a fully disjoint split from the 10 eligible task types is feasible in principle (e.g., two disjoint sets of 5), but we are constrained by the fact that only 10 task types are eligible and we use 5 for testing in each split, leaving limited room for fully disjoint configurations. We will note this constraint and frame it as future work once r_C is extended to handle multi-step collisions, which would expand the eligible pool.","revision_made":"yes","referee_comment":"The three testing splits share substantial overlap: task types 00000, 00001, 00002, and 00003 each appear in two or more splits. The paper frames the splits as verifying stability across different held-out distributions, but the overlap means the splits test largely the same task structures. The paper should either acknowledge the overlap and reframe the stability claim, or construct at least one fully disjoint split."},{"response":"The referee is correct that the manuscript does not adequately explain how phi/psi are handled for the Virtual Tool and CRAFT VQA experiments. We will clarify this in the revised manuscript. The key distinction is between training and evaluation: (1) For cross-environment transfer to Virtual Tool (Table 2) and CRAFT VQA (Table 3), the model is trained on all 25 PHYRE task types using the PHYRE-instantiated phi/psi. At evaluation time on Virtual Tool and CRAFT VQA, the same phi/psi are reused as-is — the model is prompted to produce reasoning traces in the same structured format (scene_answer, causal_actions_answer, placement_answer), and the symbolic extraction functions parse these traces identically. No environment-specific adaptation of phi/psi is performed. (2) For Virtual Tool, the prompt is adapted to reflect the different object types and shapes (e.g., trapezoids, triangles) and the different goal condition (green touches blue, without the purple static-object variant), but the symbolic extraction logic — 3x3 spatial grid, collision event tuples, placement descriptions — remains unchanged. (3) For CRAFT VQA, the model produces scene_answer and causal_actions_answer but no placement_answer (since no action is taken), and the final_answer tag is added for the VQA response. The phi/psi functions parse the same symbolic tuples. (4) Importantly, during evaluation on Virtual Tool and CRAFT VQA, the VAORA rewards (r_G, r_P, r_C) are not computed — these are training-time rewards that require simulator access. The evaluation is purely inference-based: the model generates reasoning traces and actions/answers, and we measure task success (Virtual Tool) or answer accuracy (CRAFT VQA). So the cross-environment claim is specifically that the reasoning structure learned via VA","revision_made":"no","referee_comment":"The symbolic extraction functions phi(r) and psi(o)/psi(v) are described as general but instantiated only for PHYRE (3x3 spatial grid, first-collision extraction). The cross-environment transfer to Virtual Tool and CRAFT VQA presumably reuse the same phi/psi, but the paper does not describe how these functions are adapted for environments with different object types, action spaces, or physics. The paper should clarify whether phi/psi are reused as-is or adapted, and what the adaptation entails."}],"tokens_in":27029,"tokens_out":1379,"duration_ms":262469,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Two things matter here. First, the core idea — supervising the alignment between a VLM's chain-of-thought reasoning and the actual visual outcome of its action — is a genuinely useful contribution. Second, the headline generalization claim is narrower than the paper presents, because the collision reward only models the first object the red ball hits, restricting evaluation to 10 of 25 PHYRE task types. That restriction is disclosed only in Appendix B.2, not in the main text where the claim is made. The reader and stress-test both flag this correctly; I agree it is the central issue. The three cross-validation splits also share overlapping task types (00000, 00001, 00002, 00003 each appear in two or more splits), so the stability evidence is weaker than three genuinely disjoint splits would provide. These are real limitations but do not invalidate the contribution. The reward design itself is well-motivated. The diagnostic framing of two failure modes — hallucinated CoT and reasoning-action misalignment — is clean and the ablation in Table 5 shows each component (grounding, placement, collision) contributes meaningfully. The reward breakdown in Table 4 is genuinely informative: SFT improves grounding but not placement or collision, EG alone collapses placement to zero on one split, and the full VAORA recovers all three. That is real evidence the alignment signal does what the authors claim. The DQN circularity concern (DQN as both reward source and baseline) is mitigated by the +EG vs +EG+VAORA ablation — the VAORA components add consistent gains on top of the expert-guided signal alone, so the improvement is not just the DQN leaking through. The Virtual Tool protocol modifications (color remapping, tool selection removal, unlimited attempts) are a minor concern. They are disclosed and reasonable for cross-environment transfer, but they do make the baseline comparison less clean than it could be. The CRAFT VQA results are a nice secondary check showing the learned reasoning transfers beyond action selection, though the gains are modest (~2 points overall). Who is this for? Researchers working on VLM-based physical reasoning and grounded CoT. The reward design is the contribution; the specific PHYRE numbers are secondary. The paper deserves a serious referee. The main text should state the 10/25 task-type restriction upfront, and the split overlap should be acknowledged. If the authors do that and frame the generalization claim accordingly, this is a solid paper.","headline":"VAORA grounds VLM physical reasoning in post-action visual outcomes — a solid reward design, but the headline generalization claim is narrower than presented because evaluation covers only 10 of 25 PHYRE task types.","tokens_in":28096,"tokens_out":584,"would_cite":true,"duration_ms":82267,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Aligning reasoning to physical outcomes lets 8B model beat expert agent","keywords":["vision-language models","physical reasoning","chain-of-thought alignment","reinforcement learning","reward design","cross-task generalization","PHYRE","symbolic grounding"],"falsifier":"If one tested VAORA on the 15 excluded PHYRE task types — those requiring multi-step collision chains — and found that the alignment rewards provided no signal and performance degraded to baseline levels, the central claim that outcome alignment enables generalization would be significantly weakened.","tokens_in":27133,"feed_emoji":"🎱","tokens_out":1454,"duration_ms":84641,"temperature":0.7,"pith_summary":"The paper argues that vision-language models fail at physical reasoning for two reasons: their chain-of-thought reasoning contradicts physical reality (hallucinated CoT), and their stated reasoning disconnects from their actual actions (reasoning-action misalignment). Both failures share a root cause — the model's reasoning is never checked against what actually happens after it acts. The authors propose VAORA, a reward design that projects the model's reasoning trace and visual observations into a shared symbolic space — object positions, collision events, placement descriptions — and rewards the model when its reasoning matches both the initial scene and the post-action outcome. A grounding reward checks whether the model accurately perceives the starting scene; a collision reward checks whether the model's predicted collision events match what the simulator actually produces; a placement reward checks whether the model's described ball placement matches where the ball actually lands. These visual-action alignment rewards are gated by a success-probability estimate from a pretrained DQN expert, so reasoning quality is only rewarded when the action itself is physically plausible. The authors train an 8B VLM solely on the PHYRE physics benchmark and show it surpasses the DQN expert on unseen PHYRE task types, transfers zero-shot to a different physics simulator (Virtual Tool) at the level of frontier closed-source models, and improves causal reasoning on a visual question-answering benchmark — all from a single training environment.","feed_headline":"Grounding reasoning to physical outcomes lets 8B model beat expert","feed_subtitle":"An 8B vision-language model trained only on one physics benchmark surpasses the DQN expert on unseen tasks and matches frontier closed-sot","key_machinery":"VAORA's mechanism is a symbolic projection-and-compare pipeline: the model's reasoning trace is parsed into structured symbolic tuples (object positions on a 3×3 spatial grid, predicted collision events with contact points and directions, placement descriptions relative to reference objects), and the simulator's ground-truth states are extracted into the same symbolic format. Consistency between the two is measured by soft grid scores and coordinate distances, producing dense reward signals that directly supervise whether the model's reasoning matches physical reality. The visual-action alignment rewards are gated by a DQN-derived success probability to ensure reasoning is only rewarded when","core_discovery":"The central discovery is that supervising the alignment between a VLM's reasoning trace and the actual visual consequences of its actions — not just task success — is what enables cross-task and cross-environment generalization. The paper provides direct evidence through reward breakdown analysis: supervised fine-tuning transfers static scene perception but fails on placement and collision reasoning; success-driven RL collapses into shortcut visual-to-action mappings that abandon grounded reasoning entirely. Only when both the initial-scene alignment and the post-action outcome alignment are jointly rewarded does the model develop reasoning that generalizes to unseen tasks, a different simor","pith_inferences":["The restriction of the collision reward to first-collision-only events means the generalization claim is validated on a structurally constrained subset of tasks (10 of 25 PHYRE types). Tasks requiring multi-step collision chains are excluded from held-out testing, so the generalization advantage may not extend to tasks with longer causal chains.","The symbolic extraction functions are hand-designed for 2D physics puzzles with discrete object types and simple collision dynamics. Whether the same approach scales to 3D environments, deformable objects, or fluid dynamics — where symbolic state extraction is far less tractable — remains an open question.","The reliance on a pretrained DQN expert for dense reward estimation creates a ceiling: the VLM cannot be rewarded for actions the DQN considers unlikely to succeed, potentially limiting exploration beyond the expert's competence distribution.","The single-turn, single-action setting means the model cannot learn from failed attempts. Extending to multi-turn interaction — where the model observes outcomes and updates its reasoning — could substantially expand the range of generalizable physical concepts, as the authors themselves note."],"forward_implications":["If grounding reasoning to post-action outcomes is the key to generalization, then any interactive reasoning task — robotic manipulation, navigation, tool use — could benefit from the same symbolic-alignment reward pattern, not just physics puzzles.","The finding that success-driven RL collapses into shortcut mappings suggests that reward shaping for VLMs in continuous-action spaces requires outcome-level supervision, not just task-success signals, to avoid degenerate solutions.","The zero-shot transfer from PHYRE to Virtual Tool implies that aligned physical reasoning is at least partially environment-agnostic — the model learns transferable causal concepts rather than simulator-specific visual correlations.","The improvement on CRAFT VQA from training only on PHYRE suggests that grounding reasoning to action outcomes induces broader causal understanding that extends beyond action selection to passive reasoning and question answering."],"fun_headline_variants":["Rewarding visual outcomes over task success enables VLM physical generalization","Aligning reasoning to action outcomes improves VLM physical generalization","VLMs generalize physical reasoning when rewards match visual outcomes","Visual outcome rewards outperform success-only RL for VLM physical reasoning","Grounding reasoning to visual outcomes enables cross-task VLM transfer"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The collision reward only models the first object the placed ball collides with, which means the held-out testing set is restricted to 10 of 25 PHYRE task types where the key physical interaction is a single direct collision. The generalization claim is validated only on tasks matching this structural constraint; tasks requiring multi-step collision chains are excluded from evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Rewarding visual outcomes over task success enables VLM physical generalization","Aligning reasoning to action outcomes improves VLM physical generalization","VLMs generalize physical reasoning when rewards match visual outcomes","Visual outcome rewards outperform success-only RL for VLM physical reasoning","Grounding reasoning to visual outcomes enables cross-task VLM transfer"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1270,"prompt_tokens":491,"completion_tokens":779,"prompt_tokens_details":null},"tokens_in":491,"tokens_out":779,"duration_ms":25589,"temperature":1.0,"reasoning_tokens":831,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T02:53:20.821820+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If one tested VAORA on the 15 excluded PHYRE task types — those requiring multi-step collision chains — and found that the alignment rewards provided no signal and performance degraded to baseline levels, the central claim that outcome alignment enables generalization would be significantly weakened.","supporting_citations":[],"review_version":1}