{"id":"efb689da-0723-4417-8723-96f66c29282c","arxiv_id":"2605.20246","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GROW decomposes trajectories into state-action samples to enable GRPO for multi-turn VLM agents and reports state-of-the-art results on more than 800 Minecraft tasks.","lead":"The paper introduces GROW, a reinforcement learning framework that decomposes full task trajectories into individual state-action samples so that GRPO can be applied to multi-turn vision-language model agents without excessive context lengths. This targets a practical bottleneck in training agents for open-world tasks such as Minecraft.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Surrogate analysis assumes simplified conditions that may not preserve GRPO relative signal when samples are drawn from distinct local states","rationale":"The reader's weakest_assumption correctly isolates the precise point where the theoretical justification is least secure. The empirical SOTA result on 800 tasks stands on its own as an engineering outcome, but the paper's framing ties performance to the GRPO-preserving property; confirming or refuting that property via the proposed re-derivation would directly test whether the central claim holds or requires qualification.","tokens_in":1773,"tokens_out":334,"duration_ms":20185,"concrete_test":"Re-derive the surrogate objective in the analysis section starting from the standard GRPO loss without invoking the 'identical prompt' or 'bounded variance' simplifications; substitute the actual state-conditioned sampling used in GROW and check whether the sign of the advantage term remains consistent with the original GRPO ranking for at least two distinct local states drawn from the same trajectory.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the surrogate analysis showing that decomposing trajectories into state-action pairs and computing advantages across them still recovers the core GRPO relative policy optimization. The analysis explicitly invokes simplifying assumptions (identical prompt context replaced by different local states, plus bounded variance in advantage estimates). In open-world Minecraft, visual states differ sharply across steps and trajectories, so the grouped samples no longer share the same conditioning context that GRPO relies on for stable relative ranking. If the advantage estimator under these mismatched contexts deviates from the original GRPO derivation, the claimed preservation fails and the method reduces to an unanalyzed heuristic.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents GROW, a reinforcement learning framework for vision-language model (VLM) agents in open-world tasks such as Minecraft. It modifies Group Relative Policy Optimization (GRPO) by decomposing full trajectories into state-action samples and computing advantages across these samples rather than treating entire trajectories as single entities. This is intended to avoid excessively long contexts and noise. A surrogate analysis argues that the modified objective preserves the core relative policy optimization signal of GRPO under simplifying assumptions even when samples are conditioned on different local states. Experiments report state-of-the-art performance on more than 800 Minecraft tasks.","tokens_in":1888,"tokens_out":560,"duration_ms":27330,"significance":"If the surrogate analysis is valid and the performance gains are robust, the work offers a practical route to applying relative policy optimization methods to multi-turn VLM agents, potentially improving sample efficiency and stability in long-horizon open-world settings by reducing reliance on full-trajectory training.","major_comments":[{"comment":"Surrogate Analysis section: The claim that the objective preserves the core GRPO relative policy optimization signal rests on simplifying assumptions (identical prompt context replaced by different local states plus bounded variance in advantage estimates). In open-world Minecraft, where visual states differ sharply across steps, it is unclear whether the advantage estimator maintains the original relative ranking property; the manuscript does not provide a derivation showing invariance or empirical checks that the signal is recovered outside the assumed regime. This is load-bearing for the theoretical justification of the central method.","section":"Surrogate Analysis"},{"comment":"Experiments section: The SOTA claim on more than 800 tasks is presented without quantitative details on baseline methods, exact success-rate metrics, number of independent runs, or statistical significance tests. Without these, it is difficult to assess whether the reported gains are attributable to the state-action decomposition or to other factors such as task selection or implementation details.","section":"Experiments"}],"minor_comments":[{"comment":"Abstract: The phrase 'surrogate analysis' is used without a one-sentence qualifier on the nature of the assumptions; adding this would improve readability for readers unfamiliar with the GRPO derivation.","section":"Abstract"},{"comment":"Notation: Define the state-action decomposition operator and the cross-sample advantage estimator explicitly in the preliminaries to ensure consistent interpretation across the method and analysis sections.","section":"Preliminaries"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for a machine-learning venue focused on RL and agents. The citation list appears balanced, though the authors should verify coverage of recent trajectory-decomposition work in hierarchical RL."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback on our manuscript. The comments highlight important aspects of the surrogate analysis and experimental reporting that we address point by point below. We have prepared revisions to strengthen the theoretical justification and provide the requested quantitative details.","responses":[{"response":"We agree that the surrogate analysis relies on simplifying assumptions and that additional support is needed to establish robustness outside this regime. In the original manuscript we show that, under the stated assumptions of bounded advantage variance and local-state conditioning, the relative ordering of advantages is preserved because the group-wise normalization remains monotonic with respect to the original GRPO signal. To address the referee’s concern, we will expand the appendix with a more explicit step-by-step derivation of the invariance property and add empirical checks that compare advantage rankings obtained from full-trajectory GRPO versus our state-action decomposition on a held-out subset of Minecraft tasks. These additions will clarify the conditions under which the core relative policy optimization signal is retained.","revision_made":"yes","referee_comment":"[Surrogate Analysis] Surrogate Analysis section: The claim that the objective preserves the core GRPO relative policy optimization signal rests on simplifying assumptions (identical prompt context replaced by different local states plus bounded variance in advantage estimates). In open-world Minecraft, where visual states differ sharply across steps, it is unclear whether the advantage estimator maintains the original relative ranking property; the manuscript does not provide a derivation showing invariance or empirical checks that the signal is recovered outside the assumed regime. This is load-bearing for the theoretical justification of the central method."},{"response":"The referee correctly notes that the current experimental section lacks the quantitative details necessary for rigorous evaluation. In the revised manuscript we will add comprehensive tables that report per-task success rates for GROW and all baselines, the exact number of independent runs (five random seeds), and the results of statistical significance tests (paired t-tests with p-values). We will also include an ablation isolating the contribution of the state-action decomposition from other implementation choices. These changes will make the source of the reported gains transparent.","revision_made":"yes","referee_comment":"[Experiments] Experiments section: The SOTA claim on more than 800 tasks is presented without quantitative details on baseline methods, exact success-rate metrics, number of independent runs, or statistical significance tests. Without these, it is difficult to assess whether the reported gains are attributable to the state-action decomposition or to other factors such as task selection or implementation details."}],"tokens_in":1402,"tokens_out":529,"duration_ms":19050,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that GROW decomposes full trajectories into state-action samples so GRPO can be applied without blowing up the context length in multi-turn VLM settings. They back this with a surrogate analysis and claim SOTA on a large set of Minecraft tasks. What stands out is how directly they target the practical barrier: standard GRPO wants whole trajectories, which gets noisy and too long for these agents. By regrouping into per-step samples and computing advantages across them, they make the training feasible. The scale of the evaluation, over 800 tasks, gives some credibility to the performance claims. The soft spot is the surrogate analysis itself. It says the core relative optimization signal survives even when the grouped samples come from different local states rather than one shared prompt, but only under simplifying assumptions. In open-world environments the visual states shift a lot between actions, which could break the relative comparisons that GRPO relies on. The paper would be stronger if it showed how well those assumptions match the actual data or included some sensitivity checks. This work is for people doing RL on vision-language models for interactive or robotic tasks. A reader who needs ideas for handling long sequences in agent training will find the decomposition approach and the experimental setup worth examining. I would send it to peer review. The empirical side is substantial enough to merit referee input, even if the theoretical justification needs more scrutiny.","headline":"GROW adapts GRPO to long-horizon VLM agents through state-action decomposition and reports strong Minecraft results, though the supporting analysis rests on assumptions that may not hold up.","tokens_in":2417,"tokens_out":355,"would_cite":false,"duration_ms":32412,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"GROW trajectory-decomposition RL for VLM agents lies outside RS forcing chain","alignment":"orthogonal","rationale":"Paper's central machinery (state-action decomposition of trajectories, surrogate analysis preserving GRPO relative advantages under local-state conditioning, discounted reward propagation) is standard empirical RL adaptation with no invocation of J-cost, cosh identities, ratio symmetry, φ-ladder, 8-tick periodicity, or parameter-free constant derivation. Domain (Minecraft VLM policy optimization) is one on which RS framework has no opinion.","tokens_in":52151,"confidence":"high","tokens_out":127,"duration_ms":10551,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"GROW adapts GRPO for multi-turn VLM agents by decomposing trajectories into state-action samples for advantage computation.","keywords":["GRPO","VLM agents","open-world tasks","Minecraft","reinforcement learning","trajectory decomposition","state-action modeling","multi-turn RL"],"falsifier":"An experiment that applies GROW to a long-horizon task where the local states differ substantially from any shared prompt context and measures whether the learned policy diverges from what standard GRPO would produce or loses its reported performance gains.","tokens_in":2656,"feed_emoji":"🎮","tokens_out":522,"duration_ms":37095,"temperature":0.7,"pith_summary":"The paper seeks to establish that standard GRPO cannot be directly used for multi-turn vision-language model agents in open-world settings because it treats each full trajectory as one training sample, producing excessively long contexts and noisy signals. GROW solves this by splitting collected trajectories into individual state-action samples and computing advantages across those samples instead of across whole trajectories. A surrogate analysis argues that this change still keeps the essential relative policy optimization behavior of GRPO when certain simplifying conditions hold. If the approach works, it would let researchers move beyond supervised fine-tuning with expert demonstrations and train VLM agents on hundreds of complex tasks using reinforcement learning signals that remain stable even when tasks require many perception-action turns.","feed_headline":"State-action decomposition adapts GRPO for VLM agents","feed_subtitle":"Splitting trajectories into samples for advantage calculation cuts context noise while keeping GRPO's relative signal, yielding SOTA on over","key_machinery":"Decomposition of full trajectories into state-action samples followed by inter-sample advantage computation, which aligns the GRPO objective with state-action modeling for multi-turn agents.","core_discovery":"GROW decomposes collected trajectories into state-action samples and computes advantages between these samples rather than treating a full trajectory as a single entity. A surrogate analysis indicates that even though the grouped samples are conditioned on different local states rather than an identical prompt context, the objective can preserve the core relative policy optimization signal of GRPO under simplifying assumptions. Experiments on more than 800 Minecraft tasks show that this framework achieves state-of-the-art performance for open-world VLM agents.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Trajectory decomposition aligns GRPO with VLM agents","State-action samples enable GRPO for open-world agents","GRPO signal retained in local state-action groups","GROW applies GRPO to multi-turn VLM agent tasks"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That samples drawn from different local states can still carry the same relative policy optimization signal as samples sharing an identical prompt context, provided the paper's simplifying assumptions remain valid.","fun_headline_variants_meta":{"raw":{"variants":["Trajectory decomposition aligns GRPO with VLM agents","State-action samples enable GRPO for open-world agents","GRPO signal retained in local state-action groups","GROW applies GRPO to multi-turn VLM agent tasks"]},"model":"grok-4.3","cost_usd":0.006942,"raw_usage":{"total_tokens":3224,"prompt_tokens":679,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":69424500,"prompt_tokens_details":{"text_tokens":679,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2484,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":679,"tokens_out":61,"duration_ms":23544,"temperature":1.0,"reasoning_tokens":2484,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-22T09:02:55.276742+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment that applies GROW to a long-horizon task where the local states differ substantially from any shared prompt context and measures whether the learned policy diverges from what standard GRPO would produce or loses its reported performance gains.","supporting_citations":[],"review_version":2}