{"id":"5e7541a3-bdf6-42cb-8e22-7da7aecbfd39","arxiv_id":"2608.04934","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Randomly explored environment states, used as exact-match state-reaching RL objectives, improve LLM agent performance on ALFWorld, ScienceWorld, and a mobile GUI benchmark.","lead":"State2State turns an environment into its own training data: it records states reached during random exploration and asks the agent to reach those states again, with exact-match verification. On ALFWorld, ScienceWorld, and a mobile GUI benchmark, this mid-training step improves agent success rates both alone and before downstream RL.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is confounded by extra training compute: STATE2STATE adds an 80-step RL warm-up that RL-only never receives, so Table 1 and Fig. 3 do not isolate the environment-derived objective as the cause of the gains.","rationale":"The reader's weakest assumption—that exact observation matching reliably verifies state—is a real limitation, but it is not the most load-bearing point for the central claim. Even if the verifier were perfect and reproducibility guaranteed, the main empirical comparison would still be confounded by the extra 80 training steps and extra rollouts given to STATE2STATE. The paper's own framing in Sec. 4.2 concedes that the comparison is not compute-matched; this concession means the headline improvements in Table 1 and Fig. 3 could be explained by the additional optimization budget rather than by the nature of the mid-training objective. A compute-matched control is therefore the single check that would adjudicate the claim. The cross-environment transfer result has the same confound plus a question of whether ScienceWorld exploration used the same task variations as evaluation; the paper does not specify the exploration task split. This reinforces the need for released code/data and per-seed numbers. I do not see an internal inconsistency in the method itself, and the directionally consistent results across two scales and environments justify a conditional acceptance pending these controls.","tokens_in":20152,"tokens_out":7907,"duration_ms":73917,"concrete_test":"Run the RL-only baseline with an 80-step online warm-up on the same environment using random valid actions (or a matched auxiliary objective that does not use explored state targets), then continue downstream task RL for the same number of steps as STATE2STATE+RL, and compare final success rates and the step-50/step-150 convergence curves in Fig. 3. If this compute-matched RL-only control matches or exceeds the full pipeline, the claimed benefit is attributable to extra compute rather than to the state-reaching objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 explicitly frames STATE2STATE as a mid-training stage 'rather than a compute-matched replacement,' but that caveat undercuts the very comparison used to support the headline claim. In Table 1, STATE2STATE+RL is the strongest row, and in Fig. 3 it appears to reach a high ScienceWorld score by step 50 versus step 150 for Human-Task GRPO. Yet the STATE2STATE pipeline first runs 80 GRPO steps on state-reaching tasks before downstream RL begins, while the RL-only baseline starts from the base model. If downstream training is run to the same fixed number of steps (Sec. E.1), the full pipeline has 80 additional optimizer steps and additional environment rollouts; if the x-axis in Fig. 3 counts only downstream steps, the efficiency advantage is partly an accounting artifact (80 + 50 is comparable to 150). Any extra warm-up in the same environment—even with random valid actions or a trivial auxiliary reward—could yield similar final and early-curve improvements simply from additional policy updates and environment exposure. The cross-environment comparison in Table 5 has the same structure: ScienceWorld STATE2STATE mid-training is compared with ScienceWorld human-task RL at 'same steps,' but the objectives and reward densities are not matched, so the result cannot attribute transfer to the state-reaching objective specifically. Without a compute-matched control that gives the baseline the same extra step budget, the evidence does not establish that environment-derived state-reaching objectives, rather than extra training compute, drive the reported gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STATE2STATE, an environment-derived mid-training method for LLM agents. It uses random exploration to collect reachable environment observations, filters and replays them to form reproducible state-reaching tasks, and trains the agent with GRPO using a rule-based observation-matching reward. The resulting policy is then used as initialization for downstream RL on standard human-annotated tasks. Experiments on ALFWorld and ScienceWorld with Qwen3-4B and Qwen3-8B compare standalone STATE2STATE, direct human-task RL, and STATE2STATE+RL, and also include analyses of SFT ordering, RL backbone choice, exploration strategy, cross-environment transfer, and a MobileWorld GUI extension. The central claim is that environment-derived state-reaching objectives provide useful, scalable skill priors that improve final downstream task performance and learning efficiency.","tokens_in":20394,"tokens_out":5326,"duration_ms":47586,"significance":"If the headline claims hold, the paper makes a useful and timely contribution: it demonstrates a scalable, verifiable, and supervision-light way to obtain agentic skill priors, complementing SFT and task-specific RL. The strengths include the simplicity of random exploration plus exact-match verification, the large and carefully documented constructed datasets, the explicit acknowledgment of limitations, and the inclusion of several controlled ablations (SFT ordering, backbone, exploration strategy). The method is not circular: training targets are generated by random exploration and verified independently of downstream human-task evaluation. However, the main empirical evidence is currently not clean: the full pipeline receives 80 additional RL steps before downstream training, so the reported gains do not isolate the state-reaching objective from extra training compute, and all RL results are single runs without error bars. These issues must be addressed before the central claims can be accepted.","major_comments":[{"comment":"The headline comparison does not control for training compute. STATE2STATE+RL runs 80 GRPO steps on state-reaching tasks before downstream human-task RL, whereas RL-only starts from the base model and is trained for the same fixed number of downstream steps (300 for ALFWorld, 400 for ScienceWorld, per §E.1). The full pipeline therefore has 80 additional optimizer steps and additional environment rollouts, so the gains in Table 1 and the faster early curve in Fig. 3 could reflect extra warm-up on the same environment rather than the state-reaching objective specifically. Please add a compute-matched control: give RL-only the same 80 warm-up steps on the same environments with a non-state-reaching auxiliary reward (or with repeated human-task data), and also report total optimizer steps including mid-training on the x-axis of Fig. 3. Without this, the central claim that environment-derived state-reaching objectives improve downstream RL is not isolated.","section":"§4.2, §E.1, Table 1, Fig. 3"},{"comment":"The cross-environment comparison is similarly confounded. ScienceWorld STATE2STATE and ScienceWorld human-task RL are matched in 'steps of mid-training,' but the two training signals are not matched in reward density, trajectory length, or update-to-rollout ratio; in particular, the state-reaching runs use dynamic sampling (§C.2), which changes the number of environment interactions per gradient step. As a result, the better ALFWorld transfer cannot be attributed to the state-reaching objective. A control that matches total environment interactions and total gradient steps while varying only the objective is needed; for example, mid-training on an equal number of reward-dense, non-goal exploration rollouts.","section":"§5.4, Table 5"},{"comment":"All RL results are reported as single runs without seeds, error bars, or confidence intervals. GRPO on sparse binary rewards is high-variance, and many differences are small relative to likely noise: for example, +2.56 points on ALFWorld OOD for Qwen3-4B, +3.87 points on ScienceWorld average for Qwen3-8B, and +0.033 on 117 MobileWorld tasks in Table 6. Each condition should be run with at least three random seeds and reported as mean ± std, or with a paired bootstrap over evaluation tasks. This is load-bearing for the claimed improvements in final performance, learning efficiency, and ordering conclusions in Table 2.","section":"Tables 1, 3, 4, 5 and Fig. 3"},{"comment":"The MobileWorld reward is not derived purely from the environment: it combines XML fuzzy sequence similarity and SSIM with hand-chosen weights (0.8 vs. 0.2), and the ScienceWorld exploration split into spatial and state-changing phases is also a manually specified prior. To support the claim that STATE2STATE requires no handcrafted reward components in GUI environments, the paper should ablate or justify these choices, or restrict the claim to text environments. In addition, the reported +0.033 improvement on a single run is within a range that needs uncertainty quantification before it can be interpreted as evidence of standalone improvement.","section":"§F.3, Table 6"}],"minor_comments":[{"comment":"The x-axis label 'Training Steps' is ambiguous: specify whether it counts downstream steps only or total optimizer steps including the 80-step STATE2STATE phase.","section":"Fig. 3"},{"comment":"The sentence 'with maximum prompt length between 12000 and response length 1024' appears to be missing a token count for the prompt; please clarify.","section":"§E.1"},{"comment":"GiGPO is only defined in Appendix C.3; a one-sentence description in the main text would improve readability.","section":"Table 3"},{"comment":"The ScienceWorld Case 3 example labeled 'find-animal; action feedback' shows an action-feedback observation rather than a target state; this is confusing and should be relabeled or replaced.","section":"§B.3"},{"comment":"Clarify whether the SSIM-based visual reward at training time is computed on the resized 540×1200 images only, since evaluation uses original-size images; this discrepancy should be stated explicitly.","section":"§F.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea and careful dataset construction, but the central empirical comparison currently confounds the state-reaching objective with extra RL compute, and all RL results are single-seed runs. I would require a compute-matched control and multi-seed results before accepting the headline claims. The cross-environment transfer result in Table 5 needs the same matching. The paper is within scope for cs.CL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The core idea is clean: randomly explore an environment, collect reachable observations, turn them into RL objectives where the agent must reproduce a target observation, and use that as mid-training before task-specific RL. The specific combination—random exploration, state-reaching RL with exact-match verification, then downstream human-task RL—is not in the cited prior work, and the paper correctly credits the underlying goal-relabeling idea to HER. That's the right way to position novelty.\n\nWhat the paper does well: the method description is thorough, the appendices give real detail on data construction and prompts, and the limitations section is candid. The cross-environment transfer result (ScienceWorld mid-training -> ALFWorld RL) is the most interesting result because it compares two mid-training objectives with the same step budget. The standalone S2S gains over the base model are also evidence that state-reaching objectives teach something transferable.\n\nWhere it falls short: the main comparison in Table 1 and the efficiency claim in Figure 3 are confounded by compute. The full pipeline gets 80 GRPO steps on state-reaching tasks before downstream RL starts; the RL-only baseline gets zero. If downstream runs are all 400 steps, then S2S+RL has 480 total optimizer steps in the same environment. The stress-test note is right: 'reaches the same score at step 50 instead of step 150' is partly an accounting artifact (80+50 is close to 150). The paper explicitly says it is not a compute-matched replacement, which is honest, but then uses Figure 3 to claim learning efficiency, which undercuts that caveat. The cross-environment table partially controls for extra compute, but still doesn't rule out that any additional RL in a related environment helps. On top of that, the main results are single runs without error bars, and RL at this scale is noisy. Code and data are promised but not yet available.\n\nNone of this is fatal. The direction is consistent across two model scales and multiple environments, and the authors have identified a real bottleneck. But the load-bearing claim—that the environment-derived objective, rather than extra training compute, drives the gains—is not yet established. A compute-matched control (e.g., RL-only for 480 steps, or a warm-up on a random objective) plus a couple of seeds would settle it.\n\nWho is this for: people working on agentic RL or self-supervised skill acquisition. They'll get a useful method and a fair discussion of its limits. I would cite it, and I'd send it to review with a strong request for the control above.","headline":"Useful, honest empirical paper with a clean idea, but the central comparison is confounded by extra training compute and the efficiency claim is overstated.","tokens_in":20995,"tokens_out":4380,"would_cite":true,"duration_ms":37151,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM agents learn useful environment skills by training solely on explored reachable states, and that this mid-training stage improves downstream task-specific RL.","keywords":["LLM agents","environment learning","mid-training","state-reaching objectives","reinforcement learning","rule-based verifier","ALFWorld","ScienceWorld"],"falsifier":"Build or find an environment where two different underlying states yield the same text observation (for example, a container whose visible text does not change after its contents are swapped), and show that State2State training assigns success to the wrong state; or an environment where replay from the same initial configuration diverges, so many sampled targets are unreachable and the reward is noise. If the method's gains vanish in either setting, the verifier assumption is the load-bearing part.","tokens_in":19916,"feed_emoji":"🎯","tokens_out":8668,"duration_ms":67417,"temperature":0.7,"pith_summary":"State2State is a mid-training stage that turns previously explored environment states into training objectives: the agent is shown a target observation and must act until its current observation exactly matches the target. The paper argues that these environment-derived, rule-verified state-reaching tasks give LLM agents useful perception and manipulation priors without expert demonstrations, teacher models, or human-written task instructions. In experiments on ALFWorld and ScienceWorld with Qwen3 models at two scales, the standalone stage improves performance in most settings and, when used to initialize downstream RL on human tasks, yields the best final success rates and faster learning. The paper further reports positive cross-environment transfer from ScienceWorld to ALFWorld and a gain on a mobile GUI environment, suggesting the objective itself, not merely extra environment RL, is what transfers.","feed_headline":"Turning explored states into goals lifts LLM agent RL","feed_subtitle":"A task-free mid-training stage that derives rewards from the environment improves downstream RL and learning speed.","key_machinery":"The central object is the state-reaching task, defined by an initial configuration $s_0$ and a target observation $o^*$ sampled from the explored pool, with binary reward $r_t = \\mathbf{1}[\\, o(s_t) \\text{ matches } o^*]$. The machine that makes it work is the pairing of random exploration with reproducibility filtering and diversity-oriented sampling: random exploration supplies broad state coverage without task priors, replaying each trajectory three times removes non-deterministic targets, and limiting repetition of identical observations keeps the goal pool diverse. GRPO with dynamic sampling then provides the contrastive, sparse-reward optimization that turns these targets into policy gradients.","core_discovery":"State2State claims that a large portion of agent skill can be acquired by solving self-generated state-reaching tasks. A random or lightly biased explorer collects reachable observations; invalid observations are filtered, diverse targets are sampled, and each candidate is replayed three times to confirm the state is reproducible from its initial configuration. The training prompt then says “reach this state”, and success is decided by the rule-based reward $r_t = \\mathbf{1}[\\mathrm{match}(o(s_t), o^*)]$, with normalized exact text match as the environment-specific verifier. Policies are optimized with GRPO plus dynamic sampling, which discards rollout groups whose outcomes are identical. The central empirical claim is that this stage transfers: it improves base models in most settings, improves both final performance and learning efficiency of downstream human-task RL, and transfers across environments better than training on human-specified ScienceWorld tasks.","pith_inferences":["If the textual observation is a lossy rendering of the underlying state, exact-match targets will systematically favor states that differ in text and may under-sample states whose differences are hidden; environments with richer hidden dynamics would likely need a learned or fuzzy state-equivalence matcher.","The exploration policy is a natural control knob: this paper shows random exploration beats an LLM explorer, but in larger or sparser state spaces a hierarchy that broadens then deepens exploration might combine State2State's reproducibility filter with more targeted state coverage.","The same recipe may transfer to web, software-engineering, or real-device control whenever a reproducible reset and a state matcher exist; a direct test would compare State2State mid-training against trajectory-imitation pretraining on equal compute in those domains.","The “latent benefits released by downstream RL” result suggests that state-reaching performance alone understates what the stage teaches, so an evaluation metric that probes manipulation breadth rather than just task success could make those gains visible before human-task RL."],"forward_implications":["State2State can serve as a scalable mid-training stage that reduces the need for expert demonstrations and hand-written task curricula, because both objectives and rewards come from exploration.","Downstream RL becomes more sample-efficient: on ScienceWorld with Qwen3-4B, the full pipeline reaches a comparable score around step 50 that the task-only baseline reaches near step 150.","The full pipeline achieves the best performance on both ALFWorld and ScienceWorld at both model scales, including out-of-distribution splits.","State2State remains beneficial on top of supervised fine-tuning and with a stronger RL backbone such as GiGPO, so it is complementary to existing agent-training ingredients.","Environment-derived mid-training transfers across environments: ScienceWorld State2State before ALFWorld RL improves ALFWorld success beyond using ScienceWorld human-task RL for the same number of steps."],"supporting_citations":[{"why":"Supplies ALFWorld, one of the two main evaluation environments, including the simulated households whose admissible commands the random explorer samples from.","marker":"Shridhar et al., 2021"},{"why":"Supplies ScienceWorld, the other main benchmark, with its large maps and scientific action templates used for exploration and evaluation.","marker":"Wang et al., 2022"},{"why":"Provides GRPO, the group-relative policy optimization algorithm used for both State2State mid-training and downstream task RL.","marker":"Shao et al., 2024"},{"why":"Provides DAPO's dynamic sampling mechanism, which discards rollout groups with identical outcomes to keep sparse state-reaching rewards informative.","marker":"Yu et al., 2025"},{"why":"Agent Early Experience is the mid-training baseline built from reward-free interactions that State2State is compared against.","marker":"Zhang et al., 2025a"},{"why":"Supplies GiGPO, the fine-grained group-in-group optimizer used to test whether State2State still helps under a stronger RL backbone.","marker":"Feng et al., 2025"},{"why":"Supplies MobileWorld and the MAI-UI-8B base model for the extension showing State2State works on GUI-only tasks.","marker":"Kong et al., 2025"},{"why":"Provides Qwen3-4B and Qwen3-8B, the two model scales on which the main comparisons are run.","marker":"Qwen Team, 2025b"}],"fun_headline_variants":["Self-made state goals boost LLM agent RL","Environment-derived mid-training lifts agent RL","Reaching explored states teaches agents skills","Task-free state matching accelerates agent learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole reward signal rests on the assumption that a normalized exact string match between the current observation and the target observation correctly detects that the agent has reached the intended state, and that re-running a trajectory from the same initial configuration reliably reproduces the explored state.","fun_headline_variants_meta":{"raw":{"variants":["Self-made state goals boost LLM agent RL","Environment-derived mid-training lifts agent RL","Reaching explored states teaches agents skills","Task-free state matching accelerates agent learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000491,"raw_usage":{"total_tokens":2381,"prompt_tokens":878,"completion_tokens":1503,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":1450}},"tokens_in":494,"tokens_out":1503,"duration_ms":10574,"temperature":1.0,"reasoning_tokens":1450,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:36:02.070973+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build or find an environment where two different underlying states yield the same text observation (for example, a container whose visible text does not change after its contents are swapped), and show that State2State training assigns success to the wrong state; or an environment where replay from the same initial configuration diverges, so many sampled targets are unreachable and the reward is noise. If the method's gains vanish in either setting, the verifier assumption is the load-bearing part.","supporting_citations":[],"review_version":1}