State2State: Environment-Derived Mid-Training for LLM Agents

Chenliang Li; Jieping Ye; Kaiming Liu; Ming Yan; Peng Li; Xuanyu Lei; Yang Liu; Ya-Qin Zhang; Yiqi Zhu

REVIEW 4 major objections 5 minor 13 references

State2State: Environment-Derived Mid-Training for LLM Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that LLM agents learn useful environment skills by training solely on explored reachable states, and that this mid-training stage improves downstream task-specific RL.

desk verdict Useful, honest empirical paper with a clean idea, but the central comparison is confounded by extra training compute and the efficiency claim is overstated. read the letter →

arxiv 2608.04934 v1 pith:LHW2ZNT3 submitted 2026-08-05 cs.CL cs.LG

Xuanyu Lei , Yiqi Zhu , Chenliang Li , Kaiming Liu , Peng Li , Ming Yan , Jieping Ye , Ya-Qin Zhang

show 1 more author

Yang Liu

This is my paper · ORCID

classification cs.CLcs.LG

keywords LLMagentsenvironmentlearningmid-trainingstate-reachingobjectivesreinforcementrule-basedverifierALFWorldScienceWorld

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

State2State is a mid-training stage that turns previously explored environment states into training objectives: the agent is shown a target observation and must act until its current observation exactly matches the target. The paper argues that these environment-derived, rule-verified state-reaching tasks give LLM agents useful perception and manipulation priors without expert demonstrations, teacher models, or human-written task instructions. In experiments on ALFWorld and ScienceWorld with Qwen3 models at two scales, the standalone stage improves performance in most settings and, when used to initialize downstream RL on human tasks, yields the best final success rates and faster learning. The paper further reports positive cross-environment transfer from ScienceWorld to ALFWorld and a gain on a mobile GUI environment, suggesting the objective itself, not merely extra environment RL, is what transfers.

What carries the argument

The central object is the state-reaching task, defined by an initial configuration $s_0$ and a target observation $o^*$ sampled from the explored pool, with binary reward $r_t = \mathbf{1}[\, o(s_t) \text{ matches } o^*]$. The machine that makes it work is the pairing of random exploration with reproducibility filtering and diversity-oriented sampling: random exploration supplies broad state coverage without task priors, replaying each trajectory three times removes non-deterministic targets, and limiting repetition of identical observations keeps the goal pool diverse. GRPO with dynamic sampling then provides the contrastive, sparse-reward optimization that turns these targets into policy gradients.

What would settle it

Build or find an environment where two different underlying states yield the same text observation (for example, a container whose visible text does not change after its contents are swapped), and show that State2State training assigns success to the wrong state; or an environment where replay from the same initial configuration diverges, so many sampled targets are unreachable and the reward is noise. If the method's gains vanish in either setting, the verifier assumption is the load-bearing part.

Watch

Extended reading notes

Core claim

State2State claims that a large portion of agent skill can be acquired by solving self-generated state-reaching tasks. A random or lightly biased explorer collects reachable observations; invalid observations are filtered, diverse targets are sampled, and each candidate is replayed three times to confirm the state is reproducible from its initial configuration. The training prompt then says “reach this state”, and success is decided by the rule-based reward $r_t = \mathbf{1}[\mathrm{match}(o(s_t), o^*)]$, with normalized exact text match as the environment-specific verifier. Policies are optimized with GRPO plus dynamic sampling, which discards rollout groups whose outcomes are identical. The central empirical claim is that this stage transfers: it improves base models in most settings, improves both final performance and learning efficiency of downstream human-task RL, and transfers across environments better than training on human-specified ScienceWorld tasks.

Load-bearing premise

The whole reward signal rests on the assumption that a normalized exact string match between the current observation and the target observation correctly detects that the agent has reached the intended state, and that re-running a trajectory from the same initial configuration reliably reproduces the explored state.

Editorial extensions

If this is right

State2State can serve as a scalable mid-training stage that reduces the need for expert demonstrations and hand-written task curricula, because both objectives and rewards come from exploration.
Downstream RL becomes more sample-efficient: on ScienceWorld with Qwen3-4B, the full pipeline reaches a comparable score around step 50 that the task-only baseline reaches near step 150.
The full pipeline achieves the best performance on both ALFWorld and ScienceWorld at both model scales, including out-of-distribution splits.
State2State remains beneficial on top of supervised fine-tuning and with a stronger RL backbone such as GiGPO, so it is complementary to existing agent-training ingredients.
Environment-derived mid-training transfers across environments: ScienceWorld State2State before ALFWorld RL improves ALFWorld success beyond using ScienceWorld human-task RL for the same number of steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the textual observation is a lossy rendering of the underlying state, exact-match targets will systematically favor states that differ in text and may under-sample states whose differences are hidden; environments with richer hidden dynamics would likely need a learned or fuzzy state-equivalence matcher.
The exploration policy is a natural control knob: this paper shows random exploration beats an LLM explorer, but in larger or sparser state spaces a hierarchy that broadens then deepens exploration might combine State2State's reproducibility filter with more targeted state coverage.
The same recipe may transfer to web, software-engineering, or real-device control whenever a reproducible reset and a state matcher exist; a direct test would compare State2State mid-training against trajectory-imitation pretraining on equal compute in those domains.
The “latent benefits released by downstream RL” result suggests that state-reaching performance alone understates what the stage teaches, so an evaluation metric that probes manipulation breadth rather than just task success could make those gains visible before human-task RL.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The core idea is clean: randomly explore an environment, collect reachable observations, turn them into RL objectives where the agent must reproduce a target observation, and use that as mid-training before task-specific RL. The specific combination—random exploration, state-reaching RL with exact-match verification, then downstream human-task RL—is not in the cited prior work, and the paper correctly credits the underlying goal-relabeling idea to HER. That's the right way to position novelty.

What the paper does well: the method description is thorough, the appendices give real detail on data construction and prompts, and the limitations section is candid. The cross-environment transfer result (ScienceWorld mid-training -> ALFWorld RL) is the most interesting result because it compares two mid-training objectives with the same step budget. The standalone S2S gains over the base model are also evidence that state-reaching objectives teach something transferable.

Where it falls short: the main comparison in Table 1 and the efficiency claim in Figure 3 are confounded by compute. The full pipeline gets 80 GRPO steps on state-reaching tasks before downstream RL starts; the RL-only baseline gets zero. If downstream runs are all 400 steps, then S2S+RL has 480 total optimizer steps in the same environment. The stress-test note is right: 'reaches the same score at step 50 instead of step 150' is partly an accounting artifact (80+50 is close to 150). The paper explicitly says it is not a compute-matched replacement, which is honest, but then uses Figure 3 to claim learning efficiency, which undercuts that caveat. The cross-environment table partially controls for extra compute, but still doesn't rule out that any additional RL in a related environment helps. On top of that, the main results are single runs without error bars, and RL at this scale is noisy. Code and data are promised but not yet available.

None of this is fatal. The direction is consistent across two model scales and multiple environments, and the authors have identified a real bottleneck. But the load-bearing claim—that the environment-derived objective, rather than extra training compute, drives the gains—is not yet established. A compute-matched control (e.g., RL-only for 480 steps, or a warm-up on a random objective) plus a couple of seeds would settle it.

Who is this for: people working on agentic RL or self-supervised skill acquisition. They'll get a useful method and a fair discussion of its limits. I would cite it, and I'd send it to review with a strong request for the control above.

Referee Report

4 major / 5 minor

Summary. The paper proposes STATE2STATE, an environment-derived mid-training method for LLM agents. It uses random exploration to collect reachable environment observations, filters and replays them to form reproducible state-reaching tasks, and trains the agent with GRPO using a rule-based observation-matching reward. The resulting policy is then used as initialization for downstream RL on standard human-annotated tasks. Experiments on ALFWorld and ScienceWorld with Qwen3-4B and Qwen3-8B compare standalone STATE2STATE, direct human-task RL, and STATE2STATE+RL, and also include analyses of SFT ordering, RL backbone choice, exploration strategy, cross-environment transfer, and a MobileWorld GUI extension. The central claim is that environment-derived state-reaching objectives provide useful, scalable skill priors that improve final downstream task performance and learning efficiency.

Significance. If the headline claims hold, the paper makes a useful and timely contribution: it demonstrates a scalable, verifiable, and supervision-light way to obtain agentic skill priors, complementing SFT and task-specific RL. The strengths include the simplicity of random exploration plus exact-match verification, the large and carefully documented constructed datasets, the explicit acknowledgment of limitations, and the inclusion of several controlled ablations (SFT ordering, backbone, exploration strategy). The method is not circular: training targets are generated by random exploration and verified independently of downstream human-task evaluation. However, the main empirical evidence is currently not clean: the full pipeline receives 80 additional RL steps before downstream training, so the reported gains do not isolate the state-reaching objective from extra training compute, and all RL results are single runs without error bars. These issues must be addressed before the central claims can be accepted.

major comments (4)

[§4.2, §E.1, Table 1, Fig. 3] The headline comparison does not control for training compute. STATE2STATE+RL runs 80 GRPO steps on state-reaching tasks before downstream human-task RL, whereas RL-only starts from the base model and is trained for the same fixed number of downstream steps (300 for ALFWorld, 400 for ScienceWorld, per §E.1). The full pipeline therefore has 80 additional optimizer steps and additional environment rollouts, so the gains in Table 1 and the faster early curve in Fig. 3 could reflect extra warm-up on the same environment rather than the state-reaching objective specifically. Please add a compute-matched control: give RL-only the same 80 warm-up steps on the same environments with a non-state-reaching auxiliary reward (or with repeated human-task data), and also report total optimizer steps including mid-training on the x-axis of Fig. 3. Without this, the central claim that environment-derived state-reaching objectives improve downstream RL is not isolated.
[§5.4, Table 5] The cross-environment comparison is similarly confounded. ScienceWorld STATE2STATE and ScienceWorld human-task RL are matched in 'steps of mid-training,' but the two training signals are not matched in reward density, trajectory length, or update-to-rollout ratio; in particular, the state-reaching runs use dynamic sampling (§C.2), which changes the number of environment interactions per gradient step. As a result, the better ALFWorld transfer cannot be attributed to the state-reaching objective. A control that matches total environment interactions and total gradient steps while varying only the objective is needed; for example, mid-training on an equal number of reward-dense, non-goal exploration rollouts.
[Tables 1, 3, 4, 5 and Fig. 3] All RL results are reported as single runs without seeds, error bars, or confidence intervals. GRPO on sparse binary rewards is high-variance, and many differences are small relative to likely noise: for example, +2.56 points on ALFWorld OOD for Qwen3-4B, +3.87 points on ScienceWorld average for Qwen3-8B, and +0.033 on 117 MobileWorld tasks in Table 6. Each condition should be run with at least three random seeds and reported as mean ± std, or with a paired bootstrap over evaluation tasks. This is load-bearing for the claimed improvements in final performance, learning efficiency, and ordering conclusions in Table 2.
[§F.3, Table 6] The MobileWorld reward is not derived purely from the environment: it combines XML fuzzy sequence similarity and SSIM with hand-chosen weights (0.8 vs. 0.2), and the ScienceWorld exploration split into spatial and state-changing phases is also a manually specified prior. To support the claim that STATE2STATE requires no handcrafted reward components in GUI environments, the paper should ablate or justify these choices, or restrict the claim to text environments. In addition, the reported +0.033 improvement on a single run is within a range that needs uncertainty quantification before it can be interpreted as evidence of standalone improvement.

minor comments (5)

[Fig. 3] The x-axis label 'Training Steps' is ambiguous: specify whether it counts downstream steps only or total optimizer steps including the 80-step STATE2STATE phase.
[§E.1] The sentence 'with maximum prompt length between 12000 and response length 1024' appears to be missing a token count for the prompt; please clarify.
[Table 3] GiGPO is only defined in Appendix C.3; a one-sentence description in the main text would improve readability.
[§B.3] The ScienceWorld Case 3 example labeled 'find-animal; action feedback' shows an action-feedback observation rather than a target state; this is confusing and should be relabeled or replaced.
[§F.3] Clarify whether the SSIM-based visual reward at training time is computed on the resized 540×1200 images only, since evaluation uses original-size images; this discrepancy should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the state-reaching objective and the downstream human-task evaluation are independent, and no fitted parameter or self-citation chain makes the reported result equal to its inputs.

full rationale

The derivation chain is self-contained and not circular. STATE2STATE constructs training tasks by sampling explored observations and verifying them with an exact-match reward (Eq. 1); this reward is the training objective itself, not a prediction about downstream performance. The downstream claim—improved task success on ALFWorld and ScienceWorld human tasks—is evaluated with the standard human-task success signals, which are not defined in terms of the state-matching reward. No parameter is fitted to the evaluation set and then reported as a prediction. The comparisons to RL-only are confounded by the extra 80 state-reaching GRPO steps, and the paper explicitly frames STATE2STATE as a mid-training stage 'rather than a compute-matched replacement'; this is a compute-match and experimental-control issue, not a circular reduction, because extra warm-up steps are not the same quantity as the claimed environment-derived capability. The cross-environment comparison in Table 5 states that the baselines use the same number of mid-training steps, and even if the step accounting is imperfect, the ALFWorld evaluation metric remains external to the ScienceWorld state-matching reward. Citations to GRPO, DAPO, GiGPO, and Agent Early Experience are external prior work; no load-bearing claim rests solely on a self-citation or on an author-imported uniqueness theorem. The method's reliance on reproducible states and observation matching is a stated assumption about the verifier, not an assumption of the conclusion.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical assumptions about environment reproducibility and observation fidelity, plus standard RL background. The free parameters are reward and exploration tuning choices, not fitted constants that drive the main downstream result. No new entities are postulated.

free parameters (3)

MobileWorld XML reward weight = 0.8
Hand-chosen weight for structural XML similarity in the combined GUI state-matching reward (Appendix F.3).
MobileWorld SSIM reward weight = 0.2
Hand-chosen complement to the XML weight in the same GUI reward (Appendix F.3).
ScienceWorld exploration phase split = 20% navigation / 80% manipulation
Hand-chosen sampling prior for the random explorer that defines target state diversity (Section 3.2, Appendix B.1).

assumptions (4)

domain assumption Environments are reproducible from a fixed initial configuration via seeding or replay
Required for replay-3x target filtering and for training rollouts to start from the explored initial state (Section 3.1, Appendix A).
domain assumption The textual observation fully determines the task-relevant state, so normalized exact match is a valid success verifier
Section 3.1 states 'observations provide a reliable proxy for verifying task-relevant state equivalence'; the reward in Eq. 1 depends on this.
domain assumption Random exploration with phase or verb priors yields sufficiently diverse and useful target states
The paper deliberately chooses random over LLM exploration to maximize diversity and shows it works better in Table 4, but this is an assumption about the target distribution.
standard math GRPO with dynamic sampling is a valid optimization backbone with known convergence properties
The paper uses GRPO (Shao et al. 2024) and DAPO dynamic sampling (Yu et al. 2025) as black-box optimizers (Section 3.4, Appendix C).

how reviews work

0 comments

Cite this review

Pith. "Pith review of State2State: Environment-Derived Mid-Training for LLM Agents." pith.science (2026). https://pith.science/paper/LHW2ZNT3

@misc{pith2026260804934,
  author       = {Pith},
  title        = {Pith review of: State2State: Environment-Derived Mid-Training for LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LHW2ZNT3}},
  note         = {Machine review of arXiv:2608.04934}
}

read the original abstract

Training LLM agents commonly relies on supervised fine-tuning from expert trajectories or online reinforcement learning over human-specified tasks with handcrafted verifiers. Though effective, both remain bottlenecked by externally specified tasks and supervision signals, limiting the scalability and diversity of agent training. We study an environment learning paradigm in which agents acquire interaction and manipulation capabilities solely through environment interaction, without externally specified tasks. We propose State2State, an environment-derived mid-training method that converts explored environment states into training objectives, challenging agents to reach a specified target state. By deriving tasks from environment exploration and verifying success through rule-based state matching, State2State provides scalable and verifiable training objectives without expert supervision or manual task design. Experiments on ALFWorld and ScienceWorld show that State2State improves agent performance as a standalone environment-learning stage in most settings. As initialization for downstream RL, it further improves final performance and learning efficiency, with promising evidence of cross-environment generalization.

Figures

Figures reproduced from arXiv: 2608.04934 by the authors.

**Figure 2.** Overview of STATE2STATE, an environment-derived mid-training method. STATE2STATE converts explored environment states into reproducible training objectives and trains agents with rule-based state-matching rewards. By sampling targets from explorations, STATE2STATE aims to expose agents to a different and potentially broader set of environment states, not limited by human-task distributions. ration toward states simi… view at source ↗

**Figure 3.** Training curves for Qwen3-4B on Science [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Representative MobileWorld STATE2STATE target states used for environment-derived training. The model observes a target screenshot and learns to reach a matching GUI state through multi-turn interaction. The examples cover system settings, visual search, shopping, and …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

[3]

9 Musen Lin, Minghao Liu, Taoran Lu, Lichen Yuan, Yi- wei Liu, Haonan Xu, Yu Miao, Yuhao Chao, and Zhaojian Li

Tongyi deepresearch technical report.CoRR, abs/2510.24701. 9 Musen Lin, Minghao Liu, Taoran Lu, Lichen Yuan, Yi- wei Liu, Haonan Xu, Yu Miao, Yuhao Chao, and Zhaojian Li. 2025. Gui-rewalk: Massive data gen- eration for GUI agent via stochastic exploration and intent-aware reasoning.CoRR, abs/2509.15738. Weize Liu, Minghui Liu, Sy-Tuyen Ho, Souradip Chakra...

arXiv 2025
[4]

InF orty-second International Conference on Ma- chine Learning, ICML 2025, V ancouver , BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research

Self-improving language models for evolution- ary program synthesis: A case study on ARC-AGI. InF orty-second International Conference on Ma- chine Learning, ICML 2025, V ancouver , BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net. Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Jiadai Sun, Xinyue Ya...

arXiv 2025
[6]

InProceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 3062–3077

SAND: boosting LLM agents with self-taught action deliberation. InProceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 3062–3077. Association for Com- putational Linguistics. Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei ...

arXiv 2025
[7]

Xiao Yu, Baolin Peng, Ruize Xu, Yelong Shen, Pengcheng He, Suman Nath, Nikhil Singh, Jiangfeng Gao, and Zhou Yu

DAPO: an open-source LLM reinforcement learning system at scale.CoRR, abs/2503.14476. Xiao Yu, Baolin Peng, Ruize Xu, Yelong Shen, Pengcheng He, Suman Nath, Nikhil Singh, Jiangfeng Gao, and Zhou Yu. 2026. Reinforcement world model learning for llm-based agents.CoRR, abs/2602.05842. Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen...

arXiv 2026
[12]

action":

You can explore the environment and find the items you need to complete the experiment. **4. Your thinking process should be concise, do not waste time on irrelevant details.** Your current task is: {task_description} You are now at step {current_step} and your current observation is: { current_observation} Now it's your turn to take an action. You should...

work page
[13]

You can use look around to see the surroundings of the current location

work page
[14]

All containers in the environment have already been opened, you can directly get items from the containers

work page
[15]

You can explore the environment and find the items you need to complete the experiment. **4. Your thinking process should be concise, do not waste time on irrelevant details.** The target state is: {target_state} Now it's your turn to take an action. You should first reason step-by-step about the current situation(Your thinking process should be concise a...

work page 2025

Show all 13 references

[507]

Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han

ACM. Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.CoRR, abs/2503.09516. Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chengli...

2025 arXiv
[2021]

Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 oth- ers

OpenReview.net. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 oth- ers. 2025. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267. Liangcai Su, Zhen Zhang, Guangyu Li, Zhuo C...

2025 arXiv
[2023]

action":

inference engine for high-throughput RL training. We optimize the actor with learning rate 1e−6, enable KL loss with coefficient 0.01, and do not add KL penalties directly into the environment reward. We use GRPO-style RL algorithm in both training phases. We set the train bat...
[2025]

Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jian- Guang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan

Group-in-group policy optimization for LLM agent training.CoRR, abs/2505.10978. Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jian- Guang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. 2025. Agentgen: Enhancing planning abilities for large language model based agent via envir...

2025 arXiv
[3077]

Association for Computational Linguistics. Ji Zeng, Dayuan Fu, Tiantian Mi, Yumin Zhuang, Yax- ing Huang, Xuefeng Li, Lyumanshan Ye, Muhang Xie, Qishuo Hua, Zhen Huang, Mohan Jiang, Han- ning Wang, Jifan Lin, Yang Xiao, Jie Sun, Yunze Wu, and Pengfei Liu. 2026. davinci-dev: Ag...

2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [3]

9 Musen Lin, Minghao Liu, Taoran Lu, Lichen Yuan, Yi- wei Liu, Haonan Xu, Yu Miao, Yuhao Chao, and Zhaojian Li

Tongyi deepresearch technical report.CoRR, abs/2510.24701. 9 Musen Lin, Minghao Liu, Taoran Lu, Lichen Yuan, Yi- wei Liu, Haonan Xu, Yu Miao, Yuhao Chao, and Zhaojian Li. 2025. Gui-rewalk: Massive data gen- eration for GUI agent via stochastic exploration and intent-aware reasoning.CoRR, abs/2509.15738. Weize Liu, Minghui Liu, Sy-Tuyen Ho, Souradip Chakra...

arXiv 2025

[2] [4]

InF orty-second International Conference on Ma- chine Learning, ICML 2025, V ancouver , BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research

Self-improving language models for evolution- ary program synthesis: A case study on ARC-AGI. InF orty-second International Conference on Ma- chine Learning, ICML 2025, V ancouver , BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net. Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Jiadai Sun, Xinyue Ya...

arXiv 2025

[3] [6]

InProceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 3062–3077

SAND: boosting LLM agents with self-taught action deliberation. InProceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 3062–3077. Association for Com- putational Linguistics. Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei ...

arXiv 2025

[4] [7]

Xiao Yu, Baolin Peng, Ruize Xu, Yelong Shen, Pengcheng He, Suman Nath, Nikhil Singh, Jiangfeng Gao, and Zhou Yu

DAPO: an open-source LLM reinforcement learning system at scale.CoRR, abs/2503.14476. Xiao Yu, Baolin Peng, Ruize Xu, Yelong Shen, Pengcheng He, Suman Nath, Nikhil Singh, Jiangfeng Gao, and Zhou Yu. 2026. Reinforcement world model learning for llm-based agents.CoRR, abs/2602.05842. Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen...

arXiv 2026

[5] [12]

action":

You can explore the environment and find the items you need to complete the experiment. **4. Your thinking process should be concise, do not waste time on irrelevant details.** Your current task is: {task_description} You are now at step {current_step} and your current observation is: { current_observation} Now it's your turn to take an action. You should...

work page

[6] [13]

You can use look around to see the surroundings of the current location

work page

[7] [14]

All containers in the environment have already been opened, you can directly get items from the containers

work page

[8] [15]

You can explore the environment and find the items you need to complete the experiment. **4. Your thinking process should be concise, do not waste time on irrelevant details.** The target state is: {target_state} Now it's your turn to take an action. You should first reason step-by-step about the current situation(Your thinking process should be concise a...

work page 2025