Pith. sign in

REVIEW 5 major objections 6 minor 64 references

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An offline hierarchical RL framework makes LLM agents plan sub-tasks and act on them, beating non-hierarchical baselines on long-horizon benchmarks.

desk verdict A genuinely neat parameter-efficient hierarchy, undermined by an unvalidated LLM-based reward judge and missing statistics. read the letter →

arxiv 2505.19761 v1 pith:Z6XHRTTX submitted 2025-05-26 cs.AI

Zican Hu , Wei Liu , Xiaoye Qu , Xiangyu Yue , Chunlin Chen , Zhi Wang , Yu Cheng This is my paper
classification cs.AI
keywords hierarchicalreinforcementlearningofflineRLLLMagenttemporalabstractionsub-taskplanningScienceWorldALFWorldparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GLIDER claims that LLM agents for long-horizon interactive tasks improve when the policy is split into a high-level planner that proposes sub-tasks every c steps and a low-level executor that turns each sub-task into primitive actions. The paper argues that this divide-and-conquer structure creates temporal abstraction, making exploration easier and giving the agent a chain-of-thought-like decomposition to follow. Training proceeds offline in three stages: behavior cloning, offline RL fine-tuning with an actor-critic, and optional online fine-tuning of the high-level policy only. The payoff claimed is consistent gains over strong fine-tuning baselines on ScienceWorld and ALFWorld, with the largest margins on unseen tasks, plus fast adaptation to new tasks by reusing the low-level skills.

What carries the argument

The central object is the two-level hierarchical policy with temporal abstraction: every c environment steps the high-level policy proposes one sub-task goal gt, and the low-level policy executes c primitive actions against that goal, collecting sub-task-completion intrinsic reward. The training machinery is a three-stage pipeline of SFT (behavior cloning with length regularization), ORL (offline actor-critic; IQL-style asymmetric expectile value learning, and advantage-weighted actor updates), and O2O (online fine-tuning of the high level only while the low level is refreshed with offline demonstrations). What carries the argument is that shared parameters with a hierarchy prompt make the hierarchy cheap, and the intrinsic reward makes the low level task-agnostic enough to transfer.

What would settle it

If a head-to-head rerun of the exact Llama-3-8B tables (ScienceWorld seen/unseen, ALFWorld seen/unseen, and the three online adaptation curves) were run with the sub-task-completion judge replaced by an objective state-checker (or with a judge shown to be systematically noisy), and the margins over ETO collapsed, the paper's central mechanism would be identified as reward noise rather than hierarchy.

Watch

Extended reading notes

Core claim

The core claim is that a parameter-efficient hierarchy—where high-level planner and low-level executor share one LoRA-tuned LLM backbone differentiated only by a prompt—is a reliable way to lift LLM agent performance on long-horizon decision tasks. The high-level policy receives the task description and observation and emits a sub-task goal every c steps; the low-level policy then emits primitive actions for those c steps and receives a binary intrinsic reward indicating sub-task completion derived from the observation. Both policies are trained offline from a mixture of expert and medium-quality trajectories, first by behavior cloning and then by token-level advantage-weighted policy optimization with sentence-level critics. The paper reports that with Llama-3-8B, GLIDER reaches 77.43 on seen and 68.34 on unseen ScienceWorld tasks, versus 57.90 and 52.33 for ETO; it also reports consistent gains on ALFWorld and in ablations across backbones and model scales.

Load-bearing premise

Sub-task completion can be read off the observation reliably, and this reading needs no manual design, because the low-level policy is trained entirely on binary intrinsic rewards built from that judgement; the appendix shows the judgement is actually made by a hand-written LLM prompt.

Editorial extensions

If this is right

  • If the method works as claimed, hierarchical sub-task decomposition plus offline RL could be the default recipe for grounding LLM policies in sparse-reward, long-horizon environments.
  • Unseen tasks should inherit the low-level sub-task library, so generalization to new tasks becomes mostly a planning problem, not an action-generation problem.
  • Offline-to-online adaptation should need far fewer environment interactions than monolithic fine-tuning, since only the high-level policy must be re-adapted to a new reward.
  • Even smaller models (Llama-3B) are claimed to outperform larger non-hierarchical models, which would make hierarchical decomposition a partial substitute for model scale.
  • The ablation results imply that SFT alone is not enough, and that offline RL alone is better than SFT alone, so the main source of capability is the reinforcement fine-tuning stage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension suggested by the paper's own logic: if the low-level skills are truly task-agnostic, then a planner trained on one benchmark should bootstrap a low-level library trained on another, which the paper does not directly demonstrate.
  • The hierarchical structure may also function as an implicit reward-shaping mechanism: the binary sub-task completion signal gives the low-level policy dense feedback where environment rewards are sparse, which the non-hierarchical baselines lack.
  • The claim that sub-task completion is 'easily accessible from observation' would be stronger if it were verified across language-domains with adversarial distractors, where a verification LLM prompt could confidently misjudge completion.
  • The framework's authors have implicitly connected it to the idea of language-generality for sub-tasks (e.g., reusing 'monitor until done' patterns), which suggests a next step: measuring how much of the gain survives translation to environments whose observations do not naturally decompose into named sub-tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes GLIDER, a hierarchical LLM agent framework that pairs a high-level subtask planner with a low-level action executor sharing a single LoRA-tuned backbone, and trains it via behavior cloning, offline RL (IQL-style actor-critic), and optional offline-to-online fine-tuning. Experiments on ScienceWorld and ALFWorld report consistent gains over prompt-based and fine-tuned baselines across Mistral-7B, Gemma-7B, and Llama-3-8B, together with ablations that separate the effects of the hierarchy, the training stages, model scale, and the expert-to-medium data mixture ratio.

Significance. If the central claims hold, GLIDER offers a parameter-efficient recipe for injecting temporal abstraction into LLM-based agents, and the central ablation comparing hierarchical and non-hierarchical versions on the same pipeline and backbones is a well-designed test of the hierarchy hypothesis. The cross-backbone and cross-scale consistency are genuine strengths, and the release of code would support reproducibility. However, the paper's headline claims are currently undercut by an unvalidated LLM completion judge that supplies the low-level reward, by an internal inconsistency about freezing the low-level policy in offline-to-online adaptation, and by the absence of any statistical error analysis. These issues are load-bearing because they directly affect the attribution of the reported performance gains to the proposed hierarchy.

major comments (5)
  1. [Section 3.2, Eq. (2), Appendix D] The paper states that subtask completion 'can be easily accessible from the environment observation, without requiring any manual design or domain knowledge,' and Eq. (2) defines the low-level reward r-hat as this binary completion signal. Appendix D, however, shows that completion is determined by a hand-written 'Check Subtask Complete' LLM prompt that receives the subtask, initial observation, action sequence, and final observation. No accuracy statistics, failure-mode analysis, or ablation against ground-truth completion labels is provided. Since the low-level policy is trained entirely on r-hat through Eqs. (4)-(7), a noisy or gameable judge directly threatens the central hierarchy claim; at minimum, the revision should report judge agreement with ground-truth labels on a held-out sample and include an ablation that replaces the prompt judge with oracle completion labels.
  2. [Section 3.5 and Algorithm 1, line 12] The offline-to-online description says the task-agnostic low-level policy is frozen and only the high-level policy is fine-tuned, but Algorithm 1 says 'Fix low-level policy' and the Appendix A text then admits that parameter sharing prevents strict freezing, so low-level performance is instead maintained by continually training on offline demonstration data. This internal inconsistency undermines the attribution of the O2O results in Section 4.4 to frozen, transferable skills. Please specify the exact parameter update scheme (for example, which LoRA modules or prompts are updated) and report an ablation with truly frozen low-level parameters, or reinterpret the O2O claims accordingly.
  3. [Table 1, Figures 3-5] No evaluation seeds, error bars, confidence intervals, or significance tests are reported anywhere, despite the small evaluation sets shown in Table 3 (for example, 211 unseen ScienceWorld episodes). The headline comparison of Llama-3-8B GLIDER at 68.34 versus ETO at 52.33 could in principle be within run-to-run noise. Please run at least three to five seeds per condition and report mean and standard deviation, or provide pairwise significance tests; this is essential for the paper's repeated claim of 'consistent performance gains.'
  4. [Section 4.5 and Figure 5] The expert-to-medium 1:2 mixture ratio used for all main experiments appears to have been selected because it gives the best point estimate in Figure 5. If this ratio was chosen after inspecting test performance, the reported gains carry an optimism bias, and the claim that 1:2 is best needs a defined selection protocol or a held-out validation split. Moreover, without error bars, the differences among 68.34 (1:2), 60.31 (1:1), and 60.29 (1:5) may be noise; the revision should report variance and justify the selection procedure.
  5. [Section 3.2 and Appendix B] The claim that the framework eliminates 'the necessity for any manual or task-specific design' is not supported by the implementation: the subtask labels in expert demonstrations are produced by GPT-4 decomposition (Appendix B), the completion judge is a custom LLM prompt, and the hierarchy interval c is a hyperparameter. Please either soften these autonomy claims or provide evidence that these design choices are not task-specific.
minor comments (6)
  1. [Section 4.1 and Table 1] The baseline is called 'SwiftSage' in the related work but appears as 'SwitchSage' in Table 1 and elsewhere; please standardize the name.
  2. [Table 1] The \toggle-off and \toggle-on markup artifacts appear in the rendered table; these should be removed.
  3. [Appendix C, Table 5] The hyperparameter table lists 'batch size' twice with different values (64 and a per-device batch size of 2) and also gives gradient accumulation steps; please clarify the effective global batch size.
  4. [Appendix B, O2O task setups] The text describes the setting as 'Online-to-Offline' when it should be 'Offline-to-Online' to match Section 3.5.
  5. [References] The Konda and Tsitsiklis (1999) entry lists page range 75993-76005, which appears incorrect; please verify and correct the citation details.
  6. [Appendix D] The low-level prompt asks the model to 'determine if the subtask is completed (true/false),' while the completion reward is generated separately by the 'Check Subtask Complete' prompt; please clarify how these two completion signals relate and whether the judge is the sole source of r-hat.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: benchmark scores are externally measured; the LLM-judged low-level reward is an unvalidated training-signal issue, not a collapse of prediction into input.

full rationale

GLIDER's reported results are not circular. The main comparison (Table 1) is evaluated with ScienceWorld and ALFWorld environment rewards on held-out seen/unseen tasks; these scores are external to the training objective and cannot be derived from the model's own outputs by construction. The hierarchical decomposition is trained from GPT-4-annotated demonstration subtasks (Appendix B), and the low-level policy is trained with the intrinsic reward r-hat of Eq. (2). Appendix D shows that r-hat is implemented by a hand-written 'Check Subtask Complete' LLM prompt rather than by direct observation read-off, and no accuracy statistics or ablations against ground-truth completion labels are provided. This is a genuine internal self-referential loop—an LLM judge scores an LLM executor—and it undermines the Sec. 3.2 claim that completion is 'easily accessible from the environment observation... without requiring any manual design.' However, this does not make the benchmark predictions equivalent to the training inputs: the final scores are environment rewards, and the paper does not fit any parameter to the test-set scores. There are no load-bearing self-citations or imported uniqueness theorems. The central derivation (hierarchy + SFT + offline RL) is independent of the evaluation, so the circularity score is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The approach rests on hyperparameters (c, mixture ratio, expectile, length lambda), a GPT-4-generated sub-task label space, an LLM-based completion judge, and the untested assumption that one shared model can cleanly separate two policy roles.

free parameters (4)
  • Hierarchy interval c
    High-level decisions are made every c steps, but c is not given in the hyperparameter table; the temporal abstraction and high-level reward accumulation both depend on it.
  • Expert-to-medium data mixture ratio = 1:2
    Figure 5 shows 1:2 achieves the best unseen-task score (68.34) among tested ratios, then the main results adopt 1:2; selecting the best ratio from the same evaluation curve is post-hoc.
  • IQL expectile tau (Eq. 5)
    The asymmetric loss expectile is never listed in the hyperparameter table, yet it controls how conservative the value estimate is; an unreported, hand-chosen value.
  • Length regularization lambda (Eq. 3)
    The same symbol lambda is reused for advantage weighting in Eq. 7 (reported as 0.99), but the SFT length-regularization lambda value is not reported.
assumptions (4)
  • standard math The tasks are Markovian text MDPs, so the policy-gradient and Bellman updates apply unchanged.
    Section 3.1 formulates the agent as an MDP with text observations and actions.
  • domain assumption GPT-4-generated sub-task decompositions are a valid and sufficient label space for the high-level policy.
    Appendix B: 'we utilize GPT-4 to decompose these trajectories into subtasks', so the quality of the hierarchy is inherited from an external model.
  • domain assumption Sub-task completion (the low-level intrinsic reward) is accessible from the observation without manual design.
    Section 3.2 claims this, but Appendix D implements it with the Check Subtask Complete prompt, an additional LLM-based judge.
  • domain assumption Prompt-differentiated sharing of one actor lets high- and low-level roles be learned without interference.
    Section 3.2 relies on in-context learning; Appendix A concedes interference makes strict low-level freezing impossible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning." pith.science (2026). https://pith.science/paper/Z6XHRTTX

@misc{pith2026250519761,
  author       = {Pith},
  title        = {Pith review of: Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6XHRTTX}},
  note         = {Machine review of arXiv:2505.19761}
}
read the original abstract

While showing sophisticated reasoning abilities, large language models (LLMs) still struggle with long-horizon decision-making tasks due to deficient exploration and long-term credit assignment, especially in sparse-reward scenarios. Inspired by the divide-and-conquer principle, we propose an innovative framework **GLIDER** (**G**rounding **L**anguage Models as Eff**I**cient **D**ecision-Making Agents via Offline Hi**E**rarchical **R**einforcement Learning) that introduces a parameter-efficient and generally applicable hierarchy to LLM policies. We develop a scheme where the low-level controller is supervised with abstract, step-by-step plans that are learned and instructed by the high-level policy. This design decomposes complicated problems into a series of coherent chain-of-thought reasoning sub-tasks, providing flexible temporal abstraction to significantly enhance exploration and learning for long-horizon tasks. Furthermore, GLIDER facilitates fast online adaptation to non-stationary environments owing to the strong transferability of its task-agnostic low-level skills. Experiments on ScienceWorld and ALFWorld benchmarks show that GLIDER achieves consistent performance gains, along with enhanced generalization capabilities.

Figures

Figures reproduced from arXiv: 2505.19761 by the authors.

Figure 1
Figure 1. GLIDER’s hierarchical framework, showing sig [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the GLIDER framework. (a) Hierarchical Actor-Critic architecture with prompt-controlled high [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Ablation performance on unseen tasks in Sci [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Performance on unseen tasks in ScienceWorld [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 4
Figure 4. Figure 4: Online fine-tuning performance (score/100) of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Expert demonstration for w/o and w/ hierarchical structure in ScienceWorld. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Comparison of w/o and w/ hierarchical trajectories collected through in-distribution sampling in ScienceWorld. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Comparison of w/o and w/ hierarchical trajectories collected by cross-task generalization sampling in ScienceWorld. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Hierarchical decomposition reveals shared subtask patterns across two example tasks in ScienceWorld. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 48 canonical work pages

  1. [1]

    Do as I can, not as I say: Grounding language in robotic affordances

    Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do as I can, not as I say: Grounding language in robotic affordances. In Proceedings of Conference on Robot Learning, 2022

    work page 2022

  2. [2]

    The option-critic architecture

    Bacon, P.-L., Harb, J., and Precup, D. The option-critic architecture. In Proceedings of AAAI Conference on Artificial Intelligence, volume 31, 2017

    work page 2017

  3. [3]

    _0 : A vision-language-action flow model for general robot control

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. _0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    arXiv

  4. [4]

    D., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp.\ 1877--1901, 2020

    work page 1901

  5. [5]

    Exploration by random network distillation

    Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. In Proceedings of International Conference on Learning Representations, 2019

    work page 2019

  6. [6]

    FireAct : Toward language agent fine-tuning

    Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. FireAct : Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023

    arXiv 2023

  7. [7]

    AgentVerse : Facilitating multi-agent collaboration and exploring emergent behaviors

    Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y.-H., Qian, C., et al. AgentVerse : Facilitating multi-agent collaboration and exploring emergent behaviors. In Proceedings of International Conference on Learning Representations, 2024

    work page 2024

  8. [8]

    Dietterich, T. G. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of Artificial Intelligence Research, 13: 0 227--303, 2000

    work page 2000

Show all 64 references
  1. [9]

    M., Hao, B., and Van Roy, B

    Dwaracherla, V., Asghari, S. M., Hao, B., and Van Roy, B. Efficient exploration for llms. In Proceedings of International Conference on Machine Learning, pp.\ 12215–12227, 2024

    2024

  2. [10]

    Off-policy deep reinforcement learning without exploration

    Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Proceedings of International Conference on Machine Learning, pp.\ 2052--2062, 2019

    2019

  3. [11]

    Gandhi, K., Sadigh, D., and Goodman, N. D. Strategic reasoning with language models. arXiv preprint arXiv:2305.19165, 2023

    2023 arXiv

  4. [12]

    Deepseek-coder: When the large language model meets programming--the rise of code intelligence

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  5. [13]

    DeepSeek-R1 : Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., et al. DeepSeek-R1 : Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of International Conference on Machine Learning, pp.\ 1861--1870, 2018

    2018

  7. [15]

    J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., et al

    Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., et al. LoRA : Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations, 2022

    2022

  8. [16]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  9. [17]

    and Tsitsiklis, J

    Konda, V. and Tsitsiklis, J. Actor-critic algorithms. In Advances in Neural Information Processing Systems, volume 12, pp.\ 75993--76005, 1999

    1999

  10. [18]

    Offline reinforcement learning with fisher divergence critic regularization

    Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp.\ 5774--5783, 2021

    2021

  11. [19]

    Offline reinforcement learning with implicit Q -learning

    Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit Q -learning. In Proceedings of International Conference on Learning Representations, 2022

    2022

  12. [20]

    Conservative q-learning for offline reinforcement learning

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp.\ 1179--1191, 2020

    2020

  13. [21]

    Offline-to-online reinforcement learning via balanced replay and pessimistic Q -ensemble

    Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline-to-online reinforcement learning via balanced replay and pessimistic Q -ensemble. In Proceedings of Conference on Robot Learning, pp.\ 1702--1712, 2022

    2022

  14. [22]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  15. [23]

    Learning multi-level hierarchies with hindsight

    Levy, A., Konidaris, G., Platt, R., and Saenko, K. Learning multi-level hierarchies with hindsight. In Proceedings of International Conference on Learning Representations, 2019

    2019

  16. [24]

    Sub-policy adaptation for hierarchical reinforcement learning

    Li, A., Florensa, C., Clavera, I., and Abbeel, P. Sub-policy adaptation for hierarchical reinforcement learning. In Proceedings of International Conference on Learning Representations, 2020

    2020

  17. [25]

    Pre-trained language models for interactive decision-making

    Li, S., Puig, X., Paxton, C., Du, Y., Wang, C., Fan, L., Chen, T., Huang, D.-A., Aky \"u rek, E., Anandkumar, A., et al. Pre-trained language models for interactive decision-making. In Advances in Neural Information Processing Systems, volume 35, pp.\ 31199--31212, 2022

    2022

  18. [26]

    Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks

    Li, Z., Xie, Y., Shao, R., Chen, G., Jiang, D., and Nie, L. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  19. [27]

    Y., Fu, Y., Yang, K., Brahman, F., Huang, S., Bhagavatula, C., Ammanabrolu, P., Choi, Y., and Ren, X

    Lin, B. Y., Fu, Y., Yang, K., Brahman, F., Huang, S., Bhagavatula, C., Ammanabrolu, P., Choi, Y., and Ren, X. SwiftSage : a generative agent with fast and slow thinking for complex interactive tasks. In Advances in Neural Information Processing Systems, pp.\ 23813--23825, 2023

    2023

  20. [28]

    WizardCoder : Empowering code large language models with evol-instruct

    Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. WizardCoder : Empowering code large language models with evol-instruct. In Proceedings of International Conference on Learning Representations, 2024

    2024

  21. [29]

    Large language models play StarCraft II : Benchmarks and a chain of summarization approach

    Ma, W., Mi, Q., Zeng, Y., Yan, X., Wu, Y., Lin, R., Zhang, H., and Wang, J. Large language models play StarCraft II : Benchmarks and a chain of summarization approach. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  22. [30]

    Random latent exploration for deep reinforcement learning

    Mahankali, S., Hong, Z.-W., Sekhari, A., Rakhlin, A., and Agrawal, P. Random latent exploration for deep reinforcement learning. In Proceedings of International Conference on Machine Learning, pp.\ 34219–34252, 2024

    2024

  23. [31]

    Introducing Meta Llama 3 : The most capable openly available LLM to date, 2024

    Meta. Introducing Meta Llama 3 : The most capable openly available LLM to date, 2024. https://ai.meta.com/blog/meta-llama-3/

    2024

  24. [32]

    S., Lee, H., and Levine, S

    Nachum, O., Gu, S. S., Lee, H., and Levine, S. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, pp.\ 3307–3317, 2018

    2018

  25. [33]

    AWAC : Accelerating online reinforcement learning with offline datasets

    Nair, A., Gupta, A., Dalal, M., and Levine, S. AWAC : Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  26. [34]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp.\ 27730--27744, 2022

    2022

  27. [35]

    Agent planning with world knowledge model

    Qiao, S., Fang, R., Zhang, N., Zhu, Y., Chen, X., Deng, S., Jiang, Y., Xie, P., Huang, F., and Chen, H. Agent planning with world knowledge model. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  28. [36]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pp.\ 53728–53741, 2023

    2023

  29. [37]

    Vision-language models are zero-shot reward models for reinforcement learning

    Rocamonde, J., Montesinos, V., Nava, E., Perez, E., and Lindner, D. Vision-language models are zero-shot reward models for reinforcement learning. In Proceedings of International Conference on Learning Representations, 2024

    2024

  30. [38]

    LM-Nav : Robotic navigation with large pre-trained models of language, vision, and action

    Shah, D., Osi \'n ski, B., Levine, S., et al. LM-Nav : Robotic navigation with large pre-trained models of language, vision, and action. In Proceedings of Conference on Robot Learning, pp.\ 492--504, 2023

    2023

  31. [39]

    R., and Yao, S

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pp.\ 8634–8652, 2023

    2023

  32. [40]

    ALFWorld : Aligning text and embodied environments for interactive learning

    Shridhar, M., Yuan, X., Cote, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. ALFWorld : Aligning text and embodied environments for interactive learning. In Proceedings of International Conference on Learning Representations, 2021

    2021

  33. [41]

    J., Guez, A., Sifre, L., et al

    Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529 0 (7587): 0 484--489, 2016

    2016

  34. [42]

    Silver, D., Singh, S., Precup, D., and Sutton, R. S. Reward is enough. Artificial Intelligence, 299: 0 103535, 2021

    2021

  35. [43]

    V., Kostrikov, I., Su, Y., Yang, S., and Levine, S

    Snell, C. V., Kostrikov, I., Su, Y., Yang, S., and Levine, S. Offline RL for natural language generation with implicit language Q -learning. In Proceedings of International Conference on Learning Representations, 2023

    2023

  36. [44]

    Song, Y., Yin, D., Yue, X., Huang, J., Li, S., and Lin, B. Y. Trial and error: Exploration-based trajectory optimization for llm agents. In Proceedings of Annual Meeting of the Association for Computational Linguistics, 2024

    2024

  37. [45]

    S., Precup, D., and Singh, S

    Sutton, R. S., Precup, D., and Singh, S. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112 0 (1-2): 0 181--211, 1999

    1999

  38. [46]

    D., and Toshev, A

    Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Metcalf, R., Talbott, W., Mackraz, N., Hjelm, R. D., and Toshev, A. T. Large language models as generalizable policies for embodied tasks. In Proceedings of International Conference on Learning Representations, 2024

    2024

  39. [47]

    True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning

    Tan, W., Zhang, W., Liu, S., Zheng, L., Wang, X., and An, B. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In Proceedings of International Conference on Learning Representations, 2024

    2024

  40. [48]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  41. [49]

    K.-W., and Lim, E.-P

    Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., and Lim, E.-P. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of Annual Meeting of the Association for Computational Linguistics, pp.\ 9--14, 2023 a

    2023

  42. [50]

    ScienceWorld : Is your agent smarter than a 5th grader? In Proceedings of Empirical Methods in Natural Language Processing, pp.\ 11279--11298, 2022

    Wang, R., Jansen, P., C \^o t \'e , M.-A., and Ammanabrolu, P. ScienceWorld : Is your agent smarter than a 5th grader? In Proceedings of Empirical Methods in Natural Language Processing, pp.\ 11279--11298, 2022

    2022

  43. [51]

    Learning from failure: Integrating negative examples when fine-tuning large language models as agents

    Wang, R., Li, H., Han, X., Zhang, Y., and Baldwin, T. Learning from failure: Integrating negative examples when fine-tuning large language models as agents. arXiv preprint arXiv:2402.11651, 2024

    2024 arXiv

  44. [52]

    Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning

    Wang, S., Yang, Q., Gao, J., Lin, M., Chen, H., Wu, L., Jia, N., Song, S., and Huang, G. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pp.\ 47081–47104, 2023 b

    2023

  45. [53]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp.\ 24824--24837, 2022

    2022

  46. [54]

    and Jennings, N

    Wooldridge, M. and Jennings, N. R. Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10 0 (2): 0 115--152, 1995

    1995

  47. [55]

    The rise and potential of large language model based agents: A survey

    Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023

    2023 arXiv

  48. [56]

    Language agents with reinforcement learning for strategic play in the werewolf game

    Xu, Z., Yu, C., Fang, F., Wang, Y., and Wu, Y. Language agents with reinforcement learning for strategic play in the werewolf game. In Proceedings of International Conference on Machine Learning, pp.\ 55434–55464, 2024

    2024

  49. [57]

    L., Cao, Y., and Narasimhan, K

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, pp.\ 11809–11822, 2023 a

    2023

  50. [58]

    ReAct : Synergizing reasoning and acting in language models

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct : Synergizing reasoning and acting in language models. In Proceedings of International Conference on Learning Representations, 2023 b

    2023

  51. [59]

    and Zhang, X

    Yu, Z. and Zhang, X. Actor-critic alignment for offline-to-online reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, pp.\ 40452--40474, 2023

    2023

  52. [60]

    AgentTuning : Enabling generalized agent abilities for LLMs

    Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y., and Tang, J. AgentTuning : Enabling generalized agent abilities for LLMs . arXiv preprint arXiv:2310.12823, 2023

    2023 arXiv

  53. [61]

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning

    Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  54. [62]

    V., and Chi, E

    Zhou, D., Sch \"a rli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., and Chi, E. H. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of International Conference on Learning Representations, 2023

    2023

  55. [63]

    ArCHer : Training language model agents via hierarchical multi-turn rl

    Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. ArCHer : Training language model agents via hierarchical multi-turn rl. In Proceedings of International Conference on Machine Learning, pp.\ 62178–62209, 2024

    2024

  56. [64]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.