Pith. sign in

REVIEW 4 major objections 4 minor 33 references

A two-level agent that compiles mastered goals into reusable skills learns efficiently in open-ended, compositional environments.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

HERAKLES couples a language-model planner to a small, continually retrained skill executor and outperforms three baselines on the 17-goal Crafter benchmark, scaling better to reworded and repeated goals.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely new concurrent skill-compilation architecture with a real win in Crafter, but the 'open-ended' and 'sample-efficient' claims outrun the evidence, and the paper has an internal horizon inconsistency that needs fixing. the 4 major comments →

arxiv 2508.14751 v2 pith:VPBXWEPG submitted 2025-08-20 cs.LG

HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agents

Thomas Carta , Cl\'ement Romac , Loris Gaven , Pierre-Yves Oudeyer , Olivier Sigaud , Sylvain Lamprier This is my paper
classification cs.LG
keywords hierarchical reinforcement learningopen-ended learningLLM agentsskill compilationgoal-conditioned RLcompositional generalizationCrafterautotelic agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HERAKLES is an architecture for goal-conditioned agents in environments where harder goals build on earlier ones. It pairs an LLM that chooses which subgoal to attempt next with a small neural controller that turns mastered subgoals into directly callable skills. The paper argues that this concurrent training lets an agent keep learning harder goals without restarting from scratch, and reports that in the Crafter world it reaches a difficulty-weighted score of 39.1 after 250,000 steps, well ahead of three comparison methods. It also reports that the language-structured skill space transfers to unseen goals, including repeated-collection tasks and a goal never seen in training. The reason this matters is that open-ended learning is blocked precisely by the combinatorial growth of goal complexity, and this is a mechanism aimed at amortizing that growth.

Core claim

The paper's central claim is that an open-ended agent should learn two policies at once: a high-level LLM that decomposes a goal into subgoals, and a low-level controller that executes those subgoals and progressively absorbs full successful trajectories as direct skills. As the low-level controller masters a goal, that goal becomes a new callable option for the high-level LLM, shortening the chain needed for harder goals. The authors formalize this as a bilevel coupling between the two policies and use a learned competency estimator to let the LLM propose only subgoals the low-level controller can plausibly execute. Empirically, they show that this compilation loop reaches a Crafter score o

What carries the argument

The central mechanism is the compilation loop: every successful hierarchical trajectory for a goal is replayed to the low-level policy as goal-conditioned data, so the goal becomes a skill the high-level policy can invoke directly. Around this loop sits a learned competency estimator that filters the subgoals the LLM is allowed to propose, keeping the planner focused on what the executor can actually do, and a training signal that penalizes high-level decisions to encourage temporally extended, compiled behavior rather than step-by-step reasoning.

Load-bearing premise

The load-bearing premise is that a goal's natural-language description decomposes in the same way the environment actually composes behaviors: if a goal's wording names prerequisites, those prerequisites are real and reachable milestones; when language structure and the environment's true state-transition structure diverge, the LLM cannot pick workable subgoals and the whole compilation loop loses its advantage.

What would settle it

Construct a Crafter variant where each goal's name is produced by shuffling the prerequisite names of unrelated achievements while leaving the real achievement tree intact, then compare HERAKLES against itself with truthful goal names; if scores stay high, language compositionality is not carrying the mechanism, and if they collapse, the method's advantage depends on language tracking true prerequisites.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is right, increasing goal complexity no longer forces linearly growing learning time, because mastered goals are amortized as reusable primitives.
  • Skill libraries would grow automatically from experience, removing the need for expert-defined skills or pre-trained low-level policies.
  • Natural-language descriptions could serve as a practical subgoal space for hierarchical reinforcement learning wherever goal structure is language-accessible.
  • The reported Crafter results imply a concrete gap in sample efficiency: 39.1 versus 27.6 for the best baseline after 250,000 steps.
  • The reported generalization results imply that compositional and synonymically rephrased goals can be handled without additional training, with the smallest degradation among the compared methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the mechanism should be most valuable when the goal space has an unbounded or shifting frontier; on a small fixed goal set, the overhead of training a second policy may not pay for itself.
  • Editorial inference: a testable extension is to replace the hand-built achievement tree with goals generated by another LLM; the paper's own language-compositionality assumption predicts the same compilation should work if generated goal descriptions track true prerequisites.
  • Editorial inference: the competency-filtering design suggests a general principle for LLM agents in other domains, such as robotics or tool use: constrain the planner to propose only actions whose executor has demonstrated competence, and expand that constraint set as skills are acquired.
  • Editorial inference: the experiments leave open how much of the generalization comes from the LLM's compositional priors versus the low-level distillation; an ablation with randomly chosen subgoals would separate the two.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes HERAKLES, a hierarchical goal-conditioned RL agent for partially observable environments with large, structured goal spaces. A Mistral-7B high-level policy selects subgoals from a set filtered by a learned competence estimator, while a 2M-parameter low-level policy executes them and is trained on both subgoal segments and full compiled trajectories. The authors evaluate in a modified Crafter environment over 250k steps against POAD, FUN, and BOSS, reporting a higher final Crafter score (39.1 vs 27.6 / 23.6 / 7.8) and generalization results on compositional, synonymous, and trajectory-similar goals. The method is documented in detail across appendices, with hyperparameters and a claimed code release.

Significance. If the results hold, HERAKLES is a useful contribution: it demonstrates concurrent training of an LLM high-level policy and a small low-level policy with progressive skill compilation, and it provides a concrete instantiation of language-guided hierarchical RL without expert-defined skills. The strengths include 5-seed runs, held-out evaluation, detailed ablations (e.g., the multigoal buffer mixing ablation in Fig. 12), and an unusually complete appendix. However, the central claims of sample efficiency and open-ended adaptation go beyond the current evidence, and the reliance on a hand-built language-aligned goal tree is not stress-tested.

major comments (4)
  1. [§4.2, Figure 3] The paper asks 'Is HERAKLES more sample-efficient than prior HRL approaches or LLM-only strategies?' and the conclusion repeats 'improves sample efficiency.' But Figure 3 shows baselines progressing faster early and HERAKLES surpassing them only near the end of the 250k-step budget. As stated, this is not sample efficiency in the usual sense. Please define the intended meaning, report e.g. area-under-the-curve or steps-to-threshold, and qualify the conclusion, or the abstract/conclusion will overclaim.
  2. [§3.1, §5] The mechanism rests on the assumption that linguistic compositionality corresponds to skill compositionality, i.e., that goal names in natural language track the prerequisite structure of the environment. This is load-bearing: the high-level LLM can only propose effective subgoals if this correspondence holds. The paper evaluates only in Crafter, where the achievement tree (Appendix A.1) is hand-built so that goal names transparently encode prerequisites. The conclusion mentions that the goal space is externally provided but does not acknowledge that the language–skill alignment itself is engineered. To support the 'open-ended' claim, either test in a setting where this mapping is imperfect or explicitly restrict the claim to goal spaces with language-aligned prerequisite structure.
  3. [Table 1, §4.3] The generalization claims are weakened by large standard deviations. For example, on the unseen goal 'make wood sword', HERAKLES reports 0.40 ± 0.37 against BOSS 0.33 ± 0.05; on the training goal 'make wood pickaxe', HERAKLES reports 0.47 ± 0.27 against POAD 0.75 ± 0.07. With 5 seeds, these differences are not established. Please provide per-seed data or significance tests, and avoid statements like 'HERAKLES suffers the least' without statistical support.
  4. [§4.2 vs Table 1] The text states that BOSS's most complex skill reached by one seed is 'place table', yet Table 1 reports BOSS success on 'make wood pickaxe' (0.37 ± 0.08) and 'make wood sword' (0.33 ± 0.05), which are more complex than placing a table. Clarify what 'reached' means and reconcile this with the evaluation numbers; otherwise the BOSS comparison is difficult to interpret.
minor comments (4)
  1. [Appendix A.2 vs Table 6] There is an inconsistency in the temporal limits: Appendix A.2 says the high-level controller is allowed up to 96 steps and low-level up to 64 steps, while Table 6 reports high-level max steps 64 and low-level max steps 128. Please align these numbers; this matters for reproducibility.
  2. [Eq. (4), Appendix F.2] The indicator notation `1 n(g)>2496` and `1 SR(g)>beta` should use explicit set/indicator notation, and the relationship between the epsilon floor here and the dynamic exploration term in Appendix F.5 should be clarified.
  3. [Appendix B.2] The term 'n-compositional' is used both for repeating a goal n times (Table 4) and for a goal like 'collect 4 woods' (Table 1). Please define the evaluation protocol precisely so the reader knows whether this is a single goal with a larger count or a repeated-goal curriculum.
  4. [Appendix C.2] The text says the code is released with a link, but no link is visible in the provided manuscript. Please ensure the link appears in the published version.

Circularity Check

0 steps flagged

No significant circularity; the paper's core results are empirical comparisons on held-out goals, and its assumptions are stated rather than derived from the conclusions.

full rationale

The derivation chain does not make any prediction that reduces to its inputs by construction. The hierarchical setup defines a bilevel objective (Eqs. 1–2) as a coupled fixed point, which is a formalization, not a self-fulfilling result. The adaptive subgoal filter (Section 3.4) does create a feedback loop: the competence estimator Cθk decides which subgoals are offered, and the low-level policy is trained on the offered subgoals; however, the paper explicitly identifies this loop and inserts an exploration floor ε to break it ('The exploration parameter ε avoids a self-reinforcing loop...'). This is an acknowledged bootstrap/curriculum mechanism, not a claim that a fitted quantity is being predicted. The main empirical claims are genuine held-out evaluations: the generalization targets 'collect 4 woods' and 'make wood sword' are not in the 17-goal training set and are not scored by the competence estimator, so their success rates measure actual transfer rather than reconstruction from fitted parameters. The assumption of a correspondence between linguistic compositionality and skill compositionality (Section 3.1) is stated as an assumption, not derived from the results; its validity in other environments is an external-validity concern, not circularity. The only self-citation ([8]) is a technical detail for LLM constrained decoding and is not load-bearing. Hence score 0.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The method introduces no new physical entities. Its central claim depends on a set of hand-chosen update thresholds, a competence-gated option admission rule with a tuned exploration floor, and the linguistic-to-skill compositionality assumption. The free parameters above are the ones that directly shape skill admission and low-level distillation.

free parameters (6)
  • Subgoal inclusion exploration floor epsilon = min(update_frequency_g, 0.1)
    Controls p_g = max(C, epsilon) for admitting a goal into the callable option set in Section 3.4. Appendix F.5 says a dynamic version was chosen because it outperformed a fixed epsilon in empirical evaluations.
  • Low-level update thresholds beta and L = beta = 0.1, L unspecified in text
    Equation (4) only admits a goal to the option set if its stored success rate exceeds beta and if it has been updated more than L times. These are hand-chosen scheduling constants.
  • Low-level update probability constants = minimum 2496 new transitions, exponential decay in N_ll(g)
    Equation (3) schedules low-level updates using hand-set thresholds and a decay term to avoid over-training easy goals.
  • Multigoal buffer mix fraction = 0.2
    Appendix E.3 mixes 20% transitions from other goals during low-level updates to mitigate catastrophic forgetting; an ablation shows the choice changes the final score from 10 to 15.
  • Compiled failed trajectory length filter = l < N_LL / (1 - LP_hierarchical)
    Appendix E.4 prunes long unrewarded compiled trajectories based on hierarchical learning progress; this filter is required to make low-level distillation work.
  • Competence estimator update cadence = update every 256 transitions, one epoch
    Appendix F.4 states these two hyperparameters were selected based on empirical performance, so they are fitted tuning choices supporting the main mechanism.
axioms (5)
  • domain assumption The goal space has a hidden precedence relation such that every goal trajectory decomposes into a prerequisite prefix and a suffix.
    Invoked in Section 3.1 to justify latent decomposition into subgoals; not learned or verified beyond the hand-built Crafter tree.
  • ad hoc to paper Linguistic compositionality in natural-language goal descriptions corresponds to skill compositionality in the environment.
    Section 3.1: 'assuming a correspondence between linguistic compositionality ... and skill compositionality'. This is the load-bearing assumption that makes LLM subgoal selection useful.
  • domain assumption The goal space and sparse reward functions are provided externally and do not need to be generated or discovered.
    Stated in Section 3.1 and acknowledged in the conclusion: the method handles the learning loop but not goal generation.
  • domain assumption A pretrained LLM can propose or likelihood-rank valid subgoal descriptions under constrained decoding.
    Section 3.4 defines the high-level policy as an LLM restricted to valid goal descriptions; this rests on pretrained language competence.
  • domain assumption A small neural policy trained with AWR on relabeled trajectories can internalize behaviors compiled from hierarchical rollouts.
    Section 3.3 and Appendix E.2 assume off-policy behavioral compilation works with sparse binary rewards and critic value substitution for failures.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agents." pith.science (2026). https://pith.science/paper/VPBXWEPG

@misc{pith2026250814751,
  author       = {Pith},
  title        = {Pith review of: HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VPBXWEPG}},
  note         = {Machine review of arXiv:2508.14751}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We study goal-conditioned reinforcement learning in partially observable environments with sparse rewards and large, structured goal spaces. In such settings, complex goals often require composing simpler skills, but learning these compositions efficiently remains challenging. This difficulty is particularly relevant in open-ended exploration settings, where agents are exposed to increasingly complex goals over time and must continuously expand and reorganize their skill repertoire. We assume the goal space admits prerequisite relations, enabling latent decomposition of tasks into subgoals, and leverage language to represent and reason over these goals. To exploit this structure, we propose HERAKLES, a hierarchical agent that jointly learns a high-level LLM policy and a low-level controller. The high-level policy selects subgoals among those the low-level can reliably achieve, while the low-level executes them and progressively compiles successful behaviors into reusable skills. Both policies are trained concurrently: the high-level guides exploration and structures behavior, while the low-level distills trajectories into efficient goal-conditioned skills. As training progresses, more goals become directly executable, enabling scalable skill composition. This results in a hybrid system combining planning and execution, improving efficiency and adaptation in open-ended, compositional environments.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 22 canonical work pages · 2 internal anchors

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny ...

    work page

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances, 2022

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, K...

    Pith/arXiv arXiv 2022

  3. [3]

    Hierarchical reinforcement learning with natural language subgoals

    Arun Ahuja, Kavya Kopparapu, Rob Fergus, and Ishita Dasgupta. Hierarchical reinforcement learning with natural language subgoals, 2023. URLhttps://arxiv.org/abs/2309.11564

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    Hindsight experience re- play

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 30. ...

    work page 2017

  5. [5]

    The option-critic architecture.Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb

    Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture.Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017. doi: 10.1609/aaai.v31i1.10916. URL https://ojs.aaai.org/index.php/AAAI/article/view/10916

    doi:10.1609/aaai.v31i1.10916 2017

  6. [6]

    Human-timescale adaptation in an open-ended task space, 2023

    Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley- Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rockt...

    Pith/arXiv arXiv 2023

  7. [7]

    Integration of action and language knowledge: A roadmap for developmental robotics.Autonomous Mental Development, IEEE Transactions on, 2:167 – 195, 10 2010

    Angelo Cangelosi, Giorgio Metta, Gerhard Sagerer, Stefano Nolfi, Chrystopher Nehaniv, Kerstin Fischer, Jun Tani, Tony Belpaeme, Giulio Sandini, Francesco Nori, Luciano Fadiga, Britta Wrede, Katharina Rohlfing, Elio Tuci, Kerstin Dautenhahn, Joe Saunders, and Arne Zeschel. Integration of action and language knowledge: A roadmap for developmental robotics.Aut...

    work page arXiv 2010

  8. [8]

    Grounding large language models in interactive environments with online reinforcement learning

    Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,ICML, volume 202 ofProceedings of Machine Lea...

    work page 2023

  9. [9]

    PARL: A unified framework for policy alignment in reinforcement learning from human feedback

    Souradip Chakraborty, Amrit Singh Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang. PARL: A unified framework for policy alignment in reinforcement learning from human feedback. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://ope...

    work page 2024

  10. [10]

    Learning phrase representations using RNN encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078, 2014. URLhttp://arxiv.org/abs/1406.1078

    Pith/arXiv arXiv 2014

  11. [11]

    Language and culture internalization for human-like autotelic ai.Nature Machine Intelligence, 4(12):1068–1076, December

    Cédric Colas, Tristan Karch, Clément Moulin-Frier, and Pierre-Yves Oudeyer. Language and culture internalization for human-like autotelic ai.Nature Machine Intelligence, 4(12):1068–1076, December

    work page

  12. [12]

    Feudal reinforcement learning

    Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In S. Hanson, J. Cowan, and C. Giles, editors,Advances in Neural Information Processing Systems, volume 5. Morgan- Kaufmann, 1992. URL https://proceedings.neurips.cc/paper_files/paper/1992/file/ d14220ee66aeec73c49038385428ec4c-Paper.pdf

    work page 1992

  13. [13]

    Guiding pretraining in reinforcement learning with large language models

    Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th International Conference ...

    work page 2023

  14. [14]

    Intrinsically motivated open-ended multi-task learning using transfer learning to discover task hierarchy.Applied Sciences, 11(3):975, January 2021

    Nicolas Duminy, Sao Mai Nguyen, Junshuai Zhu, Dominique Duhaut, and Jerome Kerdreux. Intrinsically motivated open-ended multi-task learning using transfer learning to discover task hierarchy.Applied Sciences, 11(3):975, January 2021. ISSN 2076-3417. doi: 10.3390/app11030975. URL http://dx.doi. org/10.3390/app11030975. 11

    work page doi:10.3390/app11030975 2021

  15. [31]

    Adaptations:V(s 1;z) is not pretrained in our set up but learned during training on collected trajectories

    Initial skill sampling.At the start of each episode, a skill z2Z is sampled proportionally to the pre-trained value functionV(s 1;z) , favouring skills that are likely to succeed from the current state s1. Adaptations:V(s 1;z) is not pretrained in our set up but learned during training on collected trajectories. For elementary actions (e.g. move left) tha...

    work page

  16. [32]

    This is repeatedN times; the actual next skill is drawn from the distribution of LLM-assigned token likelihoods, encouraging diversity

    LLM-guided skill chaining.After a successful skill execution, a large language model (LLM) is prompted with the current skill repertoire and the skills executed so far, and proposes the next skill in natural language. This is repeatedN times; the actual next skill is drawn from the distribution of LLM-assigned token likelihoods, encouraging diversity. The...

    work page

  17. [33]

    make wood pickaxe

    Skill library expansion.At the end of each episode, the collected experience is added to the replay buffer with a sparse reward of 1 per completed sub-skill. The same LLM is used to generate composite 33 language instructions describing multi-step chains (e.g., “make wood pickaxe”); these composite instructions and associated trajectories are appended to ...

    work page

  18. [54]

    Sutton, Doina Precup, and Satinder Singh

    Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artif. Intell., 112(1-2):181–211, 1999. URL http: //dblp.uni-trier.de/db/journals/ai/ai112.html#SuttonPS99

    work page 1999

  19. [55]

    Fundamental processes in sensorimotor learning: Reasoning, refinement, and retrieval.eLife, 13:e91839, aug 2024

    Jonathan S Tsay, Hyosub E Kim, Samuel D McDougle, Jordan A Taylor, Adrian Haith, Guy Avraham, John W Krakauer, Anne GE Collins, and Richard B Ivry. Fundamental processes in sensorimotor learning: Reasoning, refinement, and retrieval.eLife, 13:e91839, aug 2024. ISSN 2050-084X. doi: 10.7554/eLife.91839. URLhttps://doi.org/10.7554/eLife.91839

    work page doi:10.7554/elife.91839 2024

  20. [56]

    A. M. Turing. Computing machinery and intelligence.Mind, 59(236):433–460, 1950. ISSN 00264423. URLhttp://www.jstor.org/stable/2251299

    arXiv 1950

  21. [57]

    On the planning abilities of large language models (a critical investigation with a proposed benchmark),

    Karthik Valmeekam, Sarath Sreedharan, Matthew Marquez, Alberto Olmo, and Subbarao Kambhampati. On the planning abilities of large language models (a critical investigation with a proposed benchmark),

    work page

  22. [58]

    Feudal networks for hierarchical reinforcement learning, 2017

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning, 2017. URL https://arxiv.org/abs/1703.01161

    Pith/arXiv arXiv 2017

  23. [59]

    Will we run out of data? limits of llm scaling based on human-generated data, 2024

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of llm scaling based on human-generated data, 2024. URL https: //arxiv.org/abs/2211.04325

    Pith/arXiv arXiv 2024

  24. [60]

    V oyager: An open-ended embodied agent with large language models, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models, 2023. URL https://arxiv.org/abs/2305.16291

    Pith/arXiv arXiv 2023

  25. [61]

    Chinese Poetry Generation with Planning based Neural Network

    Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. Chinese poetry generation with planning based neural network.CoRR, abs/1610.09889, 2016. URL http://arxiv.org/ abs/1610.09889

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  26. [62]

    Reinforcing llm agents via policy optimization with action decomposition

    Muning Wen, Ziyu Wan, Jun Wang, Weinan Zhang, and Ying Wen. Reinforcing llm agents via policy optimization with action decomposition. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 103774–103805. Curran Associates, Inc., 2024. URL https://pro...

    work page 2024

  27. [63]

    Coca: Contrastive captioners are image-text foundation models, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022. URL https://arxiv.org/abs/2205. 01917

    work page 2022

  28. [64]

    Omni: Open-endedness via models of human notions of interestingness

    Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness. InThe Twelfth International Conference on Learning Representations, 2023

    work page 2023

  29. [65]

    Bootstrap your own skills: Learning to solve new tasks with large language model guidance

    Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. In7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id= a0mFRgadGO

    work page 2023

  30. [66]

    a tree is three steps north

    Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. Visualize before you write: Imagination-guided open-ended text generation, 2023. URL https: //arxiv.org/abs/2210.03765. 15 Appendices This supplementary material provides additional results, discussion, and implementation details. • Section A details our enviro...

    Pith/arXiv arXiv 2023

  31. [2022]

    doi: 10.1038/s42256-022-00591-4

    ISSN 2522-5839. doi: 10.1038/s42256-022-00591-4. URL http://dx.doi.org/10.1038/ s42256-022-00591-4

    work page doi:10.1038/s42256-022-00591-4

  32. [2023]

    URLhttps://arxiv.org/abs/2302.06706

    Pith/arXiv arXiv

  33. [2024]

    URLhttps://arxiv.org/abs/2303.08774

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.