{"id":"3a706eef-8827-4cd4-a912-68c34f266c01","arxiv_id":"2508.14751","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"HERAKLES couples a language-model planner to a small, continually retrained skill executor and outperforms three baselines on the 17-goal Crafter benchmark, scaling better to reworded and repeated goals.","lead":"The paper presents HERAKLES, an agent that splits planning and acting between a language model and a small neural network, letting the small network turn solved goals into reusable skills. It reports better results on a Minecraft-like crafting benchmark than three comparison agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Linguistic-compositionality correspondence is untested in any environment where language does not mirror the achievement tree; the central open-ended claim may rest on this engineered alignment.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I find: the method's reliance on linguistic compositionality tracking actual prerequisite structure. This is the core mechanism that differentiates HERAKLES from prior HRL and LLM-only methods, and it is only validated in Crafter, where the goal descriptions were deliberately designed to match the achievement tree. The paper does not test a scenario where language is misaligned with state-transition prerequisites, leaving open the possibility that the reported gains are an artifact of this engineered correspondence. This concern is more fundamental than the horizon inconsistency (64 vs 128) or the large variance in generalization results, which are reproducibility and evidence-strength issues but do not directly threaten the internal logic of the method. I agree with the reader's conditional verdict: the paper provides a plausible mechanism and a solid demonstration in a single environment, but the central claim about open-endedness requires evidence from a setting where the correspondence between language and task structure is not hand-crafted. The proposed permutation test would directly settle whether the linguistic-compositionality assumption is load-bearing. No ad hominem is intended; the paper is honest about its limitations, but the missing stress test of its key assumption prevents full acceptance.","tokens_in":23862,"tokens_out":4700,"duration_ms":51287,"concrete_test":"Permute the 17 goal names in Crafter so that language no longer matches the actual prerequisite structure (e.g., 'make wood pickaxe' now refers to a goal requiring a furnace), keeping the achievement tree, rewards, and observations identical. Retrain HERAKLES and BOSS for 250k steps with the same hyperparameters. If HERAKLES's advantage over BOSS vanishes or becomes statistically indistinguishable, the linguistic-compositionality correspondence is load-bearing; if it persists, the method discovers prerequisites despite the mismatch, supporting the open-ended claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.1 assumes a hidden precedence relation over G and 'a correspondence between linguistic compositionality ... and skill compositionality.' The entire method depends on this: the high-level LLM proposes subgoals from textual prompts, and the competency filter admits only goals the low-level can achieve. If language does not track the actual state-transition prerequisites, the LLM will propose decompositions that cannot be chained to the target goal, and the mechanism loses its advantage. The paper evaluates only in Crafter, where the goal tree (Appendix A.1) was hand-built so that prerequisite relations are transparent in the goal names (e.g., 'make wood pickaxe' requires 'place table' and wood). No experiment varies or breaks this correspondence. The conclusion notes the goal space is externally provided, but this limitation is not stress-tested. If the advantage of HERAKLES over POAD/BOSS comes from this engineered alignment rather than from a general skill-compilation mechanism, the central claim of improved adaptation in open-ended, compositional environments is unsupported beyond the specific testbed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HERAKLES, a hierarchical goal-conditioned RL agent for partially observable environments with large, structured goal spaces. A Mistral-7B high-level policy selects subgoals from a set filtered by a learned competence estimator, while a 2M-parameter low-level policy executes them and is trained on both subgoal segments and full compiled trajectories. The authors evaluate in a modified Crafter environment over 250k steps against POAD, FUN, and BOSS, reporting a higher final Crafter score (39.1 vs 27.6 / 23.6 / 7.8) and generalization results on compositional, synonymous, and trajectory-similar goals. The method is documented in detail across appendices, with hyperparameters and a claimed code release.","tokens_in":24098,"tokens_out":7571,"duration_ms":91022,"significance":"If the results hold, HERAKLES is a useful contribution: it demonstrates concurrent training of an LLM high-level policy and a small low-level policy with progressive skill compilation, and it provides a concrete instantiation of language-guided hierarchical RL without expert-defined skills. The strengths include 5-seed runs, held-out evaluation, detailed ablations (e.g., the multigoal buffer mixing ablation in Fig. 12), and an unusually complete appendix. However, the central claims of sample efficiency and open-ended adaptation go beyond the current evidence, and the reliance on a hand-built language-aligned goal tree is not stress-tested.","major_comments":[{"comment":"The paper asks 'Is HERAKLES more sample-efficient than prior HRL approaches or LLM-only strategies?' and the conclusion repeats 'improves sample efficiency.' But Figure 3 shows baselines progressing faster early and HERAKLES surpassing them only near the end of the 250k-step budget. As stated, this is not sample efficiency in the usual sense. Please define the intended meaning, report e.g. area-under-the-curve or steps-to-threshold, and qualify the conclusion, or the abstract/conclusion will overclaim.","section":"§4.2, Figure 3"},{"comment":"The mechanism rests on the assumption that linguistic compositionality corresponds to skill compositionality, i.e., that goal names in natural language track the prerequisite structure of the environment. This is load-bearing: the high-level LLM can only propose effective subgoals if this correspondence holds. The paper evaluates only in Crafter, where the achievement tree (Appendix A.1) is hand-built so that goal names transparently encode prerequisites. The conclusion mentions that the goal space is externally provided but does not acknowledge that the language–skill alignment itself is engineered. To support the 'open-ended' claim, either test in a setting where this mapping is imperfect or explicitly restrict the claim to goal spaces with language-aligned prerequisite structure.","section":"§3.1, §5"},{"comment":"The generalization claims are weakened by large standard deviations. For example, on the unseen goal 'make wood sword', HERAKLES reports 0.40 ± 0.37 against BOSS 0.33 ± 0.05; on the training goal 'make wood pickaxe', HERAKLES reports 0.47 ± 0.27 against POAD 0.75 ± 0.07. With 5 seeds, these differences are not established. Please provide per-seed data or significance tests, and avoid statements like 'HERAKLES suffers the least' without statistical support.","section":"Table 1, §4.3"},{"comment":"The text states that BOSS's most complex skill reached by one seed is 'place table', yet Table 1 reports BOSS success on 'make wood pickaxe' (0.37 ± 0.08) and 'make wood sword' (0.33 ± 0.05), which are more complex than placing a table. Clarify what 'reached' means and reconcile this with the evaluation numbers; otherwise the BOSS comparison is difficult to interpret.","section":"§4.2 vs Table 1"}],"minor_comments":[{"comment":"There is an inconsistency in the temporal limits: Appendix A.2 says the high-level controller is allowed up to 96 steps and low-level up to 64 steps, while Table 6 reports high-level max steps 64 and low-level max steps 128. Please align these numbers; this matters for reproducibility.","section":"Appendix A.2 vs Table 6"},{"comment":"The indicator notation `1 n(g)>2496` and `1 SR(g)>beta` should use explicit set/indicator notation, and the relationship between the epsilon floor here and the dynamic exploration term in Appendix F.5 should be clarified.","section":"Eq. (4), Appendix F.2"},{"comment":"The term 'n-compositional' is used both for repeating a goal n times (Table 4) and for a goal like 'collect 4 woods' (Table 1). Please define the evaluation protocol precisely so the reader knows whether this is a single goal with a larger count or a repeated-goal curriculum.","section":"Appendix B.2"},{"comment":"The text says the code is released with a link, but no link is visible in the provided manuscript. Please ensure the link appears in the published version.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution but the advertised scope (open-ended, sample-efficient) is broader than the evidence. I would not reject: the authors can either add experiments that vary the language–skill alignment or carefully qualify the claims. The BOSS contradiction and the N_HL/N_LL hyperparameter inconsistency should also be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something new: it trains an LLM high-level policy and a lightweight low-level policy concurrently, using a competence estimator to admit only mastered goals as callable options and compiling successful hierarchical trajectories into the low-level policy as reusable skills. That combination is not in the cited prior work, and the empirical result in Crafter is substantial—HERAKLES reaches a Crafter score of 39.1 versus 27.6 for BOSS and 23.6 for POAD, and its compositional generalization (e.g., collect 4 woods at 0.91 success, make wood sword at 0.40) is clearly better than the baselines. The appendices are thick and mostly honest, with 5-seed runs, per-task curves, and ablations. I credit that they tried to show what is happening rather than just reporting a leaderboard. The soft spots are real but not fatal. First, “sample efficiency” is the wrong label: baselines progress faster early, and HERAKLES only overtakes them later. It is a final-performance and compositional-reuse win, not an efficiency win. Second, only one environment, a fixed 17-goal tree, is tested. The stress-test worry about the linguistic-compositionality correspondence is legitimate: the method assumes that natural-language composition tracks the actual prerequisite structure, and Crafter's goal names were hand-built to make that true. No experiment varies or breaks that alignment, so the “open-ended” claim is a template, not a demonstrated general mechanism. The authors do acknowledge the externally provided goal space, but they do not stress-test it. Third, there is an internal inconsistency: the low-level horizon is 64 steps in the environment description (Appendix A.2) and 128 steps in the hyperparameter table (Appendix I). That matters for implementation and reproducibility, and it should be fixed before publication. Also, the code release is asserted but not verifiable from the provided materials, and some generalization numbers have large variance (make wood sword 0.40 ± 0.37). Overall, this is a serious paper. It shows clear thinking, engages the relevant literature, and the mechanism is plausible and self-consistent on its own terms. The limitations are the kind a good referee can work with. I would send it to peer review, with a request that the authors fix the horizon inconsistency, add at least one environment that does not have language-aligned prerequisites, and soften the wording on sample efficiency and open-endedness.","headline":"A genuinely new concurrent skill-compilation architecture with a real win in Crafter, but the 'open-ended' and 'sample-efficient' claims outrun the evidence, and the paper has an internal horizon inconsistency that needs fixing.","tokens_in":697,"tokens_out":1762,"would_cite":true,"duration_ms":39495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-level agent that compiles mastered goals into reusable skills learns efficiently in open-ended, compositional environments.","keywords":["hierarchical reinforcement learning","open-ended learning","LLM agents","skill compilation","goal-conditioned RL","compositional generalization","Crafter","autotelic agents"],"falsifier":"Construct a Crafter variant where each goal's name is produced by shuffling the prerequisite names of unrelated achievements while leaving the real achievement tree intact, then compare HERAKLES against itself with truthful goal names; if scores stay high, language compositionality is not carrying the mechanism, and if they collapse, the method's advantage depends on language tracking true prerequisites.","tokens_in":23737,"feed_emoji":"🤖","tokens_out":6250,"duration_ms":74867,"temperature":0.7,"pith_summary":"HERAKLES is an architecture for goal-conditioned agents in environments where harder goals build on earlier ones. It pairs an LLM that chooses which subgoal to attempt next with a small neural controller that turns mastered subgoals into directly callable skills. The paper argues that this concurrent training lets an agent keep learning harder goals without restarting from scratch, and reports that in the Crafter world it reaches a difficulty-weighted score of 39.1 after 250,000 steps, well ahead of three comparison methods. It also reports that the language-structured skill space transfers to unseen goals, including repeated-collection tasks and a goal never seen in training. The reason this matters is that open-ended learning is blocked precisely by the combinatorial growth of goal complexity, and this is a mechanism aimed at amortizing that growth.","feed_headline":"Compiled skills push open-ended agent to Crafter score 39.1","feed_subtitle":"LLM planner plus skill-learning controller reaches unseen and repeated-collection goals where baselines stall.","key_machinery":"The central mechanism is the compilation loop: every successful hierarchical trajectory for a goal is replayed to the low-level policy as goal-conditioned data, so the goal becomes a skill the high-level policy can invoke directly. Around this loop sits a learned competency estimator that filters the subgoals the LLM is allowed to propose, keeping the planner focused on what the executor can actually do, and a training signal that penalizes high-level decisions to encourage temporally extended, compiled behavior rather than step-by-step reasoning.","core_discovery":"The paper's central claim is that an open-ended agent should learn two policies at once: a high-level LLM that decomposes a goal into subgoals, and a low-level controller that executes those subgoals and progressively absorbs full successful trajectories as direct skills. As the low-level controller masters a goal, that goal becomes a new callable option for the high-level LLM, shortening the chain needed for harder goals. The authors formalize this as a bilevel coupling between the two policies and use a learned competency estimator to let the LLM propose only subgoals the low-level controller can plausibly execute. Empirically, they show that this compilation loop reaches a Crafter score o","pith_inferences":["Editorial inference: the mechanism should be most valuable when the goal space has an unbounded or shifting frontier; on a small fixed goal set, the overhead of training a second policy may not pay for itself.","Editorial inference: a testable extension is to replace the hand-built achievement tree with goals generated by another LLM; the paper's own language-compositionality assumption predicts the same compilation should work if generated goal descriptions track true prerequisites.","Editorial inference: the competency-filtering design suggests a general principle for LLM agents in other domains, such as robotics or tool use: constrain the planner to propose only actions whose executor has demonstrated competence, and expand that constraint set as skills are acquired.","Editorial inference: the experiments leave open how much of the generalization comes from the LLM's compositional priors versus the low-level distillation; an ablation with randomly chosen subgoals would separate the two."],"forward_implications":["If the central claim is right, increasing goal complexity no longer forces linearly growing learning time, because mastered goals are amortized as reusable primitives.","Skill libraries would grow automatically from experience, removing the need for expert-defined skills or pre-trained low-level policies.","Natural-language descriptions could serve as a practical subgoal space for hierarchical reinforcement learning wherever goal structure is language-accessible.","The reported Crafter results imply a concrete gap in sample efficiency: 39.1 versus 27.6 for the best baseline after 250,000 steps.","The reported generalization results imply that compositional and synonymically rephrased goals can be handled without additional training, with the smallest degradation among the compared methods."],"supporting_citations":[{"why":"Supplies the Crafter environment, its 17-goal achievement tree, and the difficulty-weighted score metric used for all comparisons.","marker":"[24]"},{"why":"Provides the modified Crafter version and observation setup that isolate compositional goal learning from survival dynamics.","marker":"[13]"},{"why":"Supplies the learning-progress goal sampler and the competence-estimator design the paper adapts for adaptive subgoal filtering.","marker":"[21]"},{"why":"Defines the constrained-decoding probability rule that turns the language model into a subgoal-selection policy.","marker":"[8]"},{"why":"Supplies the on-policy RL algorithm used to train the high-level policy and serves as the non-hierarchical comparison baseline.","marker":"[62]"},{"why":"Supplies the off-policy algorithm that distills hierarchical trajectories into direct low-level skills.","marker":"[44]"},{"why":"The comparison system that chains skills bottom-up from a pre-trained library rather than compiling them online.","marker":"[65]"},{"why":"The comparison hierarchical baseline that proposes subgoals in a latent embedding space instead of language.","marker":"[58]"},{"why":"Anchors the assumption that natural-language compositionality tracks skill compositionality, the load-bearing premise for subgoal selection.","marker":"[2]"},{"why":"Reinforces the same assumption and provides prior evidence that natural-language subgoals make hierarchical RL compositional and generalizable.","marker":"[3]"}],"fun_headline_variants":["Goal decomposition and skill compilation lift Crafter to 39.1","Hierarchical agent: LLM selects subgoals, low-level compiles them","Skill compilation loop lets LLM agent reach higher Crafter","Bilevel training: LLM plans, controller executes, skills accumulate","Agent compiles mastered skills into options for harder goals"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that a goal's natural-language description decomposes in the same way the environment actually composes behaviors: if a goal's wording names prerequisites, those prerequisites are real and reachable milestones; when language structure and the environment's true state-transition structure diverge, the LLM cannot pick workable subgoals and the whole compilation loop loses its advantage.","fun_headline_variants_meta":{"raw":{"variants":["Goal decomposition and skill compilation lift Crafter to 39.1","Hierarchical agent: LLM selects subgoals, low-level compiles them","Skill compilation loop lets LLM agent reach higher Crafter","Bilevel training: LLM plans, controller executes, skills accumulate","Agent compiles mastered skills into options for harder goals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":2900,"prompt_tokens":732,"completion_tokens":2168,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":2079}},"tokens_in":476,"tokens_out":2168,"duration_ms":19351,"temperature":1.0,"reasoning_tokens":2079,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:18:31.208285+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a Crafter variant where each goal's name is produced by shuffling the prerequisite names of unrelated achievements while leaving the real achievement tree intact, then compare HERAKLES against itself with truthful goal names; if scores stay high, language compositionality is not carrying the mechanism, and if they collapse, the method's advantage depends on language tracking true prerequisites.","supporting_citations":[{"cited_title":"Bootstrap your own skills: Learning to solve new tasks with large language model guidance","cited_arxiv_id":null,"evidence_quote":"The comparison system that chains skills bottom-up from a pre-trained library rather than compiling them online."},{"cited_title":"Guiding pretraining in reinforcement learning with large language models","cited_arxiv_id":null,"evidence_quote":"Provides the modified Crafter version and observation setup that isolate compositional goal learning from survival dynamics."},{"cited_title":"Grounding large language models in interactive environments with online reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Defines the constrained-decoding probability rule that turns the language model into a subgoal-selection policy."},{"cited_title":"Reinforcing llm agents via policy optimization with action decomposition","cited_arxiv_id":null,"evidence_quote":"Supplies the on-policy RL algorithm used to train the high-level policy and serves as the non-hierarchical comparison baseline."},{"cited_title":"Hierarchical reinforcement learning with natural language subgoals","cited_arxiv_id":"2309.11564","evidence_quote":"Reinforces the same assumption and provides prior evidence that natural-language subgoals make hierarchical RL compositional and generalizable."}],"review_version":1}