{"id":"96ab8335-e6a5-47af-9d48-e5ed337e07c6","arxiv_id":"2602.10085","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CODE-SHARP autonomously grows an archive of hierarchical reward programs via foundation models to train generalist RL agents that outperform baselines by up to 6x on long-horizon tasks in Craftax and XLand.","lead":"CODE-SHARP uses foundation models to automatically discover and evolve a growing library of Python programs that serve as hierarchical reward signals, allowing reinforcement learning agents to master complex skills from scratch in environments like Craftax without human-designed rewards. A smart generalist might read it to understand progress toward more autonomous AI systems that can expand their own capabilities in novel settings.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Autonomy of SHARP generation and routing depends on unexamined FM output validity and lack of post-hoc selection","rationale":"The reader's weakest_assumption directly identifies the same point. Because the paper positions itself as removing human-in-the-loop engineering, any unstated curation step is load-bearing for the central claim of open-ended autonomous evolution. The concrete test isolates that step without requiring new environments or baselines.","tokens_in":1841,"tokens_out":356,"duration_ms":25050,"concrete_test":"Extract the exact SHARP generation loop and runtime routing pseudocode from §3–4; re-execute the discovery phase on Craftax-Classic for 10k FM calls while logging every generated program, its syntactic validity, prerequisite graph acyclicity, and whether it was inserted or discarded; if >20% are filtered or if removing the filter drops diamond-mining success rate below 50% of reported value, the autonomy claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline results (6x median on Craftax-Classic, diamond mining capability, 90+ SHARPs enabling zero-shot long-horizon solving) rest on the assumption that the FM produces executable, non-redundant hierarchical reward programs whose prerequisite chains yield incremental, learnable signals. If generation involves any validity checks, rejection sampling, or manual review before insertion into the archive, or if runtime routing silently drops invalid chains, then the 'fully autonomously' and 'no pre-defined rewards' claims are not supported by the reported experiments. The abstract and generation paragraph do not quantify acceptance rate, cycle detection, or redundancy pruning, leaving open the possibility that performance derives from a filtered subset rather than raw open-ended discovery.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces CODE-SHARP, a framework that uses foundation models to continuously and autonomously discover and evolve an archive of Skills as Hierarchical Reward Programs (SHARPs). Each SHARP is a Python program encoding a local success condition together with a chain of prerequisite SHARPs; at runtime the agent is routed through the chain and receives incremental rewards only for the marginal behavior introduced by the new program. The method is evaluated on Craftax-Classic, XLand, and Craftax-Extended, where it reports 6× and 2.6× median performance gains over prior work, the first successful iron-tool and diamond-mining agents, and zero-shot long-horizon solving with >90 discovered SHARPs that matches ground-truth-reward performance.","tokens_in":2016,"tokens_out":520,"duration_ms":37407,"significance":"If the autonomy and validity claims are substantiated, the work would constitute a meaningful step toward open-ended, reward-free skill acquisition in reinforcement learning. The hierarchical program representation and dynamic routing mechanism offer a concrete route to scaling generalist agents on long-horizon tasks without hand-crafted reward functions.","major_comments":[{"comment":"Abstract: the headline performance claims (6× median on Craftax-Classic, 2.6× on XLand, unique iron-tool and diamond-mining capability) are presented without any report of the number of independent runs, statistical significance tests, variance across seeds, or the precise baseline implementations and hyper-parameters used for comparison. These details are required to assess whether the reported gains are robust.","section":"Abstract"},{"comment":"Abstract and paragraph on SHARP generation and runtime routing: the central claim that training occurs 'fully autonomously' and 'without any pre-defined rewards' rests on the assumption that the foundation model produces executable, non-redundant SHARPs whose prerequisite chains yield useful incremental signals. The manuscript provides no acceptance rate, cycle-detection procedure, redundancy-pruning method, or verification that invalid programs are never inserted into the archive; without these quantities it is impossible to rule out that performance derives from an implicitly curated subset rather than raw open-ended discovery.","section":"Abstract and paragraph on SHARP generation and runtime routing"}],"minor_comments":[{"comment":"The acronym SHARP is used in the abstract before its expansion; a parenthetical definition on first use would improve immediate readability.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback. We provide point-by-point responses to the major comments and indicate the revisions we plan to incorporate in the updated manuscript.","responses":[{"response":"We agree with this observation. The current abstract highlights key results but omits important statistical details. In the revised version, we will update the abstract to include the number of independent runs (e.g., 5 seeds), report median and interquartile ranges, and mention that statistical significance was assessed using appropriate tests. We will also add a table or section detailing baseline implementations and hyperparameters to ensure reproducibility and robustness assessment.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the headline performance claims (6× median on Craftax-Classic, 2.6× on XLand, unique iron-tool and diamond-mining capability) are presented without any report of the number of independent runs, statistical significance tests, variance across seeds, or the precise baseline implementations and hyper-parameters used for comparison. These details are required to assess whether the reported gains are robust."},{"response":"We appreciate the referee highlighting the need for more details on the autonomy mechanisms. While the framework operates without human intervention after initialization, we recognize that explicit descriptions of filtering processes are necessary. In the revision, we will add a new subsection under Methods describing the SHARP validation pipeline, including acceptance rates observed during experiments, cycle detection via topological sorting on the prerequisite graph, redundancy pruning based on program equivalence checks, and runtime verification that only valid, executable programs are added to the archive. This will substantiate that the performance gains stem from the open-ended discovery process.","revision_made":"yes","referee_comment":"[Abstract and paragraph on SHARP generation and runtime routing] Abstract and paragraph on SHARP generation and runtime routing: the central claim that training occurs 'fully autonomously' and 'without any pre-defined rewards' rests on the assumption that the foundation model produces executable, non-redundant SHARPs whose prerequisite chains yield useful incremental signals. The manuscript provides no acceptance rate, cycle-detection procedure, redundancy-pruning method, or verification that invalid programs are never inserted into the archive; without these quantities it is impossible to rule out that performance derives from an implicitly curated subset rather than raw open-ended discovery."}],"tokens_in":1540,"tokens_out":493,"duration_ms":45276,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point to take away is that CODE-SHARP uses foundation models to continuously generate and evolve hierarchical Python reward programs, called SHARPs, to train a generalist RL policy from scratch in environments like Craftax. This leads to agents that beat previous methods by significant margins and achieve capabilities like crafting iron tools and mining diamonds autonomously. The approach has some real strengths. Encoding skills as executable programs with prerequisite chains allows the system to route the agent dynamically and reward incremental progress. This seems to enable efficient learning of complex behaviors without any pre-defined rewards. The results on Craftax-Classic, XLand, and the extended version with over 90 SHARPs for zero-shot long-horizon solving are the standout empirical contributions. If the numbers hold up, this points to a scalable path for open-ended discovery. That said, the autonomy claim needs closer inspection. The generation of valid, non-redundant SHARPs by the FM is central, yet details on acceptance rates, how invalid or redundant programs are handled, and any post-generation checks are not prominent in the summary. If there's filtering involved, it could affect how 'fully autonomous' the process truly is. The performance gains are large, but without info on the number of runs or statistical tests, it's difficult to assess their robustness right away. This paper is relevant for researchers working on open-ended reinforcement learning and methods that combine foundation models with RL for skill discovery. A reader focused on automatic curriculum or reward design would get value from the specific mechanism and the reported outcomes. It deserves a serious referee because the core idea is novel and the results are ambitious enough to merit detailed feedback on the experiments and implementation. I recommend sending it out for peer review.","headline":"CODE-SHARP uses FMs to generate hierarchical Python reward programs for autonomous RL, delivering large gains on Craftax tasks, but the autonomy of the discovery process lacks key validation details.","tokens_in":2491,"tokens_out":426,"would_cite":false,"duration_ms":44673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"CODE-SHARP hierarchical reward programs and FM-driven discovery are orthogonal to RS recognition-cost forcing","alignment":"orthogonal","rationale":"The paper's core machinery (SHARP Python programs, prerequisite chains via T operator, FM propose-implement-judge loop, goal-conditioned PPO training on discovered rewards) operates entirely within empirical RL and LLM code generation. It contains no reciprocal-cost functions, golden-ratio ladders, 8-tick periodicity, or parameter-free derivations from a single distinction. No theorems from the RS corpus (reality_from_one_distinction, Jcost uniqueness, AlexanderDuality for D=3, etc.) are paralleled or contradicted.","tokens_in":60474,"confidence":"high","tokens_out":152,"duration_ms":7908,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"CODE-SHARP lets foundation models generate hierarchical Python reward programs so agents can discover and master skills from scratch without any human-designed rewards.","keywords":["skill discovery","hierarchical rewards","foundation models","reinforcement learning","autonomous learning","open-ended evolution","long-horizon tasks"],"falsifier":"An experiment in which the foundation model is replaced by one that produces mostly invalid or duplicate SHARPs and the resulting agent shows no improvement over standard RL baselines on diamond-mining success rate in Craftax.","tokens_in":2749,"feed_emoji":"🤖","tokens_out":792,"duration_ms":37397,"temperature":0.7,"pith_summary":"The paper presents CODE-SHARP as a way to let foundation models continuously create and refine an archive of Python programs called SHARPs. Each SHARP defines a local success condition for a skill and lists prerequisites that point to earlier SHARPs, so that at training time the system routes the agent through a chain of rewards and the agent only has to learn the new piece of behavior. This produces fully autonomous reinforcement learning in environments such as Craftax and XLand, where the resulting agents reach much higher performance than earlier methods and are the first to complete advanced actions like mining diamonds. The approach matters because it removes the need for hand-crafted reward functions or task curricula when building agents that keep expanding their own capabilities.","feed_headline":"AI-generated programs train agents to mine diamonds with no human rewards","feed_subtitle":"CODE-SHARP evolves over 90 hierarchical skill programs that let agents solve long tasks zero-shot at ground-truth reward levels.","key_machinery":"SHARPs, Python programs that pair a local success condition with a prerequisite list pointing to earlier programs, which at runtime dynamically route the agent and deliver incremental rewards so only the newest skill segment must be learned.","core_discovery":"CODE-SHARP leverages FMs to open-endedly grow and evolve an archive of Python programs encoding skills to train a generalist agent policy entirely from scratch via reinforcement learning, directly from source code. These programs, termed Skills as Hierarchical Reward Programs (SHARPs), each encode a local success condition and a set of prerequisites delegated to previously discovered SHARPs. At runtime, SHARPs dynamically route the agent through their prerequisite chain based on the current state, rewarding each completion along the way, requiring the agent to learn only the marginal behaviour each new SHARP introduces, enabling efficient learning of long-horizon skills without any pre-","pith_inferences":["The same generated program archive could be reused across new environments to avoid re-engineering rewards from scratch.","If the number of discovered SHARPs grows with task difficulty, the method could support continual expansion of agent capabilities without external task lists.","Physical robots might receive the same hierarchical reward chains to acquire sequences of manipulation skills with minimal human reward design.","Periodic pruning of low-utility SHARPs could keep the archive manageable as the number of programs increases."],"forward_implications":["On Craftax-Classic the trained agents reach six times the median performance of prior methods.","On XLand the same agents reach 2.6 times the median performance of prior methods.","The agents become the only ones able to craft iron tools and mine diamonds in the tested environments.","Scaling to Craftax-Extended produces a generalist policy over more than 90 discovered SHARPs that solves long-horizon tasks zero-shot at the level of agents given ground-truth rewards."],"fun_headline_variants":["CODE-SHARP evolves skills as Python reward programs from source code","Agents mine diamonds using autonomously grown hierarchical SHARPs","Open-ended discovery trains generalist agents on 90 skill programs","SHARPs route agents through prerequisite chains without predefined rewards"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The foundation model can keep producing valid, non-redundant SHARP programs whose prerequisite chains create useful incremental learning signals without any human curation or filtering.","fun_headline_variants_meta":{"raw":{"variants":["CODE-SHARP evolves skills as Python reward programs from source code","Agents mine diamonds using autonomously grown hierarchical SHARPs","Open-ended discovery trains generalist agents on 90 skill programs","SHARPs route agents through prerequisite chains without predefined rewards"]},"model":"grok-4.3","cost_usd":0.007816,"raw_usage":{"total_tokens":3628,"prompt_tokens":788,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":78162000,"prompt_tokens_details":{"text_tokens":788,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2773,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":788,"tokens_out":67,"duration_ms":27579,"temperature":1.0,"reasoning_tokens":2773,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-22T10:31:49.411745+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment in which the foundation model is replaced by one that produces mostly invalid or duplicate SHARPs and the resulting agent shows no improvement over standard RL baselines on diamond-mining success rate in Craftax.","supporting_citations":[],"review_version":1}