{"id":"7162ea14-ef3e-472d-9e78-e0013b80d623","arxiv_id":"2411.15891","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Agents that receive LLM-extracted environment rules as context or rewards explore more purposefully and score higher in the Crafter game.","lead":"This paper proposes using large language models to read records of past game plays, extract the environment's rules (what each action needs, costs, and does), and feed those rules to game-playing agents as either reasoning hints or reward signals. In the open-ended game Crafter, both a language-model agent and a reinforcement-learning agent scored higher with these extracted rules than without them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The published law-based reward functions in §D.5 contain concrete code bugs, so Table 2 may not test the method as described.","rationale":"The reader's weakest assumption concerns whether ten successful and ten failed records yield accurate laws. I find the extracted preconditions in §D.4 largely consistent with Crafter's known mechanics, so that assumption is plausible though unvalidated. The sharper, checkable defect is the reward code in §D.5. A reader cannot verify Table 2 because three reward functions for central crafting achievements are broken as written; if used verbatim they would crash or never reward the action, so the reported RL gain cannot be attributed to the described method. This does not require assuming bad faith: the appendix may simply be a non-final artifact, but the paper's quantitative support for the reward-design half of the central claim depends on it. The LLM-agent comparison in Table 1 still provides some evidence that richer textual context helps, though it does not isolate law extraction from generic information injection. I therefore keep the conditional verdict, with the concrete condition that the reward code be corrected, executed, and the per-reward firing coverage reported. This agrees only partially with the reader: they noted implementation bugs in the rationale, but their stated weakest assumption was record sufficiency rather than the reward-function artifact.","tokens_in":13019,"tokens_out":8036,"duration_ms":68493,"concrete_test":"Run the Table 2 PPO experiment with the exact reward functions printed in §D.5 and the described first-occurrence -0.5 penalty, adding per-function logging of call counts and True returns. If training crashes, or if `make_wood_pickaxe_reward`, `make_stone_pickaxe_reward`, or `make_stone_sword_reward` never return True, the published artifact is not a faithful implementation. Then fix those functions to check `'table'` and `'furnace'` in `nearby[0]` (textures) and rerun; the law-based reward claim is supported only if the corrected run reproduces 12.3 ± 0.8 and all 22 achievements have a reward function that fires during training.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that experience E can be converted into effective law-based rewards rests on Table 2, and the only artifact for that experiment is the reward code in §D.5. Several of those functions cannot work as printed. `make_stone_pickaxe_reward` checks `'table' in agent.world[target][1]`, but per the prompt in §D.3 table is a texture, not an object, and indexing the object field for a table block will not find it (and may raise on `None`). `make_wood_pickaxe_reward` searches `isinstance(obj, Table)` among `agent.world.nearby(...)[1]`, the object set, although tables are reported as textures. `make_stone_sword_reward` iterates `for texture, obj in nearby`, unpacking the `(textures, objects)` tuple incorrectly, so it will crash or misbehave. These cover core crafting achievements. If these reward functions were used in training, they would either crash or silently never fire, meaning the reported 12.3 ± 0.8 does not demonstrate the full law-based reward scheme. Since no code is released and the penalty is not specified, this is not resolvable from the paper. The LLM-agent result (Table 1) is less affected, but the reward-design component of the central claim is unverified as published.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method, 'From Laws to Motivation,' that uses interaction records from human players in the open-ended game Crafter to extract textual 'experience' E via an LLM. E is represented as preconditions, costs, and benefits for each objective, approximating the underlying environment laws. This E is then used in two ways: (1) as augmented context for an LLM agent built on SPRING, and (2) as a basis for generating code-based reward functions for a PPO agent. The reported experiments show that SPRING with E outperforms SPRING with paper-derived context or with action names only (Table 1), and that PPO trained with law-based achievement rewards plus a penalty outperforms the same agent without the penalty (Table 2, 12.3 vs 9.7). The paper concludes that E can approximate environment laws and provide internal motivation for both reasoning and reward design.","tokens_in":13333,"tokens_out":4953,"duration_ms":45355,"significance":"The idea of distilling environment 'laws' from raw interaction records and reusing them as prompts or reward criteria is timely and relevant to intrinsically motivated and open-ended learning. The paper clearly describes the pipeline, includes prompts, extracted experience text, and reward code in the appendices, which is helpful for reproducibility. However, the RL result rests on reward functions that, as published in Appendix D.5, contain concrete implementation bugs, and the LLM-agent result lacks ablations that isolate the role of E from generic extra recipe information. If the reward-function issues are fixed and the experiments clarified, the method would be a useful practical contribution; in its current form the central RL claim is not supported by the published artifact.","major_comments":[{"comment":"Several published reward functions contain errors that make them unable to behave as intended. For example, make_stone_pickaxe_reward checks `'table' in agent.world[target][1]`, but according to the prompt in D.3 `agent.world[target]` returns a `(texture, obj)` pair, so `[1]` is the object field and 'table' is a texture, not an object; this function will never return True (or may raise on a None object). Similarly, make_wood_pickaxe_reward searches for `isinstance(obj, Table)` in the object set returned by `agent.world.nearby(...)`, while tables are reported as textures. make_stone_sword_reward iterates `for texture, obj in nearby`, but `nearby` is a tuple of `(textures, objects)`, so the unpacking is incorrect. Additional functions such as defeat_zombie_reward test `'iron_sword' in agent.inventory` and `'stone_sword' in agent.inventory`, which are always true because these keys are always present with value 0 in the inventory dictionary described in the prompt. Since Table 2 is the only evidence for the RL component of the central claim, these bugs are load-bearing: as published, the experiments do not demonstrate that the proposed law-based reward scheme improves PPO.","section":"Appendix D.5"},{"comment":"The RL evaluation does not isolate the contribution of the experience E. The comparison is health+achievement+penalty vs. health+achievement vs. health, but the achievement rewards are all generated from E. There is no baseline that uses reward functions derived from the benchmark paper, from random preconditions, or from a 'shallow' version of E. Without such a control, the improvement in Table 2 could plausibly stem simply from providing any additional shaping reward, not from the law-based semantics of E. Additionally, the penalty term is only described verbally ('-0.5 point penalty upon the first occurrence') and is not specified in enough detail to reproduce the experiments.","section":"Table 2 / Section 4"},{"comment":"The paper claims in Section 5 that the experience E 'can approximate the laws governing the environment,' but the extraction accuracy is never validated. The method relies on only 10 successful and 10 failed textual records per achievement (D.1), the selection criteria for these records are not given, and no quantitative comparison of E against the true game code or a held-out set of records is conducted. Since E is the sole source for both the LLM context and the reward code, any systematic error in extraction would invalidate both sets of results. This is a correctness-risk concern that the authors should address, e.g., by measuring how often the extracted preconditions/costs/benefits match the actual game mechanics.","section":"Section D.1 / D.4"}],"minor_comments":[{"comment":"In the reward-design prompt, 'Gird is probably some kind of texture or an object' should be 'Grid is probably some kind of texture or an object.'","section":"Appendix D.3"},{"comment":"The function collect_coal_reward is defined twice with identical bodies; duplicate definitions should be removed.","section":"Appendix D.5"},{"comment":"The 'Reward' column is not defined in the text. It presumably reports average achieved reward, but this should be stated explicitly.","section":"Table 1"},{"comment":"The output line says 'Sting y' which appears to be a typo for 'String y'. Also, the algorithm's input includes 'String ug' but it is not clear how ug is used when forming preconditions y.","section":"Algorithm 3"},{"comment":"The action space is expanded from 17 to 27 actions for the RL agent, and the reward-generation loop in Algorithm 1 iterates over objectives. It would be helpful to state whether all 27 actions received reward functions and how the 'do' action split is handled in the reward code.","section":"Section 4 / Appendix C"},{"comment":"The paper reports means with standard deviations but no number of seeds or statistical significance tests. Adding these would strengthen the claims in Section 5.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style submission and the main experimental claim for RL is not reproducible from the published reward code. The appendix contains concrete bugs in several core reward functions, and the LLM-agent experiment, while more robust, would still benefit from a better-controlled baseline. The authors should be encouraged to fix the reward code, release it, and add the missing control experiments before the paper is relied upon."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my take on arXiv:2411.15891.\n\nThe novel piece is using an LLM to extract (preconditions, costs, benefits) triplets from human interaction records and then feeding those triplets to both an LLM agent as context and an RL agent as reward-generating code. That's a reasonable way to turn cheap interaction logs into reusable guidance, and the LLM-agent experiment in Table 1 is clean: swapping in experience text beats the paper-derived context (12.9 vs 8.4) and the 'actions only' baseline (2.0). The paper is honest about its limits and doesn't oversell.\n\nThe soft spots are real. The reward functions printed in Appendix D.5 contain concrete bugs: make_stone_pickaxe_reward checks for 'table' in the object slot, make_wood_pickaxe_reward looks for Table objects (but table is a texture), and make_stone_sword_reward iterates incorrectly over the nearby tuple and will crash. So as published, the RL experiment in Table 2 is not reproducible, and we can't confirm the 12.3 vs 9.7 improvement came from the described law-based reward scheme. No code is released, so this can't be checked another way. That's a load-bearing gap for the reward-design half of the paper's central claim.\n\nThe LLM-agent half stands. The method is simple, well-communicated, and the comparison against SPRING's paper-based context is a fair ablation. The main remaining weaknesses are single-benchmark evidence and the lack of any validation that the extracted experience actually matches the game's true conditions—the paper shows the experience helps, not that it approximates the laws.\n\nFor a workshop paper this is a useful contribution to discuss. The idea is worth building on, but only after the reward code is fixed and the RL numbers re-run. I'd send it to peer review because the core concept deserves scrutiny beyond a workshop, though it needs major revision before acceptance.","headline":"A clean idea for extracting environment 'laws' into LLM context and rewards, but the published reward code in Appendix D.5 cannot work as written, so the RL result needs verification.","tokens_in":13823,"tokens_out":2076,"would_cite":false,"duration_ms":18142,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM can turn raw interaction records into natural-language laws of an environment, and that these laws improve both LLM reasoning and RL reward design in Crafter.","keywords":["law-based reasoning","reward design","LLM agents","reinforcement learning","open-ended environments","Crafter","experience extraction","internal motivation"],"falsifier":"Compare the extracted experience $E$ and the generated reward functions against Crafter's actual source-code preconditions. If a substantial fraction of inferred preconditions are wrong — for example, if $E$ claims an action needs a material it does not consume or omits a required nearby object — the reported gains could not be attributed to accurate law recovery. A second test: change a crafting recipe in Crafter, collect fresh human records under the new rules, and check whether the extracted experience tracks the change or whether the LLM's prior knowledge overrides the records.","tokens_in":12830,"feed_emoji":"🎮","tokens_out":5966,"duration_ms":49427,"temperature":0.7,"pith_summary":"Large language models can read text records of past game attempts and recover the environment's rules — what each action costs, what it produces, and what must already be true for it to work. The paper calls these recovered rules \"experience\" and shows they serve as internal motivation: appended to an LLM agent's prompt they raise the Crafter score from $8.4 \\pm 1.4$ to $12.9 \\pm 2.3$, and when converted into reward code for a PPO agent they raise the score from $9.7 \\pm 0.7$ to $12.3 \\pm 0.8$ when paired with a penalty for invalid actions. The authors argue this turns blind trial-and-error into purposeful exploration without relying on external feedback.","feed_headline":"Play logs teach agents a game's laws; scores jump","feed_subtitle":"Recovered rules from past successes and failures lift both LLM reasoning and RL rewards in the open-world game Crafter.","key_machinery":"The load-bearing object is the experience set $E = \\{U, O, Y\\}$, where $U$ holds the costs, $O$ holds the benefits, and $Y$ holds the preconditions inferred for each achievable objective. An LLM extracts it by reading records $d = \\langle s, a, s', v\\rangle$ and comparing $s$ with $s'$ across successful and failed attempts, matching the paper's Constrained Markov Decision Process view that objective $g$ is achieved exactly when all constraints $c_i(s, a) \\le b_i$ hold. $E$ does double duty: as prompt context $p' = p \\oplus E$ for LLM reasoning, and as the specification from which the LLM generates code that returns True exactly when the inferred preconditions and resource checks are met, keeping reward values constant while changing only reward timing.","core_discovery":"The central claim is that interaction records alone contain enough information for an LLM to approximate the laws of an environment, and that this approximation improves both reasoning-based and reward-based agents. For each of Crafter's 22 achievements, the method collects ten successful and ten failed records, then prompts an LLM to compare the state before and after each action and aggregate the results into three sets: preconditions $Y$, costs $U$, and benefits $O$, written in natural language as the experience $E = \\{U, O, Y\\}$. The same $E$ is used two ways: it is concatenated onto the prompt of an LLM agent that reasons with a question-answer graph, and it is given to an LLM that writes boolean reward functions deciding whether an action is valid in the current state. Both uses beat the paper-text baseline, supporting the claim that language-level laws can act as self-motivation.","pith_inferences":["Going beyond the paper, the reported gains do not isolate whether the improvement comes from accurate preconditions, the penalty term, or simply from more textual detail; ablating these separately would settle it.","Going beyond the paper, if extraction is robust the same record-to-experience pipeline should transfer to other open-ended environments with text observations, since only state-pair records are needed.","Going beyond the paper, the use of human-collected successful records leaves open whether agent-generated, noisier records would work as well; testing that would show whether the loop can be closed without human data.","Going beyond the paper, the variance in the scores (e.g., $12.9 \\pm 2.3$) means practical significance depends on repeated runs and significance testing, which the paper does not report."],"forward_implications":["LLM agents can reason with environment laws without the environment's paper documentation, because experience extracted from records outperforms paper-text context in Crafter.","RL agents can receive dense, semantically meaningful rewards without iterative reward-value tuning; the LLM only decides when a reward is granted, not what its value is.","Law-based achievement rewards plus a penalty for invalid actions yields a better PPO agent than health-plus-achievement rewards alone, indicating that knowing preconditions curbs blind exploration.","Because experience is expressed in language, the same $E$ can be reused across agent paradigms and combined with other reward-shaping methods.","Law-based rewards are independent of environmental feedback, allowing an agent to evaluate likely action outcomes before executing them."],"supporting_citations":[{"why":"Supplies the SPRING LLM-agent framework and QA-DAG reasoning baseline that the context experiments extend.","marker":"[25]"},{"why":"Defines Crafter, the open-ended benchmark environment, its 22 achievements, and the score metric used in all experiments.","marker":"[9]"},{"why":"Provides the Proximal Policy Optimization backbone algorithm used to train RL agents with law-based rewards.","marker":"[18]"},{"why":"Prior work on LLM-generated reward functions that the paper contrasts with its constant-value, timing-only reward design.","marker":"[15]"},{"why":"Prior work guiding RL pretraining with LLM knowledge, referenced in the related work and used to justify decomposing the 'do' action.","marker":"[5]"},{"why":"Supplies the reward-shaping theory that motivates adding auxiliary law-based rewards.","marker":"[16]"},{"why":"Cited as the future direction for combining experience with reward machines and finite-state-machine reward design.","marker":"[10]"}],"fun_headline_variants":["LLM extracts game laws from play logs; agents excel in Crafter","From logs to laws: agents reason and reward better in Crafter","Discover the rules, then win: law-based exploration improves agents","Play logs reveal hidden rules; agents boost performance in Crafter","Agency via laws: LLM-derived rules lift agents in Crafter"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that ten successful and ten failed text records per achievement let the LLM infer accurate preconditions, costs, and benefits; if the records are unrepresentative or the LLM misreads them, every downstream use inherits the error, and the paper does not validate extraction accuracy against the true game code.","fun_headline_variants_meta":{"raw":{"variants":["LLM extracts game laws from play logs; agents excel in Crafter","From logs to laws: agents reason and reward better in Crafter","Discover the rules, then win: law-based exploration improves agents","Play logs reveal hidden rules; agents boost performance in Crafter","Agency via laws: LLM-derived rules lift agents in Crafter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1255,"prompt_tokens":838,"completion_tokens":417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":326}},"tokens_in":454,"tokens_out":417,"duration_ms":4312,"temperature":1.0,"reasoning_tokens":326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:45:48.259170+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the extracted experience $E$ and the generated reward functions against Crafter's actual source-code preconditions. If a substantial fraction of inferred preconditions are wrong — for example, if $E$ claims an action needs a material it does not consume or omits a required nearby object — the reported gains could not be attributed to accurate law recovery. A second test: change a crafting recipe in Crafter, collect fresh human records under the new rules, and check whether the extracted experience tracks the change or whether the LLM's prior knowledge overrides the records.","supporting_citations":[{"cited_title":"Spring: Studying papers and reasoning to play games","cited_arxiv_id":null,"evidence_quote":"Supplies the SPRING LLM-agent framework and QA-DAG reasoning baseline that the context experiments extend."},{"cited_title":"Using reward machines for high-level task specification and decomposition in reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Cited as the future direction for combining experience with reward machines and finite-state-machine reward design."}],"review_version":1}