{"id":"caf1b94b-4eda-4f9f-ac5f-9cafe493e1ab","arxiv_id":"2509.05448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A machine-learning system can evolve game rules, treated as axioms, until a previously unreachable goal becomes solvable, as demonstrated in blocks world.","lead":"This preprint argues that AI for scientific discovery should work by modifying existing axioms rather than learning from data, and tests the idea on logic games like blocks world. The paper builds a search framework where language models propose new game rules and planners check whether the new rules turn unreachable goals into reachable ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Blocks-world demo tests only plan-length reduction on a solvable instance; with no check that modified rules preserve original solutions, the 'valid axiom sets' claim and the Newton-to-Einstein analogy are unsupported.","rationale":"The reader's weakest assumption—that solvability is an unvalidated proxy for explanatory coverage—is correct and central. My concern makes it more concrete: the blocks-world experiment does not even test the paper's own preservation constraint (Eq. 4). Because the objective is only to reduce plan length on a single instance, a trivial precondition-deletion baseline could achieve the same result. This does not overturn the conditional verdict, because the paper is explicitly a position paper with preliminary experiments and acknowledges limitations; however, the missing regression check is the single load-bearing gap between the demo and the central claim. If the regression check fails, the paper's strongest claim collapses to 'LLMs can propose rule relaxations that shorten plans,' which is far weaker than 'evolve valid axiom sets that transform unwinnable games into solvable ones.' The reader identified a related proxy concern; my formulation differs by pointing to an internal inconsistency with the paper's own formalism rather than only an external analogy gap.","tokens_in":59631,"tokens_out":4392,"duration_ms":52496,"concrete_test":"Re-run the blocks-world beam-search experiment with a regression check: after each proposed rule modification, verify with the planner that all original benchmark goals (at minimum the original 6-step instance) remain solvable under the modified domain; report any modification that fails. Also run a trivial baseline that randomly removes one precondition from each action, with no LLM guidance, under the same 4-step budget; if the baseline succeeds as often as the LLM-driven search, then the discovered modifications provide no evidence of semantically meaningful axiom evolution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (§1) is that the system can 'evolve valid axiom sets that transform unwinnable games into solvable ones.' The only detailed evidence, §4.2, does not involve an unwinnable game: the original blocks-world instance is solvable in 6 steps, and the target is merely a 4-step budget. More importantly, the search objective (shorter plan under modified rules, §3.3/§3.6) is unconstrained with respect to previously achievable goals. The paper's own Eq. (4) requires the updated axiom set to span existing and new observations—{⟨x,g(x,ε)⟩, ⟨x̂,g(x̂,ε)⟩} ∈ span({A}_{n+1})—but no experiment checks that the discovered modified rules still solve the original 6-step instance or any other originally solvable instance. 'Simultaneous manipulation' and 'stack extraction' are plausible conservative relaxations, but without a regression test we cannot call the resulting rule sets 'valid' in the sense of preserving old explanatory coverage; the demo is equally consistent with the trivial lesson that deleting preconditions shortens plans. The Newton-to-Einstein framing therefore rests on a demonstration that does not enforce the central preservation constraint of the proposed framework.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that scientific discovery should be formulated as axiom modification rather than induction, and proposes a game-design framework in which PDDL rules play the role of axioms, a planner acts as a simulator, and a language model proposes rule modifications. The main feasibility evidence is a blocks-world case study in which the original 6-step problem instance is re-solved within a 4-step budget after discovering two rule relaxations: simultaneous manipulation of blocks and extraction of blocks from the middle of a stack. The paper also includes a large appendix with symbolic 'transformations' connecting pairs of historical theories, from flat-Earth to spherical-Earth and from Newtonian mechanics to general relativity, and argues that these illustrate the same axiom-modification pattern.","tokens_in":59980,"tokens_out":3943,"duration_ms":47304,"significance":"If the framework worked as claimed, it would be a useful contribution to interpretable, knowledge-guided AI4Science: it combines an explicit symbolic representation (PDDL), LLM-generated rule proposals, classical search, and a formal preservation constraint in Eq. (4). The flat-earth/spherical-earth worked example is pedagogically clear, and the paper is transparent about its limitations (Section 8) and about the role of LLMs in producing the appendix transformations (Appendix B). However, the central empirical claim is not yet supported: the only detailed experiment is a single blocks-world instance that is not 'unwinnable' in the stated sense, no regression test checks preservation of previously solvable goals, and the historical transformations are post-hoc narratives rather than outputs of the proposed pipeline. The significance of the proposal therefore remains conditional on substantially stronger evidence.","major_comments":[{"comment":"The central claim that the system 'evolves valid axiom sets that transform unwinnable games into solvable ones' is not supported by the reported experiment. In §4.2, the original blocks-world instance is solvable in 6 steps; the challenge is only to find a plan within a 4-step budget. This is plan-length reduction, not transformation of an unwinnable game. More importantly, Eq. (4) requires the updated axiom set to span both old and new observations, i.e., to preserve previously explained phenomena. No experiment checks that either discovered rule modification still admits the original 6-step solution or any originally solvable goal. Without such a regression test, the 'valid' in the central claim is unverified; the results are also consistent with the trivial strategy of deleting preconditions to shorten plans.","section":"§1, §4.2, Eq. (4)"},{"comment":"The formal objective in Eq. (5) is never instantiated: the paper reports no objective values, no semantic-distance measurements, no baselines, and no error bars. The blocks-world case study uses one LLM, one beam width (8), and 16 decodes, with no code released. Appendix A.3 reports qualitative success rates for BFS, MCTS, beam search, and genetic algorithms, but provides no table, no per-game results, and no definition of 'success'. Since the paper's feasibility claim rests on this single demonstration, the experimental section needs a reproducible protocol, per-game numerical results, ablations, and a clear success criterion aligned with Eq. (4).","section":"§3.6, §4, Appendix A.3"},{"comment":"The historical transformations are not outputs of the proposed framework. Appendix B states that ChatGPT was given the two discipline names and an answer template and then prompted to produce the axiom sets and transformations, with human verification afterwards. Consequently, the Newton-to-Einstein and ray-to-quantum-optics arrows are post-hoc narratives with both endpoints known, not independent predictions or validations of the game pipeline. Some of the appendix derivations are also asserted non-sequiturs; for example, Appendix F.3.2 moves from classical boundary conditions through 'global single solution' and 'global state' to the Schrödinger equation without a derivation. These appendices are best described as illustrative analogies, not evidence for the framework's discovery capability.","section":"Appendix B, Appendix F.3.2, §6"}],"minor_comments":[{"comment":"'Lorenz transformations' should read 'Lorentz transformations'.","section":"Appendix I.3.2"},{"comment":"The label 'Mechenical Engineering' contains a typo: 'Mechanical Engineering'.","section":"Figure 2"},{"comment":"The main text says that 'results from the complete 12-game suite are provided in the appendix,' but the appendix only contains PDDL domain files and qualitative algorithm discussion; no per-game numerical results are given. Either provide those results or adjust the claim.","section":"§4, Appendix A.1"},{"comment":"The notation 'span({A})' is used without formal definition; it is not clear what it means for an observation pair to lie in the span of a set of PDDL rules. A precise definition is needed for Eq. (4) to be checkable.","section":"§3.2, Eq. (4)"},{"comment":"The phrase 'previously unsolvable problems' in the abstract is stronger than the blocks-world experiment, which is only about a stricter plan-length budget on a solvable instance. The wording should be aligned with the actual evidence.","section":"Abstract, §1"}],"recommendation":"major_revision","confidential_remarks":"This is a position/demonstration paper whose core idea is potentially interesting but whose central empirical claim is currently overstated. The most serious gap is the absence of a preservation/regression check for the proposed axiom modifications; without it, the demo does not establish 'valid' axiom evolution in the sense of Eq. (4). The appendix transformations are transparently disclosed as LLM-generated post-hoc narratives, which is honest but means they cannot carry the discovery burden. If the journal is open to position papers with preliminary evidence, a major revision that reframes the claims and adds a proper blocks-world regression test plus at least one genuinely unsolvable instance could make the paper publishable; if the venue expects completed empirical validation, the paper is not yet there."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. This is a position paper with a small feasibility demo. The blocks world result is a real data point—LLM-proposed PDDL rule relaxations that shorten a plan from six steps to four or two—but the paper calls this transforming 'unwinnable games into solvable ones,' and that's not what the experiment did. The original instance was solvable; the target was a tighter step budget. And nothing checks that the modified rules still solve the original six-step instance, so the phrase 'valid axiom sets' is unsupported. Delete preconditions and plans get shorter; that's the trivial reading.\n\nWhat's genuinely new: the specific arrangement—PDDL as the axiom language, LLMs proposing rule changes, a semantic relative distance for ranking, and beam/MCTS-style search over modifications. That combination isn't in the cited prior work. The flat-Earth to spherical-Earth appendix is the strongest part of the paper: actual geometric derivations, a real failure of the flat model, and a clean limit argument R→∞ recovering flat behavior. The authors are also honest in places. Appendix B states plainly that the historical sections were generated by ChatGPT given the two discipline names and a template, so those trajectories are post-hoc narratives, not independent predictions. The limitations section is level-headed.\n\nThe soft spots are real. The historical transformations—from ray optics to quantum optics, analog to digital, Newton to GR—are mostly asserted non-sequiturs. 'Boundary conditions imply a single global state, which implies unitary evolution' is not a derivation; it's a label. Those appendices should be treated as templates for a format, not evidence. The formal optimization in Eq. 5 is never actually solved; the search is heuristic. The main text shows one puzzle, one LLM, sixteen decodes, no baselines, no error bars, and no code, despite mentioning a twelve-game suite. All fixable, but as it stands the evidence is thin.\n\nThe deeper issue: game solvability under modified rules is a position, not a demonstrated proxy for scientific explanatory coverage. That's fine for a position paper, but the demo doesn't carry the weight of the Newton-to-Einstein title.\n\nWho should read this: people working on AI for science who want an interpretable, theory-driven alternative to induction, and people building LLM-guided planning pipelines. It's a useful discussion piece, and a serious referee could help sharpen the claims. I would send it to review, with the expectation of major revisions: add a regression test for preservation, run on genuinely unsolvable instances, release code and baselines.","headline":"A real but thin blocks-world demo; the Newton-to-Einstein framing is post-hoc narrative, and the 'valid axiom sets' claim needs a preservation check.","tokens_in":60426,"tokens_out":2754,"would_cite":false,"duration_ms":29063,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that machine learning for scientific discovery should edit existing axioms rather than induce new laws from data, and it tests the idea as a game-design problem: a language-model-guided search over PDDL rules finds modifica","keywords":["axiom-based reasoning","scientific discovery","game design","rule evolution","PDDL planning","language models","blocks world","theory revision"],"falsifier":"Run the same language-model-guided search on the paper's own flat-Earth/spherical-Earth game: if the pipeline, faced with shadow-angle observations the flat model cannot produce, does not converge on the geometry change (plane→sphere) but instead finds an equally short data-fitting hack (e.g., relaxing the ray-geometry axiom so any shadow angle is allowed), the claim that the mechanism discovers meaning-preserving theory revisions would be falsified. Likewise, in blocks world, if removing the 'clear' precondition outright is ranked by the semantic distance function as closer to the reference t","tokens_in":59571,"feed_emoji":"🧩","tokens_out":7233,"duration_ms":71118,"temperature":0.7,"pith_summary":"This position paper claims that the path from Newton to Einstein is best modeled as a sequence of small, deliberate edits to a foundational axiom set, and that machine learning should be built to perform such edits rather than to infer laws from large datasets. The authors operationalize the claim by recasting science as game design: a domain's axioms are the game rules, experiments are initial setups, and an anomaly is a goal the current rules cannot reach. Their system uses a language model to propose PDDL rule modifications and an LM-based semantic distance to keep edits minimal, with search algorithms steering the proposal loop. The load-bearing experiment shows that this pipeline turns an unwinnable blocks-world configuration winnable within a four-step budget, by allowing simultaneous manipulation of multiple blocks or extraction from the middle of a stack. A sympathetic reader is meant to accept this blocks-world result as evidence that the same mechanism could, in principle, carry out theory revision in real science.","feed_headline":"Blocks-world demo: AI evolves rules, cuts a 6-step plan to 2","feed_subtitle":"The paper argues theory change is minimal rule editing, and tests the idea on a classic planning puzzle.","key_machinery":"The central object is the game tuple G = (R, I, S, T): rules, initial setup, simulator, and target goal. Rules are written in PDDL (Planning Domain Definition Language, a standard formalism for classical AI planning), which separates a domain file (the axioms/actions) from a problem file (the experiment), so rule edits are syntactic and checkable by automated planners that serve as the simulator. The novelty-bearing mechanism is the axiom-evolution loop: when no plan meets T within the step budget, a language model proposes PDDL rule modifications; a semantic relative distance function (language-model atomic comparisons, optionally hybridized with Levenshtein distance for fast filtering) sco","core_discovery":"The paper's central claim is that scientific progress—exemplified by the translation of Newton's laws into general relativity—can be reproduced computationally as axiom-based reasoning rather than induction. It advances a game-design framework where a theory is a rule set R, an experiment is an initial setup I, a simulator S derives predictions, and a goal T declares whether the theory succeeds; when S(R, I) misses T, the agent edits R while keeping I, S, and T fixed and while minimizing both the number and the semantic deviation of the edits. The experimental demonstration is in the blocks world: a 6-step goal becomes reachable in 2 steps under the rule 'lift multiple blocks at once' and in","pith_inferences":["The demo only shows rule relaxation helping a planner within a fixed step budget; it does not show the search choosing scientifically meaningful hypotheses. A natural test, closer to the paper's own flat-Earth formalization, would run the pipeline against a simulator-based game where the correct revision is structural (plane→sphere) and check whether the distance function prefers that over a shall","If the semantic distance function is the real bottleneck, the paper's own historical-transition catalogs provide a probe: ask the language model to rank pairs of theory-to-theory rewrites and measure whether its orderings match the historical 'minimal edit' narratives; inconsistency there would undermine the locality term in the optimization.","The position's strongest claim—efficiency of axiom reasoning over induction—is not directly measured in the paper; a testable extension would compare compute cost of this pipeline against symbolic-regression baselines on a small physics law-recovery task where axioms are already partly known.","Because the search optimizes path length to the goal, the system could prefer any relaxation that shortens plans, even ones that break the domain's intended physics; a conservative extension would add a physical-plausibility verifier on proposed rules."],"forward_implications":["If valid, axiom editing offers a data-lean alternative to induction-heavy discovery pipelines: new theories are generated by modifying existing rules rather than by re-fitting from scratch.","The framework yields interpretable change logs—explicit step-by-step axiom rewrites—which the authors propose as training data for future automated discovery systems.","Because PDDL planners verify rule sets automatically, modifications can be checked for formal consistency and solvability without human intervention.","The same loop (keep observations fixed, minimize deviation while expanding coverage) predicts that effective theory changes are those that preserve prior explanatory success—a testable signature in historical theory transitions.","The approach's efficiency claim depends on strong existing axiom foundations, so it targets mature fields first, not emergent ones without clear rule sets."],"supporting_citations":[{"why":"Supplies the backbone language model (gpt-4o-mini) whose rule proposals and distance judgments drive the blocks-world experiment.","marker":"[Hurst et al., 2024]"},{"why":"Defines PDDL, the formalism that makes game rules explicit, editable, and machine-checkable.","marker":"[Aeronautiques et al., 1998]"},{"why":"Provides the action-language foundation cited for PDDL-style action representation.","marker":"[Gelfond and Lifschitz, 1998]"},{"why":"Supplies the Fast Downward planner used as the automated simulator that evaluates rule modifications.","marker":"[Helmert, 2006]"},{"why":"Provides the blocks world test environment for the main experiment.","marker":"[Russell and Norvig, 2003]"},{"why":"Serves as the canonical example of the induction-based approach the paper argues against.","marker":"[Jumper et al., 2021]"},{"why":"Represents symbolic regression, the baseline that axiom-based reasoning claims to outperform in data efficiency.","marker":"[Udrescu and Tegmark, 2020]"},{"why":"Supplies the logic-programming foundation underlying the rule representation and reasoning.","marker":"[Lloyd, 2012]"}],"fun_headline_variants":["Axiom-based AI: rewrites rules to solve puzzle in 2 steps","Game-design lets AI evolve axioms, not just recognize patterns","AI edits game rules: 6-step puzzle solved in 2 steps","Newton to Einstein via rule editing: game AI demo","Blocks world: AI evolves axioms, cuts plan to 2 steps"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that making a planning game winnable under relaxed rules is a faithful proxy for expanding a scientific theory's explanatory coverage: the blocks-world demonstration shows only that a planner benefits from two rule relaxations, and the paper asserts rather than shows that this mirrors how theories like relativity emerge from Newton's axioms.","fun_headline_variants_meta":{"raw":{"variants":["Axiom-based AI: rewrites rules to solve puzzle in 2 steps","Game-design lets AI evolve axioms, not just recognize patterns","AI edits game rules: 6-step puzzle solved in 2 steps","Newton to Einstein via rule editing: game AI demo","Blocks world: AI evolves axioms, cuts plan to 2 steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000641,"raw_usage":{"total_tokens":2732,"prompt_tokens":634,"completion_tokens":2098,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":378,"completion_tokens_details":{"reasoning_tokens":2008}},"tokens_in":378,"tokens_out":2098,"duration_ms":17006,"temperature":1.0,"reasoning_tokens":2008,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:24:07.580200+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same language-model-guided search on the paper's own flat-Earth/spherical-Earth game: if the pipeline, faced with shadow-angle observations the flat model cannot produce, does not converge on the geometry change (plane→sphere) but instead finds an equally short data-fitting hack (e.g., relaxing the ray-geometry axiom so any shadow angle is allowed), the claim that the mechanism discovers meaning-preserving theory revisions would be falsified. Likewise, in blocks world, if removing the 'clear' precondition outright is ranked by the semantic distance function as closer to the reference t","supporting_citations":[],"review_version":1}