{"id":"03a91ca8-c50e-4027-94e1-96642d4b71cd","arxiv_id":"2507.21513","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Defines a world model as a simple commutative-diagram factorization through an intermediate representation, with conditions that the model be learned and emergent rather than inherited from input or output.","lead":"This paper proposes formal criteria for when a neural network's internal representation counts as a 'world model', defined as a simple readout map from activations to a model of the data-generating world. It adds non-triviality conditions to distinguish genuinely learned, emergent models from patterns already present in the input or output.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The definition's nontriviality conditions depend on unspecified choices of simple function classes, so the same network can be classified as having or lacking a learned/emergent world model depending on those choices.","rationale":"The paper's central claim is that the commutative-diagram framework plus the learned and emergent conditions provides an operational, testable meaning for 'world model'. The weakest point is exactly the choice of simple function classes and approximation tolerances. Section 3.3.1 explicitly concedes the comparability problem for spaces of different dimensionality and offers only a control-function heuristic, while Appendix A leaves 'good approximation' open. Because the learned and emergent conditions are nonexistence claims, their outcomes are not invariant under reasonable changes to FX and FY. This is a correctness risk for the operational claim, not a philosophical disagreement. I found no internal contradiction in the diagrams themselves, and the paper is careful to acknowledge many of its own limitations. The reader's conditional verdict identifies the same weakest assumption, and my concern reinforces it rather than moving it. A concrete sensitivity test varying FX, FY, and approximation tolerance on a synthetic state-tracking task would settle whether the definition is stable; if the verdict flips under those variations, the paper should either provide a principled choice procedure or soften the claim of precision. Therefore the conditional verdict remains appropriate.","tokens_in":12654,"tokens_out":8546,"duration_ms":119013,"concrete_test":"Sensitivity analysis on a fixed trained network: train a small MLP on a synthetic state-tracking task with known ground truth (e.g., an Othello-GPT style board state). Fix a candidate world model M and probe g. For the same network, evaluate the learned and emergent conditions under several plausible choices of FX and FY: coordinate projections; linear maps with parameter budget equal to dim Z; full linear maps; two-layer MLPs of bounded width; and also vary the approximation tolerance in Appendix A. If the Boolean verdict changes across reasonable choices, the definition is not operational; if it is stable, the concern is substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the classes FW, FZ, FX, FY can be chosen so that the learned/emergent conditions are meaningful and stable. Section 3.3.1 admits the comparability problem for spaces of different dimension and offers only a control-function heuristic, not a canonical rule. This is not merely a formal gap: the conditions are existential negations, so their truth is highly sensitive to the class chosen. For a fixed f, g, and M, the statement 'no h in FX satisfies h = g∘f1' can flip from true to false as FX expands. If FX is coordinate projections, almost every world model counts as learned; if FX is all linear maps on a high-dimensional X, the paper's own footnote 6 concedes that a linear projection to any moderate-dimensional M almost certainly exists, so genuine world models would be classified as not learned. The emergent condition has the analogous problem with FY on high-dimensional Y. The control-function proposal does not resolve this: it replaces one arbitrary choice with others (distribution of frandom, class FZ, training threshold). Appendix A's 'good approximation' is also left undefined, making the nonexistence conditions threshold-dependent. Until these choices are anchored, the definition is a family of definitions rather than the single operational criterion claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a formal, diagrammatic definition of what it means for a neural network f: X -> Y with intermediate representation Z to \"contain a world model\" M. The core definition in Section 3.1 requires a modeling map phi_1: W -> M and a simple readout g: Z -> M such that phi_1 = g ∘ f1 ∘ alpha, with phi_1 and g drawn from prescribed simple function classes F_W and F_Z. Sections 3.3.1 and 3.3.2 add nontriviality conditions: the model is \"learned\" if no simple function from X to M equals g ∘ f1, and \"emergent\" if no simple function from Y to M satisfies h ∘ f2 = g. The paper also defines complete and partial causal world models in Section 4 and local world models in Section 5. The authors explicitly describe the role of approximations in Appendix A, acknowledging that 'good approximation' is context-dependent.","tokens_in":12846,"tokens_out":4156,"duration_ms":51423,"significance":"If the proposed criteria could be made fully operational, this would be a genuinely useful contribution to the interpretability literature: it gives a common language for 'world model', 'learned', 'emergent', and 'causal' that is grounded in the well-established linear probing framework, and it draws a clear distinction between behavior and internal mechanism. The paper is honest about its own limitations, explicitly flagging the comparability problem across spaces of different dimensionality and the context-dependence of approximations. The main strength is the formal clarity of the diagrammatic conditions, which turn a vague debate into a set of testable existence/non-existence statements. The paper contains no machine-checked proofs or code, but it does not claim any; its contribution is conceptual and definitional.","major_comments":[{"comment":"The learned condition is an existential negation: it requires that no h in F_X satisfies h = g ∘ f1. The truth of this statement is extremely sensitive to the choice of F_X, which is left as a free parameter. As F_X expands, the condition can flip from true to false for the same network, g, and M. Section 3.3.1's own footnote 6 concedes that for high-dimensional X there is almost certainly a linear projection to any moderate-dimensional M, so the default choice 'F_X = linear maps' would make the learned condition fail for most realistic inputs. The comparability discussion in the same section only offers a control-function heuristic (Hewitt and Liang 2019), not a canonical rule for matching F_X to F_Z. The same issue applies to F_Y and the emergent condition in Section 3.3.2. Until a principled way to fix these classes, or a stability condition over a family of classes, is provided, the paper defines a family of criteria rather than the single operational criterion it announces.","section":"Section 3.3.1"},{"comment":"Appendix A states that all equations should be read as holding 'to a good approximation', and that the meaning of 'good approximation' depends on context, but no tolerance or quantitative threshold is specified. This is load-bearing because the learned and emergent conditions are non-existence conditions: under a tight tolerance, many approximate h's may exist, so the learned/emergent label fails; under a loose tolerance, few approximate h's exist, so the label holds trivially. Appendix A itself uses two different implicit standards (significantly better than chance for Li et al. 2021, and accuracy-focused analyses for Li et al. 2022), so the same network could receive opposite verdicts depending on which threshold is adopted. The authors should either propose a concrete default protocol (e.g., a fixed tolerance tied to a control model, or a sensitivity analysis over tolerances) or explicitly reframe the definition as a parameterized family whose members are chosen by the experimenter.","section":"Appendix A and Section 3.3.1"},{"comment":"The definition of a complete causal world model requires phi_2 : M -> Y with phi_2 ∘ g = f2. As written, this can hold even when g's image has little to do with the actual computation, because phi_2 can be chosen after g and f2 are fixed. The authors motivate causality by the desire to avoid spurious representations, but the formal condition itself is associational rather than interventionist: it asserts the existence of a commuting map, not that intervening on Z through g changes Y as phi_2 predicts. The later intervention examples (Section 4.2, Appendix B) are compelling, but the formal definition in 4.1 does not by itself express the causal claim it is named after. The authors should clarify whether 'causal' is meant as a shorthand for 'intervention-verified', and if so, state the required intervention condition explicitly.","section":"Section 4.1"}],"minor_comments":[{"comment":"The sentence 'We say the network uses a the \"world model\" M' contains a typo ('a the').","section":"Section 3.1"},{"comment":"The phrase 'we call it anemergent world model' is missing a space between 'it' and 'emergent'.","section":"Section 3.3.2"},{"comment":"The final paragraph of Appendix B contains garbled text: 'it because models may be built and used across multiple layers, it we may need to intervene' should be rephrased for clarity.","section":"Appendix B"},{"comment":"The reference for Mnih et al. renders the author's first initial as 'V olodymyr', with a stray space; this should be corrected to 'Volodymyr'.","section":"References"},{"comment":"The phrase 'on a networks internal activations' needs an apostrophe: 'on a network's internal activations'.","section":"Section 2.5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a definitional paper with no new empirical results, and its value depends on whether the proposed schema can be made sufficiently concrete. The central concern is the under-specification of the simple function classes and the approximation tolerance, which are load-bearing for the learned and emergent conditions. The authors are unusually candid about these issues, which is a strength, but the current text leaves the definition too open-ended to serve as a reproducible operational criterion. I would be willing to accept a revised version that either provides a concrete anchoring protocol (with a default choice of function classes and a sensitivity analysis) or explicitly reframes the contribution as a parameterized framework in which the experimenter's choices are part of the specification. The paper is otherwise clear, well-motivated, and appropriately positioned relative to the probing and world-model literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a worthwhile definitional paper that gives the field a much-needed precise vocabulary, and the stress-test concern about function-class arbitrariness is real and should be pressed, but it's not a disqualifying flaw.\n\nThe genuinely new thing is not the commutative diagram, which the authors themselves present as a clean reframing of linear probing. It's the pair of nontriviality conditions—'learned' and 'emergent'—plus the causal and local taxonomy. Those go beyond Hewitt & Liang and Belinkov in a useful way: they turn vague claims like 'the model has an internal world model' into checkable negations of simple readout claims. The paper is honest about its own limitations: appendix A says 'good approximation' is context-dependent, section 3.3.1 admits the comparability problem for different-dimensional spaces, and the authors are clear that actions are out of scope. Use of their own prior work (Othello-GPT, modular addition) reads as illustrative, not as evidence for the definition; no circular burden.\n\nWhere I part company with the reader's slightly generous framing: the stress-test note is not a nitpick. The learned condition is an existential negation: no h in FX such that h = g∘f1. Truth of that statement flips as FX changes. The paper's own footnote 6 concedes that on a high-dimensional X a linear projection to a moderate-dimensional M almost always exists, so almost nothing counts as 'learned' if FX is all linear maps; conversely, if FX is coordinate projections, almost everything counts. Section 3.3.1's control-function suggestion replaces one arbitrary choice with several others (distribution of frandom, class FZ, training threshold). So the paper really does deliver a family of definitions indexed by unanchored choices, not the single operational criterion the conclusion claims. For a definitional paper this is a serious loose end, but not a contradiction: the framework is coherent, and the resolution could be to embrace the parameters explicitly rather than hide them.\n\nWho should read it: anyone working on interpretability or arguing about whether LLMs have world models; it is more useful as a shared vocabulary than as an experimental procedure. A serious referee should engage—I'd send it out, with the main revision ask being to either anchor the simple function classes with concrete defaults or rename the contribution as a parameterized definitional framework. I would cite it for the learned/emergent distinction.","headline":"Worth engaging: the learned/emergent conditions are a real contribution, but the definition is a family of definitions until the simple-function-class choices are anchored.","tokens_in":13433,"tokens_out":1992,"would_cite":true,"duration_ms":22672,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper offers a formal, testable definition for when a neural network contains a world model.","keywords":["world model","interpretability","linear probing","internal representation","emergence","causality","state space","neural networks"],"falsifier":"Run the paper's full checklist on a state-tracking network, measuring how accurately a candidate state $M$ is linearly decodable from the intermediate representation $Z$, from the input $X$, and from the output $Y$. If $M$ is linearly readable from $Z$ but is equally readable from $X$, the paper's learned condition says the network has no learned world model; a controlled mechanistic demonstration that the network genuinely constructs $M$ during computation would show that condition is too strict. The converse test is an intervention: if editing the readout of $M$ in $Z$ changes the output, causal status is confirmed, and if the output is unchanged, the causal condition separates a genuine model from a spurious correlate.","tokens_in":12403,"feed_emoji":"🧠","tokens_out":7879,"duration_ms":79553,"temperature":0.7,"pith_summary":"The paper tries to replace vague claims that a neural network has an internal \"world model\" with a precise, experimentally checkable criterion. It defines a world model $M$ as a simple image of the true world $W$ that is also a simple readout of the network's intermediate representation $Z$, captured by the equation $\\varphi_1 = g \\circ f_1 \\circ \\alpha$. To keep the definition from being vacuous, it adds conditions: the model must be learned, meaning it is not already readable from the input; emergent, meaning it is not forced by the output labels; and it may be causal or local. A sympathetic reader should care because these conditions turn a contested philosophical term into a checklist that probing experiments can actually test.","feed_headline":"A precise, testable definition of a neural network \"world model\"","feed_subtitle":"The paper's checklist separates genuine internal state representations from artifacts of input data or output labels.","key_machinery":"The load-bearing object is the commutative diagram $X \\xrightarrow{f_1} Z \\xrightarrow{f_2} Y$ with $W \\to M$ beneath it. The diagram says the world-to-model map $\\varphi_1$ equals the composition of observation $\\alpha$, the network's front half $f_1$, and the simple readout $g$; because $\\varphi_1$ and $g$ are drawn from pre-specified classes of simple functions, typically linear, the equation is a nontrivial constraint on the network's internals. The same diagram structure generates the ancillary definitions: adding a missing simple map $h: X \\to M$ tests whether the model is learned, adding $h: Y \\to M$ tests whether it is emergent, adding $\\varphi_2: M \\to Y$ or $\\varphi_2: M \\to A$ tests causality, and restricting the world to $W' \\subset W$ gives locality.","core_discovery":"The central claim is that a network $f: X \\to Y$, factored as $f = f_2 \\circ f_1$, contains a world model $M$ if there is a simple modeling map $\\varphi_1: W \\to M$ and a simple readout $g: Z \\to M$ such that $\\varphi_1 = g \\circ f_1 \\circ \\alpha$, where $\\alpha$ is the observation map from the world to the data. The paper further argues that the model is genuinely learned only when no simple map from the input $X$ reproduces $g \\circ f_1$, and emergent only when no simple map from the output $Y$ factors through $f_2$ to give $g$; without these conditions, apparent models may be artifacts of the data or the task. It also distinguishes complete causal models, where a map $\\varphi_2: M \\to Y$ makes the model fully determine the output, from partial causal models, where the model predicts only a simplified aspect $A$ of the output, and it introduces local models restricted to a subset $W'$ of world states.","pith_inferences":["Extending the criterion, one could audit world-model claims by checking whether random or shuffled inputs still admit the same simple readout; the paper's control heuristic points in that direction, but the formal threshold is left open.","The learned and emergent conditions suggest a practical diagnostic: run input-space and output-space probes first, and only then credit the intermediate representation.","If this definition succeeds, an analogous diagram for transition operators on $M$ would define an action-aware world model, which the paper explicitly leaves to future work.","The framework could be used comparatively: two networks trained on the same task could be compared by whether one admits a simpler world-model readout after input leakage is controlled."],"forward_implications":["Probing claims become stronger: a linear probe finding a state variable counts as a world model only if the same variable is not linearly readable from the raw input and not recoverable from the output.","Researchers get a standard checklist: existence of simple $\\varphi_1$ and $g$, failure of simple input-to-model and output-to-model readouts, and an intervention test for causality.","On synthetic tasks such as modular addition, complete causal world models may be identifiable, meaning the model fully explains the network's computation.","For general-purpose systems, local world models are the most realistic target, and the definition gives a precise way to say when a model tracks state only within a restricted domain.","The framework separates internal representation from task accuracy, so a high-accuracy network need not have a world model and a low-accuracy network might."],"supporting_citations":[{"why":"Supplies the control-function heuristic used to compare simple readouts across spaces of different dimensionality.","marker":"Hewitt & Liang (2019)"},{"why":"Reviews the probing methodology that the paper reframes into a world-model definition.","marker":"Belinkov (2022)"},{"why":"Introduces linear classifier probes, the template for the simple readout function $g$.","marker":"Alain & Bengio (2016)"},{"why":"Provides the sentiment neuron example used to illustrate a partial causal world model.","marker":"Radford et al. (2017)"},{"why":"Shows via reconstruction that input data can already contain the model, motivating the learned condition.","marker":"Takens (1980)"},{"why":"Presents the Othello-GPT emergent world representation, the paper's central sequence-model example.","marker":"Li et al. (2022)"},{"why":"Proves the regulator theorem cited as the isomorphism-based precursor to this definition.","marker":"Conant & Ashby (1970)"},{"why":"Describes modular addition grokking, the suggested example of a complete causal world model.","marker":"Nanda et al. (2023a)"}],"fun_headline_variants":["Precise criteria for when a neural net really has a world model","How to tell if a neural net's world model is genuine or an artifact","Testable definition separates true world models from data artifacts","Neural net world models: a checklist to avoid trivial cases","Operational meaning for 'world model' in neural networks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything hangs on choosing the \"simple\" function classes so that the learned and emergent conditions are neither vacuous nor too narrow, and the paper leaves the comparison of simple-function spaces across different input and representation dimensions unresolved.","fun_headline_variants_meta":{"raw":{"variants":["Precise criteria for when a neural net really has a world model","How to tell if a neural net's world model is genuine or an artifact","Testable definition separates true world models from data artifacts","Neural net world models: a checklist to avoid trivial cases","Operational meaning for 'world model' in neural networks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1277,"prompt_tokens":886,"completion_tokens":391,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":303}},"tokens_in":502,"tokens_out":391,"duration_ms":4828,"temperature":1.0,"reasoning_tokens":303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:39:12.347648+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's full checklist on a state-tracking network, measuring how accurately a candidate state $M$ is linearly decodable from the intermediate representation $Z$, from the input $X$, and from the output $Y$. If $M$ is linearly readable from $Z$ but is equally readable from $X$, the paper's learned condition says the network has no learned world model; a controlled mechanistic demonstration that the network genuinely constructs $M$ during computation would show that condition is too strict. The converse test is an intervention: if editing the readout of $M$ in $Z$ changes the output, causal status is confirmed, and if the output is unchanged, the causal condition separates a genuine model from a spurious correlate.","supporting_citations":[{"cited_title":"Detecting strange attractors in turbulence","cited_arxiv_id":null,"evidence_quote":"Shows via reconstruction that input data can already contain the model, motivating the learned condition."}],"review_version":1}