Pith. sign in

REVIEW 3 major objections 5 minor 85 references

Scientific discovery is a problem of knowledge organisation: AI should be built around reusable mechanisms, not predictive mappings alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:29 UTC pith:LGJ62B5D

load-bearing objection A candid, well-scoped position paper: real gap, honest limits, and a useful if modest formalization—worth a serious referee, not a groundbreaking result. the 3 major comments →

arxiv 2607.12474 v2 pith:LGJ62B5D submitted 2026-07-14 cs.AI

From Observation to Insight: Mechanistic World Models and the Quest for Autonomous Discovery

Ingmar Posner , Anson Lei , Bernhard Sch\"olkopf This is my paper
classification cs.AI
keywords scientific discoveryworld modelsmechanismsknowledge organisationprediction vs explanationcausal representation learningparsimonycompositionality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that today's AI systems can forecast but not discover because their knowledge is organised as predictive mappings from inputs to outputs, whereas scientific explanation requires reusable mechanisms that generate observations. It proposes Mechanistic World Models, in which a model stores typed latent variables, a library of mechanisms with typed signatures, and binding structures that instantiate those mechanisms in each environment. The paper derives from the philosophy of science the pressures that would make such knowledge emerge — parsimony (explain many observations with few principles) and compositionality (recombine existing mechanisms) — and shows that existing research directions each capture pieces of this picture but none unify them. If the argument is right, the route to autonomous discovery runs through mechanism-centric representation and learning, not larger predictive models alone.

Core claim

The central claim is that scientific discovery reduces to a knowledge-organisation problem: find the smallest library of reusable mechanisms that explains the largest collection of observations. Concretely, the paper formalises a Mechanistic World Model as W = (T, Z, M, S): a set of learned semantic variable types; typed latent variables; a library of mechanisms m = (Σ_m, f_m), each a typed signature plus a transformation; and bindings that instantiate mechanism roles on variables while respecting type constraints. Learning is posed as minimising prediction loss together with parsimony and compositionality pressures over the whole tuple. The paper argues this organisation, rather than predic

What carries the argument

Mechanisms with typed signatures — m = (Σ_m, f_m), where Σ_m is a tuple of input and output variable roles drawn from semantic types T and f_m is the transformation — carried by the world model tuple W = (T, Z, M, S). Bindings b: Σ_m → Z enforce type matching and let the same mechanism be reused in different environments. The two inductive pressures, parsimony (few, simple mechanisms and variables) and compositionality (reuse across environments through shared bindings), are what turn a predictive model into an explanatory one.

Load-bearing premise

The whole edifice rests on the assumption that the systems we want to understand are nearly decomposable — generated by a modest set of mechanisms that remain approximately invariant and reusable across contexts; if that fails, a mechanism library collapses into a relabelled predictive model.

What would settle it

Train a purely predictive world model and a Mechanistic World Model on the same family of environments with known mechanisms, then test whether the MWM's recovered types, mechanisms and bindings match the ground truth and whether the predictive model can match its performance on novel environments and interventions. If a black-box predictive model, without mechanism-centric organisation, matches or exceeds the MWM on these explanation and adaptation tasks, the claim that explanatory structure must be imposed at learning time is false.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Evaluation of discovery systems should shift from prediction accuracy to whether recovered variables, mechanisms and bindings match the generative structure of the data.
  • Compositional generalisation becomes achievable: an unseen system can be explained by drawing a new binding structure over an existing mechanism library rather than retraining from scratch.
  • Continual learning is naturally localised: new evidence should update only the mechanisms implicated in that evidence, mitigating catastrophic forgetting.
  • The framework supplies a concrete target for theory: identifiability of variables and mechanisms under non-Markovian dynamics and partial observability.
  • Agentic AI-scientist systems built on predictive foundation models would inherit the wrong representational substrate; discovery requires mechanism-centric foundations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • We infer a testable consequence: if Mechanistic World Models are trained on environment families with known ground-truth mechanisms, the recovered library should approximately match the ground truth wherever parsimony and compositionality are satisfied; standard predictive baselines should fail the same match.
  • We infer that the framework predicts a 'mechanism management' subproblem (when to add, merge, or retire mechanisms) that the paper mentions but does not formalise; this could be cast as a library-selection or Bayesian model-merging problem.
  • We infer that the semantic-type alignment is the main internal unknown: whether the semantic types in equations 2–7 can be learned stably without external labels is an empirical question that a synthetic benchmark with unknown but discrete mechanisms could settle.
  • We infer that the argument implicitly requires that near-decomposability holds for real scientific domains; domains such as climate or gene regulation may force the framework to include hierarchical or emergent mechanisms, or to accept that some systems resist small reusable libraries.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This position paper argues that scientific discovery is fundamentally a problem of knowledge organisation: predictive accuracy is necessary but not sufficient, and a system that performs genuine discovery must organise its knowledge around reusable mechanisms rather than predictive mappings. Drawing on philosophy of science (explanation as invariant, parsimonious, reusable structure) and on selected machine-learning literatures (mechanistic interpretability, causal representation learning, equation discovery, mixture-of-experts), the authors propose Mechanistic World Models (MWMs). An MWM is formalised as W = (T, Z, M, S), where T are learned semantic variable types, Z are typed latent variables, M is a library of typed mechanisms m = (Σ_m, f_m), and S is a collection of bindings that instantiate mechanisms in particular environments. A schematic learning objective (Eq. 9) combines a prediction loss with parsimony and compositionality pressures. The paper surveys partial progress on variable, mechanism, and structure discovery, and closes with a candid list of open challenges: full integration, identifiability under non-Markovian dynamics and partial observability, active-inquiry machinery, and the absence of evaluation benchmarks.

Significance. If the conceptual thesis is correct, this paper provides a useful high-level unification of several research strands and a formal vocabulary for discussing mechanism-centric world models. Its main strengths are the clean formal anatomy (Eqs. 2–9), the explicit derivation of design principles from philosophy of science, the honest enumeration of unsolved problems in §5.4, and the acknowledgment (§6) that sufficiently scaled predictive models might eventually exhibit some of these properties. These features make the paper a valuable research agenda. However, the paper is entirely conceptual: no method is instantiated, no experiments are reported, and the central premises — near-decomposability of real systems and the learnability/stability of semantic types and bindings — are not tested or even stated as formal assumptions. The contribution should therefore be read as a blueprint and a call to action rather than as a demonstrated result. Within that scope, the contribution is significant, but the load-bearing assumptions need to be made explicit and testable for the paper to support its stronger claims.

major comments (3)
  1. [§2.3, §4.1 (Design Principle 3), Eq. (9)] The framework's usefulness depends on the assumption that real systems are nearly decomposable and that environments change through sparse mechanism shift. This is introduced via Simon and the independent causal mechanisms principle, but it is never stated as a formal condition or treated as a falsifiable premise. If many target systems admit no modest library of approximately invariant, reusable mechanisms, then the optimum of Eq. (9) will either sacrifice prediction accuracy or proliferate environment-specific mechanisms; in the latter case the recovered 'mechanisms' are a relabelled predictive model, not an explanation. Section 5.4 concedes unresolved identifiability and missing benchmarks, but it does not address this decomposability premise directly. Please make this assumption explicit, discuss domains where it may fail, and specify what evidence or experiment would falsify it.
  2. [§4.2, Eq. (9)] Equation (9) is a schema rather than an executable objective. The terms Ωp and Ωc are described verbally but no concrete instantiation is given, and it is unclear how the parsimony–compositionality trade-off can be balanced without degenerate solutions (for example, an overly small mechanism library that predicts poorly or an unnecessarily large one that merely memorises). If this is intended only as a blueprint, the paper should say so explicitly and give at least one concrete minimal example of how these pressures could be realised, e.g., via a description-length penalty on the mechanism library and a sparsity penalty on the binding structure.
  3. [Table 1, §5.2, §5.3] Table 1 is used to support the claim that existing methods capture only isolated ingredients of MWMs and that MWMs unify them. The selection of methods is not systematic, and four of the ten rows (VCD, COMET, SPARTAN, DIDS) are from the authors' own group; MWM is the only method receiving a full row of checkmarks. This makes the comparison appear less objective and weakens the 'convergence' argument. Either provide explicit inclusion criteria and a rating rubric, or clearly reframe the table as an illustrative grouping rather than a comparative evaluation.
minor comments (5)
  1. [Abstract/§1] The acronym MWM is used in Figure 1 before it is defined in Section 4. Define Mechanistic World Models at first use in the main text or point to Section 4 from the caption.
  2. [§4.2, Eqs. (5)–(7)] The notation Σ_m is overloaded: in Eq. (5) it denotes the pair of input/output typed roles, while in Eq. (6) it is said to denote the set of variable roles appearing in the signature. This is confusing and should be clarified, e.g., by writing roles(Σ_m).
  3. [§4.2, Eq. (9)] The minimization is written over T, Z, M, and {S_e} but the objective includes T and Z only inside Ωp and Lpred. Please restate the dependencies explicitly or simplify the notation.
  4. [§5.3] The claim that SPARTAN's sparse-attention construction scales is supported by a preprint [84]. Since this is load-bearing for the scalability argument, consider citing a published result or adding a caveat.
  5. [References] A few references are to preprints or very recent conference papers (e.g., [72], [79], [84]). This is acceptable for a forward-looking paper, but please ensure the publication status is consistently indicated.

Circularity Check

0 steps flagged

No circular derivation: MWM is a definitional blueprint, not a fitted predictive model; self-citations are supporting, not load-bearing.

full rationale

MWM is presented as a design paradigm rather than a fitted model, and the paper makes no empirical prediction claims that could reduce to fitted inputs. The formal objects (T, Z, M, S) and the learning objective in Eq. (9) are schematic definitions: variables, mechanisms, and bindings are defined so that the parsimony and compositionality pressures are exactly what a 'good explanation' was said to require in Section 2. This is internal consistency, not circular derivation. The paper explicitly concedes in Section 5.4 that no existing framework fully addresses identifiability under non-Markovian/partial-observability conditions and that evaluation benchmarks are missing; these are acknowledged open problems rather than hidden dependencies. The self-citations in Table 1 (VCD, COMET, SPARTAN, DIDS) and the scaling reference to the authors' own unpublished preprint [84] provide supporting illustrations for the proposed paradigm, but they are not load-bearing premises for the central knowledge-organization thesis. Because the paper does not claim to derive novel empirical predictions from fitted parameters or to prove uniqueness via a self-citation chain, no significant circularity is present.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 3 invented entities

No fitted parameters appear: the paper contains no data or experiments, and the pressures Ω_p and Ω_c in Eq. 9 are placeholders. The argument rests on (i) a philosophical premise about what makes explanations good; (ii) a factual premise that the world is nearly decomposable into reusable mechanisms; (iii) the paper's own, untested claim that architecture-level parsimony and compositionality induce explanatory structure; and (iv) its formal design choice of typed variable signatures. The invented computational constructs have no falsifiable handle attached in this paper.

axioms (5)
  • domain assumption Good scientific explanations are those with parsimony (few principles) and broad scope (many observations); explanation is information organization.
    Adopted from the PoS review in §2.2–2.3 (Hempel–Oppenheim, Salmon, Kitcher); this is the philosophical premise from which the design principles are argued to follow.
  • domain assumption The world is nearly decomposable: systems are generated by a small number of recurring, reusable mechanism types that are approximately invariant across contexts.
    Invoked via Simon (§2.3) and the Independent Causal Mechanisms / Sparse Mechanism Shift principles (§3.2, §4.1). If false, mechanism reuse cannot ground explanation; the paper acknowledges this family of problems as open in §5.4.
  • ad hoc to paper PoS criteria can be operationalized as architecture-level inductive pressures (parsimony + compositionality) that suffice for explanatory structure to emerge.
    The core constructive step in §4.1; asserted rather than derived, and explicitly flagged by the authors as unvalidated ('Whether the capabilities... will emerge... remains an open question').
  • ad hoc to paper Mechanisms can be typed by semantic roles (T) and reused across systems via type-consistent bindings (Eqs. 4–7).
    The formal MWM definition (Eqs. 4–8) assumes cross-domain type alignment is discoverable and binding is a learnable selection problem; no identifiability or convergence result is provided.
  • standard math Observations can be encoded into latent variables, and mechanisms act as deterministic or probabilistic maps over them.
    Background computational assumption inherited from world models and structural causal models (§4 preamble).
invented entities (3)
  • Semantic variable types T (emergent functional roles of latent variables) no independent evidence
    purpose: Enable mechanisms to be specified over roles rather than specific variables, so they can be reused across systems (Eqs. 2–5).
    A new computational abstraction introduced in §4.2; unlike a physical quantity such as position or mass, the paper provides no falsifiable handle that would constrain or confirm these types outside the framework itself.
  • Typed mechanism library M with signatures Σ_m no independent evidence
    purpose: The central knowledge unit: reusable transformation paired with a typed variable signature (Eq. 4).
    Architectural proposal; instantiated in simplified form by COMET/NEO, but the typed-signature abstraction has no evidence attached in this paper.
  • Binding structure S mapping mechanism roles to latent variables no independent evidence
    purpose: Composition: explains different systems by different bindings over a shared library (Eqs. 6–8).
    Design choice; the paper notes binding is 'fundamentally a selection problem' and suggests sparse attention as a route (§5.3), explicitly leaving scaling open.

pith-pipeline@v1.3.0-alltime-deepseek · 20760 in / 20378 out tokens · 201011 ms · 2026-08-02T06:29:26.324848+00:00 · methodology

0 comments
read the original abstract

Recent advances in foundation models have transformed AI for Science, enabling remarkably accurate predictive performance across domains ranging from protein folding to weather forecasting. Yet prediction alone does not constitute scientific discovery. Scientific understanding depends on uncovering the reusable explanatory mechanisms that generate observations, whereas contemporary machine learning remains fundamentally organised around predictive mappings rather than explanatory structure. In this paper, we argue that scientific discovery is fundamentally a problem of knowledge organisation. To this end, we introduce Mechanistic World Models, a new design paradigm that places reusable mechanisms at the centre of representation, computation and learning. Drawing on insights from the philosophy of science, we derive the computational capabilities required for discovery, identify the design principles and inductive pressures that encourage explanatory knowledge to emerge, and formalise the anatomy of a mechanism-centric world model. Finally, we show how diverse research directions including mechanistic interpretability, causal representation learning, equation discovery and modular architectures capture complementary ingredients of this paradigm while lacking a unified framework. We propose Mechanistic World Models as a conceptual foundation and computational blueprint for moving AI beyond predictive forecasting towards autonomous scientific discovery.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

85 extracted references · 5 canonical work pages

  1. [1]

    Nature630(8016), 493–500 (2024) https://doi.org/10.1038/s41586-024-07487-w

    Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Will- more, L., Ballard, A.J., Bambrick, J., Bodenstein, S.W., Evans, D.A., Hung, C.-C., O’Neill, M., Reiman, D., Tunyasuvunakool, K., Wu, Z., ˇZemgulyt˙ e, A., Arvaniti, E., Beattie, C., Bertolli, O., Bridgland, A., Cherepanov, A., Congreve, M., Cowen-Rivers, A.I., C...

    2024

  2. [2]

    Science382(6677), 1416–1421 (2023) https://doi.org/10.1126/science.adi2336

    Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., Ravuri, S., Ewalds, T., Eaton-Rosen, Z., Hu, W., Merose, A., Hoyer, S., Holland, G., Vinyals, O., Stott, J., Pritzel, A., Mohamed, S., Battaglia, P.: Learning skillful medium-range global weather forecasting. Science382(6677), 1416–1421 (2023) https://doi.org/10.1126/sci...

    doi:10.1126/science.adi2336 2023

  3. [3]

    Nature637(8044), 84–90 (2024) https://doi.org/10.1038/s41586-024-08252-9

    Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T.R., El-Kadi, A., Masters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., Lam, R., Willson, M.: Probabilistic weather forecasting with machine learning. Nature637(8044), 84–90 (2024) https://doi.org/10.1038/s41586-024-08252-9

    doi:10.1038/s41586-024-08252-9 2024

  4. [4]

    Preprint at https://arxiv.org/abs/2507.12144 (2025) 15

    Bonev, B., Kurth, T., Mahesh, A., Bisson, M., Kossaifi, J., Kashinath, K., Anandkumar, A., Collins, W.D., Pritchard, M.S., Keller, A.: FourCastNet 3: A geometric approach to probabilistic machine- learning weather forecasting at scale. Preprint at https://arxiv.org/abs/2507.12144 (2025) 15

    Pith/arXiv arXiv 2025

  5. [5]

    Nature Methods 21(1), 110–116 (2024) https://doi.org/10.1038/s41592-023-02087-4

    Terwilliger, T.C., Liebschner, D., Croll, T.I., Williams, C.J., McCoy, A.J., Poon, B.K., Afonine, P.V., Oeffner, R.D., Richardson, J.S., Read, R.J., Adams, P.D.: Alphafold predictions are valuable hypotheses and accelerate but do not replace experimental structure determination. Nature Methods 21(1), 110–116 (2024) https://doi.org/10.1038/s41592-023-02087-4

    work page doi:10.1038/s41592-023-02087-4 2024

  6. [6]

    International Journal of Molecular Sciences26(8) (2025) https://doi.org/10.3390/ijms26083671

    Krokidis, M.G., Koumadorakis, D.E., Lazaros, K., Ivantsik, O., Exarchos, T.P., Vrahatis, A.G., Kotsiantis, S., Vlamos, P.: Alphafold3: An overview of applications and performance insights. International Journal of Molecular Sciences26(8) (2025) https://doi.org/10.3390/ijms26083671

    work page doi:10.3390/ijms26083671 2025

  7. [7]

    Geophysical Research Letters51(12), 2023–107377 (2024) https://doi.org/10.1029/2023GL107377

    Bonavita, M.: On some limitations of current machine learning weather prediction models. Geophysical Research Letters51(12), 2023–107377 (2024) https://doi.org/10.1029/2023GL107377

    doi:10.1029/2023gl107377 2023

  8. [8]

    Preprint at https://arxiv.org/abs/ 2001.08361 (2020)

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. Preprint at https://arxiv.org/abs/ 2001.08361 (2020)

    Pith/arXiv arXiv 2001

  9. [9]

    In: Advances in Neural Information Processing Systems, vol

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Las Casas, D., Hendricks, L.A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J., Sifre, L.: An empirical analysis of compute-optimal large language model training. In...

    2022

  10. [10]

    Journal of Machine Learning Research23(226), 1–61 (2022)

    D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M.D., Hormozdiari, F., Houlsby, N., Hou, S., Jerfel, G., Karthikesalingam, A., Lucic, M., Ma, Y., McLean, C., Mincu, D., Mitani, A., Montanari, A., Nado, Z., Natarajan, V., Nielson, C., Osborne, T.F., Raman, R., Ramasamy, K., Sayres,...

    2022

  11. [11]

    In: Probabilistic and Causal Inference: The Works of Judea Pearl, pp

    Sch¨ olkopf, B.: Causality for machine learning. In: Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 765–804. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3501714.3501755

    arXiv 2022

  12. [12]

    In: Advances in Neural Information Processing Systems, vol

    Ha, D., Schmidhuber, J.: Recurrent world models facilitate policy evolution. In: Advances in Neural Information Processing Systems, vol. 31 (2018). https://proceedings.neurips.cc/paper files/paper/ 2018/file/2de5d16682c3c35007e4e92982f1a2ba-Paper.pdf

    2018

  13. [13]

    Nature640(8059), 647–653 (2025) https://doi.org/10.1038/s41586-025-08744-2

    Hafner, D., Pasukonis, J., Ba, J., Lillicrap, T.: Mastering diverse control tasks through world models. Nature640(8059), 647–653 (2025) https://doi.org/10.1038/s41586-025-08744-2

    doi:10.1038/s41586-025-08744-2 2025

  14. [14]

    DeepMind Blog (2024)

    Parker-Holder, J., Ball, P., Bruce, J., Dasagi, V., Holsheimer, K., Kaplanis, C., Moufarek, A., Scully, G., Shar, J., Shi, J., Spencer, S., Yung, J., Dennis, M., Kenjeyev, S., Long, S., Mnih, V., Chan, H., Gazeau, M., Li, B., Pardo, F., Wang, L., Zhang, L., Besse, F., Harley, T., Mitenkova, A., Wang, J., Clune, J., Hassabis, D., Hadsell, R., Bolton, A., S...

    2024

  15. [15]

    Clarendon Press, Oxford (1930)

    Aristotle: Physics. Clarendon Press, Oxford (1930). https://doi.org/10.1093/actrade/ 9780199540280.book.1

    doi:10.1093/actrade/ 1930

  16. [16]

    Clarendon Press, Oxford (1928)

    Aristotle: Posterior Analytics. Clarendon Press, Oxford (1928). https://global.oup.com/academic/ product/posterior-analytics-9780198240891

    1928

  17. [17]

    Oxford University Press, Oxford (2001)

    Losee, J.: A Historical Introduction to the Philosophy of Science, 4th edn. Oxford University Press, Oxford (2001)

    2001

  18. [18]

    Philosophy of Science15(2), 16 135–175 (1948)

    Hempel, C.G., Oppenheim, P.: Studies in the logic of explanation. Philosophy of Science15(2), 16 135–175 (1948). https://www.jstor.org/stable/185169

    1948

  19. [19]

    University of Pittsburgh Press, Pittsburgh, PA, USA (1990)

    Salmon, W.C.: Four Decades of Scientific Explanation. University of Pittsburgh Press, Pittsburgh, PA, USA (1990). http://www.jstor.org/stable/j.ctt5vkdm7

    1990

  20. [20]

    In: Machamer, P., Silberstein, M

    Woodward, J.: Explanation. In: Machamer, P., Silberstein, M. (eds.) The Blackwell Guide to the Philosophy of Science, pp. 37–54. Blackwell Publishers Ltd, Malden, Massachusetts (2002). Chap

    2002

  21. [21]

    https://doi.org/10.1002/9780470756614.ch3

    work page doi:10.1002/9780470756614.ch3

  22. [22]

    1214/10-STS330

    Shmueli, G.: To explain or to predict? Statistical Science25(3), 289–310 (2010) https://doi.org/10. 1214/10-STS330

    2010

  23. [23]

    Trends in Cognitive Sciences10(10), 464–470 (2006) https://doi.org/10.1016/j.tics.2006.08.004

    Lombrozo, T.: The structure and function of explanations. Trends in Cognitive Sciences10(10), 464–470 (2006) https://doi.org/10.1016/j.tics.2006.08.004

    doi:10.1016/j.tics.2006.08.004 2006

  24. [24]

    Oxford University Press, Oxford, UK (2004)

    Woodward, J.: Making Things Happen: A Theory of Causal Explanation. Oxford University Press, Oxford, UK (2004). https://doi.org/10.1093/0195155270.001.0001

    doi:10.1093/0195155270.001.0001 2004

  25. [25]

    Statistical Science16(3), 199–231 (2001) https://doi.org/10.1214/ss/1009213726

    Breiman, L.: Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science16(3), 199–231 (2001) https://doi.org/10.1214/ss/1009213726

    arXiv 2001

  26. [26]

    http://www.incompleteideas.net/IncIdeas/BitterLesson.html

    Sutton, R.: The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html. Accessed: 2026-06-28 (2019)

    2026

  27. [27]

    Transformer Circuits Thread (2021)

    Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., Olah, C.: A Mathematical Framework for Transformer Circuit...

    2021

  28. [28]

    In: International Conference on Learning Representations (2024)

    Huben, R., Cunningham, H., Smith, L.R., Ewart, A., Sharkey, L.: Sparse Autoencoders Find Highly Interpretable Features in Language Models. In: International Conference on Learning Representations (2024). https://openreview.net/forum?id=F76bwRSLeK

    2024

  29. [29]

    In: Proceedings of the 28th European Conference on Artificial Intelligence (ECAI 2025), pp

    Herasimchyk, H., Abdelhalim, A., Laue, S., Regneri, M.: Prediction Is Not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings. In: Proceedings of the 28th European Conference on Artificial Intelligence (ECAI 2025), pp. 1503–1510 (2025). https://arxiv.org/abs/2508.13729

    Pith/arXiv arXiv 2025

  30. [30]

    Nature595, 181–188 (2021) https://doi.org/10.1038/ s41586-021-03659-0

    Hofman, J.M., Watts, D.J., Athey, S., Garip, F., Griffiths, T.L., Kleinberg, J., Margetts, H., Mul- lainathan, S., Salganik, M.J., Vazire, S., Vespignani, A., Yarkoni, T.: Integrating explanation and prediction in computational social science. Nature595, 181–188 (2021) https://doi.org/10.1038/ s41586-021-03659-0

    2021

  31. [31]

    Philosophy of Science64(3), 461–477 (1997) https://doi.org/10.1086/392561

    Salmon, W.C.: Causality and Explanation: A Reply to Two Critiques. Philosophy of Science64(3), 461–477 (1997) https://doi.org/10.1086/392561

    work page doi:10.1086/392561 1997

  32. [32]

    Philosophy of Science67(1), 1–25 (2000) https://doi.org/10.1086/392759

    Machamer, P., Darden, L., Craver, C.F.: Thinking about Mechanisms. Philosophy of Science67(1), 1–25 (2000) https://doi.org/10.1086/392759

    doi:10.1086/392759 2000

  33. [33]

    In: Kitcher, P., Salmon, W

    Kitcher, P.: Explanatory unification and the causal structure of the world. In: Kitcher, P., Salmon, W. (eds.) Scientific Explanation. Minnesota Studies in the Philosophy of Science, vol. 13, pp. 410–505. University of Minnesota Press, Minneapolis (1989). https://hdl.handle.net/11299/185687

    1989

  34. [34]

    Marr, D.: Vision: A Computational Investigation Into the Human Representation and Processing of Visual Information. W. H. Freeman, San Francisco (1982). https://doi.org/10.7551/mitpress/ 9780262514620.001.0001

    doi:10.7551/mitpress/ 1982

  35. [35]

    Proceedings of the American Philosophical Society 106(6), 467–482 (1962)

    Simon, H.A.: The architecture of complexity. Proceedings of the American Philosophical Society 106(6), 467–482 (1962). https://www.jstor.org/stable/985254

    1962

  36. [36]

    In: 17 Advances in Neural Information Processing Systems, vol

    Dunefsky, J., Chlenski, P., Nanda, N.: Transcoders find interpretable LLM feature circuits. In: 17 Advances in Neural Information Processing Systems, vol. 37, pp. 24375–24410 (2024). https://doi. org/10.52202/079017-0768

    doi:10.52202/079017-0768 2024

  37. [37]

    Transformer Circuits Thread (2025)

    Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N.L., Chen, B., Citro, C., Abrahams, D., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunning- ham, H., Henighan, T., Jermyn, A., Jones, A., Persic, A., Qi, Z., Ben Thompson, T., Zimmerman, S., Rivoire, K., Conerly, T., Olah, C., Batson, J.: Circuit trac...

    2025

  38. [38]

    Nature Methods22(10), 2107–2117 (2025) https://doi.org/10.1038/ s41592-025-02836-7

    Simon, E., Zou, J.: InterPLM: discovering interpretable features in protein language models via sparse autoencoders. Nature Methods22(10), 2107–2117 (2025) https://doi.org/10.1038/ s41592-025-02836-7

    2025

  39. [39]

    Proceedings of the National Academy of Sciences 122(34), 2506316122 (2025) https://doi.org/10.1073/pnas.2506316122

    Gujral, O., Bafna, M., Alm, E., Berger, B.: Sparse autoencoders uncover biologically interpretable features in protein language model representations. Proceedings of the National Academy of Sciences 122(34), 2506316122 (2025) https://doi.org/10.1073/pnas.2506316122

    doi:10.1073/pnas.2506316122 2025

  40. [40]

    Brixi, G., Durrant, M.G., Ku, J., Naghipourfar, M., Poli, M., Sun, G., Brockman, G., Chang, D., Fanton, A., Gonzalez, G.A., King, S.H., Li, D.B., Merchant, A.T., Nguyen, E., Ricci-Tam, C., Romero, D.W., Schmok, J.C., Taghibakhshi, A., Vorontsov, A., Yang, B., Deng, M., Gorton, L., Nguyen, N., Wang, N.K., Pearce, M.T., Simon, E., Adams, E., Amador, Z.J., A...

    2026

  41. [41]

    Science379(6637), 1123–1130 (2023) https://doi.org/10.1126/science.ade2574

    Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S., Rives, A.: Evolutionary-scale prediction of atomic-level protein structure with a language model. Science379(6637), 1123–1130 (2023) https://doi.org/10.1126/science.ade2574

    doi:10.1126/science.ade2574 2023

  42. [42]

    Nature618(7965), 616–624 (2023) https://doi.org/10.1038/s41586-023-06139-9

    Theodoris, C.V., Xiao, L., Chopra, A., Chaffin, M.D., Al Sayed, Z.R., Hill, M.C., Mantineo, H., Brydon, E.M., Zeng, Z., Liu, X.S., Ellinor, P.T.: Transfer learning enables predictions in network biology. Nature618(7965), 616–624 (2023) https://doi.org/10.1038/s41586-023-06139-9

    doi:10.1038/s41586-023-06139-9 2023

  43. [43]

    Nature methods21(8), 1470–1480 (2024) https://doi.org/10.1038/s41592-024-02201-0

    Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., Duan, N., Wang, B.: scGPT: toward building a foundation model for single-cell multi-omics using generative AI. Nature methods21(8), 1470–1480 (2024) https://doi.org/10.1038/s41592-024-02201-0

    doi:10.1038/s41592-024-02201-0 2024

  44. [44]

    Preprint at https:// arxiv.org/abs/2602.17532 (2026)

    Kendiukhov, I.: Systematic Evaluation of Single-Cell Foundation Model Interpretability Reveals Attention Captures Co-Expression Rather Than Unique Regulatory Signal. Preprint at https:// arxiv.org/abs/2602.17532 (2026)

    arXiv 2026

  45. [45]

    Nature methods22(8), 1657–1661 (2025) https: //doi.org/10.1038/s41592-025-02772-6

    Ahlmann-Eltze, C., Huber, W., Anders, S.: Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. Nature methods22(8), 1657–1661 (2025) https: //doi.org/10.1038/s41592-025-02772-6

    doi:10.1038/s41592-025-02772-6 2025

  46. [46]

    In: Proceedings of the 42nd International Conference on Machine Learning

    Vafa, K., Chang, P.G., Rambachan, A., Mullainathan, S.: What has a foundation model found? Using inductive bias to probe for world models. In: Proceedings of the 42nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 267, pp. 60727–60747 (2025). https://proceedings.mlr.press/v267/vafa25a.html

    2025

  47. [47]

    Cambridge University Press, Cambridge, UK (2009)

    Pearl, J.: Causality, 2nd edn. Cambridge University Press, Cambridge, UK (2009). https://doi.org/ 10.1017/CBO9780511803161

    doi:10.1017/cbo9780511803161 2009

  48. [48]

    In: Proceed- ings of the 36th International Conference on Machine Learning

    Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨ olkopf, B., Bachem, O.: Challenging 18 Common Assumptions in the Unsupervised Learning of Disentangled Representations. In: Proceed- ings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 4114–4124 (2019). http://proceedings.mlr...

    2019

  49. [49]

    Proceedings of the IEEE109(5), 612–634 (2021) https://doi.org/10

    Sch¨ olkopf, B., Locatello, F., Bauer, S., Ke, N.R., Kalchbrenner, N., Goyal, A., Bengio, Y.: Toward causal representation learning. Proceedings of the IEEE109(5), 612–634 (2021) https://doi.org/10. 1109/JPROC.2021.3058954

    arXiv 2021

  50. [50]

    In: Proceedings of the First Conference on Causal Learning and Reasoning

    Lachapelle, S., Rodriguez Lopez, P., Sharma, Y., Everett, K.E., Le Priol, R., Lacoste, A., Lacoste- Julien, S.: Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In: Proceedings of the First Conference on Causal Learning and Reasoning. Proceedings of Machine Learning Research, vol. 177, pp. 428–484 (2022). https://p...

    2022

  51. [51]

    In: Advances in Neural Information Processing Systems, vol

    von K¨ ugelgen, J., Besserve, M., Wendong, L., Gresele, L., Keki´ c, A., Bareinboim, E., Blei, D., Sch¨ olkopf, B.: Nonparametric identifiability of causal representations from unknown interventions. In: Advances in Neural Information Processing Systems, vol. 36, pp. 48603–48638 (2023). https: //openreview.net/forum?id=V87gZeSOL4

    2023

  52. [52]

    In: Proceedings of the 39th International Conference on Machine Learning

    Lippe, P., Magliacane, S., L¨ owe, S., Asano, Y.M., Cohen, T., Gavves, S.: CITRIS: Causal identifia- bility from temporal intervened sequences. In: Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 13557–13603 (2022). https://proceedings.mlr.press/v162/lippe22a.html

    2022

  53. [53]

    In: Proceedings of the 35th International Conference on Machine Learning

    Parascandolo, G., Kilbertus, N., Rojas-Carulla, M., Sch¨ olkopf, B.: Learning independent causal mechanisms. In: Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 4036–4044 (2018). https://proceedings.mlr.press/v80/ parascandolo18a.html

    2018

  54. [54]

    MIT Press, Cambridge, MA, USA (2017)

    Peters, J., Janzing, D., Sch¨ olkopf, B.: Elements of Causal Inference — Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA (2017). https://mitpress.mit.edu/9780262037310/ elements-of-causal-inference/

    arXiv 2017

  55. [55]

    Journal of Machine Learning Research21(89), 1–53 (2020)

    Huang, B., Zhang, K., J., Z., Ramsey, J., Sanchez-Romero, R., Glymour, C., Sch¨ olkopf, B.: Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research21(89), 1–53 (2020). http://jmlr.org/papers/v21/19-232.html

    2020

  56. [56]

    Science 324(5923), 81–85 (2009) https://www.science.org/doi/pdf/10.1126/science.1165893

    Schmidt, M., Lipson, H.: Distilling free-form natural laws from experimental data. Science 324(5923), 81–85 (2009) https://www.science.org/doi/pdf/10.1126/science.1165893

    doi:10.1126/science.1165893 2009

  57. [57]

    Proceedings of the National Academy of Sciences113(15), 3932–3937 (2016) https://doi.org/10.1073/pnas.1517384113

    Brunton, S.L., Proctor, J.L., Kutz, J.N.: Discovering governing equations from data by sparse identi- fication of nonlinear dynamical systems. Proceedings of the National Academy of Sciences113(15), 3932–3937 (2016) https://doi.org/10.1073/pnas.1517384113

    doi:10.1073/pnas.1517384113 2016

  58. [58]

    In: Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks, vol

    La Cava, W., Orzechowski, P., Burlacu, B., de Franca, F.O., Virgolin, M., Jin, Y., Kom- menda, M., Moore, J.: Contemporary symbolic regression methods and their relative performance. In: Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks, vol. 1 (2021). https://datasets-benchmarks-proceedings.neurips.cc/paper files...

    2021

  59. [59]

    In: International Conference on Learning Representations (2025)

    Yu, Z., Ding, J., Li, Y., Jin, D.: Symbolic regression via MDLformer-guided search: from mini- mizing prediction error to minimizing description length. In: International Conference on Learning Representations (2025). https://openreview.net/forum?id=ljAS7cPAU0

    2025

  60. [60]

    The MIT Press, Cambridge, MA, USA (2007)

    Gr¨ unwald, P.D.: The Minimum Description Length Principle. The MIT Press, Cambridge, MA, USA (2007). https://doi.org/10.7551/mitpress/4643.001.0001

    doi:10.7551/mitpress/4643.001.0001 2007

  61. [61]

    Science Advances6(16), 2631 (2020) https://doi.org/10.1126/sciadv.aay2631 19

    Udrescu, S.-M., Tegmark, M.: AI Feynman: A physics-inspired method for symbolic regression. Science Advances6(16), 2631 (2020) https://doi.org/10.1126/sciadv.aay2631 19

    doi:10.1126/sciadv.aay2631 2020

  62. [62]

    In: Advances in Neural Information Processing Systems, vol

    Cranmer, M., Sanchez Gonzalez, A., Battaglia, P., Xu, R., Cranmer, K., Spergel, D., Ho, S.: Discov- ering symbolic models from deep learning with inductive biases. In: Advances in Neural Information Processing Systems, vol. 33, pp. 17429–17442 (2020). https://proceedings.neurips.cc/paper files/ paper/2020/hash/c9f2f917078bd2db12f23c3b413d9cba-Abstract.html

    2020

  63. [63]

    In: International Conference on Learning Representations (2017)

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In: International Conference on Learning Representations (2017). https://openreview.net/forum?id=B1ckMDqlg

    2017

  64. [64]

    Journal of Machine Learning Research23(120), 1–39 (2022)

    Fedus, W., Zoph, B., Shazeer, N.: Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research23(120), 1–39 (2022). https: //www.jmlr.org/papers/v23/21-0998.html

    2022

  65. [65]

    In: Eigth Annual Conference on Robot Learning (2024)

    Wang, Y., Zhang, Y., Huo, M., Tian, R., Zhang, X., Xie, Y., Xu, C., Ji, P., Zhan, W., Ding, M., Tomizuka, M.: Sparse diffusion policy: A sparse, reusable, and flexible policy for robot learn- ing. In: Eigth Annual Conference on Robot Learning (2024). https://openreview.net/forum?id= zeYaLS2tw5

    2024

  66. [66]

    IEEE Transac- tions on Information Theory56(10), 5168–5194 (2010) https://doi.org/10.1109/TIT.2010.2060095

    Janzing, D., Sch¨ olkopf, B.: Causal inference using the algorithmic Markov condition. IEEE Transac- tions on Information Theory56(10), 5168–5194 (2010) https://doi.org/10.1109/TIT.2010.2060095

    arXiv 2010

  67. [67]

    Nature651(8107), 914–919 (2026) https://doi.org/10.1038/ s41586-026-10265-5

    Lu, C., Lu, C., Lange, R.T., Yamada, Y., Hu, S., Foerster, J., Ha, D., Clune, J.: Towards end- to-end automation of AI research. Nature651(8107), 914–919 (2026) https://doi.org/10.1038/ s41586-026-10265-5

    2026

  68. [68]

    Nature655, 487–496 (2026) https://doi.org/10.1038/s41586-026-10644-y

    Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Sirkovic, P., Myaskovsky, A., Glowaty, G., Weis- senberger, F., Orlandi, A., Popovici, D., Palepu, A., Rong, K., Tanno, R., Saab, K., Zhang, F., Blum, J., Carroll, A., Kulkarni, K., Tomaˇ sev, N., Zverinski, D., Rendulic, I., Vedadi, E., Hasler, F., Rimanic, L., Boia, M., Budiselic, I., Feinstein, B., Bellaic...

    2026

  69. [69]

    Nature655, 497–505 (2026) https: //doi.org/10.1038/s41586-026-10652-y

    Ghareeb, A.E., Chang, B., Mitchener, L., Yiu, A., Szostkiewicz, C.J., Shved, D., Gyimesi, G.J., Laurent, J.M., Wright, S.M., Razzak, M.T., White, A.D., Finnemann, S.C., Hinks, M.M., Rodriques, S.G.: A multi-agent system for automating scientific discovery. Nature655, 497–505 (2026) https: //doi.org/10.1038/s41586-026-10652-y

    doi:10.1038/s41586-026-10652-y 2026

  70. [70]

    Automated Experimentation2(1) (2010) https://doi.org/10.1186/ 1759-4499-2-1

    Sparkes, A.C., Aubrey, W., Byrne, E., Clare, A., Khan, M.N., Liakata, M., Markham, M., Row- land, J., Soldatova, L.N., Whelan, K.E., Young, M., King, R.D.: Towards robot scientists for autonomous scientific discovery. Automated Experimentation2(1) (2010) https://doi.org/10.1186/ 1759-4499-2-1

    2010

  71. [71]

    Preprint at https://arxiv.org/abs/2406.17835 (2024)

    Gower, A.H., Korovin, K., Brunns ˚ aker, D., Kronstr¨ om, F., Reder, G.K., Tiukova, I.A., Reiserer, R.S., Wikswo, J.P., King, R.D.: The Use of AI-Robotic Systems for Scientific Discovery. Preprint at https://arxiv.org/abs/2406.17835 (2024)

    arXiv 2024

  72. [72]

    Transactions on Machine Learning Research (2023)

    Lei, A., Sch¨ olkopf, B., Posner, I.: Variational causal dynamics: Discovering modular world models from interventions. Transactions on Machine Learning Research (2023). https://openreview.net/ forum?id=V9tQKYYNK1

    2023

  73. [73]

    Preprint at https://arxiv.org/abs/2404.15109 (2024)

    Lei, A., Nolte, F., Sch¨ olkopf, B., Posner, I.: Compete and compose: Learning independent mechanisms for modular world models. Preprint at https://arxiv.org/abs/2404.15109 (2024)

    Pith/arXiv arXiv 2024

  74. [74]

    In: Advances in Neural Information Processing Systems, vol

    Lei, A., Sch¨ olkopf, B., Posner, I.: Spartan: A sparse transformer world model attending to what matters. In: Advances in Neural Information Processing Systems, vol. 38, pp. 154089–154114 (2025). https://openreview.net/forum?id=uS5ch7GjZ4 20

    2025

  75. [75]

    In: Fifth Conference on Causal Learning and Reasoning (2026)

    Baumgartner, M.W., Lei, A., Watson, J., Posner, I.: Disentangling dynamical systems: Causal rep- resentation learning meets local sparse attention. In: Fifth Conference on Causal Learning and Reasoning (2026). https://openreview.net/forum?id=5jPq695J19

    2026

  76. [76]

    In: Forty-third International Conference on Machine Learning (2026)

    Nam, H., Le Lidec, Q., Maes, L., LeCun, Y., Balestriero, R.: Causal-JEPA: Learning world models through object-level latent masking. In: Forty-third International Conference on Machine Learning (2026). https://openreview.net/forum?id=VMAHQDOtjp

    2026

  77. [77]

    In: International Conference on Learning Representations (2021)

    Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., Sch¨ olkopf, B.: Recurrent independent mechanisms. In: International Conference on Learning Representations (2021). https: //openreview.net/forum?id=mLcmdlEUxy-

    2021

  78. [78]

    In: International Conference on Learning Representations (2020)

    Kipf, T., van der Pol, E., Welling, M.: Contrastive learning of structured world models. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id= H1gax6VtDB

    2020

  79. [79]

    Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences381(2251), 20220050 (2023) https://doi.org/10.1098/rsta.2022

    Ellis, K., Wong, L., Nye, M., Sabl´ e-Meyer, M., Cary, L., Anaya Pozo, L., Hewitt, L., Solar-Lezama, A., Tenenbaum, J.B.: DreamCoder: growing generalizable, interpretable knowledge with wake–sleep bayesian program learning. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences381(2251), 20220050 (2023) https://...

    work page doi:10.1098/rsta.2022 2023

  80. [80]

    In: Forty-third International Conference on Machine Learning (2026)

    Baek, D., Lee, G., Baek, J., Lee, H., Ahn, S.: Learning to theorize the world from observation. In: Forty-third International Conference on Machine Learning (2026). https://openreview.net/forum? id=wsA8LgHU5U

    2026

Showing first 80 references.