{"id":"9305e9d8-18cc-44a2-a7d1-66544cb4994b","arxiv_id":"2505.00808","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Mechanistic interpretability is defined as producing model-level, ontic, causal-mechanistic, falsifiable explanations, and a new 'Explanatory Optimism' conjecture is introduced as a necessary precondition for its success.","lead":"This paper argues that mechanistic interpretability is a principled science because trained neural networks contain hidden, extractable explanations of their own behavior. It proposes a precise definition of mechanistic interpretability and introduces a conjecture about whether machine concepts can ever be fully understood by humans.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The uniqueness of ur-explanations is asserted, not established; the §3.2.2 activation-matching criterion is basis-dependent, so Explanatory Faithfulness is not well-defined even if the No Miracles Argument is granted.","rationale":"The paper's headline contribution is that Explanatory Faithfulness is well-defined, and the proof route goes through ur-explanations and their uniqueness in Appendix B. The reader's weakest assumption identifies the No Miracles Argument as the fragile premise. My stress-test sharpens this: even if one grants that the NMA transfers to neural networks, the uniqueness conclusion does not follow, and the paper's own operational definition makes the difficulty visible. Activations are representation-dependent; linear reparameterizations produce behaviorally identical models with different activation vectors, so an activation-matching criterion cannot pick out a unique internal explanation unless a canonical basis or equivalence class is specified. This is a correctness risk inside the formalism, not merely a contested philosophical stance. It also coheres with Section 5.1, where explanations are described as value- and theory-laden, which sits uneasily with an unqualified uniqueness claim. The fix is available: define faithfulness relative to an explicitly chosen feature basis or equivalence class, and drop or qualify the uniqueness claim in B.3. The paper therefore remains conditionally acceptable, and I leave the reader's conditional verdict unchanged; however, the stated condition should be that uniqueness must either be proven or explicitly given up, not merely that the NMA must be accepted.","tokens_in":28046,"tokens_out":7651,"duration_ms":85712,"concrete_test":"Implement two two-layer linear networks M and M' with identical input-output behavior but activations related by a random invertible U, namely W1' = U W1 and W2' = W2 U^{-1}. Let E be the exact circuit that reproduces M's intermediate activations. Evaluate the Section 3.2.2 activation-matching criterion (e.g., mean squared error between s_i and x_i) for E against M and against M'. If the scores differ for behaviorally identical models, the definition does not pick out a unique ur-explanation. Then check whether any argument in Appendix B rules out such reparameterizations; it does not. A nonzero score difference settles the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Explanatory Faithfulness is well-defined rests on the existence and uniqueness of a ur-explanation. Existence is nearly trivial, since the forward trace of M is always available (Appendix F), but uniqueness is not established. Appendix B.3 asserts 'ur-explanations are to be unique' as a consequence of the No Miracles Argument, yet the NMA, even if valid for neural networks, supports at most the approximate truth of some representation, not the uniqueness of a privileged internal decomposition. The operational definition in Section 3.2.2 makes the problem concrete: it scores faithfulness by comparing intermediate activations s_i produced by E with the model's activations x_i. Raw activations are not canonical. For any invertible matrix U, reparameterizing a layer's weights as W_i' = U W_i and the downstream weights accordingly leaves the input-output map unchanged but changes x_i by U. Two behaviorally identical networks therefore receive different faithfulness scores, so 'the' ur-explanation is not unique under the definition. This is not a dispute about scientific realism; it is a failure of well-definedness internal to the paper's own formalism. The paper's own Section 5.1.2 commitment to theory-ladenness and 'Dmitry's Koan' only reinforces that no scale-invariant canonical decomposition is supplied.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues for an Explanatory View Hypothesis: trained neural networks contain implicit explanations of their own behaviour (ur-explanations), and therefore mechanistic interpretability (MI) has a well-defined target. Explanatory Faithfulness is defined as the degree to which an algorithmic explanation's intermediate activations match the model's intermediate activations. The paper proposes a demarcation of MI as the production of Model-level, Ontic, Causal-Mechanistic, and Falsifiable explanations, discusses limits of MI arising from value-ladenness, theory-ladenness, model-level versus system-level analysis, and low-abstraction explanations, and formulates the Principle of Explanatory Optimism as a conjecture about human-understandability of machine concepts. Appendix B attempts to ground the existence and uniqueness of ur-explanations via the No Miracles Argument.","tokens_in":28214,"tokens_out":3651,"duration_ms":37955,"significance":"If the central claim were established, the paper would give MI a principled target and would usefully separate explanatory faithfulness from behavioural faithfulness, which is a substantive contribution. The paper also provides a clear and useful demarcation of MI, and it explicitly formulates Explanatory Optimism as a testable conjecture, which is valuable for focusing future work. The paper is a serious synthesis of philosophical and technical themes, and it cites relevant literature, including works that challenge its own assumptions. However, the central claim that Explanatory Faithfulness is well-defined is not currently supported: the uniqueness of ur-explanations is asserted rather than derived, and the operational activation-matching criterion is basis-dependent.","major_comments":[{"comment":"The uniqueness of the ur-explanation is load-bearing for the claim that Explanatory Faithfulness is well-defined, but it is asserted rather than established. Appendix B.3 concludes that 'ur-explanations are to be unique', yet no derivation is given; the No Miracles Argument, even if it transferred successfully to neural networks, would at most support the approximate truth of some representation or internal structure, not the uniqueness of a privileged decomposition. Appendix F provides a forward trace, which gives existence of at least one implementation-level explanation, but existence does not imply uniqueness. Without uniqueness, the definition of explanatory faithfulness as matching 'the' ur-explanation is not well-defined.","section":"Appendix B.3 and Section 3.2.2"},{"comment":"The operational criterion for Explanatory Faithfulness is basis-dependent, so it fails to define a unique target even if the No Miracles Argument is granted. The definition compares intermediate activations s_i from the explanation E with the model's activations x_i. However, for any invertible matrix U, reparameterizing the layer-i weights as W_i' = U W_i and the downstream weights accordingly leaves the input-output map unchanged while changing the activations x_i to U x_i. Two behaviorally identical parameterizations of the same model would therefore assign different faithfulness scores to the same explanation E. The paper does not supply a canonical decomposition, and its own discussion of theory-ladenness in Section 5.1.2, including Dmitry's Koan, reinforces that no scale-invariant or interpreter-independent decomposition is provided. This is a failure of well-definedness internal to the formalism, independent of the status of the No Miracles Argument.","section":"Section 3.2.2"},{"comment":"The paper claims that the Principle of Explanatory Optimism is a necessary precondition for the success of MI, but this necessity is not established. Section 4 defines MI without any reference to human understanding, and Section 7.1 argues only that 'it is not clear how MI research can proceed' if alien concepts are prevalent, which is weaker than the claim that EO is necessary. The contribution statement says 'We show that without the Principle of Explanatory Optimism the project of MI is intractable', but the body of the paper presents EO as a conjecture and explicitly says no arguments for its truth are provided. The necessity claim needs either a direct argument or a reframing as a working assumption rather than a demonstrated precondition.","section":"Section 7.2 and Contributions"}],"minor_comments":[{"comment":"There is a typo in Appendix F: 'Implementation leval' should be 'Implementation level'.","section":"Appendix F"},{"comment":"The reference list contains a typo: 'Thomasn D. Aquinas' should be 'Thomas D. Aquinas'.","section":"References"},{"comment":"The relation between 'learned internal structures' in the definition of ur-explanations and 'intermediate activations' in the operational definition of explanatory faithfulness could be stated more explicitly; as written, the two notions may not coincide, and the paper does not clarify whether the ur-explanation is the full computational trace or a compressed abstraction over it.","section":"Section 3.2.1"},{"comment":"The naming of 'Dmitry's Koan++' as 'Nora's Koan' is informal; if retained, a brief justification would help readers unfamiliar with the provenance.","section":"Section 7.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a timely and ambitious philosophy-of-MI contribution, and the demarcation and the Explanatory Optimism conjecture are likely to be useful to the community. The central problem is that the 'well-definedness' claim is not backed by a derivation: uniqueness of ur-explanations is asserted, and the activation-matching criterion is basis-dependent. I would be willing to reconsider after revision if the authors either prove uniqueness under explicit assumptions or explicitly relativize explanatory faithfulness to a chosen decomposition or basis, and if they either prove or clearly reframe the necessity claim about Explanatory Optimism. I do not see grounds for rejection on novelty or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take on arXiv:2505.00808. It is a programmatic philosophy paper, not a finished theory. The useful core is the MOCF demarcation of mechanistic interpretability and the explicit formulation of Explanatory Optimism; the weak joint is the claim that explanatory faithfulness is well-defined.\n\nWhat the paper does well: the demarcation is clean and gives researchers a concrete way to say what counts as MI. The distinction between behavioral and explanatory faithfulness is worth taking seriously. The EO conjecture is honestly labeled as unproven, and the Call to Action is a reasonable research agenda. The discussion of theory-ladenness and system-level limits is thoughtful and well-connected to existing work.\n\nThe soft spots are concentrated in the justification layer. First, uniqueness of ur-explanations is asserted, not established. Appendix B.3 says ur-explanations \"are to be unique\" as a consequence of the No Miracles Argument, but the NMA at most supports realism about some features; it does not single out a privileged decomposition of a network's computation. The activation-matching definition in Section 3.2.2 makes this concrete: raw activations depend on a basis, and reparameterizing a layer with an invertible matrix leaves the input-output map unchanged while changing every intermediate activation. To be fair, a reparameterized network is a different model, so this is not a direct counterexample to uniqueness for a fixed M. But it does show that the definition does not pin down an invariant target. If two equally good internal decompositions exist for the same M, explanatory faithfulness is not well-defined. The paper should either supply an argument for uniqueness or reframe faithfulness as relative to a chosen decomposition.\n\nSecond, the abstract's \"we hence show\" overstates what the body delivers. The paper stipulates and argues abductively; that is acceptable for a philosophy paper, but the language should match the epistemic status.\n\nThird, Rowbottom et al. 2024 is cited as part of the NMA tradition, but that paper's title signals a critical stance on applying the NMA to AI. If the authors rely on the NMA, they need to engage that critique directly. Also, the claim that EO is necessary depends on the assumption that MI's goal includes human understanding, which is not part of the Section 4 definition; that connection should be explicit.\n\nOverall, the central argument has a real gap, but the paper is serious and worth engaging. I would send it to a philosopher of science and an MI researcher. Ask for revisions on the uniqueness/basis issue and on the abstract's overclaiming; then accept. It deserves a real referee, not a desk reject.","headline":"Programmatic philosophy of MI with a useful demarcation and an honest conjecture, but the well-definedness claim rests on an unproven uniqueness assertion.","tokens_in":28860,"tokens_out":3843,"would_cite":true,"duration_ms":41105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A trained neural network contains an implicit explanation of its own behavior, so mechanistic interpretability can be a principled science with a well-defined target: explanatory faithfulness.","keywords":["mechanistic interpretability","explanatory faithfulness","ur-explanation","No Miracles Argument","Explanatory Optimism","causal explanation","neural representations","model-level explanation"],"falsifier":"For one concrete check, take a trained, generalising model and a fixed data distribution, then search over candidate explanations of increasing complexity for the best layerwise activation match; if the minimum mismatch stays bounded away from zero even as explanation complexity grows, explanatory faithfulness as defined has no attainable target.","tokens_in":1727,"feed_emoji":"🧠","tokens_out":1926,"duration_ms":72009,"temperature":0.7,"pith_summary":"The paper argues the Explanatory View Hypothesis: neural networks that generalise contain internal ur-explanations, idealised accounts of their behavior in terms of learned representations, and the job of mechanistic interpretability is to extract them. On this view, an explanation is explanatorily faithful when its layer-by-layer intermediate activations match the model's activations, making faithfulness a well-defined target rather than a behavioral or aesthetic judgment. The paper also demarcates mechanistic interpretability as the study of Model-level, Ontic, Causal-Mechanistic, and Falsifiable explanations, and it introduces Explanatory Optimism, the conjecture that the important concepts inside such networks are human-understandable, which it argues is a necessary precondition for the field's success. A sympathetic reader should care because if these claims hold, interpretability becomes an evaluable science: explanations can be right or wrong about what a model actually does.","feed_headline":"Neural networks carry implicit self-explanations","feed_subtitle":"Faithful explanations match a model's internal activity layer by layer, giving the field a measurable target.","key_machinery":"The load-bearing concept is the ur-explanation: the idealised explanation of a model's behavior over a data distribution, given in terms of the model's learned internal structures and computations over representations. It carries the argument by supplying a target for interpretability; with it, explanatory faithfulness can be defined as layerwise activation matching, and without it, claims about circuit equivalence reduce to behavioral statistics. A second mechanism is a two-step No Miracles Argument: first, a model's extraordinary predictive success justifies realism about its internal features; second, an interpretation that successfully predicts activations at each layer under causal interventions is likely to be approximately faithful to the unique ur-explanation. Representations themselves are characterised by three criteria: information, use, and the possibility of misrepresentation.","core_discovery":"The paper's central claim is that a trained, generalising neural network already contains an implicit explanation of its own behavior, called the ur-explanation, as computations over learned representations, so understanding is not imposed on the model from outside but extracted from it. Explanatory faithfulness is then defined operationally: an explanation E matches model M over data distribution D to the degree that the intermediate activations predicted by E at each layer match the model's actual intermediate activations. The paper claims this definition is only possible once ur-explanations are accepted, and it uses the No Miracles Argument, transferred to neural networks, to support the existence and uniqueness of ur-explanations. It further claims that mechanistic interpretability is demarcated as the study of Model-level, Ontic, Causal-Mechanistic, and Falsifiable explanations and that Explanatory Optimism, the conjecture that important machine concepts are human-understandable, is a necessary precondition for the project to succeed.","pith_inferences":["A natural extension is to turn explanatory faithfulness into a quantitative benchmark: average layerwise activation divergence between model and explanation over a held-out distribution; the paper gestures at this through hypothesis testing but does not propose a specific metric.","The uniqueness claim for ur-explanations could be sharpened as a testable prediction: independently trained networks that solve the same task and generalise equally well should converge on the same high-level causal structure, and robust divergence across such networks would put pressure on uniqueness.","Explanatory Optimism could be probed empirically by measuring whether increasing model scale expands the residue of concepts that resist translation into human-comprehensible abstractions; if that residue grows with capability, weak optimism would fail.","If explanatory faithfulness is accepted as the goal, then explanation quality should be evaluated by intervention-based activation prediction rather than by human ratings, which would shift interpretability evaluation toward causal and mechanistic criteria."],"forward_implications":["Mechanistic interpretability explanations can be objectively evaluated as faithful or unfaithful by comparing layerwise activations, not just matching input-output behavior.","The field gains a demarcation criterion: only explanations that are Model-level, Ontic, Causal-Mechanistic, and Falsifiable count as mechanistic interpretability.","If Explanatory Optimism is false, mechanistic interpretability cannot achieve its stated goal of understanding generalising neural networks, so the conjecture's truth value is load-bearing for the research agenda.","Models can be trusted for content-based reasons, because extracting an explanation faithful to the model's internal mechanisms provides reasons for a prediction rather than just a track record.","The success of neural networks as predictors becomes evidence that their internal representations correspond to real structure in the world, making interpretability a route to scientific discovery."],"supporting_citations":[{"why":"Supplies the original No Miracles Argument that the paper adapts to neural networks.","marker":"(Putnam, 1979)"},{"why":"Provides the Bayesian formalisation of the No Miracles Argument used to justify realism about features.","marker":"(Dawid & Hartmann, 2016)"},{"why":"Engages the question of whether the No Miracles Argument applies to AI systems.","marker":"(Rowbottom et al., 2024)"},{"why":"Supplies the Information, Use, and Misrepresentation criteria that define when activations count as representations.","marker":"(Harding, 2023)"},{"why":"Proposes hypothesis testing for circuit claims, which the paper takes as practical evidence of explanatory faithfulness.","marker":"(Shi et al., 2024)"},{"why":"Defines the three levels of analysis used to locate explanatory faithfulness at the algorithmic level.","marker":"(Marr, 1982)"},{"why":"Presents the classical inductive black-box view of machine learning against which the Explanatory View is defined.","marker":"(Andrews, 2023)"},{"why":"Provides the circuits research program and conceptual-engineering examples that motivate the Explanatory View.","marker":"(Olah et al., 2020)"},{"why":"Gives a concrete mechanistic explanation whose layerwise success illustrates what explanatory faithfulness would require.","marker":"(Nanda et al., 2023)"}],"fun_headline_variants":["Neural nets hide their own explanations","Model's inner logic is an implicit explanation","Explanation faithfulness defined via activation matching","Ur-explanations: models explain themselves","Mechanistic interpretability extracts built-in self-explanations"],"cache_read_input_tokens":30848,"weakest_assumption_plain":"The load-bearing premise is that the No Miracles Argument transfers from natural science to neural networks, so a generalising model's predictive success justifies believing that its internal representations track real entities and that its ur-explanation exists and is unique.","fun_headline_variants_meta":{"raw":{"variants":["Neural nets hide their own explanations","Model's inner logic is an implicit explanation","Explanation faithfulness defined via activation matching","Ur-explanations: models explain themselves","Mechanistic interpretability extracts built-in self-explanations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1238,"prompt_tokens":863,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":308}},"tokens_in":479,"tokens_out":375,"duration_ms":3693,"temperature":1.0,"reasoning_tokens":308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:35:29.893447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For one concrete check, take a trained, generalising model and a fixed data distribution, then search over candidate explanations of increasing complexity for the best layerwise activation match; if the minimum mismatch stays bounded away from zero even as explanation complexity grows, explanatory faithfulness as defined has no attainable target.","supporting_citations":[],"review_version":1}