pith. sign in

arxiv: 2606.29346 · v1 · pith:GOJWBNY5new · submitted 2026-06-28 · 💻 cs.LG

Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

Nick Oh , Helen Jin This is my paper

Pith reviewed 2026-06-30 08:17 UTC · model grok-4.3

classification 💻 cs.LG
keywords post-hoc explanationsscientific machine learningmodel reliabilityexplanation faithfulnessopaque modelsmechanistic correspondenceinterpretability limits
0
0 comments X

The pith

Post-hoc explanations cannot by themselves support claims about the structure of scientific phenomena.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that reliability of a model, meaning its predictions match the phenomenon's outcomes, and faithfulness of an explanation, meaning the explanation matches the model, fall short of what is needed for claims about how the phenomenon is structured. Structural claims require that the model works as the phenomenon works, a condition neither check addresses. This matters because scientists routinely treat reliable models with faithful explanations as sources of insight into real mechanisms, yet the paper shows the chain can at best suggest hypotheses that still need outside validation.

Core claim

Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.

What carries the argument

Mechanistic correspondence between model and phenomenon, the missing requirement beyond reliability and faithfulness.

If this is right

  • Explanations can generate candidate hypotheses but cannot support structural claims without external corroboration.
  • Claims that an explanation reveals how a phenomenon is structured require more than reliability and faithfulness.
  • Post-hoc methods have built-in limits when the goal is scientific insight into opaque models rather than prediction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Researchers aiming for structural understanding may need to prioritize models with built-in mechanistic constraints over post-hoc analysis of black-box models.
  • The argument suggests a practical test: examine published scientific papers that draw structural conclusions from explanations and check whether they supply independent evidence of mechanistic correspondence.
  • This distinction could apply to other interpretability techniques beyond post-hoc methods when the target is understanding real-world mechanisms.

Load-bearing premise

A claim about the structure of a phenomenon requires the model to work as the phenomenon works rather than merely matching outcomes or internal model behavior.

What would settle it

A documented scientific case in which a reliable model and faithful explanation alone establish a correct structural claim about the phenomenon with no external corroboration of mechanistic correspondence.

read the original abstract

Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper argues that post-hoc explanation methods for opaque scientific machine learning models cannot, on their own, deliver insight into the structure of the target phenomenon. Reliability ensures that model predictions match observed outcomes in the phenomenon, while faithfulness ensures that the explanation matches the model's internal behavior; neither property establishes that the model implements the same causal mechanisms or structure as the phenomenon itself. The manuscript concludes that such methods can at best generate candidate hypotheses when supplemented by external corroboration, but cannot license structural claims without additional evidence of mechanistic correspondence.

Significance. If the distinction holds, the result clarifies an important epistemic limit on the use of post-hoc interpretability techniques in scientific ML applications. It separates predictive and explanatory fidelity from mechanistic adequacy, which has direct implications for how explanations are deployed and validated when the goal is scientific understanding rather than prediction alone. The argument is grounded in clear definitions and avoids circularity or hidden assumptions about what structure requires.

minor comments (2)
  1. [Abstract] The abstract states the core distinction cleanly, but a short illustrative example (e.g., applying the reliability/faithfulness criteria to a concrete post-hoc method such as LIME or SHAP on a scientific dataset) would help readers see how the three notions diverge in practice.
  2. The manuscript would benefit from an explicit statement of the scope: whether the argument applies only to post-hoc methods or also to inherently interpretable models used for scientific claims.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive assessment of the manuscript and their recommendation to accept. The referee's summary accurately captures the central claim that reliability and faithfulness of post-hoc explanations are insufficient to license structural claims about the target phenomenon.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper advances a conceptual distinction: reliability verifies that model predictions match observed outcomes, faithfulness verifies that an explanation matches model internals, and neither verifies mechanistic correspondence between model and phenomenon, which is required for structural claims. This follows directly from the explicit definitions of the three notions without equations, fitted parameters, self-citations, or any reduction that makes the conclusion equivalent to an input assumption by construction. The derivation is therefore self-contained as a clarification of conceptual limits rather than a mathematical or empirical derivation that collapses into its premises.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on a domain assumption about what counts as a structural claim in science.

axioms (1)
  • domain assumption A claim about the structure of a phenomenon requires that the model works as the phenomenon works.
    This premise is invoked to separate structural claims from mere predictive or internal-model matching.

pith-pipeline@v0.9.1-grok · 5634 in / 1148 out tokens · 62417 ms · 2026-06-30T08:17:57.603861+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

216 extracted references · 19 canonical work pages · 5 internal anchors

  1. [1]

    International journal of nursing studies , volume=

    Towards an alternative to Benner's theory of expert intuition in nursing: a discussion paper , author=. International journal of nursing studies , volume=. 2008 , publisher=

    2008

  2. [2]

    Minds and Machines , volume=

    Expertise and intuition: A tale of three theories , author=. Minds and Machines , volume=. 2009 , publisher=

    2009

  3. [3]

    2009 , publisher=

    Expertise in nursing practice: Caring, clinical judgment, and ethics , author=. 2009 , publisher=

    2009

  4. [4]

    AJN The American Journal of Nursing , volume=

    How expert nurses use intuition , author=. AJN The American Journal of Nursing , volume=. 1987 , publisher=

    1987

  5. [5]

    1978 , publisher=

    Thought and choice in chess , author=. 1978 , publisher=

    1978

  6. [6]

    1986 , publisher=

    Mind over machine , author=. 1986 , publisher=

    1986

  7. [7]

    American Scientist , volume =

    Skill in Chess , author =. American Scientist , volume =

  8. [8]

    Science , volume=

    Expert and novice performance in solving physics problems , author=. Science , volume=. 1980 , publisher=

    1980

  9. [9]

    Memory , volume=

    Chunks in expert memory: Evidence for the magical number four… or is it two? , author=. Memory , volume=. 2004 , publisher=

    2004

  10. [10]

    Psychological science , volume=

    The roles of recognition processes and look-ahead search in time-constrained expert problem solving: Evidence from grand-master-level chess , author=. Psychological science , volume=. 1996 , publisher=

    1996

  11. [11]

    1995 , publisher=

    Remembering: A study in experimental and social psychology , author=. 1995 , publisher=

    1995

  12. [12]

    1974 , publisher=

    A framework for representing knowledge , author=. 1974 , publisher=

    1974

  13. [13]

    1972 , publisher=

    Human problem solving , author=. 1972 , publisher=

    1972

  14. [14]

    Organizational behavior and human decision processes , volume=

    Characteristics of skilled option generation in chess , author=. Organizational behavior and human decision processes , volume=. 1995 , publisher=

    1995

  15. [15]

    ICGA journal , volume=

    Adaptive expert decision making: Skilled chess players search more and deeper , author=. ICGA journal , volume=. 2004 , publisher=

    2004

  16. [16]

    , author=

    Can Deep BlueTM Make Us Happy? Reflections on Human and Artificial Expertise. , author=. Deep Blue Versus Kasparov: The Significance for Artificial Intelligence , pages=

  17. [17]

    , author=

    Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. , author=. 1954 , publisher=

    1954

  18. [18]

    Heuristics of the professional eye , year=

    Perception and memory in chess , author=. Heuristics of the professional eye , year=

  19. [19]

    American economic review , volume=

    Maps of bounded rationality: Psychology for behavioral economics , author=. American economic review , volume=. 2003 , publisher=

    2003

  20. [20]

    Harvard business review , volume=

    The experts in your midst , author=. Harvard business review , volume=

  21. [21]

    2010 , publisher=

    Complications: A surgeon's notes on an imperfect science , author=. 2010 , publisher=

    2010

  22. [22]

    Phenomenology and the Cognitive Sciences , volume=

    Concepts without intuition lose the game: commentary on Montero and Evans (2011) , author=. Phenomenology and the Cognitive Sciences , volume=. 2012 , publisher=

    2011

  23. [23]

    Management decision , volume=

    Intuition in decisions , author=. Management decision , volume=. 2003 , publisher=

    2003

  24. [24]

    Heuristics and biases: The psychology of intuitive judgment , volume=

    Representativeness revisited: Attribute substitution in intuitive judgment , author=. Heuristics and biases: The psychology of intuitive judgment , volume=

  25. [25]

    , author=

    Performance without preparation: Structure and acquisition of expert sight-reading and accompanying performance. , author=. Psychomusicology: A journal of research in music cognition , volume=. 1996 , publisher=

    1996

  26. [26]

    IEEE Transactions on systems, man, and cybernetics , volume=

    Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment , author=. IEEE Transactions on systems, man, and cybernetics , volume=. 1987 , publisher=

    1987

  27. [27]

    Hammond’s contributions to the study of judgment and decision making , author=

    Kenneth R. Hammond’s contributions to the study of judgment and decision making , author=. Judgment and Decision Making , volume=. 2018 , publisher=

    2018

  28. [28]

    Medical education , volume=

    Non-analytical models of clinical reasoning: the role of experience , author=. Medical education , volume=. 2007 , publisher=

    2007

  29. [29]

    Academy of management review , volume=

    Exploring intuition and its role in managerial decision making , author=. Academy of management review , volume=. 2007 , publisher=

    2007

  30. [30]

    , author=

    Conditions for intuitive expertise: a failure to disagree. , author=. American psychologist , volume=. 2009 , publisher=

    2009

  31. [31]

    Behavioral and brain sciences , volume=

    Building machines that learn and think like people , author=. Behavioral and brain sciences , volume=. 2017 , publisher=

    2017

  32. [32]

    Technology in Society , volume=

    From Socrates to expert systems: The limits of calculative rationality , author=. Technology in Society , volume=. 1984 , publisher=

    1984

  33. [33]

    Expertise in nursing practice: Caring, clinical judgment, and ethics , pages=

    The relationship of theory and practice in the acquisition of skill , author=. Expertise in nursing practice: Caring, clinical judgment, and ethics , pages=. 1996 , publisher=

    1996

  34. [34]

    Organization studies , volume=

    Peripheral vision: Expertise in real world contexts , author=. Organization studies , volume=. 2005 , publisher=

    2005

  35. [35]

    2016 , publisher=

    Artificial intelligence: a modern approach , author=. 2016 , publisher=

    2016

  36. [36]

    Communications of the ACM , volume=

    Computer science as empirical inquiry: symbols and search , author=. Communications of the ACM , volume=

  37. [37]

    nature , volume=

    Deep learning , author=. nature , volume=. 2015 , publisher=

    2015

  38. [38]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  39. [39]

    1972 , publisher=

    What Computers Can't Do: A Critique of Artificial Reason , author=. 1972 , publisher=

    1972

  40. [40]

    , author=

    Stagewise cognitive development: an application of catastrophe theory. , author=. Psychological review , volume=. 1992 , publisher=

    1992

  41. [41]

    de Groot , title=

    A.D. de Groot , title=. ICGA Journal , volume=. 1986 , publisher=. doi:10.3233/ICG-1986-9202 , month=

    work page doi:10.3233/icg-1986-9202 1986

  42. [42]

    1985 , publisher=

    The psychology of chess skill , author=. 1985 , publisher=

    1985

  43. [43]

    Cognitive science , volume=

    Five seconds or sixty? Presentation time in expert memory , author=. Cognitive science , volume=. 2000 , publisher=

    2000

  44. [44]

    Organizational Behavior and Human Decision Processes , volume=

    Algorithm appreciation: People prefer algorithmic to human judgment , author=. Organizational Behavior and Human Decision Processes , volume=. 2019 , publisher=

    2019

  45. [45]

    theory of machine

    The psychology of Big Data: Developing a “theory of machine” to examine perceptions of algorithms. , author=. 2022 , publisher=

    2022

  46. [46]

    Algorithm Aversion

    A Simple Explanation Reconciles “Algorithm Aversion” and “Algorithm Appreciation”: Hypotheticals vs. Real Judgments , author=. Real Judgments (January 8, 2024) , year=

    2024

  47. [47]

    , author=

    Algorithm aversion: people erroneously avoid algorithms after seeing them err. , author=. Journal of experimental psychology: General , volume=. 2015 , publisher=

    2015

  48. [48]

    Readings in artificial intelligence , pages=

    Some philosophical problems from the standpoint of artificial intelligence , author=. Readings in artificial intelligence , pages=. 1981 , publisher=

    1981

  49. [49]

    , author=

    Clinical versus mechanical prediction: a meta-analysis. , author=. Psychological assessment , volume=. 2000 , publisher=

    2000

  50. [50]

    Science , volume=

    Clinical versus actuarial judgment , author=. Science , volume=. 1989 , publisher=

    1989

  51. [51]

    Rationality and Social Responsibility , pages=

    The robust beauty of improper linear models in decision making , author=. Rationality and Social Responsibility , pages=. 2008 , publisher=

    2008

  52. [52]

    Human factors , volume=

    The perceived utility of human and automated aids in a visual detection task , author=. Human factors , volume=. 2002 , publisher=

    2002

  53. [53]

    Journal of Behavioral Decision Making , volume=

    Making sense of recommendations , author=. Journal of Behavioral Decision Making , volume=. 2019 , publisher=

    2019

  54. [54]

    , author=

    Is overconfidence a motivated bias? Experimental evidence. , author=. Journal of Experimental Psychology: General , volume=. 2018 , publisher=

    2018

  55. [55]

    The Wiley Blackwell handbook of judgment and decision making , volume=

    Overprecision in judgment , author=. The Wiley Blackwell handbook of judgment and decision making , volume=. 2015 , publisher=

    2015

  56. [56]

    2014 , publisher=

    Aspects of the Theory of Syntax , author=. 2014 , publisher=

    2014

  57. [57]

    2011 , publisher=

    Thinking, fast and slow , author=. 2011 , publisher=

    2011

  58. [58]

    , author=

    Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. , author=. Psychological Review , volume=. 1983 , publisher=

    1983

  59. [59]

    , author=

    Judgment under Uncertainty: Heuristics and Biases: Biases in judgments reveal some heuristics of thinking under uncertainty. , author=. science , volume=. 1974 , publisher=

    1974

  60. [60]

    Philosophy, Expertise, and the Myth of Neutrality , pages=

    Trustworthy experts and untrustworthy experts: Insights from the cognitive psychology of expertise , author=. Philosophy, Expertise, and the Myth of Neutrality , pages=. 2024 , publisher=

    2024

  61. [61]

    2017 , publisher=

    Expert political judgment: How good is it? How can we know?-New edition , author=. 2017 , publisher=

    2017

  62. [62]

    2001 , publisher=

    The social psychology of expertise: Case studies in research, professional domains, and expert roles , author=. 2001 , publisher=

    2001

  63. [63]

    AlphaZero’s Groundbreaking Chess Strategies and the Promise of AI

    Game changer , author=. AlphaZero’s Groundbreaking Chess Strategies and the Promise of AI. Alkmaar. The Netherlands. New in Chess , year=

  64. [64]

    also John W

    General Systems Theory, New York, Brazillier, 1968 , author=. also John W. Sutherland, A General Systems Philosophy for the Social and Behavioral Sciences, New York, Brazillier , pages=

    1968

  65. [65]

    , author=

    Computer power and human reason: From judgment to calculation. , author=. 1976 , publisher=

    1976

  66. [66]

    Journal of Consciousness Studies , volume=

    Beyond the computer metaphor: Behaviour as interaction , author=. Journal of Consciousness Studies , volume=. 1999 , publisher=

    1999

  67. [67]

    Ethics and Information Technology , volume=

    Artificial intelligence and responsibility gaps: what is the problem? , author=. Ethics and Information Technology , volume=. 2022 , publisher=

    2022

  68. [68]

    Philosophy & Technology , volume=

    Zombies in the loop? Humans trust untrustworthy AI-advisors for ethical decisions , author=. Philosophy & Technology , volume=. 2022 , publisher=

    2022

  69. [69]

    Journal of Medical Internet Research , volume=

    Investigating the impact of user trust on the adoption and use of ChatGPT: survey analysis , author=. Journal of Medical Internet Research , volume=. 2023 , publisher=

    2023

  70. [70]

    Phenomenology and the Cognitive Sciences , volume=

    Intuitions without concepts lose the game: mindedness in the art of chess , author=. Phenomenology and the Cognitive Sciences , volume=. 2011 , publisher=

    2011

  71. [71]

    Proceedings of the AISB Convention 2023 , year=

    Trust in Cognitive Models: Understandability and Computational Reliabilism , author=. Proceedings of the AISB Convention 2023 , year=

    2023

  72. [72]

    Psychological Research , volume=

    Pattern recognition makes search possible: Comments on Holding (1992) , author=. Psychological Research , volume=. 1998 , publisher=

    1992

  73. [73]

    Cognitive psychology , volume=

    Templates in chess memory: A mechanism for recalling several boards , author=. Cognitive psychology , volume=. 1996 , publisher=

    1996

  74. [74]

    The science of synthesis: exploring the social implications of general systems theory , volume=

    General systems theory , author=. The science of synthesis: exploring the social implications of general systems theory , volume=. 2010 , publisher=

    2010

  75. [75]

    Epistemology: new essays , pages=

    Immediate justification and process reliabilism , author=. Epistemology: new essays , pages=. 2008 , publisher=

    2008

  76. [76]

    De Groot, A. D. , title =. Personality Psychology in Europe , editor =

  77. [77]

    The road to excellence , pages=

    Perceptual and memory processes in the acquisition of expert performance: The EPAM model , author=. The road to excellence , pages=. 2014 , publisher=

    2014

  78. [78]

    Cognitive psychology , volume=

    Inflexibility of experts—Reality or myth? Quantifying the Einstellung effect in chess masters , author=. Cognitive psychology , volume=. 2008 , publisher=

    2008

  79. [79]

    Cognition , volume=

    Why good thoughts block better ones: The mechanism of the pernicious Einstellung (set) effect , author=. Cognition , volume=. 2008 , publisher=

    2008

  80. [80]

    Minds and Machines , volume=

    Grounds for trust: Essential epistemic opacity and computational reliabilism , author=. Minds and Machines , volume=. 2018 , publisher=

    2018

Showing first 80 references.