Recognition: no theorem link
"What Is It That You Don't Understand?" Language Games and Black Box Algorithms
Pith reviewed 2026-05-14 23:50 UTC · model grok-4.3
The pith
Black box algorithms cannot be fully explained because their rules resist complete interpretation.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper establishes that machines face the problem of the inscrutability of reference, in the same way that the linguist cannot precisely determine what a term refers to in a situation of radical translation. There is no rule for the application of language, except for language games. The hope of achieving complete explicability and transparency of algorithms is undoubtedly in vain: we can only rely on partial and broad interpretations that will never fully explain the underlying rules.
What carries the argument
The direct transfer of the inscrutability of reference from radical translation scenarios and the limits of rule-following from language games to the internal operations of machine learning models. This transfer demonstrates why no interpretation can capture the models' rules without remainder.
Load-bearing premise
The challenges of determining exact references in unknown languages and following rules only through use apply directly to the internal decision processes of machine learning models.
What would settle it
A documented case of a black box model whose complete set of decision rules is extracted and verified with no remaining ambiguity or need for partial interpretation.
read the original abstract
The aim of this article is to understand the problem of "black box" algorithms, an issue inherent to the nascent field of Explainable Artificial Intelligence (XAI). While it is relatively easy to understand something someone explained to us, it becomes more complicated when no one can fully grasp the issue. Our purpose is however to highlight: (1) that we should speak of interpretability rather than explainability when we seek to understand models, mainly because we never have complete and unambiguous access to information; (2) that the machines face the problem of the inscrutability of reference, in the same way that the linguist imagined by Willard Van Orman Quine cannot precisely determine what the term "gavagai" refers to in a situation of radical translation; (3) that there is no rule for the application of language, except for "language games", as Ludwig Wittgenstein's linguistics teaches us. The hope of achieving complete explicability and transparency of algorithms is undoubtedly in vain: we can only rely on partial and broad interpretations that will never fully explain the underlying rules.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that complete explicability of black-box algorithms is impossible and that the field should focus on interpretability instead. Drawing on Quine's inscrutability of reference (illustrated via the 'gavagai' example in radical translation) and Wittgenstein's language games, it claims that machine learning models encounter analogous indeterminacies: reference cannot be fixed unambiguously and rule application lacks independent criteria beyond interpretive practices. The conclusion is that only partial and broad interpretations are feasible.
Significance. If the analogies are made rigorous, the paper would provide a philosophical grounding for why XAI techniques cannot deliver full transparency, shifting emphasis toward bounded interpretability methods and their epistemic limits. This could inform regulatory and design discussions in AI ethics by highlighting inherent constraints rather than technical shortcomings.
major comments (2)
- [Abstract] Abstract and the section applying Quine: the assertion that machines 'face the problem of the inscrutability of reference, in the same way' as the linguist with 'gavagai' is load-bearing for the central claim but remains an unargued transfer. The manuscript does not address how a model's fixed parameter vector, explicit architecture, loss function, and training data provide disambiguation mechanisms absent in radical translation, leaving the analogy interpretive rather than derived.
- [Wittgenstein discussion / Conclusion] The Wittgenstein language-games section and conclusion: the claim that 'there is no rule for the application of language, except for language games' is applied directly to algorithmic rule-following without showing why the deterministic forward pass and optimization process are governed solely by such games rather than by the public, inspectable criteria of the training regime and evaluation metrics.
minor comments (1)
- [Introduction] Clarify the distinction between interpretability and explainability earlier and more precisely, as the current phrasing risks conflating the two without operational definitions.
Simulated Author's Rebuttal
We thank the referee for their constructive comments, which help clarify where our philosophical analogies require additional rigor. We respond to each major comment below.
read point-by-point responses
-
Referee: [Abstract] Abstract and the section applying Quine: the assertion that machines 'face the problem of the inscrutability of reference, in the same way' as the linguist with 'gavagai' is load-bearing for the central claim but remains an unargued transfer. The manuscript does not address how a model's fixed parameter vector, explicit architecture, loss function, and training data provide disambiguation mechanisms absent in radical translation, leaving the analogy interpretive rather than derived.
Authors: We accept that the analogy to Quine's inscrutability of reference needs explicit elaboration to move beyond assertion. Although a model's parameters, architecture, loss function, and training data impose strong behavioral constraints, these elements still leave open multiple possible interpretations of what the internal states refer to, in the same manner that observable behavior underdetermines reference in radical translation. We will revise the abstract and the Quine section to include a direct discussion of these disambiguation mechanisms and why they do not eliminate the relevant form of indeterminacy. revision: yes
-
Referee: [Wittgenstein discussion / Conclusion] The Wittgenstein language-games section and conclusion: the claim that 'there is no rule for the application of language, except for language games' is applied directly to algorithmic rule-following without showing why the deterministic forward pass and optimization process are governed solely by such games rather than by the public, inspectable criteria of the training regime and evaluation metrics.
Authors: The referee is right to note that the connection requires clarification. Our position is that the training regime and evaluation metrics are themselves constituted within human language games and interpretive practices; they supply public criteria only insofar as those practices are already in play. The deterministic forward pass and optimization therefore operate inside, rather than outside, such games. We will expand the Wittgenstein section and the conclusion to make this embedding explicit and to address the inspectable criteria directly. revision: yes
Circularity Check
No circularity: philosophical analogy asserted without self-referential reduction or fitted inputs
full rationale
The paper applies Quine's radical translation and Wittgenstein's language games to black-box algorithms through direct analogy, as stated in the abstract. No mathematical derivations, parameter fits, self-citations, or internal definitions exist that would render the central claim equivalent to its premises by construction. The argument is interpretive and relies on external philosophical sources rather than any closed loop within the paper's own content or data.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption There is no rule for the application of language except for language games
- domain assumption Machines face the inscrutability of reference in the same way as in radical translation
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.