REVIEW 3 major objections 4 minor 9 references
Reinforcement Learning is not a Causal problem
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper argues that standard reinforcement learning is not a causal problem, on the basis of non-isomorphic algebras.
desk verdict A short, clearly written opinion piece whose algebraic analogy doesn't carry the load; the conclusion may be true, but the paper doesn't prove it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a pair of binary operators: $\bigoplus$ for the associative algebra, which captures correlations without any distinguished order, and $\bigotimes(Z, \succcurlyeq)$ for the causal algebra, which carries an implicit order and a context for interventions. These operators encode the relation $aRb$ that defines a problem; in RL the relation uses only $\bigoplus$, placing $\beta$ in the associative algebra. The argument is carried by the analogy with $G_1 = (\mathbb{Z}, +)$ and $G_2 = (\mathbb{Z}, \cdot)$: non-isomorphic structures over the same set cannot exchange their solution methods, so associative and causal tools cannot be mixed.
What would settle it
Construct a small Markov decision process with a confound and compare the policy obtained from the standard Bellman equation with the policy obtained from the equation using $P_g(s' \mid s, \mathrm{do}(a))$; if the policies diverge, the paper's separation is supported, but if they coincide, the two algebras can be mixed and the argument's core step is undercut.
Extended reading notes
Core claim
The paper's central claim is that the relation $R$ expressing the RL objective is an associative operation $\bigoplus$, and that any legitimate transfer of tools from causal reasoning would require an isomorphism between the associative algebra and the causal algebra. Since the associative algebra has no order while the causal algebra is defined by an implicit order $\succcurlyeq$ and a context $Z$, the paper asserts that no such isomorphism exists. Consequently, the target quantity $\beta = f[aRb]$ with $R = \bigoplus$ cannot be obtained with causal machinery, and standard reinforcement learning is not a causal problem. The authors locate RL on the first, associational level of causal reasoning and argue that even the intuition of an agent taking actions does not change the mathematical formulation. They close by speculating that a genuinely causal version of RL would require modifying the Bellman equations to use an interventional distribution $P_g(s' \mid s, \mathrm{do}(a))$.
Load-bearing premise
The entire claim rests on an undefined pair of algebras: if the associative and causal operations are not well-defined, non-isomorphic algebraic structures, then the conclusion has no formal basis.
Editorial extensions
If this is right
- If the claim is correct, attempts to attach causal machinery to standard RL by relabeling actions as interventions are solving a different equation, not the RL equation.
- Current RL algorithms that learn from correlations can support associational claims about state and reward, but policy claims about the effect of an action are not established unless the model is changed to include interventions.
- A causal variant of RL would be a different problem from standard RL, requiring an interventional distribution such as $P_g(s' \mid s, \mathrm{do}(a))$ inside the Bellman equations.
- The debate over whether RL is causal becomes a formulation-level question: the same real-world situation can be modeled associatively or causally, but the mathematical problem and its solution change with the algebra.
- Success of standard RL algorithms does not by itself license causal interpretations of the policies they learn.
Reading between the lines
- Extension: if the non-isomorphism were made rigorous, the argument would imply a separation principle: no value function computed from observational correlations can identify interventional quantities without adding structure, a claim that resembles identifiability results but is here framed algebraically.
- Extension: the analogy could be tested concretely by defining the Bellman update operator as an element of an associative algebra and the intervention-modified Bellman operator as an element of a causal algebra, then checking whether any homomorphism between them exists; the paper does not provide that construction.
- Extension: the authors' proposed interventional Bellman equations suggest a testable comparison, in a confounded Markov decision process, of the policies learned by standard Q-learning and by an interventional variant; divergence would support the algebraic separation, agreement would weaken it.
- Extension: if the paper is right, causal reinforcement learning is not a refinement of RL but a new formulation with additional structure, one that changes the optimality criterion rather than merely adding a causal module to an existing agent.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This short position paper argues that reinforcement learning (RL), in its current mathematical formulation, is not a causal problem. The authors draw an analogy between non-isomorphic algebraic structures—specifically the groups (Z,+) and (Z,·)—and two 'algebras' they associate with associative information (correlation) and causal information (intervention and order). They introduce an operator ⨁ for the associative algebra and ⨂(Z,≽) for the causal algebra, assert that the two algebras are not isomorphic because the associative algebra lacks order, and conclude that since RL uses associative operations (e.g., the Q-function), causal tools cannot be used to solve RL problems. The paper ends with speculative modified Bellman equations involving do-calculus.
Significance. If the central claim were rigorously established, the paper would contribute a useful conceptual clarification to the ongoing debate about whether RL is a causal problem, potentially preventing conceptual confusion between correlation-level and intervention-level reasoning. The authors are right to point out that the intuitive framing of RL in terms of agents acting in an environment does not by itself make the mathematical formulation causal, and they correctly invoke Pearl's hierarchy of causal reasoning. However, the argument's force depends entirely on a formal analogy that is never made precise: the algebraic structures are not defined, the non-isomorphism is asserted rather than proved, and the step from non-isomorphism to the impossibility of using causal tools is a logical gap. The paper therefore does not currently deliver a sound argument, though the underlying question is significant.
major comments (3)
- [Section 3] The operators ⨁ and ⨂ are not defined with the precision needed to support the claimed non-isomorphism. The text says only that correlated variables are represented as ⨁(A,B) and that a cause-effect pair is represented as ⨂(A,B), with ⨂ more specifically written as ⨂(Z,≽). No algebraic axioms, operations, identities, or homomorphisms are specified, so 'non-isomorphism' is not a well-defined mathematical claim. Without rigorous definitions, the central contrast between an 'associative algebra' and a 'causal algebra' reduces to an intuitive metaphor, and the conclusion that they are 'clearly not isomorphic' is unsupported.
- [Section 4, Eq. (4)-(5)] Even granting the non-isomorphism of ⨁ and ⨂, the inference that 'we can not use causal tools to solve for β' is a non-sequitur. Non-isomorphism of two algebraic structures means there is no bijective homomorphism between them; it does not imply that a numeric quantity β obtainable via one structure cannot be computed by an algorithm that uses the other structure. To make the inference valid, the paper would need the additional premise that every RL solution β is expressible solely within the ⨁-algebra and that no causal method can output the same value. That premise is asserted in the sentence 'current reinforcement learning problems... rely only on associative tools' but is not derived from the MDP formulation. The Bellman equations involve sums, maxima, conditional probabilities, and discounting—not a single binary operation—so the (Z,+) versus (Z,·) analogy, which concerns solving an equation written with one operation using the other operation, does not transfer directly.
- [Section 5, Eqs. (6)-(7)] The proposed modified Bellman equations are presented as speculation without any derivation or definition of the quantity Pg. If the paper's point is that classical RL is not causal, then presenting 'causal RL' equations as a possible modification is a significant caveat that interacts with the main claim; the relationship between these equations and the preceding argument that 'we can not use causal tools to solve for β' is left unclear. As written, Eqs. (6)-(7) suggest that a do-operator can be inserted into Bellman equations, which would seem to offer a causal tool for RL, undermining the paper's categorical conclusion unless the scope of 'current formulation' is carefully delineated.
minor comments (4)
- [Introduction and Conclusion] There are numerous typographical errors and inconsistent formatting (e.g., 'comunidad', 'isomo rtphic', 'Spirte s’ Causation', 'independently if the motivation') that should be corrected in any revision.
- [Section 3] The phrase 'the variable, or event A which causes some other B' should be 'a variable or event A which causes some other event B'; the missing comma and article make the sentence harder to read.
- [Section 5] The notation Pg is introduced without definition; the paper should specify the causal graphical model g and the meaning of do(a) in this context, or explicitly refer to Pearl's do-calculus.
- [References] The reference to Szepesvari (2018) is listed as a workshop item but lacks a stable identifier or DOI; several other references are incomplete (e.g., no page numbers for the Gershman chapter) and would need to be completed for publication.
Circularity Check
RL-not-causal conclusion is a definitional restatement, not a derived result
-
self definitional
[Section 4, Eq. (3) and the main-argument paragraph]
"In particular, in RL we must find β = f [aRb] (3) where R = ⨁ , and f a function which depends on the state and reward of the system through only associative operations (e.g., the Q function). And here in this point we have our main argument: since R is the associative algebra, and such algebra can not be isomorphic to the causal algebra because of the lack of order, then we can not use causal tools to solve for β , and therefore RL is not a causal problem."
The paper stipulates that RL's relation R is ⨁, the 'associative algebra,' and defines associative information as having no distinguishable order, whereas the causal algebra is defined as ⨂(Z,≽) with an implicit order and context. The asserted non-isomorphism is then explained 'because of the lack of order'—that is, it follows immediately from the definitions, not from a proof. The conclusion 'RL is not a causal problem' is a restatement of the initial classification of RL as associative. The additional bridge that causal tools 'can not solve for β' is not derived from any Bellman or Q-learning equation; it is asserted by identifying every RL solution with an expression in the single binary operation ⨁.
full rationale
The paper's central inference is circular in the self-definitional sense. It defines the associative algebra as lacking order and the causal algebra as carrying an implicit order, then assigns RL's R to be ⨁, and finally concludes that RL cannot be causal because ⨁ is not isomorphic to ⨂. The non-isomorphism is a built-in feature of the definitions rather than an independently established theorem. The external (Z,+) vs (Z,·) analogy does not rescue the argument because RL is not shown to be expressible as one binary operation; the proposed causal Bellman equations (6)-(7) are explicitly speculative. There is no load-bearing self-citation chain here; the circularity is internal and definitional. Because the central claim is forced by the paper's own stipulations rather than derived from a falsifiable analysis of the MDP/Bellman formalism, the circularity score is 8.
Assumptions & free parameters
assumptions (5)
- domain assumption Manipulationist notion of causation (Woodward 2003) includes an implicit order ≽ and a context Z.
- domain assumption Associative information has no distinguishable order; any joint distribution can be factored either way.
- ad hoc to paper The operators ⨁ and ⨂ are not isomorphic because of the lack of order.
- domain assumption RL's standard formulation relies only on associative operations.
- ad hoc to paper The group example (Z,+) vs (Z,·) provides a valid analogy to associative vs causal information.
invented entities (2)
-
Associative algebra operator ⨁
-
Causal algebra operator ⨂
Cite this review
Pith. "Pith review of Reinforcement Learning is not a Causal problem." pith.science (2026). https://pith.science/paper/6YMZOJT6
@misc{pith2026190807617,
author = {Pith},
title = {Pith review of: Reinforcement Learning is not a Causal problem},
year = {2026},
howpublished = {\url{https://pith.science/paper/6YMZOJT6}},
note = {Machine review of arXiv:1908.07617}
}
read the original abstract
We use an analogy between non-isomorphic mathematical structures defined over the same set and the algebras induced by associative and causal levels of information in order to argue that Reinforcement Learning, in its current formulation, is not a causal problem, independently if the motivation behind it has to do with an agent taking actions.
Reference graph
Works this paper leans on
-
[1]
Clark, A. (2015). Surfing uncertainty: Prediction, action, and the embodied m ind. Oxford University Press
work page 2015
-
[2]
Gershman, S. J. (2015). Reinforcement learning and causal m odels. In The Oxford Handbook of Causal Reasoning
work page 2015
-
[3]
Hungerford, T. W . (1974). Algebra. Springer-V erlag New Y ork
work page 1974
-
[4]
Pearl, J. (2009). Causality: Models, Reasoning and Inference . Cambridge University Press, New Y ork, NY , USA, 2nd edition
work page 2009
-
[5]
Spirtes, P ., Glymour, C. N., and Scheines, R. (2000). Causation, prediction and search. MIT Press
work page 2000
-
[6]
Sutton, R. S. and Barto, A. G. (1998). Reinforcement Learning: An introduction . MIT Press
1998
-
[7]
Sutton, R. S., Barto, A. G., and Williams, R. J. (1992). Reinf orcement learning is direct adaptive optimal control. IEEE Control Systems Magazine , 12(2):19–22
work page 1992
-
[8]
Szepesvari, C. (2018). Causality from the perspective of re inforcement learning. Machine Learning for Causal Inference, Counterfactual Prediction, and Auto nomous Action (CausalML) Workshop, ICML
work page 2018
Show all 9 references
-
[9]
Woodward, J. (2003). Making things happen: A theory of causal explanation . Oxford Studies in Philosophy of Science. Oxford University Press. 3
2003
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.