{"id":"e88b6923-a0ae-4c17-a3d7-42d23acf9b5e","arxiv_id":"1908.07617","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RL's standard Bellman-equation formulation operates only on associative information, so treating RL as a causal problem rests on a category error, the paper argues.","lead":"Reinforcement learning as normally formulated learns from correlations in data, not from causal interventions, and this paper argues these are mathematically different problems. It uses an analogy with non-isomorphic algebraic structures to claim that treating RL as a causal problem is a category mistake.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central inference in Section 4 is a non-seguitur: even granting non-isomorphic algebras, it does not follow that causal tools cannot solve an RL problem.","rationale":"I read the paper as attempting to establish a categorical distinction between RL and causal reasoning, using an analogy with non-isomorphic algebraic structures. The strongest claim is indeed Section 4's inference from non-isomorphism to the impossibility of using causal tools for RL. My concern is not merely that the algebras are undefined, although that is true; it is that even with fully precise definitions, the logical bridge from 'non-isomorphic structures' to 'one structure's tools cannot solve the other's problems' is missing. The reader's weakest_assumption focused on the undefined algebras and unsupported non-isomorphism. I partially agree, but I see a second, more load-bearing gap: the conclusion would not follow even if those definitions were supplied. The paper's own modified Bellman equations in the conclusion (Eqs. 6-7) suggest a causal variant exists, which further undermines the categorical reading. I therefore do not change the reader's REJECT verdict, but I would sharpen the reason: the paper has not shown a formal obstruction, only asserted one. The proposed proof-assistant check would settle whether the inference can be made rigorous, and I expect it would expose the hidden premise that Q-values are exclusively associative expressions.","tokens_in":2710,"tokens_out":5320,"duration_ms":141615,"concrete_test":"Formalize the Section 4 argument in a proof assistant such as Lean. Define the associative algebra as the algebraic structure generated by the Bellman optimality expressions, define the causal algebra with signature (⊗, ≽, Z), and assume only that the two algebras are non-isomorphic. Attempt to derive the theorem 'for every MDP, no term built from causal operators computes the optimal Q-function.' If the derivation requires an extra axiom such as 'Q-values are representable only in the associative algebra', then the central inference has a hidden premise. Alternatively, construct a simple two-state MDP and show that do-calculus can compute the same optimal Q-value as Q-learning; that would refute the categorical version of the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main argument (Section 4) claims: since R = ⊕ is not isomorphic to the causal algebra ⊗(Z, ≽), 'we can not use causal tools to solve for β, and therefore RL is not a causal problem.' Even granting fully rigorous definitions of both algebras and a proof of their non-isomorphism, the conclusion does not follow. Non-isomorphism says only that there is no bijective homomorphism preserving the algebraic operations; it says nothing about whether a function computable by causal tools can output the same β as a function built from associative operations. To make the inference, an additional premise is required: every RL solution β is expressible entirely within the ⊕-algebra, and no causal tool can compute that value. This premise is asserted ('current reinforcement learning problems... rely only on associative tools') but not derived from the MDP formulation. The Bellman equations involve sums, maxima, conditional probabilities, and discounting; they are not a single binary operation. The analogy with (Z,+) versus (Z,·) shows that an equation written with '+' cannot be solved with '·', but RL is not presented as a single equation over one binary operation. Thus the argument either assumes what it sets out to prove or leaves a substantial gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This short position paper argues that reinforcement learning (RL), in its current mathematical formulation, is not a causal problem. The authors draw an analogy between non-isomorphic algebraic structures—specifically the groups (Z,+) and (Z,·)—and two 'algebras' they associate with associative information (correlation) and causal information (intervention and order). They introduce an operator ⨁ for the associative algebra and ⨂(Z,≽) for the causal algebra, assert that the two algebras are not isomorphic because the associative algebra lacks order, and conclude that since RL uses associative operations (e.g., the Q-function), causal tools cannot be used to solve RL problems. The paper ends with speculative modified Bellman equations involving do-calculus.","tokens_in":2985,"tokens_out":2134,"duration_ms":466473,"significance":"If the central claim were rigorously established, the paper would contribute a useful conceptual clarification to the ongoing debate about whether RL is a causal problem, potentially preventing conceptual confusion between correlation-level and intervention-level reasoning. The authors are right to point out that the intuitive framing of RL in terms of agents acting in an environment does not by itself make the mathematical formulation causal, and they correctly invoke Pearl's hierarchy of causal reasoning. However, the argument's force depends entirely on a formal analogy that is never made precise: the algebraic structures are not defined, the non-isomorphism is asserted rather than proved, and the step from non-isomorphism to the impossibility of using causal tools is a logical gap. The paper therefore does not currently deliver a sound argument, though the underlying question is significant.","major_comments":[{"comment":"The operators ⨁ and ⨂ are not defined with the precision needed to support the claimed non-isomorphism. The text says only that correlated variables are represented as ⨁(A,B) and that a cause-effect pair is represented as ⨂(A,B), with ⨂ more specifically written as ⨂(Z,≽). No algebraic axioms, operations, identities, or homomorphisms are specified, so 'non-isomorphism' is not a well-defined mathematical claim. Without rigorous definitions, the central contrast between an 'associative algebra' and a 'causal algebra' reduces to an intuitive metaphor, and the conclusion that they are 'clearly not isomorphic' is unsupported.","section":"Section 3"},{"comment":"Even granting the non-isomorphism of ⨁ and ⨂, the inference that 'we can not use causal tools to solve for β' is a non-sequitur. Non-isomorphism of two algebraic structures means there is no bijective homomorphism between them; it does not imply that a numeric quantity β obtainable via one structure cannot be computed by an algorithm that uses the other structure. To make the inference valid, the paper would need the additional premise that every RL solution β is expressible solely within the ⨁-algebra and that no causal method can output the same value. That premise is asserted in the sentence 'current reinforcement learning problems... rely only on associative tools' but is not derived from the MDP formulation. The Bellman equations involve sums, maxima, conditional probabilities, and discounting—not a single binary operation—so the (Z,+) versus (Z,·) analogy, which concerns solving an equation written with one operation using the other operation, does not transfer directly.","section":"Section 4, Eq. (4)-(5)"},{"comment":"The proposed modified Bellman equations are presented as speculation without any derivation or definition of the quantity Pg. If the paper's point is that classical RL is not causal, then presenting 'causal RL' equations as a possible modification is a significant caveat that interacts with the main claim; the relationship between these equations and the preceding argument that 'we can not use causal tools to solve for β' is left unclear. As written, Eqs. (6)-(7) suggest that a do-operator can be inserted into Bellman equations, which would seem to offer a causal tool for RL, undermining the paper's categorical conclusion unless the scope of 'current formulation' is carefully delineated.","section":"Section 5, Eqs. (6)-(7)"}],"minor_comments":[{"comment":"There are numerous typographical errors and inconsistent formatting (e.g., 'comunidad', 'isomo rtphic', 'Spirte s’ Causation', 'independently if the motivation') that should be corrected in any revision.","section":"Introduction and Conclusion"},{"comment":"The phrase 'the variable, or event A which causes some other B' should be 'a variable or event A which causes some other event B'; the missing comma and article make the sentence harder to read.","section":"Section 3"},{"comment":"The notation Pg is introduced without definition; the paper should specify the causal graphical model g and the meaning of do(a) in this context, or explicitly refer to Pearl's do-calculus.","section":"Section 5"},{"comment":"The reference to Szepesvari (2018) is listed as a workshop item but lacks a stable identifier or DOI; several other references are incomplete (e.g., no page numbers for the Gershman chapter) and would need to be completed for publication.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central argument is an analogy rather than a formal derivation, and the key step—from non-isomorphic algebras to the impossibility of using causal tools—is a logical gap that cannot be fixed by local revision. The paper would require a substantially new formal framework to make its claim rigorous; as it stands, it does not meet the standard for publication in a mainstream AI journal. I would suggest the authors consider a workshop or a more carefully scoped position paper if they can supply the missing definitions and at least a partial formal argument."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked what I make of 1908.07617. It's a conceptual note arguing that RL, as currently formulated, sits on the associative level of Pearl's ladder and therefore is not a causal problem. I think the claim is plausible and worth taking seriously. But the paper's job is to justify the claim, and it doesn't.\n\nThe good parts: the authors draw a clean distinction between a real-world situation and its mathematical model, they correctly point out that the MDP formalism doesn't include interventions or counterfactuals, and their example comparing (Z,+) with (Z,·) is pedagogically useful. They also gesture at what a causal Bellman update would look like with do-operators, which is a nice starting point. The writing is accessible and the references are appropriate.\n\nThe soft spot is the central inference. Section 4 asserts that since R = ⨁ is not isomorphic to ⨂(Z,≽), causal tools cannot solve for β. The stress-test note has this right: even granting the algebras are well-defined and non-isomorphic, that conclusion doesn't follow. Non-isomorphism tells you about structure-preserving maps between algebraic systems; it says nothing about whether one way of computing a quantity can be simulated by another. The analogy would only work if every RL solution β were literally an equation in one binary associative operation. But Bellman equations are built from sums, max, probabilities, and discounting—they aren't a single operation. So the argument either assumes what it tries to prove or leaves a gap.\n\nThere's also the problem that ⨁ and ⨂ are never formally defined. The claim that the associative algebra has no order while the causal algebra has one is stated in prose, not derived. The non-isomorphism is asserted, not proved. This makes the paper an argument from analogy, not a derivation.\n\nSo where does that leave the paper? As a position statement, it's fine—similar points appear in Pearl's own writing and in the work the authors cite. As an original formal result, it doesn't hold up. I wouldn't cite it as evidence for anything, but I might bring it up as an example of how not to formalize the associative/causal distinction.\n\nMy recommendation: don't send this to a serious referee as a research paper. If a venue is specifically hosting short conceptual position pieces, then a light review could push the authors to tighten the algebra definition and fix the logical gap. But in its current form, the core argument doesn't survive scrutiny.","headline":"A short, clearly written opinion piece whose algebraic analogy doesn't carry the load; the conclusion may be true, but the paper doesn't prove it.","tokens_in":3461,"tokens_out":1502,"would_cite":false,"duration_ms":124719,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that standard reinforcement learning is not a causal problem, on the basis of non-isomorphic algebras.","keywords":["reinforcement learning","causality","causal inference","associative algebra","causal algebra","Bellman equations","interventions","category error"],"falsifier":"Construct a small Markov decision process with a confound and compare the policy obtained from the standard Bellman equation with the policy obtained from the equation using $P_g(s' \\mid s, \\mathrm{do}(a))$; if the policies diverge, the paper's separation is supported, but if they coincide, the two algebras can be mixed and the argument's core step is undercut.","tokens_in":1691,"feed_emoji":"🧠","tokens_out":2211,"duration_ms":66934,"temperature":0.7,"pith_summary":"The paper sets out to settle the debate over whether reinforcement learning is a causal problem by comparing the algebraic structures that associative and causal information each induce. Its central claim is that the operations used by RL—expectations, value functions, Bellman updates—live in an associative algebra with no built-in order, whereas causal reasoning requires an ordered algebra equipped with a context for interventions. Because the authors argue these two algebras are not isomorphic, they conclude that no causal tool can be applied to solve the RL parameter $\\beta$ as currently formulated. The practical stakes are that RL and causal inference should not be treated as interchangeable, and that mixing their languages is a category error, even when the motivating scenario involves an agent intervening in an environment.","feed_headline":"Reinforcement learning is not a causal problem","feed_subtitle":"Two order-free and order-carrying operations form non-isomorphic algebras, so RL tools cannot solve causal queries.","key_machinery":"The central machinery is a pair of binary operators: $\\bigoplus$ for the associative algebra, which captures correlations without any distinguished order, and $\\bigotimes(Z, \\succcurlyeq)$ for the causal algebra, which carries an implicit order and a context for interventions. These operators encode the relation $aRb$ that defines a problem; in RL the relation uses only $\\bigoplus$, placing $\\beta$ in the associative algebra. The argument is carried by the analogy with $G_1 = (\\mathbb{Z}, +)$ and $G_2 = (\\mathbb{Z}, \\cdot)$: non-isomorphic structures over the same set cannot exchange their solution methods, so associative and causal tools cannot be mixed.","core_discovery":"The paper's central claim is that the relation $R$ expressing the RL objective is an associative operation $\\bigoplus$, and that any legitimate transfer of tools from causal reasoning would require an isomorphism between the associative algebra and the causal algebra. Since the associative algebra has no order while the causal algebra is defined by an implicit order $\\succcurlyeq$ and a context $Z$, the paper asserts that no such isomorphism exists. Consequently, the target quantity $\\beta = f[aRb]$ with $R = \\bigoplus$ cannot be obtained with causal machinery, and standard reinforcement learning is not a causal problem. The authors locate RL on the first, associational level of causal reasoning and argue that even the intuition of an agent taking actions does not change the mathematical formulation. They close by speculating that a genuinely causal version of RL would require modifying the Bellman equations to use an interventional distribution $P_g(s' \\mid s, \\mathrm{do}(a))$.","pith_inferences":["Extension: if the non-isomorphism were made rigorous, the argument would imply a separation principle: no value function computed from observational correlations can identify interventional quantities without adding structure, a claim that resembles identifiability results but is here framed algebraically.","Extension: the analogy could be tested concretely by defining the Bellman update operator as an element of an associative algebra and the intervention-modified Bellman operator as an element of a causal algebra, then checking whether any homomorphism between them exists; the paper does not provide that construction.","Extension: the authors' proposed interventional Bellman equations suggest a testable comparison, in a confounded Markov decision process, of the policies learned by standard Q-learning and by an interventional variant; divergence would support the algebraic separation, agreement would weaken it.","Extension: if the paper is right, causal reinforcement learning is not a refinement of RL but a new formulation with additional structure, one that changes the optimality criterion rather than merely adding a causal module to an existing agent."],"forward_implications":["If the claim is correct, attempts to attach causal machinery to standard RL by relabeling actions as interventions are solving a different equation, not the RL equation.","Current RL algorithms that learn from correlations can support associational claims about state and reward, but policy claims about the effect of an action are not established unless the model is changed to include interventions.","A causal variant of RL would be a different problem from standard RL, requiring an interventional distribution such as $P_g(s' \\mid s, \\mathrm{do}(a))$ inside the Bellman equations.","The debate over whether RL is causal becomes a formulation-level question: the same real-world situation can be modeled associatively or causally, but the mathematical problem and its solution change with the algebra.","Success of standard RL algorithms does not by itself license causal interpretations of the policies they learn."],"supporting_citations":[{"why":"Defines the standard RL problem and its value-function solution, the target that the paper argues is purely associative.","marker":"Sutton and Barto (1998)"},{"why":"States the opposing position that RL is a causal problem because agents intervene, which the paper aims to refute.","marker":"Szepesvari (2018)"},{"why":"Supplies the three-level hierarchy of association, intervention, and counterfactual, used to locate RL at the associative level.","marker":"Pearl (2009)"},{"why":"Provides the manipulationist notion of causation with implicit order and context, from which the causal algebra is derived.","marker":"Woodward (2003)"},{"why":"Offers another manipulationist theory of causation and causal structure that informs the causal algebra's content.","marker":"Spirtes et al. (2000)"},{"why":"Supplies the group-versus-semigroup non-isomorphism examples, the analogy carrying the entire argument.","marker":"Hungerford (1974)"},{"why":"Argues reinforcement learning is direct adaptive optimal control, an instance of the control/causal framing the paper opposes.","marker":"Sutton et al. (1992)"}],"fun_headline_variants":["RL's associative algebra blocks causal tool transfer","Causal reasoning can't map onto RL's operation","Reinforcement learning fails causal isomorphism test","RL is associational, not causal, by algebra","Why causal machinery can't solve RL's objective"],"cache_read_input_tokens":5632,"weakest_assumption_plain":"The entire claim rests on an undefined pair of algebras: if the associative and causal operations are not well-defined, non-isomorphic algebraic structures, then the conclusion has no formal basis.","fun_headline_variants_meta":{"raw":{"variants":["RL's associative algebra blocks causal tool transfer","Causal reasoning can't map onto RL's operation","Reinforcement learning fails causal isomorphism test","RL is associational, not causal, by algebra","Why causal machinery can't solve RL's objective"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000125,"raw_usage":{"total_tokens":1015,"prompt_tokens":763,"completion_tokens":252,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":379,"completion_tokens_details":{"reasoning_tokens":181}},"tokens_in":379,"tokens_out":252,"duration_ms":3293,"temperature":1.0,"reasoning_tokens":181,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:00:41.413937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a small Markov decision process with a confound and compare the policy obtained from the standard Bellman equation with the policy obtained from the equation using $P_g(s' \\mid s, \\mathrm{do}(a))$; if the policies diverge, the paper's separation is supported, but if they coincide, the two algebras can be mixed and the argument's core step is undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"States the opposing position that RL is a causal problem because agents intervene, which the paper aims to refute."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the three-level hierarchy of association, intervention, and counterfactual, used to locate RL at the associative level."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the manipulationist notion of causation with implicit order and context, from which the causal algebra is derived."},{"cited_title":"N., and Scheines, R","cited_arxiv_id":null,"evidence_quote":"Offers another manipulationist theory of causation and causal structure that informs the causal algebra's content."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the group-versus-semigroup non-isomorphism examples, the analogy carrying the entire argument."},{"cited_title":"S., Barto, A","cited_arxiv_id":null,"evidence_quote":"Argues reinforcement learning is direct adaptive optimal control, an instance of the control/causal framing the paper opposes."}],"review_version":1}