Q-function Decomposition with Intervention Semantics with Factored Action Spaces

Debarun Bhattacharjya; Elliot Nelson; Junkyu Lee; Miao Liu; Songtao Lu; Tian Gao

REVIEW 4 major objections 6 minor 46 references

Q-function Decomposition with Intervention Semantics with Factored Action Spaces

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that when a factored MDP's action blocks have non-interacting effects on the next state, the global Q-function can be decomposed without bias into per-block projected Q-functions, with sample-complexity and convergence…

desk verdict Definition 2 has the importance weight inverted, which makes the advertised unbiased decomposition false as written, but the algorithms sidestep the error and the separable-effects idea is worth a serious look. read the letter →

arxiv 2504.21326 v1 pith:XSARQZM3 submitted 2025-04-30 cs.LG cs.AI

Junkyu Lee , Tian Gao , Elliot Nelson , Miao Liu , Debarun Bhattacharjya , Songtao Lu This is my paper · ORCID

classification cs.LGcs.AI

keywords reinforcementlearningfactoredactionspacesQ-functiondecompositioninterventionsemanticscausalinferencesamplecomplexityofflinemodel-based

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether reinforcement learning over a combinatorial action space, where a full action is a tuple of lower-dimensional choices, can be done by learning one small Q-function per action block instead of one huge Q-function over all combinations. It argues yes, provided each action block changes only its own slice of the next state, so the effect of a full action can be written as a product of independent per-block interventions. The central result is an unbiased decomposition: the full Q-function can be recovered by weighting each per-block projected Q-function with a propensity factor computed from the system's no-op dynamics, and a model-based algorithm using this decomposition converges to the optimal policy when the underlying Q-function is monotonic. The sample-complexity bounds grow with factor sizes, not with the product of all action dimensions. A practical action-decomposed framework then augments DQN and BCQ with this idea and demonstrates faster learning on a 2D control task and an offline sepsis-treatment dataset.

What carries the argument

The load-bearing object is the weighted projected Q-function $\tilde Q^{\pi_k}(s,a_k)$: a Q-function that lives on one action block $A_k$ but accounts for the other blocks by forcing them to follow the evaluation policy and reweighting next-state probabilities by the no-op propensity $\rho_{-k}(s,s')$. This object carries the argument because it turns the full action's intervention into a product of independent per-block interventions, converting joint learning over a combinatorial action space into separate per-block learning problems. In the model-based version, MB-FPI alternates per-block policy evaluation and improvement; in the deep versions, AD-DQN and AD-BCQ replace the single critic with $K$ sub-critics plus a mixer network that implements the combination rule.

What would settle it

Run MB-FPI on a small tabular MDP in which the next value of one state variable depends on actions from two different blocks, violating the non-interaction assumption; if per-block Q-functions still recover the exact optimal value with unlimited data, the assumption is unnecessary, and if they do not, the unbiasedness claim fails exactly in the regime the paper excludes.

Watch

Extended reading notes

Core claim

Under the paper's non-interacting-effects assumption, the interventional transition for a factored action $a=(a_1,\dots,a_K)$ factors as $P(S'|S,\mathrm{do}(a))=P(S'_{K+1}|S,\mathrm{Eff}(A))\prod_{k=1}^K P(S'_k|S,\mathrm{do}(a_k))$, so each action block's effect is confined to its own slice of the next state. The paper defines a projected MDP per block and proves that the global Q-function can be represented through weighted projected Q-functions $\tilde Q^{\pi_k}(s,a_k)$, each reweighted by the no-op propensity $\rho_{-k}(s,s')=\prod_{i\ne k}P(s'_i=\sigma_{A_i}(\mathrm{Pre}(A_i))|s)$. In the tabular model-based setting, MB-FPI learns these per-block pieces; Theorem 1 states that it converges to a locally optimal policy and to the globally optimal policy whenever $Q^\pi$ is monotonic, while Theorem 2 gives Hoeffding-style sample bounds that scale with factor sizes instead of the full joint action space. The authors then instantiate the same decomposition as AD-DQN and AD-BCQ, using learned projected dynamics to synthesize training samples, and report sample-efficiency improvements in continuous control and in an offline sepsis-treatment dataset.

Load-bearing premise

The decomposition stands or falls on the assumption that each action block changes only its own part of the next state, with no interaction between blocks; if two blocks' effects touch the same variable, or the reward depends on more than one block, the per-block estimates become biased.

Editorial extensions

If this is right

Value-based RL can learn one critic per action block and still recover the full action's value, so algorithms no longer need to enumerate the combinatorial product of discrete actions.
MB-FPI always reaches a locally optimal policy, and it reaches the global optimum when the underlying Q-function is monotonic, matching ordinary policy iteration in that case.
Sample needs for learning dynamics shrink from the full state-action product $|S|^2|A|$ to factor-sized terms such as $|\mathrm{Eff}(A_k)||\mathrm{Pre}(A_k)|/\epsilon^2\log(2|\mathrm{Pre}(A_k)|/\delta)$.
Augmenting DQN and BCQ with per-block critics and projected-dynamics data augmentation improves sample efficiency on a 2D control task and on an offline sepsis-treatment dataset, with AD-BCQ dominating baselines in WIS/ESS across action-space sizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Learning the action-effect partition from data, rather than assuming it is given, would let the decomposition apply where causal structure is unknown; the paper explicitly leaves this to future work.
The unbiasedness guarantee is proved for the tabular model-based setting; in the deep variants, learned dynamics and reward models inject approximation error, so a sensitivity analysis of how per-block model error propagates into the global Q estimate would be a direct stress test.
The no-op propensity view suggests a way to multiply sample reuse: any transition whose action block matches the no-op action already carries information about that block's projected dynamics, so replay can be expanded before new data arrives.
Where monotonicity fails, per-block improvement is only local; auditing or constraining the mixer to enforce monotonicity would tell a practitioner whether the global optimality guarantee actually applies.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Definition 2 inverts the importance weight. To bridge the projected transition and the full interventional transition, you need to multiply by the reciprocal of the no-op probability; the paper multiplies by the probability itself, so Eq. (11) does not define an unbiased estimator and the advertised decomposition is false as written. The good news is that the practical algorithms never use this weight. Algorithm 1 directly simulates the full do(A) transition with the other blocks fixed to the current policy, and the deep RL variants train unweighted per-component critics with augmented data. So the model-based idea can be repaired with a one-line fix and the empirical results do not rest on the faulty equation.

I want to give credit where it is due. The separable-effects condition (Assumption 1) is a genuine extension of the fully separable MDPs in Tang et al. and Seyde et al.: the action partition matches a partition of next-state variables, so per-block interventions do not interact even when the dynamics and reward are not fully separable. The causal intervention framing is a useful lens for thinking about Q-decomposition, and MB-FPI is a clean way to evaluate per-block Q-functions. Theorem 2's sample-complexity bound for learning the no-op dynamics is a reasonable contribution, and the experiments show consistent improvements over factored baselines, which is encouraging.

The soft spots are not minor. The Definition 2 error is load-bearing because the paper explicitly claims unbiasedness; it is not just a typo, because it invalidates the theoretical justification for the weighted estimator, though the fix is obvious. Second, the paper never states a separable-reward assumption. Definition 2 and Algorithm 2 evaluate R(s, a_k, s') even though the original reward may depend on the full joint action; if reward coupling exists, the per-block Q functions do not approximate the full Q. Third, there is a theory-practice gap: the weighted projected Q-functions are not what AD-DQN and AD-BCQ train, so the practical algorithms lack a complete theoretical guarantee. Finally, the paper ships no code, no data, and no confidence intervals; the sepsis results are a single evaluation of a few seeds, and the authors themselves flag that the outcome needs domain-expert review.

Who is this for? Researchers working on factored action spaces and Q-decomposition. The core idea is plausible and the separable-effects condition is a useful step beyond full separability, but the current write-up has a central theoretical error and the experiments are not fully reproducible. I would not desk reject this: it deserves a serious referee who can check the math and ask for a revision. If the authors fix the weight, add the reward assumption, and release code, it could be a solid contribution. As is, treat the theoretical claims as provisional.

Referee Report

4 major / 6 minor

Summary. The paper studies Q-function decomposition for factored action spaces in reinforcement learning. It models actions as interventions in a factored MDP under a no-unobserved-confounder semantics and assumes that the effects of disjoint action blocks are non-interacting (Assumption 1). It defines a projected action-space MDP M_k for each action block A_k and a weighted projected Q-function (Definition 2, Eq. (11)) claimed to equal the full Q-function under a propensity weighting. It then presents model-based factored policy iteration (MB-FPI), with Theorem 1 (local convergence, plus global convergence under a monotonicity condition) and Theorem 2 (sample complexity growing with factor sizes rather than the joint action space), and practical algorithms AD-DQN and AD-BCQ that train per-block critics with learned dynamics and reward models. Experiments on a 2D point-mass control task and a MIMIC-III sepsis treatment environment report improved sample efficiency over DECQN and factored-BCQ baselines.

Significance. The core idea, using intervention semantics to justify per-block Q-learning in MDPs with non-interacting action effects, is timely and potentially useful; if correct, it would reduce the sample complexity from the full joint action space to the product of factor sizes. The paper explicitly states its structural assumptions, includes attempted proofs, and provides a sample-complexity analysis. However, the central unbiasedness derivation contains an inverted importance weight and an unstated reward-separability assumption, and the practical algorithms do not implement the weighting. These issues must be resolved before the theoretical claims can be accepted. The empirical results are suggestive but are not connected to the theory as written.

major comments (4)

[Section 3.1, Definition 2, Eq. (11)] Definition 2, Eq. (11), uses the wrong importance weight. Eq. (9) shows that changing measure from P(s'|s,do(a_k)) to P(s'|s,do(A)) requires multiplying by the product over i≠k of I[S'_i=σ_{A_i}(Pre_i)] / P(S'_i=σ_{A_i}(Pre_i)|s); therefore rho_{-k}(s,s') should be the reciprocal of the product of the no-op probabilities. As printed, rho_{-k} is the product itself. For deterministic interventions, E_{s'~P(·|s,do(a_k))}[rho_{-k} I[Eff(A)=σ_A(Pre)]] = ∏_{i≠k} P(S'_i=σ_{A_i}(Pre_i)|s), which is less than 1 under Assumption 2 unless the no-op dynamics are deterministic. Hence the weighted projected Q-function in Eq. (11) does not equal Q^pi(s,a) as claimed. Algorithm 1 line 5 avoids this issue by constructing the full do(A) transition directly, but then Algorithm 1 does not implement the object analyzed in Definition 2.
[Proposition 1, Definition 2, Algorithm 2 line 13] Proposition 1, Definition 2, and Algorithm 2 line 13 all write the reward as R(s,a_k,s'), whereas the full Q-function in Eq. (6) depends on the joint action through R(s,a,s'). Assumption 1 factorizes only the transition, not the reward. Without an explicit separability or action-block-reward condition, the projected Q-function is the value of a different MDP, and the proof of Theorem 1 in Appendix A.1 silently substitutes R(s,a,s') for R(s,a_k,s') when concluding that the projected Q equals the full Q. This missing assumption is load-bearing for both the theory and the practical reward-model training.
[Algorithms 2 and 3] Algorithms 2 and 3 (lines 13–14 and 7–9) train the per-block critics Q^{pi_k}(s,a_k) on batches B_k that are modified to follow P(S'|S,do(A_k)), with no propensity weighting of the form in Eq. (11). The practical AD-DQN and AD-BCQ therefore do not implement the weighted projected Q-function whose unbiasedness Section 3.1 aims to establish. If the unweighted version is claimed to be valid under additional structure, that claim should be stated and proved; as presented, the experiments do not test the theoretical mechanism.
[Theorem 1 and Appendix A.1] Theorem 1's global-convergence clause depends on a 'monotonic Q' condition that is not defined in the paper. The proof in Appendix A.1 asserts that block-wise improvement is equivalent to joint improvement under monotonicity, but this equivalence is precisely the property needed for the conclusion; no argument shows it follows from Assumption 1 or Assumptions 2–3. This makes the global-optimality result conditional on an essentially tautological assumption. The statement should be made precise, or the theorem should be weakened to local convergence.

minor comments (6)

[Eq. (10)] Eq. (10) drops the indicator functions from Eq. (9); it should write the ratio with I[S'_i=σ_{A_i}(Pre_i)] in the numerator.
[Algorithm 1, line 7] Algorithm 1, line 7: the right-hand side contains Q^{pi_k}(s,a_k) rather than Q^{pi_k}(s',pi_k(s')); as printed this is not a valid Bellman backup.
[Appendix A.1, Proposition 1 proof] The proof of Proposition 1 in Eqs. (13)–(15) has inconsistent discount factors and summation indices; for example, the term prod_{j=0}^t gamma^t does not match the nested sums in the preceding line.
[Definition 2, notation] Definition 2 switches between sigma_{A_i}(Pre(A_i)) and sigma_{a_i}(Pre(a_i)); clarify whether the intervention policy is indexed by the action variable or by its value and whether Pre may depend on the full state.
[Section 5.2 and Figure 4] There are typos in Section 5.2: 'vassopressors' should be 'vasopressors' and Figure 4 captions say 'binds' where 'bins' is intended.
[Theorem 2] Theorem 2 uses the expression |S\S_{K+1}| without definition; if it denotes the cardinality of the product of state variables outside block K+1, this should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the decomposition is a conditional change-of-measure identity under explicit structural assumptions.

full rationale

The paper's theoretical chain is conditional on explicit structural assumptions rather than on its own conclusions. Assumption 1 (Eq. 5) states a factorization of the interventional transition, and Eqs. (8)-(10) algebraically rewrite P(S'|S,do(A)) relative to P(S'|S,do(A_k)); the weighted projected Q-function in Definition 2 is then an importance-weighted Bellman object built from that ratio, not a fitted parameter renamed as a prediction. Algorithm 1's line 5 uses the full interventional transition directly, so the model-based policy-evaluation argument is a substitution identity under Assumption 1, not a circular definition. Theorem 1's global optimality clause explicitly imports monotonicity as an additional assumption from the external multi-agent-RL literature (Rashid et al., 2020), and the proof merely shows coordinate ascent reaches the joint fixed point under that assumption; this is a conditional theorem, not a self-citation or an assumed conclusion. Theorem 2 follows a standard Hoeffding/union-bound sample-complexity argument. There are no load-bearing self-citations, no fitted quantities later reported as predictions, and no imported uniqueness theorem. The apparent reciprocal misplacement of rho in Definition 2 (the importance weight is printed as the product of probabilities rather than their reciprocals) is a correctness concern about the formula as written, but it is not a circularity: even if Eq. (11) is biased as printed, that does not make the derivation equivalent to its own input. Overall, the central claim rests on stated structural assumptions and external mathematical conditions, so no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The theoretical results are conditional on four structural assumptions (no hidden confounders, separable effects, positive no-op dynamics, support equivalence), plus a monotonicity condition for global optimality and an unstated separable-reward assumption for the practical algorithms. The sample-complexity theorem relies on standard Hoeffding and union-bound arguments. No new physical entities are introduced.

free parameters (2)

Dynamics model noise variance = 0.0001
Used in DynamicsModelDelta2l.evaluate in Appendix A.2.1 for point-mass data augmentation; a hand-chosen constant that affects synthetic transitions and the reported speedups.
No-op fraction probability = 0.1
AD-DQN exploration hyperparameter from Appendix A.2.1 chosen by grid search; it affects sample collection in the point-mass experiments.

assumptions (7)

domain assumption No unobserved confounder setting: state S is the only confounder between action do(A) and next state S' and reward R.
Invoked in Section 2 to justify the interventional factorization P(S'|S,do(A)) and Eq. (3); hidden confounders would break the causal effect estimation and propensity weighting.
domain assumption Assumption 1: action variables can be partitioned so that each block A_k affects only its own next-state block S'_k = Eff(A_k) with no interaction.
Section 3.1, Eq. (5); this exact factorization is the basis for projected MDPs, weighted Q-functions, and all theorems.
domain assumption Assumption 2: no-op dynamics P(S'|S) is positive.
Section 3.2; used to define propensity weights rho_{-k}(s,s') and to make the ratios in Eqs. (8)-(10) well-defined.
domain assumption Assumption 3: supports of next states induced by behavior policy pi_b and factored policy pi_b^k coincide.
Section 3.2; required so that exploring projected action spaces does not miss next states reachable under the full action space.
domain assumption Monotonicity of the joint Q-function for global optimality of block-wise policy improvement.
Theorem 1 and Section 7; without it MB-FPI is only guaranteed a local optimum.
ad hoc to paper Reward in the projected MDP can be evaluated with R(s,a_k,s') rather than the full-action R(s,a,s').
Definition 1, Definition 2, and Algorithm 2 line 13 use R(s,a_k,s') while the original reward may depend on the joint action; no separable-reward assumption is stated or verified in the experiments.
standard math Hoeffding's inequality and union bound apply to the tabular counts.
Theorem 2 proof in Appendix A.1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Q-function Decomposition with Intervention Semantics with Factored Action Spaces." pith.science (2026). https://pith.science/paper/XSARQZM3

@misc{pith2026250421326,
  author       = {Pith},
  title        = {Pith review of: Q-function Decomposition with Intervention Semantics with Factored Action Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XSARQZM3}},
  note         = {Machine review of arXiv:2504.21326}
}

read the original abstract

Many practical reinforcement learning environments have a discrete factored action space that induces a large combinatorial set of actions, thereby posing significant challenges. Existing approaches leverage the regular structure of the action space and resort to a linear decomposition of Q-functions, which avoids enumerating all combinations of factored actions. In this paper, we consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions using causal effect estimation from the no unobserved confounder setting in causal statistics. This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms. The proposed approach is shown to improve sample complexity in a model-based reinforcement learning setting. We demonstrate improvements in sample efficiency compared to state-of-the-art baselines in online continuous control environments and a real-world offline sepsis treatment environment.

Figures

Figures reproduced from arXiv: 2504.21326 by the authors.

**Figure 2.** Comparing the values in the test environment on three action spaces. The X-axis shows 2 million steps [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3a and 3b visualize the number of samples for two discrete action spaces in the training set. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (11 more)

**Figure 4.** Figure 4: Discretized action spaces ranging from 5x5 to 14x14 [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

**Figure 5.** Figure 5: The train and validation loss (Gaussian NLL) for 5 different action spaces ranging from 5x5 to 14x14. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

**Figure 6.** Figure 6: The train and validation mean squared error (MSE) for 5 different action spaces ranging from 5x5 to [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

**Figure 7.** Figure 7: Validation performance of action space 5x5 in terms of WIS and ESS for all BCQ threshold parameters [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

**Figure 8.** Figure 8: Validation performance of action space 10x10 in terms of WIS and ESS for all BCQ threshold parameters [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

**Figure 9.** Figure 9: Validation performance of action space 11x11 in terms of WIS and ESS for all BCQ threshold parameters [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]

**Figure 10.** Figure 10: Validation performance of action space 13x13 in terms of WIS and ESS for all BCQ threshold [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]

**Figure 11.** Figure 11: Validation performance of action space 14x14 in terms of WIS and ESS for all BCQ threshold [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]

**Figure 12.** Figure 12: Model Selection Scores: the X-axis is minimum ESS cutoff values and the Y-axis is ESS computed in [PITH_FULL_IMAGE:figures/full_fig_p029_12.png]

**Figure 13.** Figure 13: Model Selection Scores: the X-axis is minimum ESS cutoff values and the Y-axis is ESS computed in [PITH_FULL_IMAGE:figures/full_fig_p030_13.png]

**Figure 14.** Figure 14: Performance Scores of Selected Models: Each point represents the performance score of the selected [PITH_FULL_IMAGE:figures/full_fig_p031_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 32 canonical work pages

[1]

M., and Sun, W

Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. (2019). Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , 32:96

work page 2019
[2]

Brehmer, J., De Haan, P., Lippe, P., and Cohen, T. S. (2022). Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems

work page 2022
[3]

Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679

arXiv 2015
[4]

Fujimoto, S., Meger, D., and Precup, D. (2019). Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052--2062. PMLR

2019
[5]

Gao, C., Zheng, Y., Wang, W., Feng, F., He, X., and Li, Y. (2024). Causal inference in recommender systems: A survey and future directions. ACM Transactions on Information Systems , 42(4):1--32

work page 2024
[6]

L., Amaral, L

Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. E. (2000). Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation , 101(23):e215--e220

2000
[7]

M., Glymour, C., Sch \"o lkopf, B., and Zhang, K

Huang, B., Lu, C., Leqi, L., Hern \'a ndez-Lobato, J. M., Glymour, C., Sch \"o lkopf, B., and Zhang, K. (2022a). Action-sufficient state representation learning for control with structural constraints. In International Conference on Machine Learning

work page 2022
[8]

Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. (2022b). Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1--18

work page 2022

Show all 46 references

[9]

D., Shalit, U., and Sontag, D

Johansson, F. D., Shalit, U., and Sontag, D. (2016). Learning representations for counterfactual inference. In 33rd International Conference on Machine Learning, ICML 2016 , pages 4407--4418. International Machine Learning Society (IMLS)

2016
[10]

E., Pollard, T

Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., and Mark, R. G. (2016). Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1--9

2016
[11]

W., Zhang, H., Subramanian, J., Fatemi, M., and Ghassemi, M

Killian, T. W., Zhang, H., Subramanian, J., Fatemi, M., and Ghassemi, M. (2020). An empirical study of representation learning for reinforcement learning in healthcare. arXiv preprint arXiv:2011.11235

2020 arXiv
[12]

A., Badawi, O., Gordon, A

Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., and Faisal, A. A. (2018). The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine , 24(11):1716--1720

2018
[13]

Lange, S., Gabel, T., and Riedmiller, M. (2012). Batch reinforcement learning. In Reinforcement learning: State-of-the-art , pages 45--73. Springer

2012
[14]

Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643

2020 arXiv
[15]

Luczkow, V. (2021). Structural Causal Models for Reinforcement Learning . McGill University (Canada)

2021
[16]

Martino, L., Elvira, V., and Louzada, F. (2016). Alternative effective sample size measures for importance sampling. In 2016 IEEE Statistical Signal Processing Workshop (SSP) , pages 1--5. IEEE

2016
[17]

Mitrovic, J., McWilliams, B., Walker, J., Buesing, L., and Blundell, C. (2021). Representation learning via invariant causal mechanisms. In International Conference on Learning Representation

2021
[18]

A., Veness, J., Bellemare, M

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature , 518(7540):529--533

2015
[19]

Pearl, J. (2009). Causality . Cambridge University Press

2009
[20]

Pearl, J. (2019). The seven tools of causal inference, with reflections on machine learning. Communications of the ACM , 62(3):54–60

2019
[21]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in P ython. Jou...

2011
[22]

Pitis, S., Creager, E., and Garg, A. (2020). Counterfactual data augmentation using locally factored dynamics. In Advances in Neural Information Processing Systems

2020
[23]

Pitis, S., Creager, E., Mandlekar, A., and Garg, A. (2022). Mocoda: Model-based counterfactual data augmentation. In Advances in Neural Information Processing Systems

2022
[24]

S., Farquhar, G., Foerster, J., and Whiteson, S

Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. (2020). Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research , 21(1):7234--7284

2020
[25]

P., Tang, S., Wiens, J., and Parbhoo, S

Rebello, A. P., Tang, S., Wiens, J., and Parbhoo, S. (2023). Leveraging factored action spaces for off-policy evaluation. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems

2023
[26]

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology , 66(5):688

1974
[27]

Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association , 100(469):322--331

2005
[28]

Russell, S. J. and Zimdars, A. (2003). Q-decomposition for reinforcement learning agents. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) , pages 656--663

2003
[29]

Saito, Y., Ren, Q., and Joachims, T. (2023). Off-policy evaluation for large action spaces via conjunct effect modeling. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org

2023
[30]

Sch \"o lkopf, B. (2022). Causality for machine learning. In Probabilistic and Causal Inference: The Works of Judea Pearl , pages 765--804

2022
[31]

and Poupart, P

Schulte, O. and Poupart, P. (2024). Why online reinforcement learning is causal. arXiv preprint arXiv:2403.04221

2024 arXiv
[32]

Seyde, T., Werner, P., Schwarting, W., Gilitschenski, I., Riedmiller, M., Rus, D., and Wulfmeier, M. (2022). Solving continuous control via Q -learning. In Proceedings of the International Conference on Learning Representations

2022
[33]

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017). Mastering the game of Go without human knowledge. Nature , 550(7676):354--359

2017
[34]

J., Hostallero, D

Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y. (2019). Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning , pages 5887--5896. PMLR

2019
[35]

Subramanian, J., Sinha, A., Seraj, R., and Mahajan, A. (2022). Approximate information state for approximate planning and reinforcement learning in partially observed systems. Journal of Machine Learning Research , 23(12):1--83

2022
[36]

M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J

Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. (2018). Value-decomposition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonom...

2018
[37]

Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction . MIT press

2018
[38]

Tang, S., Makar, M., Sjoding, M., Doshi-Velez, F., and Wiens, J. (2022). Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems , 35:34272--34286

2022
[39]

and Wiens, J

Tang, S. and Wiens, J. (2021). Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Conference , pages 2--35. PMLR

2021
[40]

and Wiens, J

Tang, S. and Wiens, J. (2023). Counterfactual-augmented importance sampling for semi-offline policy evaluation. Advances in Neural Information Processing Systems , 36:11394--11429

2023
[41]

Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018). Deepmind control suite. arXiv preprint arXiv:1801.00690

2018 arXiv
[42]

Tavakoli, A., Fatemi, M., and Kormushev, P. (2020). Learning to represent action values as a hypergraph on the action vertices. In Proceedings of the International Conference on Learning Representations

2020
[43]

Tavakoli, A., Pardo, F., and Kormushev, P. (2018). Action branching architectures for deep reinforcement learning. In Proceedings of the aaai conference on artificial intelligence , volume 32

2018
[44]

Uehara, M., Shi, C., and Kallus, N. (2022). A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355

2022 arXiv
[45]

Van Hasselt, H., Guez, A., and Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30

2016
[46]

Wang, J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. (2020). Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations

2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

M., and Sun, W

Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. (2019). Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , 32:96

work page 2019

[2] [2]

Brehmer, J., De Haan, P., Lippe, P., and Cohen, T. S. (2022). Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems

work page 2022

[3] [3]

Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679

arXiv 2015

[4] [4]

Fujimoto, S., Meger, D., and Precup, D. (2019). Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052--2062. PMLR

2019

[5] [5]

Gao, C., Zheng, Y., Wang, W., Feng, F., He, X., and Li, Y. (2024). Causal inference in recommender systems: A survey and future directions. ACM Transactions on Information Systems , 42(4):1--32

work page 2024

[6] [6]

L., Amaral, L

Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. E. (2000). Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation , 101(23):e215--e220

2000

[7] [7]

M., Glymour, C., Sch \"o lkopf, B., and Zhang, K

Huang, B., Lu, C., Leqi, L., Hern \'a ndez-Lobato, J. M., Glymour, C., Sch \"o lkopf, B., and Zhang, K. (2022a). Action-sufficient state representation learning for control with structural constraints. In International Conference on Machine Learning

work page 2022

[8] [8]

Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. (2022b). Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1--18

work page 2022