Pith. sign in

REVIEW 3 major objections 5 minor 49 references

IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read IADD-TR claims that decomposing transitions into an action-intervention stage and an action-free natural-evolution stage, anchored by a zero action, plus a targeted density-scaled critic correction, reduces confounding bias and improves…

desk verdict A genuinely new MBRL idea with plausible empirical gains, but the headline double-robustness theorem overclaims as stated and the zero-action anchor is never verified in the MuJoCo environments. read the letter →

arxiv 2608.10634 v1 pith:LURVJ6B7 submitted 2026-08-11 cs.LG

Zefeng Liang , Jie Qiao , Ruichu Cai , Weilin Chen , Zhifeng Hao This is my paper · ORCID
classification cs.LG
keywords model-basedreinforcementlearningconfoundingbiascausalinferencedynamicsmodelingintervention-awaredecouplingtargetedregularizationefficientinfluencefunctionsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Model-based reinforcement learning usually treats the dynamics model and the critic as monolithic estimators, and the paper argues that this hides a policy-induced confounding bias: the agent's own action choices correlate actions with states, and uneven replay coverage distorts counterfactual value estimates. IADD-TR is proposed as a unified fix: factor each transition into an action-intervention stage followed by an action-free natural-evolution stage, anchor the factorization with the zero-action identity $p_{\mathrm{act},o}(s_t,a_0)=s_t$, and regularize the critic with a density-scaled residual derived from the efficient influence function of the policy-gradient target. The paper claims this reduces both model-learning and policy-learning bias, that the observable post-intervention state is pointwise identifiable while the latent part is identifiable up to componentwise invertible maps, and that the resulting policy-gradient estimate is doubly robust. The payoff would be model-based agents that learn more sample-efficiently on control tasks where a meaningful zero action exists.

What carries the argument

The load-bearing object is the zero-action-anchored two-stage transition model: an action-intervention mechanism $p_{\mathrm{act}}$ that produces an observable intermediate state $\tilde{s}_{t,o}$ plus a latent block $\tilde{s}_{t,l}$, followed by an action-free natural-evolution mechanism $p_{\mathrm{env}}$. The hard identity $p_{\mathrm{act},o}(s_t,a_0)=s_t$ is what makes the split physical rather than arbitrary. The second mechanism is the targeted critic $Q_{\mathrm{adj}}=Q_\omega+\epsilon_\xi(a)/\pi_\beta(a|s)$, a density-scaled residual correction derived from the efficient influence function of the policy-gradient functional. Theorem 1 (pointwise identifiability of the observable block) and Theorem 2 (componentwise identifiability of the latent block) carry the model-side argument, while Theorem 4 (double robustness of the targeted policy-gradient estimator) carries the policy-side argument.

What would settle it

Apply the zero-torque action to a held-out set of states in a MuJoCo environment for one control timestep using the paper's integration settings and compare the resulting state with the original. If the mean squared displacement is not negligible relative to the state scale, the anchor $p_{\mathrm{act},o}(s_t,a_0)=s_t$ is violated and the toy-system validation does not transfer. A second check would instrument the simulator to expose the true post-intervention state and compare it with the decoded observable block of the learned model.

Watch

Extended reading notes

Core claim

The central claim is that the confounding bias in MBRL is structural and can be reduced by deconstructing the two estimators. For dynamics, each transition is written as $s_{t+1}=p_{\mathrm{env}}(p_{\mathrm{act},o}(s_t,a_t),p_{\mathrm{act},l}(s_t,a_t,\varepsilon_t))$, with the zero-action anchor $p_{\mathrm{act},o}(s_t,a_0)=s_t$ enforced by construction; Theorems 1 and 2 then assert that, on the zero-action-covered reachable support, the observable post-intervention state is recovered pointwise and the latent coordinates are recovered up to permutation and one-dimensional diffeomorphisms. For policy learning, the paper derives the efficient influence function of the replay-state policy gradient and builds an adjusted critic $Q_{\mathrm{adj}}=Q_\omega+\epsilon_\xi(a)/\pi_\beta(a|s)$; Theorem 4 asserts the corrected policy-gradient estimator is consistent if either the critic is consistent for $Q^{\pi_\theta}$ or the replay action-density estimate is consistent. Empirically, the paper claims improved sample efficiency and final returns over SAC, PPO, SLBO, and MBPO on five MuJoCo tasks, with diagnostics showing that zero-action data steer the learned intermediate state toward the simulator-defined post-intervention state and that targeted regularization increases cosine similarity to a Monte Carlo policy-gradient reference.

Load-bearing premise

The load-bearing premise is that a zero action exists and truly leaves the observable state unchanged over the intervention interval in the target environment; if the zero-torque action does not correspond to a physical no-op, the learned intermediate state may not be the intended post-intervention state and the identifiability claims lose their physical meaning.

Editorial extensions

If this is right

  • Existing Dyna-style MBRL pipelines can be modified to collect zero-action transitions within the same interaction budget and use them to debias dynamics learning rather than to change exploration.
  • The critic regularizer offers a principled way to correct uneven action coverage without pessimistic policy constraints, so it could complement or replace conservatism in offline and low-data settings.
  • Because the correction uses the logged behavior density, it applies in any off-policy actor-critic method that stores the action-selection score with each transition, with no additional propensity-model fitting.
  • The identifiability guarantees imply that the learned intermediate state can serve as a calibrated representation of the action's direct effect, which is useful for model interpretation and for transferring model components across environments with the same physical structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Any environment with a true no-op action that leaves the observable state invariant over the control timestep should inherit the same decoupling and double-robustness benefits, provided the anchor is verified empirically rather than assumed.
  • Because the MuJoCo benchmarks never record a simulator-defined post-intervention state, the physical interpretation of the gains there rests on the synthetic validation; an instrumented benchmark that exposes that state would directly test whether the anchor is the mechanism behind the improvement.
  • The double robustness result suggests a fixed-data off-policy evaluation test: on a logged replay buffer with known action densities, the adjusted critic should yield policy-gradient estimates whose directional agreement with a long-horizon Monte Carlo reference exceeds a TD-trained critic when action coverage is skewed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes IADD-TR, a model-based reinforcement learning method with two components. IADD factorizes one-step transitions into an action-intervention stage and an action-free natural-evolution stage, using a hard zero-action anchor to identify the intermediate representation. TR augments the critic with a density-scaled residual correction derived from the efficient influence function of a replay-state policy-gradient functional, with a claimed double-robustness guarantee. The authors provide identifiability theorems for the intermediate representation, a double-robustness theorem for the policy-gradient estimator, a controlled synthetic experiment for the zero-action anchor, a paired diagnostic for the targeted critic, and MuJoCo experiments on five tasks showing improved sample efficiency over MBPO and other baselines.

Significance. If the theoretical claims held, IADD-TR would be a meaningful contribution: it gives a concrete causal-decomposition approach to dynamics modeling in MBRL and connects targeted learning ideas to actor-critic policy-gradient estimation. The paper has several genuine strengths: the two-stage dynamics formulation is clearly motivated, the identifiability theorems are nontrivial and come with proofs, the synthetic study directly tests the zero-action mechanism, and the Q4 diagnostic isolates the critic-level effect rather than reporting only end-to-end returns. The paper also states its limitations explicitly, including the dependence on a meaningful zero-action anchor. However, the central double-robustness theorem is false as stated, and the zero-action anchor is not verified in the actual MuJoCo environments, so the theoretical foundations of both main components need substantial repair.

major comments (3)
  1. [Theorem 4 / Appendix F] The double-robustness claim is not established for the estimator in Eq. (21). The proof in Appendix F defines the target as the action-wise marginal conditional-mean curve μ0(a)=E_S[Qπθ(S,a)] in Eq. (46), not the policy-gradient functional ψpg of Eq. (13). The density-correct branch only enforces E_S[h(S,a)]=0 for each action a, where h=Qadj−Qπθ, but Eq. (21) is biased by B=E_{S,a∼πθ}[h(S,a)gθ(S,a)]. The TR loss in Eq. (19) is unweighted by gθ, so the fluctuation solves the score equation for μ0(a), not for the EIF of ψpg derived in Theorem 3. A finite counterexample confirms the issue: take S∈{0,1} with P(S=1)=1/2, binary actions with πβ uniform, true Q(s,a)=as, baseline critic bm=0, and known replay density be=e0; then the density branch gives μ*(a)=μ0(a), yet B=−0.25 p1(1−p1)≠0 for p1=sigmoid(θ). Thus Theorem 4 as stated is false; only the critic-consistent branch follows from the appendix.
  2. [Section IV-A and Section V-A / Appendix B] The zero-action anchor is the load-bearing premise of the IADD identifiability results, but it is verified only in the synthetic ODE system of Appendix B, which is constructed to satisfy \tilde s_{t,o}=s_t when a_t=0. In the MuJoCo experiments, an all-zero torque vector is substituted into a physics simulator where gravity and contact forces continue to act over the control interval, so the hard constraint p_act,o(s_t,a0)=s_t is an imposed model convention rather than a verified property of the data-generating process. Consequently, Theorem 1's pointwise identifiability conclusion does not transfer to the benchmark environments, and the Q3 synthetic result cannot justify the physical interpretation of the MuJoCo improvements. The paper should either provide a MuJoCo-specific check of the anchor, such as diagnostic evidence that the learned zero-action natural-evolution stage makes the identity stage sufficient, or explicitly restrict the identifiability claims to environments with a meaningful zero-action anchor and present the MuJoCo gains as heuristic.
  3. [Section IV-B, Eq. (17) and Theorem 3] There is a mismatch between the theoretical replay-action model and the implementation's logged densities. Theorem 3 and Theorem 4 model the replay action density as a single conditional density πβ(a|s), but in the algorithm eπβ in Eq. (17) is the per-transition logged density of the behavior policy that generated that particular replay action. In an agent that collects data with time-varying behavior policies, the marginal replay action density is a mixture, and the per-transition logged density is not the oracle density used in the EIF derivation. If the authors intend the logged density as the exact per-transition behavior density, the EIF proof and the double-robustness statement should be redone for a mixture model with a behavior-policy index; as written, the theoretical guarantee does not cover the actual replay distribution.
minor comments (5)
  1. [Section IV-A] The phrase "natural natural evolution stage" appears in the second paragraph of Section IV-A and should be corrected to "natural evolution stage."
  2. [Abstract and Theorem 1] The abstract states that the state-aligned component is identifiable pointwise, but the theorem requires Assumption A1, zero-action support coverage, and a deterministic two-stage model; these qualifications should be reflected in the summary claims.
  3. [Section V-B, Q3] The controlled synthetic study is reported for a single random seed, and the paper explicitly disclaims cross-seed uncertainty; this limitation should be moved into the main text or supplemented with additional seeds before publication.
  4. [Theorem 3, Eq. (16)] The EIF formula uses a Dirac delta notation for continuous actions; the authors should clarify that the expression is a density ratio and specify the measure with respect to which the influence function is defined.
  5. [Appendix F] The assumption that the bootstrapped Bellman target's contribution to the targeted risk is o_P(1) is stated without proof; given that the practical method uses such targets in Eq. (19), this assumption should be either proved or explicitly listed as a regularity condition in Theorem 4.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the derivations are self-contained; the Theorem 4 double-robustness overclaim is a proof gap, not a circular reduction.

full rationale

The paper's methodological derivation chain is not circular. The IADD model class is defined by the zero-action anchor (Section IV-A; Appendix B, Eq. (27)), and Theorem 1's pointwise identifiability is a conditional consequence of that definition plus injectivity assumptions, not a prediction of its own inputs. The EIF for the replay-state policy-gradient functional (Theorem 3, Eq. (16)) is derived by pathwise differentiation of Eq. (13) under the nonparametric replay model, and Theorem 4's consistency claim is stated conditionally on the fluctuation solving the corresponding EIF score equation. Appendix F proves that the fitted fluctuation makes the action-wise marginal conditional-mean curve μ*(a) double-robust (Eqs. (52)-(57)); whether that suffices for the ψpg estimator of Eq. (21) is a genuine logical gap, as the skeptical reading notes, because the proof does not control E_S[h(S,a)gθ(S,a)]. However, a missing or false inference step is a correctness problem, not a circularity. Nothing in the derivation renames a fit as a prediction: the adjusted critic is fitted by the targeted loss (19) and then used in a separately reported held-out diagnostic (Q4), and the zero-action recovery experiment (Q3) uses a simulator-recorded target never used in training, so the measured MSE and correlation are not forced by construction. There are no load-bearing self-citations: references [21], [42], [43], and [49] are external targeted-learning literature, and no uniqueness theorem is imported from the authors' prior work. The paper explicitly acknowledges limitations (end of Section VI; Appendix B's note that the sweep does not isolate the cause of nonzero-ratio differences), further indicating no hidden self-reference. Therefore the appropriate finding is no significant circularity; a low score is justified despite the Theorem 4 proof gap.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claim rests on two fitted parameters (zero-action ratio and regularization weight), a domain assumption that the zero action is truly intervention-free in the target environments, and unverified regularity conditions for the identifiability theorems. The invented intermediate representation is the key modeling device but has no independent evidence outside the controlled toy system.

free parameters (2)
  • lambda_0 (zero-action collection ratio) = 0.1
    The fraction of environment interactions where the SAC action is replaced by the zero vector. Set to 0.1 for all MuJoCo tasks. The synthetic study varies it from 0 to 0.4, showing results depend on this choice. It is an ad hoc parameter introduced by the method.
  • beta (targeted regularization weight) = 1
    Controls the strength of the targeted regularization term in the critic loss. Fixed to 1 across all MuJoCo environments. A single value is used, with no sensitivity analysis or tuning procedure reported.
assumptions (5)
  • domain assumption p_act,o(st, a0) = st, the zero-action anchor holds in the real environment
    Used in Section IV-A and the proof of Theorem 1. The model class enforces this by construction, but the paper does not verify that a zero action leaves the observable state unchanged in MuJoCo. In physics simulators, gravity and contact forces act even with zero applied torque.
  • domain assumption A1: natural evolution mechanisms are injective, have continuous inverses, and are twice continuously differentiable
    Assumed in Theorem 1 and used in the proof. Not verified for the MuJoCo environments.
  • domain assumption B1 and B2: latent conditional densities are positive, smooth, and sufficiently variable over state-action contexts
    Used in Theorem 2, following standard nonlinear ICA conditions. Reasonable but unverified in the benchmark environments.
  • domain assumption The replay action density e_pi_beta is logged for every transition
    Required by TR in Eq. (17) to compute the adjusted critic. The paper states this is logged, but in practice the behavior policy is a stochastic SAC policy, and its density must be computed exactly, which is standard but not always trivial.
  • domain assumption Overlap and moment conditions in Appendix F for Theorem 4
    The double robustness result relies on these conditions. They are standard in semiparametric theory but are not checked empirically.
invented entities (1)
  • Intermediate representation s_tilde_t = (s_tilde_t,o, s_tilde_t,l)
    purpose: Captures the post-intervention state after the action stage, with an observable block and an auxiliary latent block, used by the natural evolution stage to predict the next state.
    This latent representation is the core abstraction of the IADD model. It has no falsifiable handle outside the paper. In the synthetic study, the simulator records a matching quantity, but the representation is not directly observable in the MuJoCo benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning." pith.science (2026). https://pith.science/paper/LURVJ6B7

@misc{pith2026260810634,
  author       = {Pith},
  title        = {Pith review of: IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LURVJ6B7}},
  note         = {Machine review of arXiv:2608.10634}
}
read the original abstract

Model-based reinforcement learning (MBRL), which learns environment dynamics to generate synthetic experience, is a promising approach to sample-efficient decision making. Numerous methods have been developed to improve dynamics prediction and policy optimization for MBRL through uncertainty estimation, model regularization, and conservative value learning. However, these methods typically treat the transition model and critic as monolithic predictors, overlooking the policy-induced data bias. Consequently, action can become entangled with environmental evolution, while uneven action coverage may distort the counterfactual value estimates used for policy improvement. To address this, we propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR). IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization. Its latent and state-aligned components are identifiable up to an invertible within-block transformation and pointwise, respectively. For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional. TR augments the critic with an action-density-scaled residual correction and optimizes a targeted loss, yielding doubly robust policy-gradient estimation when either the critic or the replay action density is consistently specified. Extensive experiments on five MuJoCo tasks show that IADD-TR achieves competitive returns with improved sample efficiency.

Figures

Figures reproduced from arXiv: 2608.10634 by the authors.

Figure 1
Figure 1. An example of confounding bias in a car-driving scenario, together [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the proposed IADD-TR method. The framework consists of two key components: IADD and targeted regularization (TR). (a) In the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison on MuJoCo continuous-control benchmarks. We compare the proposed method (IADD-TR) with two model-free baselines [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation results on MuJoCo continuous-control benchmarks. We compare MBPO, MBPO with targeted regularization (MBPO+TR), the proposed [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Controlled synthetic dynamics study. The panels show the observable post-intervention-state MSE (left), action-effect Pearson correlation (middle), [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Policy-gradient alignment diagnostic on Hopper. We independently [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 31 canonical work pages

  1. [1]

    R. S. Sutton, A. G. Bartoet al.,Introduction to reinforcement learning. MIT press Cambridge, 1998, vol. 135

    work page 1998

  2. [2]

    Integrated architectures for learning, planning, and reacting based on approximating dynamic programming,

    R. S. Sutton, “Integrated architectures for learning, planning, and reacting based on approximating dynamic programming,” inMachine learning proceedings 1990. Elsevier, 1990, pp. 216–224

    work page 1990

  3. [3]

    A survey on model-based reinforcement learning,

    F.-M. Luo, T. Xu, H. Lai, X.-H. Chen, W. Zhang, and Y . Yu, “A survey on model-based reinforcement learning,”Science China Information Sciences, vol. 67, no. 2, p. 121101, 2024

    2024

  4. [4]

    Pilco: A model-based and data-efficient approach to policy search,

    M. Deisenroth and C. E. Rasmussen, “Pilco: A model-based and data-efficient approach to policy search,” inProceedings of the 28th International Conference on machine learning (ICML-11), 2011, pp. 465–472

    work page 2011

  5. [5]

    Guided policy search,

    S. Levine and V . Koltun, “Guided policy search,” inInternational conference on machine learning. PMLR, 2013, pp. 1–9

    work page 2013

  6. [6]

    Dream to control: Learning behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” inInternational Conference on Learning Representations, 2020

    2020

  7. [7]

    Model- based reinforcement learning: A survey,

    T. M. Moerland, J. Broekens, A. Plaat, C. M. Jonkeret al., “Model- based reinforcement learning: A survey,”Foundations and Trends® in Machine Learning, vol. 16, no. 1, pp. 1–118, 2023

    work page 2023

  8. [8]

    Mastering atari, go, chess and shogi by planning with a learned model,

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepelet al., “Mastering atari, go, chess and shogi by planning with a learned model,” Nature, vol. 588, no. 7839, pp. 604–609, 2020

    2020

Show all 49 references
  1. [9]

    When to trust your model: Model-based policy optimization,

    M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,”Advances in neural information pro- cessing systems, vol. 32, 2019

    2019

  2. [10]

    Dyna, an integrated architecture for learning, planning, and reacting,

    R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,”ACM Sigart Bulletin, vol. 2, no. 4, pp. 160–163, 1991

    1991

  3. [11]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” inInternational conference on machine learning. PMLR, 2018, pp. 1861–1870

    2018

  4. [12]

    Model- ensemble trust-region policy optimization,

    T. Kurutach, I. Clavera, Y . Duan, A. Tamar, and P. Abbeel, “Model- ensemble trust-region policy optimization,” inInternational Conference on Learning Representations, 2018

    2018

  5. [13]

    Improving multi-step prediction of learned time series models,

    A. Venkatraman, M. Hebert, and J. Bagnell, “Improving multi-step prediction of learned time series models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 29, no. 1, 2015

    2015

  6. [14]

    Lipschitz continuity in model- based reinforcement learning,

    K. Asadi, D. Misra, and M. Littman, “Lipschitz continuity in model- based reinforcement learning,” inInternational conference on machine learning. PMLR, 2018, pp. 264–273

    2018

  7. [15]

    Adversarial counterfactual environment model learning,

    X.-H. Chen, Y . Yu, Z. Zhu, Z. Yu, C. Zhenjun, C. Wang, Y . Wu, R.- J. Qin, H. Wu, R. Dinget al., “Adversarial counterfactual environment model learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 70 654–70 706, 2023

    2023

  8. [16]

    Generative adversarial imitation learning,

    J. Ho and S. Ermon, “Generative adversarial imitation learning,”Ad- vances in neural information processing systems, vol. 29, 2016

    2016

  9. [17]

    Off-policy deep reinforcement learning without exploration,

    S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” inInternational conference on machine learning. PMLR, 2019, pp. 2052–2062

    2019

  10. [18]

    Conservative q-learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,”Advances in neural information processing systems, vol. 33, pp. 1179–1191, 2020

    2020

  11. [19]

    Trust the model where it trusts itself - model-based actor-critic with uncertainty-aware rollout adaption,

    B. Frauenknecht, A. Eisele, D. Subhasish, F. Solowjow, and S. Trimpe, “Trust the model where it trusts itself - model-based actor-critic with uncertainty-aware rollout adaption,” inForty-first International Confer- ence on Machine Learning, 2024

    2024

  12. [20]

    M. A. Hernan,Causal Inference: What If, J. M. Robins, Ed. Boca Raton: Taylor & Francis, 2024

    2024

  13. [21]

    M. J. Van der Laan, S. Roseet al.,Targeted learning: causal inference for observational and experimental data. Springer, 2011, vol. 4

    2011

  14. [22]

    Continuous deep q- learning with model-based acceleration,

    S. Gu, T. Lillicrap, I. Sutskever, and S. Levine, “Continuous deep q- learning with model-based acceleration,” inInternational conference on machine learning. PMLR, 2016, pp. 2829–2838

    2016

  15. [23]

    Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning,

    A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine, “Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning,” in2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 7559–7566

    2018

  16. [24]

    Improving pilco with bayesian neural network dynamics models,

    Y . Gal, R. McAllister, and C. E. Rasmussen, “Improving pilco with bayesian neural network dynamics models,” inData-efficient machine learning workshop, ICML, vol. 4, no. 34, 2016, p. 25

    2016

  17. [25]

    Sample- efficient reinforcement learning with stochastic ensemble value expan- sion,

    J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee, “Sample- efficient reinforcement learning with stochastic ensemble value expan- sion,”Advances in neural information processing systems, vol. 31, 2018

    2018

  18. [26]

    Deep reinforce- ment learning in a handful of trials using probabilistic dynamics models,

    K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforce- ment learning in a handful of trials using probabilistic dynamics models,” Advances in neural information processing systems, vol. 31, 2018

    2018

  19. [27]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,”arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  20. [28]

    Uncertainty-based offline reinforcement learning with diversified q-ensemble,

    G. An, S. Moon, J.-H. Kim, and H. O. Song, “Uncertainty-based offline reinforcement learning with diversified q-ensemble,”Advances in neural information processing systems, vol. 34, pp. 7436–7447, 2021

    2021

  21. [29]

    Adversarially trained actor critic for offline reinforcement learning,

    C.-A. Cheng, T. Xie, N. Jiang, and A. Agarwal, “Adversarially trained actor critic for offline reinforcement learning,” inInternational Confer- ence on Machine Learning. PMLR, 2022, pp. 3852–3878

    2022

  22. [30]

    Mopo: Model-based offline policy optimization,

    T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y . Zou, S. Levine, C. Finn, and T. Ma, “Mopo: Model-based offline policy optimization,”Advances in Neural Information Processing Systems, vol. 33, pp. 14 129–14 142, 2020

    2020

  23. [31]

    Morel: Model-based offline reinforcement learning,

    R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, “Morel: Model-based offline reinforcement learning,”Advances in neural infor- mation processing systems, vol. 33, pp. 21 810–21 823, 2020

    2020

  24. [32]

    A review of off-policy evaluation in reinforcement learning,

    M. Uehara, C. Shi, and N. Kallus, “A review of off-policy evaluation in reinforcement learning,”Statistical Science, 2025

    2025

  25. [33]

    Marginal mean models for dynamic regimes,

    S. A. Murphy, M. J. van der Laan, J. M. Robins, and C. P. P. R. Group, “Marginal mean models for dynamic regimes,”Journal of the American Statistical Association, vol. 96, no. 456, pp. 1410–1423, 2001

    2001

  26. [34]

    Doubly robust policy evaluation and learning,

    M. Dud ´ık, J. Langford, and L. Li, “Doubly robust policy evaluation and learning,” inProceedings of the 28th International Conference on International Conference on Machine Learning, 2011, pp. 1097–1104

    2011

  27. [35]

    Estimation of the causal effect of a time-varying exposure on the marginal mean of a repeated bi- nary outcome,

    J. M. Robins, S. Greenland, and F.-C. Hu, “Estimation of the causal effect of a time-varying exposure on the marginal mean of a repeated bi- nary outcome,”Journal of the American Statistical Association, vol. 94, no. 447, pp. 687–700, 1999

    1999

  28. [36]

    Eligibility traces for off-policy policy evaluation,

    D. Precup, R. S. Sutton, and S. P. Singh, “Eligibility traces for off-policy policy evaluation,” inProceedings of the Seventeenth International Conference on Machine Learning, ser. ICML ’00, 2000, p. 759–766

    2000

  29. [37]

    The central role of the propensity score in observational studies for causal effects,

    P. R. Rosenbaum and D. B. Rubin, “The central role of the propensity score in observational studies for causal effects,”Biometrika, vol. 70, no. 1, pp. 41–55, 1983

    1983

  30. [38]

    More robust doubly robust off-policy evaluation,

    M. Farajtabar, Y . Chow, and M. Ghavamzadeh, “More robust doubly robust off-policy evaluation,” inInternational Conference on Machine Learning. PMLR, 2018, pp. 1447–1456

    2018

  31. [39]

    Estimation of regression coefficients when some regressors are not always observed,

    J. M. Robins, A. Rotnitzky, and L. P. Zhao, “Estimation of regression coefficients when some regressors are not always observed,”Journal of the American statistical Association, vol. 89, no. 427, pp. 846–866, 1994

    1994

  32. [40]

    Doubly robust off-policy value evaluation for rein- forcement learning,

    N. Jiang and L. Li, “Doubly robust off-policy value evaluation for rein- forcement learning,” inInternational conference on machine learning. PMLR, 2016, pp. 652–661

    2016

  33. [41]

    Data-efficient off-policy policy evaluation for reinforcement learning,

    P. Thomas and E. Brunskill, “Data-efficient off-policy policy evaluation for reinforcement learning,” inInternational conference on machine learning. PMLR, 2016, pp. 2139–2148

    2016

  34. [42]

    More efficient off-policy evaluation through regularized targeted learning,

    A. Bibaut, I. Malenica, N. Vlassis, and M. Van Der Laan, “More efficient off-policy evaluation through regularized targeted learning,” in JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 15 International Conference on Machine Learning. PMLR, 2019, pp. 654– 663

    2021

  35. [43]

    Targeted maximum likelihood learning,

    M. J. van der Laan and D. Rubin, “Targeted maximum likelihood learning,”The International Journal of Biostatistics, vol. 2, no. 1, 2006

    2006

  36. [44]

    Policy gradi- ent methods for reinforcement learning with function approximation,

    R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour, “Policy gradi- ent methods for reinforcement learning with function approximation,” Advances in neural information processing systems, vol. 12, 1999

    1999

  37. [45]

    Nonlinear ica using auxiliary variables and generalized contrastive learning,

    A. Hyvarinen, H. Sasaki, and R. Turner, “Nonlinear ica using auxiliary variables and generalized contrastive learning,” inThe 22nd interna- tional conference on artificial intelligence and statistics. PMLR, 2019, pp. 859–868

    2019

  38. [46]

    Mujoco: A physics engine for model- based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model- based control,” in2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033

    2012

  39. [47]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  40. [48]

    Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees,

    Y . Luo, H. Xu, Y . Li, Y . Tian, T. Darrell, and T. Ma, “Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees,” inInternational Conference on Learning Representations, 2019

    2019

  41. [49]

    Vcnet and functional targeted regularization for learning causal effects of continuous treatments,

    L. Nie, M. Ye, Q. Liu, and D. Nicolae, “Vcnet and functional targeted regularization for learning causal effects of continuous treatments,”arXiv preprint arXiv:2103.07861, 2021

    2021 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.