Pith. sign in

REVIEW 2 major objections 4 minor 56 references

Semi-gradient DICE for Offline Constrained Reinforcement Learning

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Semi-gradient DICE actually computes policy corrections, not stationary distribution corrections, so it cannot estimate costs; CORSDICE recovers the true distribution from the policy correction, restoring cost evaluation and constraint…

desk verdict A clear-eyed paper that correctly identifies the policy-correction nature of semi-gradient DICE and offers a practical extraction method; the theoretical gap at the function-approximation level is real but not fatal. read the letter →

arxiv 2506.08644 v1 pith:TWYRDE7G submitted 2025-06-10 cs.LG

Woosung Kim , JunHo Seo , Jongmin Lee , Byung-Jun Lee This is my paper
classification cs.LG
keywords offlineconstrainedreinforcementlearningstationarydistributioncorrectionestimationDICEsemi-gradientoptimizationoff-policyevaluationpolicybehavior-regularizedMDPcostconstraints
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain why semi-gradient DICE — the stabilization trick behind several of the strongest distribution-correction (DICE) offline RL methods — fails at off-policy evaluation, and to repair it. Its diagnosis is that the correction such methods compute is a policy correction, the ratio of the learned policy to the dataset policy, rather than the stationary distribution correction the DICE framework is built on. Because a policy correction carries no information about how often states are visited, any cost estimate computed from it is invalid, which is exactly why naive semi-gradient constrained DICE violates cost limits. The paper then gives CORSDICE, which extracts a valid stationary distribution correction from the policy correction through a convex program, and reports that on the DSRL benchmark CORSDICE is the only algorithm that satisfies cost constraints in every environment while reaching the highest average return.

What carries the argument

The load-bearing object is the distinction between the two corrections a DICE-style dual can output. A stationary distribution correction $w(s,a)=d(s,a)/d_D(s,a)$ satisfies the Bellman flow equality and supports off-policy evaluation of any reward or cost; a policy correction $w(a|s)=\pi(a|s)/\pi_D(a|s)$ satisfies the normalization $\sum_a w(a|s)\pi_D(a|s)=1$ instead. The mechanism that flips one into the other is the semi-gradient update: dropping the gradient through the next-state $\nu(s')$ and replacing the initial-state term with the dataset distribution makes $\nu$ lose its role as the Bellman-flow Lagrange multiplier, so the closed-form correction becomes a policy correction. CORSDICE then runs a second convex program — maximize $-\sum_s d_D(s)f(w(s))$ subject to the Bellman flow constraint with $w(a|s)$ held fixed — whose inner maximization over $w(s)$ has the closed form $w^*(s)=\max(0,(f')^{-1}(\mathbb{E}_{a\sim\pi_D}[w(a|s)e_\mu(s,a)]))$, and it removes single-sample bias from the dual by estimating the expectation inside the convex conjugate $f^*_0$ with an auxiliary network $A(s)$ (Proposition 5.1). The product $w(s)w(a|s)$ is the recovered stationary distribution correction that makes cost evaluation valid.

What would settle it

On a tabular MDP with known dynamics, run SemiDICE and its extraction step, then compare the extracted $w(s)w(a|s)$ against the exact stationary distribution correction of the policy $\pi = w(a|s)\pi_D$. If the normalized cost estimate $\mathbb{E}_{d_D}[w(s)w(a|s)c]$ systematically misses the true cost as dataset size shrinks, while $\sum_a w(a|s)\pi_D(a|s)$ drifts away from 1, the assumed exactness of the policy correction is the failure point. Conversely, the cleanest direct test is whether the equality-constrained extraction problem (15) ever becomes infeasible on a standard benchmark dataset with the learned $w(a|s)$; infeasibility would contradict the paper's reconstruction claim.

Watch

Extended reading notes

Core claim

The central claim, stated as Proposition 4.1, is that the object optimized by SemiDICE is not the object the DICE literature assumed. At the optimum of its $\nu$-loss, the closed-form correction $w^*(s,a)=\max(0,(f')^{-1}((Q(s,a)-\nu^*(s))/\alpha))$ violates the Bellman flow constraint, yet it satisfies $\sum_a w^*(s,a)\pi_D(a|s)=1$ and $w^*(s,a)\ge 0$ — exactly the defining conditions of a policy correction $w(a|s)=\pi(a|s)/\pi_D(a|s)$, not of a stationary distribution correction $w(s,a)=d(s,a)/d_D(s,a)$. It follows that treating $\mathbb{E}_{d_D}[w\,c]$ as a cost estimate is invalid, and the authors confirm the failure experimentally: on D4RL value prediction, SemiDICE's raw correction gives RMSE of 87.6–111.6, while the extracted stationary distribution gives 9.2–26.4. The paper additionally shows (Proposition 4.2) that SemiDICE is an approximate solver for a behavior-regularized MDP, which explains both the policy-correction structure and its strong offline RL performance, and proves (Corollary 4.3) that this structure avoids the state-sparsity failure that leaves OptiDICE with undefined actions in some states. The constructive half is CORSDICE: given the learned $w(a|s)$, it solves a convex program whose dual yields the missing state stationary distribution correction $w(s)$, debiased by an auxiliary approximator $A(s)$, and uses $w(s)w(a|s)$ for off-policy cost evaluation and the cost-constraint multiplier update.

Load-bearing premise

The method assumes the learned correction multiplying the data policy is a perfect probability ratio that sums to one at every state in the dataset, so that a well-defined state distribution obeying the flow of probability can be reconstructed; with finite data and neural networks this exactness is never guaranteed, and the algorithm neither enforces nor repairs it.

Editorial extensions

If this is right

  • The policy correction itself is still usable for policy extraction: because $\pi^*(a|s)=w(a|s)\pi_D(a|s)$, weighted behavior cloning with $w(a|s)$ recovers the implicit policy, which is why SemiDICE-style methods work well as offline RL even though they cannot do evaluation.
  • Using the raw semi-gradient correction for off-policy evaluation is invalid: on D4RL benchmarks SemiDICE's value-prediction RMSE is 87.6–111.6, while the extracted stationary distribution correction brings RMSE down to 9.2–26.4.
  • Any constrained RL built on raw semi-gradient DICE will fail to honor cost limits because its cost estimate is not an expectation under any policy-induced distribution; the extraction step is what makes the Lagrangian multiplier update meaningful.
  • On the DSRL benchmark, CORSDICE is the only method among the compared baselines that satisfies the cost constraint in every environment, records the best return among safe agents in 27 of 38 tasks, and uses less cost as the cost limit tightens; the diffusion-based D-CORSDICE also stays safe in all tasks while beating strong diffusion baselines on average return.
  • The extraction procedure is not specific to costs: the authors note it can be attached to other DICE-based algorithms that need off-policy evaluation, such as return-on-investment maximization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Proposition 4.1 diagnosis is right, then the practical success of SQL, XQL, and f-DVL is best explained as behavior-regularized MDP solving, not as improved DICE; a direct test would be whether SemiDICE's returns match an explicit solver for the behavior-regularized MDP with the same f-divergence.
  • The extraction step is structurally a Q-LP-style problem solved in-sample, so the same machinery should estimate the return of the implicit policy $\pi = w(a|s)\pi_D$, not just its cost — which would turn any semi-gradient DICE variant into a general off-policy evaluation tool for its own policies.
  • Everything hangs on $w(a|s)$ being an exact policy correction, yet nothing in the algorithm enforces $\sum_a w(a|s)\pi_D(a|s)=1$ under function approximation; a robustness audit measuring this violation and the Bellman-flow residual of the extracted $w(s)w(a|s)$ product on the actual deep-RL datasets, and correlating them with cost-estimation bias, would settle how much the finite-sample assumptio
  • The paper's bias-reduction idea handles both transition and policy bias inside the convex conjugate; the same trick could be ported back to full-gradient OptiDICE and COptiDICE, which only address transition bias, potentially making the full-gradient family viable in continuous domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies semi-gradient DICE methods for offline constrained RL. It claims that semi-gradient variants (SemiDICE, f-DVL, ODICE) do not produce stationary distribution corrections but rather policy corrections satisfying \sum_a w(s,a)\pi_D(a|s)=1, and therefore cannot directly perform off-policy evaluation. The paper proposes CORSDICE, which first obtains a policy correction via SemiDICE and then extracts a state stationary distribution correction w(s) by solving a convex optimization dual, enabling cost evaluation for the Lagrange multiplier update in constrained RL. Empirical results are reported on finite MDP diagnostics, D4RL OPE tasks, and the DSRL constrained-RL benchmark, where CORSDICE and its diffusion variant D-CORSDICE achieve strong constraint satisfaction and return.

Significance. The paper identifies a genuinely important phenomenon: the correction produced by semi-gradient DICE is normalized as a policy ratio, not as a stationary-distribution ratio, which explains the OPE failures of prior semi-gradient DICE methods and their empirical success in unconstrained offline RL. The proposed extraction method is novel and empirically effective, substantially improving OPE RMSE on D4RL and achieving state-of-the-art cost-constrained performance on DSRL. The proofs of Proposition 4.1 (for a fixed Q at the coupled fixed point) and Proposition 5.1 (gradient equivalence when A(s) equals its exact target) are clean and well presented. If the representational and feasibility gaps discussed below are addressed, this will be a valuable contribution to offline constrained RL.

major comments (2)
  1. [Section 4, Proposition 4.1] The proposition claims that the SemiDICE correction w*(s,a) "violates the Bellman flow constraint" while satisfying the policy-correction condition. The proof, however, only establishes the policy-correction condition \sum_a w*(s,a)\pi_D(a|s)=1; it does not show that Bellman flow (Eq. 1b) is violated. The statement as written is false as a universal claim: in a single-state MDP with r=0 and a symmetric dataset policy, w*=1 satisfies both the normalization condition and Bellman flow. The paper should weaken the statement to "does not in general satisfy the Bellman flow constraint" or prove the violation under explicit additional assumptions. This matters because the paper uses the proposition to conclude that SemiDICE "fundamentally" cannot perform OPE; the proof as written only shows the correction has the policy-correction property, which does not logically preclude it from also being a stationary distribution correction in special cases.
  2. [Section 5, Eq. (15b) and Algorithm 1] The extraction step (15) is the linchpin of CORSDICE, and its theoretical guarantee requires the learned w(a|s) to be an exact policy correction with \sum_a w(a|s)\pi_D(a|s)=1 for every state in the dataset. Proposition 4.1 provides this only at the exact minimizer of the \nu-loss with a perfect Q(s,a). In the deep RL implementation, Q, \nu, A, and \mu are function approximators trained on finite samples, and the algorithm never projects onto (15b) or normalizes w(a|s). Consequently, the equality-constrained problem (15b) can become infeasible, strong duality invoked in Appendix D need not hold, and the recovered w(s)=max(0,(f')^{-1}(A(s))) is not guaranteed to satisfy the Bellman flow equality. The cost estimate in Eq. (14) and the subsequent \lambda update are then not guaranteed to be unbiased for E_{d_{\pi_w}}[c]. The limitation section (Sec. 7) mentions only increased complexity from the extra approximator, not this normalization/infeasibility issue. The paper should either add a projection or normalization step that enforces (or approximately enforces) the policy-correction condition, or explicitly discuss the bias incurred when (15b) is infeasible.
minor comments (4)
  1. [Section 5, Eq. (15)] The text says the |S| constraints (15b) "uniquely determine w(s), making the problem over-constrained." With |S| unknowns and |S| equality constraints, the system is not over-constrained when consistent; the real issue is potential inconsistency/infeasibility, which is exactly the point raised in the major comment on Eq. (15b). Rephrasing would avoid confusion.
  2. [Section 5, Proposition 5.1] The proposition shows equality of gradients at the fixed point where A(s) equals the exact conditional expectation E_{a,s'}[w(a|s)(\gamma\mu(s')-\mu(s))]. It does not establish convergence of the coupled A/\mu updates or provide a finite-sample bias bound. The paper should state clearly that the bias-reduction guarantee is asymptotic/fixed-point in nature and that the practical claim "bias is significantly smaller" rests on the empirical results in Section 6.2.
  3. [Table 1] The OPE comparison reports RMSE point estimates without standard errors or confidence intervals across seeds. Given the small number of tasks, the claim of superiority over DualDICE and IHOPE would be strengthened by reporting inter-seed variability.
  4. [Throughout] There are numerous typographical errors and ungrammatical phrases, e.g., "Despite being were adopted" (Section 1), "intital" (Section 4), "behvaior" (Section 6.1), "graident" (Appendix B.2), "staionary" (Section 5), "mdthods" (Appendix G.2), "Largrangian" (Section 6.3), and "Noramlized" (Appendix I.2). A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SemiDICE's policy-correction analysis and CORSDICE's extraction step are proven from explicit objectives, and cost estimates are direct expectation evaluations rather than fitted to target returns.

full rationale

Proposition 4.1 is derived from the first-order condition of the SemiDICE objective (5), so the policy-correction property is a mathematical consequence of the loss, not an assumed input. The extraction step (15)-(19) constructs w(s) by imposing the Bellman flow equation; the equality E_{dD}[w(s)w(a|s)c] = E_{dπ_w}[c] holds by construction, and the reward/cost functions enter only as evaluation functionals, not as fitting labels. Proposition 5.1 gives an independent gradient-equivalence proof for the auxiliary A(s); the citations to Kim et al. [14] and other same-author works are inspirational/contextual and are not load-bearing. The D4RL OPE results and DSRL constrained-RL results are external benchmarks, not constants used to fit the method. The skeptic's concern that Eq. (15b) may be infeasible under finite-sample function approximation is a real correctness/robustness limitation, and the paper's limitation section indeed underplays it, but infeasibility under approximation is not circularity: the exact derivation does not presuppose its own conclusion. No load-bearing step in the claimed derivation chain reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard MDP/Bellman machinery, the f-divergence duality toolkit from prior DICE work, strong duality, and two unverified function-approximation premises: exact policy-correction normalization and convergence of the auxiliary A(s) regression. The only genuinely tuned numeric quantity in CORSDICE is α, plus guidance parameters for D-CORSDICE; λ is optimized, not fitted.

free parameters (3)
  • α (DICE regularization strength) = tuned per task via random search in [0.001, 100.0]
    Controls the f-divergence penalty in the SemiDICE objective (Eq. 5) and therefore the policy correction and the extracted stationary distribution; the reported CORSDICE results depend on this choice.
  • Guidance scale τ (D-CORSDICE) = grid searched over {1.0, 2.0, 4.0}
    Scales the diffusion guidance score in D-CORSDICE; used only in the advanced-function-approximator variant.
  • Number of inference actions (D-CORSDICE) = grid searched over {1, 32, 64, 128}
    Samples multiple actions and selects the best Q-value action; used only in D-CORSDICE.
assumptions (6)
  • standard math Discounted infinite-horizon MDP with γ∈[0,1) and standard Bellman flow equations.
    Used throughout Section 2 and Appendix A; the DICE framework relies on the single-step transposed Bellman recurrence.
  • standard math f is a valid convex f-divergence generator with f(1)=0, and its convex conjugate f*_0 is used to derive closed-form corrections.
    Required for Props 4.1 and 4.2 and the closed-form solutions in Eqs. (12) and (17).
  • domain assumption Strong duality (Slater's condition) holds for the DICE primal and for the extraction optimization (15).
    Invoked to swap max and min in Appendices A and D. For (15) the feasible set is a singleton, so Slater's condition as stated is not obviously applicable.
  • domain assumption Dataset policy πD and optimized policy share the same initial state distribution p0, allowing replacement of (1-γ)p0(s) with dataset-flow terms.
    Appendix B.1 uses this to derive SemiDICE from OptiDICE. If it fails, the derivation of the SemiDICE loss changes, though Prop 4.1 may still hold for the algorithm itself.
  • domain assumption The learned correction w(a|s) is an exact policy correction with Σ_a w(a|s)πD(a|s)=1.
    Needed for Eq. (15b) to be feasible. Not guaranteed under finite-sample function approximation, and the paper does not provide a normalization or projection step.
  • domain assumption The auxiliary network A(s) can represent, and the alternating updates converge to, the conditional expectation E_{a∼πD}[w(a|s)e_μ(s,a)].
    Proposition 5.1 only proves gradient equivalence when A equals this expectation. Realizability and convergence are assumed in the deep RL implementation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semi-gradient DICE for Offline Constrained Reinforcement Learning." pith.science (2026). https://pith.science/paper/TWYRDE7G

@misc{pith2026250608644,
  author       = {Pith},
  title        = {Pith review of: Semi-gradient DICE for Offline Constrained Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TWYRDE7G}},
  note         = {Machine review of arXiv:2506.08644}
}
read the original abstract

Stationary Distribution Correction Estimation (DICE) addresses the mismatch between the stationary distribution induced by a policy and the target distribution required for reliable off-policy evaluation (OPE) and policy optimization. DICE-based offline constrained RL particularly benefits from the flexibility of DICE, as it simultaneously maximizes return while estimating costs in offline settings. However, we have observed that recent approaches designed to enhance the offline RL performance of the DICE framework inadvertently undermine its ability to perform OPE, making them unsuitable for constrained RL scenarios. In this paper, we identify the root cause of this limitation: their reliance on a semi-gradient optimization, which solves a fundamentally different optimization problem and results in failures in cost estimation. Building on these insights, we propose a novel method to enable OPE and constrained RL through semi-gradient DICE. Our method ensures accurate cost estimation and achieves state-of-the-art performance on the offline constrained RL benchmark, DSRL.

Figures

Figures reproduced from arXiv: 2506.08644 by the authors.

Figure 1
Figure 1. Policy Return (Left), Bellman flow constraint violation (Middle), and policy correction constraint violation (Right), averaged over 300 runs. Hyperparameters α (SemiDICE, SQL, XQL, OptiDICE) and β (f-DVL and ODICE) control f-divergence regularization strength (increasing with α, decreasing with β). Performance of XQL for small α values is omitted due to numerical stability. 6 Experiment 6.1 Examining algorithm chara… view at source ↗
Figure 2
Figure 2. Visualization of state-action stationary distribution from OptiDICE. Arrow indicates the [PITH_FULL_IMAGE:figures/full_fig_p028_2.png] view at source ↗
Figure 3
Figure 3. Ablation on the sensitivity of constrained RL algorithms on 3 different cost limits. While [PITH_FULL_IMAGE:figures/full_fig_p029_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Root mean squared error (RMSE) of off-policy evaluation of SemiDICE policy, with [PITH_FULL_IMAGE:figures/full_fig_p029_4.png]
Figure 5
Figure 5. Figure 5: Early learning curves of CORSDICE and baselines on four Safety Gymnasium [ [PITH_FULL_IMAGE:figures/full_fig_p030_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 35 canonical work pages

  1. [1]

    Constrained Markov decision processes

    Eitan Altman. Constrained Markov decision processes. Stochastic Modeling Series. Chapman & Hall/CRC, Philadelphia, PA, March 1999

    work page 1999

  2. [2]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  3. [3]

    JAX: composable transformations of Python+NumPy programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/jax-ml/jax

    2018

  4. [4]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021

    2021

  5. [5]

    Tree-based batch mode reinforcement learning

    Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005

    2005

  6. [6]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    arXiv 2004

  7. [7]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052–2062. PMLR, 2019

    2019

  8. [8]

    Extreme q-learning: Maxent RL without entropy

    Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations,

    work page

Show all 56 references
  1. [9]

    A theory of regularized markov decision processes

    Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning, pages 2160–2169. PMLR, 2019

    2019

  2. [10]

    Bullet-safety-gym: A framework for constrained reinforcement learning

    Sven Gronauer. Bullet-safety-gym: A framework for constrained reinforcement learning. Technical report, mediaTUM, 2022

    2022

  3. [11]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  4. [12]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  5. [13]

    Safety gymnasium: A unified safe reinforce- ment learning benchmark

    Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforce- ment learning benchmark. In Thirty-seventh Conference on Neural Information Processing Systems Datase...

    2023

  6. [14]

    Relaxed stationary distribution correction estimation for improved offline policy optimization

    Woosung Kim, Donghyeon Ki, and Byung-Jun Lee. Relaxed stationary distribution correction estimation for improved offline policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13185–13192, 2024

    2024

  7. [15]

    Roidice: Offline return on investment maximization for efficient decision making

    Woosung Kim, Hayeong Lee, Jongmin Lee, and Byung-Jun Lee. Roidice: Offline return on investment maximization for efficient decision making. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  8. [16]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  9. [17]

    Stabilizing off- policy q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019. 10

    2019

  10. [18]

    Safe policy improvement with baseline bootstrapping

    Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International conference on machine learning, pages 3652–3661. PMLR, 2019

    2019

  11. [19]

    Batch rein- forcement learning with hyperparameter gradients

    Byungjun Lee, Jongmin Lee, Peter Vrancx, Dongho Kim, and Kee-Eung Kim. Batch rein- forcement learning with hyperparameter gradients. In International Conference on Machine Learning, pages 5725–5735. PMLR, 2020

    2020

  12. [20]

    Optidice: Offline policy optimization via stationary distribution correction estimation

    Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120–6130. PMLR, 2021

    2021

  13. [21]

    Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation

    Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2021

    2021

  14. [22]

    Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning

    Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3):3461–3475, 2022

    2022

  15. [23]

    Safe offline reinforcement learning with real-time budget constraints

    Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, and Dong Wang. Safe offline reinforcement learning with real-time budget constraints. In International Conference on Machine Learning, pages 21127–21152. PMLR, 2023

    2023

  16. [24]

    Breaking the curse of horizon: Infinite-horizon off-policy estimation

    Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in neural information processing systems, 31, 2018

    2018

  17. [25]

    Constrained decision transformer for offline safe reinforcement learning

    Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pages 21611–21630. PMLR, 2023

    2023

  18. [26]

    Datasets and benchmarks for offline safe reinforcement learning

    Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, and Ding Zhao. Datasets and benchmarks for offline safe reinforcement learning. Journal of Data-centric Machine Learning Research, 2024

    2024

  19. [27]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  20. [28]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022

    2022

  21. [29]

    Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning

    Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning. arXiv preprint arXiv:2407.20109, 2024

    2024 arXiv

  22. [30]

    Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradient update

    Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradient update. In International Conference on Learning Representations, 2024

    2024

  23. [31]

    Benchmarking safe deep re- inforcement learning in aquatic navigation

    Enrico Marchesini, Davide Corsi, and Alessandro Farinelli. Benchmarking safe deep re- inforcement learning in aquatic navigation. CoRR, abs/2112.10593, 2021. URL https: //arxiv.org/abs/2112.10593

    2021 arXiv

  24. [32]

    Reinforcement learning via fenchel-rockafellar duality.arXiv preprint arXiv:2001.01866, 2020

    Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality.arXiv preprint arXiv:2001.01866, 2020

    2001 arXiv

  25. [33]

    Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections

    Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019. 11

    2019

  26. [34]

    Revisiting the linear- programming framework for offline rl with general function approximation

    Asuman E Ozdaglar, Sarath Pattathil, Jiawei Zhang, and Kaiqing Zhang. Revisiting the linear- programming framework for offline rl with general function approximation. In International Conference on Machine Learning, pages 26769–26791. PMLR, 2023

    2023

  27. [35]

    Constrained offline policy optimization

    Nicholas Polosky, Bruno C Da Silva, Madalina Fiterau, and Jithin Jagannath. Constrained offline policy optimization. In International Conference on Machine Learning, pages 17801–17810. PMLR, 2022

    2022

  28. [36]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part ...

    2015

  29. [37]

    Dual rl: Unification and new methods for reinforcement and imitation learning

    Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. In Sixteenth European Workshop on Reinforcement Learning, 2023

    2023

  30. [38]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  31. [39]

    Responsive safety in reinforcement learning by pid lagrangian methods

    Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. InInternational Conference on Machine Learning, pages 9133–9143. PMLR, 2020

    2020

  32. [40]

    Reinforcement Learning: An Introduction

    Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . Adap- tive Computation and Machine Learning series. Bradford Books, Cambridge, MA, 2 edition, November 2018

    2018

  33. [41]

    Revisiting the minimalist approach to offline reinforcement learning

    Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022

    2022 arXiv

  35. [43]

    Offline rl with no ood actions: In-sample learning via implicit value regularization

    Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2022

    2022

  36. [44]

    An optimal discrimina- tor weighted imitation perspective for reinforcement learning

    Haoran Xu, Shuozhe Li, Harshit Sikchi, Scott Niekum, and Amy Zhang. An optimal discrimina- tor weighted imitation perspective for reinforcement learning. arXiv preprint arXiv:2504.13368, 2025

    2025 arXiv

  37. [45]

    Off-policy evaluation via the regularized lagrangian

    Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551– 6561, 2020

    2020

  38. [46]

    Offline reinforcement learning with realizability and single-policy concentrability

    Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730–2775. PMLR, 2022

    2022

  39. [47]

    Safe and efficient: A primal-dual method for offline convex cmdps under partial data coverage

    Haobo Zhang, Xiyue Peng, Honghao Wei, and Xin Liu. Safe and efficient: A primal-dual method for offline convex cmdps under partial data coverage. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  40. [48]

    Safe offline reinforcement learning with feasibility-guided diffusion model

    Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. arXiv preprint arXiv:2401.10700, 2024. A OptiDICE and COptiDICE In this section, we provide full de...

    2024 arXiv

  41. [50]

    Difficulty in choosing an appropriate η: It is challenging to determine a sufficiently large η that ensures the equal convergence point of orthogonal and full gradient descent

  42. [51]

    Bias in the objective function: ODICE is based on the biased objective (37), where the expectation over the transition probability T appears outside the convex function f ∗ 0 (x). This implies that unless the transition probability is deterministic for all states and actions, ...

  43. [52]

    XQL is an instance of semi- gradient DICE with initial state distribution replacement, where the f-divergence is the reverse KL divergence

    Mismatch in the gradients: In (37), the first term replaces the initial state distribution p0(s) with the dataset distribution dD. This substitution can be interpreted as the application of a semi-gradient method, as demonstrated in the derivation of SemiDICE from (36). Conse-...

  44. [53]

    Convexity of f (x) in its domain: • f (θx + (1 − θ)y) ≤ θf (x) + (1 − θ)f (y), with 0 ≤ θ ≤ 1, ∀x, y∈ dom f

  45. [54]

    We define c = θx + (1 − θ)y

    f (1) = 0 and strict convexity of f (x) at 1 • If θx + (1 − θ)y = 1, f (1) = f (θx + (1 − θ)y) < θf(x) + (1 − θ)f (y), with 0 ≤ θ ≤ 1, ∀x, y∈ dom f We show that g(x) = xf (1/x) also satisfies these properties by using the properties of f. We define c = θx + (1 − θ)y

  46. [55]

    Convexity of g(x) in its domain: • g(θx + (1− θ)y) = cf (1/c) = cf ( θx c 1 x + (1−θ)y c 1 y ) ≤ θg(x) + (1− θ)g(y), with 0 ≤ θ ≤ 1, ∀x, y∈ dom g

  47. [56]

    U (s) + X a π∗(a|s)(Q(s, a) − U (s)) # − X s dD(s)

    g(1) = 0 and strict convexity of g(x) at 1 • g(1) = f (1) = 0 • If c = 1, g(1) = f (1) = f ( θx 1 1 x + (1−θ)y 1 1 y ) < θg(x) + (1 − θ)g(y), with 0 ≤ θ ≤ 1, ∀x, y∈ dom g As f-divergence characterized byg(x) is a valid f-divergence, we now show their reverse relationship. Df (...

  48. [2023]

    URL https://openreview.net/forum?id=SJ0Lde3tRL

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.