Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning

Ali Baheri

REVIEW 5 major objections 5 minor 16 references

Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that composing the optimal Bellman operator with a proximal projection onto a convex constraint set yields an offline RL critic that remains a γ-contraction, has a unique fixed point, and enforces structural constraints…

desk verdict Novel proximal-Bellman idea, but the contraction proof has a norm-mismatch hole and the auction environment's monotonicity premise is false; worth sending to review but not ready as is. read the letter →

arxiv 2506.14058 v1 pith:KDUKCPME submitted 2025-06-16 eess.SY cs.SY

Ali Baheri This is my paper · ORCID

classification eess.SYcs.SY

keywords offlinereinforcementlearningproximalBellmanoperatorconstraint-awarevaluefunctionmonotonicityconstraintsimplicitdifferentiationdifferentiableoptimizationlayerconservativeQ-learning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning can drift into value estimates that violate known structure, such as monotonicity in a bid. This paper proposes to fold such structure into the learning dynamics itself: instead of penalizing violations in the loss, it composes the optimal Bellman operator with a proximal projection onto a convex constraint set. The new operator is claimed to remain a γ-contraction with a unique fixed point, so the usual convergence guarantees of dynamic programming survive, while each Bellman backup now lands on the constrained manifold. On a synthetic advertising auction, the method reports zero monotonicity violations and better return, regret, and sample efficiency than conservative and implicit Q-learning. The contribution is a mechanism, not just a regularizer: a differentiable projection layer with implicit gradients that can be inserted into an offline actor-critic loop.

What carries the argument

The load-bearing object is the composed operator $\Psi_\lambda = \Phi_\lambda \circ T^*$, where $T^*$ is the optimal Bellman operator and $\Phi_\lambda = \mathrm{prox}_{\lambda C}$ is the proximal map of the convex constraint functional $C$ (e.g., the squared violation of monotonicity). The proximal map is firmly non-expansive, which the paper uses to argue that the composition remains a $\gamma$-contraction with a unique fixed point, and the projection step at each Bellman backup keeps the value estimate inside the feasible set. Differentiability comes from the implicit function theorem applied to the fixed-point condition $u_\theta - T^* u_\theta + \lambda \nabla C(u_\theta) = 0$, with the Jacobian solve done by conjugate gradients. A single warm-started proximal step is used in practice, and a dual update adjusts $\lambda$ to balance constraint fidelity against Bellman residual.

What would settle it

A reader could settle the empirical claim by computing $\partial \mathbb{E}[r|s,a]/\partial a = 2\sigma'(2a+0.5x) - c$ for the Bid-Click environment; since this derivative is negative for large bids at $c=0.4$, the environment's optimal value is not provably monotone, and the paper's 'provable monotonicity' premise would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a proximal Bellman operator, defined by $\Phi_\lambda(v) = \arg\min_u \tfrac{1}{2}\lVert u - T^*v\rVert_2^2 + \lambda C(u)$, simultaneously preserves the contraction property of dynamic programming and makes the fixed point obey the convex structural constraint encoded by $C$. Because $\Phi_\lambda$ is the proximal map of a convex, lower-semicontinuous functional, it is firmly non-expansive; composing it with the optimal Bellman operator yields a $\gamma$-contraction with a unique fixed point $v^*_\lambda$ that converges to the unconstrained optimum $v^*$ as $\lambda \to 0$. In implementation, each gradient step redefines the critic output as the solution of this proximal subproblem, and implicit differentiation through the fixed point provides gradients at the cost of one conjugate-gradient solve. The practical consequence claimed is that the critic enforces the prescribed structure exactly at every step, so the actor inherits the guarantee without a separate safety critic. On the Bid-Click auction, the method achieves return $0.851 \pm 0.006$, normalized regret $0.067 \pm 0.004$, and zero monotonicity errors, beating implicit Q-learning and conservative Q-learning.

Load-bearing premise

The whole argument leans on the claim that composing the projection with the Bellman update stays a contraction, yet the two pieces are contractive in different mathematical norms and the paper gives no norm in which the composition is a contraction.

Editorial extensions

If this is right

If the contraction claim holds, any convex structural prior can be embedded in the critic's Bellman update without losing the existence or uniqueness of the fixed point.
Because the projected critic defines the actor's Q-values, the policy inherits the structural guarantees without a separate safety critic.
The advantage widens as data become scarce: at 25% of the original buffer the absolute return lead over implicit Q-learning grows from 6.9% to 12.4%.
The dual update on the penalty weight automatically balances constraint fidelity against Bellman residual, and fixed weak or strong penalties both underperform.
Computational cost stays comparable to implicit Q-learning because differentiating the projection requires only one conjugate-gradient solve per step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's proof of contraction combines a firmly non-expansive map in $L^2$ with a sup-norm contraction; whether the composition is a contraction in a common norm is not established, so the guarantee should be stress-tested on other convex constraint sets before being taken as general.
If the Bid-Click environment's optimal value is not actually monotone under the stated reward, the reported zero violations measure how strongly the projection enforces a possibly incorrect prior, not how well the method recovers true structure; a corrected environment with provable monotonicity would give a cleaner test.
A natural testable extension is to apply the same proximal Bellman operator to non-convex priors (e.g., lower bounds on Q-values or safety constraints) and check numerically whether the fixed point and policy quality survive; the paper's theory only covers convex $C$.
In practice the paper uses a single warm-started proximal step, so 'exact' constraint enforcement is approximate at finite training time; the gap between one-step projection and the exact proximal solution is worth measuring before relying on zero-violation guarantees in deployment.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this one because the construction is actually new. Composing the proximal operator of a convex structural penalty with the optimal Bellman operator, and differentiating through that composition, is not in the cited literature. The paper also does a reasonable thing in its ablations: showing that soft penalties leave feasibility violations while projection removes them, and that warm-started inner solves keep the overhead low. That part is honest and useful.

The problems are in the load-bearing claims. Section III says the composition Ψλ = prox_λC ∘ T* is a γ-contraction because prox is firmly nonexpansive. But prox is firmly nonexpansive in L2, while T* is a contraction in the sup norm. The paper never supplies a norm in which both properties hold. The stress-test note gives explicit counterexamples: a two-state MDP where the L2 contraction constant is γ/√(μ_min) > γ, and a sup-norm counterexample with a projection onto span{x} that has Lipschitz constant 1.5γ. So the fixed-point uniqueness and convergence remarks rest on an unproven step. The Moreau–Yosida envelope remark in the text also seems to confuse the proximal operator with its envelope; that is fixable, but it is not a typo-level issue.

The empirical section has a more basic problem. The paper claims the optimal value is monotone in the bid because ∂r/∂a ≥ 0. The actual expected reward derivative is 2σ'(2a+0.5x) − c, which is negative for large bids and larger costs. So the auction's true optimal value is not provably monotone in the action. On top of that, the constraint functional C(v) = Σ(max{0,−∂v/∂a_i})² differentiates V with respect to the action, but V was defined as a function of state only. That is a type mismatch that the paper never resolves. The algorithm pseudocode uses V̄θ(s',a'), suggesting V is actually a Q-function, but the theory is written for state values. This needs to be straightened out.

The zero-monotonicity-violation result is a consistency check rather than independent evidence, since the projection forces feasibility. The return/regret gains over the baselines are the only real signal, but if the constraint does not match the environment, those gains are hard to interpret.

The equations in Section III and Algorithm 1 are also internally inconsistent: Eq. (5) conflates the fixed-point condition with the first-order optimality of the prox step, Eq. (7) omits a transpose, and the algorithm's gradient for the critic does not match Eq. (6). These are correctable, but as written they block reproduction.

Who is this for? A reader working on provably constrained offline RL will see the idea and immediately ask whether a common-norm contraction proof exists. The paper is not there, but it is worth a serious referee: the construction is novel, the flaws are identifiable, and the author could plausibly fix them. I would send it to review with a request for a corrected norm argument, a corrected environment, and a consistent value/Q formulation. Not ready for acceptance, but not a desk reject either.

Referee Report

5 major / 5 minor

Summary. The paper proposes an offline RL framework in which a convex structural prior is encoded as a functional C and every Bellman backup is mapped through the proximal operator Φλ of C. It claims that the composed operator Ψλ = Φλ ∘ T* remains a γ-contraction, possesses a unique fixed point, and enforces the prior exactly, and that implicit differentiation through the proximal step yields gradients at cost comparable to implicit Q-learning. The experimental section applies the method to a synthetic single-slot auction with a monotonicity constraint and reports zero monotonicity violations, higher return, and lower regret than CQL, IQL, and behavior cloning, together with ablations on the penalty weight, warm-starting, inner iterations, spectral normalization, and critic-side versus actor-side constraint enforcement.

Significance. Should the contraction and fixed-point results hold, the proximal Bellman operator would be an elegant and potentially useful mechanism for embedding hard convex constraints into offline value learning, and the implicit-gradient implementation is a plausible way to make it computationally feasible. The paper also provides a fairly complete ablation study and compares against two standard offline RL baselines. However, the principal theoretical assertion is not proven in the stated generality and appears to be false in concrete examples; the auction experiment is built on a monotonicity claim that does not follow from the reward model; and the zero-violation headline is a direct artifact of the projection. These issues are load-bearing for the paper's central claims, so the current manuscript is not publishable in its present form.

major comments (5)

[Section III, Eq. (2) and following paragraph] The claim that Ψλ = Φλ ∘ T* is a γ-contraction because Φλ is firmly non-expansive is unsupported: Φλ is firmly non-expansive in L2(S), while T* is a contraction in the sup norm, and no norm is supplied in which both properties hold. The composition need not be a contraction in either norm. For C the indicator of the nonnegative cone, the two-state MDP with μ({1})=0.9, μ({2})=0.1, P(s'=2|s)=1, γ=0.9, and v=(0,1), w=(0,-1) gives ||Ψλv − Ψλw||_L2 = 0.9 > ||v − w||_L2 = √0.4. For C the indicator of span{x} in L2[-1,1] together with T*=γI, the composed operator has sup-norm Lipschitz constant 1.5γ > γ. Consequently the fixed-point existence, uniqueness, and convergence statements in this section and in the Convergence remarks do not follow.
[Section III, Eqs. (5)-(7)] The first-order condition in Eq. (5) is written as g(uθ,θ) = uθ − T*uθ + λ∇C(uθ), but Eq. (2) defines the proximal solution as a function of T*v, so the optimality condition should involve T*Vθ (the current value function), not T*uθ. As written, the derivation mixes the proximal solution with the Bellman target; the implicit-differentiation formulas in Eqs. (6) and (7) inherit this inconsistency. The definition of uθ and the exact objective whose gradient is computed must be stated precisely before the algorithm's update rules can be verified.
[Section IV] The environment section asserts that ∂r/∂a ≥ 0 and concludes that the optimal value is monotone in the bid, but with r = σ(2a+0.5x) − ca we have ∂E[r]/∂a = 2σ'(2a+0.5x) − c, which is negative for large a because σ' decays exponentially and c ∈ [0.2,0.4]. Hence the optimal value is not provably monotone in the bid, and the constraint functional C(v) may encode an incorrect prior. In addition, v* is defined as a function of state s only, whereas C(v) contains derivatives ∂v/∂a_i; the notation for the inputs of the value function should be harmonized before this experiment can support the paper's claims.
[Tables I-II and Section IV] The reported zero monotonicity errors for the proposed method are enforced by construction: Eq. (2) projects every Bellman backup onto the monotone cone that the error metric checks. This makes the headline result a consistency check of the optimizer (i.e., that the projection is implemented correctly), not independent evidence that the learned value function is structurally correct. The comparison against baselines should rely on return and regret; the monotonicity column should be reframed as a constraint-satisfaction audit, and the experiment should include a setting where the true value is known to satisfy the constraint.
[Section III, Regularized MDP convergence] The sentence 'Regularized MDP theory guarantees that vλ⋆ → v⋆ point-wise as λ → 0' is not supported by the cited theory in the form used here: regularized MDP results concern entropy or Legendre-type regularization of the policy, not the proximal penalty λC on the value-function space. For an indicator constraint C, the proximal operator is the same for every λ>0, so the limit as λ→0 would not recover the unconstrained optimum. The claim needs a direct proof or a precise statement of the assumptions on C.

minor comments (5)

[Section III, paragraph after Eq. (2)] The sentence 'Because 2 is the Moreau-Yosida envelope of C' is inaccurate: the Moreau-Yosida envelope is the optimal value of the minimization in Eq. (2), whereas Φλ is its argmin (the proximal operator).
[Section III, Eq. (7)] Equation (7) omits the outer transpose in the gradient of the squared Bellman residual; with the standard Jacobian convention the factor multiplying ∂uθ/∂θ should be (uθ − T*uθ)ᵀ(I − γP).
[Algorithm 1] The line '∇θL ← −(uθ−y)z' does not match Eq. (7), and the symbol y is used both for the per-sample target y(s) and as an aggregate vector; please define the notation consistently and align the algorithm with the derivation.
[Related Work] The sentence 'such regularizes temper gradients' contains a typo: 'regularizes' should be 'regularizers'.
[Section IV and Tables I-II] The paper does not specify the number of seeds, the exact policy used for evaluation, or the precise definition of normalized regret; these details are needed to interpret the confidence intervals and to assess statistical significance.

Circularity Check

1 steps flagged · score 4.0 of 10

Zero-violation empirical result is enforced by construction; the contraction theorem is non-circular but unproven.

self definitional [Section IV (Results), Bid-Click environment setup and results paragraph; cf. Eq. (2) and Eq. (3)]
"Because ∂r/∂a≥0, the optimal state value v⋆(s)=max_a E[r|s,a] is monotone non-decreasing in the bid. This monotonicity is enforced in our critic through the constraint functional C(v)=∑_i (max{0,−∂v/∂a_i})^2. ... The constraint-aware agent achieves the highest return and the lowest regret, while entirely eliminating monotonicity violations."

The reported metric 'monotonicity errors' counts violations of exactly the derivative-sign condition that defines the constraint functional C. The proximal Bellman operator in Eq. (2) projects every Bellman backup onto the feasible set of C, and Eq. (3) redefines the critic output as that projection, so C(Vθ)=0 holds at every gradient step by construction. Thus 'eliminating all monotonicity violations' is not an empirical discovery about the environment or a learned policy; it is a tautological consistency check of the projection operator. The return and regret comparisons with baselines are independent, but the zero-violation headline is forced by the algorithm's definition.

full rationale

The paper's central theoretical construction is not circular relative to external facts: the proximal Bellman operator is defined explicitly, and the claim that composition with T* remains a γ-contraction would be a meaningful mathematical assertion if proven. There is no self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction; all references are to standard external literature. The genuine circularity is the empirical demonstration: the monotonicity-error metric checks the same derivative-sign condition that the constraint functional C enforces via projection, so the zero-error result is guaranteed by design. The return, regret, and sample-efficiency comparisons against CQL and IQL provide non-circular empirical content that stands independently of the zero-error claim. I also note, without treating it as circularity, that the environment's monotonicity premise '∂r/∂a≥0' is numerically false for large bids (e.g., a=1, c=0.4), which is a correctness risk rather than a circular reduction. The contraction claim also inherits an unproven norm-mismatch step (firm non-expansiveness in L2 versus T* contraction in sup norm), but that is an unsupported theorem, not an equivalence-by-construction, so it does not raise the circularity score beyond the partial self-definitional step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The framework rests on one adaptive penalty parameter, an unproven norm-compatibility assumption, and a false monotonicity premise; no new physical entities are introduced.

free parameters (2)

Constraint penalty weight λ = λ0=0.1, updated by λ_t = max(λ_{t-1} + η_λ C(u_θ), 0)
Controls the strength of the structural projection; it is adaptively tuned during training rather than derived from first principles, and the headline zero-violation result depends on it being large enough.
Inner proximal-gradient step size (α in Algorithm 1) = not reported; notation collides with entropy temperature α in Eq. (8)
The single warm-started proximal-gradient update in Algorithm 1 uses a step size α that is never specified, so the exact projection behavior is not reproducible.

assumptions (4)

domain assumption The constraint functional C is convex and lower-semicontinuous on L2(S), vanishing exactly when the desired structure holds.
The proximal operator and nonexpansiveness require this; the paper does not verify that the squared violation of a global Lipschitz bound or a monotone cone is well-defined and closed in L2(S).
ad hoc to paper Φλ is firmly non-expansive and T* is a contraction in the same norm, so Ψλ is a γ-contraction.
The paper asserts this in Section III without specifying a common norm; prox is L2-nonexpansive and the Bellman optimality operator is a sup-norm contraction.
domain assumption The Monte Carlo Bellman objective with one warm-started proximal-gradient step is an adequate solution of Eq. (2).
The convergence remarks require the inner optimization error to vanish on average, but Algorithm 1 takes a single proximal-gradient step and no error bound is provided.
ad hoc to paper The auction's optimal value is monotone non-decreasing in the bid because ∂r/∂a≥0.
False from the paper's own reward definition: ∂E[r]/∂a = 2σ'(2a+0.5x) - c, which is negative for large bid values.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/KDUKCPME

@misc{pith2026250614058,
  author       = {Pith},
  title        = {Pith review of: Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDUKCPME}},
  note         = {Machine review of arXiv:2506.14058}
}

abstract

Offline reinforcement learning promises policy improvement from logged interaction data alone, yet state-of-the-art algorithms remain vulnerable to value over-estimation and to violations of domain knowledge such as monotonicity or smoothness. We introduce implicit constraint-aware off-policy correction, a framework that embeds structural priors directly inside every Bellman update. The key idea is to compose the optimal Bellman operator with a proximal projection on a convex constraint set, which produces a new operator that (i) remains a $\gamma$-contraction, (ii) possesses a unique fixed point, and (iii) enforces the prescribed structure exactly. A differentiable optimization layer solves the projection; implicit differentiation supplies gradients for deep function approximators at a cost comparable to implicit Q-learning. On a synthetic Bid-Click auction -- where the true value is provably monotone in the bid -- our method eliminates all monotonicity violations and outperforms conservative Q-learning and implicit Q-learning in return, regret, and sample efficiency.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages

[1]

Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643 , 2020

arXiv 2005
[2]

A survey on offline reinforcement learning: Taxonomy, re- view, and open problems,

R. F. Prudencio, M. R. Maximo, and E. L. Colombini, “A survey on offline reinforcement learning: Taxonomy, re- view, and open problems,” IEEE Transactions on Neural Networks and Learning Systems , 2023

work page 2023
[3]

Beyond uniform sampling: Offline rein- forcement learning with imbalanced datasets,

Z.-W. Hong, A. Kumar, S. Karnik, A. Bhandwaldar, A. Srivastava, J. Pajarinen, R. Laroche, A. Gupta, and P. Agrawal, “Beyond uniform sampling: Offline rein- forcement learning with imbalanced datasets,” Advances in Neural Information Processing Systems , vol. 36, pp. 4985–5009, 2023

work page 2023
[4]

The importance of pessimism in fixed-dataset policy opti- mization,

J. Buckman, C. Gelada, and M. G. Bellemare, “The importance of pessimism in fixed-dataset policy opti- mization,” arXiv preprint arXiv:2009.06799 , 2020

arXiv 2009
[5]

A minimalist approach to offline reinforcement learning,

S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforcement learning,” Advances in neural in- formation processing systems , vol. 34, pp. 20132–20145, 2021

work page 2021
[6]

Safe offline reinforcement learning with feasibility-guided diffusion model,

Y . Zheng, J. Li, D. Yu, Y . Yang, S. E. Li, X. Zhan, and J. Liu, “Safe offline reinforcement learning with feasibility-guided diffusion model,” arXiv preprint arXiv:2401.10700, 2024

arXiv 2024
[7]

Spectral normalization for lipschitz-constrained poli- cies on learning humanoid locomotion,

J. Shin, W. Cha, D. Kim, J. Cha, and J. Park, “Spectral normalization for lipschitz-constrained poli- cies on learning humanoid locomotion,” arXiv preprint arXiv:2504.08246, 2025

work page arXiv 2025
[8]

Off-policy deep reinforcement learning without exploration,

S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in Interna- tional conference on machine learning , pp. 2052–2062, PMLR, 2019

work page 2019

Show all 16 references

[9]

Conser- vative q-learning for offline reinforcement learning,

A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conser- vative q-learning for offline reinforcement learning,” Ad- vances in neural information processing systems , vol. 33, pp. 1179–1191, 2020

2020
[10]

Uncertainty-based offline reinforcement learning with diversified Q-ensemble,

G. An, S. Moon, J.-H. Kim, and H. O. Song, “Uncertainty-based offline reinforcement learning with diversified Q-ensemble,” Advances in neural information processing systems, vol. 34, pp. 7436–7447, 2021

2021
[11]

IDQL: Implicit Q-learning as an actor- critic method with diffusion policies,

P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine, “IDQL: Implicit Q-learning as an actor- critic method with diffusion policies,” arXiv preprint arXiv:2304.10573, 2023

2023 arXiv
[12]

L2c2: Locally lipschitz continuous con- straint towards stable and smooth reinforcement learn- ing,

T. Kobayashi, “L2c2: Locally lipschitz continuous con- straint towards stable and smooth reinforcement learn- ing,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 4032–4039, IEEE, 2022

2022
[13]

Monotonic value function factorisation for deep multi-agent reinforcement learn- ing,

T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learn- ing,” Journal of Machine Learning Research , vol. 21, no. 178, pp. 1–51, 2020

2020
[14]

Optnet: Differentiable opti- mization as a layer in neural networks,

B. Amos and J. Z. Kolter, “Optnet: Differentiable opti- mization as a layer in neural networks,” in International conference on machine learning , pp. 136–145, PMLR, 2017

2017
[15]

A theory of regularized Markov decision processes,

M. Geist, B. Scherrer, and O. Pietquin, “A theory of regularized Markov decision processes,” in International conference on machine learning , pp. 2160–2169, PMLR, 2019

2019
[16]

Offline reinforce- ment learning with implicit Q-learning,

I. Kostrikov, A. Nair, and S. Levine, “Offline reinforce- ment learning with implicit Q-learning,” arXiv preprint arXiv:2110.06169, 2021

2021 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643 , 2020

arXiv 2005

[2] [2]

A survey on offline reinforcement learning: Taxonomy, re- view, and open problems,

R. F. Prudencio, M. R. Maximo, and E. L. Colombini, “A survey on offline reinforcement learning: Taxonomy, re- view, and open problems,” IEEE Transactions on Neural Networks and Learning Systems , 2023

work page 2023

[3] [3]

Beyond uniform sampling: Offline rein- forcement learning with imbalanced datasets,

Z.-W. Hong, A. Kumar, S. Karnik, A. Bhandwaldar, A. Srivastava, J. Pajarinen, R. Laroche, A. Gupta, and P. Agrawal, “Beyond uniform sampling: Offline rein- forcement learning with imbalanced datasets,” Advances in Neural Information Processing Systems , vol. 36, pp. 4985–5009, 2023

work page 2023

[4] [4]

The importance of pessimism in fixed-dataset policy opti- mization,

J. Buckman, C. Gelada, and M. G. Bellemare, “The importance of pessimism in fixed-dataset policy opti- mization,” arXiv preprint arXiv:2009.06799 , 2020

arXiv 2009

[5] [5]

A minimalist approach to offline reinforcement learning,

S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforcement learning,” Advances in neural in- formation processing systems , vol. 34, pp. 20132–20145, 2021

work page 2021

[6] [6]

Safe offline reinforcement learning with feasibility-guided diffusion model,

Y . Zheng, J. Li, D. Yu, Y . Yang, S. E. Li, X. Zhan, and J. Liu, “Safe offline reinforcement learning with feasibility-guided diffusion model,” arXiv preprint arXiv:2401.10700, 2024

arXiv 2024

[7] [7]

Spectral normalization for lipschitz-constrained poli- cies on learning humanoid locomotion,

J. Shin, W. Cha, D. Kim, J. Cha, and J. Park, “Spectral normalization for lipschitz-constrained poli- cies on learning humanoid locomotion,” arXiv preprint arXiv:2504.08246, 2025

work page arXiv 2025

[8] [8]

Off-policy deep reinforcement learning without exploration,

S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in Interna- tional conference on machine learning , pp. 2052–2062, PMLR, 2019

work page 2019