Pith. sign in

REVIEW 4 major objections 5 minor 80 references

ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ACL-QL argues that letting each state-action pair choose its own conservative level keeps offline Q-values in a mild range between the ordinary Q-function and CQL's conservative Q-function, and that this improves D4RL returns.

desk verdict A real idea and strong D4RL results, but the implemented surrogate losses relax the claimed interval guarantee instead of enforcing it. read the letter →

arxiv 2412.16848 v2 pith:46BPFQR6 submitted 2024-12-22 cs.LG cs.AIcs.RO

Kun Wu , Yinuo Zhao , Zhiyuan Xu , Zhengping Che , Chengxiang Yin , Chi Harold Liu , Feiferi Feng , Jian Tang This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords offlinereinforcementlearningQ-learningconservativeadaptiveleveldistributionshiftD4RLbenchmarkweightfunctionsmonotonicityloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning has to stop Q-values from ballooning on unseen actions without squeezing them so much that the policy becomes useless. This paper proposes ACL-QL, which replaces the single fixed conservatism constant used by CQL with two learned weight functions, one for out-of-distribution actions and one for in-dataset actions. The central claim is that these per-transition weights keep the learned Q-function between the ordinary Q-function and CQL's conservative Q-function, and that this mild range is what lets one set of hyperparameters work across dataset types. On D4RL, ACL-QL reports total normalized returns of 241.6, 337.9, and 311.5 on HalfCheetah, Hopper, and Walker, above CQL's 199.7, 268.2, and 261.0.

What carries the argument

The engine is a pair of adaptive weight functions $w_\mu(s,a)$ and $w_{\pi_\beta}(s,a)$, implemented as a single neural network with two outputs, trained against three objectives: a monotonicity loss that aligns weight differences with transition-quality differences, surrogate hinge losses that encode the two sandwich inequalities, and a positivity regularizer. The transition-quality pseudo-label $m(s,a)$ combines normalized Monte Carlo returns and immediate rewards for in-dataset pairs, and uses distance to the in-dataset action for OOD pairs. The paper also swaps probabilities for $1+\ln\mu$ and $1+\ln\pi_\beta$ in the surrogate hinge losses to avoid arithmetic underflow, invoking $\ln x \le x-1$.

What would settle it

Run ACL-QL on HalfCheetah-medium-v2 while logging, for every dataset transition, the argument inside the hinge losses in Eqs. (20) and (21); if a non-negligible fraction of transitions has the wrong sign, so the learned Q lies outside the claimed sandwich between ordinary Q and CQL Q, the central claim fails. A complementary check is to replace the logarithmic surrogate with the exact probabilities and see whether the reported performance gap vanishes.

Watch

Extended reading notes

Core claim

The paper's discovery is that the conservative level of Q-learning can be made pointwise adaptive while remaining theoretically anchored. For any distribution $\mu$ supported inside the behavioral policy, the gap between ACL-QL's Q-function and the ordinary Q-function is $(w_\mu \mu - w_{\pi_\beta}\pi_\beta)/\pi_\beta$, and the gap to CQL's Q-function is $((\alpha-w_\mu)\mu - (\alpha-w_{\pi_\beta})\pi_\beta)/\pi_\beta$. Forcing the first gap positive and the second positive sandwiches the learned Q-values between ordinary Q and CQL Q. ACL-QL sets the desired gaps from a normalized transition-quality score $m(s,a)$ and trains the weights with monotonicity, surrogate hinge, and positivity losses, with CQL as the special case where both weights equal the constant $\alpha$.

Load-bearing premise

The paper assumes that replacing the action probabilities in the conservative-level losses by the logarithmic expressions $1+\ln\mu$ and $1+\ln\pi_\beta$ leaves the claimed sandwich condition intact, even though those expressions can become negative.

Editorial extensions

If this is right

  • If the sandwich claim holds, offline RL becomes less sensitive to the choice of conservatism hyperparameter; ACL-QL uses the same $\alpha$ across tasks in one environment and still outperforms CQL at every fixed $\alpha$ on the MuJoCo suite.
  • The learned Q-values stay in a mild range, so selecting a checkpoint by the dataset's average Q-value does not drift toward overestimation; the reported Q-values sit between SAC's ordinary Q-values and CQL-10's values.
  • Because CQL is the constant-weight special case, ACL-QL's improvements over CQL are direct consequences of letting conservative levels vary per transition rather than changing the underlying pessimism principle.
  • The framework allows custom weight functions beyond neural networks, so hand-designed rules could be substituted where learning weights is undesirable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural follow-up is to test whether the logarithmic surrogate, not the sandwich theory, is what drives the gains: replacing Eqs. (20)-(21) with the exact hinge losses (16)-(17) would separate the two effects, since the paper does not prove the surrogate preserves the inequalities.
  • The same sandwich construction could be layered onto other conservative offline algorithms that use a KL or distributional penalty, because the identities depend only on the CQL-style regularizer and not on the actor-critic backbone.
  • The transition-quality pseudo-label is a placeholder: upper-envelope estimates, learned uncertainty, or learned Q-based scores could replace it, and making $\lambda$ in Eq. (13) depend on dataset quality might remove the last tuned hyperparameter.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ACL-QL, an offline reinforcement learning algorithm that aims to keep the learned Q-function in a mild range between the ordinary Q-function and CQL's conservative Q-function. It introduces two learnable weight functions, w_mu and w_pi_beta, that modulate the conservative penalty per state-action pair, and it derives conditions on these weights under simplified assumptions. The practical algorithm uses transition-quality pseudo-labels, a monotonicity loss, and surrogate hinge losses to train the weights, Q-function, and policy. Experiments on D4RL Gym-MuJoCo, Adroit, and Franka Kitchen report state-of-the-art or competitive results, and ablations study the effect of each loss component and the transition-quality measurements.

Significance. The framework is well motivated, and the idea of per-transition conservative levels is a useful direction for reducing sensitivity to conservatism hyperparameters. The paper's main theoretical contribution, however, is not established: the implemented surrogate losses do not enforce the derived conditions, and the closed-form comparisons to CQL assume an idealized objective. If these gaps were closed, the method would offer a principled way to control conservatism in a fine-grained manner. The experimental study is extensive and includes a useful comparison against CQL with different alpha values, and the ablations are informative. The aggregate Q-value statistics in Table VII are suggestive but do not verify the pointwise 'mild range' claim. The flexibility of the proposed framework and the breadth of the D4RL evaluation are notable strengths.

major comments (4)
  1. [Section V-C, Eqs. (20)-(21)] The surrogate losses do not enforce the conditions in Eqs. (16)-(17). Lemma V.1 gives ln x <= x - 1, so 1 + ln x <= x, with equality only at x = 1; replacing mu and pi_beta by 1 + ln(mu) and 1 + ln(pi_beta) changes the inequality in an uncontrolled way because the same quantity appears with both positive and negative signs. Concretely, with mu = 0.8, pi_beta = 0.5, w_mu = 1.1, w_pi_beta = 1.0, and d_ord = 1.0, Eq. (20) equals 0 while Eq. (16) equals 0.12 > 0, so a zero surrogate loss does not imply the original condition; the same failure applies to Eq. (21) and d_cql. Since Eq. (24) is a sum of nonnegative hinge terms, the optimizer can drive these losses to zero even when the conservative-level conditions of Propositions IV.1 and IV.3 are violated. The theoretical guarantees are therefore not connected to the implemented objective, and the authors need either a stable version of the original probabilities or a proof of the intended implication.
  2. [Section IV, Propositions IV.1 and IV.3] The closed-form expressions for the CQL Q-function and for d_cql assume a simplified objective that omits the max over mu, the regularizer R(mu), and the expectation over states; the proof in the appendix (Eq. (47)) starts from such a simplified problem. Consequently, the statement that Q^pi_CQL = Q^pi - alpha(mu - pi_beta)/pi_beta does not describe the actual CQL solution, and the pointwise interval Q_CQL <= Q_hat <= Q^pi is not established for the algorithms being compared. The authors should either state the precise idealizations under which Eqs. (5) and (10) hold or present them as design heuristics rather than formal guarantees.
  3. [Section V-B, Eq. (15)] The monotonicity loss in Eq. (15) enforces a relationship between softmax outputs over pairs in a batch, whereas Definition IV.1 specifies monotonicity with respect to the optimal Q-function for actions in the same state. The replacement of Q* by the transition-quality measurement m(s,a) and the use of cross-state pairs are not derived from the definition, so the loss is a heuristic. If the monotonicity property is central to the adaptive conservative-level mechanism, the authors should justify why the batch-level softmax equality approximates the per-state condition.
  4. [Section VI-B, Table VII and Figure 4] The empirical support for the 'mild range' claim reports only average Q-values over the dataset (Table VII) and a sampled visualization (Figure 4). An average can lie between the SAC and CQL averages even if the pointwise interval is violated for most transitions. To substantiate the claim that ACL-QL Q-values lie between the ordinary and CQL Q-values pointwise, the authors should report distributional evidence such as quantiles or per-transition Q differences.
minor comments (5)
  1. [Appendix, Proof of Proposition IV.2] The heading reads 'Proof of Proposition VI.2' but should be 'Proof of Proposition IV.2'.
  2. [Proposition IV.2, Eq. (9)] The denominator in the definition of err(s,a) is written as (1 - gamma sqrt(|D|)), which appears to be a typesetting error; the appendix Eq. (30) gives the correct form (1 - gamma) sqrt(|D|).
  3. [Proposition IV.2 and Appendix] The proof uses the symbol g(s,a) in Eqs. (38) and (46) where the main text defines h(s,a); please unify the notation.
  4. [Section VI-B, Table II] The baseline results are taken from different original papers, and standard deviations are not reported for the baselines, making it difficult to assess whether the reported improvements are statistically significant; this limitation should be acknowledged.
  5. [Section VI-A and Table I] The text states that the same hyperparameters are used for all tasks on the same environment, but alpha is set to 10, 20, or 1 depending on the environment; please clarify whether alpha is considered a per-environment hyperparameter.

Circularity Check

1 steps flagged · score 3.0 of 10

The 'mild range' validation is self-referential: the interval is encoded in the hinge losses, so Table VII checks that the objective took effect rather than independently predicting the interval; the D4RL benchmark comparisons are external and not circular.

  1. self definitional [Section VI-C, Table VII, with Eqs. (16)-(17) and Propositions IV.1, IV.3]
    "The Q-values of ACL-QL are between SAC and CQL-10 and are closer to the Q-values of online SAC, showing that the losses Lord, Lcql in Equations (16) and (17) take effect."

    The interval 'between SAC and CQL-10' is not an independent prediction. Equations (16)-(17) are hinge losses constructed to enforce d_ord(s,a) >= 0 and d_cql(s,a) >= 0, and by Eqs. (5) and (10) those inequalities are precisely the condition that the learned Q-function lies between the ordinary Q-function and the CQL Q-function. Thus the reported average Q-values only confirm that the optimized weights approximately satisfy the constraints the losses were designed to impose. The mild-range property is an input encoded in the training objective, and Table VII is a sanity check of optimization, not an external confirmation of the framework. The D4RL score comparisons retain independent content because they benchmark against external algorithms.

full rationale

The paper's main benchmark claims are not circular: ACL-QL is compared against external baselines on D4RL, and the reported returns are independent of the paper's own construction. No load-bearing self-citation chain was found; the only self-citation visible in the references is prior autonomous-driving work cited as background, not used to justify the ACL-QL mechanism. The CQL closed-form anchor and concentration bounds are imported from external prior work (Kumar et al.), so no uniqueness theorem is being imported from the authors' own earlier papers. The one genuine circularity is the validation of the 'mild range' property: the hinge losses in Eqs. (16)-(17) are derived from Propositions IV.1 and IV.3 to enforce exactly d_ord >= 0 and d_cql >= 0, which by definition place the Q-function between the ordinary and CQL Q-functions. Therefore Table VII's observation that ACL-QL Q-values lie between SAC and CQL-10 is a check that the losses took effect, not a prediction of an independent phenomenon. This is a mild self-referential validation, not a statistically forced benchmark result. Separately, there is a correctness risk that is not circularity: the surrogate losses Eqs. (20)-(21) replace probabilities with 1+ln(p), and Lemma V.1 gives 1+ln(p) <= p, with the substituted quantities possibly negative; minimizing the surrogates can therefore leave the exact conditions of Eqs. (16)-(17) violated. That is a derivation gap affecting whether the implemented objective enforces the claimed interval guarantee, but it is not an equivalence-by-construction circularity, so it does not by itself raise the circularity score.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The algorithm has several hand-set free parameters and its theoretical guarantees depend on at least two unproved simplifications: a closed-form CQL anchor and a log-substitution that is assumed to preserve hinge inequalities. These choices do not by themselves invalidate the empirical D4RL results, but they mean the paper's theoretical contribution is weaker than stated.

free parameters (4)
  • CQL anchor scale alpha = 10 for halfcheetah and walker, 20 for hopper and kitchen, 1/10/20 for Adroit tasks
    Hand-set per environment. It defines the CQL anchor in Proposition IV.3 and enters the Lcql surrogate in Eq. (21), so it directly controls the claimed mild range.
  • lambda in transition quality measurement = 0.5
    Selected after ablation (Table IX): 'setting it to 0.5 is a simple and good choice'. Not derived from theory.
  • dord and dcql gap functions = dord=(1-m)*rmax, dcql=m*rmax
    Chosen because rmean was too small and Qup too large to distinguish transitions. This is a hand-designed mapping from the quality pseudo-label to Q-gap targets in Eqs. (18)-(19).
  • transition quality pseudo-label m(s,a) = lambda*g_norm+(1-lambda)*r_norm; OOD uses T(m,L2)
    A proxy for the optimal Q* used in Definition IV.1. It is a heuristic normalization of Monte Carlo returns, single-step rewards, and L2 distances, and the monotonicity loss is trained against it.
assumptions (6)
  • domain assumption Support of mu is contained in support of pi_beta so mu/pi_beta is finite
    Propositions IV.1 and IV.3 assume supp(mu) subset supp(pi_beta). In the practical algorithm pi is a Gaussian policy that can sample actions outside the finite dataset support, so the ratios used in the derivation are not guaranteed to exist.
  • ad hoc to paper CQL Q-function has closed form Q_CQL = Q_pi - alpha(mu-pi_beta)/pi_beta
    Proposition IV.3 states this as given, and Appendix I derives it from an objective without CQL's R(mu) and max over mu. The full CQL objective in Eq. (3) does not generally yield this simple pointwise form, so the anchor is an unproved simplification.
  • ad hoc to paper Replacing probabilities by 1+ln(probability) preserves the conservative-level inequalities
    Eqs. (20)-(21) are presented as the result of Lemma V.1, but the lemma only bounds ln x by x-1. No proof shows that the hinge losses with 1+ln(mu) and 1+ln(pi_beta) enforce the original conditions; the substituted values can be negative.
  • standard math Bellman concentration bound used in Proposition IV.2
    The inequality |Bhat_pi Q - B_pi Q| <= C Rmax/((1-gamma) sqrt(|D|)) is imported from CQL [18] and Auer/Osband concentration results. The paper does not rederive it for its own objective.
  • domain assumption Behavioral cloning gives a good estimate of pi_beta
    pi_beta is trained by Eq. (22) and its log-density appears in the surrogate losses. Errors in this estimate change the effective weights and the claimed bounds.
  • domain assumption Transition quality m(s,a) preserves the ordering of the optimal Q-function
    Definition IV.1 requires monotonicity in Q*. Section V replaces Q* by m(s,a) computed from Monte Carlo returns, rewards, and L2 distance. The paper assumes this proxy preserves enough ordering for the monotonicity loss to be meaningful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/46BPFQR6

@misc{pith2026241216848,
  author       = {Pith},
  title        = {Pith review of: ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46BPFQR6}},
  note         = {Machine review of arXiv:2412.16848}
}
read the original abstract

Offline Reinforcement Learning (RL), which operates solely on static datasets without further interactions with the environment, provides an appealing alternative to learning a safe and promising control policy. The prevailing methods typically learn a conservative policy to mitigate the problem of Q-value overestimation, but it is prone to overdo it, leading to an overly conservative policy. Moreover, they optimize all samples equally with fixed constraints, lacking the nuanced ability to control conservative levels in a fine-grained manner. Consequently, this limitation results in a performance decline. To address the above two challenges in a united way, we propose a framework, Adaptive Conservative Level in Q-Learning (ACL-QL), which limits the Q-values in a mild range and enables adaptive control on the conservative level over each state-action pair, i.e., lifting the Q-values more for good transitions and less for bad transitions. We theoretically analyze the conditions under which the conservative level of the learned Q-function can be limited in a mild range and how to optimize each transition adaptively. Motivated by the theoretical analysis, we propose a novel algorithm, ACL-QL, which uses two learnable adaptive weight functions to control the conservative level over each transition. Subsequently, we design a monotonicity loss and surrogate losses to train the adaptive weight functions, Q-function, and policy network alternatively. We evaluate ACL-QL on the commonly used D4RL benchmark and conduct extensive ablation studies to illustrate the effectiveness and state-of-the-art performance compared to existing offline DRL baselines.

Figures

Figures reproduced from arXiv: 2412.16848 by the authors.

Figure 1
Figure 1. Performance gaps of CQL [18] with different conser [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. ACL-QL can limit the Q-values into the interval [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. To eliminate the impact of gaps in results due to the quality of the dataset itself and fairly show the comparison for [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of the Q-values Qµ(s, a), where the actions a are from the learned policy π or dataset D, on HalfCheetah￾medium-expert-v2 and HalfCheetah-random datasets. The Q-values are sorted in ascending order according to relative transition quality measurement m(s,…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 61 canonical work pages

  1. [1]

    Mastering the game of go with deep neural networks and tree search,

    D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanctot et al., “Mastering the game of go with deep neural networks and tree search,” Nature, 2016

    work page 2016

  2. [2]

    Self-paced prioritized curriculum learning with coverage penalty in deep reinforcement learning,

    Z. Ren, D. Dong, H. Li, and C. Chen, “Self-paced prioritized curriculum learning with coverage penalty in deep reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems , 2018

    work page 2018

  3. [3]

    Feature control as intrinsic motivation for hierarchical reinforcement learning,

    N. Dilokthanakul, C. Kaplanis, N. Pawlowski, and M. Shanahan, “Feature control as intrinsic motivation for hierarchical reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, 2019

    work page 2019

  4. [4]

    Scalable deep reinforcement learning for vision-based robotic manipulation,

    D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V . Vanhoucke et al. , “Scalable deep reinforcement learning for vision-based robotic manipulation,” in Conference on Robot Learning . PMLR, 2018

    work page 2018

  5. [5]

    Solving rubik’s cube with a robot hand,

    I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas et al., “Solving rubik’s cube with a robot hand,” arXiv preprint arXiv:1910.07113, 2019

    arXiv 1910

  6. [6]

    Rein- forcement learning for mobile robotics exploration: A survey,

    L. C. Garaffa, M. Basso, A. A. Konzen, and E. P. de Freitas, “Rein- forcement learning for mobile robotics exploration: A survey,” IEEE Transactions on Neural Networks and Learning Systems , 2021

    work page 2021

  7. [7]

    Deductive reinforcement learning for visual autonomous urban driving navigation,

    C. Huang, R. Zhang, M. Ouyang, P. Wei, J. Lin, J. Su, and L. Lin, “Deductive reinforcement learning for visual autonomous urban driving navigation,” IEEE Transactions on Neural Networks and Learning Systems, 2021

    work page 2021

  8. [8]

    Deep reinforcement learning on autonomous driving policy with auxiliary critic network,

    Y . Wu, S. Liao, X. Liu, Z. Li, and R. Lu, “Deep reinforcement learning on autonomous driving policy with auxiliary critic network,” IEEE Transactions on Neural Networks and Learning Systems , 2021

    work page 2021

Show all 80 references
  1. [9]

    Cadre: A cascade deep reinforcement learning framework for vision- based autonomous urban driving,

    Y . Zhao, K. Wu, Z. Xu, Z. Che, Q. Lu, J. Tang, and C. H. Liu, “Cadre: A cascade deep reinforcement learning framework for vision- based autonomous urban driving,” Proceedings of the AAAI Conference on Artificial Intelligence , 2022

    2022

  2. [10]

    Batch reinforcement learning,

    S. Lange, T. Gabel, and M. Riedmiller, “Batch reinforcement learning,” in Reinforcement learning. Springer, 2012

    2012

  3. [11]

    A survey on offline reinforcement learning: Taxonomy, review, and open problems,

    R. F. Prudencio, M. R. Maximo, and E. L. Colombini, “A survey on offline reinforcement learning: Taxonomy, review, and open problems,” IEEE Transactions on Neural Networks and Learning Systems , 2023

    2023

  4. [12]

    Stabilizing off- policy q-learning via bootstrapping error reduction,

    A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine, “Stabilizing off- policy q-learning via bootstrapping error reduction,” Advances in Neural Information Processing Systems , 2019

    2019

  5. [13]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  6. [14]

    Way off-policy batch deep reinforce- ment learning of implicit human preferences in dialog,

    N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard, “Way off-policy batch deep reinforce- ment learning of implicit human preferences in dialog,” arXiv preprint arXiv:1907.00456, 2019

    1907 arXiv

  7. [15]

    Behavior regularized offline reinforcement learning,

    Y . Wu, G. Tucker, and O. Nachum, “Behavior regularized offline reinforcement learning,” arXiv preprint arXiv:1911.11361 , 2019

    1911 arXiv

  8. [16]

    Keep doing what worked: Behavior modelling priors for offline reinforcement learning,

    N. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Ne- unert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller, “Keep doing what worked: Behavior modelling priors for offline reinforcement learning,” in International Conference on Learning Representations , 2020

    2020

  9. [17]

    Off-policy deep reinforcement learning without exploration,

    S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in International Conference on Machine Learning. PMLR, 2019

    2019

  10. [18]

    Conservative q-learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,” Advances in Neural Information Processing Systems, 2020

    2020

  11. [19]

    Conservative data sharing for multi-task offline reinforcement learning,

    T. Yu, A. Kumar, Y . Chebotar, K. Hausman, S. Levine, and C. Finn, “Conservative data sharing for multi-task offline reinforcement learning,” Advances in Neural Information Processing Systems , 2021

    2021

  12. [20]

    Combo: Conservative offline model-based policy optimization,

    T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn, “Combo: Conservative offline model-based policy optimization,” Ad- vances in Neural Information Processing Systems , 2021

    2021

  13. [21]

    Conservative offline distribu- tional reinforcement learning,

    Y . Ma, D. Jayaraman, and O. Bastani, “Conservative offline distribu- tional reinforcement learning,” Advances in Neural Information Pro- cessing Systems, 2021

    2021

  14. [22]

    Bail: Best-action imitation learning for batch deep reinforcement learning,

    X. Chen, Z. Zhou, Z. Wang, C. Wang, Y . Wu, and K. Ross, “Bail: Best-action imitation learning for batch deep reinforcement learning,” Advances in Neural Information Processing Systems , 2020

    2020

  15. [23]

    Critic regularized regression,

    Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess et al. , “Critic regularized regression,” Advances in Neural Information Processing Systems, 2020

    2020

  16. [24]

    Curriculum offline imitating learning,

    M. Liu, H. Zhao, Z. Yang, J. Shen, W. Zhang, L. Zhao, and T.-Y . Liu, “Curriculum offline imitating learning,” Advances in Neural Information Processing Systems, 2021

    2021

  17. [25]

    Issues in using function approximation for reinforcement learning,

    S. Thrun and A. Schwartz, “Issues in using function approximation for reinforcement learning,” in Proceedings of the 1993 connectionist models summer school . Psychology Press, 2014

    1993

  18. [26]

    Diagnosing bottlenecks in deep q-learning algorithms,

    J. Fu, A. Kumar, M. Soh, and S. Levine, “Diagnosing bottlenecks in deep q-learning algorithms,” in International Conference on Machine Learning. PMLR, 2019

    2019

  19. [27]

    Towards characterizing divergence in deep q-learning,

    J. Achiam, E. Knight, and P. Abbeel, “Towards characterizing divergence in deep q-learning,” arXiv preprint arXiv:1903.08894 , 2019

    1903 arXiv

  20. [28]

    Non-delusional q-learning and value-iteration,

    T. Lu, D. Schuurmans, and C. Boutilier, “Non-delusional q-learning and value-iteration,” Advances in neural information processing systems, 2018

    2018

  21. [29]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in International Conference on Machine Learning. PMLR, 2018

    2018

  22. [30]

    Addressing function approxi- mation error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approxi- mation error in actor-critic methods,” in International conference on machine learning. PMLR, 2018

    2018

  23. [31]

    Adaptive qq -learning for data-based optimal output regulation with experience replay,

    B. Luo, Y . Yang, and D. Liu, “Adaptive qq -learning for data-based optimal output regulation with experience replay,” IEEE Transactions on Cybernetics, 2018

    2018

  24. [32]

    Prioritized experience replay,

    T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” arXiv preprint arXiv:1511.05952 , 2015

    2015 arXiv

  25. [33]

    Playing atari with deep reinforcement learn- ing,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wier- stra, and M. Riedmiller, “Playing atari with deep reinforcement learn- ing,” arXiv preprint arXiv:1312.5602 , 2013

    2013 arXiv

  26. [34]

    Distributed prioritized experience replay,

    D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver, “Distributed prioritized experience replay,” in International Conference on Learning Representations , 2018

    2018

  27. [35]

    An equivalence between loss functions and non-uniform sampling in experience replay,

    S. Fujimoto, D. Meger, and D. Precup, “An equivalence between loss functions and non-uniform sampling in experience replay,” Advances in neural information processing systems , 2020

    2020

  28. [36]

    Model-augmented prioritized experience replay,

    Y . Oh, J. Shin, E. Yang, and S. J. Hwang, “Model-augmented prioritized experience replay,” in International Conference on Learning Represen- tations, 2021

    2021

  29. [37]

    Demysti- fying reinforcement learning in time-varying systems,

    P. Hamadanian, M. Schwarzkopf, S. Sen, and M. Alizadeh, “Demysti- fying reinforcement learning in time-varying systems,” arXiv preprint arXiv:2201.05560, 2022

    2022 arXiv

  30. [38]

    Towards continual reinforcement learning: A review and perspectives,

    K. Khetarpal, M. Riemer, I. Rish, and D. Precup, “Towards continual reinforcement learning: A review and perspectives,” The Journal of Artificial Intelligence Research , 2022. 14

    2022

  31. [39]

    Pseudo- rehearsal: Achieving deep reinforcement learning without catastrophic forgetting,

    C. Atkinson, B. McCane, L. Szymanski, and A. Robins, “Pseudo- rehearsal: Achieving deep reinforcement learning without catastrophic forgetting,” Neurocomputing, 2021

    2021

  32. [40]

    Under- standing the impact of entropy on policy optimization,

    Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans, “Under- standing the impact of entropy on policy optimization,” in International conference on machine learning . PMLR, 2019

    2019

  33. [41]

    Offline reinforcement learning with implicit q-learning,

    I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” in International Conference on Learning Representations, 2022

    2022

  34. [42]

    Monotonic quantile network for worst-case offline reinforcement learning,

    C. Bai, T. Xiao, Z. Zhu, L. Wang, F. Zhou, A. Garg, B. He, P. Liu, and Z. Wang, “Monotonic quantile network for worst-case offline reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, 2022

    2022

  35. [43]

    Mild policy evaluation for offline actor–critic,

    L. Huang, B. Dong, J. Lu, and W. Zhang, “Mild policy evaluation for offline actor–critic,” IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  36. [44]

    Plas: Latent action space for offline reinforcement learning,

    W. Zhou, S. Bajracharya, and D. Held, “Plas: Latent action space for offline reinforcement learning,” in Conference on Robot Learning, 2020

    2020

  37. [45]

    Offline re- inforcement learning with fisher divergence critic regularization,

    I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum, “Offline re- inforcement learning with fisher divergence critic regularization,” in International Conference on Machine Learning . PMLR, 2021

    2021

  38. [46]

    A minimalist approach to offline reinforce- ment learning,

    S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforce- ment learning,” Advances in Neural Information Processing Systems , 2021

    2021

  39. [47]

    An emphatic approach to the problem of off-policy temporal-difference learning,

    R. S. Sutton, A. R. Mahmood, and M. White, “An emphatic approach to the problem of off-policy temporal-difference learning,” The Journal of Machine Learning Research , 2016

    2016

  40. [48]

    Algaedice: Policy gradient from arbitrary experience,

    O. Nachum, B. Dai, I. Kostrikov, Y . Chow, L. Li, and D. Schuurmans, “Algaedice: Policy gradient from arbitrary experience,” arXiv preprint arXiv:1912.02074, 2019

    1912 arXiv

  41. [49]

    Accelerating on- line reinforcement learning with offline datasets,

    A. Nair, M. Dalal, A. Gupta, and S. Levine, “Accelerating on- line reinforcement learning with offline datasets,” arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  42. [50]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine, “Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,”arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  43. [51]

    Safe policy improvement with an estimated baseline policy,

    T. D. Simão, R. Laroche, and R. Tachet des Combes, “Safe policy improvement with an estimated baseline policy,” in Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, 2020

    2020

  44. [52]

    An optimistic perspec- tive on offline reinforcement learning,

    R. Agarwal, D. Schuurmans, and M. Norouzi, “An optimistic perspec- tive on offline reinforcement learning,” in International Conference on Machine Learning. PMLR, 2020

    2020

  45. [53]

    S4rl: Surprisingly simple self- supervision for offline reinforcement learning in robotics,

    S. Sinha, A. Mandlekar, and A. Garg, “S4rl: Surprisingly simple self- supervision for offline reinforcement learning in robotics,” in Conference on Robot Learning . PMLR, 2022

    2022

  46. [54]

    Offline rl without off-policy evaluation,

    D. Brandfonbrener, W. Whitney, R. Ranganath, and J. Bruna, “Offline rl without off-policy evaluation,” Advances in Neural Information Pro- cessing Systems, 2021

    2021

  47. [55]

    Offline reinforcement learning as one big sequence modeling problem,

    M. Janner, Q. Li, and S. Levine, “Offline reinforcement learning as one big sequence modeling problem,” Advances in neural information processing systems, 2021

    2021

  48. [56]

    Decision transformer: Reinforcement learning via sequence modeling,

    L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,” Advances in Neural Information Processing Systems, 2021

    2021

  49. [57]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems , 2017

    2017

  50. [58]

    Hundreds guide millions: Adaptive offline reinforcement learning with expert guidance,

    Q. Yang, S. Wang, Q. Zhang, G. Huang, and S. Song, “Hundreds guide millions: Adaptive offline reinforcement learning with expert guidance,” IEEE Transactions on Neural Networks and Learning Systems , 2023

    2023

  51. [59]

    When to trust your model: Model-based policy optimization,

    M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,” Advances in Neural Information Processing Systems, 2019

    2019

  52. [60]

    Morel: Model-based offline reinforcement learning,

    R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, “Morel: Model-based offline reinforcement learning,” Advances in Neural Infor- mation Processing Systems , 2020

    2020

  53. [61]

    Mopo: Model-based offline policy optimization,

    T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y . Zou, S. Levine, C. Finn, and T. Ma, “Mopo: Model-based offline policy optimization,” Advances in Neural Information Processing Systems , 2020

    2020

  54. [62]

    Deployment-efficient reinforcement learning via model-based offline optimization,

    T. Matsushima, H. Furuta, Y . Matsuo, O. Nachum, and S. S. Gu, “Deployment-efficient reinforcement learning via model-based offline optimization,” in International Conference on Learning Representations, 2021

    2021

  55. [63]

    Agnostic system identification for model- based reinforcement learning,

    S. Ross and J. A. Bagnell, “Agnostic system identification for model- based reinforcement learning,” in International Conference on Machine Learning, 2012

    2012

  56. [64]

    Synthesis and stabilization of complex behaviors through online trajectory optimization,

    Y . Tassa, T. Erez, and E. Todorov, “Synthesis and stabilization of complex behaviors through online trajectory optimization,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2012

    2012

  57. [65]

    A survey of monte carlo tree search methods,

    C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton, “A survey of monte carlo tree search methods,” IEEE Transactions on Computational Intelligence and AI in Games , 2012

    2012

  58. [66]

    Near-optimal regret bounds for reinforcement learning,

    P. Auer, T. Jaksch, and R. Ortner, “Near-optimal regret bounds for reinforcement learning,” Advances in Neural Information Processing Systems, 2008

    2008

  59. [67]

    Deep exploration via bootstrapped dqn,

    I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped dqn,” Advances in Neural Information Processing Systems, 2016

    2016

  60. [68]

    Semi-supervised classification with graph convolutional networks,

    T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Rep- resentations, 2017

    2017

  61. [69]

    Deep learning using rectified linear units (relu),

    A. F. Agarap, “Deep learning using rectified linear units (relu),” arXiv preprint arXiv:1803.08375, 2018

    2018 arXiv

  62. [70]

    D4rl: Datasets for deep data-driven reinforcement learning,

    J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine, “D4rl: Datasets for deep data-driven reinforcement learning,” 2020

    2020

  63. [71]

    Mujoco: A physics engine for model-based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2012

    2012

  64. [72]

    Openai gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  65. [73]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,

    A. Rajeswaran, V . Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” in Proceedings of Robotics: Science and Systems (RSS) , 2018

    2018

  66. [74]

    Alvinn: An autonomous land vehicle in a neural network,

    D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” Advances in neural information processing systems , 1988

    1988

  67. [75]

    Reproducibil- ity of benchmarked deep reinforcement learning tasks for continuous control,

    R. Islam, P. Henderson, M. Gomrokchi, and D. Precup, “Reproducibil- ity of benchmarked deep reinforcement learning tasks for continuous control,” arXiv preprint arXiv:1708.04133 , 2017

    2017 arXiv

  68. [76]

    A workflow for offline model-free robotic reinforcement learning,

    A. Kumar, A. Singh, S. Tian, C. Finn, and S. Levine, “A workflow for offline model-free robotic reinforcement learning,” arXiv preprint arXiv:2109.10813, 2021

    2021 arXiv

  69. [77]

    Relay pol- icy learning: Solving long horizon tasks via imitation and reinforcement learning,

    A. Gupta, V . Kumar, C. Lynch, S. Levine, and K. Hausman, “Relay pol- icy learning: Solving long horizon tasks via imitation and reinforcement learning,” Conference on Robot Learning (CoRL) , 2019

    2019

  70. [78]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018

    2018

  71. [79]

    Stability analysis of discrete-time infinite-horizon optimal control with discounted cost,

    R. Postoyan, L. Bu¸ soniu, D. Neši ´c, and J. Daafouz, “Stability analysis of discrete-time infinite-horizon optimal control with discounted cost,” IEEE Transactions on Automatic Control , 2016. 15 APPENDIX Proof of Proposition IV .1 . Without considering the sam- pling error ...

    2016

  72. [80]

    >" to " <

    If we want to relax the conservative level, like requiring control over the V-values or the empirical MDP, we can also easily relax the Equation (29) to the integration over each state or the whole empirical MDP respectively. For the conditions where we want to make the learne...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.