Pith. sign in

REVIEW 4 major objections 4 minor 37 references

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Annealing the critic's expectile parameter from near 1 to 0.5 gradually converts the Bellman optimality operator into the Bellman operator, which the paper shows accelerates online actor-critic learning while keeping final Q-value bias low.

desk verdict Real, well-validated empirical gains from annealing the expectile parameter in the critic loss; the operator-transition narrative is looser than advertised, but the paper deserves a serious referee. read the letter →

arxiv 2506.05968 v2 pith:2FQELIDN submitted 2025-06-06 cs.LG cs.AIcs.RO

Motoki Omura , Kazuki Ota , Takayuki Osa , Yusuke Mukuta , Tatsuya Harada This is my paper
classification cs.LGcs.AIcs.RO
keywords onlinereinforcementlearningactor-criticBellmanoptimalityoperatorexpectilelossannealingscheduleoverestimationbiascontinuouscontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to settle a trade-off inside continuous-action actor-critic reinforcement learning: updating the critic toward the optimal value via a max operator learns fast but overestimates, while updating it toward the current policy's value is unbiased but slow. The proposed method, Annealed Q-learning (AQ-L), replaces the critic's ordinary squared loss with an expectile loss whose parameter $\tau$ starts close to 1 — approximating the Bellman optimality operator — and linearly decays to 0.5, which recovers the Bellman operator. In the paper's experiments, TD3 and SAC modified this way outperform their base versions and a fixed-optimality baseline on DM Control locomotion tasks and Meta-World manipulation tasks, with better mean scores and markedly less sensitivity to the choice of $\tau$. The paper's core message is that early overestimation is not just a nuisance but an exploration bonus, and annealing lets an agent consume that bonus early and discard it once the policy has converged.

What carries the argument

The load-bearing object is the expectile loss $L^\tau_2(u) = |\tau - \mathbf{1}\{u < 0\}|\,u^2$, used as the critic's regression loss over actions sampled from the current policy. At $\tau = 0.5$ it equals the squared error and implements the Bellman operator; as $\tau \to 1$ it estimates the upper expectile of the target-return distribution, which in continuous action spaces stands in for the intractable max over actions and thereby implements a Bellman-optimality-style update. The annealing schedule $\tau(t) = \tau_{\mathrm{init}} - (\tau_{\mathrm{init}} - 0.5) t/T$ linearly moves the operator from optimality to expectation over training, carrying the tabular interpolation of Q-learning and SARSA targets from the paper's preliminary experiments into the function-approximation setting.

What would settle it

Train AQ-SAC in a continuous-control task, and at regular checkpoints compute both the implicit expectile target (the value that minimizes the $\tau$-weighted loss over policy-sampled actions) and the true $\max_{a'} Q(s', a')$ evaluated on a dense action grid; if at $\tau = 0.9$ the expectile target deviates from the grid max by a large margin that does not shrink during training, then the loss is not actually implementing a transition from optimality to expectation, and the reported gains would need a different explanation.

Watch

Extended reading notes

Core claim

The central claim is that a single hyperparameter, the expectile level $\tau$ in the critic's loss, can be used to interpolate between the Bellman optimality operator $T^*$ and the Bellman operator $T^\pi$, and that annealing $\tau$ from a high value to 0.5 over the course of training captures the best of both. With $\tau$ near 1, minimizing the expectile loss over actions sampled from the current policy pushes Q-values toward $\max_{a'} Q(s',a')$, so the critic approaches the optimal value function before the policy has improved — the acceleration shown in the paper's tabular experiments. The accompanying overestimation bias, which appears at high $\tau$ and grows with noise, is treated as beneficial early because it widens exploration, then is removed by the anneal down to $\tau = 0.5$, where the loss is the standard L2 loss and the target is the expectation under the current policy. The paper's quantitative evidence is that AQ-SAC with annealing from 0.9 reaches a final mean of 746.1 over ten DM Control tasks, above the best fixed $\tau = 0.7$ at 730.7, and that the estimated Q-bias of AQ-SAC converges to SAC's level by the end of training.

Load-bearing premise

The load-bearing premise is that, over actions sampled from the current policy, the expectile loss at high $\tau$ behaves enough like the exact maximum over actions to deliver the claimed early-learning acceleration, yet the paper never measures the approximation error between the expectile-implied target and the true max.

Editorial extensions

If this is right

  • If the central claim holds, any continuous-action actor-critic that regresses a critic target with an L2 loss can be upgraded by swapping in the expectile loss and annealing $\tau$, with no other hyperparameter changes.
  • Annealed AQ-TD3 and AQ-SAC improve mean score and interquartile mean over TD3, SAC, and XQL on the ten DM Control locomotion tasks at both 1M and 3M steps, with the largest gains on tasks where the baselines make little progress (hopper-hop, humanoid-run, humanoid-walk).
  • The measured Q-bias under annealing converges to the base algorithm's bias level, so the final value estimates are as trustworthy as those of TD3 or SAC despite the early optimistic phase.
  • The robustness result implies that practitioners no longer need to tune the optimality level precisely: scores stay high for $\tau_{\mathrm{init}}$ between 0.7 and 0.95, whereas fixed $\tau$ collapses at 0.8 and above.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The expectile-as-max approximation is never directly measured; a plausible extension would be to compute, at fixed checkpoints, the gap between the $\tau=0.9$ expectile target and the true $\max_a Q(s',a)$ on a dense action grid, and to use that gap to schedule $\tau$ adaptively instead of linearly.
  • The bias-exploration link suggests the method's gains will concentrate in tasks with poor early exploration; in dense-reward, easily explorable domains the anneal may be neutral, so the method should transfer best to sparse-reward manipulation or long-horizon control.
  • Because AQ-L is stated in terms of the loss on the critic, it should compose with orthogonal improvements such as replay-ratio adjustments or ensembling; whether the early optimistic bias strengthens or interferes with those mechanisms is an open question.
  • A direct test of the exploration story would be to replace the early high-$\tau$ phase with an explicit entropy bonus at the same magnitude and keep $\tau = 0.5$ throughout; if the entropy bonus reproduces the gains, the mechanism is exploration rather than the operator transition itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Annealed Q-learning (AQ-L), a modification of actor-critic methods such as TD3 and SAC, in which the critic's squared-error loss is replaced by an expectile loss with parameter τ, and τ is annealed linearly from a value near 1 (e.g., 0.9) down to 0.5 over training. The authors argue that this annealing gradually transitions the critic update from the Bellman optimality operator to the Bellman operator, accelerating early learning through optimistic value estimates while removing overestimation bias in the later stages. Experiments on 10 DM Control locomotion tasks and 10 Meta-World manipulation tasks show that AQ-TD3 and AQ-SAC achieve higher average returns and success rates than TD3, SAC, and XQL, and that annealing provides more robust performance across τ settings than fixed τ. The paper includes ablations on annealing duration, nonlinear schedules, max-backup alternatives, and bias measurements.

Significance. If the empirical claims hold, the proposed modification is simple, broadly applicable, and yields substantial improvements on standard continuous-control benchmarks, which would be a valuable contribution to online RL. The experimental protocol is generally solid: 10 seeds for DM Control and 5 for Meta-World, confidence intervals computed with the rliable methodology, ablations on fixed-τ baselines, annealing duration, and scheduling. The paper also ships code. However, the central theoretical interpretation—that Eq. (6) literally interpolates between the Bellman optimality operator and the Bellman operator—is not supported, and a key ablation comparison (annealed vs. best fixed τ) does not show statistically separated confidence intervals. These issues need to be resolved before the claims can be accepted as stated.

major comments (4)
  1. [§3.4, Eq. (6)] The claim that setting τ close to 1 in the expectile loss (Eq. (6)) estimates a Bellman optimality target is not correct. Minimizing L_τ^2 over q with Y = r + γQ̄(s',a'), where s'~P(·|s,a) and a'~π(·|s'), yields an upper expectile of Y under the joint distribution of transition and action noise. This is not T*Q(s,a) = r + γ E_{s'}[max_{a'} Q̄(s',a')]; in stochastic environments it systematically overshoots any such target because it also up-weights favorable transition noise, and even with deterministic transitions it only reaches an upper expectile over actions in the support of the current policy, not the global maximum. The statement in §3.4 that "setting τ = 1 allows for estimating a Q-learning-based target" is therefore not literally correct, and no theorem or quantitative bound connects Eq. (6) to the advertised interpolation between T* and T^π. Please provide a correct fixed-point characterization of the expectile update, such as a soft-max or expectile-Bellman operator, and adjust the claims accordingly.
  2. [Appendix D] The max-backup comparison is presented as evidence that expectile-based maximization is superior to a direct sampling approximation of the max. However, max-backup with n action samples is a more faithful estimator of E_{s'}[max_{a'} Q̄(s',a')] than the expectile loss is. The observation that AQ-SAC outperforms max-backup on the hardest tasks (hopper-hop, humanoid-run) therefore indicates that the performance gain of AQ-L is not explained by more accurate approximation of the Bellman optimality operator. This undercuts the proposed mechanism and should be confronted head-on, e.g., by analyzing what property of the expectile update (bias, variance, implicit entropy) actually drives the improvement.
  3. [Table 2] The central ablation claim that annealing τ from 0.9 outperforms the best fixed τ is not supported by the reported confidence intervals. Annealed (0.9) has mean 746.1 with CI (732.0–758.5), while Fixed (0.7) has mean 730.7 with CI (715.3–745.7); these intervals overlap substantially. The same is true for the IQM values. Thus the evidence for the specific benefit of annealing over the best fixed-τ setting is weak. Please report pairwise confidence intervals or a significance test (e.g., bootstrap or Mann-Whitney with proper correction) for the annealing-vs-best-fixed comparison, or soften the claim to "comparable or slightly better performance with greater robustness."
  4. [Figure 12 and §4.3] The bias measurement shows that AQ-SAC's bias eventually converges to SAC's level. This is partly by construction, because τ is annealed to 0.5, at which point the expectile loss reduces to the L2 loss and the update becomes exactly the SAC (Bellman operator) update. The interesting scientific claim is the early-training acceleration, not the final debiasing, which is forced by the schedule. The paper should clarify that the debiasing result is a direct consequence of the annealing endpoint, and focus the bias discussion on the transient behavior during training.
minor comments (4)
  1. [Appendix B] Typo: "expetile" should be "expectile" in the sentence describing the initial value of the expectile parameter.
  2. [Figure 17 caption] Typo: "averate" should be "average".
  3. [§3.4] The phrase "smooth interpolation between the Bellman optimality operator and the Bellman operator" is used before the formal definition in Eq. (6); consider moving the precise definition of the expectile loss to the first mention to avoid ambiguity.
  4. [§2.3] The notation L_τ^2(u) is defined as |τ − 1(u < 0)| u^2, but the indicator function variable is not explicitly redefined; please give the full expression with the indicator argument, e.g., 1{u<0}, for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the annealed-expectile method is an empirical algorithm whose reported gains are not encoded in its loss or annealing schedule by construction.

full rationale

The paper's central claim is that annealing the expectile parameter tau from near 1 to 0.5 in the critic loss (Eq. 6) improves online RL performance relative to fixed-tau and standard actor-critic baselines. This is an empirical claim, and the derivation chain does not reduce to its inputs. At tau = 0.5, the loss is exactly the standard L2 critic loss, and the paper explicitly states that in this case AQ-TD3 and AQ-SAC are precisely TD3 and SAC (Sec. 4.1), so the comparison against TD3/SAC is not circular. The annealing schedule (Eq. 7) is a linear function of the timestep and is not derived from the benchmark scores; the measured improvements in Table 2 and Appendix C come from comparing annealed runs against fixed-tau runs over a range of tau/tau_init values. No fitted parameter is renamed as a prediction: tau_init is a hyperparameter, and the paper reports performance across several values rather than claiming a specific value is predicted. The expectile loss is adopted from IQL (Kostrikov et al., 2022) as an approximation to max-style backups; the fact that the high-tau expectile is not literally equivalent to the Bellman optimality operator in stochastic environments is a correctness/validation limitation, not a circularity, because the paper does not define the operator as equivalent to Eq. 6 by fiat. The only self-citation (Omura et al., 2024, MXQL) appears in related work and an ablative appendix combination; no load-bearing premise or central result depends on it. Accordingly, no circular step can be exhibited with the required quote-and-reduction standard.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard RL operator theory and on two domain assumptions: that the expectile loss is a valid surrogate for the max operator in continuous action spaces, and that early overestimation aids exploration. The method introduces no new entities, but has two fitted hyperparameters (τinit per benchmark, annealing duration) that affect the reported results.

free parameters (4)
  • τ_init for AQ-SAC on DM Control = 0.9
    Chosen in Appendix B, Table 3; robustness shown across 0.7-0.95 in Table 2 but the default affects the reported main result.
  • τ_init for AQ-TD3 on DM Control = 0.8
    Chosen in Appendix B, Table 3; different from AQ-SAC's 0.9, suggesting per-algorithm tuning.
  • τ_init for Meta-World (both) = 0.7
    Selected for the manipulation benchmark; robustness not reported on Meta-World.
  • Annealing duration T = 3M steps for DMC, 10M for Meta-World
    Set equal to total training steps; Appendix E shows robustness on two tasks across 1M-5M, but T is still a schedule parameter affecting results.
assumptions (3)
  • domain assumption The expectile loss with τ close to 1 estimates a value close to the maximum of the target distribution, effectively approximating the Bellman optimality operator for continuous actions.
    Section 3.4, Eq. (6); the approximation is not formally quantified and is validated only indirectly via max-backup comparison in Appendix D.
  • domain assumption Overestimation bias in early training promotes exploration and is beneficial, while later bias is harmful.
    Section 3.3 and Appendix I; support is indirect: entropy measurements correlate with τ, but a causal link is assumed.
  • standard math Bellman operators are contractive and iterative application converges as described.
    Section 2.2, standard RL background.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning." pith.science (2026). https://pith.science/paper/2FQELIDN

@misc{pith2026250605968,
  author       = {Pith},
  title        = {Pith review of: Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FQELIDN}},
  note         = {Machine review of arXiv:2506.05968}
}
read the original abstract

For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL). However, unlike RL algorithms for discrete actions, which generally model the optimal value function using the Bellman optimality operator, RL algorithms for continuous actions typically model Q-values for the current policy using the Bellman operator. These algorithms for continuous actions rely exclusively on policy updates for improvement, which often results in low sample efficiency. This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks. Experiments in a simple environment show that modeling optimal values accelerates learning but leads to overestimation bias. To address this, we propose an annealing approach that gradually transitions from the Bellman optimality operator to the Bellman operator, thereby accelerating learning while mitigating bias. Our method, combined with TD3 and SAC, significantly outperforms existing approaches across various locomotion and manipulation tasks, demonstrating improved performance and robustness to hyperparameters related to optimality. The code for this study is available at https://github.com/motokiomura/annealed-q-learning.

Figures

Figures reproduced from arXiv: 2506.05968 by the authors.

Figure 1
Figure 1. A simple MDP used in preliminary experiments. A re￾ward ri is obtained upon reaching state si , and the episode termi￾nates when the agent reaches either state s3 or s4. The discount rate γ is set to 0.9. 3. Gradual Transition from Bellman Optimality Operator to Bellman Operator In this section, we first discuss the effectiveness of the Bell￾man optimality operator T ∗ and the Bellman operator T π in actor-critic me… view at source ↗
Figure 2
Figure 2. Left: The estimated values of Q(s0, a0) and Q(s0, a1) when using tabular actor-critic with a Q-learning-based critic up￾date under T ∗ , compared to a SARSA-based critic update based on T π in the environment of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The average scores across the 10 locomotion tasks in DM Control. learning and that annealing this optimality allows for ef￾ficient estimation of less biased values in the final stage. However, in continuous action tasks, it is generally not fea￾sible to directly compute the maximum Q-value. To address this, we use the expectile loss, as introduced in Kostrikov et al. (2022), which allows for the implicit computation… view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: The average return for each task in DM Control. Gumbel regression, assuming a Gumbel distribution for the error distribution, and estimates the soft-optimal value in entropy-maximizing RL, offering a comparison as another method for estimating the optimal value. While …
Figure 5
Figure 5. Figure 5: The average success rate across the 10 manipulation tasks in Meta-World. 4.3. Effects of τ Annealing We evaluated the effectiveness of annealing τ using AQ￾SAC [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The average success rate for each task in Meta-World. AQ-SAC Fixed (0.6) Fixed (0.7) Fixed (0.8) Fixed (0.9) Fixed (0.95) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Environment Steps ×106 0 100 200 300 400 500 Average Return hopper-hop 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Environment Steps ×1…
Figure 7
Figure 7. Figure 7: The average return of AQ-SAC on hopper-hop and humanoid-run when τ is annealed compared to when τ is fixed. Annealing τ improves asymptotic performance and stability. bias. Incorporating these ensemble techniques into our pro￾posed method could potentially enhance samp…
Figure 8
Figure 8. Figure 8: The estimated Q-values for s1 and s2 in the experiment where noise was added to reproduce the randomness of Q-value estimation in the environment of [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: The estimated Q-values obtained using various standard deviations σ of Gaussian noise in the environment of [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: It was observed that when the difference between r3 and r4 is large, the bias decreases. Optimal Q(s0, a0) Optimal Q(s0, a1) Q-learning Q(s0, a0) Q-learning Q(s0, a1) SARSA Q(s0, a0) SARSA Q(s0, a1) Annealed Q(s0, a0) Annealed Q(s0, a1) 0 10000 20000 30000 40000 50000…
Figure 11
Figure 11. Figure 11: The estimated Q-values obtained using r1 = 0.25 and r2 = −0.25 with the various standard deviations σ of Gaussian noise in the environment of [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: The bias of the estimated Q-value with respect to the Monte Carlo return. The larger τ , the greater the bias, and in AQ-SAC, the bias eventually became comparable to that of SAC. D. Experimental Results of Max-backup In continuous action tasks, one straightforward wa…
Figure 13
Figure 13. Figure 13: The average return for each task when annealing with various τinit values in AQ-SAC. while the right figure corresponds to fixed τ . The values used were τinit = 0.9 for annealing and τ = 0.7 for the fixed case, as these settings yielded the best performance. In both …
Figure 14
Figure 14. Figure 14: The average return for each task when using various fixed τ values in AQ-SAC [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: The average return for each task when using max-backup with various action sampling numbers [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: The average return of AQ-SAC when the annealing duration is varied. The dashed line represents the step count at which annealing ends, after which learning proceeds with τ = 0.5, the same as the SAC. 0.0 0.2 0.4 0.6 0.8 1.0 Environment Steps ×107 0 100 200 300 400 Ave…
Figure 17
Figure 17. Figure 17: The averate return of SAC and AQ-SAC. In AQ-SAC, training continues with τ = 0.5 after 3M steps of annealing. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Environment Steps ×106 0 100 200 300 400 Average Return hopper-hop (a) Annealed τ from 0.9 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Environment…
Figure 18
Figure 18. Figure 18: The average scores for AQ-SAC trained using only the Q-function compared to AQ-SAC utilizing both the Q-function and V-function in a stochastic hopper-hop environment. The stochastic environment is created by adding zero-mean Gaussian noise to the actions fed into the…
Figure 19
Figure 19. Figure 19: The annealing patterns of τ used in the experiments. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Environment Steps ×106 0 100 200 300 400 500 600 700 Average Return AQ-TD3 (MXQL) AQ-SAC (MXQL) TD3 SAC [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 21
Figure 21. Figure 21: The average return for each task in DM Control. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 26 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    S., Courville, A., and Bellemare, M

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021

    work page 2021

  3. [3]

    P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z

    Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z. D., and Blundell, C. Agent57: Outperforming the A tari human benchmark. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 507--517. PMLR, 13--18 Jul 2020

    work page 2020

  4. [4]

    Chen, X., Wang, C., Zhou, Z., and Ross, K. W. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations, 2021

    work page 2021

  5. [5]

    G., and Courville, A

    D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023

    work page 2023

  6. [6]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 1587--1596. PMLR, 10--15 Jul 2018

    work page 2018

  7. [7]

    Extreme q-learning: Maxent RL without entropy

    Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme q-learning: Maxent RL without entropy. In International Conference on Learning Representations, 2023

    work page 2023

  8. [8]

    Reinforcement learning with deep energy-based policies

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 1352--1361. PMLR, 06--11 Aug 2017

    work page 2017

Show all 37 references
  1. [9]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings...

    2018

  2. [10]

    and Montana, G

    Ireland, D. and Montana, G. Revalued: Regularised ensemble value-decomposition for factorisable markov decision processes. In The Twelfth International Conference on Learning Representations, 2024

    2024

  3. [11]

    Seizing serendipity: exploiting the value of past success in off-policy actor-critic

    Ji, T., Luo, Y., Sun, F., Zhan, X., Zhang, J., and Xu, H. Seizing serendipity: exploiting the value of past success in off-policy actor-critic. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  4. [12]

    Scalable deep reinforcement learning for vision-based robotic manipulation

    Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pp.\ 651--673. PMLR, 2018

    2018

  5. [13]

    Offline reinforcement learning with implicit q-learning

    Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022

    2022

  6. [14]

    Stabilizing off-policy q-learning via bootstrapping error reduction

    Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems,...

    2019

  7. [15]

    Conservative q-learning for offline reinforcement learning

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 1179--1191. Curran Associ...

    2020

  8. [16]

    Maxmin q-learning: Controlling the estimation bias of q-learning

    Lan, Q., Pan, Y., Fyshe, A., and White, M. Maxmin q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2020

    2020

  9. [17]

    Continuous control with deep reinforcement learning

    Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  10. [18]

    Playing atari with deep reinforcement learning, 2013

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning, 2013

    2013

  11. [19]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518 0 (7540): 0 529--533, 2015

    2015

  12. [20]

    Curriculum dropout

    Morerio, P., Cavazza, J., Volpi, R., Vidal, R., and Murino, V. Curriculum dropout. In ICCV, 2017

    2017

  13. [21]

    Stabilizing extreme q-learning by maclaurin expansion

    Omura, M., Osa, T., Mukuta, Y., and Harada, T. Stabilizing extreme q-learning by maclaurin expansion. In Reinforcement Learning Conference, 2024

    2024

  14. [22]

    and Niranjan, M

    Rummery, G. and Niranjan, M. On-line q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166, 11 1994

    1994

  15. [23]

    Trust region policy optimization

    Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp.\ 1889--1897, Lil...

    2015

  16. [24]

    Proximal policy optimization algorithms, 2017

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017

    2017

  17. [25]

    Solving continuous control via q-learning

    Seyde, T., Werner, P., Schwarting, W., Gilitschenski, I., Riedmiller, M., Rus, D., and Wulfmeier, M. Solving continuous control via q-learning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  18. [26]

    Growing Q -networks: S olving continuous control tasks with adaptive control resolution

    Seyde, T., Werner, P., Schwarting, W., Wulfmeier, M., and Rus, D. Growing Q -networks: S olving continuous control tasks with adaptive control resolution. In Abate, A., Cannon, M., Margellos, K., and Papachristodoulou, A. (eds.), Proceedings of the 6th Annual Learning for Dyna...

    2024

  19. [27]

    Dual RL : Unification and new methods for reinforcement and imitation learning

    Sikchi, H., Zheng, Q., Zhang, A., and Niekum, S. Dual RL : Unification and new methods for reinforcement and imitation learning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  20. [28]

    Learning to predict by the method of temporal differences

    Sutton, R. Learning to predict by the method of temporal differences. Machine Learning, 3: 0 9--44, 08 1988. doi:10.1007/BF00115009

    1988 doi

  21. [29]

    Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018

    2018

  22. [30]

    Deepmind control suite, 2018

    Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., and Riedmiller, M. Deepmind control suite, 2018

    2018

  23. [31]

    Action branching architectures for deep reinforcement learning

    Tavakoli, A., Pardo, F., and Kormushev, P. Action branching architectures for deep reinforcement learning. In AAAI Conference on Artificial Intelligence, pp.\ 4131--4138, 2018

    2018

  24. [32]

    A deep hierarchical approach to lifelong learning in minecraft

    Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., and Mannor, S. A deep hierarchical approach to lifelong learning in minecraft. Proceedings of the AAAI Conference on Artificial Intelligence, 31 0 (1), Feb. 2017. doi:10.1609/aaai.v31i1.10744

    2017 doi

  25. [33]

    and Schwartz, A

    Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In In Fourth Connectionist Models Summer School, 10 1993

    1993

  26. [34]

    dm\_control: Software and tasks for continuous control

    Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. dm\_control: Software and tasks for continuous control. Software Impacts, 6: 0 100022, 2020. ISSN 2665-9638. doi:https://doi.org/10.1016/j.simpa.2020.100022

    2020

  27. [35]

    Watkins, C. J. C. H. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, UK, 05 1989

    1989

  28. [36]

    Xu, H., Jiang, L., Li, J., Yang, Z., Wang, Z., Chan, V. W. K., and Zhan, X. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023

    2023

  29. [37]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.