Pith. sign in

REVIEW 3 major objections 4 minor 70 references

EVAL: EigenVector-based Average-reward Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EVAL learns the entropy-regularized average-reward policy by stochastic approximation of the dominant eigenvector of a tilted transition matrix, and a posterior policy iteration loop recovers the unregularized average-reward optimum…

desk verdict A promising neural extension of the ERAR eigenvalue framework, but the experiments freeze θ=0 and therefore don't validate the algorithm as written. read the letter →

arxiv 2501.09770 v1 pith:BNJCYQLU submitted 2025-01-15 cs.LG cs.AI

Jacob Adamczyk , Volodymyr Makarenko , Stas Tiomkin , Rahul V. Kulkarni This is my paper · ORCID
classification cs.LGcs.AI
keywords average-rewardreinforcementlearningentropyregularizationtiltedtransitionmatrixPerroneigenvectorposteriorpolicyiterationfunctionapproximationoff-policylargedeviationtheory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EVAL is an off-policy algorithm that solves the entropy-regularized average-reward (ERAR) reinforcement-learning problem in continuous state spaces using neural-network function approximation, extending a tabular large-deviation framework in which the optimal policy and reward rate are read off from the dominant eigenvector of a tilted transition matrix. The paper claims that the same eigenvector equation can be learned stochastically by a TD-style update with no discount factor, and that iterating the prior policy (Posterior Policy Iteration, PPI) recovers the unregularized average-reward optimum without temperature annealing. If correct, this gives value-based average-reward RL a practical, DQN-compatible implementation and a principled way to choose a discount factor from the spectral gap of the tilted matrix. The paper supports the claim with classic-control experiments showing better reward and lower variance than DQN and soft Q-learning, including a continuing CartPole task in which EVAL+PPI keeps the pole balanced for very long horizons.

What carries the argument

The load-bearing object is the tilted transition matrix $\tilde P_{(s',a'),(s,a)}=p(s'|s,a)\pi_0(a'|s')e^{\beta r(s,a)}$, whose Perron root $e^{\beta\theta}$ and left eigenvector $u(s,a)$ determine the entropy-regularized average reward rate, the differential value function $Q(s,a)=\beta^{-1}\log u(s,a)$, and the optimal policy $\pi(a|s)\propto \pi_0(a|s)u(s,a)$. EVAL carries this identity into a neural-network stochastic approximation of the fixed-point equation $u(s,a)=e^{\beta(r(s,a)-\theta)}\mathbb{E}[u(s',a')]$, using two online networks, two lagging target networks, and a batch update for $\theta$. The PPI loop treats this solver as a subroutine and iteratively replaces the prior policy with the current optimal-policy estimate, which the paper argues converges to the unregularized average-reward solution without temperature annealing.

What would settle it

Run EVAL on a deterministic continuing MDP whose ERAR solution can be computed exactly (for instance a small discretized chain whose tilted matrix can be diagonalized), train it to convergence, and compare the learned $u(s,a)$, $\theta$, and greedy policy against the exact Perron eigenvector, eigenvalue, and ERAR-optimal policy; a mismatch in $\theta$ or a policy that fails the ERAR Bellman equation would refute the claim, and repeating the test with $\theta$ unfrozen would settle whether the frozen-$\theta$ experiments changed the fixed point.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the entropy-regularized average-reward problem is equivalent to a Perron eigenproblem for the tilted transition matrix $\tilde P_{(s',a'),(s,a)} = p(s'|s,a)\,\pi_0(a'|s')\,e^{\beta r(s,a)}$, and that this equivalence can be turned into a stochastic-approximation learning rule rather than remaining a tabular identity. The eigenvector equation is written as $u(s,a)=e^{\beta(r(s,a)-\theta)}\,\mathbb{E}_{s'\sim p,\,a'\sim\pi_0}[u(s',a')]$, with $u$ parameterized by two deep networks, trained by minimizing the squared TD error in Eq. (14) against a lagging target network, while the reward rate $\theta$ is updated from the batch average in Eq. (16). The paper then shows that replacing the prior policy $\pi_0$ with the current estimate of the optimal policy, at a fixed schedule (posterior policy iteration), yields the greedy average-reward policy without an annealing schedule; it reports that EVAL and EVAL+PPI compare favorably with DQN and soft Q-learning on CartPole, Acrobot, and MountainCar. It asserts that this is the first solution to deterministic ERAR MDPs in continuous state spaces via function approximation.

Load-bearing premise

The load-bearing premise is that the neural-network TD updates converge to the dominant eigenvector of the tilted matrix even with function approximation, and that freezing the reward-rate estimate at zero in the experiments still leaves the ERAR-optimal fixed point; the paper supplies no convergence proof and defers it to future work.

Editorial extensions

If this is right

  • Average-reward RL becomes a value-based, off-policy problem solvable with the same replay-buffer and target-network machinery as DQN, so long-horizon continuing tasks no longer require picking a discount factor.
  • EVAL+PPI solves the unregularized average-reward MDP without discounting or an annealing schedule, by iterating the prior policy toward the current optimal-policy estimate.
  • The spectral gap of the tilted matrix gives a principled discount factor: it marks where discounted soft Q-learning has essentially converged to the average-reward solution, so increasing $\gamma$ further no longer pays.
  • Learning the normalized left eigenvector yields a centered differential value function, avoiding the arbitrary additive constant that many average-reward algorithms carry.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the experiments freeze the reward-rate estimate at $\theta=0$, the empirical results are compatible with a fixed point slightly different from the stated ERAR objective; an ablation that unfreezes $\theta$ and measures the true entropy-regularized rate would clarify what the method actually optimizes.
  • If the eigenvector view is right, the same tilted matrix could serve as a diagnostic unifying discounted and average-reward objectives, for instance by estimating the spectral gap online to set an effective horizon without tuning $\gamma$.
  • PPI's convergence guarantee assumes each ERAR subproblem is solved before the prior is updated, while the experiments update the prior on a fixed schedule; varying the subproblem solve budget would show how much of the success depends on the schedule.
  • The method is restricted to deterministic transitions as presented; taking up the paper's suggested iterative reward/dynamics biasing to handle stochastic dynamics would be a natural stress test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes EVAL, an off-policy algorithm for entropy-regularized average-reward reinforcement learning (ERAR MDPs) based on stochastic approximation of the dominant eigenvector u of a tilted transition matrix. The optimal policy and the entropy-regularized reward rate are recovered from u via Eq. (10) and the Perron root. The paper also introduces a posterior policy iteration (PPI) scheme to recover the unregularized average-reward policy, and reports experiments on classic control environments comparing EVAL and EVAL+PPI against DQN and Soft Q-Learning. The theoretical derivation follows Arriojas et al. (2023b); the algorithmic contribution is a deep-RL implementation with a temporal-difference update for u (Eqs. 14--15) and an online estimate of the reward rate θ (Eq. 16).

Significance. If the proposed stochastic approximation converges, the tilted-eigenvector formulation is an elegant and principled way to solve ERAR MDPs without a discount factor, and the PPI trick provides a potentially useful route to unregularized average-reward control. The paper is clearly written, and the authors state that the code is publicly available. However, the experimental evaluation as reported does not test the algorithm as described: the appendix states that θ is frozen at zero in all non-tabular benchmarks, which changes the fixed point of the learning update and invalidates the theoretical grounding of the experiments. This is a serious gap that must be addressed before the empirical claims can be accepted.

major comments (3)
  1. [Appendix, Experimental Details; Eqs. (15)--(16); Algorithm 1] The appendix states 'The ERAR rate, θ is kept frozen at zero (except for irreducible tabular dynamics).' This directly contradicts Algorithm 1 (lines 17 and 21) and Eq. (16), which update θ online. With θ fixed at zero, the TD target in Eq. (15) becomes e^{βr} E[u], and the corresponding fixed-point equation is u = e^{βr} E[u]. This is not the eigenvector equation (13) unless e^{βθ} = 1; for a positive tilted matrix with Perron root e^{βθ} ≠ 1, the only nonnegative fixed point is u = 0. Therefore the reported benchmark results do not validate EVAL as presented. The authors should either rerun all experiments with the θ update active, or provide a separate theoretical justification for the θ = 0 variant and show that its policy still solves the ERAR MDP.
  2. [Limitations and Future Work; Proposed Algorithms, Eqs. (14)--(16)] No convergence analysis is provided for the coupled stochastic approximation of u and θ, and the paper explicitly defers this to future work. Since the central claim is a 'solution' to deterministic ERAR MDPs with general function approximators, the paper should at least state the conditions under which the updates converge (for example, via two-timescale stochastic approximation with appropriate step-size schedules) or temper the claim to a heuristic algorithm with empirical support. Without this, the theoretical grounding of the function-approximation setting remains incomplete.
  3. [Introduction, Main Contributions; Conclusion] The paper claims to provide 'the first solution to deterministic ERAR MDPs in continuous state spaces by use of function approximation.' Given that the core Lemma 1 is imported from an overlapping-author publication (Arriojas et al. 2023b) and no new convergence result is proven here, the novelty is the algorithmic adaptation plus the empirical study. This is valuable, but the wording overstates the theoretical contribution; please scope the claim accordingly, especially given the issue in the first major comment.
minor comments (4)
  1. [Algorithm 1] Line 9 of Algorithm 1 samples actions from π0, but the text in 'Solution to ERAR-MDP' says the replay buffer is collected using the learnt policy (Eq. (10)). Please align the pseudocode with the text, as is done in Algorithm 3.
  2. [Eq. (16)] The θ update averages the ratio e^{βr} E_{a'~π0} u(s',a') / u(s,a) over the batch. Although u uses a softplus output so the denominator is positive, it can be very small and cause large variance or overflow; consider clipping the ratio or using a log-space update to improve numerical stability.
  3. [Experimental Details, Table 2] The caption states that the final two columns show hyperparameters specific to PPI, but the table appears to use the same column structure as the preceding hyperparameter table; please clarify which columns are PPI-specific.
  4. [Experiments] The text refers to 'EVAL-PPI' while Figure 2 and Algorithm 3 use 'EVAL+PPI'; please standardize the notation throughout the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EVAL is a direct stochastic approximation of the tilted-matrix eigenvector equation, Lemma 1 is an independent published result, and the frozen-θ experiments are a non-circular theory/implementation gap.

full rationale

The derivation chain is self-contained as an algorithm and does not reduce any prediction to its inputs. Equation (15) is literally the right-hand side of the fixed-point/eigenvector equation (13), and Eq. (16) is the matching eigenvalue estimator, so the learned pair (u, θ) is by construction a stochastic approximation to the dominant eigenpair of the tilted matrix; this is standard TD-style self-consistency, not a fitted parameter relabeled as a prediction. The bridge from that eigenpair to the ERAR optimal policy is Lemma 1, imported from Arriojas et al. (2023b); although that work shares authors with the present paper, it is a published, parameter-free derivation rather than a fit performed here, and the present paper also sketches the same spectral/log-probability argument around Eq. (11), so the citation is independent support rather than a self-referential definition. PPI's convergence is imported from Rawlik, Toussaint, and Vijayakumar (2012), an external theorem, and the empirical comparisons are against standard baselines (DQN, SQL) rather than against quantities fit by the method. Two flagged caveats are real but non-circular: the paper defers a convergence proof for the stochastic approximation to future work, and the Appendix freezes θ at zero in the main experiments, so the learned u is not the Eq. (13) Perron eigenvector unless e^{βθ}=1, creating a theory/experiment gap. Neither caveat makes a claimed result equivalent to its inputs by construction, so no circular step is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim relies on the eigenvalue characterization from prior work (Arriojas et al. 2023b) and on several unverified domain assumptions. The main free parameters are the inverse temperature beta, tuned per environment, and the reward-rate theta, frozen at zero in experiments despite the algorithm's update rules. The tilted-matrix framework requires deterministic transitions and Perron-Frobenius conditions, neither of which is verified in the tested environments.

free parameters (4)
  • Inverse temperature beta = CartPole 2, Acrobot 0.01, MountainCar 20
    Controls the strength of entropy regularization in the ERAR objective; chosen by hyperparameter sweeps per environment.
  • Reward-rate theta = 0 (frozen in all benchmark runs, except tabular)
    The TD target in Equation (15) depends on theta, but the experimental appendix freezes theta at zero, which changes the fixed point being learned.
  • Network aggregation function = max over two value networks
    Tuned over min, max, and mean; max was found best empirically but is not justified theoretically.
  • Standard DQN hyperparameters = learning rate, batch size, target update interval, gradient steps, hidden dimension per environment
    Tuned via roughly 200 random-search runs per environment; these affect the reported performance.
assumptions (5)
  • domain assumption The Markov chain induced by the dynamics and any stationary policy is irreducible and aperiodic (Assumption 1).
    Ensures well-defined stationary distributions and the Perron-Frobenius theorem; not verified for the tested continuous-state environments.
  • domain assumption The reward function is upper bounded and can be shifted to be non-positive (Assumption 2).
    Used to guarantee theta is negative and the exponential terms are bounded; the classic-control environments use positive rewards, so the shift is not described.
  • ad hoc to paper Transition dynamics are deterministic.
    The tilted-matrix policy is optimal only for deterministic dynamics; the paper restricts to this case and lists stochastic dynamics as future work.
  • domain assumption The prior policy pi0 has full support over actions.
    Ensures the KL divergence is finite and the tilted matrix is defined; satisfied by softmax parameterization but not stated as a checked condition.
  • domain assumption The tilted matrix in Equation (8) has a unique positive dominant eigenvector.
    Perron-Frobenius is invoked, but irreducibility and aperiodicity of the tilted matrix are not proved from Assumption 1 because of the tilting factor exp(beta r).

how reviews work

0 comments
Cite this review

Pith. "Pith review of EVAL: EigenVector-based Average-reward Learning." pith.science (2026). https://pith.science/paper/BNJCYQLU

@misc{pith2026250109770,
  author       = {Pith},
  title        = {Pith review of: EVAL: EigenVector-based Average-reward Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNJCYQLU}},
  note         = {Machine review of arXiv:2501.09770}
}
read the original abstract

In reinforcement learning, two objective functions have been developed extensively in the literature: discounted and averaged rewards. The generalization to an entropy-regularized setting has led to improved robustness and exploration for both of these objectives. Recently, the entropy-regularized average-reward problem was addressed using tools from large deviation theory in the tabular setting. This method has the advantage of linearity, providing access to both the optimal policy and average reward-rate through properties of a single matrix. In this paper, we extend that framework to more general settings by developing approaches based on function approximation by neural networks. This formulation reveals new theoretical insights into the relationship between different objectives used in RL. Additionally, we combine our algorithm with a posterior policy iteration scheme, showing how our approach can also solve the average-reward RL problem without entropy-regularization. Using classic control benchmarks, we experimentally find that our method compares favorably with other algorithms in terms of stability and rate of convergence.

Figures

Figures reproduced from arXiv: 2501.09770 by the authors.

Figure 1
Figure 1. Performance of discounted soft Q-Learning (SQL) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Classic control benchmark comparing soft Q-learning (SQL), deep Q network (DQN) and our two proposed methods [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. As a demonstration of the usefulness of EVAL+PPI, we consider a modified version of CartPole which represents a continuing task. After training for 5000 steps (in the standard CartPole-v1 environment with a maxi￾mum episode length of 500), we compare the evaluation per￾formance of SQL with EVAL+PPI. Specifically, we set the time-limit of the environment much higher: to 100,000 steps. We find that EVAL+PPI consistent… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: For EVAL (without PPI) we compare the performance for training multiple networks in parallel. All networks are [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 44 canonical work pages

  1. [1]

    Abbasi-Yadkori, Y.; Bartlett, P.; Bhatia, K.; Lazic, N.; Szepesvári, C.; and Weisz, G. 2019. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning, 3692--3702. PMLR

    work page 2019

  2. [2]

    Abounadi, J.; Bertsekas, D.; and Borkar, V. S. 2001. Learning Algorithms for M arkov Decision Processes with Average Cost. SIAM Journal on Control and Optimization, 40(3): 681--698

    work page 2001

  3. [3]

    Adamczyk, J.; Arriojas, A.; Tiomkin, S.; and Kulkarni, R. V. 2023. Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6): 6658--6665

    work page 2023

  4. [4]

    Adamczyk, J.; Makarenko, V.; Tiomkin, S.; and Kulkarni, R. V. 2024. Boosting Soft Q-Learning by Bounding. Reinforcement Learning Journal, 5: 2373--2399

    work page 2024

  5. [5]

    Ahmed, Z.; Le Roux, N.; Norouzi, M.; and Schuurmans, D. 2019. Understanding the impact of entropy on policy optimization. In International conference on machine learning, 151--160. PMLR

    work page 2019

  6. [6]

    Andrychowicz, M.; Raichuk, A.; Sta \'n czyk, P.; Orsini, M.; Girgin, S.; Marinier, R.; Hussenot, L.; Geist, M.; Pietquin, O.; Michalski, M.; et al. 2021. What matters for on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations

    work page 2021

  7. [7]

    Arriojas, A.; Adamczyk, J.; Tiomkin, S.; and Kulkarni, R. V. 2023 a . Bayesian inference approach for entropy regularized reinforcement learning with stochastic dynamics. In Evans, R. J.; and Shpitser, I., eds., Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216, 99--109. PMLR

    work page 2023

  8. [8]

    Arriojas, A.; Adamczyk, J.; Tiomkin, S.; and Kulkarni, R. V. 2023 b . Entropy regularized reinforcement learning using large deviation theory. Phys. Rev. Res., 5: 023085

    work page 2023

Show all 70 references
  1. [9]

    Arriojas, A. A. 2022. Analytical Framework for Entropy Regularized Reinforcement Learning Using Probabilistic Inference. Ph.D. thesis, University of Massachusetts Boston

    2022

  2. [10]

    G.; Naddaf, Y.; Veness, J.; and Bowling, M

    Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253--279

    2013

  3. [11]

    Bertsekas, D. 2012. Dynamic programming and optimal control: Volume I, volume 4. Athena scientific

    2012

  4. [12]

    Blackwell, D. 1962. Discrete dynamic programming. The Annals of Mathematical Statistics, 719--726

    1962

  5. [13]

    Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016. Open AI gym. arXiv preprint arXiv:1606.01540

    2016 arXiv

  6. [14]

    Ceron, J. S. O.; and Castro, P. S. 2021. Revisiting Rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139, 1373--1383. PMLR

    2021

  7. [15]

    Chen, X.; Wang, C.; Zhou, Z.; and Ross, K. W. 2021. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. In International Conference on Learning Representations

    2021

  8. [16]

    Chetrite, R.; and Touchette, H. 2015 a . Nonequilibrium Markov Processes Conditioned on Large Deviations . Annales Henri Poincaré, 16(9): 2005--2057

    2015

  9. [17]

    Chetrite, R.; and Touchette, H. 2015 b . Variational and optimal control representations of conditioned and driven processes. Journal of Statistical Mechanics: Theory and Experiment, 2015(12): P12001

    2015

  10. [18]

    Dewanto, V.; Dunn, G.; Eshragh, A.; Gallagher, M.; and Roosta, F. 2020. Average-reward model-free reinforcement learning: a systematic review and literature mapping. arXiv preprint arXiv:2010.08920

    2020 arXiv

  11. [19]

    M.; and Mansour, Y

    Even-Dar, E.; Kakade, S. M.; and Mansour, Y. 2009. Online M arkov decision processes. Mathematics of Operations Research, 34(3): 726--736

    2009

  12. [20]

    Eysenbach, B.; Gupta, A.; Ibarz, J.; and Levine, S. 2018. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070

    2018 arXiv

  13. [21]

    Eysenbach, B.; and Levine, S. 2022. Maximum Entropy RL (Provably) Solves Some Robust RL Problems. In International Conference on Learning Representations

    2022

  14. [22]

    Fujimoto, S.; Hoof, H.; and Meger, D. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 1587--1596. PMLR

    2018

  15. [23]

    Geist, M.; Scherrer, B.; and Pietquin, O. 2019. A theory of regularized M arkov decision processes. In International Conference on Machine Learning, 2160--2169. PMLR

    2019

  16. [24]

    Grau-Moya, J.; Leibfried, F.; and Vrancx, P. 2018. Soft q-learning with mutual-information regularization. In International conference on learning representations

    2018

  17. [25]

    Haarnoja, T.; Pong, V.; Zhou, A.; Dalal, M.; Abbeel, P.; and Levine, S. 2018 a . Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE international conference on robotics and automation (ICRA), 6244--6251. IEEE

    2018

  18. [26]

    Haarnoja, T.; Tang, H.; Abbeel, P.; and Levine, S. 2017 a . Reinforcement Learning with Deep Energy-Based Policies. In Precup, D.; and Teh, Y. W., eds., Proceedings of the 34th International Conference on Machine Learning, volume 70, 1352--1361. PMLR

    2017

  19. [27]

    Haarnoja, T.; Tang, H.; Abbeel, P.; and Levine, S. 2017 b . Reinforcement learning with deep energy-based policies. In International conference on machine learning, 1352--1361. PMLR

    2017

  20. [28]

    Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S. 2018 b . Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Dy, J.; and Krause, A., eds., Proceedings of the 35th International Conference on Machine Learning, volume 80, 1861...

    2018

  21. [29]

    Haarnoja, T.; Zhou, A.; Hartikainen, K.; Tucker, G.; Ha, S.; Tan, J.; Kumar, V.; Zhu, H.; Gupta, A.; Abbeel, P.; et al. 2018 c . Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905

    2018 arXiv

  22. [30]

    Hafner, D.; Pasukonis, J.; Ba, J.; and Lillicrap, T. 2023. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104

    2023 arXiv

  23. [31]

    Hessel, M.; Modayil, J.; Van Hasselt, H.; Schaul, T.; Ostrovski, G.; Dabney, W.; Horgan, D.; Piot, B.; Azar, M.; and Silver, D. 2018. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    2018

  24. [32]

    Jiang, N.; Kulesza, A.; Singh, S.; and Lewis, R. 2015. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 1181--1189

    2015

  25. [33]

    Kakade, S. M. 2003. On the sample complexity of reinforcement learning. Ph.D. thesis, University College London

    2003

  26. [34]

    Kingma, D.; and Ba, J. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR). San Diega, CA, USA

    2015

  27. [35]

    Lee, K.; Laskin, M.; Srinivas, A.; and Abbeel, P. 2021. SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139, 6131--6141. PMLR

    2021

  28. [36]

    Levine, S. 2018. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909

    2018 arXiv

  29. [37]

    Ma, X.; Tang, X.; Xia, L.; Yang, J.; and Zhao, Q. 2021. Average-Reward Reinforcement Learning with Trust Region Methods. In Zhou, Z.-H., ed., Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , 2797--2803. International Joint Conf...

    2021

  30. [38]

    Mahadevan, S. 1996. Average reward reinforcement learning: Foundations, algorithms, and empirical results. Machine learning, 22: 159--195

    1996

  31. [39]

    M \'e l \'e ard, S.; and Villemonais, D. 2012. Quasi-stationary distributions and population processes. Probability Surveys, 9: 340--410

    2012

  32. [40]

    K.; and Newton, N

    Mitter, S. K.; and Newton, N. 2000. The duality between estimation and control. Published in Festschrift for A. Bennoussan

    2000

  33. [41]

    A.; Veness, J.; Bellemare, M

    Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540): 529--533

    2015

  34. [42]

    Naik, A.; Shariff, R.; Yasui, N.; Yao, H.; and Sutton, R. S. 2019. Discounted reinforcement learning is not an optimization problem. arXiv preprint arXiv:1910.02140

    2019 arXiv

  35. [43]

    Naik, A.; Wan, Y.; Tomar, M.; and Sutton, R. S. 2024. Reward Centering. arXiv preprint arXiv:2405.09999

    2024 arXiv

  36. [44]

    Neu, G.; Jonsson, A.; and G \'o mez, V. 2017. A unified view of entropy-regularized M arkov decision processes. arXiv preprint arXiv:1705.07798

    2017 arXiv

  37. [45]

    Park, S.; Lee, K.; Lee, Y.; and Abbeel, P. 2023. Controllability-Aware Unsupervised Skill Discovery. In Krause, A.; Brunskill, E.; Cho, K.; Engelhardt, B.; Sabato, S.; and Scarlett, J., eds., Proceedings of the 40th International Conference on Machine Learning, volume 202 of P...

    2023

  38. [46]

    Raffin, A.; Hill, A.; Gleave, A.; Kanervisto, A.; Ernestus, M.; and Dormann, N. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268): 1--8

    2021

  39. [47]

    Rawlik, K.; Toussaint, M.; and Vijayakumar, S. 2012. On stochastic optimal control and reinforcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII

    2012

  40. [48]

    Rawlik, K. C. 2013. On probabilistic inference approaches to stochastic optimal control. Ph.D. thesis, The University of Edinburgh

    2013

  41. [49]

    C.; Mair, J

    Rose, D. C.; Mair, J. F.; and Garrahan, J. P. 2021. A reinforcement learning approach to rare trajectory sampling. New Journal of Physics, 23(1): 013013

    2021

  42. [50]

    Saxena, N.; Khastagir, S.; Kolathaya, S.; and Bhatnagar, S. 2023 a . Off-policy average reward actor-critic with deterministic policy search. In International Conference on Machine Learning, 30130--30203. PMLR

    2023

  43. [51]

    Saxena, N.; Khastagir, S.; Shishir, N.; and Bhatnagar, S. 2023 b . Off-policy average reward actor-critic with deterministic policy search. In International Conference on Machine Learning, 30130--30203. PMLR

    2023

  44. [52]

    Schaul, T.; Quan, J.; Antonoglou, I.; and Silver, D. 2015. Prioritized experience replay. arXiv preprint arXiv:1511.05952

    2015 arXiv

  45. [53]

    Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; and Moritz, P. 2015. Trust region policy optimization. In International conference on machine learning, 1889--1897. PMLR

    2015

  46. [54]

    Schulman, J.; Moritz, P.; Levine, S.; Jordan, M.; and Abbeel, P. 2016. High-Dimensional Continuous Control Using Generalized Advantage Estimation. In Proceedings of the International Conference on Learning Representations (ICLR)

    2016

  47. [55]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  48. [56]

    Schwartz, A. 1993. A reinforcement learning method for maximizing undiscounted rewards. In Proceedings of the tenth international conference on machine learning, volume 298, 298--305

    1993

  49. [57]

    S.; and Barto, A

    Sutton, R. S.; and Barto, A. G. 2018. Reinforcement learning: An introduction. MIT press

    2018

  50. [58]

    Todorov, E. 2006. Linearly-solvable M arkov decision problems. In Sch\" o lkopf, B.; Platt, J.; and Hoffman, T., eds., Advances in Neural Information Processing Systems, volume 19. MIT Press

    2006

  51. [59]

    Todorov, E. 2009. Efficient computation of optimal actions. Proceedings of the national academy of sciences, 106(28): 11478--11483

    2009

  52. [60]

    Todorov, E.; Erez, T.; and Tassa, Y. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, 5026--5033. IEEE

    2012

  53. [61]

    Van Hasselt, H.; Guez, A.; and Silver, D. 2016. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30

    2016

  54. [62]

    Wan, Y.; Naik, A.; and Sutton, R. S. 2021. Learning and planning in average-reward M arkov decision processes. In International Conference on Machine Learning, 10653--10662. PMLR

    2021

  55. [63]

    Wang, Z.; Schaul, T.; Hessel, M.; Hasselt, H.; Lanctot, M.; and Freitas, N. 2016. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, 1995--2003. PMLR

    2016

  56. [64]

    Wu, Y.; Tucker, G.; and Nachum, O. 2019. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361

    2019 arXiv

  57. [65]

    B.; and Wang, J

    Yan, X.; Song, Y.; Feng, X.; Yang, M.; Zhang, H.; Ammar, H. B.; and Wang, J. 2024. Efficient Reinforcement Learning with Large Language Model Priors. arXiv:2410.07927

    2024 arXiv

  58. [66]

    S.; and Whiteson, S

    Zhang, S.; Wan, Y.; Sutton, R. S.; and Whiteson, S. 2021. Average-reward off-policy policy evaluation with function approximation. In international conference on machine learning, 12578--12588. PMLR

    2021

  59. [67]

    Zhang, Y.; and Ross, K. W. 2021. On-policy deep reinforcement learning for the average-reward criterion. In International Conference on Machine Learning, 12535--12545. PMLR

    2021

  60. [68]

    Ziebart, B. D. 2010. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University

    2010

  61. [69]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.