Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learning

Dominik Baumann; Xinyi Sheng

REVIEW 3 major objections 4 minor 33 references

Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learning

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A reinforcement learning objective that mixes expected return with a sliding-window geometric mean can improve long-term per-trajectory performance.

desk verdict The paper's heuristic regularizer might help in some tasks, but the theoretical bridge it claims — MGM estimates the time-average growth rate — does not hold up, leaving the central objective unsupported. read the letter →

arxiv 2508.21443 v1 pith:74TUT2VC submitted 2025-08-29 cs.LG cs.SYeess.SY

Xinyi Sheng , Dominik Baumann This is my paper · ORCID

classification cs.LGcs.SYeess.SY

keywords geometricmeantime-averagegrowthratenon-ergodicrewardsreinforcementlearningmulti-stepQ-learningregularizedMDPslidingwindowBrownianmotion

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard RL objective—expected cumulative reward over many rollouts—can mislead when rewards are non-ergodic, because a policy that looks good on average may fail on most individual trajectories. To fix this, it proposes optimizing a weighted combination of the expected return and the time-average growth rate of a single trajectory, estimating the latter with a modified geometric mean over an N-step sliding window of rewards. The authors derive a regularized Bellman operator for this combined objective and prove that the idealized operator is a contraction whose fixed point is the optimal Q-function. In Lunar Lander and Cart-Pole simulations, the regularized multi-step Q-learning algorithm raises the median and mean cumulative reward compared with standard multi-step Q-learning. A reader would care because this is a drop-in modification to existing Q-learning that targets what an agent actually experiences over its lifetime, not just the ensemble average.

What carries the argument

The engine of the method is a modified geometric mean (MGM) over an N-sliding window of rewards, Ĝ_t = sgn(R̂_t)(|R̂_t|)^{1/N}, with R̂_t the sum of the next N rewards. It is meant to estimate the time-average growth rate of a trajectory without knowing the reward dynamics. This estimator is injected as a regularizer into the multi-step Q-learning target, weighted by λ, and the resulting objective is analyzed through an N-step regularized Bellman optimality operator (TG)^N. The operator, which replaces each per-step reward with (1−λ)r_t + (1−γ)λG∞, is shown to be a γ^N-contraction with fixed point the optimal regularized Q-function; this is what licenses the practical algorithm.

What would settle it

Run two policies on the multiplicative coin-toss process from the paper, chosen so their true time-average growth rates are rank-ordered and separated. For each policy, collect long reward trajectories, compute the MGM estimator on N-windows for several N, and check whether the estimator orders the policies in the same direction as the true growth rates. If the ordering flips for any N, or if the estimator does not converge to a quantity monotone in the growth rate, the regularizer is optimizing a proxy unrelated to its stated target.

Watch

Extended reading notes

Core claim

The central claim is that the time-average growth rate—the long-run growth of rewards along one trajectory—can be estimated by a modified geometric mean of cumulative rewards over a sliding window, and that adding this estimate as a regularizer to the standard RL objective improves long-term policy performance. Concretely, the paper defines Ĝ_t = sgn(R̂_t)(|R̂_t|)^{1/N} with R̂_t = Σ_{i=0}^{N-1} r_{t+i}, embeds λĜ into the multi-step Q-learning target, and shows that under geometric Brownian motion reward increments a stationary Markov chain has a fixed growth rate ḡ = Σ_i d_i^π(μ_i − σ_i²/2). The idealized regularized Bellman operator is proved to be a γ^N-contraction with the optimal Q-

Load-bearing premise

The whole algorithm rests on the assumption that taking the N-th root of a sliding-window sum of rewards actually measures how fast a trajectory grows over time; if that estimator does not track the growth rate, the regularizer optimizes the wrong quantity.

Editorial extensions

If this is right

If the estimator is faithful, setting λ=1 makes the objective directly maximize the invariant growth rate ḡ, connecting RL to ergodicity-theoretic results without requiring an explicit ergodicity transformation.
Because λ=0 recovers standard multi-step Q-learning, the regularizer can be added to existing multi-step return estimators by changing only the target computation.
The γ^N-contraction property means the idealized regularized operator has a unique fixed point and inherits the convergence behavior of standard multi-step Q-learning when the growth rate is known.
The sliding window makes the value update depend on the last N rewards, effectively turning the MDP into a path-dependent process, so the policy can react to recent reward trends rather than only the current state.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to replace the raw sliding-window sum with a discounted window, aligning the estimator's horizon with γ and making it more natural for off-policy corrections.
The same MGM regularizer could be applied to policy-gradient methods as a path-dependent intrinsic reward; the paper only demonstrates Q-learning.
The dynamic-awareness flag e suggests a curriculum—non-overlapping windows during exploration so estimates are consistent with one policy, then overlapping windows as the policy stabilizes—which is plausible but not analyzed.
Because the sign function handles negative rewards by preserving direction, the estimator's scale sensitivity in environments with large negative rewards is worth testing; a scale-invariant normalization may be needed.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know: the paper combines a sensible motivation with a broken theoretical link. The core claim is that a modified geometric mean over an N-step window estimates the time-average growth rate, and that a regularized Bellman operator can optimize it. On reading the math, neither step holds together.

What's genuinely new: the specific MGM regularizer embedded into multi-step Q-learning is not in the prior literature. The authors are honest about borrowing the GBM growth-rate result, and the algorithm is simple enough to reproduce — they even provide code. The empirical comparison against a λ=0 baseline shows consistent median improvements on two control tasks, which is something.

The problems start with Equation (8). The MGM is the N-th root of an additive window sum, but Proposition 1's growth rate is the logarithmic per-period growth of a multiplicative process. For positive growth, the N-th root of (R_{t+N} - R_t) converges to exp(ḡ), not ḡ; for negative growth it goes to -1. So the estimator is not estimating the quantity the theory defines. That severs the practical algorithm from the stated objective.

Then there's the fixed-point issue. Definition 1 bakes in G∞, the optimal growth rate, as a parameter. That makes the optimality equation near-tautological; the proof of Proposition 2 substitutes Gπ and never establishes the fixed point for the actual definition. Under Assumption 1, the growth rate is independent of the initial (s,a) anyway, so even a perfectly known G∞ would be constant across actions and cannot change greedy selection. That undermines the whole mechanism.

The empirical section is also thin: no error bars, no seeds, and the λ sweep is done on the same test set. The gains are modest and on two basic benchmarks. So the paper doesn't meet the bar for an archival venue as it stands.

That said, the underlying question — how to optimize per-trajectory long-term performance under non-ergodic reward dynamics — is real and worth attention. The heuristic might work for reasons not captured by the theory. But the current manuscript's central claims don't survive scrutiny.

Recommendation: send to peer review, because the topic is important and the flaws deserve a careful referee report. With the theory properly fixed or repositioned as a heuristic, there might be a useful paper here. For now, though, I'd expect rejection without major revision.

Referee Report

3 major / 4 minor

Summary. The paper proposes a reinforcement learning objective that adds a time-average growth-rate regularizer to the conventional expected discounted return. The authors define an N-step regularized Bellman optimality operator for an idealized known growth rate, introduce a 'modified geometric mean' (MGM) estimator over an N-sliding window to estimate this rate from observed rewards, and embed it in a multi-step Q-learning algorithm. They report improved median cumulative rewards on Lunar Lander and Cart-Pole compared with a multi-step Q-learning baseline for several values of the balance parameter λ.

Significance. The motivation is important: in non-ergodic reward dynamics, optimizing the ensemble mean can yield policies that perform poorly on individual trajectories, and a practical regularizer targeting long-run per-trajectory growth would be a useful contribution. The paper also ships code and evaluates on standard benchmarks, which is commendable. However, the central theoretical bridge—the claim that the MGM estimator (Eq. 8) estimates the time-average growth rate G∞ used in the Bellman operator—is not established, and the regularized Bellman operator is defined using the very quantity it is meant to compute. If these gaps cannot be repaired, the algorithm is an unexplained heuristic and the theoretical contribution is unsupported.

major comments (3)

[Sec. IV-B2, Eq. (8)] The MGM estimator does not estimate the time-average growth rate. With r_t = R_t - R_{t-1}, the N-window sum is R̂_t = R_{t+N} - R_t. The growth rate in Prop. 1 is asymptotically (1/N) log(R_{t+N}/R_t), whereas |R_{t+N} - R_t|^{1/N} is a different functional: for large R_t it behaves like R_t^{1/N}(e^{Ng}-1)^{1/N}, not g, and when R_{t+N} < R_t it returns a large negative value rather than a negative log-ratio. Thus Eq. (8) is not an estimator of G∞, and Algorithm 1 is not optimizing objective (3) in the stated sense.
[Definition 1 and Appendix B] The regularized Bellman optimality operator (TG)^N is defined using G∞, 'the time-average growth rate of the best policy'—the object that the operator is supposed to find. In the proof of Prop. 2(1), the authors prove a fixed point for T^π_G with a policy-dependent G^π_∞ and then substitute it into the optimality equation, treating G∞ as a fixed external parameter. The contraction proof likewise treats G∞ as independent of q. As stated, (TG)^N is not a well-defined operator on Q-functions, and the fixed-point and contraction claims do not establish convergence of the proposed regularized Q-learning algorithm.
[Sec. III and Definition 1] Under Assumption 1 (irreducible and aperiodic chain), the time-average growth rate of a fixed policy is almost surely independent of the initial state-action pair. Hence G^π_∞(s,a) is a constant across (s,a) for each π. In Definition 1, the regularizer (1-γ)λG∞(s,a) then adds the same value to every action at every state and cancels in the max over actions. The theoretical regularizer therefore cannot change the greedy policy in the idealized setting, undermining the claimed mechanism. Any empirical benefit must come from the finite-window heuristic, not from the stated objective.

minor comments (4)

[Sec. IV-B] The relationship between the continuous-time Δt in Eq. (7) and the discrete window size N in Eq. (6) is never made precise; the paper moves between continuous SDE notation and discrete-time RL without a clear discretization.
[Sec. IV-C1, after Definition 1] The sentence 'For λ = 1, this operator maps any q to a fixed point, G∞' is unclear: an operator maps a Q-function to a Q-function, not to a scalar. This should be rephrased or removed.
[Sec. III and IV-A] Assumption 2 (existence of an ergodicity transformation) is stated as essential but is never used in any proof or derivation; the paper should either use it or state that it is only motivational.
[Sec. V] The experimental comparison reports only medians and density plots without error bars or statistical tests. The claim that the method 'outperforms conventional RL methods' would be strengthened by repeated-seed statistics and significance tests.

Circularity Check

1 steps flagged · score 6.0 of 10

The regularized Bellman optimality operator is defined in terms of the optimal growth rate G∞, which is exactly the quantity the operator is supposed to compute, so the fixed-point result is true by construction.

self definitional [Section IV-C1, Definition 1 (Regularized Bellman optimality operator)]
"Now, let π∗ represent the optimal policy such that π∗ = arg maxπ qπ_G and q∗_G := max π qπ_G = qπ∗_G be the optimal Q-value. Then, we can define the corresponding N-step regularized Bellman optimality operator. Definition 1 ... where G∞ is the time-average growth rate of the best policy that can maximize qπ_G over π."

The operator (TG)^N embeds G∞(s,a), which is defined as the time-average growth rate of the optimal policy π* (the very policy the operator is meant to produce). Thus the fixed point q*_G is not derived; it is inserted as a parameter. Classical Bellman optimality operators construct the optimal value via max over actions without presupposing the optimal policy; here the regularization term requires G^{π*}_∞ before the operator can be written. Proposition 2(1) therefore reduces to "q*_G is fixed under an operator containing q*_G's own growth rate," true by construction. Contractivity (Prop. 2.2) still holds, but the claimed derivation of a Bellman operator for the time-average growth rate is circular.

full rationale

The paper's central theoretical fixed-point result is circular: the N-step regularized Bellman operator is parameterized by G∞, the optimal policy's time-average growth rate, so the fixed point q*_G is baked into the definition. The practical algorithm instead uses the MGM estimator Ĝ_t = sgn(R̂_t)|R̂_t|^{1/N}, but while no derivation connects this estimator to the GBM growth rate of Prop. 1, that is an unsupported leap rather than a demonstrated reduction, so it is not scored as a separate circular step under the strict rules. No load-bearing self-citation is present: reference [8] is motivational, and the existence of ergodicity transformations is attributed to external work [24]. The empirical comparison against conventional multi-step Q-learning is an independent benchmark comparison, not a fitted prediction. Overall, partial circularity: the theoretical guarantee reduces by construction, while the empirical claim retains independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The paper's central contribution rests on an unvalidated estimator (MGM), a Bellman operator that requires the optimal growth rate as known input, and two assumptions that are not operationalized. The GBM model in Prop. 1 is the only place a growth rate is rigorously defined, yet the algorithm does not use that structure, creating a gap between theory and practice.

free parameters (3)

lambda (balance parameter) = swept over [0,1] in both environments, no principled selection
Section V reports performance across lambda values; the claim of outperformance depends on the chosen lambda for each task.
N (sliding window size) = 2 for Lunar Lander, 5 for Cart-Pole
Chosen by hand in Section V; no rule for selecting N is given.
G_infinity (ideal growth rate) = assumed known in Definition 1, replaced by MGM estimate in practice
The Bellman operator is defined using the optimal growth rate as input, which is the unknown solution; in the practical algorithm it is replaced by G_t_hat from Eq. (8), itself fitted to windowed rewards.

assumptions (4)

domain assumption Assumption 1: The Markov chain underlying the MDP is irreducible and aperiodic.
Invoked in Section IV-B to ensure a stationary distribution exists; the practical algorithm is applied to environments (e.g., Cart-Pole) where this is not obviously verified.
domain assumption Assumption 2: An ergodicity transformation exists for the reward dynamics.
Stated in Section IV as essential for the framework, but the method never computes it; justification rests on a citation to [24].
ad hoc to paper Per-state reward increments follow a geometric Brownian motion (GBM).
Section IV-B2 assumes GBM to derive Prop. 1, but the MGM estimator is then applied to general additive rewards in the experiments, so the assumption's role is unclear.
standard math Standard finite MDP with stationary transition kernel and discount factor gamma.
Section III defines the MDP; the Bellman machinery relies on standard contraction and fixed point results.

invented entities (1)

Modified geometric mean (MGM) estimator
purpose: Estimates the time-average growth rate for the regularizer in the RL objective.
Eq. (8) is introduced as an estimator without a derivation or external validation; it does not correspond to the log-growth rate of the GBM model it claims to represent.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learning." pith.science (2026). https://pith.science/paper/74TUT2VC

@misc{pith2026250821443,
  author       = {Pith},
  title        = {Pith review of: Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/74TUT2VC}},
  note         = {Machine review of arXiv:2508.21443}
}

abstract

Reinforcement learning (RL) algorithms typically optimize the expected cumulative reward, i.e., the expected value of the sum of scalar rewards an agent receives over the course of a trajectory. The expected value averages the performance over an infinite number of trajectories. However, when deploying the agent in the real world, this ensemble average may be uninformative for the performance of individual trajectories. Thus, in many applications, optimizing the long-term performance of individual trajectories might be more desirable. In this work, we propose a novel RL algorithm that combines the standard ensemble average with the time-average growth rate, a measure for the long-term performance of individual trajectories. We first define the Bellman operator for the time-average growth rate. We then show that, under multiplicative reward dynamics, the geometric mean aligns with the time-average growth rate. To address more general and unknown reward dynamics, we propose a modified geometric mean with $N$-sliding window that captures the path-dependency as an estimator for the time-average growth rate. This estimator is embedded as a regularizer into the objective, forming a practical algorithm and enabling the policy to benefit from ensemble average and time-average simultaneously. We evaluate our algorithm in challenging simulations, where it outperforms conventional RL methods.

Figures

Figures reproduced from arXiv: 2508.21443 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 26 canonical work pages

[1]

Human- level control through deep reinforcement learning,

V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, et al., “Human- level control through deep reinforcement learning,” Nature, 2015

work page 2015
[2]

Benchmarking deep reinforcement learning for continuous control,

Y . Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” in Int. Conf. on Machine Learning , 2016

work page 2016
[3]

Research priorities for robust and beneficial artificial intelligence,

S. Russell, D. Dewey, and M. Tegmark, “Research priorities for robust and beneficial artificial intelligence,” AI Magazine, 2015

work page 2015
[4]

Concrete problems in AI safety,

D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Man ´e, “Concrete problems in AI safety,” arXiv preprint arXiv:1606.06565, 2016

arXiv 2016
[5]

Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures,

J. Uesato, A. Kumar, C. Szepesvari, T. Erez, A. Ruderman, K. An- derson, N. Heess, P. Kohli, et al. , “Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures,” arXiv preprint arXiv:1812.01647, 2018

arXiv 2018
[6]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduc- tion. MIT press, 2018

2018
[7]

Bertsekas, Reinforcement Learning and Optimal Control

D. Bertsekas, Reinforcement Learning and Optimal Control . Athena Scientific, 2019

2019
[8]

Reinforcement learning with non-ergodic reward in- crements: robustness via ergodicity transformations,

D. Baumann, E. Noorani, J. Price, O. Peters, C. Connaughton, and T. B. Sch ¨on, “Reinforcement learning with non-ergodic reward in- crements: robustness via ergodicity transformations,” Transactions on Machine Learning Research , 2025

work page 2025

Show all 33 references

[9]

Linearly-solvable markov decision problems,

E. Todorov, “Linearly-solvable markov decision problems,” Advances in Neural Information Processing Systems , 2006

2006
[10]

A theory of regularized Markov decision processes,

M. Geist, B. Scherrer, and O. Pietquin, “A theory of regularized Markov decision processes,” in Int. Conf. on Machine Learning, 2019

2019
[11]

Approximate modified policy iteration and its application to the game of tetris

B. Scherrer, M. Ghavamzadeh, V . Gabillon, B. Lesner, and M. Geist, “Approximate modified policy iteration and its application to the game of tetris.” Journal of Machine Learning Research , 2015

2015
[12]

Munchausen reinforcement learning,

N. Vieillard, O. Pietquin, and M. Geist, “Munchausen reinforcement learning,” Advances in Neural Information Processing Systems , 2020

2020
[13]

Maximum a posteriori policy optimisation,

A. Abdolmaleki, J. T. Springenberg, Y . Tassa, R. Munos, N. Heess, and M. Riedmiller, “Maximum a posteriori policy optimisation,” arXiv preprint arXiv:1806.06920, 2018

2018 arXiv
[14]

Leverage the average: an analysis of KL regularization in reinforcement learning,

N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist, “Leverage the average: an analysis of KL regularization in reinforcement learning,” Advances in Neural Information Processing Systems, 2020

2020
[15]

Incremental multi-step q-learning,

J. Peng and R. J. Williams, “Incremental multi-step q-learning,” in Machine Learning Proceedings 1994 . Elsevier, 1994

1994
[16]

Multi- step reinforcement learning: A unifying algorithm,

K. De Asis, J. Hernandez-Garcia, G. Holland, and R. Sutton, “Multi- step reinforcement learning: A unifying algorithm,” in AAAI Conf. on Artificial Intelligence, 2018

2018
[17]

Multi-bellman operator for convergence of q-learning with linear function approximation,

D. S. Carvalho, P. A. Santos, and F. S. Melo, “Multi-bellman operator for convergence of q-learning with linear function approximation,” arXiv preprint arXiv:2309.16819 , 2023

2023 arXiv
[18]

A novel multi-step q-learning method to improve data efficiency for deep reinforcement learning,

Y . Yuan, Z. L. Yu, Z. Gu, Y . Yeboah, W. Wei, X. Deng, J. Li, and Y . Li, “A novel multi-step q-learning method to improve data efficiency for deep reinforcement learning,” Knowledge-Based Systems, 2019

2019
[19]

Rainbow: Combining improvements in deep reinforcement learning,

M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in AAAI Conf. on Artificial Intelligence , 2018

2018
[20]

Policy invariance under reward transformations: Theory and application to reward shaping,

A. Y . Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in Int. Conf. on Machine Learning , 1999

1999
[21]

Self- supervised online reward shaping in sparse-reward environments,

F. Memarian, W. Goo, R. Lioutikov, S. Niekum, and U. Topcu, “Self- supervised online reward shaping in sparse-reward environments,” in IEEE/RSJ Int. Conf. on Intelligent Robots and Systems , 2021

2021
[22]

On learning intrinsic rewards for policy gradient methods,

Z. Zheng, J. Oh, and S. Singh, “On learning intrinsic rewards for policy gradient methods,” Advances in Neural Information Processing Systems, 2018

2018
[23]

Microfoun- dations of discounting,

A. Adamou, Y . Berman, D. Mavroyiannis, and O. Peters, “Microfoun- dations of discounting,” Decision Analysis, 2021

2021
[24]

The time interpretation of expected utility theory,

O. Peters and A. Adamou, “The time interpretation of expected utility theory,” arXiv preprint arXiv:1801.03680 , 2018

2018 arXiv
[25]

On-policy deep reinforcement learning for the average-reward criterion,

Y . Zhang and K. W. Ross, “On-policy deep reinforcement learning for the average-reward criterion,” in Int. Conf. on Machine Learning , 2021

2021
[26]

A provably-efficient model-free algo- rithm for infinite-horizon average-reward constrained markov decision processes,

H. Wei, X. Liu, and L. Ying, “A provably-efficient model-free algo- rithm for infinite-horizon average-reward constrained markov decision processes,” in AAAI Conf. on Artificial Intelligence , 2022

2022
[27]

Robust average-reward markov decision processes,

Y . Wang, A. Velasquez, G. Atia, A. Prater-Bennette, and S. Zou, “Robust average-reward markov decision processes,” in AAAI Conf. on Artificial Intelligence , 2023

2023
[28]

Proof of the ergodic theorem,

G. D. Birkhoff, “Proof of the ergodic theorem,” Proceedings of the National Academy of Sciences , 1931

1931
[29]

D. A. Levin and Y . Peres, Markov Chains and Mixing Times . Amer- ican Mathematical Soc., 2017

2017
[30]

Finite-time analysis of natural actor- critic for POMDPs,

S. Cayci, N. He, and R. Srikant, “Finite-time analysis of natural actor- critic for POMDPs,” SIAM Journal on Mathematics of Data Science , 2024

2024
[31]

The ergodicity solution of the cooperation puzzle,

O. Peters and A. Adamou, “The ergodicity solution of the cooperation puzzle,” Philosophical Transactions of the Royal Society A , 2022

2022
[32]

OpenAI Gym,

G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, “OpenAI Gym,” arXiv preprint arXiv:1606.01540, 2016

2016 arXiv
[33]

Playing Atari with deep reinforce- ment learning,

V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with deep reinforce- ment learning,” arXiv preprint arXiv:1312.5602 , 2013

2013 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Human- level control through deep reinforcement learning,

V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, et al., “Human- level control through deep reinforcement learning,” Nature, 2015

work page 2015

[2] [2]

Benchmarking deep reinforcement learning for continuous control,

Y . Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Benchmarking deep reinforcement learning for continuous control,” in Int. Conf. on Machine Learning , 2016

work page 2016

[3] [3]

Research priorities for robust and beneficial artificial intelligence,

S. Russell, D. Dewey, and M. Tegmark, “Research priorities for robust and beneficial artificial intelligence,” AI Magazine, 2015

work page 2015

[4] [4]

Concrete problems in AI safety,

D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Man ´e, “Concrete problems in AI safety,” arXiv preprint arXiv:1606.06565, 2016

arXiv 2016

[5] [5]

Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures,

J. Uesato, A. Kumar, C. Szepesvari, T. Erez, A. Ruderman, K. An- derson, N. Heess, P. Kohli, et al. , “Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures,” arXiv preprint arXiv:1812.01647, 2018

arXiv 2018

[6] [6]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduc- tion. MIT press, 2018

2018

[7] [7]

Bertsekas, Reinforcement Learning and Optimal Control

D. Bertsekas, Reinforcement Learning and Optimal Control . Athena Scientific, 2019

2019

[8] [8]

Reinforcement learning with non-ergodic reward in- crements: robustness via ergodicity transformations,

D. Baumann, E. Noorani, J. Price, O. Peters, C. Connaughton, and T. B. Sch ¨on, “Reinforcement learning with non-ergodic reward in- crements: robustness via ergodicity transformations,” Transactions on Machine Learning Research , 2025

work page 2025