Pith. sign in

REVIEW 3 major objections 4 minor 28 references

Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that a robust average-reward Bellman operator with no discount factor still contracts strictly in a designed semi-norm, yielding Q-learning and actor-critic algorithms with $\widetilde{O}(\epsilon^{-2})$ sample complexity.

desk verdict The uniform critic bounds and actor-critic analysis look genuinely useful, but the central contraction proof for Q-learning has a real gap in the max-operator step that needs attention before the headline Q-learning result is established. read the letter →

arxiv 2506.07040 v4 pith:6SZWATWX submitted 2025-06-08 cs.LG cs.AIstat.ML

Yang Xu , Swetha Ganesh , Vaneet Aggarwal This is my paper
classification cs.LGcs.AIstat.ML MSC 68Q3290C40
keywords distributionallyrobustMDPaverage-rewardreinforcementlearningQ-learningactor-criticsamplecomplexityBellmanoperatorcontaminationuncertaintyWasserstein
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes the first non-asymptotic sample-complexity guarantees for model-free control in distributionally robust infinite-horizon average-reward MDPs. It proves that the optimal robust Bellman operator, which has no discount factor, is nevertheless a strict contraction under a specially constructed semi-norm, provided the nominal kernels mix uniformly and the uncertainty radius is not too large. That contraction lets a synchronous Q-learning update learn the robust optimal Q-function in $\widetilde{O}(\epsilon^{-2})$ samples, and a matching actor-critic scheme learns an $\epsilon$-optimal robust policy in the same order, with critic error bounds that hold uniformly across all policies. This matters because planners can now certify robust long-run policies from simulation data of a single nominal model.

What carries the argument

The load-bearing object is the semi-norm $\|\cdot\|_H$ defined on Q-functions as a maximum over actions of an extremal norm applied to fluctuation matrices $F = P^\pi - E_P$, plus a small infimal-convolution term that quotients out constant shifts. Under the uniform minorization assumption (Assumption A.1) and the stated radius restrictions, the joint spectral radius of all such fluctuation matrices is strictly below one, which yields the Dobrushin-coefficient bound and the extremal norm; the max over actions is non-expansive in this semi-norm, and the three-step factorization $Q \mapsto V_Q \mapsto T(V_Q) \mapsto HQ$ composes to a strict contraction with $\gamma_H = \alpha + \epsilon < 1$. This contraction is what converts the robust Bellman equation into a contractive stochastic approximation with bounded noise, and it is also what makes the critic bounds uniform over all stationary policies.

What would settle it

Take a two-state nominal MDP with two deterministic policies, one forcing the chain to stay in state 1 and the other forcing it to stay in state 2; then no length-$L$ product can place a common $\epsilon_L \nu$ mass under all policy sequences, so Assumption A.1 fails. Run the anchored robust Q-learning update under a small contamination radius on this MDP: if the iterates fail to converge to the robust fixed point (or the anchored error does not decrease at the predicted rate), the contraction mechanism is doing the work; if they still converge, then the stated sufficient condition is not necessary.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the absence of discounting does not doom contraction: for contamination, total-variation, and Wasserstein uncertainty sets, the optimal robust average-reward Bellman operator $H$ contracts strictly in a semi-norm $\|\cdot\|_H$ that kills constant shifts, with a uniform factor $\gamma_H < 1$ across all deterministic policies and admissible kernels. This one-step contraction (Theorem 4.3) is what makes the stochastic-approximation Q-learning update converge, yielding $\mathbb{E}[\|Q_T - Q^*\|_\infty^2] = O(1/T^2 + t_{\mathrm{mix}}^2/(T(1-\gamma_H)^2))$ and an $\widetilde{O}(\epsilon^{-2})$ sample complexity for an $\epsilon$-optimal robust Q-function (Theorem 4.4). The same machinery, extended to policy evaluation with policy-independent constants, feeds a robust mirror-ascent actor that returns an $\epsilon$-optimal robust policy within $\widetilde{O}(\epsilon^{-2})$ total samples (Theorem 5.6).

Load-bearing premise

The load-bearing premise is Assumption A.1, that the nominal kernels, under every deterministic policy sequence, spread at least a fixed small amount of probability onto a common distribution within $L$ steps, together with the stated radius limits for TV and Wasserstein sets; if that uniform minorization fails, the strict contraction factor $\gamma_H < 1$ in Theorem 4.3 is not established.

Editorial extensions

If this is right

  • A synchronous robust Q-learning update with step sizes $\eta_t = O(1/t)$ learns the $\epsilon$-optimal robust Q-function using $O(SA t_{\mathrm{mix}}^2/(\epsilon^2 (1-\gamma_H)^2))$ samples for contamination sets and $\widetilde{O}(\cdot)$ samples for TV and Wasserstein sets, which is order-optimal in $\epsilon$.
  • The robust actor-critic runs $K = O(\log(1/\epsilon))$ policy updates with an inexact critic satisfying $\mathbb{E}\|\hat{Q}^\pi - Q^\pi\|_\infty \le \epsilon$ and returns an $\epsilon$-optimal robust policy with $\widetilde{O}(\epsilon^{-2})$ total simulator samples.
  • Critic estimation (value function, average reward, and Q-function) converges simultaneously for all stationary policies with policy-independent constants, so the critic can be re-solved at each actor step without degrading the overall rate.
  • For contamination uncertainty, no smallness restriction on the radius is needed; for TV and Wasserstein uncertainty, the radius must lie below explicit thresholds tied to the mixing parameter and the metric's minimum state distance.
  • The contamination, TV, and Wasserstein support-function estimators fit into one algorithm template, sharing the same stochastic-approximation analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The uniform contraction argument does not obviously require these three specific uncertainty sets; any rectangular set whose worst-case transition family keeps the joint spectral radius of fluctuation matrices below one should inherit $\widetilde{O}(\epsilon^{-2})$ Q-learning and actor-critic guarantees.
  • If Assumption A.1 fails but the MDP is still ergodic policy-by-policy, the semi-norm contraction may fail even though robust average-reward values exist; a practical diagnostic is to measure the Dobrushin coefficient of length-$L$ products from simulator data before deploying these algorithms.
  • The mismatch factor $M$ and mixing time $t_{\mathrm{mix}}$ suggest that performance degrades with state-space diameter and policy aggressiveness; variance-reduced estimators or multi-level Monte Carlo could plausibly shave the $N_{\max}$ factors in the TV and Wasserstein bounds.
  • These finite-sample results create a testable prediction: in queueing or ride-hailing simulators with rare reset episodes, both algorithms should show the predicted $\epsilon^{-2}$ scaling, while removing the reset mechanism should break the guaranteed rate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript studies model-free distributionally robust reinforcement learning for infinite-horizon average-reward MDPs. It proposes a robust Q-learning algorithm and a robust actor-critic algorithm for contamination, total-variation, and Wasserstein uncertainty sets, and claims sample complexity tilde-O(epsilon^{-2}) for returning an epsilon-optimal robust Q-function or policy. The analysis is built on a claimed strict contraction of the optimal robust Bellman operator in a specially constructed semi-norm, and on uniform finite-sample robust TD bounds for the critic. Numerical experiments on two small MDPs are provided.

Significance. If the contraction result and the subsequent stochastic-approximation analyses were correct, the paper would be a substantial contribution: it would provide the first non-asymptotic guarantees for model-free Q-learning and actor-critic in robust average-reward MDPs, with near-optimal epsilon-dependence, and would introduce a reusable semi-norm contraction framework. The paper is also careful to cite prior work by the same group and others, and it provides explicit radius restrictions for the three uncertainty classes. However, as discussed below, the central contraction proof contains an invalid inequality, and the theorem statements do not match the proven mode of convergence; these issues currently prevent the claimed advances from being established.

major comments (3)
  1. [Theorem 4.3 vs. Assumption A.1] The proof of Theorem 4.3 requires the inequality sup_{F in F} ||F (V_Q1 - V_Q2)||_ext <= max_{a in A} sup_{F in F} ||F (Q1 - Q2)(.,a)||_ext. This inequality is asserted in the final step of the proof and is false in general. To see this, take S={1,2,3}, A={1,2}, Q2=0, Q1(.,1)=[0,10,0]^T, Q1(.,2)=[0,0,10]^T, so V_Q1 - V_Q2 = [0,10,10]^T, and let the nominal kernel be P = [[0.9,0.05,0.05],[0.1,0.8,0.1],[0.1,0.1,0.8]], which satisfies Assumption A.1 with L=1 and epsilon_L=0.2. With F={P-E_P} (e.g., an action-independent MDP with contamination radius delta=0, which is allowed by the paper's radius conditions), direct computation gives ||(P-E_P)(V_Q1 - V_Q2)||_2 = sqrt(48) ≈ 6.93, whereas max_a ||(P-E_P)(Q1 - Q2)(.,a)||_2 = sqrt(36.5) ≈ 6.04. For the extremal norm constructed in (48) with alpha in (0.8,1), the ordering persists because both sequences decay with rate equal to the second-largest eigenvalue of P. Hence the step bounding sup_F ||F(V_Q1 - V_Q2)||_ext by max_a sup_F ||F(Q1 - Q2)(.,a)||_ext is invalid, and the claimed strict contraction gamma_H < 1 is not established by the written proof. The proof supplies no compensating argument using the epsilon term of the H semi-norm; it simply drops it. Since Theorem 4.3 is the engine for both Theorems 4.4 and 5.6, this gap is load-bearing.
  2. [Theorem 4.4 / Theorem A.8] Theorem 4.3 is stated under Assumption 4.2 only, but the proof in Appendix A relies crucially on Assumption A.1 (uniform L-step minorization for products of nominal kernels over all deterministic policy sequences). This assumption is not stated in Theorem 4.3 or in Section 4, and it is not implied by Assumption 4.2: the latter only requires each individual induced center kernel to be irreducible and aperiodic, which does not control products of different policies. The radius restrictions in Lemmas A.3-A.5 (e.g., delta < epsilon_L/(2L) for TV sets) are expressed in terms of epsilon_L from Assumption A.1, and the joint spectral radius bound in (25)-(28) fails without a uniform minorization. Thus the theorem's hypotheses are incomplete; Assumption A.1 must be incorporated into the theorem statement or derived from Assumption 4.2.
  3. [Section 5 / Theorem B.7] The formal convergence result in Appendix A (Theorem A.8) is an expectation bound: E[||Q_T - Q*||_inf^2] <= ... . The main text of Theorem 4.4 is phrased in terms of sample complexity to ensure epsilon convergence, and the proof sketch states that the guarantee holds with high probability. These are different statements; applying Markov's inequality to the expectation bound does not yield a high-probability guarantee of epsilon-accurate output with the stated sample complexity unless an additional probability parameter appears in the bound. The same mismatch occurs in Remark 5.3 versus Theorem B.7, where the remark claims with high probability but the theorem gives expectation bounds. The authors should state the precise mode of convergence and, if high-probability guarantees are claimed, provide them.
minor comments (4)
  1. [Section 3, Eqs. (11)-(14)] The estimators for sigma are introduced before Algorithm 4 is defined; Algorithm 4 is only presented in Appendix A. Please point to it explicitly as Algorithm 4 in the main text.
  2. [Appendix B, Lemma B.5] The notation ||x||_{inf,0} is used without definition; define it as min_c ||x - c e||_inf before first use.
  3. [Proof of Theorem 4.4] The constants c_H, C_H, c_u, c_l, rho_2, and G are used in the main text but defined only later in Appendix C; this makes the formal bounds difficult to verify.
  4. [Throughout the manuscript] There are several LaTeX artifacts in the text (e.g., '/bracehtipupleft/bracehtipdownright', 'stepsize stepsize', and 'est' for 'ext') that should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the contraction factor is derived from uniform mixing and radius restrictions, not assumed or fitted.

full rationale

The paper's central Theorem 4.3 constructs a policy- and kernel-dependent extremal semi-norm and derives gamma_H < 1 from Assumption A.1 (uniform L-step minorization) and the radius restrictions in Lemmas A.3-A.5, which force the joint spectral radius of the fluctuation family F below one. This is a derivation, not an assumption of the conclusion. The Q-learning and actor-critic sample-complexity theorems are consequences of Theorem 4.3 and standard stochastic-approximation arguments, with no parameter fitted to data. The paper does rely on same-group prior work (Xu, Mondal, and Aggarwal 2025) for the single-policy evaluation semi-norm, the truncated MLMC estimator, and its bias/variance bounds, but those are cited tools with stated assumptions that do not include the target optimal-control contraction or the actor-critic bound; they are not restatements of the paper's main results. The skeptical concern about the max-operator non-expansiveness step in the proof of Theorem 4.3 is a correctness or invalid-proof issue, not a circularity, because the claimed reduction is not equivalent to an input or fit. No self-definitional pattern or fitted-input-as-prediction pattern is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The contraction arguments depend on hand-chosen constants alpha and epsilon defining the semi-norm, and on uniform ergodicity and minorization assumptions over policy classes. None of these are fitted to data; they are analytical parameters chosen to make the proof work. The paper introduces no new physical or ontological entities.

free parameters (2)
  • semi-norm parameter alpha
    Chosen in (r*, 1) to construct the extremal norm in Lemma A.6; it controls the contraction factor gamma_H = alpha + epsilon and enters all norm-equivalence constants.
  • semi-norm parameter epsilon
    Chosen in (0, 1 - alpha) in the definition of the semi-norm ||.||_H; it sets the contraction factor gamma_H and affects the constants c_H, C_H in the norm equivalence.
assumptions (4)
  • domain assumption Finite tabular MDP with (s,a)-rectangular compact uncertainty sets and a known deterministic reward function.
    Used throughout Section 3 to define the robust average reward, uncertainty sets, and Bellman operators.
  • domain assumption Assumptions 4.2 and 5.4: the nominal transition kernels induced by all policies are irreducible and aperiodic.
    Stated in Sections 4 and 5; ensures uniform geometric ergodicity, finite mixing times, and a well-defined robust average reward.
  • ad hoc to paper Assumption A.1: uniform L-step minorization of products of nominal kernels over all deterministic policy sequences.
    Introduced in Appendix A to upper-bound the joint spectral radius of the fluctuation matrices; it is load-bearing for Theorem 4.3 and is not stated in the main text.
  • standard math Dobrushin coefficient and joint spectral radius bounds, plus Moreau envelope machinery from cited prior work.
    Used in Appendices A and B to convert mixing estimates into contraction of the robust Bellman operator; follows Berger and Wang, Zhang et al. 2021, and Chen et al. 2020.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning." pith.science (2026). https://pith.science/paper/6SZWATWX

@misc{pith2026250607040,
  author       = {Pith},
  title        = {Pith review of: Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SZWATWX}},
  note         = {Machine review of arXiv:2506.07040}
}
abstract

We study model-free methods for distributionally robust infinite-horizon average-reward Markov decision processes (MDPs). We present non-asymptotic convergence analyses of Q-learning and actor-critic algorithms for robust average-reward MDPs under contamination, total-variation distance, and Wasserstein uncertainty sets. A key ingredient of our analysis is showing that the optimal robust Bellman operator is a strict contraction with respect to a carefully designed semi-norm. This property enables a stochastic approximation update that learns the optimal robust $Q$-function with $\tilde{\mathcal{O}}(\epsilon^{-2})$ dependence on the target accuracy. We also establish robust TD convergence bounds whose constants are uniform over all stationary policies, yielding an efficient data-driven routine for robust critic estimation. Building on this, we introduce an actor-critic algorithm that learns an $\epsilon$-optimal robust policy with $\tilde{\mathcal{O}}(\epsilon^{-2})$ dependence on the target accuracy. We provide numerical simulations to illustrate the qualitative behavior of the proposed algorithms. Our results contribute to the theoretical foundations of robust planning under model misspecification and to model-free approaches for building robust long-run policies directly from simulation data.

Figures

Figures reproduced from arXiv: 2506.07040 by the authors.

Figure 1
Figure 1. Ride-hailing via robust Q-learning 0 5000 10000 15000 20000 25000 30000 35000 40000 Total simulator samples 0.60 0.65 0.70 0.75 0.80 0.85 Robust average reward g Contamination =0.005 =0.01 =0.02 g * ( =0.005) g * ( =0.01) g * ( =0.02) 0 5000 10000 15000 20000 25000 30000 35000 40000 Total simulator samples 0.60 0.65 0.70 0.75 0.80 0.85 Robust average reward g Total Variation =0.005 =0.01 =0.02 g * ( =0.005) g * ( =0… view at source ↗
Figure 2
Figure 2. Control loop via robust actor-critic g π ∗ P −g πk P that decays at rate O(ϵ) after K = Θ(log(1/ϵ)) iterations; full details are given in Appendix B. This matches, up to logarithmic factors, the O˜(ϵ −2 ) sample complexity required to obtain an ϵ-accurate critic in Theorem 5.2, and improves on the oracle-gradient setting of Sun et al. (2024); Wang et al. (2025), where access to Qπ (or its robust gradient) is assumed… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

  1. [1]

    On the theory of policy gradient methods: Optimality, approximation, and distribution shift

    Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1--76, 2021

    work page 2021

  2. [2]

    Bounded semigroups of matrices

    Marc A Berger and Yang Wang. Bounded semigroups of matrices. Linear Algebra and its Applications , 166:21--27, 1992

    work page 1992

  3. [3]

    Single sample path-based optimization of markov chains

    Xi-Ren Cao. Single sample path-based optimization of markov chains. Journal of optimization theory and applications , 100:527--548, 1999

    work page 1999

  4. [4]

    Sample complexity of distributionally robust average-reward reinforcement learning

    Zijun Chen, Shengbo Wang, and Nian Si. Sample complexity of distributionally robust average-reward reinforcement learning. arXiv preprint arXiv:2505.10007 , 2025

    arXiv 2025

  5. [5]

    Distributionally robust stochastic optimization with W asserstein distance

    Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with W asserstein distance. Mathematics of Operations Research , 2022

    work page 2022

  6. [6]

    Distributionally robust stochastic optimization with wasserstein distance

    Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research , 48(2):603--655, 2023

    work page 2023

  7. [7]

    Sim2real in robotics and automation: Applications and challenges

    Sebastian H \"o fer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, et al. Sim2real in robotics and automation: Applications and challenges. IEEE transactions on automation science and engineering , 18(2):398--400, 2021

    work page 2021

  8. [8]

    A robust version of the probability ratio test

    Peter J Huber. A robust version of the probability ratio test. The Annals of Mathematical Statistics , pages 1753--1758, 1965

    work page 1965

Show all 28 references
  1. [9]

    Robust dynamic programming

    Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2):257--280, 2005

    2005

  2. [10]

    Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018

    Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018

    2018

  3. [11]

    Learning robust policy against disturbance in transition dynamics via state-conservative policy optimization

    Yufei Kuang, Miao Lu, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Learning robust policy against disturbance in transition dynamics via state-conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7247--7254, 2022

    2022

  4. [12]

    Policy gradient for rectangular robust markov decision processes

    Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Y Levy, and Shie Mannor. Policy gradient for rectangular robust markov decision processes. Advances in Neural Information Processing Systems , 36:59477--59501, 2023

    2023

  5. [13]

    First-order policy optimization for robust markov decision process

    Yan Li, Guanghui Lan, and Tuo Zhao. First-order policy optimization for robust markov decision process. arXiv preprint arXiv:2209.10579 , 2022

    2022 arXiv

  6. [14]

    Reinforcement learning in robust markov decision processes

    Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust markov decision processes. Advances in Neural Information Processing Systems , 26, 2013

    2013

  7. [15]

    Robustness in markov decision problems with uncertain transition matrices

    Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices. Advances in neural information processing systems , 16, 2003

    2003

  8. [16]

    A finite-sample analysis of distributionally robust average-reward reinforcement learning

    Zachary Roch, Chi Zhang, George Atia, and Yue Wang. A finite-sample analysis of distributionally robust average-reward reinforcement learning. arXiv preprint arXiv:2505.12462 , 2025

    2025 arXiv

  9. [17]

    Policy optimization for robust average reward mdps

    Zhongchang Sun, Sihong He, Fei Miao, and Shaofeng Zou. Policy optimization for robust average reward mdps. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  10. [18]

    u nderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, J \

    Niko S \"u nderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, J \"u rgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, et al. The limits and potentials of deep learning for robotics. The International journal of robotics research , 37(4...

    2018

  11. [19]

    Near sample-optimal reduction-based policy learning for average reward mdp

    Jinghan Wang, Mengdi Wang, and Lin F Yang. Near sample-optimal reduction-based policy learning for average reward mdp. arXiv preprint arXiv:2212.00603 , 2022

    2022 arXiv

  12. [20]

    A finite sample complexity bound for distributionally robust q-learning

    Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. A finite sample complexity bound for distributionally robust q-learning. In International Conference on Artificial Intelligence and Statistics , pages 3370--3398. PMLR, 2023

    2023

  13. [21]

    Sample complexity of variance-reduced distributionally robust q-learning

    Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. Sample complexity of variance-reduced distributionally robust q-learning. Journal of Machine Learning Research , 25(341):1--77, 2024

    2024

  14. [22]

    Robust average-reward markov decision processes

    Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, and Shaofeng Zou. Robust average-reward markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 15215--15223, 2023

    2023

  15. [23]

    Robust average-reward reinforcement learning

    Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, and Shaofeng Zou. Robust average-reward reinforcement learning. Journal of Artificial Intelligence Research , 80:719--803, 2024

    2024

  16. [24]

    Model-free robust average-reward reinforcement learning

    Yue Wang, Alvaro Velasquez, George K Atia, Ashley Prater-Bennette, and Shaofeng Zou. Model-free robust average-reward reinforcement learning. In International Conference on Machine Learning , pages 36431--36469. PMLR, 2023

    2023

  17. [25]

    Policy gradient method for robust reinforcement learning

    Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International conference on machine learning , pages 23484--23526. PMLR, 2022

    2022

  18. [26]

    Model-free reinforcement learning in infinite-horizon average-reward markov decision processes

    Chen-Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. Model-free reinforcement learning in infinite-horizon average-reward markov decision processes. In International conference on machine learning , pages 10170--10180. PMLR, 2020

    2020

  19. [27]

    Finite-sample analysis of policy evaluation for robust average reward reinforcement learning

    Yang Xu, Washim Uddin Mondal, and Vaneet Aggarwal. Finite-sample analysis of policy evaluation for robust average reward reinforcement learning. arXiv preprint arXiv:2502.16816 , 2025

    2025

  20. [28]

    Natural actor-critic for robust reinforcement learning with function approximation

    Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, PR Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. Advances in neural information processing systems , 36, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.