Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that optimizing a pessimistic lower-bound proxy—not an optimistic one—guarantees a bounded suboptimality gap for zero-shot transfer reinforcement learning and inherently avoids negative transfer.

desk verdict Solid conservative-proxy construction with a correct lower-bound lemma, but the advertised monotonic no-negative-transfer claim does not follow from the math. read the letter →

arxiv 2505.18447 v2 pith:DHNV5S3T submitted 2025-05-24 cs.LG

Chi Zhang , Ziying Jia , George K. Atia , Sihong He , Yue Wang This is my paper
classification cs.LG
keywords transferreinforcementlearningzero-shotpessimismprinciplerobustMarkovdecisionprocessnegativedistributedconservativeproxyBellmanoperator
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Zero-shot transfer reinforcement learning aims to deploy a policy in a target environment using only data from related source environments. This paper argues that the right objective is not a proxy that estimates target performance optimistically, but a conservative proxy $f(\pi) \le V^\pi_{P_0}$ that underestimates every policy's true target value. The central guarantee, Lemma 4.1, is that the transferred policy $\pi_f = \arg\max_\pi f(\pi)$ suffers a target suboptimality gap of at most $\max_\pi (V^\pi_{P_0} - f(\pi))$, the worst-case degree of pessimism, so less pessimistic conservative proxies monotonically improve the guarantee. Built on worst-case value functions under uncertainty sets around each source kernel, the paper constructs two such proxies—an averaged-operator proxy and a minimal-pessimism proxy—and gives distributed algorithms that converge to them while sharing only Q-tables. If the framework's assumptions hold, transfer is safe by construction and negative transfer is ruled out despite no target-domain data.

What carries the argument

The load-bearing object is the pessimistic proxy: a functional $f(\pi) \le V^\pi_{P_0}$ whose optimization bounds the target gap through $\max_\pi (V^\pi_{P_0} - f(\pi))$. Concretely, the paper uses robust Bellman operators $T_k^\pi$ for each source domain with $(s,a)$-rectangular uncertainty sets containing the target kernel; the averaged-operator proxy is the fixed point of $T^\pi_{AO} = (1/K)\sum_k T^\pi_k$, and the minimal-pessimism proxy is the fixed point of $T^\pi_{MP} = \max_k T^\pi_k$. Both operators are $\gamma$-contractions, so their fixed points exist and can be approached by iterative updates, and the max operator's fixed point serves as a 'minimal pessimism' selector that up-weights informative sources. The distributed algorithms alternate local Q-table updates with global aggregation (average or max), which is enough because the averaged and maxed operators factor through per-source updates.

What would settle it

Run an exact tabular search over small MDPs: enumerate target kernels $P_0$ and source kernels $P_1,\dots,P_K$ satisfying $D(P_0,P_k)\le\Gamma$ for a known $\Gamma$, compute the true target values $V^\pi_{P_0}$ exactly, and check whether the optimal policy of the averaged-operator proxy ever violates the central conservatism claim $V^\pi_{AO}\le V^\pi_{P_0}$ for some policy $\pi$; any single violation would refute Theorem 5.2.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a theorem about proxy optimization: whenever a proxy $f$ satisfies $f(\pi) \le V^\pi_{P_0}$ for every policy $\pi$, the policy $\pi_f$ that maximizes $f$ is within $\|\zeta\| = \max_\pi (V^\pi_{P_0} - f(\pi))$ of the target-optimal policy, a gap that shrinks monotonically as the proxy becomes more accurate. It then shows that robust value functions under $(s,a)$-rectangular uncertainty sets centered at the source kernels are concrete conservative proxies whenever the target kernel $P_0$ lies inside every local uncertainty set. The averaged-operator proxy $V^\pi_{AO}$, defined as the unique fixed point of the average of the source robust Bellman operators, is provably conservative, dominates proximal robust domain randomization, and can be optimized by a distributed algorithm with rate $\tilde{O}(1/(TK) + (E-1)\Gamma/T)$. The minimal-pessimism proxy $V^\pi_{MP}$, the fixed point of a max of robust Bellman operators, is shown to dominate both the averaged proxy and every single-source robust value, so maximizing it selects the most informative source domains and thereby avoids negative transfer. Distributed algorithms MDTL-Avg and MDTL-Max converge to these fixed points with partial linear speedup, and an unbiased multilevel Monte-Carlo aggregation scheme makes the max operator stochastically unbiased.

Load-bearing premise

The load-bearing premise is that the learner knows a number $\Gamma$ with $D(P_0, P_k) \le \Gamma$ for every source $k$, so the target kernel $P_0$ lies inside each local uncertainty set; if $\Gamma$ is unknown or set too loosely, the conservative guarantee becomes vacuous or extremely conservative, and the theory also assumes identical rewards across source and target domains.

Editorial extensions

If this is right

  • Deploying the MDTL-Avg policy guarantees the target value is at least the optimized conservative proxy, with the optimality gap bounded by the level of pessimism.
  • With the known bound $\Gamma$ on source-target kernel distance, both algorithms converge at $\tilde{O}(1/(TK) + (E-1)\Gamma/T)$, giving partial linear speedup in the number of source domains.
  • Because $V^\pi_{AO} \le V^\pi_{MP} \le V^\pi_{P_0}$, switching from the averaged proxy to the minimal-pessimism proxy can only improve or preserve the transfer guarantee.
  • The minimal-pessimism proxy automatically down-weights dissimilar sources, so adding a poor source domain cannot degrade the transferred policy.
  • With the model-free variant, the same guarantee holds with high probability at rate $\tilde{O}((E-1)\Gamma/(T(1-\gamma)^3) + \sqrt{\log(SATK/\delta)}/((1-\gamma)^3\sqrt{TK}))$.
  • The monotonic gap bound suggests a practical adaptive scheme: any small amount of target data or a tighter estimate of $\Gamma$ can be fed back to shrink the uncertainty-set radii, improving the transferred policy's worst-case gap without changing the algorithm family.
  • The known-$\Gamma$ assumption could be replaced by an online estimation procedure that starts conservative and tightens the radii as similarity information accumulates; the paper's monotonicity guarantee implies such tightening never hurts.
  • The same operator-level pessimism recipe—average or max of robust Bellman operators—could apply to other transfer settings, such as sim-to-real with a family of simulators, where each simulator is a source domain and the real environment is the unobserved target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The monotonic gap bound suggests a practical adaptive scheme: any small amount of target data or a tighter estimate of $\Gamma$ can be fed back to shrink the uncertainty-set radii, improving the transferred policy's worst-case gap without changing the algorithm family.
  • The known-$\Gamma$ assumption could be replaced by an online estimation procedure that starts conservative and tightens the radii as similarity information accumulates; the paper's monotonicity guarantee implies such tightening never hurts.
  • The same operator-level pessimism recipe—average or max of robust Bellman operators—could apply to other transfer settings, such as sim-to-real with a family of simulators, where each simulator is a source domain and the real environment is the unobserved target.
  • There is a direct tie to offline RL: pessimism there fights distribution shift in data, while here it fights model mismatch across domains, so a unified 'pessimism level' calculus could connect both literatures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a pessimism-based framework for zero-shot transfer reinforcement learning. It constructs conservative proxies f(pi) that lower-bound the target-domain value V^pi_P0, proves a suboptimality bound in Lemma 4.1 in terms of the maximum pessimism, and builds two distributed algorithms (MDTL-Avg and MDTL-Max) around averaged and max-aggregated robust Bellman operators. The manuscript claims that this framework guarantees an optimized lower bound on target performance and, crucially, that it exhibits monotonic improvement with source-domain quality, thereby avoiding negative transfer. Convergence rates of order O~(1/(TK)+(E-1)Gamma/T) are stated for both algorithms, with experiments on recycling robot, FrozenLake, CartPole, HPC, and DVRP domains.

Significance. The lower-bound perspective is a natural and potentially useful way to connect robust RL with transfer learning, and the fixed-point results for the averaged and max operators (Theorems 5.2, 5.6, 6.1) are correctly argued given the assumption that the target kernel lies in every local uncertainty set. The distributed formulations and the explicit convergence proofs are also valuable: they go beyond the informal guarantees typical of domain randomization and provide a concrete algorithmic template. However, the paper's headline claim of monotonic improvement and avoidance of negative transfer is not supported by the proven results, and the convergence theorem contains a constraint that makes the advertised communication-efficiency tradeoff vacuous for K>=2. The framework's genuine contribution is the conservative lower-bound guarantee plus the algorithmic machinery; the stronger transfer-improvement claims need either additional assumptions or removal.

major comments (2)
  1. [Section 4.2 (Lemma 4.1), abstract] The claim that Lemma 4.1 establishes a 'monotonic dependence' between the pessimism level and the transferred policy's target performance is not implied by the lemma, and it is false as a general statement. Lemma 4.1 only bounds V^pi*_P0 - V^pi_f_P0 by max_pi (V^pi_P0 - f(pi)); it does not say that if f2 >= f1 pointwise, then the argmax of f2 has higher V_P0 than the argmax of f1. A concrete counterexample: let target values be V_P0(A)=1 and V_P0(B)=0.95, with f1(A)=0.5, f1(B)=0.2, f2(A)=0.7, f2(B)=0.9. Then f1 <= f2 <= V_P0 pointwise, argmax f1 is A with target value 1, argmax f2 is B with target value 0.95, while max_pi (V_P0 - f) drops from 0.75 to 0.3. Thus the bound in Lemma 4.1 can improve while the deployed policy's target value worsens. Consequently, the statements in Section 4.2, Section 6, and the abstract that a less conservative proxy 'directly translates into better performance' and that the method 'avoids negative transfer' are unsupported by the theory. The paper needs either a strictly stronger theorem (e.g., under additional structure on the proxy family) or a reformulation of the contribution as a lower-bound guarantee only, with negative-transfer avoidance supported empirically.
  2. [Theorems 5.7 and 6.4] The stated conditions on the number of local update steps make the claimed communication-efficiency tradeoff vacuous. Both theorems require E-1 <= min{ (1/lambda) * gamma/(1-gamma), 1/K } with lambda = 4 log^2(TK)/(T(1-gamma)). For any K>=2, the second term is at most 1/2, and since E is an integer, the condition forces E=1. The convergence bound then reduces to O~(1/(TK)) with no (E-1)Gamma/T term, and the 'reduce the aggregation frequency by E times' claim in Section 5.2 is not permitted under the theorem's own assumptions. As written, this is an internal inconsistency in the load-bearing convergence result; the constraint set or the rate expression must be corrected before the partial linear speedup and communication-efficiency claims can be evaluated.
minor comments (4)
  1. [Section 2.1] The notation for the domain-similarity assumption is inconsistent: the text writes Gamma >= D(P0||Pk) in one place and D(P0, Pk) elsewhere; the divergence should be defined once in the preliminaries and used consistently.
  2. [Section 4.1 (Lemma 4.1)] The proof of Lemma 4.1 is deferred to 'Appendix 4.1', but the proof actually appears in Appendix C; the cross-reference should be corrected.
  3. [Algorithm 1 and Section 6] The pseudocode line for MDTL-Max simply says 'Max-Aggregation' and the accompanying text refers to Appendix E.1 for the MLMC construction. For a self-contained paper, it would help to either state the MLMC estimator in the main text or give a precise pointer in the algorithm box.
  4. [Section 7 (Theorem 7.2)] Theorem 7.2 is labeled 'Informal' and then followed by a formal statement in Corollary G.7; the numbering and presentation could be aligned so that the main text states the formal result directly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central lower-bound guarantee follows from the stated conservatism assumption, and the cited external results are supporting rather than load-bearing; the monotonic-improvement claim is an inference gap, not a circular reduction.

full rationale

The paper's derived claims are consequences of the assumptions it states, not of the conclusions it advertises. Lemma 4.1 defines the pessimism slack as zeta^pi = V^pi_P0 - f(pi), assumes f(pi) <= V^pi_P0 for all pi, and takes pi_f to maximize f. Appendix C then proves V^pi*_P0 - V^pi_f_P0 <= V^pi*_P0 - f(pi*) <= max_pi zeta^pi using only these definitions and optimality of pi_f. This is a valid algebraic consequence, not a circular use of the conclusion. Theorem 5.2 and Theorem 6.1 establish V^pi_AO <= V^pi_P0 and V^pi_MP <= V^pi_P0 from the construction of uncertainty sets with P0 in each P_k; the robust-value definition already gives V^pi_Pk <= V^pi_P0, and the proofs for the averaged and max operators add contraction arguments that do not presuppose the theorem. The convergence theorems (5.7, 6.4, 7.2) are proved from the stated unbiased-estimator assumption and explicit recursion analyses; citations to Wang et al. 2023b, Kumar et al. 2023b, Yang et al. 2023, and Liu et al. 2022 support existence of unbiased robust Bellman estimators but are not the sole justification, and the self-cited items are not load-bearing for the paper's central transfer guarantee. No parameter is fitted to data and then renamed as a prediction, no uniqueness theorem is imported from the authors' prior work to force a choice, and no ansatz is disguised through citation. The paper does contain a logical overreach when it infers monotonic improvement and avoidance of negative transfer from the upper bound in Lemma 4.1; that pointwise increases in f can fail to increase the deployed value is a correctness concern rather than a circularity, so it does not affect the circularity score.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper's central proofs assume standard RMDP theory, a rectangular uncertainty structure, a known distributional-distance bound Gamma, identical rewards across domains, and access to unbiased robust Bellman-operator estimators. No new physical or formal entities are introduced.

free parameters (1)
  • Uncertainty radius Gamma = Unknown; assumed known or set to 1
    The entire conservative guarantee requires D(P0,Pk) <= Gamma for all k; the algorithm suboptimality bounds depend directly on Gamma. In practice the learner must supply it or fall back to the maximal value 1, which yields a very conservative policy.
assumptions (5)
  • domain assumption The target MDP shares the same state/action spaces and reward function with the source domains.
    Assumed in Section 2.1; the theory does not cover reward shifts or different action spaces.
  • domain assumption An upper bound Gamma >= D(P0||Pk) is known for all source domains k.
    Stated in Section 2.1; the conservativeness guarantees rely on P0 lying in each local uncertainty set.
  • domain assumption The uncertainty sets are (s,a)-rectangular and the distance D is total variation, Wasserstein, or l_p norm for the main results.
    Section 2 defines rectangular RMDPs; Proposition 5.4 and Remark 5.5 rely on these specific distance measures.
  • domain assumption Unbiased estimators of the robust Bellman operators are available to each local agent.
    Assumed in Section 5.2; needed for the stochastic convergence Theorems 5.7 and 6.4, following cited works.
  • standard math Contraction and fixed-point properties of robust Bellman operators are taken as known from standard RMDP theory.
    Used throughout Sections 5 and 6 to establish unique fixed points and convergence of the proposed operators.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning." pith.science (2026). https://pith.science/paper/DHNV5S3T

@misc{pith2026250518447,
  author       = {Pith},
  title        = {Pith review of: Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHNV5S3T}},
  note         = {Machine review of arXiv:2505.18447}
}
read the original abstract

Transfer reinforcement learning aims to derive a near-optimal policy for a target environment with limited data by leveraging abundant data from related source domains. However, it faces two key challenges: the lack of performance guarantees for the transferred policy, which can lead to undesired actions, and the risk of negative transfer when multiple source domains are involved. We propose a novel framework based on the pessimism principle, which constructs and optimizes a conservative estimation of the target domain's performance. Our framework effectively addresses the two challenges by providing an optimized lower bound on target performance, ensuring safe and reliable decisions, and by exhibiting monotonic improvement with respect to the quality of the source domains, thereby avoiding negative transfer. We construct two types of conservative estimations, rigorously characterize their effectiveness, and develop efficient distributed algorithms with convergence guarantees. Our framework provides a theoretically sound and practically robust solution for transfer learning in reinforcement learning.

Figures

Figures reproduced from arXiv: 2505.18447 by the authors.

Figure 2
Figure 2. Negative Transfer under FrozenLake Gym environment 9. Conclusion In this paper, we studied zero-shot transfer reinforcement learning and identified two critical limitations of existing methods: the lack of guarantees for the safety and per￾formance of transferred policies and the inability to miti￾gate negative transfer when multiple source domains are involved. To overcome these challenges, we incorporate a pessimi… view at source ↗
Figure 3
Figure 3. Robot: Values of MDTL-Avg under Different Uncertainty Levels [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Robot: Values of MDTL-Max under Different Uncertainty Levels [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: HPC Cluster Management Problem Our ablation study highlights the substantial advantage of our approaches in handling different levels of uncertainty in target domain. Across different levels of model uncertainty, our method consistently outperforms the non-robust basel…
Figure 6
Figure 6. Figure 6: HPC: Values of MDTL-Avg under Different Uncertainty Levels [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: HPC: Values of MDTL-Max under Different Uncertainty Levels [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Effect of Aggregation Bias for MDTL-Max B.5. Additional Experiments on Negative Transfer We further tested our MDTL-Avg and MDTL-Max algorithms on two additional environments, aiming to validate whether MDTL-Max can effectively mitigate negative transfer. For the CartP…
Figure 9
Figure 9. Figure 9: Effect of Negative Transfer under CartPole Gym environment [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Effect of Negative Transfer under Recycling Robot B.6. Comparison to Related Robust RL Approaches Note that our methods are based on distributionally robust RL, where our uncertainty set is constructed to account for the potential distributional shift. We thus compare…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Diagnosing Simulation and Hardware Barriers to Cross-Size Transfer in Equivariant Quantum Reinforcement Learning

    quant-ph 2025-10 reject novelty 6.0 of 10

    Abstract claims zero-shot 5-to-10-city EQC transfer beats target-size training only in exact simulation, degrading by 31.3% under sampling noise and 45.3% on hardware; the supplied body omits these experiments.

Reference graph

Works this paper leans on

100 extracted references · 52 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Provable benefits of representational transfer in reinforcement learning

    Agarwal, A., Song, Y., Sun, W., Wang, K., Wang, M., and Zhang, X. Provable benefits of representational transfer in reinforcement learning. In The Thirty Sixth Annual Conference on Learning Theory, pp.\ 2114--2187. PMLR, 2023

    2023

  3. [3]

    G., Aggarwal, V., Singh, A

    Arivazhagan, M. G., Aggarwal, V., Singh, A. K., and Choudhary, S. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019

    arXiv 1912

  4. [4]

    R., Schmidhuber, J., and Srivastava, R

    Arulkumaran, K., Ashley, D. R., Schmidhuber, J., and Srivastava, R. K. All you need is supervised learning: From imitation learning to meta-rl with upside down rl. arXiv preprint arXiv:2202.11960, 2022

    arXiv 2022

  5. [5]

    Badrinath, K. P. and Kalathil, D. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In Proc. International Conference on Machine Learning (ICML), pp.\ 511--520. PMLR, 2021

    2021

  6. [6]

    Contextualize me--the case for context in reinforcement learning

    Benjamins, C., Eimer, T., Schubert, F., Mohan, A., D \"o hler, S., Biedenkapp, A., Rosenhahn, B., Hutter, F., and Lindauer, M. Contextualize me--the case for context in reinforcement learning. arXiv preprint arXiv:2202.04500, 2022

    arXiv 2022

  7. [7]

    Dynamics generalisation in reinforcement learning via adaptive context-aware policies

    Beukman, M., Jarvis, D., Klein, R., James, S., and Rosman, B. Dynamics generalisation in reinforcement learning via adaptive context-aware policies. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    F., Eimer, T., Hutter, F., and Lindauer, M

    Biedenkapp, A., Bozkurt, H. F., Eimer, T., Hutter, F., and Lindauer, M. Dynamic algorithm configuration: Foundation of a new meta-algorithmic framework. In ECAI 2020, pp.\ 427--434. IOS Press, 2020

    2020

Show all 100 references
  1. [9]

    Blanchet, J. H. and Glynn, P. W. Unbiased M onte C arlo for optimization and functions of expectations via multi-level randomization. In 2015 Winter Simulation Conference (WSC), pp.\ 3656--3667. IEEE, 2015

    2015

  2. [10]

    H., Glynn, P

    Blanchet, J. H., Glynn, P. W., and Pei, Y. Unbiased multilevel M onte C arlo: Stochastic optimization, steady-state simulation, quantiles, and other applications. arXiv preprint arXiv:1904.09929, 2019

    1904 arXiv

  3. [11]

    Understanding domain randomization for sim-to-real transfer

    Chen, X., Hu, J., Jin, C., Li, L., and Wang, L. Understanding domain randomization for sim-to-real transfer. arXiv preprint arXiv:2110.03239, 2021

    2021 arXiv

  4. [12]

    Provable benefit of multitask representation learning in reinforcement learning

    Cheng, Y., Feng, S., Yang, J., Zhang, H., and Liang, Y. Provable benefit of multitask representation learning in reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 31741--31754, 2022

    2022

  5. [13]

    Near-optimal distributionally robust reinforcement learning with general l\_p norms

    Clavier, P., Shi, L., Le Pennec, E., Mazumdar, E., Wierman, A., and Geist, M. Near-optimal distributionally robust reinforcement learning with general l\_p norms. Advances in Neural Information Processing Systems, 37: 0 1750--1810, 2024

    2024

  6. [14]

    M., and Mahdavi, M

    Deng, Y., Kamani, M. M., and Mahdavi, M. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461, 2020

    2003 arXiv

  7. [15]

    Twice regularized MDP s and the equivalence between robustness and regularization

    Derman, E., Geist, M., and Mannor, S. Twice regularized MDP s and the equivalence between robustness and regularization. In Proc. Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  8. [16]

    An imitation from observation approach to transfer learning with dynamics mismatch

    Desai, S., Durugkar, I., Karnan, H., Warnell, G., Hanna, J., and Stone, P. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33: 0 3917--3929, 2020

    2020

  9. [17]

    Online policy optimization for robust mdp

    Dong, J., Li, J., Wang, B., and Zhang, J. Online policy optimization for robust mdp. arXiv preprint arXiv:2209.13841, 2022

    2022 arXiv

  10. [18]

    Personalized federated learning: A meta-learning approach

    Fallah, A., Mokhtari, A., and Ozdaglar, A. Personalized federated learning: A meta-learning approach. arXiv preprint arXiv:2002.07948, 2020

    2002 arXiv

  11. [19]

    Contextual markov decision processes

    Hallak, A., Di Castro, D., and Mannor, S. Contextual markov decision processes. arXiv preprint arXiv:1502.02259, 2015

    2015 arXiv

  12. [20]

    Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning

    Hua, J., Zeng, L., Li, G., and Ju, Z. Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning. Sensors, 21 0 (4): 0 1278, 2021

    2021

  13. [21]

    Offline multitask representation learning for reinforcement learning

    Ishfaq, H., Nguyen-Tang, T., Feng, S., Arora, R., Wang, M., Yin, M., and Precup, D. Offline multitask representation learning for reinforcement learning. Advances in Neural Information Processing Systems, 37: 0 70557--70616, 2024

    2024

  14. [22]

    Iyengar, G. N. Robust dynamic programming. Mathematics of Operations Research, 30 0 (2): 0 257--280, 2005

    2005

  15. [23]

    Variance reduced domain randomization for policy gradient, 2022

    Jiang, Y., Li, C., Dai, W., Zou, J., and Xiong, H. Variance reduced domain randomization for policy gradient, 2022. URL https://openreview.net/forum?id=vnF5gDNvcKX

    2022

  16. [24]

    Variance reduced domain randomization for reinforcement learning with policy gradient

    Jiang, Y., Li, C., Dai, W., Zou, J., and Xiong, H. Variance reduced domain randomization for reinforcement learning with policy gradient. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  17. [25]

    Federated reinforcement learning with environment heterogeneity

    Jin, H., Peng, Y., Yang, W., Wang, S., and Zhang, Z. Federated reinforcement learning with environment heterogeneity. In International Conference on Artificial Intelligence and Statistics, pp.\ 18--37. PMLR, 2022

    2022

  18. [26]

    Is pessimism provably efficient for offline RL ? arXiv preprint arXiv:2012.15085, 2020

    Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline RL ? arXiv preprint arXiv:2012.15085, 2020

    2012 arXiv

  19. [27]

    Probabilistic active meta-learning

    Kaddour, J., S mundsson, S., et al. Probabilistic active meta-learning. Advances in Neural Information Processing Systems, 33: 0 20813--20822, 2020

    2020

  20. [28]

    Khodadadian, S., Sharma, P., Joshi, G., and Maguluri, S. T. Federated reinforcement learning: Linear speedup under markovian sampling. In International Conference on Machine Learning, pp.\ 10997--11057. PMLR, 2022

    2022

  21. [29]

    Training robots without robots: deep imitation learning for master-to-robot policy transfer

    Kim, H., Ohmura, Y., Nagakubo, A., and Kuniyoshi, Y. Training robots without robots: deep imitation learning for master-to-robot policy transfer. IEEE Robotics and Automation Letters, 8 0 (5): 0 2906--2913, 2023

    2023

  22. [30]

    Understanding the effects of rlhf on llm generalisation and diversity

    Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., and Raileanu, R. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023 a

    2023 arXiv

  23. [31]

    A survey of zero-shot generalisation in deep reinforcement learning

    Kirk, R., Zhang, A., Grefenstette, E., and Rockt \"a schel, T. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 0 201--264, 2023 b

    2023

  24. [32]

    and Whiteson, S

    Koppejan, R. and Whiteson, S. Neuroevolutionary reinforcement learning for generalized helicopter control. In Proceedings of the 11th Annual conference on Genetic and evolutionary computation, pp.\ 145--152, 2009

    2009

  25. [33]

    Y., and Mannor, S

    Kumar, N., Derman, E., Geist, M., Levy, K. Y., and Mannor, S. Policy gradient for rectangular robust markov decision processes. Advances in Neural Information Processing Systems, 36: 0 59477--59501, 2023 a

    2023

  26. [34]

    An efficient solution to s-rectangular robust markov decision processes

    Kumar, N., Levy, K., Wang, K., and Mannor, S. An efficient solution to s-rectangular robust markov decision processes. arXiv preprint arXiv:2301.13642, 2023 b

    2023 arXiv

  27. [35]

    Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review

    Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., and Dresp-Langley, B. Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review. Robotics, 10 0 (1): 0 22, 2021

    2021

  28. [36]

    Micro: Model-based offline reinforcement learning with a conservative bellman operator

    Liu, X.-Y., Zhou, X.-H., Li, G.-T., Li, H., Gui, M.-J., Xiang, T.-Y., Huang, D.-X., and Hou, Z.-G. Micro: Model-based offline reinforcement learning with a conservative bellman operator. arXiv preprint arXiv:2312.03991, 2023

    2023 arXiv

  29. [37]

    and Xu, P

    Liu, Z. and Xu, P. Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation. arXiv preprint arXiv:2402.15399, 2024

    2024 arXiv

  30. [38]

    Distributionally robust Q -learning

    Liu, Z., Bai, Q., Blanchet, J., Dong, P., Xu, W., Zhou, Z., and Zhou, Z. Distributionally robust Q -learning. In International Conference on Machine Learning, pp.\ 13623--13643. PMLR, 2022

    2022

  31. [39]

    Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithm

    Lu, M., Zhong, H., Zhang, T., and Blanchet, J. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithm. arXiv preprint arXiv:2404.03578, 2024

    2024 arXiv

  32. [40]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.\ 1273--1282. PMLR, 2017

    2017

  33. [41]

    J., and Paull, L

    Mehta, B., Diaz, M., Golemo, F., Pal, C. J., and Paull, L. Active domain randomization. In Conference on Robot Learning, pp.\ 1162--1176. PMLR, 2020

    2020

  34. [42]

    Playing Atari with deep reinforcement learning

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  35. [43]

    Markov decision processes with continuous side information

    Modi, A., Jiang, N., Singh, S., and Tewari, A. Markov decision processes with continuous side information. In Algorithmic Learning Theory, pp.\ 597--618. PMLR, 2018

    2018

  36. [44]

    and La, H

    Nguyen, H. and La, H. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE international conference on robotic computing (IRC), pp.\ 590--595. IEEE, 2019

    2019

  37. [45]

    and El Ghaoui, L

    Nilim, A. and El Ghaoui, L. Robustness in M arkov decision problems with uncertain transition matrices. In Proc. Advances in Neural Information Processing Systems (NIPS), pp.\ 839--846, 2004

    2004

  38. [46]

    Dr2l: Surfacing corner cases to robustify autonomous driving via domain randomization reinforcement learning

    Niu, H., Hu, J., Cui, Z., and Zhang, Y. Dr2l: Surfacing corner cases to robustify autonomous driving via domain randomization reinforcement learning. In Proceedings of the 5th International Conference on Computer Science and Application Engineering, pp.\ 1--8, 2021

    2021

  39. [47]

    and Kalathil, D

    Panaganti, K. and Kalathil, D. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics, pp.\ 9582--9602. PMLR, 2022

    2022

  40. [48]

    Robust reinforcement learning using offline data

    Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. Robust reinforcement learning using offline data. arXiv preprint arXiv:2208.05129, 2022

    2022 arXiv

  41. [49]

    Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  42. [50]

    Hybrid transfer reinforcement learning: Provable sample efficiency from shifted-dynamics data

    Qu, C., Shi, L., Panaganti, K., You, P., and Wierman, A. Hybrid transfer reinforcement learning: Provable sample efficiency from shifted-dynamics data. arXiv preprint arXiv:2411.03810, 2024

    2024 arXiv

  43. [51]

    A., Colmenarejo, S

    Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., and Hadsell, R. Policy distillation. arXiv preprint arXiv:1511.06295, 2015

    2015 arXiv

  44. [52]

    Salvato, E., Fenu, G., Medvet, E., and Pellegrino, F. A. Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access, 9: 0 153171--153187, 2021

    2021

  45. [53]

    Sam, T., Chen, Y., and Yu, C. L. The limits of transfer reinforcement learning with latent low-rank structure. Advances in Neural Information Processing Systems, 37: 0 108262--108330, 2024

    2024

  46. [54]

    Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning

    Seo, Y., Lee, K., Clavera Gilaberte, I., Kurutach, T., Shin, J., and Abbeel, P. Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 12968--12979, 2020

    2020

  47. [55]

    Shakerimov, A., Alizadeh, T., and Varol, H. A. Efficient sim-to-real transfer in reinforcement learning through domain randomization and domain adaptation. IEEE Access, 11: 0 136809--136824, 2023

    2023

  48. [56]

    Sharma, A. R. and Kaushik, P. Literature survey of statistical, deep and reinforcement learning in natural language processing. In 2017 International conference on computing, communication and automation (ICCCA), pp.\ 350--354. IEEE, 2017

    2017

  49. [57]

    Pessimistic Q -learning for offline reinforcement learning: Towards optimal sample complexity

    Shi, L., Li, G., Wei, Y., Chen, Y., and Chi, Y. Pessimistic Q -learning for offline reinforcement learning: Towards optimal sample complexity. International Conference on Machine Learning, pp.\ 19967--20025, 2022

    2022

  50. [58]

    The curious price of distributional robustness in reinforcement learning with a generative model

    Shi, L., Li, G., Wei, Y., Chen, Y., Geist, M., and Chi, Y. The curious price of distributional robustness in reinforcement learning with a generative model. arXiv preprint arXiv:2305.16589, 2023

    2023 arXiv

  51. [59]

    Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., a...

    2016

  52. [60]

    Mastering the game of G o without human knowledge

    Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of G o without human knowledge. Nature, 550 0 (7676): 0 354--359, 2017

    2017

  53. [61]

    B., Clements, W

    Slaoui, R. B., Clements, W. R., Foerster, J. N., and Toth, S. Robust domain randomization for reinforcement learning. 2019

    2019

  54. [62]

    Multi-task reinforcement learning with context-based representations

    Sodhani, S., Zhang, A., and Pineau, J. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pp.\ 9767--9779. PMLR, 2021

    2021

  55. [63]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018

    2018

  56. [64]

    Z., Yu, H., Cui, L., and Yang, Q

    Tan, A. Z., Yu, H., Cui, L., and Yang, Q. Towards personalized federated learning. IEEE transactions on neural networks and learning systems, 34 0 (12): 0 9587--9603, 2022

    2022

  57. [65]

    M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R

    Teh, Y., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

  58. [66]

    Action robust reinforcement learning and applications in continuous control

    Tessler, C., Efroni, Y., and Mannor, S. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pp.\ 6215--6224. PMLR, 2019

    2019

  59. [67]

    Domain randomization via entropy maximization

    Tiboni, G., Klink, P., Peters, J., Tommasi, T., D'Eramo, C., and Chalvatzaki, G. Domain randomization via entropy maximization. arXiv preprint arXiv:2311.01885, 2023

    2023 arXiv

  60. [68]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp.\ 23--30. IEEE, 2017

    2017

  61. [69]

    Online vs

    Tommasi, T. Online vs. offline adaptive domain randomization benchmark. In Human-Friendly Robotics 2022: HFR: 15th International Workshop on Human-Friendly Robotics, volume 26, pp.\ 158. Springer Nature, 2023

    2022

  62. [70]

    Discorl: Continual reinforcement learning via policy distillation

    Traor \'e , R., Caselles-Dupr \'e , H., Lesort, T., Sun, T., Cai, G., D \' az-Rodr \' guez, N., and Filliat, D. Discorl: Continual reinforcement learning via policy distillation. arXiv preprint arXiv:1907.05855, 2019 a

    1907 arXiv

  63. [71]

    Continual reinforcement learning deployed in real-life using policy distillation and sim2real transfer

    Traor \'e , R., Caselles-Dupr \'e , H., Lesort, T., Sun, T., D \' az-Rodr \' guez, N., and Filliat, D. Continual reinforcement learning deployed in real-life using policy distillation and sim2real transfer. arXiv preprint arXiv:1906.04452, 2019 b

    1906 arXiv

  64. [72]

    Survey on reinforcement learning for language processing

    Uc-Cetina, V., Navarro-Guerrero, N., Martin-Gonzalez, A., Weber, C., and Wermter, S. Survey on reinforcement learning for language processing. Artificial Intelligence Review, 56 0 (2): 0 1543--1575, 2023

    2023

  65. [73]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Vershynin, R. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  66. [74]

    and Mahmoud, Q

    Vithayathil Varghese, N. and Mahmoud, Q. H. A survey of multi-task deep reinforcement learning. Electronics, 9 0 (9): 0 1363, 2020

    2020

  67. [75]

    Vuong, Q., Vikram, S., Su, H., Gao, S., and Christensen, H. I. How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? arXiv preprint arXiv:1903.11774, 2019

    1903 arXiv

  68. [76]

    and Wang, T

    Wang, G. and Wang, T. Unbiased multilevel M onte C arlo methods for intractable distributions: Mlmc meets mcmc. arXiv preprint arXiv:2204.04808, 2022

    2022 arXiv

  69. [77]

    Sample complexity of offline distributionally robust linear markov decision processes

    Wang, H., Shi, L., and Chi, Y. Sample complexity of offline distributionally robust linear markov decision processes. arXiv preprint arXiv:2403.12946, 2024 a

    2024 arXiv

  70. [78]

    On the convergence rates of federated q-learning across heterogeneous environments

    Wang, M., Yang, P., and Su, L. On the convergence rates of federated q-learning across heterogeneous environments. arXiv preprint arXiv:2409.03897, 2024 b

    2024 arXiv

  71. [79]

    P., and Petrik, M

    Wang, Q., Ho, C. P., and Petrik, M. Policy gradient in robust mdps with global convergence guarantee, 2023 a

    2023

  72. [80]

    and Zou, S

    Wang, Y. and Zou, S. Online robust reinforcement learning with model uncertainty. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp.\ 7193--7206, 2021

    2021

  73. [81]

    and Zou, S

    Wang, Y. and Zou, S. Policy gradient method for robust reinforcement learning. In Proc. International Conference on Machine Learning (ICML), volume 162, pp.\ 23484--23526. PMLR, 2022

    2022

  74. [82]

    K., Prater-Bennette, A., and Zou, S

    Wang, Y., Velasquez, A., Atia, G. K., Prater-Bennette, A., and Zou, S. Model-free robust average-reward reinforcement learning. In International Conference on Machine Learning, pp.\ 36431--36469. PMLR, 2023 b

    2023

  75. [83]

    A unified principle of pessimism for offline reinforcement learning under model mismatch

    Wang, Y., Sun, Z., and Zou, S. A unified principle of pessimism for offline reinforcement learning under model mismatch. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 c . URL https://openreview.net/forum?id=cBY66CKEbq

    2024

  76. [84]

    Robust average-reward reinforcement learning

    Wang, Y., Velasquez, A., Atia, G., Prater-Bennette, A., and Zou, S. Robust average-reward reinforcement learning. Journal of Artificial Intelligence Research, 80: 0 719--803, 2024 d

    2024

  77. [85]

    Finite-time analysis for conflict-avoidant multi-task reinforcement learning

    Wang, Y., Xiao, P., Ban, H., Ji, K., and Zou, S. Finite-time analysis for conflict-avoidant multi-task reinforcement learning. arXiv preprint arXiv:2405.16077, 2024 e

    2024 arXiv

  78. [86]

    Model-free robust reinforcement learning with sample complexity analysis

    Wang, Y., Zou, S., and Wang, Y. Model-free robust reinforcement learning with sample complexity analysis. In The 40th Conference on Uncertainty in Artificial Intelligence, 2024 f . URL https://openreview.net/forum?id=brZRvwK58H

    2024

  79. [87]

    Distributionally robust convex optimization

    Wiesemann, W., Kuhn, D., and Sim, M. Distributionally robust convex optimization. Operations research, 62 0 (6): 0 1358--1376, 2014

    2014

  80. [88]

    Multi-task reinforcement learning: a hierarchical bayesian approach

    Wilson, A., Fern, A., Ray, S., and Tadepalli, P. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pp.\ 1015--1022, 2007

    2007

  81. [89]

    The blessing of heterogeneity in federated q-learning: Linear speedup and beyond

    Woo, J., Joshi, G., and Chi, Y. The blessing of heterogeneity in federated q-learning: Linear speedup and beyond. In International Conference on Machine Learning, pp.\ 37157--37216. PMLR, 2023

    2023

  82. [90]

    Improved sample complexity bounds for distributionally robust reinforcement learning

    Xu, Z., Panaganti, K., and Kalathil, D. Improved sample complexity bounds for distributionally robust reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 9728--9754. PMLR, 2023

    2023

  83. [91]

    Multi-task reinforcement learning with soft modularization

    Yang, R., Xu, H., Wu, Y., and Wang, X. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33: 0 4767--4777, 2020

    2020

  84. [92]

    Rorl: Robust offline reinforcement learning via conservative smoothing

    Yang, R., Bai, C., Ma, X., Wang, Z., Zhang, C., and Han, L. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 35: 0 23851--23866, 2022

    2022

  85. [93]

    Towards theoretical understandings of robust M arkov decision processes: Sample complexity and asymptotics

    Yang, W., Zhang, L., and Zhang, Z. Towards theoretical understandings of robust M arkov decision processes: Sample complexity and asymptotics. arXiv preprint arXiv:2105.03863, 2021

    2021 arXiv

  86. [94]

    M., and Zhang, Z

    Yang, W., Wang, H., Kozuno, T., Jordan, S. M., and Zhang, Z. Robust markov decision processes without model estimation. arXiv preprint arXiv:2302.01248, 2023

    2023 arXiv

  87. [95]

    A gift from knowledge distillation: Fast optimization, network minimization and transfer learning

    Yim, J., Joo, D., Bae, J., and Kim, J. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4133--4141, 2017

    2017

  88. [96]

    and Pan, S

    Yin, H. and Pan, S. Knowledge transfer for deep reinforcement learning with hierarchical experience replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017

    2017

  89. [97]

    M., Khosravi, A., and Nahavandi, S

    Zare, M., Kebria, P. M., Khosravi, A., and Nahavandi, S. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics, 2024

    2024

  90. [98]

    U., Atia, G

    Zhang, C., Farhat, Z. U., Atia, G. K., and Wang, Y. Model-free offline reinforcement learning with enhanced robustness. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  91. [99]

    P., and Westerlund, T

    Zhao, W., Queralta, J. P., and Westerlund, T. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pp.\ 737--744. IEEE, 2020

    2020

  92. [100]

    Natural actor-critic for robust reinforcement learning with function approximation

    Zhou, R., Liu, T., Cheng, M., Kalathil, D., Kumar, P., and Tian, C. Natural actor-critic for robust reinforcement learning with function approximation. Advances in neural information processing systems, 36: 0 97--133, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.