Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Linear Mixture Distributionally Robust Markov Decision Processes

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Linear mixture distributionally robust MDPs are statistically learnable from offline data: uncertainty sets placed on mixture weights give $\widetilde{O}(dH^2C_{\pi^\star}/\sqrt{K})$ suboptimality for TV divergence and analogous rates for…

desk verdict A genuinely new robust RL setting with a plausible 1/sqrt(K) story, but the stated dimension dependence and the data-dependent coverage coefficient both need fixing before the guarantees are as advertised. read the letter →

arxiv 2505.18044 v1 pith:MO3PJLQN submitted 2025-05-23 cs.LG cs.AIcs.ROstat.ML

Zhishuai Liu , Pan Xu This is my paper · ORCID
classification cs.LGcs.AIcs.ROstat.ML MSC 90C4068Q32
keywords distributionallyrobustMDPlinearmixturemodelofflinereinforcementlearningf-divergenceuncertaintysetdoublepessimismpartialcoveragetransition-targetedridgeregressionsamplecomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a new class of distributionally robust MDPs in which the uncertainty set is a ball around the mixture-weighting parameter of a linear mixture transition model, rather than a ball around the transition kernel at each state-action pair. It argues that when the nominal dynamics are known to be linear mixtures of basis modes, this linear mixture uncertainty set captures plausible target-domain shifts more tightly than standard $(s,a)$-rectangular or $d$-rectangular sets, and can recover tabular $(s,a)$-rectangular sets by construction. The main theoretical target is statistical learnability: the paper designs a meta-algorithm based on double pessimism and transition-targeted ridge regression, and proves offline suboptimality bounds of order $\widetilde{O}(dH^2C_{\pi^\star}/\sqrt{K})$ for TV divergence, $\widetilde{O}(dH^2C_{\pi^\star}e^{H/\lambda}/(\rho\sqrt{K}))$ for KL divergence, and $\widetilde{O}(d(\sqrt{\rho}H^3+H^2)C_{\pi^\star}/\sqrt{K})$ for $\chi^2$ divergence. A sympathetic reader should care because this is the paper's first analysis showing that robustness and linear-mixture structure are compatible with efficient offline learning, giving an alternative to over-conservative rectangular uncertainty sets.

What carries the argument

The load-bearing object is the linear mixture uncertainty set $U^\rho(s,a;\theta_0)=\{P(\cdot|s,a)=\langle\phi(\cdot|s,a),\theta\rangle : \theta\in\Delta^{d-1},\ D(\theta\|\theta_0)\leq\rho\}$, a ball in mixture-weight space that induces a set of transition kernels. Around it, the algorithm stacks two mechanisms: transition-targeted ridge regression, which estimates the weights $\theta_0$ from per-transition targets and yields a confidence set $\widehat{\mathcal{P}}$ of plausible kernels, and the double-pessimism value estimator, which takes an infimum over $\widehat{\mathcal{P}}$ and then over each plausible kernel's own uncertainty set. The duality theorems for the three divergences convert the inner worst-case expectation into a one-dimensional scalar optimization, which is what makes the practical algorithms in Section 5 possible.

What would settle it

On the Section 6 five-state environment, fix the TV uncertainty set, compute the ratio $\sup_{\alpha,P,x} x^\top \Lambda^{TV}_h x / (x^\top \Lambda_0 x)$ from the collected dataset, and then measure SubOpt for DRTTR-TV as $K$ grows. The theorem predicts the gap decays as $K^{-1/2}$ whenever that ratio is bounded; observing a plateau in suboptimality while the ratio stays finite, or an unbounded ratio on a dataset where the bound is invoked, would refute the claimed rates.

Watch

Extended reading notes

Core claim

The central claim is that placing the distributional-robustness ball on the mixture-weighting parameter rather than on the transition kernel preserves provable offline learnability. Under robust partial coverage assumptions, Algorithm 1 returns a policy whose suboptimality is at most $\widetilde{O}(dH^2 C_{\pi^\star}/\sqrt{K})$ for TV divergence (Theorem 4.6), $\widetilde{O}(dH^2 C_{\pi^\star}e^{H/\lambda}/(\rho\sqrt{K}))$ for KL divergence (Theorem 4.11), and $\widetilde{O}(d(\sqrt{\rho}H^3+H^2)C_{\pi^\star}/\sqrt{K})$ for $\chi^2$ divergence (Theorem 4.13), up to logarithmic factors. The paper also claims that the linear mixture uncertainty set is a strictly more refined model of target-domain shift than $(s,a)$-rectangular and $d$-rectangular sets in the instances compared, and that any tabular $(s,a)$-rectangular DRMDP can be recovered as a special case.

Load-bearing premise

The entire rate rests on the robust partial coverage assumptions (4.4, 4.9, 4.12), which require the offline sampling covariance to dominate the worst-case robust value covariance for every transition in a confidence region built from that same offline data; this cannot be checked from the data alone before the algorithm runs, and if it fails the $1/\sqrt{K}$ rates do not follow.

Editorial extensions

If this is right

  • Offline data from the source domain suffice to learn a policy whose worst-case suboptimality under TV-divergence robustness decays as $\widetilde{O}(dH^2 C_{\pi^\star}/\sqrt{K})$, matching the non-robust linear mixture MDP rate in $d$ and $H$.
  • Because the $\chi^2$ bound reduces to the non-robust rate when $\rho=O(1/H^2)$, the framework degrades gracefully to standard linear mixture MDPs as the uncertainty level shrinks.
  • The KL bound's extra $e^{H/\lambda}/\rho$ factor is presented by the paper as an intrinsic price of KL robustness, not an artifact of the analysis, consistent with earlier tabular and $d$-rectangular KL results.
  • In tabular MDPs the linear mixture uncertainty set can be designed to reproduce exactly the $(s,a)$-rectangular set, so the new framework contains the standard tabular model as a special case.
  • The meta-algorithm's guarantees run through a planning oracle; the paper's practical DRTTR and DRVTR algorithms, which replace the oracle with value-iteration and Nelder-Mead solves, are validated experimentally but are not covered by the theorems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Testable extension: the same weight-space uncertainty construction should apply to other structured transition families besides linear mixtures, and the $f$-divergence dual formulation suggests the sample complexity would again be governed by a coverage ratio in feature space.
  • The robust partial coverage assumption being defined over the data-dependent $\widehat{\mathcal{P}}$ means the framework inherits a self-referential check: in practice one cannot certify $C_{\pi^\star}$ from the same dataset, so the stated rates should be read as conditional on a property that may require domain knowledge to validate.
  • Choosing basis modes becomes a statistical design decision: basis functions that concentrate the worst-case robust value directions in the sampled directions would reduce $C_{\pi^\star}$, potentially improving the bound more than additional samples would.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces linear mixture distributionally robust Markov decision processes (DRMDPs), in which the nominal transition kernel is a linear mixture of known basis modes and the uncertainty set is a ball around the mixture-weighting parameter under a general f-divergence. The authors argue that this uncertainty set is less conservative than (s,a)-rectangular and d-rectangular sets in specific examples, and they prove that dynamic programming principles hold. For offline learning, they propose a meta-algorithm based on double pessimism and transition-targeted ridge regression, and they state finite-sample suboptimality bounds for TV, KL, and chi-squared divergences (Theorems 4.6, 4.11, 4.13). They also present two computationally tractable heuristic algorithms and numerical experiments.

Significance. If the stated results are correct, this is a useful first step for robust policy learning in structured transition models: it proposes a new uncertainty-set geometry, extends double-pessimism analysis to linear mixture DRMDPs, and gives explicit finite-sample rates under three divergences. The comparison with existing rectangular uncertainty sets, the DPP results, and the use of transition-targeted regression are all valuable. However, the central quantitative claims currently have two load-bearing weaknesses: the confidence-radius lemma appears to produce an extra factor of d in the dimension dependence, and the robust partial coverage coefficients are defined through the data-dependent confidence set without any control on their magnitude. These issues affect the abstract's O~(d H^2 C*/sqrt(K)) rates and the paper's 'statistical learnability' conclusion, so the theorems need substantive revision before the contribution can be assessed as stated.

major comments (3)
  1. [Lemma 4.3 and Theorem 4.6] The confidence radius in Lemma 4.3 is beta_h = (5/4) sqrt(lambda) + 2 sqrt(lambda)(2 log(H/zeta) + d log(4 + 4 ceil(1/pmin) K/(lambda d))). With lambda = d, the second term is 2 d^{3/2} log(4 + 4 ceil(1/pmin) K/d^2), which is O~(d^{3/2}), not O~(sqrt(d)). The proof of Theorem 4.6 later bounds beta_h by c sqrt(d) log(K/(pmin d^2 zeta)) and drops the extra factor d inside the logarithm. Consequently the displayed bound c d H^2 C* K^{-1/2} log(...) does not follow from the stated lemma; the lemma gives O~(d^{3/2} H^2 C* / sqrt(K)) for TV divergence, and the same issue propagates to Theorems 4.11 and 4.13. This is load-bearing because the abstract and contribution list claim linear dependence on d.
  2. [Assumptions 4.4, 4.9, 4.12 and Theorems 4.6, 4.11, 4.13] The robust partial coverage constants C* are defined through a supremum over P in bP, where bP is the data-dependent confidence region of (4.2), and through the empirical covariance Lambda0_h built from the same offline dataset. No lemma in the paper controls the magnitude or growth of C*, so the stated bounds are conditional on a random, data-dependent coefficient; if C* grows with K, the displayed 1/sqrt(K) rates do not establish statistical learnability. Remark 4.5 explicitly acknowledges that bP depends on the dataset and that replacing bP by the fixed set of all feasible kernels would give a stronger assumption, but the proof still needs the supremum over bP. Please either reformulate the coverage assumptions with respect to a fixed set, or provide a high-probability bound on C*, and discuss verifiability of the condition before algorithm execution.
  3. [Assumption 4.7 and Theorem 4.11] The KL-divergence bound contains the factor e^{H/lambda}/rho, where lambda is a lower bound on the optimal dual variable. Assumption 4.7 only postulates that such a positive lower bound exists; the paper gives no example, sufficient condition, or verification procedure for it. Since the rate is exponentially sensitive to 1/lambda, this assumption is load-bearing for the KL result. The authors should at least provide natural instances where the lower bound provably holds, and discuss how lambda may depend on rho, H, and the feature geometry.
minor comments (5)
  1. [Theorems 4.6, 4.11, 4.13 and remarks] The theorem statements say 'Assume Theorems 3.1, 4.1 and 4.4 hold' and the remarks refer to 'Theorem 4.4', 'Theorem 4.9', and 'Theorem 4.12'; these are Assumptions, not Theorems, and should be cross-referenced consistently.
  2. [Appendix A] The appendix headers 'Proof of Theorem 3.4' and 'Proof of Theorem 3.5' should refer to Lemmas 3.4 and 3.5.
  3. [Equation (5.4)] Equation (5.4) is missing the minus sign before the square-root term in the chi-squared robust estimator, inconsistent with the formula given in the preceding paragraph.
  4. [Appendix B] The proof of the suboptimality decomposition begins 'For any P in ~P', but the set ~P is not defined; it should be bP.
  5. [Proposition 3.3] There is a typo in 'pilicy' which should be 'policy'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the sample-complexity bounds are derived from concentration and duality lemmas under explicit coverage assumptions; the data-dependent coverage coefficient weakens the guarantee but is not a circular reduction.

full rationale

The derivation chain for the main theorems is self-contained relative to its inputs. Lemma 4.3 is obtained from the concentration lemma of Li et al. [16], and Theorems 4.6, 4.11 and 4.13 are proved using standard strong-duality characterizations (Lemmas C.4-C.6) and a trace comparison lemma (C.3). The robust partial coverage assumptions 4.4, 4.9 and 4.12 introduce a coefficient C_pi* that appears in the final bound, but C_pi* is not fitted from the data and is not defined in terms of the suboptimality being bounded; it is an assumed upper bound on a covariance ratio. The proof uses this ratio through Lemma C.3 and Jensen's inequality, not by substituting the target inequality into itself. The main caveat, explicitly flagged by the authors in Remark 4.5, is that the confidence region bP is constructed from the same offline dataset, making C_pi* random and the coverage condition unverifiable a priori; the paper even notes that replacing bP by the fixed set of all feasible kernels would give a stronger assumption. This is a real weakness in the strength of the 'statistical learnability' claim, but it is not circular: the theorem is still a conditional implication and does not assume its conclusion. The use of the authors' prior work [18] to justify the robust Bellman equation in Proposition 3.2 is a minor self-citation, but this is standard rectangular-type dynamic programming and is not the load-bearing step for the sample-complexity guarantees. The citation of [34] is contextual, noting that similar KL-divergence terms appeared before. No equation in the paper reduces to its own input by construction, and no fitted parameter is relabeled as a prediction. Score 2 reflects only minor, non-load-bearing self-citations; there is no significant circularity in the central derivation.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central results rest on the linear mixture structure, finite-support transitions, data-dependent coverage, and KL dual regularity. These are plausible but restrictive assumptions. None are fitted parameters: the robust level rho and regularizer lambda are user-specified inputs, and C_pi* is an unknown problem-dependent constant.

assumptions (6)
  • domain assumption The nominal transition kernel is a linear mixture: P0(·|s,a) = <phi(·|s,a), theta0_h> with known basis modes phi and unknown weights theta0 in the simplex (Assumption 3.1).
    Foundation of the framework. If the basis modes are misspecified, the uncertainty set (3.2) is misaligned with the true target dynamics.
  • domain assumption Finite reachable next-state support with all positive transition probabilities bounded below by pmin (Assumption 4.1).
    Needed for the transition-targeted ridge regression and the concentration bound; excludes continuous and large-support transitions.
  • domain assumption Robust partial coverage: the offline data covariance dominates the worst-case robust value covariance over the data-dependent confidence region bP (Assumptions 4.4, 4.9, 4.12).
    Standard-pattern coverage condition, but quantified over bP, which is built from the dataset, so it is not checkable a priori.
  • ad hoc to paper The optimal KL dual variable is bounded below by lambda > 0 (Assumption 4.7).
    Needed to control the exponential e^{H/lambda} term in the KL bound. No constructive way to verify or estimate lambda is given.
  • standard math Dynamic programming holds for rectangular uncertainty sets, following Propositions 3.2 and 3.3 of [18].
    Rectangularity preserves Bellman optimality. The paper inherits this from prior work, but the cited source is the same research group.
  • standard math Concentration lemmas from [16] and the trace inequality from [35] are valid for the transition-targeted errors and covariance matrices used in the proof.
    These lemmas are external background results. The paper's own Lemma 4.3 is presented as a direct adaptation of [16].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Linear Mixture Distributionally Robust Markov Decision Processes." pith.science (2026). https://pith.science/paper/MO3PJLQN

@misc{pith2026250518044,
  author       = {Pith},
  title        = {Pith review of: Linear Mixture Distributionally Robust Markov Decision Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MO3PJLQN}},
  note         = {Machine review of arXiv:2505.18044}
}
abstract

Many real-world decision-making problems face the off-dynamics challenge: the agent learns a policy in a source domain and deploys it in a target domain with different state transitions. The distributionally robust Markov decision process (DRMDP) addresses this challenge by finding a robust policy that performs well under the worst-case environment within a pre-specified uncertainty set of transition dynamics. Its effectiveness heavily hinges on the proper design of these uncertainty sets, based on prior knowledge of the dynamics. In this work, we propose a novel linear mixture DRMDP framework, where the nominal dynamics is assumed to be a linear mixture model. In contrast with existing uncertainty sets directly defined as a ball centered around the nominal kernel, linear mixture DRMDPs define the uncertainty sets based on a ball around the mixture weighting parameter. We show that this new framework provides a more refined representation of uncertainties compared to conventional models based on $(s,a)$-rectangularity and $d$-rectangularity, when prior knowledge about the mixture model is present. We propose a meta algorithm for robust policy learning in linear mixture DRMDPs with general $f$-divergence defined uncertainty sets, and analyze its sample complexities under three divergence metrics instantiations: total variation, Kullback-Leibler, and $\chi^2$ divergences. These results establish the statistical learnability of linear mixture DRMDPs, laying the theoretical foundation for future research on this new setting.

Figures

Figures reproduced from arXiv: 2505.18044 by the authors.

Figure 2
Figure 2. An illustration of the linear mixture uncertainty set and the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The source and the target linear MDP environments. The value on each arrow represents [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Simulation results of DRTTR under different source domains. Policies are learned [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Simulation results of DRVTR under different source domains. Policies are learned from [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Simulation results of DRTTR under different source domains. Policies are learned from the nominal environment featuring θ1 = (0.1, 0.8, 0.1). Numbers in parenthesis represent (δ, ∥ξ∥1, ρTV, ρKL, ρχ2 ), respectively. The x-axis represents the perturbation level correspo…
Figure 7
Figure 7. Figure 7: Simulation results of DRVTR under different source domains. Policies are learned [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 44 canonical work pages

  1. [1]

    Improved algorithms for linear stochastic bandits.Advances in Neural Information Processing Systems, 24, 2011

    Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits.Advances in Neural Information Processing Systems, 24, 2011. 6

    work page 2011

  2. [2]

    Model-based rein- forcement learning with value-targeted regression

    Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based rein- forcement learning with value-targeted regression. InInternational Conference on Machine Learning, pages 463–474. PMLR, 2020. 2, 3, 4, 6, 8, 9, 10

    work page 2020

  3. [3]

    Robust reinforcement learning using least squares policy iteration with provable performance guarantees

    Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. InInternational Conference on Machine Learning, pages 511–520. PMLR, 2021. 3

    work page 2021

  4. [4]

    Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage.Advances in Neural Information Processing Systems, 36, 2024. 2, 3, 7, 8, 9

    work page 2024

  5. [5]

    Provably efficient exploration in policy optimization

    Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. InInternational Conference on Machine Learning, pages 1283–1294. PMLR,

    work page

  6. [6]

    A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models.arXiv preprint arXiv:2502.13187, 2025

    Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, and Hua Wei. A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models.arXiv preprint arXiv:2502.13187, 2025. 1

    arXiv 2025

  7. [7]

    Off-dynamics reinforcement learning: Training for transfer with domain classifiers

    Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations, 2021. 1

    work page 2021

  8. [8]

    Birkhauser Boston Inc., 1989

    Boris Vladimirovich Gnedenko and Igor Nikolaevich Kovalenko.Introduction to queueing theory. Birkhauser Boston Inc., 1989. 3

    work page 1989

Show all 51 references
  1. [9]

    Robust markov decision processes: Beyond rectangu- larity.Mathematics of Operations Research, 48(1):203–226, 2023

    Vineet Goyal and Julien Grand-Clement. Robust markov decision processes: Beyond rectangu- larity.Mathematics of Operations Research, 48(1):203–226, 2023. 2, 3, 4, 5

    2023

  2. [10]

    Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation

    Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. InAdvances in Neural Information Processing Systems, volume 37, pages 136326–136360, 2024. 1 15

    2024

  3. [11]

    Kullback-leibler divergence constrained distributionally robust optimization.Available at Optimization Online, 1(2):9, 2013

    Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization.Available at Optimization Online, 1(2):9, 2013. 26

    2013

  4. [12]

    Robust dynamic programming.Mathematics of Operations Research, 30(2): 257–280, 2005

    Garud N Iyengar. Robust dynamic programming.Mathematics of Operations Research, 30(2): 257–280, 2005. 2, 3, 4, 5

    2005

  5. [13]

    Model-based reinforcement learning with value-targeted regression

    Zeyu Jia, Lin Yang, Csaba Szepesvari, and Mengdi Wang. Model-based reinforcement learning with value-targeted regression. InLearning for Dynamics and Control, pages 666–686. PMLR,

  6. [14]

    Reinforcement learning in robotics: A survey

    Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013. 3

    2013

  7. [15]

    The transferability approach: Crossing the reality gap in evolutionary robotics.IEEE Transactions on Evolutionary Computa- tion, 17(1):122–145, 2012

    Sylvain Koos, Jean-Baptiste Mouret, and Stéphane Doncieux. The transferability approach: Crossing the reality gap in evolutionary robotics.IEEE Transactions on Evolutionary Computa- tion, 17(1):122–145, 2012. 1

    2012

  8. [16]

    Improved algorithm for adversarial linear mixture mdps with bandit feedback and unknown transition

    Long-Fei Li, Peng Zhao, and Zhi-Hua Zhou. Improved algorithm for adversarial linear mixture mdps with bandit feedback and unknown transition. InInternational Conference on Artificial Intelligence and Statistics, pages 3061–3069. PMLR, 2024. 2, 6, 26

    2024

  9. [17]

    Policy gradient algorithms for robust mdps with non-rectangular uncertainty sets.arXiv preprint arXiv:2305.19004, 2023

    Mengmeng Li, Daniel Kuhn, and Tobias Sutter. Policy gradient algorithms for robust mdps with non-rectangular uncertainty sets.arXiv preprint arXiv:2305.19004, 2023. 3

    2023

  10. [18]

    Distributionally robust off-dynamics reinforcement learning: Prov- able efficiency with linear function approximation

    Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Prov- able efficiency with linear function approximation. InInternational Conference on Artificial Intelligence and Statistics, pages 2719–2727. PMLR, 2024. 1, 4, 5, 11

    2024

  11. [19]

    Minimax optimal and computationally efficient algorithms for distributionally robust offline reinforcement learning

    Zhishuai Liu and Pan Xu. Minimax optimal and computationally efficient algorithms for distributionally robust offline reinforcement learning. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2, 3, 5

    2024

  12. [20]

    Upper and lower bounds for distributionally robust off-dynamics reinforcement learning.arXiv preprint arXiv:2409.20521, 2024

    Zhishuai Liu, Weixin Wang, and Pan Xu. Upper and lower bounds for distributionally robust off-dynamics reinforcement learning.arXiv preprint arXiv:2409.20521, 2024. 3

    2024 arXiv

  13. [21]

    Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithms

    Miao Lu, Han Zhong, Tong Zhang, and Jose Blanchet. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithms. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 3

    2024

  14. [22]

    Distributionally robust offline reinforcement learning with linear function approximation.arXiv preprint arXiv:2209.06620, 2022

    Xiaoteng Ma, Zhipeng Liang, Li Xia, Jiheng Zhang, Jose Blanchet, Mingwen Liu, Qianchuan Zhao, and Zhengyuan Zhou. Distributionally robust offline reinforcement learning with linear function approximation.arXiv preprint arXiv:2209.06620, 2022. 2, 3, 5

    2022 arXiv

  15. [23]

    Finite mixture models.Annual review of statistics and its application, 6(1):355–378, 2019

    Geoffrey J McLachlan, Sharon X Lee, and Suren I Rathnayake. Finite mixture models.Annual review of statistics and its application, 6(1):355–378, 2019. 2

    2019

  16. [24]

    A simplex method for function minimization.The computer journal, 7(4):308–313, 1965

    John A Nelder and Roger Mead. A simplex method for function minimization.The computer journal, 7(4):308–313, 1965. 10

    1965

  17. [25]

    Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005

    Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005. 2, 4

    2005

  18. [26]

    Robustness in markov decision problems with uncertain transition matrices.Advances in Neural Information Processing Systems, 16, 2003

    Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices.Advances in Neural Information Processing Systems, 16, 2003. 3

    2003

  19. [27]

    Assessing generalization in deep reinforcement learning.arXiv preprint arXiv:1810.12282,

    Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning.arXiv preprint arXiv:1810.12282,

    arXiv

  20. [28]

    Bridging distributionally robust learning and offline rl: An approach to mitigate distribution shift and partial data coverage

    Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Bridging distributionally robust learning and offline rl: An approach to mitigate distribution shift and partial data coverage. InICML 2024 Workshop: Foundations of Reinforcement Learning and Control–Conne...

    2024

  21. [29]

    The infinite gaussian mixture model.Advances in Neural Information Processing Systems, 12, 1999

    Carl Rasmussen. The infinite gaussian mixture model.Advances in Neural Information Processing Systems, 12, 1999. 2

    1999

  22. [30]

    Gaussian mixture models.Encyclopedia of biometrics, 741(659-663): 3, 2009

    Douglas A Reynolds et al. Gaussian mixture models.Encyclopedia of biometrics, 741(659-663): 3, 2009. 2

    2009

  23. [31]

    Markovian decision processes with uncertain transition probabilities.Operations Research, 21(3):728–740, 1973

    Jay K Satia and Roy E Lave Jr. Markovian decision processes with uncertain transition probabilities.Operations Research, 21(3):728–740, 1973. 2

    1973

  24. [32]

    Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity.Journal of Machine Learning Research, 25(200):1–91,

    Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity.Journal of Machine Learning Research, 25(200):1–91,

  25. [33]

    The curious price of distributional robustness in reinforcement learning with a generative model.Advances in Neural Information Processing Systems, 36, 2024

    Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, Matthieu Geist, and Yuejie Chi. The curious price of distributional robustness in reinforcement learning with a generative model.Advances in Neural Information Processing Systems, 36, 2024. 3, 9, 26

    2024

  26. [34]

    Robust offline reinforcement learning with linearly structuredf-divergence regularization.arXiv preprint arXiv:2411.18612, 2024

    Cheng Tang, Zhishuai Liu, and Pan Xu. Robust offline reinforcement learning with linearly structuredf-divergence regularization.arXiv preprint arXiv:2411.18612, 2024. 3, 5, 8, 9

    2024

  27. [35]

    Pessimistic model-based offline reinforcement learning under partial coverage

    Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. InInternational Conference on Learning Representations, 2021. 7, 26

    2021

  28. [36]

    Sample complexity of offline distributionally robust linear markov decision processes

    He Wang, Laixi Shi, and Yuejie Chi. Sample complexity of offline distributionally robust linear markov decision processes. InReinforcement Learning Conference, 2024. 5

    2024

  29. [37]

    Return augmented decision transformer for off-dynamics reinforcement learning.arXiv preprint arXiv:2410.23450, 2024

    Ruhan Wang, Yu Yang, Zhishuai Liu, Dongruo Zhou, and Pan Xu. Return augmented decision transformer for off-dynamics reinforcement learning.arXiv preprint arXiv:2410.23450, 2024. 1

    2024

  30. [38]

    Robust markov decision processes

    Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research, 38(1):153–183, 2013. 2, 3, 4

    2013

  31. [39]

    Mutual alignment transfer learning

    Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. In Conference on Robot Learning, pages 281–290. PMLR, 2017. 1

    2017

  32. [40]

    The robustness-performance tradeoff in markov decision processes

    Huan Xu and Shie Mannor. The robustness-performance tradeoff in markov decision processes. Advances in Neural Information Processing Systems, 19, 2006. 2, 3

    2006

  33. [41]

    Improved sample complexity bounds for distributionally robust reinforcement learning

    Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distributionally robust reinforcement learning. InInternational Conference on Artificial Intelligence and Statistics, pages 9728–9754. PMLR, 2023. 5

    2023

  34. [42]

    Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics.The Annals of Statistics, 50 (6):3223–3248, 2022

    Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics.The Annals of Statistics, 50 (6):3223–3248, 2022. 3

    2022

  35. [43]

    Reward-free model-based reinforcement learning with linear function approximation.Advances in Neural Information Processing Systems, 34:1582–1593, 2021

    Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Reward-free model-based reinforcement learning with linear function approximation.Advances in Neural Information Processing Systems, 34:1582–1593, 2021. 6

    2021

  36. [44]

    Learning adversarial linear mixture markov decision processes with bandit feedback and unknown transition

    Canzhe Zhao, Ruofeng Yang, Baoxiang Wang, and Shuai Li. Learning adversarial linear mixture markov decision processes with bandit feedback and unknown transition. InThe Eleventh International Conference on Learning Representations, 2023. 6

    2023

  37. [45]

    Sim-to-real transfer in deep reinforcement learning for robotics: a survey

    Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE, 2020. 1

    2020

  38. [46]

    Nearly minimax optimal reinforcement learning for linear mixture markov decision processes

    Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. InConference on Learning Theory, pages 4532–4576. PMLR, 2021. 2, 3, 6, 8, 10

    2021

  39. [47]

    Provably efficient reinforcement learning for discounted mdps with feature mapping

    Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. InInternational Conference on Machine Learning, pages 12793–12802. PMLR, 2021. 5 17

    2021

  40. [48]

    Natural actor-critic for robust reinforcement learning with function approximation

    Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. InThirty-seventh Conference on Neural Information Processing Systems, 2023. 3

    2023

  41. [49]

    Finite-sample regret bound for distributionally robust offline tabular reinforcement learning

    Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3331–3339. PMLR,

  42. [50]

    Time- constrained robust mdps.Advances in Neural Information Processing Systems, 37:35574–35611,

    Adil Zouitine, David Bertoin, Pierre Clavier, Matthieu Geist, and Emmanuel Rachelson. Time- constrained robust mdps.Advances in Neural Information Processing Systems, 37:35574–35611,

  43. [2024]

    A.1 Proof of Theorem 3.4 Proof

    3 18 A Proof in Section 3 In this section, we prove Theorem 3.4 and Theorem 3.5. A.1 Proof of Theorem 3.4 Proof. To see this, on the one hand, for any (s, a)∈ S × A, assume P 0(s′|s, a) =ϕ(s′, s, a)⊤θ0 is the nominal kernel. For any θ∈ {θ∈∆ d−1|DTV(θ||θ0)≤ρ} , we have P(s ′|s,...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.