Pith. sign in

REVIEW 3 major objections 3 minor 53 references

Solving Robust Markov Decision Processes: Generic, Reliable, Efficient

T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A framework solves robust Markov decision processes with anytime precision guarantees, without constructing the induced stochastic game.

desk verdict The pmin worry is a red herring; the real flaw is the sign-reversed Lp update proof in Lemma 1, but the anytime-VI framework is valuable and deserves a rigorous review. read the letter →

arxiv 2412.10185 v1 pith:OZIHNZPC submitted 2024-12-13 cs.AI cs.LG

Tobias Meggendorfer , Maximilian Weininger , Patrick Wienhöft This is my paper · ORCID
classification cs.AIcs.LG
keywords robustMarkovdecisionprocessesstochasticgamesvalueiterationanytimealgorithmstoppingcriterionlong-runaveragerewardtotalimplicitupdates
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to make robust Markov decision processes (RMDPs) practically solvable while retaining formal correctness guarantees, even when transition probabilities are only known to lie in general uncertainty sets. It claims that RMDPs can be solved through a reduction to stochastic games without ever materializing the game, so non-polytopic sets such as L2-balls become tractable. The central proof target is an anytime value-iteration algorithm that maintains lower and upper bounds on the true value and provably closes the gap for total-reward, stochastic-shortest-path, and long-run-average objectives. If correct, this closes the stopping-criterion gap left by earlier robust value iteration and makes reliable RMDP solving scale to models with a million states.

What carries the argument

The Constant-Support Assumption—that all distributions in one uncertainty set share the same set of possible successors—together with closedness of the uncertainty sets. This combination guarantees continuity of value functions with respect to the environment policy, which yields memoryless deterministic optimal policies and allows end components to be collapsed so that Bellman updates have a unique fixpoint and upper bounds converge. The second key mechanism is the implicit Bellman update: instead of enumerating the possibly uncountably infinite actions of the induced stochastic game, the algorithm optimizes a linear function over the uncertainty set in polynomial time for the supported representations.

What would settle it

Run Algorithm 2 on a closed constant-support RMDP whose uncertainty set is, for example, {$P : 0 \le P(s_1) \le 1,\ P(s_2)=1-P(s_1)$}, so probabilities approach zero while support stays constant; if for some $\varepsilon$ the gap between upper and lower bounds never drops below $\varepsilon$, the anytime claim fails for that instance.

Watch

Extended reading notes

Core claim

The paper establishes that, for every closed constant-support RMDP with a total-reward objective and any precision epsilon > 0, its Algorithm 2 is an anytime algorithm: it keeps sound lower and upper bounds whose difference converges to zero, and it works implicitly, without constructing the induced stochastic game. The long-run-average variant is likewise anytime. The paper generalizes the RMDP-to-SG connection to arbitrary uncertainty sets and to total-reward and stochastic-shortest-path objectives, proves that robust Bellman updates converge in the limit, and gives polynomial-time implicit update formulas for polytopes in H- or V-representation and for Lp-norm balls. It also proves that in closed constant-support RMDPs optimal policies exist and are memoryless deterministic, and it reports experimental evidence of solving RMDPs with over a million states in under a minute.

Load-bearing premise

The anytime guarantee's contraction argument needs all transition probabilities in every uncertainty set to be bounded away from zero, and the Constant-Support Assumption alone does not ensure that for every closed set.

Editorial extensions

If this is right

  • Explicit construction of the induced stochastic game becomes unnecessary, removing the exponential space blowup that limited earlier polytopic approaches.
  • Uncertainty sets such as L2-balls, which state-of-the-art tools could not handle, become solvable with formal precision guarantees.
  • RMDPs with total reward, stochastic shortest path, and long-run average reward objectives all fall under one implicit value-iteration framework with a sound stopping criterion.
  • The reported runtimes suggest that reliable RMDP solving can scale to large verification benchmarks with millions of states and actions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader relying on the anytime guarantee should verify that all transition probabilities in their uncertainty sets are bounded away from zero, because the termination proof uses such a positive minimum probability; the paper's theorem statement does not spell out this condition.
  • The implicit-update technique likely transfers beyond the listed objectives to discounted rewards and to optimistic or best-case environment semantics, since the paper notes those cases reduce to the same machinery.
  • For distance-based uncertainty sets such as KL-divergence or Wasserstein balls, the Constant-Support Assumption holds for sufficiently small radii, suggesting a practical route from this framework to distributionally robust reinforcement learning with guarantees.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes a framework for solving robust Markov decision processes (RMDPs) that is generic over uncertainty sets (polytopes, intervals, Lp balls) and objectives (total reward, long-run average reward, stochastic shortest path), reliable through anytime algorithms with stopping criteria, and efficient by performing value iteration implicitly on the RMDP rather than by explicitly constructing the induced stochastic game. The main results are a formal reduction from RMDPs to stochastic games (Theorem 1), existence of memoryless deterministic optimal policies under closed constant-support uncertainty (Theorem 2), convergence of robust value iteration (Theorem 3), polynomial-time implicit Bellman updates for several uncertainty representations (Lemma 1), and anytime algorithms for constant-support and, in part, non-constant-support RMDPs (Theorems 4 and 5). The paper also reports a prototype implementation, artefacts, and experiments on benchmarks with up to a million states. One reviewer concern about a missing lower bound pmin>0 is not supported: for closed constant-support sets, the map P ↦ min_{s in the support} P(s) is positive and continuous on a compact subset of the simplex, so a positive minimum exists without an additional axiom.

Significance. If the results hold, the framework would be a substantial advance: it unifies several previously separate RMDP settings, provides the first practical anytime guarantees for undiscounted and average-reward robust objectives, and demonstrates scalability through an implicit implementation. The strengths of the submission include a concrete artefact with code and models, comparison against several existing tools, and a clear theoretical reduction to stochastic games that is used throughout. However, the central technical lemma for Lp-ball uncertainty sets, which is advertised as a key novelty (including L2 balls), contains a sign error and an optimality argument that only works for p=2. Since the paper's genericity and experimental claims for L2/Lp uncertainty rest on this lemma, the significance is conditional on a corrected and complete proof. The remaining framework, especially for polytopic and constant-support interval uncertainty, remains plausible and potentially valuable.

major comments (3)
  1. [Appendix D, Lemma 1, Lp-ball case] The proof defines P_opt(s,a) = P*(s,a) - ζ x / ||x||^w_p with x = L_i - (||L_i||_1/k)·1 and claims that this point maximizes the linear objective. Since x is positive on above-average-reward states and negative on below-average-reward states, subtracting ζx moves probability mass away from high-reward states and toward low-reward states; this constructs a minimizer, not a maximizer. The containment calculation then writes P_opt - P* = ζ x / ||x||^w_p, which contradicts the definition immediately above it. For maximization the sign must be plus. As printed, Lemma 1 does not prove the implicit L2 update, and the experiments in Section 6 that advertise L2 uncertainty sets rely on this formula.
  2. [Appendix D, Lemma 1, general Lp optimality] Even after correcting the sign, the optimality argument is only valid for p=2. At a boundary point P of an Lp ball, the outward normal is componentwise sign(P-P*)·|P-P*|^{p-1}, which is parallel to P-P* only for p=2. For general integer p, the KKT conditions require solving for a Lagrange multiplier, and the paper provides neither a closed form nor a polynomial-time numerical procedure for that multiplier. Consequently, the claim that the implicit update can be evaluated in polynomial time for every p ∈ N ∪ {∞} is not established; the proof as written supports only L1, L∞, and L2 with corrected sign. Since the abstract, introduction, and experiments present general Lp balls as a central feature, this is load-bearing for the paper's genericity and reliability claims.
  3. [Appendix E, proof of Theorem 4, termination paragraph] The termination proof breaks off mid-sentence. After the contraction argument, the text states that there is some n for which |U_n(s) - L_n(s)| and then continues with the incomplete fragment 'as U_i(s) ≥ L_i(s) (see Correctness paragraph) this implies U_n(s) − L_n(s),' without completing the epsilon-bound argument or specifying how n is chosen. Because termination for every ε > 0 is part of Definition 4, the printed proof of Theorem 4 is incomplete. This is readily fixable, but it should be written out explicitly.
minor comments (3)
  1. [Algorithm 2, line 4] The while condition 'U(s_i) - L(s_i) > ε' uses an undefined identifier s_i; it should quantify over all states, e.g. 'max_{s ∈ S'} (U(s) - L(s)) > ε'.
  2. [Appendix D, proof of Theorem 3, LRA part] The equality LG_i(s) = 2·L_{i/2}(s) is only meaningful for even i and has the index off by one; it should be LG_{2j}(s) = 2·L_j(s). The limit conclusion is unaffected, but the indexing should be corrected.
  3. [Section 6, Table 3] The column header 'Solving Time' in Table 3 does not state units; the text should explicitly repeat that times are in seconds, as is done for Tables 1 and 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: core claims reduce to independently established SG theory, not to the paper's own inputs.

full rationale

After walking the derivation chain, I find no step in which a prediction or first-principles result is equivalent to its inputs by construction. The RMDP-to-SG reduction (Thm. 1) is a genuine theorem with a policy-mapping proof; it is not defined into existence. The anytime algorithm (Thm. 4) is proved from a contraction argument (Lem. 2) using the paper's own collapse construction and the Banach fixed-point theorem; the key pmin bound is entailed by closedness plus the Constant-Support Assumption via compactness, as the paper itself states in Lem. 5, so no concealed fitted input is renamed as a prediction. The stopping-criterion machinery cites Kretínský, Meggendorfer, and Weininger 2023a/b, which includes the present authors, but that cited work is an independent, parameter-free SG result with stated assumptions that do not include the RMDP claim being proved; applying it to the induced SG is standard and externally checkable. The Lp-ball update proof in App. D has an apparent sign/generality flaw (and thus is a correctness/rigor risk for the L2/Lp genericity claim), but an incorrect proof step is not a circular dependency: the claimed formula is not assumed as an input, nor is the L2 result fitted from the RMDP data. The experimental comparison to PRISM and to Chatterjee et al.'s RPPI provides external falsifiability. Overall circularity score 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities. The main load-bearing inputs are the rectangularity, closedness and Constant-Support assumptions, plus an unstated uniform lower bound on transition probabilities (pmin > 0) needed in the contraction proof of the anytime algorithm. Standard game-theoretic convergence theorems are used as external background.

assumptions (5)
  • domain assumption (s,a)-rectangularity of uncertainty sets
    Used to define RMDP semantics and the reduction to SG; stated as a classical assumption in Sec. 2.
  • domain assumption Closedness of uncertainty sets
    Needed for existence of optimal environment policies (Thm 2); assumed in Sec. 2.
  • domain assumption Constant-Support Assumption
    Assumed for the anytime algorithms (Alg. 2, 3); requires identical support for all distributions in each uncertainty set.
  • ad hoc to paper Uniform positive lower bound pmin > 0
    Used in App. E contraction proof to get γ < 1 and unique fixpoint; not stated in the Constant-Support Assumption or the theorems.
  • standard math Known convergence theorems for SG value iteration (Chen et al. 2013; Kretínský et al. 2023b)
    Used to prove Thm 3 via the SG reduction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Solving Robust Markov Decision Processes: Generic, Reliable, Efficient." pith.science (2026). https://pith.science/paper/OZIHNZPC

@misc{pith2026241210185,
  author       = {Pith},
  title        = {Pith review of: Solving Robust Markov Decision Processes: Generic, Reliable, Efficient},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OZIHNZPC}},
  note         = {Machine review of arXiv:2412.10185}
}
abstract

Markov decision processes (MDP) are a well-established model for sequential decision-making in the presence of probabilities. In robust MDP (RMDP), every action is associated with an uncertainty set of probability distributions, modelling that transition probabilities are not known precisely. Based on the known theoretical connection to stochastic games, we provide a framework for solving RMDPs that is generic, reliable, and efficient. It is *generic* both with respect to the model, allowing for a wide range of uncertainty sets, including but not limited to intervals, $L^1$- or $L^2$-balls, and polytopes; and with respect to the objective, including long-run average reward, undiscounted total reward, and stochastic shortest path. It is *reliable*, as our approach not only converges in the limit, but provides precision guarantees at any time during the computation. It is *efficient* because -- in contrast to state-of-the-art approaches -- it avoids explicitly constructing the underlying stochastic game. Consequently, our prototype implementation outperforms existing tools by several orders of magnitude and can solve RMDPs with a million states in under a minute.

Figures

Figures reproduced from arXiv: 2412.10185 by the authors.

Figure 1
Figure 1. RMDP without optimal environment-policy. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. An MDP where value iteration from above does [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. MEC in an SG where not all states have the same [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 48 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ashok, P.; Chatterjee, K.; Daca, P.; Kret \' nsk \' y , J.; and Meggendorfer, T. 2017. Value Iteration for Long-Run Average Reward in M arkov Decision Processes. In CAV (1) , volume 10426 of Lecture Notes in Computer Science, 201--221. Springer

    work page 2017

  4. [4]

    Ashok, P.; Kret \' nsk \' y , J.; and Weininger, M. 2019. PAC Statistical Model Checking for M arkov Decision Processes and Stochastic Games. In CAV, Part I , volume 11561 of LNCS, 497--519. Springer

    work page 2019

  5. [5]

    Azeem, M.; Evangelidis, A.; Kret \' nsk \' y , J.; Slivinskiy, A.; and Weininger, M. 2022. Optimistic and Topological Value Iteration for Simple Stochastic Games. In ATVA , volume 13505 of Lecture Notes in Computer Science, 285--302. Springer

    work page 2022

  6. [6]

    S.; Sim \ a o, T

    Badings, T. S.; Sim \ a o, T. D.; Suilen, M.; and Jansen, N. 2023. Decision-making under uncertainty: beyond probabilities. Int. J. Softw. Tools Technol. Transf., 25(3): 375--391

    work page 2023

  7. [7]

    Baier, C.; and Katoen, J. 2008. Principles of model checking. MIT Press. ISBN 978-0-262-02649-9

    2008

  8. [8]

    Baier, C.; Klein, J.; Leuschner, L.; Parker, D.; and Wunderlich, S. 2017. Ensuring the Reliability of Your Model Checker: Interval Iteration for M arkov Decision Processes. In CAV (1) , volume 10426 of Lecture Notes in Computer Science, 160--180. Springer

    work page 2017

Show all 53 references
  1. [9]

    Bart, A.; Delahaye, B.; Fournier, P.; Lime, D.; Monfroy, \' E .; and Truchet, C. 2018. Reachability in parametric Interval Markov Chains using constraints. Theor. Comput. Sci., 747: 48--74

    2018

  2. [10]

    Bertrand, N.; Bouyer-Decitre, P.; Fijalkow, N.; and Skomra, M. 2023. Stochastic Games. In Fijalkow, N., ed., Games on Graphs

    2023

  3. [11]

    P.; and Tsitsiklis, J

    Bertsekas, D. P.; and Tsitsiklis, J. N. 1991. An Analysis of Stochastic Shortest Path Problems. Math. Oper. Res., 16(3): 580--595

    1991

  4. [12]

    Z.; Parker, D.; and Ujma, M

    Br \' a zdil, T.; Chatterjee, K.; Chmelik, M.; Forejt, V.; K r et \' nsk \' y , J.; Kwiatkowska, M. Z.; Parker, D.; and Ujma, M. 2014. Verification of M arkov Decision Processes Using Learning Algorithms. In ATVA , 98--114. Springer

    2014

  5. [13]

    Buffet, O. 2005. Reachability Analysis for Uncertain SSPs. In ICTAI , 515--522. IEEE Computer Society

    2005

  6. [14]

    K.; Karrabi, M.; Novotn \' y , P.; and Zikelic, D

    Chatterjee, K.; Goharshady, E. K.; Karrabi, M.; Novotn \' y , P.; and Zikelic, D. 2023. Solving Long-run Average Reward Robust MDPs via Stochastic Games. CoRR, abs/2312.13912

    2023 arXiv

  7. [15]

    K.; Karrabi, M.; Novotn \' y , P.; and Zikelic, D

    Chatterjee, K.; Goharshady, E. K.; Karrabi, M.; Novotn \' y , P.; and Zikelic, D. 2024. Solving Long-run Average Reward Robust MDPs via Stochastic Games. In Larson, K., ed., Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 , 6...

    2024

  8. [16]

    Chatterjee, K.; Sen, K.; and Henzinger, T. A. 2008. Model-Checking omega-Regular Properties of Interval Markov Chains. In FoSSaCS, volume 4962 of Lecture Notes in Computer Science, 302--317. Springer

    2008

  9. [17]

    Z.; Parker, D.; and Simaitis, A

    Chen, T.; Forejt, V.; Kwiatkowska, M. Z.; Parker, D.; and Simaitis, A. 2013. Automatic verification of competitive stochastic systems. Formal Methods Syst. Des., 43(1): 61--92

    2013

  10. [18]

    Condon, A. 1992. The Complexity of Stochastic Games. Inf. Comput., 96(2): 203--224

    1992

  11. [19]

    A.; Kret \' nsk \' y , J.; and Petrov, T

    Daca, P.; Henzinger, T. A.; Kret \' nsk \' y , J.; and Petrov, T. 2017. Faster Statistical Model Checking for Unbounded Temporal Properties. ACM Trans. Comput. Log. , 18(2): 12:1--12:25

    2017

  12. [20]

    De Alfaro, L. 1997. Formal verification of probabilistic systems. Ph.D. thesis, Stanford university

    1997

  13. [21]

    de Alfaro, L.; and Henzinger, T. A. 2000. Concurrent Omega-Regular Games. In LICS , 141--154. IEEE Computer Society

    2000

  14. [22]

    Eisentraut, J.; Kelmendi, E.; Kret \' nsk \' y , J.; and Weininger, M. 2022. Value iteration for simple stochastic games: Stopping criterion and learning algorithm. Inf. Comput., 285(Part): 104886

    2022

  15. [23]

    Givan, R.; Leach, S.; and Dean, T. 2000. Bounded-parameter M arkov decision processes. Artificial Intelligence, 122(1-2): 71--109

    2000

  16. [24]

    Grand - Cl \' e ment, J.; Petrik, M.; and Vieille, N. 2023. Beyond discounted returns: Robust Markov decision processes with average and Blackwell optimality. CoRR, abs/2312.03618

    2023 arXiv

  17. [25]

    Gr\"unbaum, B. 2003. Convex Polytopes. Springer, second edition. ISBN 978-0-387-00424-2

    2003

  18. [26]

    Haddad, S.; and Monmege, B. 2018. Interval iteration algorithm for MDP s and IMDP s. Theor. Comput. Sci., 735: 111--131

    2018

  19. [27]

    Hartmanns, A.; Junges, S.; Quatmann, T.; and Weininger, M. 2023. A Practitioner's Guide to MDP Model Checking Algorithms. In TACAS (1) , volume 13993 of Lecture Notes in Computer Science, 469--488. Springer

    2023

  20. [28]

    Hartmanns, A.; and Kaminski, B. L. 2020. Optimistic Value Iteration. In CAV (2) , volume 12225 of Lecture Notes in Computer Science, 488--511. Springer

    2020

  21. [29]

    Hartmanns, A.; Klauck, M.; Parker, D.; Quatmann, T.; and Ruijters, E. 2019. The Quantitative Verification Benchmark Set. In TACAS (1) , volume 11427 of Lecture Notes in Computer Science, 344--350. Springer

    2019

  22. [30]

    Hensel, C.; Junges, S.; Katoen, J.; Quatmann, T.; and Volk, M. 2022. The probabilistic model checker Storm. Int. J. Softw. Tools Technol. Transf., 24(4): 589--610

    2022

  23. [31]

    P.; Petrik, M.; and Wiesemann, W

    Ho, C. P.; Petrik, M.; and Wiesemann, W. 2018. Fast Bellman Updates for Robust MDPs. In ICML , volume 80 of Proceedings of Machine Learning Research, 1984--1993. PMLR

    2018

  24. [32]

    Hölder, O. 1889. Ueber einen Mittelwerthabsatz. Nachrichten von der Königl. Gesellschaft der Wissenschaften und der Georg-Augusts-Universität zu Göttingen, 38--47

  25. [33]

    Iyengar, G. N. 2005. Robust Dynamic Programming. Math. Oper. Res., 30(2): 257--280

    2005

  26. [34]

    Jansen, N.; Junges, S.; and Katoen, J. 2022. Parameter Synthesis in Markov Models: A Gentle Survey. In Principles of Systems Design, volume 13660 of Lecture Notes in Computer Science, 407--437. Springer

    2022

  27. [35]

    Karmarkar, N. 1984. A new polynomial-time algorithm for linear programming. Comb., 4(4): 373--396

    1984

  28. [36]

    Kret \' nsk \' y , J.; Meggendorfer, T.; and Weininger, M. 2023 a . Stopping Criteria for Value Iteration on Stochastic Games with Quantitative Objectives. In LICS , 1--14. IEEE

    2023

  29. [37]

    Kret \' nsk \' y , J.; Meggendorfer, T.; and Weininger, M. 2023 b . Stopping Criteria for Value Iteration on Stochastic Games with Quantitative Objectives. CoRR, abs/2304.09930

    2023 arXiv

  30. [38]

    Kwiatkowska, M.; Norman, G.; and Parker, D. 2011. PRISM 4.0: Verification of Probabilistic Real-time Systems. In Gopalakrishnan, G.; and Qadeer, S., eds., Proc. 23rd International Conference on Computer Aided Verification (CAV'11), volume 6806 of LNCS, 585--591. Springer

    2011

  31. [39]

    B.; and Belta, C

    Lahijanian, M.; Andersson, S. B.; and Belta, C. 2015. Formal Verification and Synthesis for Discrete-Time Stochastic Systems. IEEE Trans. Autom. Control. , 60(8): 2031--2045

    2015

  32. [40]

    B.; Lahijanian, M.; and Laurenti, L

    Mathiesen, F. B.; Lahijanian, M.; and Laurenti, L. 2024. IntervalMDP.jl: Accelerated Value Iteration for Interval Markov Decision Processes. IFAC-PapersOnLine, 58(11): 1--6. ADHS 2024

    2024

  33. [41]

    Meggendorfer, T. 2024. Solving Robust Markov Decision Processes: Generic, Reliable, Efficient (artefact). Zenodo. h ttps://doi.org/10.5281/zenodo.14385450

    2024 doi

  34. [42]

    Meggendorfer, T.; and Weininger, M. 2024 a . Playing Games with Your PET: Extending the Partial Exploration Tool to Stochastic Games. In CAV (3) , volume 14683 of Lecture Notes in Computer Science, 359--372. Springer

    2024

  35. [43]

    Meggendorfer, T.; and Weininger, M. 2024 b . Playing Games with your PET: Extending the Partial Exploration Tool to Stochastic Games. CoRR, abs/2405.03885

    2024 arXiv

  36. [44]

    Nilim, A.; and Ghaoui, L. E. 2005. Robust Control of M arkov Decision Processes with Uncertain Transition Matrices. Oper. Res., 53(5): 780--798

    2005

  37. [45]

    Puterman, M. L. 1994. M arkov decision processes: Discrete stochastic dynamic programming . John Wiley and Sons

    1994

  38. [46]

    Sen, K.; Viswanathan, M.; and Agha, G. 2006. Model-Checking Markov Chains in the Presence of Uncertainties. In TACAS , volume 3920 of Lecture Notes in Computer Science, 394--410. Springer

    2006

  39. [47]

    L.; and Littman, M

    Strehl, A. L.; and Littman, M. L. 2004. An empirical evaluation of interval estimation for M arkov decision processes. In 16th IEEE International Conference on Tools with Artificial Intelligence, 128--135. IEEE

    2004

  40. [48]

    Tewari, A.; and Bartlett, P. L. 2007. Bounded Parameter Markov Decision Processes with Average Reward Criterion. In COLT , volume 4539 of Lecture Notes in Computer Science, 263--277. Springer

    2007

  41. [49]

    K.; Prater - Bennette, A.; and Zou, S

    Wang, Y.; Velasquez, A.; Atia, G. K.; Prater - Bennette, A.; and Zou, S. 2023. Robust Average-Reward M arkov Decision Processes. In Williams, B.; Chen, Y.; and Neville, J., eds., Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on I...

    2023

  42. [50]

    K.; Prater - Bennette, A.; and Zou, S

    Wang, Y.; Velasquez, A.; Atia, G. K.; Prater - Bennette, A.; and Zou, S. 2024. Robust Average-Reward Reinforcement Learning. J. Artif. Intell. Res., 80: 719--803

    2024

  43. [51]

    Wu, D.; and Koutsoukos, X. D. 2008. Reachability analysis of uncertain systems using bounded-parameter Markov decision processes. Artif. Intell., 172(8-9): 945--954

    2008

  44. [52]

    Yang, I. 2017. A Convex Optimization Approach to Distributionally Robust Markov Decision Processes With Wasserstein Distance. IEEE Control Systems Letters, 1(1): 164--169

    2017

  45. [53]

    Zhang, Y.; Steimle, L.; and Denton, B. T. 2017. Robust Markov decision processes for medical treatment decisions. Optimization online

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.