Inside madupite: Technical Design and Performance

John Lygeros; Matilde Gargiani; Philip Pawlowsky; Robin Sieber

REVIEW 3 major objections 5 minor 34 references

Inside madupite: Technical Design and Performance

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Madupite is a distributed solver for finite-state Markov decision processes that, the authors argue, is the only one able to compute exact solutions for problems too large to fit in a laptop's memory, even at discount factors close to one.

desk verdict Real engineering value in a distributed MDP solver, but a mischaracterized 2-norm stopping condition and overbroad uniqueness claims need to be fixed. read the letter →

arxiv 2507.22538 v1 pith:6JH5NJ27 submitted 2025-07-30 cs.SE

Matilde Gargiani , Robin Sieber , Philip Pawlowsky , John Lygeros This is my paper · ORCID

classification cs.SE MSC 90C4065F10

keywords Markovdecisionprocessesinexactpolicyiterationdistributedmemoryparallelismexactdynamicprogrammingdiscountedinfinite-horizonMDPsKrylovsubspacemethodshigh-performancecomputinglarge-scale

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces madupite, a solver for discounted infinite-horizon Markov decision processes with finite state and action spaces, and argues that it is the only solver able to compute exact solutions at a scale that exceeds the memory of a modern laptop, including settings with discount factors close to one. The central bet is that inexact policy iteration, in which each policy evaluation is only approximately solved by an iterative linear solver, can be combined with distributed-memory parallelism to make exact dynamic programming competitive at scale. If the bet holds, practitioners facing large real-world MDPs no longer have to choose between quantifiable optimality guarantees and tractability. The paper backs the claim with benchmarks on epidemic control, inverted pendulum stabilization, and maze navigation, reporting exact solutions for state spaces in the millions.

What carries the argument

The load-bearing object is the solver madupite itself: a distributed implementation of inexact policy iteration built on a portable scientific-computing toolkit that supplies sparse distributed matrices, vectors, and a library of iterative linear solvers. Inexact policy iteration alternates a greedy-policy extraction step with an approximate policy evaluation step, where the inner loop runs an iterative linear solver on (I - gamma P^pi) V = g^pi until a residual-based stopping condition is met; the algorithm's outer tolerance and inner-accuracy parameter $\alpha$ regulate the trade-off. Two implementation details carry the scalability: the transition probability tensor is flattened into an n m by n row-stacked sparse matrix and partitioned so that all rows belonging to the same state live on the same process, and the coefficient matrix is applied matrix-free through a shell operator, avoiding explicit assembly unless a preconditioner such as SOR needs it. The inner stopping condition is computed from the 2-norm of the linear residual, which the solver already produces internally, rather than the infinity norm used in the theoretical description.

What would settle it

Solve a collection of small MDPs with known optimal value functions using the 2-norm inner stopping rule, and compare the infinity-norm Bellman residual at termination with the user-specified tolerance; if the residual exceeds the tolerance for any instance, the exactness claim as stated is false for that configuration.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that madupite establishes a new baseline for exact MDP solving: by storing the transition tensor in a row-block layout distributed over parallel processes and forming the operator I - gamma P^pi matrix-free, it can evaluate policies and extract greedy policies without ever assembling the full system on one machine. The outer loop is an inexact policy iteration whose inner loop solves the policy-evaluation linear system with a user-selected Krylov solver (GMRES, BiCGStab, TFQMR, Richardson, and others), and the stopping rule uses the 2-norm of the linear residual instead of the infinity norm to cut per-iteration overhead. Benchmarks show the solver solving SIS epidemic models of one million states in under ten seconds, an inverted pendulum discretization of about ten million states in about seventeen minutes, and a maze of one million states in under two minutes, while a 95 percent parallelizable workload gives a theoretical maximum speedup near twenty. The paper also demonstrates speedups of roughly two to three times over existing single-machine MDP toolkits, and emphasizes that customizing the inner solver is what unlocks fast convergence in high-discount regimes.

Load-bearing premise

The paper assumes the modified inner-loop stopping rule is at least as demanding as the theoretical one, even though the rule it actually implements can stop earlier, so if that assumption fails, the returned policy may not be as accurate as claimed.

Editorial extensions

If this is right

If the central claim holds, medium-to-large MDPs can be solved exactly on clusters, giving practitioners certified near-optimal policies instead of function approximations of unknown quality.
The high-discount-factor regime, historically the slowest for dynamic programming, becomes tractable when the inner solver is chosen to exploit problem structure.
Distributed storage rather than laptop RAM becomes the practical limit on problem size, so the solver can absorb sparse or dense MDPs that previously had to be simplified or aggregated.
The ability to recover classical algorithms (value iteration, policy iteration, optimistic policy iteration) through parameter settings makes madupite a single testbed for comparing DP variants at scale.
The reported speedups over existing single-machine toolkits suggest that simply upgrading the solution algorithm, not just the hardware, carries much of the benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The switch from infinity-norm to 2-norm inner residuals is described as a performance optimization; an obvious next step is a worst-case analysis showing whether the theoretical convergence guarantee survives, since the 2-norm can hide a large single-component error.
A portable sanity check for the 'exact solution' claim is to run the solver on small MDPs with analytically known value functions and verify the infinity-norm Bellman residual on exit, which the paper does not report.
Because the outer loop is generic, the same distributed machinery is plausibly reusable for risk-averse MDPs or stochastic shortest-path problems, as the paper hints for the risk-averse case.
Amdahl's 95 percent parallelizable estimate implies a speedup ceiling near 20 for the current implementation, so reaching hundreds of cores would likely demand reducing synchronization or overlapping communication with computation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Matilde and co-authors have built a real distributed MDP solver, and the paper gives a level of implementation detail you rarely see in this area. The matrix layout decisions, the matrix-free policy evaluation, and the PETSc integration are explained carefully, and the benchmarks—median over 10 runs, open code, realistic SIS and inverted pendulum problems—support the basic claim that madupite solves million-state, high-discount MDPs on a cluster that would not fit on a laptop. The core iPI algorithm is not new; it is the authors' own prior work [18], and the paper is honest about that. The contribution here is engineering, and the engineering looks solid.

The main soft spot is the inner-loop stopping criterion in Section III-B.3. The text says the 2-norm of the linear residual is a 'more stringent criterion' than the infinity-norm condition in Algorithm 1. That is backwards: for any vector, the infinity-norm is bounded by the 2-norm, so a 2-norm relative-residual threshold is weaker, by up to a factor of sqrt(n). The convergence guarantees for inexact policy iteration in the cited [18] are established for the infinity-norm inner test; the paper provides no proof they survive this change. In practice the outer loop still checks the infinity-norm of the full residual, so if the algorithm terminates, the returned value is within tolerance. But the efficiency and the 'exact solution' claim rest on inner solves being accurate enough for the greedy step to make steady progress, and the 2-norm condition could let the inner solver exit too early. This is a genuine flaw in the argument, not a nitpick. The fix is straightforward: prove the guarantee under the 2-norm, go back to the infinity-norm, or present the 2-norm as a heuristic with supporting experiments. The authors should be asked to address this.

The 'only solver' and 'unique' claims are also overbroad. The comparison is against two toolboxes, pymdptoolbox and mdpsolver. That is enough to say madupite is competitive and scalable, but not enough for 'only solver capable.' Same with the customization claim—unique among the compared set, not unique in the world. The Amdahl p=0.95 estimate comes from one benchmark, so the ~20x speedup ceiling is a fitted number, not a general law. That is a minor caveat, not a fatal one.

Who is this for? Anyone working on exact dynamic programming for large MDPs, or on HPC numerical software. It deserves a serious referee. I would recommend sending it to peer review with a request to fix the stopping-criterion issue and tone down the uniqueness claims.

Referee Report

3 major / 5 minor

Summary. This paper presents madupite, an open-source distributed-memory solver for discounted infinite-horizon MDPs with finite state and action spaces. The solver is built on PETSc and implements inexact policy iteration, with a Python API and user-selectable inner linear solvers, preconditioners, and stopping parameters. The paper describes the algorithmic background (Algorithm 1), data layout and parallelization choices, and reports benchmarks: Amdahl speedup analysis (p≈0.95), discount-factor sensitivity with different inner solvers, scaling to MDPs beyond laptop memory, a comparison with pymdptoolbox and mdpsolver, and case studies in epidemiology (SIS), inverted pendulum control, and 2D maze navigation. The headline claims are that madupite is the only solver that can efficiently compute exact solutions at large scale and near-undiscounted settings and that it is unique in allowing users to customize the inner solution method.

Significance. If the performance and algorithmic claims are substantiated, madupite would be a practically valuable contribution: it is open source, builds on a mature HPC library, provides reproducible benchmarks with median runtimes over 10 runs, and demonstrates solving MDPs with millions of states on a cluster. The case studies are realistic and the comparisons, though narrow, are run on the same hardware. The paper does not provide machine-checked proofs, and the theoretical guarantees are inherited from the cited iPI theory [18]; the main value is engineering and empirical. The strength of the claims in the abstract is currently ahead of the evidence, and the modified inner stopping criterion introduces a correctness gap that must be resolved before the exactness claims can be accepted.

major comments (3)

[III-B.3, Algorithm 1 step 8] The statement that the 2-norm inner stopping criterion is 'more stringent' than the infinity-norm criterion is mathematically incorrect: for any vector x, ∥x∥∞ ≤ ∥x∥2, so the test ∥r_i∥2 < α∥r_0∥2 can be satisfied while ∥r_i∥∞ < α∥r_0∥∞ fails, potentially by a factor up to sqrt(n). Since the iPI convergence and suboptimality guarantees cited from [18] are formulated for the infinity-norm relative-residual stopping rule, the manuscript must either prove that the guarantees remain valid under the 2-norm replacement, revert to the infinity-norm rule, or provide empirical evidence (e.g., reported infinity-norm residuals of returned solutions across all benchmarks) that the modification does not change the convergence behavior. This is load-bearing because the abstract's 'exact solution' claim and the near-undiscounted scalability results in Section IV-C rely on inner solves being accurate enough for the greedy step to make guaranteed progress.
[Abstract; Section IV-A.4] The uniqueness claims—'only solver capable of efficiently computing exact solutions...' and Section IV-A.2's 'only existing MDP solver that enables users to customize'—are supported by comparison with only two open-source Python toolkits (pymdptoolbox and mdpsolver). The evidence establishes superiority on one benchmark instance against these two comparators, not uniqueness in the field. Please broaden the survey/comparison or qualify the claims to the compared toolkits, and report the configuration of mdpsolver's parallel mode (number of threads/cores) and the variance of the runtimes in Table 3.
[Algorithm 1 lines 11-12, 15-16; Section IV] Algorithm 1 returns V_k after breaking on the outer or inner iteration caps (N_o, N_i), without any residual certificate. The evaluation sections do not state whether any benchmark run terminated by reaching a cap rather than by satisfying the tolerance; given the weakened inner stopping criterion, this is a real possibility. The paper should report cap activity for all experiments and, when a cap is hit, either continue to convergence or clearly label the output as not certified. Without this, the reported runtimes cannot be interpreted as times to an exact (tolerance-certified) solution.

minor comments (5)

[Throughout] There are several typos and word-choice issues, e.g., 'benchmakrs' in Section IV-A.1, 'loosing' in Section III-C, and 'without requiring the minimal change' in Section III-B.1 (should be 'without requiring any change').
[Figure 1 and Section IV-A.1] The text says the blue dotted line plots median runtime as a function of the number of ranks, while the caption says it plots median speedup; the axis and caption should be made consistent.
[Section IV-A.4 and Table 3] Please report the mdpsolver parallel configuration (number of threads or cores) and provide standard deviations or confidence intervals; a single benchmark instance with means over 10 runs gives limited evidence for the speedup claims.
[Section III-B.3] MINRES and CG are listed as supported inner solvers, but I−γPπ is generally nonsymmetric; clarify whether these are intended only for MDPs with special structure or are provided as user options without a convergence guarantee.
[Abstract and Section IV] The word 'exact' is used for solutions computed to a finite tolerance (e.g., tol=1e-8); the paper should state explicitly that 'exact' means 'within the user-specified residual tolerance' to avoid overclaiming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: madupite's performance claims rest on external benchmarks of an open-source implementation, not on equations that reduce to their inputs; self-citations point to a separate theoretical paper and a prior code release.

full rationale

I found no circular step. The paper's central claims are empirical: runtimes measured against pymdptoolbox and mdpsolver, strong-scaling results, and the SIS, pendulum, and maze case studies are external measurements of a public codebase, not quantities derived from the conclusions they support. The iPI algorithm is stated explicitly in Algorithm 1, and its convergence properties are cited to [18], a separate theory paper by the same group, rather than assumed as the target result of this paper. Citing one's own prior theory is not circular here because [18] has its own stated assumptions and does not include madupite's benchmark outcomes as an input. The 'only solver' and 'unique customization' statements are unsupported uniqueness claims, which are evidence or correctness issues, not circularity. The one substantive technical concern is Section III-B.3, where the paper describes the 2-norm inner-loop residual criterion as 'more stringent' than the infinity-norm criterion; mathematically the 2-norm criterion is generally weaker, so the cited iPI guarantees from [18] may not transfer unchanged. That is a correctness risk, not a circularity: the solver's returned residual is still checked in the outer loop, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the iPI theory of the same authors (preprint [18]), on an unproved modification of the stopping criterion, and on one fitted parameter (p) used to state the scalability ceiling.

free parameters (1)

Amdahl parallelizable fraction p = 0.95
Estimated by minimizing absolute error between Amdahl's law and measured runtimes on a 100,000-state, 100-action MDP (Section IV-A.1). Used to compute the claimed maximum speedup of ~20.

assumptions (3)

standard math Standard Bellman equation theory for discounted MDPs (contraction property, optimality conditions)
Used implicitly throughout Section II to justify the problem formulation and residual-based stopping.
domain assumption Convergence and complexity guarantees of inexact policy iteration methods as established in [18]
The inner/outer loop structure of Algorithm 1 and its stopping rules are taken from the authors' prior work [18]; the paper explicitly says it is 'out of the scope' to review the mathematical details.
ad hoc to paper The 2-norm inner stopping criterion (Section III-B.3) preserves the theoretical guarantees of iPI
The paper asserts the replacement is 'more stringent' without proof; this is necessary for the claim that madupite computes exact solutions within the specified tolerance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Inside madupite: Technical Design and Performance." pith.science (2026). https://pith.science/paper/6JH5NJ27

@misc{pith2026250722538,
  author       = {Pith},
  title        = {Pith review of: Inside madupite: Technical Design and Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6JH5NJ27}},
  note         = {Machine review of arXiv:2507.22538}
}

read the original abstract

In this work, we introduce and benchmark madupite, a newly proposed high-performance solver designed for large-scale discounted infinite-horizon Markov decision processes with finite state and action spaces. After a brief overview of the class of mathematical optimization methods on which madupite relies, we provide details on implementation choices, technical design and deployment. We then demonstrate its scalability and efficiency by showcasing its performance on the solution of Markov decision processes arising from different application areas, including epidemiology and classical control. Madupite sets a new standard as, to the best of our knowledge, it is the only solver capable of efficiently computing exact solutions for large-scale Markov decision processes, even when these exceed the memory capacity of modern laptops and operate in near-undiscounted settings. This is possible as madupite can work in a fully distributed manner and therefore leverage the memory storage and computation capabilities of modern high-performance computing clusters. This key feature enables the solver to efficiently handle problems of medium to large size in an exact manner instead of necessarily resorting to function approximations. Moreover, madupite is unique in allowing users to customize the solution algorithm to better exploit the specific structure of their problem, significantly accelerating convergence especially in large-discount factor settings. Overall, madupite represents a significant advancement, offering unmatched scalability and flexibility in solving large-scale Markov decision processes.

Figures

Figures reproduced from arXiv: 2507.22538 by the authors.

**Figure 3.** FIGURE 3: Mean runtime (in seconds) over 10 runs versus [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** FIGURE 4: Runtime versus number of states when madupite is used to solve MDPs arising from dynamic SIS models. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: FIGURE 5: Runtime versus number of states when madupite is used to solve MDPs arising from the discretization of an [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: FIGURE 6: Runtime versus number of states when madupite is used to solve MDPs representing navigation tasks in 2D [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: FIGURE 7: Graphical representation of a [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: FIGURE 8: Visualizations of the optimal cost (left) and [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages

[18]

Inexact Policy Iteration Methods for Large-Scale Markov Decision Processes

M. Gargiani, R. Sieber, E. Balta, D. Liao-McPherson, and J. Lygeros. Inexact policy iteration methods for large-scale Markov decision processes.https://arxiv.org/abs/2404.06136, 2024

work page Pith review arXiv 2024
[1]

G. M. Amdahl. Validity of the single processor approach to achieving large-scale computing capabilities. InProceedings of the April 18-20, 1967, Spring Joint Computer Conference, AFIPS ’67 (Spring), pages 483–485. ACM, 1967

work page 1967
[2]

A. R. Andersen and J. F. Andersen. MDPSolver: An efficient solver for Markov decision processes.Journal of Open Source Software, 10(109):7544, 2025

work page 2025
[3]

Balay, S

S. Balay, S. Abhyankar, M. F. Adams, J. Brown, P. Brune, K. Buschel- man, L. Dalcin, V . Eijkhout, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, K. Rupp, B. F. Smith, S. Zampini, and H. Zhang. PETSc users manual. Technical Report ANL-95/11 - Revision 3.6, Argonne National Laboratory, 2015

work page 2015
[4]

Balay, S

S. Balay, S. Abhyankar, M. F. Adams, J. Brown, P. Brune, K. Buschel- man, L. Dalcin, V . Eijkhout, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, K. Rupp, B. F. Smith, S. Zampini, and H. Zhang. PETSc Web page. http://www.mcs.anl.gov/petsc, 2015

work page 2015
[5]

Balay, W

S. Balay, W. D. Gropp, L. C. McInnes, and B. F. Smith. Efficient man- agement of parallelism in object oriented numerical software libraries. In E. Arge, A. M. Bruaset, and H. P. Langtangen, editors,Modern Software Tools in Scientific Computing, pages 163–202. Birkh ¨auser Press, 1997

work page 1997
[6]

Bellman.Dynamic Programming

R. Bellman.Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1 edition, 1957

work page 1957
[7]

D. P. Bertsekas.Dynamic Programming and Optimal Control, Vol. II. Athena Scientific, Belmont, Massachusets, 4th edition, 2012

work page 2012

Show all 34 references

[8]

Boubaker

O. Boubaker. The inverted pendulum: A fundamental benchmark in control theory and robotics.International Conference on Education and e-Learning Innovations, pages 1–6, 2012

2012
[9]

B ¨auerle and U

N. B ¨auerle and U. Rieder.Markov Decision Processes with Applica- tions to Finance. Springer Berlin Heidelberg, 2011

2011
[10]

Chad `es, G

I. Chad `es, G. Chapron, M. Cros, F. Garcia, and R. Sabbadin. Mdptoolbox: A multi-platform toolbox to solve stochastic dynamic programming problems.Ecography, 37(9):916–920, Sept. 2014. 12 VOLUME 00 2021 FIGURE 7: Graphical representation of a33×33-maze. The heatmap shows the ...

2014
[11]

Clarke, I

L. Clarke, I. Glendinning, and R. Hempel. The MPI message passing interface standard. In K. M. Decker and R. M. Rehmann, editors,Programming Environments for Massively Parallel Distributed Systems, pages 213–218, Basel, 1994. Birkh ¨auser Basel

1994
[12]

Euler Computing Cluster

ETH Zurich. Euler Computing Cluster. https://scicomp.ethz.ch/wiki/ Euler. Accessed: 2025-07-24

2025
[13]

R. W. Freund. Transpose-free quasi-minimal residual methods for non- hermitian linear systems. In G. Golub, M. Luskin, and A. Greenbaum, editors,Recent Advances in Iterative Methods, pages 69–94, New York, NY , 1994. Springer New York

1994
[14]

Gargiani, D

M. Gargiani, D. Liao-McPherson, A. Zanelli, and J. Lygeros. Inexact GMRES policy iteration for large-scale Markov decision processes. IFAC-PapersOnLine, 56(2):11249–11254, 2023. 22nd IFAC World Congress

2023
[15]

Gargiani, A

M. Gargiani, A. Martinelli, M. R. Martinez, and J. Lygeros. Parallel and flexible dynamic programming via the mini-batch Bellman oper- ator.IEEE Transactions on Automatic Control, 69(1):455–462, 2024

2024
[16]

Gargiani, F

M. Gargiani, F. Micheli, A. Tsiamis, and J. Lygeros. Semis- mooth Newton methods for risk-averse Markov decision processes. https://arxiv.org/abs/2501.13612, 2025

2025 arXiv
[17]

Gargiani, P

M. Gargiani, P. Pawlowsky, R. Sieber, V . Hapla, and J. Lygeros. madupite: A high-performance distributed solver for large-scale markov decision processes.Journal of Open Source Software, 10(108):7411, 2025

2025
[19]

Gargiani, R

M. Gargiani, R. Sieber, P. Pawlowsky, and J. Lygeros. madupite’s documentation. https://madupite.github.io/, 2024

2024
[20]

Gargiani, A

M. Gargiani, A. Zanelli, D. Liao-McPherson, T. Summers, and J. Lygeros. Dynamic programming through the lens of semismooth Newton-type methods.IEEE Control Systems Letters, 6:1–1, 01 2022

2022
[21]

Gargiani, A

M. Gargiani, A. Zanelli, D. Liao-McPherson, T. Summers, and J. Lygeros. Dynamic programming through the lens of semismooth Newton-type methods (extended version). https://arxiv.org/abs/2203.08678, 2022

2022 arXiv
[22]

Model Predictive Control and Reinforcement Learning

S. Gros. Adaptation of MPC to RL: fundamental principles. Lecture for the course “Model Predictive Control and Reinforcement Learning ” at IMTEK, Faculty of Engineering, University of Freiburg, 2021

2021
[23]

H. W. Hethcote. The mathematics of infectious diseases.SIAM Review, 42(4):599–653, 2000

2000
[24]

Israilov, L

S. Israilov, L. Fu, J. Sanchez-Rodriguez, F. Fusco, G. Allibert, C. Ra- ufaste, and M. Argentina. Reinforcement learning approach to control an inverted pendulum: A general framework for educational purposes, feb 2023

2023
[25]

W. Jakob. nanobind: tiny and efficient c++/python bindings, 2022. https://github.com/wjakob/nanobind

2022
[26]

Lauri, D

M. Lauri, D. Hsu, and J. Pajarinen. Partially observable Markov decision processes in robotics: A survey.IEEE Transactions on Robotics, 39(1):21–40, 2023

2023
[27]

S. Liu, G. Mahajan, D. Kane, S. Lovett, G. Weisz, and C. Szepesv ´ari. Exponential hardness of reinforcement learning with linear function approximation. InProceedings of the Thirty-Sixth Conference on Learning Theory, volume 195, pages 1588–1617, 2023

2023
[28]

Nielsen, E

L. Nielsen, E. Jørgensen, and S. Højsgaard. Embedding a state space model into a Markov decision process.Annals OR, 190:289–309, 10 2011

2011
[29]

A. V . Oppenheim. Lecture 26: Feedback Example: The Inverted Pendulum, 1987

1987
[30]

M. L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994

1994
[31]

Saad.Iterative methods for sparse linear systems

Y . Saad.Iterative methods for sparse linear systems. SIAM, 2003

2003
[32]

Saad and M

Y . Saad and M. H. Schultz. GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems.SIAM Journal on Scientific and Statistical Computing, 7(3):856–869, 1986

1986
[33]

R. S. Sutton and A. G. Barto.Reinforcement Learning: An Introduc- tion. The MIT Press, second edition, 2018

2018
[34]

Yaesoubi and T

R. Yaesoubi and T. Cohen. Generalized Markov models of infectious disease spread: A novel framework for developing dynamic health policies.European Journal of Operational Research, 215(3):679–687, 2011. VOLUME 00 2021 13

2011

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [18]

Inexact Policy Iteration Methods for Large-Scale Markov Decision Processes

M. Gargiani, R. Sieber, E. Balta, D. Liao-McPherson, and J. Lygeros. Inexact policy iteration methods for large-scale Markov decision processes.https://arxiv.org/abs/2404.06136, 2024

work page Pith review arXiv 2024

[2] [1]

G. M. Amdahl. Validity of the single processor approach to achieving large-scale computing capabilities. InProceedings of the April 18-20, 1967, Spring Joint Computer Conference, AFIPS ’67 (Spring), pages 483–485. ACM, 1967

work page 1967

[3] [2]

A. R. Andersen and J. F. Andersen. MDPSolver: An efficient solver for Markov decision processes.Journal of Open Source Software, 10(109):7544, 2025

work page 2025

[4] [3]

Balay, S

S. Balay, S. Abhyankar, M. F. Adams, J. Brown, P. Brune, K. Buschel- man, L. Dalcin, V . Eijkhout, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, K. Rupp, B. F. Smith, S. Zampini, and H. Zhang. PETSc users manual. Technical Report ANL-95/11 - Revision 3.6, Argonne National Laboratory, 2015

work page 2015

[5] [4]

Balay, S

S. Balay, S. Abhyankar, M. F. Adams, J. Brown, P. Brune, K. Buschel- man, L. Dalcin, V . Eijkhout, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, K. Rupp, B. F. Smith, S. Zampini, and H. Zhang. PETSc Web page. http://www.mcs.anl.gov/petsc, 2015

work page 2015

[6] [5]

Balay, W

S. Balay, W. D. Gropp, L. C. McInnes, and B. F. Smith. Efficient man- agement of parallelism in object oriented numerical software libraries. In E. Arge, A. M. Bruaset, and H. P. Langtangen, editors,Modern Software Tools in Scientific Computing, pages 163–202. Birkh ¨auser Press, 1997

work page 1997

[7] [6]

Bellman.Dynamic Programming

R. Bellman.Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1 edition, 1957

work page 1957

[8] [7]

D. P. Bertsekas.Dynamic Programming and Optimal Control, Vol. II. Athena Scientific, Belmont, Massachusets, 4th edition, 2012

work page 2012