Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Bayesian Composite Risk Approach for Stochastic Optimal Control and Markov Decision Processes

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes a Bayesian composite risk framework for Markov decision problems that unifies risk-averse, distributionally robust, and Bayes-adaptive models, and proves that the learned optimal values and policies converge to the true…

desk verdict A useful synthesis of Bayesian learning and risk-averse MDPs, but the asymptotic convergence theorem rests on an unproven Bayesian consistency assumption and a technical gap in the proof. read the letter →

arxiv 2412.16488 v3 pith:6NUR5DAZ submitted 2024-12-21 math.OC

Wentao Ma , Zhiping Chen , Huifu Xu This is my paper · ORCID
classification math.OC MSC 90C4090C1593E2049K4562F1591B30
keywords BayesiancompositeriskmeasureMarkovdecisionprocessstochasticoptimalcontrolepistemicuncertaintyaleatoricdynamicmeasuresBayes-adaptiveMDPasymptoticconvergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the two sources of difficulty in sequential decision-making under uncertainty — the environment's inherent randomness (aleatoric uncertainty) and the decision-maker's ignorance of the true model parameters (epistemic uncertainty) — can be treated by a single object: a Bayesian composite risk measure, i.e. an outer risk measure applied to the posterior distribution of the unknown parameter composed with an inner risk measure applied to the environmental randomness. The resulting model, called BCR-SOC/MDP, lets the chosen action depend on the evolving belief about the unknown parameter rather than only on the physical state, unlike the episodic Bayesian models it extends. The paper shows that this one framework contains conventional risk-averse, distributionally robust, Bayes-adaptive, and Bayesian-risk Markov decision models as special cases, and that it is solvable: dynamic programming for finite horizons, value iteration on a Bellman equation for infinite horizons. Its central result is that as observations accumulate and the posterior concentrates on the true parameter, the optimal value function converges uniformly in the state to the true model's value function, and the optimal actions converge almost surely to the true optimal action set. If true, this offers a middle path between optimistic point-estimate learning and pessimistic worst-case robustness, with the decision-maker's risk attitude itself allowed to evolve as information arrives.

What carries the argument

The load-bearing object is the Bayesian composite risk (BCR) measure $\rho_{\mu_t} \circ \rho_{P_\theta}$: an outer risk measure over the posterior distribution $\mu_t$ of the unknown parameter $\theta$ composed with an inner law-invariant risk measure over the distribution $P_\theta$ of the randomness. It does two jobs at once — it prices aleatoric risk conditional on each candidate parameter value, and it aggregates those conditional prices according to the decision-maker's stance toward epistemic uncertainty. The argument is carried by three supporting mechanisms. First, the dynamic programming recursion on the augmented state $(s,\mu)$, whose Bellman operator $TV(s,\mu) = \min_a \rho_\mu \circ \rho_{P_\theta}[C(s,a,\xi) + \gamma V(g(s,a,\xi),\mu')]$ is contractive (giving a unique fixed point) and monotone (giving time-consistent policies). Second, Assumption 5, a local H\"older condition $|\rho_{P_\theta}(Z) - \rho_{P_{\theta_c}}(Z)| \leq L^{\mathrm{in}}_\rho |\theta - \theta_c|^k$ on the inner risk measure near the true parameter, which is the bridge converting posterior concentration into value-function convergence. Third, the hyper-parameter discretization of the belief space, which replaces the infinite-dimensional posterior by a finite adaptive grid of sufficient-statistic vectors $h$ updated by $h' = h + H(\xi)$, with error controlled by Kantorovich-distance bounds.

What would settle it

Two concrete checks would settle the claim. First, run the infinite-horizon inventory experiment with a deliberately misspecified likelihood (for example, true demand drawn from a negative-binomial while the manager believes it is Poisson): the posterior mean then converges to the wrong parameter, yet all other assumptions of Theorem 2 still hold, so the theorem predicts the gap $\sup_s |V^*(s,\mu_t) - V^*(s,\delta_{\theta_c})|$ should not vanish — observing it stay bounded away from zero confirms the theorem's dependence on concentration, while observing it vanish would refute it. Second, in the correctly specified Poisson-Gamma setting, Example 10 predicts $t\max\{(m_t-\theta_c)^2, v_t\} \to \theta_c$, so with the expectation outer measure and $k=2, M_1=1$ the empirical value gap should decay like $O(1/t)$; measuring the slope of the log-gap over episodes and finding it far from $-1$ would falsify the rate claim of Theorem 3.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a Markov decision process whose objective is a Bayesian composite risk measure is a well-defined, time-consistent, computationally solvable model that asymptotically learns the true environment. Theorem 2 is the anchor: under Assumptions 1, 3, 4 and 5, the infinite-horizon optimal value $V^*(s,\mu_t)$ converges to $V^*(s,\delta_{\theta_c})$ uniformly in the state $s$, and the optimal policy $\pi^*(s,\mu_t)$ converges almost surely to the set of actions optimal under the true parameter $\theta_c$, as the episode index $t$ grows. Theorem 3 quantifies this: when the posterior mean and variance obey the concentration condition $\lim_t t^{M_1}\max\{(m_t-\theta_c)^2, v_t\} = M_2$, the value gap is bounded by explicit powers of the posterior variance plus squared bias — a rate $O(t^{-kM_1/2})$ for outer measures of expectation, VaR, and robust spectral type, and $O(t^{-kM_1/4})$ for the AVaR outer measure. The paper further argues that the model subsumes distributionally robust and Bayes-adaptive SOC/MDP models as special cases, connects to preference-robust models, and that its hyper-parameter discretization of the belief space makes the augmented-state Bellman recursion tractable with provable approximation error bounds and logarithmic SAA sample sizes.

Load-bearing premise

The convergence result stands on the assumption that the decision-maker's belief about the unknown parameter actually collapses onto the true value as data accumulate (posterior mean to $\theta_c$, posterior variance to zero) — a Bayesian-consistency property the paper assumes and illustrates on conjugate examples but does not derive from primitive conditions; with a poor prior or a misspecified model it fails, and everything built on Theorem 2 gives way.

Editorial extensions

If this is right

  • A single model now covers the standard families: a point-mass outer measure recovers conventional risk-averse SOC/MDP, a worst-case (VaR at level zero) outer measure recovers distributionally robust SOC/MDP, and expectation for both measures recovers Bayes-adaptive MDP — so the algorithmic and convergence results carry over to all of these lines of work.
  • Risk-averse learning is safe in the limit: with accumulating data the decision-maker's value function converges uniformly to the true environment's value function and the chosen actions converge almost surely to the true optimal action set, so the adaptive model is not merely heuristic.
  • The convergence is quantitative: under the posterior rate $t^{-M_1}$ the value gap decays as $O(t^{-kM_1/2})$ for expectation, VaR, and robust spectral outer measures and as $O(t^{-kM_1/4})$ for AVaR, improving on the $O(1/\sqrt{t})$ baseline of the episodic Bayesian model when the inner risk measure is locally smooth.
  • The belief space does not have to be discretized directly: conjugate-prior hyper-parameters, updated by adding a sufficient statistic, support a finite adaptive grid whose Kantorovich error is bounded with high probability, making value iteration and dynamic programming implementable in continuous-state problems.
  • Finite-horizon solutions are time-consistent: the nested composite risk structure matches the nested Bellman recursion exactly, so an action chosen early remains optimal as later observations arrive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The model parameterizes a continuum of conservativeness between pure learning and pure worst-case hedging (the outer measure interpolates from expectation to essential supremum), which suggests that risk levels could be tuned adaptively as data accumulate rather than fixed in advance — a design rule the paper does not state but its examples imply.
  • Theorem 2 splits consistency into a statistical part (posterior concentration) and an analytic part (local smoothness of the risk functional in the parameter); this modularity predicts that swapping in a heavy-tailed or misspecified likelihood degrades the value bound exactly to the slower actual concentration rate, a quantitative claim that could be tested numerically.
  • Because the bound depends on the posterior through $\mathbb{E}_{\mu_t}[|\theta-\theta_c|^k]$ alone, the posterior mean and variance could serve as a stopping certificate in online settings — collect data until the estimated sup-norm gap falls below a target; this is directly checkable in the inventory experiment with its Poisson-Gamma posterior.
  • The paper's finding that the inner (aleatoric) risk level $\beta$ shifts decisions more than the outer (epistemic) level $\alpha$ in spread betting suggests an asymmetry between the two uncertainty types that, if it generalizes, would guide how much effort to spend on reducing each kind of uncertainty in practice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces BCR-SOC/MDP, a stochastic optimal control / Markov decision process framework in which both aleatoric uncertainty (through an inner law-invariant risk measure ρ_{P_θ}) and epistemic uncertainty (through an outer law-invariant risk measure ρ_{μ_t} over the unknown parameter θ) are quantified by a Bayesian composite risk measure. The model is shown to subsume risk-averse, distributionally robust, Bayes-adaptive, and preference-robust SOC/MDP models. For the finite-horizon case, the authors prove well-definedness and give a dynamic programming recursion; for the infinite-horizon case they establish a Bellman equation, a fixed-point characterization, and an asymptotic convergence result (Theorem 2) stating that the optimal value and optimal policy converge, almost surely, to the counterparts for the true parameter as the posterior concentrates. They also provide quantitative convergence rates for specific outer risk measures (Theorem 3), a hyper-parameter discretization scheme for the posterior space (Section 6), SAA-based algorithms for VaR and A VaR composite risk models, and numerical experiments on spread betting and inventory control problems.

Significance. The BCR-SOC/MDP framework is a genuinely useful synthesis: it cleanly unifies several previously separate modeling paradigms (risk-averse, distributionally robust, Bayes-adaptive, and preference-robust SOC/MDP), and it provides algorithms and numerical evidence that the framework is computationally tractable in nontrivial problems. The paper is strong on modeling breadth and on the algorithmic side (the hyper-parameter adaptive grid and the SAA treatments are well motivated). The central theoretical claim, however, is the asymptotic learning theorem (Theorem 2), and that result is currently conditional on a Bayesian consistency assumption that is not derived, plus a technical gap in the application of Lemma 4 to unbounded functions. If those issues are repaired, the paper would be a useful contribution to risk-averse adaptive decision-making; in its current form, the main convergence result is not self-contained and not fully proven.

major comments (4)
  1. [Section 5.2 / Assumption 4] Theorem 2, the paper's main asymptotic claim, rests entirely on Assumption 4, which postulates that the posterior mean m_t converges to θ_c and the posterior variance v_t converges to 0 almost surely. This is a Bayesian consistency statement, not a consequence of the BCR-SOC/MDP structure. The paper cites the Bernstein–von Mises theorem and verifies the assumption in a conjugate Poisson–Gamma example, but it never states primitive conditions (prior support, identifiability, likelihood regularity, or an explicit i.i.d. sampling model) under which Assumption 4 holds in the general setting of the paper. If the posterior does not concentrate, the bound in (5.23) has no reason to vanish and Theorem 2 collapses. The authors need either to prove Assumption 4 from more basic assumptions or to state it explicitly as a statistical hypothesis and discuss its scope.
  2. [Section 5.2, Lemma 4 and Theorem 2 proof] Lemma 4 is stated for bounded continuous h, but in the proof of Theorem 2 (the fourth inequality in (5.23)) the lemma is applied with h(θ) = L_in^ρ |θ − θ_c|^k, which is unbounded on typical parameter spaces such as Θ = R_+ in the inventory control example. The proof therefore has a gap: one needs additional uniform-integrability or moment conditions to conclude that ∫ h(θ) μ_t(dθ) → 0 from weak convergence of μ_t to δ_{θ_c}. This is a load-bearing step in the qualitative convergence result, and it should be fixed by either extending Lemma 4 to unbounded h under suitable growth conditions or by proving directly that E_{μ_t}|θ − θ_c|^k → 0 under Assumption 4.
  3. [Section 5.1, 5.3 (Lemmas 2, 3 and Theorem 4)] The proofs of Lemma 2 (equivalence of deterministic and randomized Bellman operators), Lemma 3 (monotonicity and γ-contraction), and Theorem 4 (uniform convergence of value iteration) are deferred to 'the online version of this paper [45]'. Reference [45] is the very same arXiv preprint, so this is a self-deferral. These results are central to the infinite-horizon theory: contraction and fixed-point properties underpin the Bellman equation and the value iteration algorithm. The paper is not self-contained as a proof of its main infinite-horizon claims unless these proofs are included in the manuscript or the reference is a published independent source.
  4. [Section 6, Assumption 7] Assumption 7 asserts that every posterior belief depends on a finite-dimensional hyper-parameter h with an additive update h' = h + H(ξ). This is substantially stronger than the conjugate-exponential-family motivation given in the text and is not satisfied by some standard conjugate families discussed earlier in the paper. For instance, in Example 2(b) (normal-normal with unknown mean), the posterior mean update is m_t = λ_t m_{t−1} + (1 − λ_t)ξ_{t−1}, which is not of the form h + H(ξ) unless additional restrictions are imposed. Since the hyper-parameter discretization and projection guarantees in Section 6 rely on Assumption 7, the scope of the algorithmic results should be qualified accordingly, or the assumption should be relaxed to cover the normal-normal case used in the motivating examples.
minor comments (5)
  1. [Section 5.2, Lemma 4] The statement 'since μ_t converges to δ_{θ_c} weakly and by Lemma 1, ρ_{μ_t}(h(θ)) converges to h(θ_c)' is imprecise: Lemma 1 concerns continuity of a risk measure in the distribution of a random variable entering through a parametric function f(z,X), not directly the continuity of the outer risk functional ρ_μ(h(θ)) with respect to weak convergence of μ. The intended result is standard (Lipschitz continuity of law-invariant monetary risk measures with respect to the Kantorovich metric), but the citation to Lemma 1 should be replaced or supplemented.
  2. [Section 5.2.2, Theorem 3] In part (ii), the derivation of the VaR bound treats the quantile of |ψ(θ)| via the inequality Q_{μ_t}(|ψ(θ)| ≤ L_in^ρ ε) ≥ 1 − (v_t + (m_t − θ_c)^2)/ε^{2/k}. This is stated without proof; since it is a key step in the quantitative bound, a short derivation or a reference would improve readability.
  3. [Section 7.1, Table 1] The DR-SOC/MDP row reports mean and variance exactly zero for all sample sizes. The text explains this as over-conservatism that leads to 'refraining from betting', but a reader would benefit from explicit confirmation in the table caption or the surrounding text that the optimal policy is a = 0 for all tested instances.
  4. [General] There are several typographical issues, including 'Lipschtz' for 'Lipschitz' (Proposition 4), 'folloowing' (Proposition 9), and inconsistent use of 'A VaR' vs 'AVaR' vs 'A VaR' in Section 6. These should be corrected in a final revision.
  5. [Section 3.1, Example 4] The line 'if we set the inner risk measure ρ_{P_θ} as the expectation E_{P_θ} and the outer risk measure as essential supremum (VaR_μ^α with α = 0)' is slightly confusing, since VaR_μ^α at α = 0 is a quantile, not the essential supremum. Clarify the intended limiting interpretation.

Circularity Check

1 steps flagged · score 2.0 of 10

No definitional or fitted-input circularity; central convergence theorem is a genuine derivation from Assumptions 4 and 5, but foundational proofs are repeatedly deferred to the paper's own arXiv version, a minor self-citation provenance issue.

  1. self citation load bearing [Section 5.1, Lemma 2, Lemma 3, Theorem 4 (proofs deferred to reference [45])]
    "The proof is somewhat standard in the literature of MDP, see details in the online version of this paper [45]. ... The proof for this lemma is standard in the literature of MDP, see details in the online version of this paper [45]. ... The proof is standard in the literature of MDP, see details in the online version of this paper [45]."

    Reference [45] is the paper's own arXiv version, arXiv:2412.16488, by the same three authors. Lemma 3 (monotonicity and contraction of the Bellman operator) is load-bearing for Theorem 1, which establishes the existence and uniqueness of the optimal value function, and Theorem 4 (uniform convergence of value iteration) likewise rests on a proof cited to [45]. The paper does not supply independent verification of these foundational results in the current text. However, these are standard dynamic-programming facts, not the paper's Bayesian-learning conclusion, so the self-citation is a provenance weakness rather than a forced circularity.

full rationale

The central claim, Theorem 2, derives uniform convergence of V*(s, mu_t) to V*(s, delta_theta_c) and almost-sure policy convergence from explicit assumptions: posterior mean/variance consistency (Assumption 4), local Holder continuity of the inner risk measure (Assumption 5), and the contraction properties of the Bellman operator. No parameter is fitted to make the theorem true, and no quantity is defined in terms of the conclusion. The proof of the key estimate (5.23) genuinely uses Lemma 4, the weak convergence of the posterior, and Assumption 5; it does not assume the value-function convergence it proves. A separate correctness concern is that Lemma 4 is stated only for bounded continuous h, while the proof applies it to the unbounded h(theta)=L|theta-theta_c|^k over parameter spaces like Theta=R_+, requiring extra moment or uniform-integrability control not supplied by the assumptions; this is a rigor gap, not circularity. The only circularity-adjacent issue is the repeated deferral of standard lemmas and theorems to the paper's own online version [45], which is a self-citation used for foundational dynamic-programming proofs. These citations do not smuggle in the paper's intended conclusion, and the central convergence result retains independent content, so the appropriate score is 2.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The framework rests on standard stochastic control regularity conditions. The most important load-bearing assumption is Assumption 4, which asserts posterior consistency (m_t→θ_c, v_t→0); this is a statistical regularity condition rather than a derived result. Assumption 7 restricts the tractable algorithms to conjugate-exponential families. No free parameters are fitted to data; numerical risk levels α and β are user-specified risk preferences.

assumptions (8)
  • domain assumption Assumption 1: cost functions satisfy growth conditions (3.4)-(3.5), ξ_t are i.i.d. with unknown distribution P_θc in a parametric family, and the action space is compact and convex.
    Invoked in Proposition 2 to establish well-definedness of the BCR-SOC/MDP objective.
  • domain assumption Assumption 2: C_t and g_t are jointly convex in (s_t, a_t) and non-decreasing in s_t.
    Used in Proposition 4 to guarantee the Bellman recursion is convex and the infimum is attained.
  • domain assumption Assumption 3: for infinite horizon, transition and cost are time-invariant, cost is bounded, and discount factor γ<1.
    Standard bounded-cost condition for the infinite-horizon contraction arguments.
  • domain assumption Assumption 4: posterior mean m_t converges to θ_c and posterior variance v_t converges to 0 almost surely.
    The load-bearing statistical regularity for Theorem 2's asymptotic convergence; not derived from primitive prior/likelihood conditions.
  • domain assumption Assumption 5: the inner risk measure is Hölder continuous in θ at θ_c (inequality 5.17).
    Needed for quantitative rates in Theorem 3 and the uniform bound in Theorem 2.
  • domain assumption Assumption 6: C is strongly convex in a, g is affine in a, and both risk measures are coherent.
    Used in Proposition 8 to show the ε-optimal policy is unique and continuous.
  • ad hoc to paper Assumption 7: every posterior belief is uniquely parameterized by a finite-dimensional hyper-parameter h, with update h' = h + H(ξ).
    Restricts the practical algorithms to conjugate/exponential-family models so the belief space can be discretized; introduced specifically to make Algorithms 1-3 tractable.
  • standard math Banach fixed-point theorem and standard contraction arguments.
    Used in Theorem 1 and Theorem 4 to prove existence and uniqueness of the value function.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Bayesian Composite Risk Approach for Stochastic Optimal Control and Markov Decision Processes." pith.science (2026). https://pith.science/paper/6NUR5DAZ

@misc{pith2026241216488,
  author       = {Pith},
  title        = {Pith review of: A Bayesian Composite Risk Approach for Stochastic Optimal Control and Markov Decision Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NUR5DAZ}},
  note         = {Machine review of arXiv:2412.16488}
}
read the original abstract

Inspired by Shapiro et al.~\cite{shapiro2023episodic}, we consider a stochastic optimal control (SOC) and Markov decision process (MDP) where the risks arising from epistemic and aleatoric uncertainties are assessed using Bayesian composite risk (BCR) measures (Qian et al.~\cite{qian2019composite}). The time dependence of the risk measures allows us to capture the decision maker's (DM) dynamic risk preferences opportunely as increasing information about both uncertainties is obtained. This makes the new BCR-SOC/MDP model more flexible than conventional risk-averse SOC/MDP models. Unlike \cite{shapiro2023episodic} where the control/action at each episode is based on the current state alone, the new model allows the control to depend on the probability distribution of the epistemic uncertainty, which reflects the fact that in many practical instances the cumulative information about epistemic uncertainty often affects the DM's belief about the future aleatoric uncertainty and hence the DM's action \cite{strens2000bayesian}. The new modeling paradigm incorporates several existing SOC/MDP models including distributionally robust SOC/MDP models and Bayes-adaptive MDP models and generates so-called preference robust SOC/MDP models. Moreover, we derive conditions under which the BCR-SOC/MDP model is well-defined, demonstrate that finite-horizon BCR-SOC/MDP models can be solved using dynamic programming techniques, and extend the discussion to the infinite-horizon case. By using Bellman equations, we show that under some standard conditions, asymptotic convergence of the optimal values and optimal actions as the episodic variable goes to infinity is achieved. Finally, we carry out numerical tests on a finite horizon spread betting problem and an inventory control problem and show the effectiveness of the proposed model and numerical schemes.

Figures

Figures reproduced from arXiv: 2412.16488 by the authors.

Figure 1
Figure 1. Posterior distribution of θ based on observations with 2 up days and 3 down days It is possible to express BCR-SOC/MDP as a preference robust SOC/MDP under some specific circumstances. To see this, we consider a BCR where ρPθ = VaRα Pθ for some α ∈ (0, 1) and ρµ = Eµ. The resulting BCR is Eµ ◦ VaRα Pθ (X) = Z Θ VaRα Pθ (X)µ(θ)dθ. We will show shortly that the BCR may be expressed as an SRM. Assume, for fixed α, that… view at source ↗
Figure 2
Figure 2. Histogram of performance frequencies over 100 replications for the AVaR [PITH_FULL_IMAGE:figures/full_fig_p047_2.png] view at source ↗
Figure 3
Figure 3. The distribution of δI and theoretical error bound with different tolerance ϵ and max grid size Mmax. in various environments with the demand following different Poisson distributions. To illustrate the convergence of optimal value function, we use Algorithm 2 to find the unique optimal value function V ∗ of the Bellman equation corresponding to (7.2a). Then, we generate sample sequences ξ t across different episode… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Actual performances of different approaches with different [PITH_FULL_IMAGE:figures/full_fig_p051_4.png]
Figure 5
Figure 5. Figure 5: Convergence of the optimal value function of the VaR [PITH_FULL_IMAGE:figures/full_fig_p051_5.png]
Figure 6
Figure 6. Figure 6: Convergence of the optimal value function of the AVaR [PITH_FULL_IMAGE:figures/full_fig_p052_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Stability Analysis of an Integrated Multistage Stochastic Programming and Markov Decision Process Problem

    math.OC 2025-09 conditional novelty 6.0 of 10

    An integrated MSP-MDP model is shown to admit a nested dynamic reformulation, and its optimal value and solution sets are proven stable under Kantorovich and Fortet-Mourier distribution perturbations.

Reference graph

Works this paper leans on

96 extracted references · 75 canonical work pages · cited by 1 Pith paper

  1. [45]

    Y. Lin, Y. Ren, and E. Zhou , Bayesian risk Markov decision processes , Advances in Neural Information Processing Systems, 35 (2022), pp. 17430–17442

    work page 2022

  2. [1]

    As in Algorithm 1, this requires the set S ×D0 1 to be a finite discrete set

    Steps 2-7 are designed to obtain an approximate optimal value function of problem (5.1). As in Algorithm 1, this requires the set S ×D0 1 to be a finite discrete set. 31 Algorithm 2 Value Iteration Algorithm for Infinite-horizon BCR-SOC/MDP (5.1). 1: Initialization: the precision parameter ε, the iteration number I and the value functions V0 = 0 and V1 = ...

  3. [2]

    Based on this mild assumption, we have the following result

    (6.2) Suppose there exists a p >0 such that the p-th moment of ξ is finite, thus for all R >0, there exists Mp such that Pθc (∥ξ∥ > R) ≤ Mp Rp . Based on this mild assumption, we have the following result. 35 Algorithm 3 Step-Wise Adaptive Grid for Hyper-Parameters 1: Initialize candidate set Υ ← ∅, support size B, max grid size Mmax, projection tolerance...

  4. [3]

    To address the issue, the authors propose an algorithm to solve the BR-MDP (A VaR α µ-EPθ ) which is an approximation of the BR-MDP model (VaRα µ-EPθ )

    is excessively large (10 6), which may result in prohibitively expensive computational cost in terms of CPU time for the proposed BR-MDP model (VaR α µ-EPθ ). To address the issue, the authors propose an algorithm to solve the BR-MDP (A VaR α µ-EPθ ) which is an approximation of the BR-MDP model (VaRα µ-EPθ ). The resulting CPU time is significantly lower...

    2025

  5. [4]

    Acerbi, Spectral measures of risk: A coherent representation of subjective risk aversion, Journal of Banking & Finance, 26 (2002), pp

    C. Acerbi, Spectral measures of risk: A coherent representation of subjective risk aversion, Journal of Banking & Finance, 26 (2002), pp. 1505–1518

    2002

  6. [5]

    Ahmadi, U

    M. Ahmadi, U. Rosolia, M. D. Ingham, R. M. Murray, and A. D. Ames, Constrained risk- averse Markov decision processes, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 11718–11725

    2021

  7. [6]

    C. D. Aliprantis and K. C. Border , Infinite dimensional analysis: a hitchhiker’s guide , Springer Science & Business Media, 2006

    2006

  8. [7]

    B ¨auerle and J

    N. B ¨auerle and J. Ott , Markov decision processes with average-value-at-risk criteria , Mathe- matical Methods of Operations Research, 74 (2011), pp. 361–379

    2011

Show all 96 references
  1. [8]

    R. E. Bellman and S. E. Dreyfus , Applied Dynamic Programming, vol. 2050, Princeton Uni- versity Press, 2015

    2015

  2. [9]

    Bertsekas, Dynamic Programming and Optimal Control: Volume I , Athena scientific, 2012

    D. Bertsekas, Dynamic Programming and Optimal Control: Volume I , Athena scientific, 2012

    2012

  3. [10]

    S. P. Bhat and P. LA , Concentration of risk measures: A wasserstein distance approach , Ad- vances in neural information processing systems, 32 (2019)

    2019

  4. [11]

    J. R. Birge, Y. Feng, N. B. Keskin, and A. Schultz , Dynamic learning and market making in spread betting markets with informed bettors , Operations Research, 69 (2021), pp. 1746–1766

    2021

  5. [12]

    Carpin, Y.-L

    S. Carpin, Y.-L. Chow, and M. Pavone, Risk aversion in finite Markov decision processes using total cost criteria and average value at risk , in 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2016, pp. 335–342

    2016

  6. [13]

    Chen and W

    Z. Chen and W. Ma , A Bayesian approach to data-driven multi-stage stochastic optimization , Journal of Global Optimization, 90 (2024), pp. 401–428

    2024

  7. [14]

    Y. Chow, A. Tamar, S. Mannor, and M. Pavone, Risk-sensitive and robust decision-making: a CVaR optimization approach , Advances in Neural Information Processing Systems, 28 (2015)

    2015

  8. [15]

    Claus, Advancing Stability Analysis of Mean-risk Stochastic programs: Bilevel and Two-stage Models, PhD thesis, Dissertation, Duisburg, Essen, Universit¨ at Duisburg-Essen, 2016

    M. Claus, Advancing Stability Analysis of Mean-risk Stochastic programs: Bilevel and Two-stage Models, PhD thesis, Dissertation, Duisburg, Essen, Universit¨ at Duisburg-Essen, 2016

    2016

  9. [16]

    Dearden, N

    R. Dearden, N. Friedman, and D. Andre , Model-based Bayesian exploration, arXiv preprint arXiv:1301.6690, (2013)

    2013 arXiv

  10. [17]

    Delage and D

    E. Delage and D. A. Iancu , Robust multistage decision making , in The operations research revolution, INFORMS, 2015, pp. 20–46. 53

    2015

  11. [18]

    Delage and S

    E. Delage and S. Mannor, Percentile optimization for Markov decision processes with parameter uncertainty, Operations Research, 58 (2010), pp. 203–213

    2010

  12. [19]

    Delage and Y

    E. Delage and Y. Ye , Distributionally robust optimization under moment uncertainty with application to data-driven problems , Operations Research, 58 (2010), pp. 595–612

    2010

  13. [20]

    Dommel and A

    P. Dommel and A. Pichler, Foundations of Multistage Stochastic Programming, arXiv preprint arXiv:2102.07464, (2021)

    2021 arXiv

  14. [21]

    M. O. Duff , Optimal Learning: Computational procedures for Bayes-adaptive Markov Decision Processes, University of Massachusetts Amherst, 2002

    2002

  15. [22]

    Dupaˇcov´a and K

    J. Dupaˇcov´a and K. Sladk `y, Comparison of multistage stochastic programs with recourse and stochastic dynamic programs with discrete time , ZAMM-Journal of Applied Mathematics and Me- chanics, 82 (2002), pp. 753–765

    2002

  16. [23]

    E. A. Feinberg and A. Shwartz , Handbook of Markov Decision Processes: Methods and Ap- plications, vol. 40, Springer Science & Business Media, 2012

    2012

  17. [24]

    J. A. Filar, D. Krass, and K. W. Ross , Percentile performance criteria for limiting average Markov decision processes, IEEE Transactions on Automatic Control, 40 (1995), pp. 2–10

    1995

  18. [25]

    Gao and A

    R. Gao and A. Kleywegt , Distributionally robust stochastic optimization with Wasserstein distance, Mathematics of Operations Research, 48 (2023), pp. 603–655

    2023

  19. [26]

    Gelman, J

    A. Gelman, J. Carlin, H. Stern, D. Dunson, A. Vehtari, and D. Rubin , Bayesian Data Analysis, Third Edition , Chapman & Hall/CRC Texts in Statistical Science, Taylor & Francis, 2013

    2013

  20. [27]

    Guigues, A

    V. Guigues, A. Shapiro, and Y. Cheng , Risk-averse stochastic optimal control: an efficiently computable statistical upper bound , Operations Research Letters, 51 (2023), pp. 393–400

    2023

  21. [28]

    Guo and H

    S. Guo and H. Xu, Robust spectral risk optimization when the subjective risk aversion is ambigu- ous: a moment-type approach , Mathematical Programming, 194 (2022), pp. 305–340

    2022

  22. [29]

    Gupta, Near-optimal bayesian ambiguity sets for distributionally robust optimization , Manage- ment Science, 65 (2019), pp

    V. Gupta, Near-optimal bayesian ambiguity sets for distributionally robust optimization , Manage- ment Science, 65 (2019), pp. 4242–4260

    2019

  23. [30]

    Gzyl and S

    H. Gzyl and S. Mayoral, On a relationship between distorted and spectral risk measures., (2008)

    2008

  24. [31]

    Hallak, D

    A. Hallak, D. Di Castro, and S. Mannor , Contextual markov decision processes , arXiv preprint arXiv:1502.02259, (2015)

    2015 arXiv

  25. [32]

    J. L. Hau, M. Petrik, and M. Ghavamzadeh, Entropic risk optimization in discounted MDPs , in International Conference on Artificial Intelligence and Statistics, PMLR, 2023, pp. 47–76

    2023

  26. [33]

    Hong and R

    L. Hong and R. Martin , A review of Bayesian asymptotics in general insurance applications , European Actuarial Journal, 7 (2017), pp. 231–255

    2017

  27. [34]

    R. A. Howard, Dynamic Programming and Markov Processes , John Wiley, 1960

    1960

  28. [35]

    R. A. Howard and J. E. Matheson , Risk-sensitive Markov decision processes , Management Science, 18 (1972), pp. 356–369

    1972

  29. [36]

    D. A. Iancu, M. Petrik, and D. Subramanian , Tight approximations of dynamic risk mea- sures, Mathematics of Operations Research, 40 (2015), pp. 655–682

    2015

  30. [37]

    G. N. Iyengar , Robust dynamic programming, Mathematics of Operations Research, 30 (2005), pp. 257–280. 54

    2005

  31. [38]

    Jalota and Y

    D. Jalota and Y. Ye , Stochastic online fisher markets: Static pricing limits and adaptive en- hancements, Operations Research, (2024)

    2024

  32. [39]

    D. R. Jiang and W. B. Powell , Risk-averse approximate dynamic programming with quantile- based risk measures, Mathematics of Operations Research, 43 (2018), pp. 554–579

    2018

  33. [40]

    L. V. Kantorovich and S. Rubinshtein , On a space of totally additive functions , Vestnik of the St. Petersburg University: Mathematics, 13 (1958), pp. 52–59

    1958

  34. [41]

    P. Kern, A. Simroth, and H. Z ¨ahle, First-order sensitivity of the optimal value in a Markov decision model with respect to deviations in the transition probability function, Mathematical Meth- ods of Operations Research, 92 (2020), pp. 165–197

    2020

  35. [42]

    Kusuoka , On law invariant coherent risk measures , Advances in mathematical economics, (2001), pp

    S. Kusuoka , On law invariant coherent risk measures , Advances in mathematical economics, (2001), pp. 83–95

    2001

  36. [43]

    M. Li, X. Tong, and H. Xu , Randomization of spectral risk measure and distributional robust- ness, arXiv preprint arXiv:2212.08871, (2022)

    2022 arXiv

  37. [44]

    Liang and Z

    H. Liang and Z. Luo , Regret bounds for risk-sensitive reinforcement learning with lipschitz dy- namic risk measures , in International Conference on Artificial Intelligence and Statistics, PMLR, 2024, pp. 1774–1782

    2024

  38. [46]

    Luedtke and S

    J. Luedtke and S. Ahmed, A sample approximation approach for optimization with probabilistic constraints, SIAM Journal on Optimization, 19 (2008), pp. 674–699

    2008

  39. [47]

    Ma and Z

    W. Ma and Z. Chen , Multi-stage distributionally robust convex stochastic optimization with Bayesian-type ambiguity sets, Mathematical Methods of Operations Research, 100 (2024), pp. 553– 600

    2024

  40. [48]

    W. Ma, Z. Chen, and H. Xu , A bayesian composite risk approach for stochastic optimal control and markov decision processes , arXiv preprint arXiv:2412.16488, (2024)

    2024 arXiv

  41. [49]

    Mannor, D

    S. Mannor, D. Simester, P. Sun, and J. N. Tsitsiklis , Bias and variance in value function estimation, in Proceedings of the twenty-first International Conference on Machine learning, 2004, p. 72

    2004

  42. [50]

    Mannor and J

    S. Mannor and J. Tsitsiklis , Mean-variance optimization in Markov decision processes , arXiv preprint arXiv:1104.5601, (2011)

    2011 arXiv

  43. [51]

    Mannor and H

    S. Mannor and H. Xu , Data-driven methods for Markov decision problems with parameter un- certainty, in Operations Research & Management Science in the Age of Analytics, INFORMS, 2019, pp. 101–129

    2019

  44. [52]

    Nilim and L

    A. Nilim and L. El Ghaoui , Robust control of Markov decision processes with uncertain transi- tion matrices, Operations Research, 53 (2005), pp. 780–798

    2005

  45. [53]

    Osband, D

    I. Osband, D. Russo, and B. V an Roy , (More) efficient reinforcement learning via posterior sampling, Advances in Neural Information Processing Systems, 26 (2013)

    2013

  46. [54]

    Osogami and T

    T. Osogami and T. Morimura , Time-consistency of optimization problems , in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 26, 2012, pp. 1945–1953

    2012

  47. [55]

    Petrik and D

    M. Petrik and D. Subramanian, An approximate solution method for large risk-averse Markov decision processes, arXiv preprint arXiv:1210.4901, (2012). 55

    2012 arXiv

  48. [56]

    G. C. Pflug and A. Pichler , Multistage Stochastic Optimization , vol. 1104, Springer, 2014

    2014

  49. [57]

    A. B. Philpott, V. L. de Matos, and L. Kapelevich , Distributionally robust SDDP , Com- putational Management Science, 15 (2018), pp. 431–454

    2018

  50. [58]

    Pichler and R

    A. Pichler and R. Schlotter , Risk-averse optimal control in continuous time by nesting risk measures, Mathematics of Operations Research, 48 (2023), pp. 1657–1678

    2023

  51. [59]

    W. B. Powell, H. P. Simao, and B. Bouzaiene-Ayari, Approximate dynamic programming in transportation and logistics: a unified framework , EURO Journal on Transportation and Logistics, 1 (2012), pp. 237–284

    2012

  52. [60]

    M. L. Puterman , Markov Decision Processes: Discrete Stochastic Dynamic Programming , John Wiley & Sons, 2014

    2014

  53. [61]

    P. Qian, Z. W ang, and Z. Wen, A composite risk measure framework for decision making under uncertainty, Journal of the Operations Research Society of China, 7 (2019), pp. 43–68

    2019

  54. [62]

    Rigter, B

    M. Rigter, B. Lacerda, and N. Hawes , Risk-averse Bayes-adaptive reinforcement learning , Advances in Neural Information Processing Systems, 34 (2021), pp. 1142–1154

    2021

  55. [63]

    R. T. Rockafellar, S. Uryasev, et al. , Optimization of conditional value-at-risk , Journal of risk, 2 (2000), pp. 21–42

    2000

  56. [64]

    S. Ross, J. Pineau, B. Chaib-draa, and P. Kreitmann, A Bayesian approach for learning and planning in partially observable Markov decision processes., Journal of Machine Learning Research, 12 (2011)

    2011

  57. [65]

    H. Ruan, Z. Chen, and C. P. Ho , Risk-averse MDPs under reward ambiguity , arXiv preprint arXiv:2301.01045, (2023)

    2023 arXiv

  58. [66]

    Ruszczy´nski, Risk-averse dynamic programming for Markov decision processes, Mathematical Programming, 125 (2010), pp

    A. Ruszczy´nski, Risk-averse dynamic programming for Markov decision processes, Mathematical Programming, 125 (2010), pp. 235–261

    2010

  59. [67]

    Saha and P

    E. Saha and P. K. Ray, Modelling and analysis of inventory management systems in healthcare: A review and reflections , Computers & Industrial Engineering, 137 (2019), p. 106051

    2019

  60. [68]

    Shapiro, Tutorial on risk neutral, distributionally robust and risk averse multistage stochastic programming, European Journal of Operational Research, 288 (2021), pp

    A. Shapiro, Tutorial on risk neutral, distributionally robust and risk averse multistage stochastic programming, European Journal of Operational Research, 288 (2021), pp. 1–13

    2021

  61. [69]

    Shapiro, Distributionally robust modeling of optimal control , Operations Research Letters, 50 (2022), pp

    A. Shapiro, Distributionally robust modeling of optimal control , Operations Research Letters, 50 (2022), pp. 561–567

    2022

  62. [70]

    Shapiro, D

    A. Shapiro, D. Dentcheva, and A. Ruszczynski, Lectures on Stochastic Programming: Mod- eling and Theory , SIAM, 2021

    2021

  63. [71]

    Shapiro, E

    A. Shapiro, E. Zhou, and Y. Lin , Bayesian distributionally robust optimization , SIAM Journal on Optimization, 33 (2023), pp. 1279–1304

    2023

  64. [72]

    Shapiro, E

    A. Shapiro, E. Zhou, Y. Lin, and Y. W ang, Episodic Bayesian optimal control with unknown randomness distributions, arXiv preprint arXiv:2308.08478, (2023)

    2023 arXiv

  65. [73]

    Sharma, J

    A. Sharma, J. Harrison, M. Tsao, and M. Pavone, Robust and adaptive planning under model uncertainty, in Proceedings of the international conference on automated planning and scheduling, vol. 29, 2019, pp. 410–418

    2019

  66. [74]

    Sniedovich, A variance-constrained reservoir control problem, Water Resources Research, 16 (1980), pp

    M. Sniedovich, A variance-constrained reservoir control problem, Water Resources Research, 16 (1980), pp. 271–274. 56

    1980

  67. [75]

    Strasser, Mathematical Theory of Statistics: Statistical Experiments and Asymptotic Decision Theory, vol

    H. Strasser, Mathematical Theory of Statistics: Statistical Experiments and Asymptotic Decision Theory, vol. 7, Walter de Gruyter, 1985

    1985

  68. [76]

    Strens, A Bayesian framework for reinforcement learning , in ICML, 2000, pp

    M. Strens, A Bayesian framework for reinforcement learning , in ICML, 2000, pp. 943–950

    2000

  69. [77]

    Tamar, Y

    A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor , Sequential decision making with coherent risk, IEEE Transactions on Automatic Control, 62 (2016), pp. 3323–3338

    2016

  70. [78]

    Y. Tao, E. Delage, and H. Xu , Risk-averse decision making with contextual information: Model, sample average approximation, and kernelization , arXiv preprint arXiv:2502.16607, (2025)

    2025 arXiv

  71. [79]

    W ang, L

    H. W ang, L. He, R. Gao, and F. Calmon , Aleatoric and epistemic discrimination: Funda- mental limits of fairness interventions , Advances in Neural Information Processing Systems, 36 (2024)

    2024

  72. [80]

    W ang and S

    W. W ang and S. Ahmed, Sample average approximation of expected value constrained stochastic programs, Operations Research Letters, 36 (2008), pp. 515–519

    2008

  73. [81]

    W ang and H

    W. W ang and H. Xu , Robust spectral risk optimization when information on risk spectrum is incomplete, SIAM Journal on Optimization, 30 (2020), pp. 3198–3229

    2020

  74. [82]

    W ang, H

    W. W ang, H. Xu, and T. Ma, Quantitative statistical robustness for tail-dependent law invariant risk measures, Quantitative Finance, 21 (2021), pp. 1669–1685

    2021

  75. [83]

    W ang and E

    Y. W ang and E. Zhou , Bayesian risk-averse Q-Learning with streaming observations , arXiv preprint arXiv:2305.11300, (2023)

    2023 arXiv

  76. [84]

    S. Weber , Distribution-invariant risk measures, information, and dynamic consistency , Mathe- matical Finance: An International Journal of Mathematics, Statistics and Financial Economics, 16 (2006), pp. 419–441

    2006

  77. [85]

    Weitzel and C

    T. Weitzel and C. H. Glock , Energy management for stationary electric energy storage sys- tems: A systematic literature review , European Journal of Operational Research, 264 (2018), pp. 582–606

    2018

  78. [86]

    Wiesemann, D

    W. Wiesemann, D. Kuhn, and B. Rustem , Robust Markov decision processes, Mathematics of Operations Research, 38 (2013), pp. 153–183

    2013

  79. [87]

    Wiesemann, D

    W. Wiesemann, D. Kuhn, and M. Sim, Distributionally robust convex optimization , Operations Research, 62 (2014), pp. 1358–1376

    2014

  80. [88]

    Wozabal, Robustifying convex risk measures for linear portfolios: A nonparametric approach , Operations Research, 62 (2014), pp

    D. Wozabal, Robustifying convex risk measures for linear portfolios: A nonparametric approach , Operations Research, 62 (2014), pp. 1302–1315

    2014

  81. [89]

    D. Wu, H. Zhu, and E. Zhou , A Bayesian risk approach to data-driven stochastic optimization: Formulations and asymptotics , SIAM Journal on Optimization, 28 (2018), pp. 1588–1612

    2018

  82. [90]

    Q. Wu, H. Xu, and H. Zheng , Multistage Robust Average Randomized Spectral Risk Optimiza- tion, arXiv preprint arXiv:2409.00892, (2024)

    2024 arXiv

  83. [91]

    H. Xu, Y. Liu, and H. Sun, Distributionally robust optimization with matrix moment constraints: Lagrange duality and cutting plane methods , Mathematical Programming, 169 (2018), pp. 489–529

    2018

  84. [92]

    Xu and S

    H. Xu and S. Mannor , Distributionally robust Markov decision processes , Advances in Neural Information Processing Systems, 23 (2010)

    2010

  85. [93]

    Xu and D

    H. Xu and D. Zhang , Smooth sample average approximation of stationary points in nonsmooth stochastic optimization and applications , Mathematical Programming, 119 (2009), pp. 371–401. 57

    2009

  86. [94]

    Yang , Wasserstein distributionally robust stochastic control: A data-driven approach , IEEE Transactions on Automatic Control, 66 (2020), pp

    I. Yang , Wasserstein distributionally robust stochastic control: A data-driven approach , IEEE Transactions on Automatic Control, 66 (2020), pp. 3863–3870

    2020

  87. [95]

    Zhang, H

    S. Zhang, H. Xu, and H. Sun, Statistical Robustness of Kernel Learning Estimator with Respect to Data Perturbation , arXiv preprint arXiv:2406.10555, (2024)

    2024 arXiv

  88. [96]

    Zipkin, On the structure of lost-sales inventory models, Operations research, 56 (2008), pp

    P. Zipkin, On the structure of lost-sales inventory models, Operations research, 56 (2008), pp. 937– 944. 9 Appendix Definition 2 (Fortet-Mourier metric). Let Fp(Z) := {h : Z → R : |h(z′) − h(z′′)| ≤Lp(z′, z′′)∥z′ − z′′∥, ∀z′, z′′ ∈ Z} (9.1) be the set of locally Lipschitz con...

    2008

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.