Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Distributionally Robust Deep Q-Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that distributionally robust Q-learning can be made practical for continuous state spaces by replacing the Wasserstein ball with a Sinkhorn ball and dualizing the robust Bellman operator into a scalar maximization.

desk verdict A coherent Sinkhorn-ball DQN for continuous state spaces; the prior ν is a load-bearing dial the experiments never turn. read the letter →

arxiv 2505.19058 v1 pith:H7VNZEOD submitted 2025-05-25 cs.LG math.OCq-fin.PMstat.ML

Chung I Lu , Julian Sester , Aijia Zhang This is my paper
classification cs.LGmath.OCq-fin.PMstat.ML
keywords distributionallyrobustQ-learningSinkhorndistanceDeepQ-NetworkmodeluncertaintyMarkovdecisionprocessentropicregularizationneuralnetworkapproximationportfoliooptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to do deep Q-learning when the state-transition model of the environment is not known exactly. Instead of assuming one transition distribution, it builds an ambiguity set, a ball of nearby distributions around a reference measure measured by the Sinkhorn distance, and optimizes against the worst-case transition in that ball. The paper proves that the corresponding value function satisfies a robust Bellman equation, dualizes the robust Bellman operator into a maximization over a single scalar, and packages the result as a modified DQN algorithm called RDQN. If the construction works as claimed, reinforcement learning agents can be trained in continuous state spaces to be conservative against model misspecification, with a tunable radius controlling how much robustness is bought.

What carries the argument

The carrying object is the Sinkhorn ball $B_{\varepsilon,\delta}(\hat{P}(x,a)) = \{P : W_\delta(\hat{P}(x,a),P)\le \varepsilon\}$, where $W_\delta$ is the entropically regularized Wasserstein distance taken against a prior $\nu$. The entropy term makes the robust Bellman operator dualizable: following the Sinkhorn DRO duality, the inner worst-case expectation becomes a maximum over a scalar Lagrange multiplier $\lambda$, with inner expectations sampling from $\nu$ rather than from the environment. That sampling trick is what lets the algorithm estimate targets from a replay buffer and update a deep Q-network by gradient descent.

What would settle it

On a one-dimensional MDP with a known reference Beta(2,2), choose $\nu$=Beta(1,5) and compute the RDQN worst-case cumulative distribution function at small $\delta$; if it agrees with the true infimum over the Sinkhorn ball rather than drifting as in the paper's Figure 2, the claimed $\nu$-sensitivity is falsified, and if the dual Monte Carlo target disagrees with a direct discretized solution of the Bellman infimum, the tractability claim fails.

Watch

Extended reading notes

Core claim

The central claim is that distributionally robust Q-learning is tractable in continuous state spaces when the ambiguity set is a Sinkhorn ball. Concretely, Proposition 2.7 shows that the worst-case value function satisfies $V_\delta(x) = \sup_a \inf_{P \in B_{\varepsilon,\delta}(\hat{P}(x,a))} \mathbb{E}_P[r(x,a,X_1)+\alpha V_\delta(X_1)]$, so dynamic programming applies. Proposition 3.1 rewrites the inner infimum as a supremum over a scalar Lagrange multiplier $\lambda$ of an expression involving only expectations under the reference measure and a user-chosen prior $\nu$, converting an intractable optimization over measures into a convex scalar problem. The paper then parameterizes $Q^*_\delta$ by a neural network, replaces DQN's target with the dual expression, and calls the resulting algorithm RDQN. Under bounded state space, Proposition 3.5 gives existence of a neural-network solution and closeness to the true robust Q-function, and to the Wasserstein-robust Q-function as $\delta\downarrow 0$.

Load-bearing premise

The load-bearing premise is that the user-chosen prior $\nu$ covers the relevant support of the reference transition and satisfies the exponential moment condition; if $\nu$ misses regions where the reference puts mass, the worst-case distribution computed by the algorithm is wrong, as the paper's own Section 3.4 figures show.

Editorial extensions

If this is right

  • An agent trained by RDQN is optimizing against the worst transition within radius $\varepsilon$ of the reference, so its value function is a lower bound on performance across that ambiguity set.
  • Because the dual target samples from $\nu$ instead of the environment, the algorithm can estimate robust targets from replay-buffer transitions plus i.i.d. prior samples, avoiding repeated sampling of the unknown transition.
  • Small $\delta$ approximates the Wasserstein-robust Q-function (Corollary 3.2), giving a practical route to Wasserstein-ball robustness without the intractable $\lambda$-$c$ transform.
  • On bounded state spaces, a solution of the neural-network optimization problem is within $\mathrm{TOL}/(1-\alpha)$ of $Q^*_\delta$ and, for small enough $\delta$, within $2\cdot\mathrm{TOL}/(1-\alpha)$ of the Wasserstein $Q^*_0$.
  • In the reported experiments, the robust agent improves worst-case quantiles in a gambling task and risk-adjusted returns in S&P 500 portfolio optimization, at the cost of lower performance on the reference model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The prior $\nu$ acts as a design choice that blends Bayesian and adversarial robustness: larger $\delta$ pulls the worst-case distribution toward $\nu$, so choosing $\nu$ with heavy tails (as in the portfolio experiment) encodes a subjective belief about where the true distribution might sit, and the paper's figures show the failure mode when $\nu$ under-weights the reference support.
  • The same dualization should extend to other cost geometries and to continuous action spaces, but the scalar-$\lambda$ structure and the Monte Carlo target would need re-derivation; the paper leaves this open.
  • A testable diagnostic is to monitor the warning condition $\bar{\varepsilon}<0$ and compare the learned robust Q-values against a direct discretized solution on a small one-dimensional MDP before trusting the neural approximation.
  • The robustness guarantee is only as good as the support of $\nu$: practitioners should treat $\nu$ as a modeling assumption to be stress-tested, not as a free parameter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Robust DQN (RDQN), a distributionally robust deep Q-learning algorithm for continuous state spaces and discrete action spaces. The uncertainty set is a Sinkhorn ball around a reference transition kernel, and the robust Bellman operator is dualized into a scalar maximization over a Lagrange multiplier λ, following the Sinkhorn DRO duality of [55]. The Q-function is parameterized by a neural network, and Algorithm 1 gives a DQN-style training procedure whose targets are computed via the dual formula with Monte Carlo sampling from a chosen prior ν. Theoretical results include a robust Bellman equation (Proposition 2.7), a duality result (Proposition 3.1), and an existence/approximation result for neural-network fixed points on compact state spaces (Proposition 3.5). Experiments cover a toy gambling environment on the unit square and an S&P 500 portfolio optimization task, where RDQN is compared with DQN in terms of tail rewards and risk-adjusted returns.

Significance. If the derivation and algorithm are correct, this is one of the few practical distributionally robust deep Q-learning methods for continuous state spaces, and the use of the Sinkhorn distance with a flexible prior ν is a useful alternative to Wasserstein-ball formulations whose duals are often intractable. The paper gives a fairly complete transfer of known results [38, 55] to the Sinkhorn-ball setting, including weak compactness and continuity of the ambiguity set, and it ships a reproducible code link. The central theoretical statements are conditional on standard assumptions (compactness/continuity, exponential moments, prior absolute continuity), and the paper is reasonably explicit about the role of ν in Remark 2.3 and Section 3.4. The main weaknesses are a mismatch between Proposition 3.1 and the actual target computation in Algorithm 1, the absence of a convergence or consistency analysis connecting Proposition 3.5 to the stochastic algorithm, and a heavy dependence of the practical robustness claim on the unvalidated prior ν.

major comments (4)
  1. [Section 3.3, Algorithm 1 and Proposition 3.1] The target computation in Algorithm 1 does not match the dual objective in Proposition 3.1. In Algorithm 1, the inner quantity C_{i,j} is computed using the raw parameter λ_i, while the outer expression cHδQNN uses λ_i^+ = log(1+exp(λ_i)). If λ_i is the unconstrained optimization variable, then the inner exponent must use the same positive parameter λ_i^+; if λ_i is already intended to be the positive dual variable, then the outer softplus wrapper changes the objective. As written, the optimized target is not the quantity in Proposition 3.1, so the gradient updates optimize a different objective. This affects every target update and therefore the learned Q-function; please correct the algorithm and clarify the role of λ_i versus λ_i^+.
  2. [Definition 2.2, Remark 2.3, Section 3.4] The robustness guarantee is relative to the chosen prior ν: by Definition 2.2, every probability measure in the Sinkhorn ball B_{ε,δ}(bP(x,a)) must be absolutely continuous with respect to ν, and Section 3.4 (Figures 1 and 2) explicitly demonstrates that a misspecified ν, e.g., Beta(1,5) instead of Uniform(0,1), yields the wrong worst-case distribution even for small δ. Algorithm 1 inherits this dependence because the inner expectation in the target is approximated by sampling x^ν ∼ ν. The experimental sections choose ν heuristically (uniform in §4.1, Student-t in §4.2) with no sensitivity analysis, so the practical claim of robustness to transition misspecification is conditional on an unvalidated prior. Please either provide a principled way to choose or adapt ν, or add a sensitivity analysis and temper the robustness claims accordingly.
  3. [Proposition 3.5 and Algorithm 1] Proposition 3.5 is not connected to Algorithm 1. The proposition assumes an exact solution Q*_NN of Optimization Problem 3.4 under the true operator Hδ, whereas Algorithm 1 replaces Hδ by a single-sample Monte Carlo estimate of the outer expectation, a finite-sample estimate of the inner expectation over ν, and a per-sample stochastic-gradient maximization over λ. No proof is given that the stochastic target converges to the dual objective in Proposition 3.1, nor that the algorithm's iterates converge to a fixed point of Hδ. Consequently, the theoretical guarantee in Proposition 3.5 does not apply to the proposed algorithm as stated. Please either provide a consistency/convergence analysis or state explicitly that Algorithm 1 is a heuristic approximation whose theoretical grounding is only at the level of the exact optimization problem.
  4. [Proposition 3.1] The assumptions in Proposition 3.1 are ill-formulated. The condition ν({0 ≤ ∥y − X^ν_1∥ < ∞}) = 1 uses an undefined variable y, and the exponential moment condition is stated for 'bP(x,a)-almost every y' without defining y or specifying the integrability with respect to which measure. This makes it impossible for a reader to verify the assumptions. In addition, the definition ε := ε + δ E[log E exp(-∥X^P_1 - X^ν_1∥/δ)] ≥ 0 reuses the symbol ε on both sides; use a different symbol, such as ε̄, throughout the paper and in Algorithm 1.
minor comments (5)
  1. [Section 3.3.3] The warning condition is described as 'ε < 0' in the text, but the algorithm computes ¯ε_i; the text should consistently refer to ¯ε_i < 0.
  2. [Sections 3.3.1 and 3.3.4] The two subsections 'Optimisation of λ' are duplicated verbatim; please merge them into one.
  3. [Throughout] There are several typos, including 'theoreticctical gaurantees' and 'Proprosition' in Sections 1.1 and 5, and 'for all |delta ∈ (0, δ′)' in Proposition 3.5(iii).
  4. [Algorithm 1] The stopping criterion 'while ∇λ_i cHδQNN does not change sign' is unclear; please define the sign change over iterations and state how the step size scheduler interacts with this criterion.
  5. [Lemma 6.5] The decomposition Wδ(P1,P2) = Sδ(P1,P2) + δH(P2|ν) should be stated explicitly with the assumption P2 ≪ ν, since it is used later in the proof of lower hemicontinuity.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: core Bellman and duality results are imported from independently published theorems ([38], [55]) with assumptions verified; self-citations are present but not used as circular evidence.

full rationale

The paper's load-bearing steps do not reduce to their inputs. Proposition 2.7 (robust Bellman equation) is proved by verifying weak compactness and continuity of the Sinkhorn ball (Lemmas 6.4, 6.6, 6.7) and then invoking the external theorem [38, Theorem 2.7]; [38] is prior published work by a coauthor, but it is an independent theorem with stated assumptions, and the paper supplies the verification needed to apply it, so this is legitimate import rather than circularity. Proposition 3.1 is explicitly derived from [55, Theorem I], an external Sinkhorn-DRO duality result, and the paper only adapts notation and checks measurability; no parameter of the reported experiments is fed back into the dual formula as a 'prediction'. The fixed-point equation (2.10) is definitional (Q*_δ is defined so that HδQ*_δ=Q*_δ), but that is the standard Q-learning construction, not a claimed empirical prediction. The paper itself flags the prior-dependence limitations: Section 3.3.3 states that when the adjusted radius ε is negative, 'the duality in Proposition 3.1 does not hold and the resulting target is not valid', and Section 3.4 shows that a misspecified ν 'can lead to the wrong worst case distribution'. These passages identify correctness/robustness caveats, not circular reasoning. Self-citations ([36], [38], [30]) are used for the Wasserstein DPP, the generative simulator, and the DQN template, but each is an independent, externally checkable result; the robust Q-function is not fitted to the reported S&P 500 or gambling performance metrics. Accordingly there is no exhibited Eq.-X-equals-Eq.-Y reduction, and the appropriate finding is no significant circularity, with a low score reflecting only the presence of minor self-citations.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The core derivation imports two major external results: the robust Bellman equation framework of [38] and the Sinkhorn dual of [55]. The paper's own contribution is to verify the required compactness and continuity conditions and to turn the dual into a DQN target. The free parameters epsilon, delta, and nu are essential to the empirical results and are tuned by hand; the prior nu is especially consequential because the dual and the Monte Carlo target depend on it. No new entities are postulated.

free parameters (4)
  • Sinkhorn radius epsilon = 0.05, 0.1, 0.2 (toy); 2.5e-3, 3.0e-3, 3.5e-3 (portfolio)
    Controls the size of the ambiguity ball. Chosen by hand and swept across tables; the conclusion openly states that selecting ε relies on empirical tuning.
  • Sinkhorn regularization delta = 1e-4, 1e-5, 1e-6, 0.01, 0.0001 (sweeps)
    Controls smoothness of the Sinkhorn distance and approximation to the Wasserstein case. Swept empirically in Tables 1, 2, and 4.
  • Sampling prior nu parameters = Toy: Uniform(0,1); portfolio: Student's t with mu=0, sigma=0.03, nu=2
    The prior is used in the dual expectation and fixes the support of measures in the ambiguity set. Section 3.4 shows the worst-case distribution is sensitive to this choice; the portfolio parameters were chosen based on visual observation of the simulator.
  • Neural network and training hyperparameters = not fully reported
    Architecture, learning rate, replay buffer size, target network update frequency, and training steps are not fully specified; training length was set based on observing when wealth plateaued.
assumptions (6)
  • domain assumption The reference transition map (x,a) -> bP(x,a) is continuous in the Wasserstein-1 topology and has finite first moment for all (x,a).
    Assumption 2.4 is used throughout to prove compactness and continuity of the Sinkhorn ball and to invoke [38, Theorem 2.7].
  • ad hoc to paper The Sinkhorn radius satisfies epsilon >= sup_{x,a} W_delta(bP(x,a), bP(x,a)).
    Assumption 2.5 is needed because W_delta(P,P) is not zero in general, so the ball may not contain the reference measure. The paper never verifies the inequality in experiments; Algorithm 1 only warns when the empirical epsilonbar is negative.
  • domain assumption The reward function is continuous, bounded, and Lipschitz in (x,a), and the discount factor satisfies 0<alpha<1.
    Standing Assumption 2.6 is required by Proposition 2.7, Lemma 6.8, and the dual derivation.
  • domain assumption Distributions in the ambiguity set are absolutely continuous with respect to the prior nu, and an exponential moment condition holds.
    Proposition 3.1 requires P2 << nu and E[exp(-||y - X^nu||/delta)] finite. If the prior misses critical support, the dual target is biased, as shown in Section 3.4.
  • standard math The robust Bellman equation framework of [38, Theorem 2.7] and the Sinkhorn duality of [55, Theorem I] are correct.
    The central mathematical claims of this paper are imported from these external references; any error in them is inherited by the present derivation.
  • domain assumption Log returns of the S&P 500 are bounded (for the portfolio experiment).
    Used to satisfy Assumption 2.6 in Proposition 4.2; it is stated as a reasonable assumption, not verified from data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributionally Robust Deep Q-Learning." pith.science (2026). https://pith.science/paper/H7VNZEOD

@misc{pith2026250519058,
  author       = {Pith},
  title        = {Pith review of: Distributionally Robust Deep Q-Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H7VNZEOD}},
  note         = {Machine review of arXiv:2505.19058}
}
abstract

We propose a novel distributionally robust $Q$-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterized with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S\&{P}~500 data.

Figures

Figures reproduced from arXiv: 2505.19058 by the authors.

Figure 1
Figure 1. The worst case cumulative distribution function for different values of δ with ν = Uniform(0, 1) Choosing ν with the wrong support or where critical parts of the support of Pb are low in probabil￾ity can lead to the wrong worst case distribution as we see in [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. The worst case cumulative distribution function for different values of δ with ν = Beta(1, 5) In the following, we describe the true distribution of the environment. The initial state X0 ∼ Beta(α ′ , β′ ), i.e., it is distributed according to a Beta distribution with parameters α ′ = 1.2, β′ = 2. The realised state x0 and action a are used to determine the parameters of the Beta distribution used to sample the next … view at source ↗
Figure 3
Figure 3. The expected reward for each action at+1 based on the current state xt and action at under P(xt , at). The x-axis is the state x and the y-axis is the expected reward for the action taken [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: RDQN agents trained and evaluated with varied transaction cost on the S&P 500 index from 3 Jan 1995 to 28 Dec 2023. Both agents use ε = 0.0025, δ = 0.0001 with the same seed. Within each figure, the top graph shows the wealth trajectory of the RDQN agent (blue line) an…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Robustness in Sequential Decision Making under Evolving Uncertainty: Evidence from High-Frequency Market Making

    q-fin.TR 2026-07 conditional novelty 6.0 of 10

    In high-frequency market making, action robustness (Sinkhorn regularization) dominates uncertainty tolerance in reshaping sequential quoting and inventory, while excessive robustness can cut execution opportunities in...

Reference graph

Works this paper leans on

66 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [55]

    Sinkhorn distributionally robust optimization.arXiv preprint arXiv:2109.11926, 2021

    Jie Wang, Rui Gao, and Yao Xie. Sinkhorn distributionally robust optimization.arXiv preprint arXiv:2109.11926, 2021

    arXiv 2021

  2. [1]

    Investigating the parameters of the beta distribution

    H Ali, CN Akanihu, and J Felix. Investigating the parameters of the beta distribution. World Journal of Advanced Research and Reviews, 19(1):815–830, 2023

    work page 2023

  3. [2]

    Infinite dimensional analysis: a hitchhiker’s guide

    Charalambos D Aliprantis and Kim C Border. Infinite dimensional analysis: a hitchhiker’s guide . Springer Science & Business Media, 2006

    work page 2006

  4. [3]

    Computational aspects of robust optimized certainty equiv- alents and option pricing

    Daniel Bartl, Samuel Drapeau, and Ludovic Tangpi. Computational aspects of robust optimized certainty equiv- alents and option pricing. Mathematical Finance, 30(1):287–309, 2020

    work page 2020

  5. [4]

    On the theory of dynamic programming

    Richard Bellman. On the theory of dynamic programming. Proceedings of the national Academy of Sciences , 38(8):716–719, 1952

    work page 1952

  6. [5]

    Dota 2 with large scale deep reinforcement learning

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, and Chris Hesse. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019

    arXiv 1912

  7. [6]

    Convex optimization

    Stephen Boyd. Convex optimization. Cambridge UP , 2004

    work page 2004

  8. [7]

    Distributionally robust Markov decision processes and their connection to risk measures

    Nicole B¨ auerle and Alexander Glauner. Distributionally robust Markov decision processes and their connection to risk measures. Mathematics of Operations Research, 47(3):1757–1780, 2022

    work page 2022

Show all 66 references
  1. [8]

    R. Cont. Empirical properties of asset returns: stylized facts and statistical issues. Quantitative Finance, 1(2):223– 236, 2001. DISTRIBUTIONALLY ROBUST DEEP Q-LEARNING 29

    2001

  2. [9]

    Sinkhorn distances: Lightspeed computation of optimal transport, 2013

    Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport, 2013

    2013

  3. [10]

    Robust Q-learning for finite ambiguity sets

    C´ ecile Decker and Julian Sester. Robust Q-learning for finite ambiguity sets. arXiv preprint arXiv:2407.04259 , 2024

    2024

  4. [11]

    Twice regularized Markov decision processes: The equivalence between robustness and regularization

    Esther Derman, Yevgeniy Men, Matthieu Geist, and Shie Mannor. Twice regularized Markov decision processes: The equivalence between robustness and regularization. arXiv preprint arXiv:2303.06654 , 2023

    2023 arXiv

  5. [12]

    Maximum entropy RL (provably) solves some robust RL problems

    Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. arXiv preprint arXiv:2103.06257 , 2021

    2021 arXiv

  6. [13]

    A theoretical analysis of deep Q-learning

    Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep Q-learning. In Learning for dynamics and control , pages 486–489. PMLR, 2020

    2020

  7. [14]

    In- terpolating between optimal transport and mmd using Sinkhorn divergences

    Jean Feydy, Thibault S´ ejourn´ e, Fran¸ cois-Xavier Vialard, Shun-ichi Amari, Alain Trouv´ e, and Gabriel Peyr´ e. In- terpolating between optimal transport and mmd using Sinkhorn divergences. InThe 22nd international conference on artificial intelligence and statistics , pag...

    2019

  8. [15]

    Sample complexity of Sinkhorn divergences

    Aude Genevay, L´ enaic Chizat, Francis Bach, Marco Cuturi, and Gabriel Peyr´ e. Sample complexity of Sinkhorn divergences. In The 22nd international conference on artificial intelligence and statistics, pages 1574–1583. PMLR, 2019

    2019

  9. [16]

    Stability of entropic optimal transport and Schr¨ odinger bridges

    Promit Ghosal, Marcel Nutz, and Espen Bernton. Stability of entropic optimal transport and Schr¨ odinger bridges. Journal of Functional Analysis , 283(9):109622, 2022

    2022

  10. [17]

    Robust Markov decision processes: Beyond rectangularity.Mathematics of Operations Research, 48(1):203–226, 2023

    Vineet Goyal and Julien Grand-Cl´ ement. Robust Markov decision processes: Beyond rectangularity.Mathematics of Operations Research, 48(1):203–226, 2023

    2023

  11. [18]

    Double Q-learning

    Hado Hasselt. Double Q-learning. Advances in neural information processing systems , 23, 2010

    2010

  12. [19]

    Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural networks, 3(5):551–560, 1990

    1990

  13. [20]

    Learning to utilize shaping rewards: A new approach of reward shaping

    Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931–15941, 2020

    2020

  14. [21]

    Robust dynamic programming

    Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280, 2005

    2005

  15. [22]

    Probability essentials

    Jean Jacod and Philip Protter. Probability essentials. Springer Science & Business Media, 2012

    2012

  16. [23]

    Universal approximation with deep narrow networks

    Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. In Conference on learning theory, pages 2306–2327. PMLR, 2020

    2020

  17. [24]

    Probability theory: a comprehensive course

    Achim Klenke. Probability theory: a comprehensive course . Springer Science & Business Media, 2013

    2013

  18. [25]

    An efficient solution to s-rectangular robust Markov decision processes

    Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor. An efficient solution to s-rectangular robust Markov decision processes. arXiv preprint arXiv:2301.13642 , 2023

    2023 arXiv

  19. [26]

    Playing fps games with deep reinforcement learning

    Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In Pro- ceedings of the AAAI conference on artificial intelligence , volume 31, 2017

    2017

  20. [27]

    Policy gradient algorithms for robust MDPs with non-rectangular uncertainty sets

    Mengmeng Li, Daniel Kuhn, and Tobias Sutter. Policy gradient algorithms for robust MDPs with non-rectangular uncertainty sets. arXiv preprint arXiv:2305.19004 , 2023

    2023

  21. [28]

    On the efficiency of entropic regularized algorithms for optimal transport

    Tianyi Lin, Nhat Ho, and Michael I Jordan. On the efficiency of entropic regularized algorithms for optimal transport. Journal of Machine Learning Research , 23(137):1–42, 2022

    2022

  22. [29]

    Distri- butionally robust Q-learning

    Zijian Liu, Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, and Zhengyuan Zhou. Distri- butionally robust Q-learning. In International Conference on Machine Learning , pages 13623–13643. PMLR, 2022

    2022

  23. [30]

    Generative model for financial time series trained with MMD using a signature kernel

    Chung I Lu and Julian Sester. Generative model for financial time series trained with MMD using a signature kernel. arXiv preprint arXiv:2407.19848 , 2024

    2024

  24. [31]

    Robust MDPs with k-rectangular uncertainty.Mathematics of Operations Research, 41(4):1484–1509, 2016

    Shie Mannor, Ofir Mebel, and Huan Xu. Robust MDPs with k-rectangular uncertainty.Mathematics of Operations Research, 41(4):1484–1509, 2016

    2016

  25. [32]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

    2015

  26. [33]

    Robust SGLD algorithm for solving non-convex distri- butionally robust optimisation problems

    Ariel Neufeld, Matthew Ng Cheng En, and Ying Zhang. Robust SGLD algorithm for solving non-convex distri- butionally robust optimisation problems. arXiv preprint arXiv:2403.09532 , 2024

    2024 arXiv

  27. [34]

    Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

    Ariel Neufeld and Philipp Schmocker. Universal approximation results for neural networks with non-polynomial activation function over non-compact domains. arXiv preprint arXiv:2410.14759 , 2024

    2024 arXiv

  28. [35]

    Neural networks can detect model-free static arbitrage strategies

    Ariel Neufeld and Julian Sester. Neural networks can detect model-free static arbitrage strategies. Applied Math- ematics & Optimization , 90(2):41, 2024

    2024

  29. [36]

    Robust Q-learning algorithm for markov decision processes under Wasserstein uncertainty

    Ariel Neufeld and Julian Sester. Robust Q-learning algorithm for markov decision processes under Wasserstein uncertainty. Automatica, 168:111825, 2024

    2024

  30. [37]

    Non-concave stochastic optimal control in finite discrete time under model uncertainty

    Ariel Neufeld and Julian Sester. Non-concave stochastic optimal control in finite discrete time under model uncertainty. arXiv preprint arXiv:2404.05230 , 2025

    2025 arXiv

  31. [38]

    Markov decision processes under model uncertainty.Mathematical Finance, 33(3):618–665, 2023

    Ariel Neufeld, Julian Sester, and Mario ˇSiki´ c. Markov decision processes under model uncertainty.Mathematical Finance, 33(3):618–665, 2023

    2023

  32. [39]

    Robust control of Markov decision processes with uncertain transition matrices

    Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005

    2005

  33. [40]

    Introduction to entropic optimal transport

    Marcel Nutz. Introduction to entropic optimal transport. Lecture notes, Columbia University , 2021

    2021

  34. [41]

    Robust reinforcement learning using offline data

    Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. Advances in neural information processing systems , 35:32211–32224, 2022. 30 DISTRIBUTIONALLY ROBUST DEEP Q-LEARNING

    2022

  35. [42]

    Approximation theory of the MLP model in neural networks

    Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta numerica, 8:143–195, 1999

    1999

  36. [43]

    Distributionally robust optimization: A review

    Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659, 2019

    1908 arXiv

  37. [44]

    Distributionally robust model-based reinforcement learning with large state spaces

    Shyam Sundhar Ramesh, Pier Giuseppe Sessa, Yifan Hu, Andreas Krause, and Ilija Bogunovic. Distributionally robust model-based reinforcement learning with large state spaces. In International Conference on Artificial Intelligence and Statistics , pages 100–108. PMLR, 2024

    2024

  38. [45]

    Principles of mathematical analysis , volume 3

    Walter Rudin. Principles of mathematical analysis , volume 3. McGraw-hill New York, 1964

    1964

  39. [46]

    Structural estimation of Markov decision processes

    John Rust. Structural estimation of Markov decision processes. Handbook of econometrics, 4:3081–3143, 1994

    1994

  40. [47]

    Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results

    Franco Scarselli and Ah Chung Tsoi. Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results. Neural networks, 11(1):15–37, 1998

    1998

  41. [48]

    A relationship between arbitrary positive matrices and doubly stochastic matrices

    Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics , 35(2):876–879, 1964

    1964

  42. [49]

    Distributionally robust reinforcement learning

    Elena Smirnova, Elvis Dohmatob, and J´ er´ emie Mary. Distributionally robust reinforcement learning. arXiv preprint arXiv:1902.08708, 2019

    1902 arXiv

  43. [50]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  44. [51]

    Sinkhorn divergences for unbalanced optimal transport

    Thibault S´ ejourn´ e, Jean Feydy, Fran¸ cois-Xavier Vialard, Alain Trouv´ e, and Gabriel Peyr´ e. Sinkhorn divergences for unbalanced optimal transport. arXiv preprint arXiv:1910.12958 , 2019

    1910 arXiv

  45. [52]

    Deep reinforcement learning: From Q-learning to deep Q-learning

    Fuxiao Tan, Pengfei Yan, and Xinping Guan. Deep reinforcement learning: From Q-learning to deep Q-learning. In Neural Information Processing: 24th International Conference, ICONIP 2017, Guangzhou, China, November 14–18, 2017, Proceedings, Part IV 24 , pages 475–483. Springer, 2017

    2017

  46. [53]

    Deep reinforcement learning with double Q-learning

    Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016

    2016

  47. [54]

    Springer, 2009

    C´ edric Villani.Optimal transport: old and new , volume 338. Springer, 2009

    2009

  48. [56]

    Policy gradient in robust MDPs with global convergence guarantee, 2023

    Qiuhao Wang, Chin Pang Ho, and Marek Petrik. Policy gradient in robust MDPs with global convergence guarantee, 2023

    2023

  49. [57]

    A finite sample complexity bound for distribu- tionally robust Q-learning

    Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. A finite sample complexity bound for distribu- tionally robust Q-learning. In International Conference on Artificial Intelligence and Statistics , pages 3370–3398. PMLR, 2023

    2023

  50. [58]

    Sample complexity of variance-reduced distribu- tionally robust Q-learning

    Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. Sample complexity of variance-reduced distribu- tionally robust Q-learning. Journal of Machine Learning Research , 25(341):1–77, 2024

    2024

  51. [59]

    Online robust reinforcement learning with model uncertainty

    Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems, 34:7193–7206, 2021

    2021

  52. [60]

    Policy gradient method for robust reinforcement learning, 2022

    Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning, 2022

    2022

  53. [61]

    Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279–292, 1992

    1992

  54. [62]

    Robust Markov decision processes

    Wolfram Wiesemann, Daniel Kuhn, and Ber¸ c Rustem. Robust Markov decision processes. Mathematics of Oper- ations Research, 38(1):153–183, 2013

    2013

  55. [63]

    Distributionally robust Markov decision processes

    Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. Advances in Neural Information Processing Systems, 23, 2010

    2010

  56. [64]

    A convex optimization approach to distributionally robust Markov decision processes with Wasser- stein distance

    Insoon Yang. A convex optimization approach to distributionally robust Markov decision processes with Wasser- stein distance. IEEE control systems letters , 1(1):164–169, 2017

    2017

  57. [65]

    Wasserstein distributionally robust stochastic control: A data-driven approach

    Insoon Yang. Wasserstein distributionally robust stochastic control: A data-driven approach. IEEE Transactions on Automatic Control , 66(8):3863–3870, 2020

    2020

  58. [66]

    On linear optimization over Wasserstein balls

    Man-Chung Yue, Daniel Kuhn, and Wolfram Wiesemann. On linear optimization over Wasserstein balls. Mathe- matical Programming, 195(1):1107–1122, 2022. DISTRIBUTIONALLY ROBUST DEEP Q-LEARNING 31 Appendix A. Performance of agents in the reference distribution in the toy example T...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.