Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Entropy regularization in continuous-time RL yields explicit worst-case robustness guarantees whose certified sets grow with the temperature.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 04:22 UTC pith:G7QVK74S

load-bearing objection First clean continuous-time robustness certificates for entropy regularization, without the discrete-time state-entropy term and without step-size degeneracy. the 2 major comments →

arxiv 2607.03168 v1 pith:G7QVK74S submitted 2026-07-03 math.OC cs.LG

Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning

Jialun Cao , Fernando Acero , David \v{S}i\v{s}ka , Yufei Zhang This is my paper
classification math.OC cs.LG MSC 90C4093E2060J27
keywords entropy regularizationpolicy robustnesscontinuous-time Markov decision processesrobust reinforcement learningrelative entropy rateoccupancy measuresqueueing controlmarket making
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the common practice of adding entropy to continuous-time reinforcement learning objectives actually buys robustness, and against what kinds of model error. The authors prove that maximizing an entropy-regularized objective on a continuous-time Markov decision process lower-bounds the worst-case performance of the same policy under joint reward and transition-rate perturbations. They give two explicit descriptions of the admissible uncertainty sets—one global, through discounted occupancy measures, and one local, through relative-entropy rates of jump intensities—and prove both sets expand as the regularization strength increases. Unlike earlier discrete-time arguments, the continuous-time certificates avoid an intractable state-distribution entropy term and stay non-degenerate when the agent can act more frequently. Experiments on queueing control and market making show that moderate entropy yields policies that hold up better under intensity misspecification than both greedy and ε-greedy baselines.

Core claim

Maximizing the entropy-regularized continuous-time objective is equivalent to a robust control problem under pure reward uncertainty and provides a certified lower bound under joint reward-and-dynamics uncertainty; the corresponding robust sets expand monotonically with the temperature, so stronger entropy enlarges the class of perturbations against which the policy is protected.

What carries the argument

Two analytically characterized robust sets: an occupancy-based set that measures global distortion of discounted state-action measures, and a local set built from the relative-entropy rate of transition intensities together with a log-reward cost; both are defined by a soft-max (log-sum-exp) constraint that widens with temperature.

Load-bearing premise

The local certificate requires that every perturbed model keeps exactly the same possible jumps as the baseline model; if new transitions can appear or existing ones can vanish, that certificate no longer applies.

What would settle it

Train entropy-regularized and greedy policies on a continuous-time MDP whose jump support can change under realistic misspecification, then measure whether worst-case performance still improves with temperature on the enlarged support; if it does not, the local robustness claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper establishes the first robustness guarantees for entropy-regularized continuous-time MDPs with controlled CTMC dynamics. Maximizing the entropy-regularized objective J_τ(π) is shown to yield a lower bound on worst-case performance under joint reward and transition perturbations (Theorems 3.2 and 3.6), with exact equivalence under reward-only uncertainty (Theorem 3.1). The induced robust sets are characterized both via discounted occupancy measures and via local relative-entropy rates of transition intensities, and are proved to expand monotonically with the regularization strength τ (Propositions 3.4 and 3.8). The continuous-time certificates avoid the intractable state-distribution entropy term that appears in discrete-time analyses and remain non-degenerate as action frequency increases (Example 3.7). Experiments on criss-cross queueing control and market making support the qualitative claims.

Significance. If the results hold, the paper supplies a clean theoretical foundation for a practice that is already widespread in continuous-time RL (entropy regularization for robustness) without requiring an explicit adversary or minimax solver. The continuous-time certificates are genuinely better adapted to the setting than discrete-time analogues: they remove the state-entropy term, stay non-degenerate under refinement of the action grid, and admit an event-driven implementation. The proofs (Appendices B.2–B.7) are complete and use standard tools (Lagrange duality, Jensen, Girsanov for CTMCs, Donsker–Varadhan). The market-making and queueing experiments, together with the certificate-tightness study in Appendix E, give concrete evidence that moderate τ improves worst-case performance over greedy and ε-greedy baselines. The main limitations (common jump support, finite spaces, certificate conservatism) are already acknowledged by the authors and do not undermine the formal lower-bound statements.

major comments (2)
  1. Assumption 3.5 (identical jump support of baseline and perturbed rates) is load-bearing for Theorem 3.6 and the local certificate C^π_τ,ε. The Girsanov/semi-martingale argument in Appendix B.6 and the definition of the local relative-entropy rate ℓ require that no new transitions appear and none vanish. The paper should state more prominently (in the introduction or after Theorem 3.6) that the local certificate does not cover support-changing misspecification, while the global occupancy certificate of Theorem 3.2 remains valid without this assumption. A short remark on how one might extend the local construction (e.g., via absolute continuity of path measures) would strengthen the scope discussion.
  2. The experimental gains, while directionally consistent with the theory, are modest and temperature-sensitive (Tables 3–4, Figures 2–3 and 6–9). Worst-case improvements of order 0.3–2% over π_std are statistically significant for carefully chosen small τ, but the inverted-U pattern shows that larger τ quickly degrades both nominal and worst-case performance. The manuscript should more clearly separate the formal lower-bound claims (which hold for any τ) from the practical claim that moderate entropy regularization improves robustness; the latter is supported only for a narrow temperature range and should be presented as such.
minor comments (5)
  1. Notation for the robust sets (bC, eC, C) is dense; a short table or paragraph summarizing the three constructions and the assumptions each requires would help the reader.
  2. In the market-making illustration (Section 3.3 and Appendix C), the reward is shifted by a large constant C to enforce positivity. The effect of this shift on the log-transformed objective and on the numerical size of the robust sets should be briefly discussed.
  3. Example 3.7 is useful; making the continuous-time limit of the discrete-time constraint fully rigorous (or citing the appropriate large-deviations reference) would remove any residual ambiguity.
  4. Appendix F on event-driven versus fixed-grid discretization is valuable but somewhat long relative to the main contribution; a shorter summary in the main text with the full comparison left in the appendix would improve balance.
  5. A few minor typos appear (e.g., “ε-greedy” spacing, occasional missing articles). A careful proof-reading pass is recommended.

Circularity Check

0 steps flagged

No significant circularity: dual robust-set constructions and Girsanov/KL lower bounds are self-contained mathematical characterizations, not fits or self-citation chains.

full rationale

The load-bearing claims (Theorems 3.1, 3.2, 3.6; Propositions 3.4, 3.8) are proved from standard tools—Lagrange duality for reward-only KL balls, Jensen on occupancy measures, Girsanov for CTMC likelihood ratios, Donsker–Varadhan/KL duality, and log-sum-exp monotonicity—without reducing the target to a fitted parameter or to an unverified self-citation. Defining robust sets so that the entropy-regularized objective lower-bounds worst-case performance is the usual dual characterization of entropy regularization, not circular prediction: the sets are analytically characterized (occupancy ratios vs local relative-entropy rates), shown to expand with τ, and contrasted with discrete-time constructions that degenerate (Example 3.7). Experiments choose τ and perturbation grids independently and compare against greedy/ε-greedy baselines; they do not reverse-engineer parameters to force the theorems. Self-citations concern continuous-time RL algorithms and related work, not uniqueness or the robustness dualities themselves. Assumption 3.5 and R>0 restrict scope but do not create circularity. Score 0 with empty steps is the honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 2 invented entities

The theory rests on standard CTMDP structure, positivity of rewards, common jump support under perturbation, and classical variational/change-of-measure tools. Free parameters are the regularization temperature, adversary budget, and experimental environment/training knobs; none of these are fitted to force the theorems. No new physical entities are postulated—the robust sets and local relative-entropy rate are mathematical constructions built from known path-measure KL rates.

free parameters (4)
  • entropy temperature τ
    Regularization strength chosen by the user; theory shows robust sets expand in τ, but optimal τ is environment-dependent and selected by grid search in experiments.
  • adversary budget ε
    Defines the size of the robust sets; free design parameter of the certificate, fixed to 0.01 in the market-making illustration.
  • reference policy μ
    Enters the KL penalty and the log-sum-exp robust-set criteria; taken uniform over actions in experiments.
  • experimental environment scales and network hyperparameters
    Arrival/service rates, fill parameters, MLP sizes, learning rates, and seed counts are set by hand for the queueing and market-making studies; they do not enter the theorems.
axioms (6)
  • domain assumption Finite state and action spaces with stationary randomized Markov policies and discounted infinite-horizon CTMDP structure.
    Stated in §2; all occupancy measures and generators are finite-dimensional.
  • domain assumption Instantaneous rewards strictly positive for all models in Θ.
    Required for log-transformed objectives in Theorems 3.2 and 3.6; enforced by reward shifting in the market-making example.
  • domain assumption Assumption 3.5: baseline and perturbed transition rates share the same support.
    Needed for the local relative-entropy rate and Girsanov argument in Theorem 3.6.
  • standard math Girsanov theorem / likelihood-ratio semi-martingale decomposition for controlled CTMCs.
    Invoked in the proof of Theorem 3.6 (Appendix B.6), citing Øksendal–Sulem.
  • standard math Donsker–Varadhan variational formula for KL divergence.
    Lemma B.3; used to dualize the local perturbation cost into the entropy penalty.
  • standard math Joint convexity of the relative-entropy rate function f(x,y)=x log(x/y)−x+y.
    Lemma B.2; bounds policy-aggregated relative entropy by action-wise rates.
invented entities (2)
  • Local relative entropy rate ℓ(λ^a_θ, λ^a_˜θ) and joint cost c_θ,˜θ independent evidence
    purpose: Provide an interpretable continuous-time model-level perturbation cost for the robust set C^π_τ,ε.
    Continuous-time analogue of one-step KL between transition kernels; related quantities appear in prior CTMC variational inference (Opper & Sanguinetti), so not a free-standing new physical entity.
  • Policy-dependent robust sets bC, eC, C parameterized by τ and ε independent evidence
    purpose: Characterize the uncertainty sets for which the entropy-regularized objective lower-bounds worst-case performance.
    Mathematical constructions defined from occupancy measures or local costs; falsifiable by checking membership and empirical performance, as done in Appendix E.

pith-pipeline@v1.1.0-grok45 · 46563 in / 3547 out tokens · 36685 ms · 2026-07-12T04:22:33.022550+00:00 · methodology

0 comments
read the original abstract

Entropy regularization is widely used in continuous-time reinforcement learning (RL) to reduce sensitivity to environmental perturbations, yet its robustness benefits lack a rigorous theoretical foundation. This paper establishes the first robustness guarantees for entropy-regularized continuous-time Markov decision processes. We show that maximizing an entropy-regularized objective yields a lower bound on a worst-case robust RL problem with joint reward and transition perturbations. We analytically characterize the induced robust sets and prove that they expand monotonically with the regularization strength, justifying the empirical observation that stronger entropy improves robustness. In contrast to prior discrete-time analyses, our results remove the intractable state-distribution entropy term and provide guarantees invariant to action frequency. Experiments on queueing network control and market making confirm our theory, showing that entropy-regularized policies outperform greedy and $\epsilon$-greedy baselines under dynamics perturbations.

Figures

Figures reproduced from arXiv: 2607.03168 by David \v{S}i\v{s}ka, Fernando Acero, Jialun Cao, Yufei Zhang.

Figure 1
Figure 1. Figure 1: Robust regions for the market making example with r = 1 and ε = 0.01. Each panel shows simultaneous perturbations of αΛ and βΛ for a fixed τ . The green region is the local intensity-based robust set C π τ,ε(θ), and the purple region is the global occupancy-based robust set Ceπ τ,ε(θ); the black cross marks the baseline parameter [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Robustness to arrival and service intensity perturbations in the criss￾cross queueing network. Left: gain of the entropy-regularized policy over the standard deterministic policy. Right: gain over a noisy standard policy (ϵ-greedy policy with ϵ = 0.1). Black lines denote sign changes in ∆J. A natural question is whether the improved robustness is simply due to added randomness. To test this, we compare aga… view at source ↗
Figure 3
Figure 3. Figure 3: Effect of the regularization temperature in the criss-cross queueing network. Left: nominal and worst-case performance over the perturbation grid. Right: corresponding gains relative to πstd. indicates that, under adverse intensity perturbations, πstd deteriorates more severely than the entropy-regularized policy. Analogous results for market making experiments are reported in Appendix D, where formal stat… view at source ↗
Figure 4
Figure 4. Figure 4: Robust regions for the market making example under the same setting as [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Robust regions for the market making example under the heuristic policy as [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Market-making: reward gain heatmaps across the volatility scale × arrival rate scale perturbation grid. Positive (blue) regions indicate where the entropy-regularised policy outperforms the baseline. The entropy-regularised policy gains most in the high-volatility region (top of grid), where the optimal spread changes qualitatively from the nominal setting. 10 4 10 3 10 2 10 1 10 0 Temperature 2 1 0 1 J( )… view at source ↗
Figure 7
Figure 7. Figure 7: Market-making: worst-case and nominal gain vs. πstd as a function of τ (10 seeds, mean ± SE). The inverted-U shape confirms that moderate entropy regularisation improves worst-case robustness, while excessive regularisation de￾grades performance. D.4. Queueing Scheduling Results [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Queueing: reward gain heatmaps across the λ1 scale × λ3 scale perturbation grid. The entropy-regularised policy gains most in the high-arrival corner (s1 = s3 = 3.0), where the load balance between classes 1 and 3 for server 1 shifts from the nominal setting [PITH_FULL_IMAGE:figures/full_fig_p030_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Queueing: worst-case and nominal gain vs. πstd as a function of τ (5 seeds, mean ± SE). Small τ values ( 10−4 ) improve worst-case performance, while τ = 10−1 and greater slightly degrades it. D.5. Discussion. Across both domains, entropy regularisation with appropriately chosen τ significantly improves worst-case performance under model perturbation without degrading nominal performance. The optimal τ is … view at source ↗
Figure 10
Figure 10. Figure 10: Certificate value G( ˜θ) vs empirical performance ratio for three entropy temperatures and three policy types (analytical softmax, Soft-PG, Soft￾AC). All three cluster tightly, confirming that learned policies match the theoretical optimum. Certified points (G ≤ 0, black squares) all lie above the 0.9 threshold (dashed grey) [PITH_FULL_IMAGE:figures/full_fig_p032_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Certified region size (number of grid cells with G ≤ 0) as a function of τ . Both perturbation planes show monotone growth, confirming the theoretical prediction. Green diamonds show the empirical robust region size for reference. E.3. Transition vs Reward Decomposition. The certificate function decomposes as Xδ(q) = ℓδ(q) − r · log(Rθ˜(q, δ)/Rθ(q, δ)), where ℓa is the local relative entropy (KL divergenc… view at source ↗
Figure 12
Figure 12. Figure 12: Decomposition of the certificate bound into transition KL (ℓa) and reward perturbation (|r · log(Rθ˜/Rθ)|) contributions, shown on a log scale. The transition KL dominates by 1.5–3 orders of magnitude at all perturbation points. The transition KL term dominates overwhelmingly: it exceeds the reward term by a factor of 150–800× across all grid points. This has two implications. First, the policy is inheren… view at source ↗
Figure 13
Figure 13. Figure 13: One-dimensional cross-sections at τ = 1.0. Left: sweep βΛ with αΛ = −1.0 fixed. Right: sweep αΛ with βΛ = 10.0 fixed. The certificate boundary (G = 0, blue) is crossed well before the empirical 90% performance boundary (red), quantifying the conservatism along each perturbation direction. Along the βΛ-axis (left panel), the certificate allows perturbations of roughly ±0.5 around the nominal βΛ = 10, while… view at source ↗
Figure 14
Figure 14. Figure 14: confirms this prediction. The reward-only certificate Grw certifies 401–409 out of 441 grid cells across all three τ values, compared to 18–24 for the joint certificate G. 0 1 2 3 4 5 Greward( ) 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 J( , )/J( , ) = 0.05 Not certified Certified (401/441) 0.5 0.0 0.5 1.0 1.5 2.0 2.5 Greward( ) = 1.0 Not certified Certified (409/441) 0.4 0.2 0.0 0.2 0.4 0.6 Greward( ) = 50… view at source ↗
Figure 15
Figure 15. Figure 15: Certified region size vs τ : reward-only certificate (green) vs joint G (blue). The large gap confirms that certificate conservatism is entirely driven by the transition KL term. This analysis directly identifies the source of conservatism: the local relative entropy of logistic fill intensities grows quadratically with parameter deviation, producing large KL values even for moderate changes in (αΛ, βΛ). … view at source ↗
Figure 16
Figure 16. Figure 16: Global certificate Φglob vs empirical performance ratio. The vertical line marks the certificate threshold r log(1/r) ≈ 0.37. All certified points (Φglob ≤ r log(1/r), red squares) achieve J/Jnom > 0.94. 10 2 10 1 10 0 10 1 Temperature 0 100 200 300 400 Number of robust cells (out of 441) Certificate Comparison: G vs loc vs glob G local max (Thm 3.6) loc occ-weighted glob (Prop 3.4) Empirical robust [PIT… view at source ↗
Figure 17
Figure 17. Figure 17: Certified region size vs τ for three certificate variants: G (state￾max, blue), Gw (occupancy-weighted, orange), and Φglob (global, green). Green diamonds show the empirical robust region. All three certificates show monotone growth, with the occupancy-weighted variants certifying slightly more cells. The modest improvement from occupancy weighting reflects the structure of the inventory￾stabilising polic… view at source ↗
Figure 18
Figure 18. Figure 18: Certificate coverage heatmap at τ = 1.0. Green cells are certified by both G and Φglob; blue cells are certified only by Φglob. The black star marks the nominal operating point [PITH_FULL_IMAGE:figures/full_fig_p038_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: further confirms that Gw ≤ G pointwise across all 441 grid cells, with most points clustering near the diagonal—indicating that the worst-case state dominates the average. That the improvement from occupancy-weighting is small (only 1–2 additional cells) confirms that the conservatism of G is not driven by the max-over-states aggregation. Rather, the bottleneck lies in the per-state bound F(q), which aggr… view at source ↗
Figure 20
Figure 20. Figure 20: Certified region sizes for G (state-max), Gw (occupancy-weighted), and the empirical robust region. Occupancy-weighting provides a small improve￾ment (1–2 additional cells), but the ∼10× gap to the empirical region remains. E.8. Policy-Weighted KL Decomposition. The certificate computes an effective per-state KL contribution via the soft-max aggregation ℓcert(q) = τ [PITH_FULL_IMAGE:figures/full_fig_p039… view at source ↗
Figure 21
Figure 21. Figure 21: Comparison of three KL aggregation methods at q = 0 (τ = 1.0) across 8 representative perturbation points. The certificate’s logsumexp aggregation (blue) exceeds the policy-weighted KL (orange) and the uniform average (green), but the amplification is modest (1–2.5×) [PITH_FULL_IMAGE:figures/full_fig_p039_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: KL amplification ratio ℓcert/ℓπ across the αΛ-βΛ grid at q = 0, τ = 1.0. The ratio ranges from 1.0 (at small perturbations) to 2.5 (at grid extremes). The black star marks the nominal point. The amplification factor is modest (1.0–2.5×), indicating that at τ = 1.0 the softmax policy distributes weight broadly enough that the logsumexp aggregation closely approximates the policy-weighted KL. At smaller τ (… view at source ↗
Figure 23
Figure 23. Figure 23: Certified region size vs action space cardinality at τ = 1.0. Both the recomputed Cshift (blue) and fixed Cshift (red) variants converge by |A| ≈ 225, confirming that the continuous-time certificate remains non-degenerate [PITH_FULL_IMAGE:figures/full_fig_p040_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Action space convergence for three entropy temperatures. All three converge rapidly and plateau at their respective certified region sizes (18–24 cells). The certificate is non-degenerate across 400× variation in |A|. The reward shift Cshift is constant at 38.66 across all action grid sizes because the minimum reward always occurs at the grid boundary δ = δmin = 0.12 with extreme inventory q = ±10. 20 40 … view at source ↗
Figure 25
Figure 25. Figure 25: Reward shift Cshift vs action grid resolution Nδ. The shift is constant because the minimum reward is determined by the grid boundary, not the interior resolution. The convergence is rapid: the certified region stabilises by Nδ = 15 (|A| = 225) for all τ values. This confirms the theoretical advantage of the continuous-time certificate formulation over its discrete-time counterpart: the log |A| normalisat… view at source ↗
Figure 26
Figure 26. Figure 26: Fixed-grid A2C discretization sweep on the market-making environ￾ment (5 seeds, 2,000 epochs). The dashed red line shows the arrival-driven A2C baseline. Finer temporal resolution (∆t → 0) does not improve RL performance; instead, the credit assignment problem dominates, and the agent fails to learn effective policies despite much higher computational cost [PITH_FULL_IMAGE:figures/full_fig_p043_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Performance vs training cost for fixed-grid A2C at various ∆t (blue circles) and arrival-driven A2C (red square). The arrival-driven agent achieves strong performance without requiring grid resolution tuning. No fixed￾grid configuration simultaneously dominates the arrival-driven agent in both reward and training time by a large margin. F.4. Queueing Scheduling Experiment. We also compare discretizations … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Feedback Cycles in Exploratory Equilibria

    math.OC 2026-07 conditional novelty 7.0

    In exploratory equilibria, low-temperature policy susceptibility grows polynomially on causal chains but exponentially on positive feedback cycles, with Lambert-W critical temperature τ* = βT/W(βT√n).

Reference graph

Works this paper leans on

68 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    The reality gap in robotics: Challenges, solutions, and best practices.Annual Review of Control, Robotics, and Autonomous Systems, 9:403–432, 2026

    Elie Aljalbout, Jiaxu Xing, Angel Romero, Iretiayo Akinola, Caelan Reed Garrett, Eric Heiden, Abhishek Gupta, Tucker Hermans, Yashraj Narang, Dieter Fox, Davide Scaramuzza, and Fabio Ramos. The reality gap in robotics: Challenges, solutions, and best practices.Annual Review of Control, Robotics, and Autonomous Systems, 9:403–432, 2026

    2026

  2. [2]

    State entropy regularization for robust reinforcement learning

    Yonatan Ashlag, Uri Koren, Mirco Mutti, Esther Derman, Pierre-Luc Bacon, and Shie Mannor. State entropy regularization for robust reinforcement learning. InAdvances in Neural Information Processing Systems, volume 38, 2025

    2025

  3. [3]

    Algorithmic market making in dealer markets with hedging and market impact.Mathematical Finance, 33(1):41–79, 2023

    Alexander Barzykin, Philippe Bergault, and Olivier Guéant. Algorithmic market making in dealer markets with hedging and market impact.Mathematical Finance, 33(1):41–79, 2023

    2023

  4. [4]

    Continuous-time q-learning in jump-diffusion models under Tsallis entropy, 2024

    Lijun Bo, Yijie Huang, Xiang Yu, and Tingting Zhang. Continuous-time q-learning in jump-diffusion models under Tsallis entropy, 2024

    2024

  5. [5]

    Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, and Pedro A. Ortega. Your policy regularizer is secretly an adversary.Transactions on Machine Learning Research, 2022

    2022

  6. [6]

    Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms

    Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang, and Tuo Zhao. Robust multi-agent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms. InAdvances in Neural Information Processing Systems, volume 36, pages 68121–68133, 2023

    2023

  7. [7]

    Cambridge University Press, 2015

    Álvaro Cartea, Sebastian Jaimungal, and José Penalva.Algorithmic and High-Frequency Trading. Cambridge University Press, 2015

    2015

  8. [8]

    Robust reinforcement learning with general utility

    Ziyi Chen, Yan Wen, Zhengmian Hu, and Heng Huang. Robust reinforcement learning with general utility. In Advances in Neural Information Processing Systems, volume 37, pages 11290–11344, 2024

    2024

  9. [9]

    Deterministic policy gradient for reinforcement learning with continuous time and state, 2026

    Ziheng Cheng, Xin Guo, and Yufei Zhang. Deterministic policy gradient for reinforcement learning with continuous time and state, 2026

    2026

  10. [10]

    Dai and Mark Gluzman

    Jim G. Dai and Mark Gluzman. Queueing network controls via deep reinforcement learning.Stochastic Systems, 12(1):30–67, 2022

    2022

  11. [11]

    Twice regularized MDPs and the equivalence between robustness and regularization.Advances in Neural Information Processing Systems, 34:22274–22287, 2021

    Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDPs and the equivalence between robustness and regularization.Advances in Neural Information Processing Systems, 34:22274–22287, 2021

    2021

  12. [12]

    Robustness and regularization in rein- forcement learning

    Esther Derman, Yevgeniy Men, Matthieu Geist, and Shie Mannor. Robustness and regularization in rein- forcement learning. InNeurIPS 2023 Workshop on Generalization in Planning, 2023

    2023

  13. [13]

    Entropy regularization in mean-field games of optimal stopping.arXiv preprint arXiv:2509.18821, 2025

    Jodi Dianetti, Roxana Dumitrescu, Giorgio Ferrari, and Renyuan Xu. Entropy regularization in mean-field games of optimal stopping.arXiv preprint arXiv:2509.18821, 2025

    arXiv 2025

  14. [14]

    Donsker and S

    Monroe D. Donsker and S. R. Srinivasa Varadhan. Asymptotic evaluation of certain Markov process expecta- tions for large time—iii.Communications on Pure and Applied Mathematics, 29(4):389–461, 1976. ENTROPY REGULARIZATION IMPROVES ROBUSTNESS IN CONTINUOUS-TIME RL 12

    1976

  15. [15]

    Maximum entropy RL (provably) solves some robust RL problems

    Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. InInternational Conference on Learning Representations, 2022

    2022

  16. [16]

    Actor-critic learning for mean-field control in continuous time.Journal of Machine Learning Research, 26(127):1–42, 2025

    Noufel Frikha, Maximilien Germain, Mathieu Laurière, Huyên Pham, and Xuanye Song. Actor-critic learning for mean-field control in continuous time.Journal of Machine Learning Research, 26(127):1–42, 2025

    2025

  17. [17]

    Reinforcement learning for jump-diffusions, with financial applications.Mathematical Finance, 2026

    Xuefeng Gao, Lingfei Li, and Xun Yu Zhou. Reinforcement learning for jump-diffusions, with financial applications.Mathematical Finance, 2026. Early View

    2026

  18. [18]

    A theory of regularized Markov decision processes

    Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 2160–2169, 2019

    2019

  19. [19]

    Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems.SIAM Journal on Control and Optimization, 62(2):1060–1092, 2024

    Michael Giegrich, Christoph Reisinger, and Yufei Zhang. Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems.SIAM Journal on Control and Optimization, 62(2):1060–1092, 2024

    2024

  20. [20]

    Scalable first-order methods for robust MDPs

    Julien Grand-Clément and Christian Kroer. Scalable first-order methods for robust MDPs. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12086–12094, 2021

    2021

  21. [21]

    Continuous-time Markov decision processes

    Xianping Guo and Onésimo Hernández-Lerma. Continuous-time Markov decision processes. InContinuous- Time Markov Decision Processes: Theory and Applications, pages 9–18. Springer, 2009

    2009

  22. [22]

    Entropy regularization for mean field games with learning

    Xin Guo, Renyuan Xu, and Thaleia Zariphopoulou. Entropy regularization for mean field games with learning. Mathematics of Operations Research, 47(4):3239–3260, 2022

    2022

  23. [23]

    Soft actor-critic algorithms and applications

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, abs/1812.05905, 2018

    Pith/arXiv arXiv 2018

  24. [24]

    Continuous-time reinforcement learning for optimal switching over multiple regimes, 2025

    Yijie Huang, Mengge Li, Xiang Yu, and Zhou Zhou. Continuous-time reinforcement learning for optimal switching over multiple regimes, 2025

    2025

  25. [25]

    Regularized policies are reward robust

    Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. InInternational Conference on Artificial Intelligence and Statistics, pages 64–72. PMLR, 2021

    2021

  26. [26]

    Garud N. Iyengar. Robust dynamic programming.Mathematics of Operations Research, 30(2):257–280, 2005

    2005

  27. [27]

    Continuous-time risk-sensitive reinforcement learning via quadratic variation penalty.Applied Mathematics & Optimization, 93(2):58, 2026

    Yanwei Jia. Continuous-time risk-sensitive reinforcement learning via quadratic variation penalty.Applied Mathematics & Optimization, 93(2):58, 2026

    2026

  28. [28]

    Accuracy of discretely sampled stochastic policies in continuous-time reinforcement learning.arXiv preprint arXiv:2503.09981, 2025

    Yanwei Jia, Du Ouyang, and Yufei Zhang. Accuracy of discretely sampled stochastic policies in continuous-time reinforcement learning.arXiv preprint arXiv:2503.09981, 2025

    arXiv 2025

  29. [29]

    Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach.Journal of Machine Learning Research, 23(154):1–55, 2022

    Yanwei Jia and Xun Yu Zhou. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach.Journal of Machine Learning Research, 23(154):1–55, 2022

    2022

  30. [30]

    Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms.Journal of Machine Learning Research, 23(275):1–50, 2022

    Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms.Journal of Machine Learning Research, 23(275):1–50, 2022

    2022

  31. [31]

    q-learning in continuous time.Journal of Machine Learning Research, 24(161):1–61, 2023

    Yanwei Jia and Xun Yu Zhou. q-learning in continuous time.Journal of Machine Learning Research, 24(161):1–61, 2023

    2023

  32. [32]

    A Fisher–Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces.Foundations of Computational Mathematics, pages 1–75, 2025

    Bekzhan Kerimkulov, James-Michael Leahy, David Siska, Lukasz Szpruch, and Yufei Zhang. A Fisher–Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces.Foundations of Computational Mathematics, pages 1–75, 2025

    2025

  33. [33]

    Policy gradient for rectangular robust Markov decision processes

    Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Levy, and Shie Mannor. Policy gradient for rectangular robust Markov decision processes. InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  34. [34]

    Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes.Mathematical Programming, 198:1059–1106, 2023

    Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes.Mathematical Programming, 198:1059–1106, 2023

    2023

  35. [35]

    Policy gradient algorithms for robust MDPs with nonrectan- gular uncertainty sets.SIAM Journal on Optimization, 36(1):120–151, 2026

    Mengmeng Li, Daniel Kuhn, and Tobias Sutter. Policy gradient algorithms for robust MDPs with nonrectan- gular uncertainty sets.SIAM Journal on Optimization, 36(1):120–151, 2026

    2026

  36. [36]

    Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning.Advances in Neural Information Processing Systems, 35:22547–22561, 2022

    Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning.Advances in Neural Information Processing Systems, 35:22547–22561, 2022

    2022

  37. [37]

    Reinforcement learning in robust Markov decision processes

    Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust Markov decision processes. InAdvances in Neural Information Processing Systems, volume 26, 2013

    2013

  38. [38]

    Robust value iteration for continuous control tasks

    Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, and Animesh Garg. Robust value iteration for continuous control tasks. InRobotics: Science and Systems, 2021

    2021

  39. [39]

    Reinforcement learning for intensity control: An application to choice-based network revenue management.arXiv preprint arXiv:2406.05358, 2024

    Huiling Meng, Ningyuan Chen, and Xuefeng Gao. Reinforcement learning for intensity control: An application to choice-based network revenue management.arXiv preprint arXiv:2406.05358, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Robust reinforcement learning.Neural Computation, 17(2):335–359, 2005

    Jun Morimoto and Kenji Doya. Robust reinforcement learning.Neural Computation, 17(2):335–359, 2005

    2005

  41. [41]

    Robust control of Markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005

    Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005

    2005

  42. [42]

    Springer, 2007

    Bernt Øksendal and Agnès Sulem.Applied Stochastic Control of Jump Diffusions, volume 3. Springer, 2007

    2007

  43. [43]

    Variational inference for Markov jump processes.Advances in Neural Information Processing Systems, 20, 2007

    Manfred Opper and Guido Sanguinetti. Variational inference for Markov jump processes.Advances in Neural Information Processing Systems, 20, 2007

    2007

  44. [44]

    Robustness and risk-sensitivity in Markov decision processes

    Takayuki Osogami. Robustness and risk-sensitivity in Markov decision processes. InAdvances in Neural Information Processing Systems, volume 25, 2012. ENTROPY REGULARIZATION IMPROVES ROBUSTNESS IN CONTINUOUS-TIME RL 13

    2012

  45. [45]

    Sim-to-real transfer of robotic control with dynamics randomization

    Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3803–3810. IEEE, 2018

    2018

  46. [46]

    Continuous-time reinforcement learning for robust control under worst-case uncertainty.International Journal of Systems Science, 52(4):770–784, 2021

    Adolfo Perrusquía and Wen Yu. Continuous-time reinforcement learning for robust control under worst-case uncertainty.International Journal of Systems Science, 52(4):770–784, 2021

    2021

  47. [47]

    Robust adversarial reinforcement learning

    Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. InInternational Conference on Machine Learning, 2017

    2017

  48. [48]

    Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming

    Martin L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994

    1994

  49. [49]

    On stochastic optimal control and reinforcement learning by approximate inference

    Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. InProceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, pages 3052–3056, 2013

    2013

  50. [50]

    Regularity and stability of feedback relaxed controls.SIAM Journal on Control and Optimization, 59(5):3118–3151, 2021

    Christoph Reisinger and Yufei Zhang. Regularity and stability of feedback relaxed controls.SIAM Journal on Control and Optimization, 59(5):3118–3151, 2021

    2021

  51. [51]

    Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations, 2026

    Zhenjie Ren, Xiaoli Wei, Xiang Yu, and Xun Yu Zhou. Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations, 2026

    2026

  52. [52]

    Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms, 2026

    Zhenjie Ren, Xiaoli Wei, Xiang Yu, and Xun Yu Zhou. Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms, 2026

    2026

  53. [53]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. InInternational Conference on Machine Learning, pages 1889–1897. PMLR, 2015

    2015

  54. [54]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  55. [55]

    Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning.SIAM Journal on Control and Optimization, 62(1):135–166, 2024

    Lukasz Szpruch, Tanut Treetanthiploet, and Yufei Zhang. Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning.SIAM Journal on Control and Optimization, 62(1):135–166, 2024

    2024

  56. [56]

    Action robust reinforcement learning and applications in continuous control

    Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. InInternational Conference on Machine Learning, pages 6215–6224. PMLR, 2019

    2019

  57. [57]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30. IEEE, 2017

    2017

  58. [58]

    Linearly-solvable Markov decision problems

    Emanuel Todorov. Linearly-solvable Markov decision problems. InAdvances in Neural Information Processing Systems, volume 19, pages 1369–1376, 2006

    2006

  59. [59]

    Policy gradient in robust MDPs with global convergence guarantee

    Qiuhao Wang, Chin Pang Ho, and Marek Petrik. Policy gradient in robust MDPs with global convergence guarantee. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 35763–35797. PMLR, 2023

    2023

  60. [60]

    Continuous time q-learning for mean-field control problems.Applied Mathematics & Optimization, 91(1):10, 2025

    Xiaoli Wei and Xiang Yu. Continuous time q-learning for mean-field control problems.Applied Mathematics & Optimization, 91(1):10, 2025

    2025

  61. [61]

    Robust Markov decision processes.Mathematics of Operations Research, 38(1):153–183, 2013

    Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes.Mathematics of Operations Research, 38(1):153–183, 2013

    2013

  62. [62]

    Continuous-time q-learning for Markov regime switching system under Tsallis entropy, 2026

    Minghui Zhang, Xun Li, and Xin Zhang. Continuous-time q-learning for Markov regime switching system under Tsallis entropy, 2026

    2026

  63. [63]

    Policy optimization for continuous reinforcement learning

    Hanyang Zhao, Wenpin Tang, and David Yao. Policy optimization for continuous reinforcement learning. Advances in Neural Information Processing Systems, 36:13637–13663, 2023

    2023

  64. [64]

    X s∈S ¯dπ ρ(s) X a∈As µ(a|s) exp R(s, a)−R ˜θ∗(s, a) τ # =τlog

    Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, Panganamala R. Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. InAdvances in Neural Information Processing Systems, volume 36, 2023. ENTROPY REGULARIZATION IMPROVES ROBUSTNESS IN CONTINUOUS-TIME RL 14 AppendixA.Extended Related Work Continuous-time R...

    2023

  65. [65]

    The test statistic is t= WC(πτ)−WC(π std)q SE2 τ + SE2 0 , where SEτ and SE0 are the standard errors (across seeds) at the respective worst-case grid cells

    = arg mini,j ¯R0(i, j)denote the worst-case grid cells forπ τ andπ std respectively, where ¯Ris the seed-averaged reward. The test statistic is t= WC(πτ)−WC(π std)q SE2 τ + SE2 0 , where SEτ and SE0 are the standard errors (across seeds) at the respective worst-case grid cells. Under H0, this follows an approximatet-distribution (Welch’st-test with unequa...

    2000

  66. [66]

    In both market making and queueing, performance peaks at an intermediate∆t and degrades for both coarser and finer grids

    Fixed-grid performance is non-monotone in∆ t. In both market making and queueing, performance peaks at an intermediate∆t and degrades for both coarser and finer grids. This is not a training budget artefact: all configurations receive the same number of epochs. The root cause is a trade-off betweendiscretization bias(large∆t, too few decisions) andcredit ...

  67. [67]

    The arrival-driven implementation has no grid resolution hyperparameter

    Event-based discretization eliminates∆ t. The arrival-driven implementation has no grid resolution hyperparameter. It automatically operates at the natural time scale of the stochastic process, making decisions only when events occur. This yields competitive performance without tuning: event-driven PG achieves95.2%of J ∗ in market making (vs.97.5%for the ...

  68. [68]

    Computational cost scales with physics. In the event-based approach, the number of steps per episode is determined by the event rate of the environment (e.g., order arrival intensity in market making, transition rates in queueing), not by an artificial grid. This avoids wasting computation on time intervals where no events occur. In the market-making disc...