Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A data-driven actor–critic algorithm for continuous-time indefinite LQ control learns its own exploration schedule and still provably achieves the same $O(N^{3/4})$ regret bound as fixed-schedule methods.

desk verdict Solid companion extension with adaptive exploration and zero-initial-state analysis, but the central regret theorem is proved for the continuous-time recursion while the discretized algorithm's bias conditions are explicitly left unchecked. read the letter →

arxiv 2507.00358 v2 pith:KZBI7AVB submitted 2025-07-01 cs.LG cs.AIcs.SYeess.SYmath.OC

Yilie Huang , Xun Yu Zhou This is my paper · ORCID
classification cs.LGcs.AIcs.SYeess.SYmath.OC MSC 93E2093E3568T05
keywords continuous-timereinforcementlearningindefinitelinear-quadraticcontroldata-drivenexplorationentropyregularizationpolicygradientregretboundstochasticapproximationactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that exploration in continuous-time reinforcement learning can be made fully data-driven without sacrificing worst-case performance. For a class of scalar-state stochastic linear–quadratic control problems with state- and control-dependent volatility and no running control reward, it proposes an actor–critic algorithm that adaptively lowers the entropy-temperature parameter $\gamma_n$ and the Gaussian policy variance $\Gamma_n$ as learning progresses. The main result is that the cumulative regret, measured against the oracle optimal policy, grows at most as $O(N^{3/4})$ up to logarithmic factors, matching the best-known model-free bound that previously required a fixed or deterministic exploration schedule. The paper also removes the nonzero-initial-state assumption needed by the earlier fixed-schedule analysis. If correct, this means an RL agent can tune its own exploration from observed data and still match the theoretically established sublinear regret for this problem class.

What carries the argument

The argument is carried by two exact conditional-mean identities plus the adaptive schedule. For the actor variance, the expected increment of the stochastic-approximation update is $h^{(\Gamma)}(\Gamma_n;\theta_n,\gamma_n) = \tfrac12\big(\int_0^T k_1(t;\theta_n)\,dt\big)\Gamma_n\big(\sum_j D_jD_j^\top\big)\Gamma_n - \tfrac{\gamma_n T}{2}\Gamma_n$, and this quadratic form becomes contractive once $\gamma_n = c_\gamma\int k_1/(b_n T)$ makes the target $\Gamma_n^* = (\gamma_n T/\int k_1)(\sum_j D_jD_j^\top)^{-1} = (c_\gamma/b_n)(\sum_j D_jD_j^\top)^{-1}$ lie inside the projection set. For the policy mean, the expected increment is $h^{(\phi)}(\phi_n,\Gamma_n;\theta_n,\gamma_n) = -l(\phi_n,\Gamma_n)(\phi_n-\phi^*)$, where $l$ is bounded below by a positive multiple of $|\Gamma_n|$ when the initial state is zero. These identities convert the updates into stochastic approximations with effective learning rates $a_n^{(\Gamma)}/b_n$ and $a_n^{(\phi)}/b_n$; the choice $b_n = (n+\beta)^{1/4}/\alpha^{1/4}$ balances the bias from the shrinking target, the expanding projection bounds, and the variance bound, yielding the $O(N^{3/4})$ regret.

What would settle it

Run Algorithm 1 with a fixed seed and a long horizon, and estimate the conditional mean of $Z_n(T)$ and $Y_n(T)$ empirically at many iteration counts. If the fitted mean increment for $\Gamma_n$ differs from $\tfrac12(\int k_1)\Gamma_n(\sum_j D_jD_j^\top)\Gamma_n - \tfrac{\gamma_n T}{2}\Gamma_n$ by a term that does not decay faster than the effective learning rate $a_n^{(\Gamma)}/b_n$, or if the mean increment for $\phi_n$ is not collinear with $-(\phi_n-\phi^*)$, the contraction argument in Theorems 5.3 and 5.7 fails and the regret bound would not hold for the implemented scheme.

Watch

Extended reading notes

Core claim

The central claim is Theorem 5.9: for both zero and nonzero initial states, Algorithm 1 yields regret bounds of the form $\sum_{n=1}^N E[\bar{J}(\phi^*,0) - \bar{J}(\phi_n,\Gamma_n)] \le c + c N^{3/4}(\log N)^p(\log\log N)$, with the exponent $p$ depending on the case. The exploration parameters are not hand-tuned sequences: the critic's temperature is set by $\gamma_n = c_\gamma \int_0^T k_1(t;\theta_n)\,dt/(b_n T)$, and the actor's variance $\Gamma_n$ is updated by stochastic approximation using a policy-gradient estimator. Under this adaptive schedule the paper proves that $\gamma_n$ and $\Gamma_n$ converge almost surely to zero, that $\Gamma_n$ has mean-squared error $O((\log n)^{p_2}(\log\log n)^{4/3}/n^{1/2})$, that $\phi_n$ converges almost surely to the optimal mean coefficient $\phi^*$, and that the regret matches the fixed-schedule bound of [1]. The zero-initial-state case is handled by using $a_n^{(\phi)}/b_n$ as the effective learning rate, which compensates for the fact that the contraction coefficient for $\phi$ vanishes like $|\Gamma_n|$.

Load-bearing premise

The proof assumes that after time-discretization and projection, the expected parameter updates are exactly the clean mean-increment formulas in (35) and (50), with any extra bias small enough to be absorbed by the stated hyperparameter conditions; the discretization analysis needed to justify this for the implemented algorithm is deferred in Remark 5.10.

Editorial extensions

If this is right

  • The same $O(N^{3/4})$-type regret bound that was previously available only for fixed exploration schedules is preserved when both exploration parameters are learned from data.
  • Learning can start from the zero state $x_0 = 0$, which the earlier fixed-schedule analysis could not handle.
  • Exploration levels $\gamma_n$ and $\Gamma_n$ shrink to zero almost surely, so the algorithm asymptotically converges to the deterministic optimal policy $N(\cdot|\phi^*x, 0)$.
  • In numerical tests the adaptive schedule converges faster and accumulates less regret than both model-free fixed-exploration and model-based benchmarks.
  • Because exploration is driven by data rather than a pre-set schedule, the amount of manual tuning of exploration hyperparameters is reduced.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same adaptive-temperature and adaptive-variance mechanism could plausibly carry over to multi-dimensional states or to LQ problems with running control rewards, where the optimal policy is still Gaussian but the Riccati structure changes; the paper does not address these cases.
  • Numerical experiments show regret slopes around 0.73 in one setting, below the theoretical $3/4$ exponent; an unproved but testable question is whether data-driven exploration actually improves the exponent rather than only the constants.
  • The paper defers the discretization-error analysis to earlier references (Remark 5.10), so an explicit treatment of the discretized algorithm's bias would be needed for a fully implementation-ready guarantee.
  • Because the actor and critic reuse the same trajectory data in each iteration, the adaptive schedule may interact with off-policy or replay-based sampling; testing the algorithm with mini-batched or asynchronous updates would show whether the contraction arguments survive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies model-free continuous-time indefinite linear-quadratic reinforcement learning with state- and control-dependent volatility. It proposes an actor-critic algorithm in which the critic temperature gamma_n and actor variance Gamma_n are updated adaptively from data, using stochastic approximation with projections, and it proves almost-sure convergence of (gamma_n, Gamma_n, phi_n), mean-squared-error rates for Gamma_n and phi_n, and a cumulative regret bound of O(N^{3/4} polylog) that matches the best-known fixed-schedule bound of [1] and also covers the case x0 = 0. The analysis follows the continuous-time policy-gradient/temporal-difference machinery of [1] and reduces to stochastic approximation with time-varying bounds; numerical experiments compare the adaptive schedule with fixed-schedule baselines.

Significance. If the results are fully established, the paper makes a meaningful contribution: it replaces manually tuned deterministic exploration schedules by a provably convergent data-driven schedule, removes the nonzero-initial-state restriction, and preserves the best-known sublinear regret order. The theoretical rates are corroborated by log-log slopes (-0.51, -0.52, 0.73) that match the predicted -1/2, -1/2, and 3/4, and the experimental protocol reports random seeds for reproducibility. The main reservations concern the gap between the discretized Algorithm 1 and the continuous-time recursions that are actually analyzed, and the compressed proof of the x0 = 0 convergence rate.

major comments (3)
  1. [§4.4, Eqs. (27)–(30); Remark 5.10; Theorems 5.3 and 5.7] Theorem 5.9 is stated for Algorithm 1, which implements the discretized updates (27)–(30), but the proofs of Theorems 5.3 and 5.7 analyze the continuous-time recursions (31) and (49) driven by Z_n(T) and Y_n(T), whose exact mean increments are (35) and (50). Remark 5.10 asserts that discretization bias terms beta_n^{(Gamma)} and beta_n^{(phi)} 'should' be O(n^{-3/8}) and that choosing Delta t_n = T(n+1)^{-5/8} achieves this, with details omitted. Theorems 5.3 and 5.7 impose specific conditions on these bias terms, namely summability of a_n|beta_n| in (38)(i), the variance bound (37), and monotonicity of b_n^3 |beta_n|^2 / a_n, and none of these is verified for the discretized scheme in (27)–(30). Without this verification, the O(N^{3/4}) regret bound is proved only for the continuous-time recursion, not for the algorithm whose output Theorem 5.9 concerns.
  2. [§5.2, Theorem 5.8(b), Eq. (52)] The removal of the nonzero-initial-state assumption is one of the paper's advertised improvements over [1], yet the proof of the x0 = 0 case in Theorem 5.8(b) is only a sketch. After deriving the lower bound l(phi_n, Gamma_n) >= c_bar_1 |Gamma_n| I in (52), the text says to reinterpret a_n^{(phi)}/b_n as the effective learning rate and to 'follow the strategy outlined in Theorem 5.7', with the analysis of [1, Appendix B.6] carried over. The resulting MSE rate n^{-1/4} is then used in the x0 = 0 regret bound in Theorem 5.9. No recursion analogous to (45)–(47) is written for rho_n = E|phi_n - phi*|^2, and the role of the condition sum a_n^{(phi)}/b_n = infinity is not demonstrated. This is a load-bearing step and needs a complete derivation rather than an appeal to a different theorem.
  3. [§5.1.2, Eq. (35); §5.2, Eq. (50)] The contraction argument for the entire algorithm rests on the exact identities (35) and (50), both of which are asserted with only a pointer to 'taking integration and expectation in (33)' or to [1, Section B]. Because gamma_n is now a stochastic, data-dependent sequence (12) and Gamma_n is updated by policy gradient rather than by a deterministic schedule, these identities are not immediate corollaries of [1] even though the strategy is similar. The paper should either reproduce the derivations, which are conditional on the current parameter iterates and are short, or state precisely which results of [1] apply verbatim and why the adaptive gamma_n and Gamma_n do not alter them.
minor comments (4)
  1. [§6 and Appendix A] The numerical experiments do not implement Algorithm 1 exactly as analyzed: Appendix A states that the projections are 'slightly modified' to [−2.25, −1.1] for phi_n and [0, 1] for Gamma_n, with theta and gamma left unbounded, whereas Theorem 5.1 specifies expanding projection sets K_n^{(phi)} and K_n^{(Gamma)}. The paper should state clearly that the experiments validate a variant of the algorithm and discuss whether the theoretical guarantees are believed to extend to these fixed projection sets.
  2. [Algorithm 1] There is a typo in Algorithm 1: 'Uupdate state' should read 'Update state'.
  3. [Eq. (22)] In the projected update for theta, the entropy term is written as gamma_n p(t; phi_n, gamma_n); this should presumably be gamma_n p(t; phi_n, Gamma_n), consistent with Eqs. (18), (29), and (30).
  4. [References] Reference [1] is cited as 'Forthcoming'; if it has now appeared in SIAM Journal on Control and Optimization, the full publication details should be given.

Circularity Check

0 steps flagged · score 2.0 of 10

No constructional circularity: the O(N^{3/4}) regret bound is genuinely derived by stochastic approximation from the adaptive schedule; self-cited mean-increment formulas are parameter-free technical lemmas, and the deferred discretization analysis in Remark 5.10 is a correctness gap, not a reduction to inputs.

full rationale

The paper's derivation chain is: Algorithm 1 (Eqs. 27–30) is analyzed through the idealized continuous-time recursions (16)–(20); the two load-bearing mean-increment identities (35) and (50) are asserted and credited to [1, Section B]; Theorems 5.3 and 5.7 convert them, under the adaptive temperature rule (12) and the hyperparameter choices of Theorem 5.1, into the MSE rate E|Γ_n|^2 = O(n^{-1/2} polylog); Theorem 5.8 does the same for φ_n; Theorem 5.9 sums the per-iteration objective gap via (58)–(59) to obtain O(N^{3/4} polylog). I can exhibit no Eq.-equals-Eq. reduction: the regret (53) is the gap in the unregularized objective, not a relabeling of any fitted parameter or schedule. The Section 3.2 admission that rule (12) is 'chosen to ensure convergence and achieve the desired sublinear regret bounds' is an honest design attribution: the bound's order is targeted by b_n = Θ(n^{-1/4}) and a_n = Θ(n^{-3/4}), and Theorems 5.3–5.9 certify that the stochastic-approximation contraction (variance bound (37), coercivity (44), summability (38)) actually delivers that order; that is ordinary schedule tuning, not constructional circularity. The self-citations to the same group's [1] for (35), (50), Lemma 5.2, and Lemmas B.7–B.8 are load-bearing, but they are parameter-free computations from the stated estimator definitions that do not contain the adaptive-exploration claim, hence they count as real evidence under the review rules and do not raise the circularity score. The genuine weakness is Remark 5.10: Theorems 5.3–5.8 require bias terms β_n of order n^{-3/8}, the paper asserts Δt_n = T(n+1)^{-5/8} achieves this, and explicitly omits the verification ('We omit the details here'), so Theorem 5.9 is proven for the continuous-time recursion rather than the discretized Algorithm 1; Appendix A further deviates by fixing projection sets in the experiments. That is a correctness gap, not circularity, and it is what the score understates. Verdict: no significant circularity (score 2).

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper's theorems depend on the problem class, on structural results inherited from the authors' companion paper [1], on boundedness and mean-increment assumptions, and on standard stochastic approximation theorems. The free parameters are hyperparameters of the schedule and experimental choices; none are fitted to data in the sense of empirical model fitting, but several are hand-chosen to make the desired O(N^{3/4}) rate attainable.

free parameters (5)
  • c_γ = unspecified; must satisfy 1/c_γ < λ_min((ΣD_jD_j^T)^{−1})
    Sets the scale of the adaptive temperature γ_n = c_γ∫k1/(b_nT) and hence the target Γ*_n = c_γ/b_n (ΣD_jD_j^T)^{−1}; the regret order does not depend on it, but the algorithm's trajectory does.
  • α, β = α^{3/4}=1 for a^{(Γ)}, α^{3/4}=0.05 for a^{(φ)}, β=1 in experiments
    Define the learning rates a_n = α^{3/4}/(n+β)^{3/4} and the sequence b_n = 1∨(n+β)^{1/4}/α^{1/4}; these constants set the convergence rate and are tuned by hand.
  • c^{(θ)}, c^{(γ)} = unbounded in experiments
    Projection bounds enforcing the boundedness conditions (11) of the critic; the order of the regret bound is unaffected, but the theory requires them finite while the experiments leave them unbounded.
  • Experimental projection bounds for φ and Γ = φ∈[-2.25,-1.1], Γ∈[0,1]
    Used in all experiments instead of the theoretically required expanding sets K^{(φ)}_n and K^{(Γ)}_n; this deviation is acknowledged but not analyzed.
  • Trivial critic k1=1, k3=0 = k1(t;θ)=1, k3(t;θ,γ)=0
    Chosen in experiments for computational efficiency; the theory requires k1,k3 only to satisfy (11), but the experiments do not exercise the critic parameterization the algorithm is designed for.
assumptions (6)
  • domain assumption The state process follows the SDE (1) with scalar A,C_j, vector B,D_j, and ΣD_jD_j^T>0, objective (2).
    Defines the problem class; imported from [1].
  • domain assumption The optimal value function is quadratic and the optimal randomized policy is Gaussian, Eqs. (7)-(9).
    Inherited from the theory in [1], Section 2.2; the algorithm is built on this structure.
  • ad hoc to paper k1, k3 parameterizations are bounded with |k1|≥1/c2, |k1'|≤c1, |k3'|≤c3 (Eq. 11).
    Imposed to make the SA analysis tractable; the comment after (11) notes these follow from the known solution's properties only when parameters are known, so the assumption is a design constraint on the approximator.
  • ad hoc to paper The mean increments of the stochastic updates have the exact forms (35) and (50).
    These structural identities underlie Theorems 5.3, 5.7, and 5.8; they are not re-derived here but taken from the machinery of [1].
  • standard math The sequence conditions in (38) hold: Σa_n^{(Γ)}=∞, Σa_n^{(Γ)}|β_n|<∞, Σ(a_n^{(Γ)})^2(...)e^{c(...)}<∞, Σa_n^{(Γ)}/b_n=∞, Σ|1/b_n−1/b_{n+1}|<∞.
    These are standard stochastic-approximation conditions verified for the chosen hyperparameters in Remark 5.4; they are assumptions in Theorem 5.3.
  • standard math Robbins-Siegmund almost-supermartingale theorem and SA convergence theorems [45, 46, 47].
    Unproved background results used for convergence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems." pith.science (2026). https://pith.science/paper/KZBI7AVB

@misc{pith2026250700358,
  author       = {Pith},
  title        = {Pith review of: Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZBI7AVB}},
  note         = {Machine review of arXiv:2507.00358}
}
read the original abstract

We study reinforcement learning (RL) for the same class of continuous-time stochastic linear--quadratic (LQ) control problems as in \cite{huang2024sublinear}, where volatilities depend on both states and controls while states are scalar-valued and running control rewards are absent. We propose a model-free, data-driven exploration mechanism that adaptively adjusts entropy regularization by the critic and policy variance by the actor. Unlike the constant or deterministic exploration schedules employed in \cite{huang2024sublinear}, which require extensive tuning for implementations and ignore learning progresses during iterations, our adaptive exploratory approach boosts learning efficiency with minimal tuning. Despite its flexibility, our method achieves a sublinear regret bound that matches the best-known model-free results for this class of LQ problems, which were previously derived only with fixed exploration schedules. Numerical experiments demonstrate that adaptive explorations accelerate convergence and improve regret performance compared to the non-adaptive model-free and model-based counterparts.

Figures

Figures reproduced from arXiv: 2507.00358 by the authors.

Figure 1
Figure 1. Log-log plot of Algorithm 1. numerical results are consistent with the theoretical analysis, confirming the long-term effectiveness of our learning algorithm. 6.2 Model-Free vs. Model-Based Under the same setting, we compare our model-free LQ-RL algorithm with data-driven explo￾ration to the recently developed model-based method with a deterministic exploration schedule [34], which estimates parameters A and B in th… view at source ↗
Figure 2
Figure 2. Log-log plot of model-based LQ-RL algorithm with fixed exploration schedule. considerably larger than that of ours.3 6.3 Adaptive vs. Fixed Explorations Now, we compare two model-free continuous-time LQ-RL algorithms: ours with data-driven exploration (LQRL Adaptive) and the one by [1] with fixed exploration schedules (LQRL Fixed). Both algorithms follow the same fundamental framework, with the key distinction being… view at source ↗
Figure 3
Figure 3. Comparison of exploration level and cumulative regret under excessive initial exploration. [PITH_FULL_IMAGE:figures/full_fig_p030_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of exploration level and cumulative regret under insufficient initial explo [PITH_FULL_IMAGE:figures/full_fig_p030_4.png]
Figure 5
Figure 5. Figure 5: Comparison of regrets under randomized environments. [PITH_FULL_IMAGE:figures/full_fig_p031_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learning

    cs.LG 2026-07 unverdicted novelty 7.0 of 10

    ART-RL learns adaptive diffusion sampling timesteps via continuous-time control and Gaussian actor–critic RL, improving and transferring over hand-designed grids at matched budgets.

  2. Continuous-Time Reinforcement Learning for $N$-Player Stochastic Differential Games with Exploratory Policies

    math.OC 2026-07 conditional novelty 6.0 of 10

    For entropy-regularized N-player differential games, a Nash-type equilibrium exists exactly when the Gibbs conditional best responses are jointly compatible, checkable via a cross-partial criterion on the learned q-functions.

Reference graph

Works this paper leans on

49 extracted references · 39 canonical work pages · cited by 2 Pith papers

  1. [1]

    Sublinear regret for a class of continuous-time linear– quadratic reinforcement learning problems,

    Y. Huang, Y. Jia, and X. Y. Zhou, “Sublinear regret for a class of continuous-time linear– quadratic reinforcement learning problems,”SIAM Journal on Control and Optimization, 2025. Forthcoming. Available at https://arxiv.org/abs/2407.17226

    arXiv 2025

  2. [2]

    B. D. Anderson and J. B. Moore, Optimal Control: Linear Quadratic Methods . Courier Cor- poration, 2007. 33

    work page 2007

  3. [3]

    Yong and X

    J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations . New York, NY: Spinger, 1999

    work page 1999

  4. [4]

    Stochastic linear quadratic regulators with indefinite control weight costs,

    S. Chen, X. Li, and X. Y. Zhou, “Stochastic linear quadratic regulators with indefinite control weight costs,” SIAM Journal on Control and Optimization , vol. 36, no. 5, pp. 1685–1702, 1998

    work page 1998

  5. [5]

    Well-posedness and attainability of indefinite stochastic linear quadratic control in infinite time horizon,

    M. Ait Rami, X. Y. Zhou, and J. Moore, “Well-posedness and attainability of indefinite stochastic linear quadratic control in infinite time horizon,” Systems & Control Letters, vol. 41, no. 2, pp. 123–133, 2000

    work page 2000

  6. [6]

    Linear matrix inequalities, riccati equations, and indefinite stochastic linear quadratic controls,

    M. A. Rami and X. Y. Zhou, “Linear matrix inequalities, riccati equations, and indefinite stochastic linear quadratic controls,” IEEE Transactions on Automatic Control, vol. 45, no. 6, pp. 1131–1143, 2000

    work page 2000

  7. [7]

    Solvability and asymptotic behavior of generalized riccati equations arising in indefinite stochastic lq controls,

    M. A. Rami, X. Chen, J. B. Moore, and X. Y. Zhou, “Solvability and asymptotic behavior of generalized riccati equations arising in indefinite stochastic lq controls,” IEEE Transactions on Automatic Control , vol. 46, no. 3, pp. 428–440, 2001

    work page 2001

  8. [8]

    A primal-dual semi-definite programming approach to linear quadratic control,

    D. D. Yao, S. Zhang, and X. Y. Zhou, “A primal-dual semi-definite programming approach to linear quadratic control,” IEEE Transactions on Automatic Control , vol. 46, no. 9, pp. 1442– 1447, 2001

    work page 2001

Show all 49 references
  1. [9]

    Stabilization control for itˆ o stochastic system with indefinite state and control weight costs,

    H. Li, Q. Qi, and H. Zhang, “Stabilization control for itˆ o stochastic system with indefinite state and control weight costs,” International Journal of Control , vol. 95, no. 2, pp. 295–302, 2022

    2022

  2. [10]

    Optimal regulators for a class of nonlinear stochastic systems,

    B. Gashi and H. Hua, “Optimal regulators for a class of nonlinear stochastic systems,” Inter- national Journal of Control , vol. 96, no. 1, pp. 136–146, 2023

    2023

  3. [11]

    Indefinite linear-quadratic optimal control of mean-field stochas- tic differential equation with jump diffusion: an equivalent cost functional method,

    G. Wang and W. Wang, “Indefinite linear-quadratic optimal control of mean-field stochas- tic differential equation with jump diffusion: an equivalent cost functional method,” IEEE Transactions on Automatic Control , 2024

    2024

  4. [12]

    Stochastic linear quadratic optimal control problems with regime- switching jumps in infinite horizon,

    F. Wu, X. Li, and X. Zhang, “Stochastic linear quadratic optimal control problems with regime- switching jumps in infinite horizon,”SIAM Journal on Control and Optimization, vol. 63, no. 2, pp. 852–891, 2025. 34

    2025

  5. [13]

    On estimating the expected return on the market: An exploratory investiga- tion,

    R. C. Merton, “On estimating the expected return on the market: An exploratory investiga- tion,” Journal of Financial Economics , vol. 8, no. 4, pp. 323–361, 1980

    1980

  6. [14]

    D. G. Luenberger, Investment Science. Oxford University Press, 1998

    1998

  7. [15]

    Rustem and M

    B. Rustem and M. Howe, Algorithms for worst-case design and applications to risk manage- ment. Princeton University Press, 2009

    2009

  8. [16]

    R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction . Cambridge, MA: MIT Press, 2018

    2018

  9. [17]

    A survey on intrinsic motivation in reinforcement learning,

    A. Aubret, L. Matignon, and S. Hassas, “A survey on intrinsic motivation in reinforcement learning,” arXiv preprint arXiv:1908.06976 , 2019

    1908 arXiv

  10. [18]

    Formal theory of creativity, fun, and intrinsic motivation (1990–2010),

    J. Schmidhuber, “Formal theory of creativity, fun, and intrinsic motivation (1990–2010),” IEEE Transactions on Autonomous Mental Development , vol. 2, no. 3, pp. 230–247, 2010

    1990

  11. [19]

    Surprise-based intrinsic motivation for deep reinforcement learning,

    J. Achiam and S. Sastry, “Surprise-based intrinsic motivation for deep reinforcement learning,” arXiv preprint arXiv:1703.01732 , 2017

    2017 arXiv

  12. [20]

    Curiosity-driven exploration by self- supervised prediction,

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self- supervised prediction,” in International Conference on Machine Learning , pp. 2778–2787, PMLR, 2017

    2017

  13. [21]

    Large-scale study of curiosity-driven learning,

    Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros, “Large-scale study of curiosity-driven learning,” arXiv preprint arXiv:1808.04355 , 2018

    2018 arXiv

  14. [22]

    Exploration by random network distilla- tion,

    Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distilla- tion,” arXiv preprint arXiv:1810.12894 , 2018

    2018 arXiv

  15. [23]

    Randomized prior functions for deep reinforcement learning,

    I. Osband, J. Aslanides, and A. Cassirer, “Randomized prior functions for deep reinforcement learning,” Advances in neural information processing systems , vol. 31, 2018

    2018

  16. [24]

    Fast active learning for pure exploration in reinforcement learning,

    P. M´ enard, O. D. Domingues, A. Jonsson, E. Kaufmann, E. Leurent, and M. Valko, “Fast active learning for pure exploration in reinforcement learning,” in International Conference on Machine Learning, pp. 7599–7608, PMLR, 2021

    2021

  17. [25]

    # exploration: A study of count-based exploration for deep reinforcement learning,

    H. Tang, R. Houthooft, D. Foote, A. Stooke, O. Xi Chen, Y. Duan, J. Schulman, F. DeTurck, and P. Abbeel, “# exploration: A study of count-based exploration for deep reinforcement learning,” Advances in neural information processing systems , vol. 30, 2017. 35

    2017

  18. [26]

    Go-explore: A new approach for hard-exploration problems,

    A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, “Go-explore: A new approach for hard-exploration problems,” arXiv preprint arXiv:1901.10995 , 2019

    1901 arXiv

  19. [27]

    First return, then explore,

    A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, “First return, then explore,” Nature, vol. 590, no. 7847, pp. 580–586, 2021

    2021

  20. [28]

    Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards,

    M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Roth¨ orl, T. Lampe, and M. Riedmiller, “Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards,” arXiv preprint arXiv:1707.08817 , 2017

    2017 arXiv

  21. [29]

    A comprehensive survey on safe reinforcement learning,

    J. Garcıa and F. Fern´ andez, “A comprehensive survey on safe reinforcement learning,”Journal of Machine Learning Research, vol. 16, no. 1, pp. 1437–1480, 2015

    2015

  22. [30]

    Trial without error: Towards safe reinforcement learning via human intervention,

    W. Saunders, G. Sastry, A. Stuhlmueller, and O. Evans, “Trial without error: Towards safe reinforcement learning via human intervention,” arXiv preprint arXiv:1707.05173 , 2017

    2017 arXiv

  23. [31]

    Policy gradient in continuous time,

    R. Munos, “Policy gradient in continuous time,” Journal of Machine Learning Research, vol. 7, pp. 771–791, 2006

    2006

  24. [32]

    Making deep Q-learning methods robust to time dis- cretization,

    C. Tallec, L. Blier, and Y. Ollivier, “Making deep Q-learning methods robust to time dis- cretization,” in International Conference on Machine Learning , pp. 6096–6104, PMLR, 2019

    2019

  25. [33]

    Time discretization-invariant safe action repetition for policy gradient methods,

    S. Park, J. Kim, and G. Kim, “Time discretization-invariant safe action repetition for policy gradient methods,” Advances in Neural Information Processing Systems, vol. 34, pp. 267–279, 2021

    2021

  26. [34]

    Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning,

    L. Szpruch, T. Treetanthiploet, and Y. Zhang, “Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning,” SIAM Journal on Control and Optimization, vol. 62, no. 1, pp. 135–166, 2024

    2024

  27. [35]

    Indefinite stochastic riccati equations,

    Y. Hu and X. Y. Zhou, “Indefinite stochastic riccati equations,” SIAM Journal on Control and Optimization , vol. 42, no. 1, pp. 123–137, 2003

    2003

  28. [36]

    Existence of solutions to a class of indefinite stochastic riccati equations,

    Z. Qian and X. Y. Zhou, “Existence of solutions to a class of indefinite stochastic riccati equations,” SIAM journal on Control and Optimization , vol. 51, no. 1, pp. 221–229, 2013

    2013

  29. [37]

    Reinforcement learning in continuous time and space: A stochastic control approach,

    H. Wang, T. Zariphopoulou, and X. Y. Zhou, “Reinforcement learning in continuous time and space: A stochastic control approach,” Journal of Machine Learning Research, vol. 21, no. 198, pp. 1–34, 2020. 36

    2020

  30. [38]

    Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models,

    L. Szpruch, T. Treetanthiploet, and Y. Zhang, “Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models,” arXiv preprint arXiv:2112.10264, 2021

    2021 arXiv

  31. [39]

    Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms,

    Y. Jia and X. Y. Zhou, “Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms,” Journal of Machine Learning Research , vol. 23, no. 154, pp. 1–55, 2022

    2022

  32. [40]

    Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach,

    Y. Jia and X. Y. Zhou, “Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach,” Journal of Machine Learning Research , vol. 23, no. 154, pp. 1–55, 2022

    2022

  33. [41]

    A stochastic approximation method,

    H. Robbins and S. Monro, “A stochastic approximation method,” The Annals of Mathematical Statistics, pp. 400–407, 1951

    1951

  34. [42]

    Stochastic approximation,

    T. L. Lai, “Stochastic approximation,” The Annals of Statistics , vol. 31, no. 2, pp. 391–406, 2003

    2003

  35. [43]

    V. S. Borkar, Stochastic approximation: A dynamical systems viewpoint , vol. 48. Springer, 2009

    2009

  36. [44]

    An overview of stochastic approximation,

    M. Chau and M. C. Fu, “An overview of stochastic approximation,” Handbook of Simulation Optimization, pp. 149–178, 2014

    2014

  37. [45]

    A stochastic approximation algorithm with varying bounds,

    S. Andrad´ ottir, “A stochastic approximation algorithm with varying bounds,”Operations Re- search, vol. 43, no. 6, pp. 1037–1048, 1995

    1995

  38. [46]

    General bounds and finite-time improvement for the Kiefer-Wolfowitz stochastic approximation algorithm,

    M. Broadie, D. Cicek, and A. Zeevi, “General bounds and finite-time improvement for the Kiefer-Wolfowitz stochastic approximation algorithm,” Operations Research, vol. 59, no. 5, pp. 1211–1224, 2011

    2011

  39. [47]

    A convergence theorem for non negative almost supermartin- gales and some applications,

    H. Robbins and D. Siegmund, “A convergence theorem for non negative almost supermartin- gales and some applications,” in Optimizing Methods in Statistics , pp. 233–257, Elsevier, 1971

    1971

  40. [48]

    P. E. Kloeden and E. Platen, Numerical Solution of Stochastic Differential Equations , vol. 23. Springer, 1992

    1992

  41. [49]

    Regret of exploratory policy improvement and q-learning,

    W. Tang and X. Y. Zhou, “Regret of exploratory policy improvement and q-learning,” arXiv preprint arXiv:2411.01302, 2024. 37

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.