Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gaussian mixtures can replace deep nets as Q-function approximators in policy iteration while keeping a fixed parameter budget and provable error bounds.

desk verdict A fresh parametric Q-function idea with clean theory and an unproven optimization oracle in the main bound; peer-reviewable, but only after artifacts and statistical rigor. read the letter →

arxiv 2512.18763 v2 pith:FZFG4CUP submitted 2025-12-21 cs.LG

Minh Vu , Konstantinos Slavakis This is my paper
classification cs.LG MSC 68T0590C4049M37
keywords GaussianmixturemodelsQ-functionapproximationpolicyiterationBellmanresidualRiemannianoptimizationuniversalreinforcementlearningpositive-definitemanifold
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes using Gaussian mixture models (GMMs) directly as Q-function approximators in reinforcement learning, rather than as density estimators. It claims that a fixed-size mixture of Gaussians—with learnable weights, means, and covariances—forms a universal approximator for continuous Q-functions on compact state-action spaces, and that fitting it by minimizing Bellman residuals is as good as searching over all continuous functions. The fitting is done by Riemannian optimization on a product manifold, where covariance matrices live on the positive-definite manifold. Under a set of assumptions, the paper proves that policy iteration built on these estimators keeps the estimated Q-functions within a bounded distance of the optimal Q-function. If true, this gives a lightweight, parameter-frugal alternative to deep Q-networks that does not need experience replay.

What carries the argument

The named object is the GMM-QF class Q_K := {Σ_k ξ_k G(·|m_k,C_k)}. The carrying mechanism is the product Riemannian manifold M_K, whose factors are Euclidean spaces for mixing weights and means, and the manifold of positive-definite matrices for covariances; retractions are implemented with matrix exponentials (AffI metric) or Lyapunov solves (BW metric). The universal-approximation result (Theorem 3) justifies restricting attention to Q_K, while the gradient formulas (Proposition 7) make Riemannian steepest descent with Armijo backtracking concrete. The error analysis (Proposition 14) turns the Bellman-residual decrease of Algorithm 2 into an L2 bound on Q-function error.

What would settle it

Take a small MDP (e.g., a two-state, two-action chain) with known optimal Q-function; for each policy iteration, run Algorithm 2 with increasing K and compare the empirical Bellman residual it returns against the infimum over Q_K computed by exhaustive or many-restart global search. If the gap exceeds a preset small Δ, or grows with K or policy index n, Assumption 13(ii) fails and the Proposition 14 bound becomes vacuous for that setting.

Watch

Extended reading notes

Core claim

The central object is Q_K, the set of functions of the form Σ_{k=1}^K ξ_k G(·|m_k,C_k), with no constraint on weights and with learnable covariance matrices. The paper proves Q_K is dense in C(Z) on compact Z—and in L2 on all of R^{D_z}—so representational capacity is not a bottleneck. It then embeds Q_K in the Bellman residual loss, treats the parameter tuple as a point on the product Riemannian manifold M_K = R^K × (R^{D_z})^K × (S_{++}^{D_z})^K, and derives closed-form gradients for the weights, means, and covariances under two metrics. The main performance theorem says that, under assumptions on uniform approximation of the loss, near-global minimization of the residual, sufficient K, an

Load-bearing premise

Assumption 13(ii): a finite run of nonconvex Riemannian steepest descent on the Bellman-residual loss returns, at every policy-iteration step, a Q-function within a small additive gap Δ of the global infimum over Q_K.

Editorial extensions

If this is right

  • Policy evaluation becomes parametric: a fixed number of Gaussians yields a Q-function estimate whose parameter count does not grow with the number of collected transitions.
  • Because ∪_K Q_K is dense in C(Z), any continuous Q-function can be approximated to arbitrary precision, so the parametric class does not cap expressiveness.
  • The method provides explicit gradients on a Riemannian manifold, enabling first-order optimization for covariance matrices while preserving positive-definiteness.
  • Under Assumptions 13, approximate policy iteration with GMM-QFs stays within a bounded band of the optimal Q-function; the band shrinks as Bellman-residual error and policy-drift constants shrink.
  • The benchmarks suggest the method can match or beat deep RL baselines with orders of magnitude fewer learnable parameters and without replay-buffer experience data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • In practice the Δ in Assumption 13(ii) is the crux: since Algorithm 2 only guarantees critical points, the method's reliability depends on how benign the nonconvex Bellman-residual landscape is—this is testable on small MDPs by comparing against global or many-restart search.
  • The observed non-monotonicity of performance in K (K=500 underperforms K=50 on the acrobot task) hints at overfitting to the biased Bellman-residual objective; automatic K selection via structured sparsification is a natural next step.
  • If the bounded-error guarantee holds online, GMM-QFs may be useful for fast adaptation to non-stationary environments, where retraining a deep network at each policy step is impractical.
  • The discrete-action greedy improvement step is separate from the smooth GMM parameterization; a continuous-action extension would need a different policy-improvement rule, but the density result suggests the representation itself would carry over.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Gaussian-mixture-model Q-functions (GMM-QFs) as a parametric, fixed-size function-approximation class for Q-functions in policy iteration. The Q-function is represented as a weighted sum of Gaussians with learnable means, covariances, and weights, trained by minimizing the empirical Bellman residual via Riemannian steepest descent on the product manifold (R^K × (R^{D_z})^K × (S_{++}^{D_z})^K). The authors prove a universal-approximation theorem for this class, derive closed-form Riemannian gradients, and give an approximate-PI performance bound under a set of assumptions. Numerical experiments on inverted pendulum, mountain car, and acrobot compare the method with KLSPI, OBR, DQN, dueling DDQN, PPO, and EM-based GMM reinforcement learning.

Significance. If the theoretical results were correct, the paper would offer a useful lightweight parametric alternative to RKHS-based nonparametric Q-estimators and deep Q-networks, with a novel connection to Riemannian optimization. Strengths include the detailed universal-approximation analysis (Theorem 3), explicit gradient formulas (Proposition 7), and a serious attempt at a performance bound for the policy-iteration scheme. However, the central quantitative guarantee depends critically on two unproven ingredients: an L2 contraction property for the Bellman optimality operator, and an oracle-like assumption on the finite-budget nonconvex optimizer. The contraction claim appears to be incorrect as stated, and the optimization assumption is unverified. The empirical section also lacks confidence intervals, seed counts, and released code. Thus, although the proposal is interesting, the main correctness and reproducibility claims are not currently established.

major comments (3)
  1. [Section IV-C, Proposition 11, Eq. (20b)] The assertion that the Bellman optimality operator T^⋄ is an α-contraction on L2(P) is not justified by the stated assumptions. In the proof, Fact 15 selects an auxiliary action a^♯(s') that nearly realizes the difference between the two minima, and then the term E{|Q1(z')-Q2(z')|^2} is identified with ||Q1-Q2||_{L2(P)}^2 where z' = ζ(s', a^♯(s')). But the distribution of ζ(s', a^♯(s')) is the state-action distribution of the auxiliary policy a^♯, which is not the same as the data distribution P in general. As a concrete illustration, take a finite MDP with deterministic transitions and let P be supported only on action a1 at state s0. Let Q1=0 and let Q2(s0,a1)=0, Q2(s0,a2)=-c, with Q2=0 elsewhere. Then ||Q1-Q2||_{L2(P)}=0, but at the observed pair (s0,a1) we have T^⋄Q2(s0,a1) = -αc, so ||T^⋄Q1 - T^⋄Q2||_{L2(P)} = α c > 0, contradicting (20b). Assumption 10(i) does not prevent this; if
  2. [Assumptions 13(ii) and Appendix G, Eq. (35b)] The main performance bound hinges on Assumption 13(ii), which asserts that the finite-budget Riemannian steepest descent in Algorithm 2 returns an empirical-loss value within an additive Δ of the global infimum over Q_K, for every n and almost surely. Algorithm 2 is a nonconvex Riemannian descent method; Theorem 8 guarantees only that infinite-run accumulation points are critical points, and the paper itself acknowledges that critical points need not be global minimizers. No bound on the optimization gap is proved, and the Armijo condition only gives sufficient decrease per step. Moreover, the wording "for any observed data, there exists Δ" allows Δ to depend on the data and on n, while the proof in (35b) uses a single uniform Δ. If Δ is large or data-dependent, the limsup bound in Proposition 14 is vacuous. The numerical section does not report any estimate of Δ, δ_inf, or σ_V^2. I woul
  3. [Section V, Figures 5–9 and Table I] The empirical claim that GMM-QFs are competitive with, and sometimes outperform, deep RL methods is supported only by single learning curves without error bars or confidence intervals. The text says results are averaged over multiple independent runs, but it does not state the number of seeds, the standard deviation, or the hyperparameter sensitivity. Figure 6 even shows DQN achieving the best performance in the mountain-car task, so the strength of the claimed advantage over DQN is unclear without uncertainty quantification. Furthermore, no code or repository is provided, and Figure 9 reports cumulative number of gradient evaluations rather than wall-clock time, which weakens the computational-efficiency claim. Please add seed counts, confidence intervals, hyperparameter settings, and either release code or give a detailed reproducible protocol.
minor comments (4)
  1. [Throughout] There are typographical errors that should be corrected: "trails" instead of "trials" in Figures 5–8, "Nuber" in Figure 8, "conmputational" in Section IV-D, and "experiement" in Section V-C.
  2. [Section IV-C, Proposition 14] The text after Proposition 14 says that Assumption 13(v) is required for Proposition 14(i), but the proof of part (i) does not use Assumption 13(v); it is used only in part (ii). This should be clarified.
  3. [Section V] The paper does not report the chosen values of Algorithm 2's J, Armijo parameters, or the number of independent runs for each benchmark. These are essential for reproducibility and for interpreting the complexity plot in Figure 9.
  4. [References] Reference [48] is formatted incompletely ("N. J. Ehsan Saleh"), and some URLs/repositories are not dated or versioned. Please complete the bibliography entries.

Circularity Check

0 steps flagged · score 2.0 of 10

No meaningful circularity: the GMM-QF density theorem (Thm. 3) is proved from first principles, and the PI error bound (Prop. 14) is a conditional oracle inequality in constants that are explicitly assumed, not fitted or re-derived. Only minor, non-load-bearing self-citations ([15], [34]) appear; numerics are validated on external benchmarks.

full rationale

Walking the derivation chain: (1) The function class Q_K in Eq. (7) is defined independently of any target Q^⋐; Theorem 3 is proved in Appendix B from Stone-Weierstrass for (i), a density-of-C(Z)-in-L2(P) argument for (ii), and a Hahn-Banach/Fourier argument for (iii), with the debt to prior RBF/EBF results [33,38] acknowledged in the text. No target result is assumed inside these proofs. (2) Proposition 4 follows directly from the density result and is proved in Appendix C; it is not a renaming of a known result but a consequence of the self-contained Theorem 3. (3) Proposition 14 is a conditional oracle inequality: the chain (35a)-(35g) bounds the Bellman residual of the iterate by inf over C(Z) plus the assumed constants δ_E♯, Δ, δ_inf, then the variance proxy σ_V^2 and the successive-fixed-point gap Δ_2. All of these are declared in Assumptions 13 as assumptions; none is fitted from data or renamed as an output, and every term on the right-hand side is transparently an input. This is the standard structure of approximate-PI analyses (the paper itself notes the parallel to [1, (5.11)]), not an Eq.-X-equals-Eq.-Y reduction. (4) The self-citations are not load-bearing: Fact 15 from [15] is quoted verbatim and is an elementary finite-action epsilon lemma that can be checked without consulting [15]; Assumption 13(v) is an explicit assumption whose motivation cites [15] but whose content is not derived from it; [34] is disclosed as the conference precursor. (5) The paper itself flags the main validity concern in the text after Theorem 8: 'due to the non-convexity of the objective \widehat{L}_{μ_n[T]}, locating its critical points does not guarantee the identification of its global minimizers.' Assumption 13(ii) is a strong, possibly vacuous oracle condition because Δ is only asserted to exist and is never measured; a bound controlled by an unexamined Δ may be vacuous, but that is a correctness/assumption-strength issue, not circularity. (6) The numerical claims rest on external benchmark tasks (inverted pendulum, mountain car, acrobot) with standard dynamics, independent of the theory's constants; no quantity called a 'prediction' is reconstructed from a fitted parameter. Hence no circular step is identified; the score of 2 reflects only the minor, non-load-bearing self-citations.

Assumptions & free parameters 7 free parameters · 11 assumptions · 0 invented entities

The theoretical core rests on standard approximation theory plus a set of strong existence assumptions whose constants are not computed or empirically checked. The method itself has user-chosen hyperparameters—K, J, metric, initialization, and dataset size—that materially affect the reported behavior.

free parameters (7)
  • K (number of Gaussian components) = 5, 20, 50, 200, 500 across experiments
    User-chosen model size; performance depends strongly on K (Figure 7), and Assumption 13(iii) assumes K is large enough.
  • J (gradient iteration budget in Algorithm 2) = 10–30 in Figure 9
    Finite-budget optimization; Assumption 13(ii) effectively assumes the budget is sufficient to near-minimize the nonconvex loss.
  • Discount factor alpha = not reported in experiments
    Needed for contraction; chosen by hand; not stated in the numerical section.
  • Armijo line-search constants (alpha_bar, beta, sigma_A) = not reported
    Control step sizes in Algorithm 2; no concrete values are given.
  • Riemannian metric on positive-definite matrices = AffI or BW
    Changes gradients and retractions; both are tested but no selection principle is provided.
  • Dataset size T per policy iteration = 1400, 1000, 1400 for the three tasks
    Exploration effort is chosen per task and affects optimization and bounds through the empirical-loss approximation.
  • Initialization Omega_0 = arbitrary, unspecified
    Nonconvex optimization is initialization-sensitive; no initialization strategy is described.
assumptions (11)
  • standard math Stone-Weierstrass theorem, Hahn-Banach theorem, Riesz representation theorem
    Used in Appendices B and C to prove universal approximation of GMM-QFs and Proposition 4.
  • standard math Contraction mapping theorem and Banach-Picard iteration
    Used in Section IV-C to obtain uniqueness of fixed points for the Bellman mappings on L2(P).
  • domain assumption Assumption 2: state-action space Z is compact
    Needed for Theorem 3(i) and (ii) and Proposition 4.
  • domain assumption Assumption 10(i): z and z' are identically distributed; Assumption 10(ii): g ∈ L2(P)
    Defines the L2(P) setting and makes the Bellman mappings contractions on L2(P).
  • domain assumption All relevant random variables admit PDFs
    Used in Section IV-C to bridge expectations over the sample space and integrals over Z.
  • domain assumption Assumption 13(i): uniform sup-norm closeness of empirical and ensemble Bellman losses over C(Z)×M
    A strong uniform-convergence condition; used in equations (35a) and (35e).
  • ad hoc to paper Assumption 13(ii): Algorithm 2 returns a near-global minimizer of the empirical loss up to additive Δ
    This is the weakest load-bearing premise; it is assumed, not proven, for a nonconvex finite-budget optimization.
  • ad hoc to paper Assumption 13(iii): for sufficiently large K, the Q_K approximation gap is at most δ_inf
    Assumes the user picks K well; no rate or check is provided.
  • ad hoc to paper Assumption 13(iv): bounded conditional variance around the fixed points
    Imposes low stochasticity near the fixed points; the bound depends on σ_V² without estimation.
  • ad hoc to paper Assumption 13(v): fixed points of successive policies are close in L2(P)
    A smoothness condition on the policy-evaluation sequence; motivated by [15] but not verified.
  • domain assumption Finite discrete action set A and deterministic greedy policy improvement
    Used throughout the policy-improvement step and in Fact 15 for the min Bellman operator.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning." pith.science (2026). https://pith.science/paper/FZFG4CUP

@misc{pith2026251218763,
  author       = {Pith},
  title        = {Pith review of: Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZFG4CUP}},
  note         = {Machine review of arXiv:2512.18763}
}
read the original abstract

Unlike their conventional use as estimators of probability density functions in reinforcement learning (RL), this paper introduces a novel function-approximation role for Gaussian mixture models (GMMs) as direct surrogates for Q-function losses. The proposed estimators, termed GMM-QFs, possess substantial representational capacity, as they are shown to be universal approximators over a broad class of functions. They are further embedded within Bellman residuals, where their learnable parameters--a fixed number of mixing weights, together with Gaussian mean vectors and covariance matrices--are inferred from data via optimization on a product Riemannian manifold. This geometric perspective on the parameter space naturally introduces Riemannian optimization into the policy-evaluation step of standard policy-iteration (PI) frameworks. Moreover, rigorous theoretical analysis establishes performance bounds on Q-function estimation error under the proposed PI scheme. Supporting numerical tests show that GMM-QFs deliver competitive performance and, in some cases, outperform state-of-the-art approaches across a range of benchmark RL tasks, all while maintaining a significantly smaller computational footprint than deep-learning methods.

Figures

Figures reproduced from arXiv: 2512.18763 by the authors.

Figure 1
Figure 1. RL as a sequential decision-making process: at state s, the RL agent takes decision/action a := µ(s), suffers the one-step loss g(s, a), and moves to the next state s ′ according to some transition probability. Function µ(·) denotes the policy or decision-making mechanism. The agent seeks to identify a policy that minimizes the cumulative (long-term) loss—quantified by the Q￾function Q(·)—incurred over its sequence … view at source ↗
Figure 2
Figure 2. Policy iteration consists of two steps: policy evaluation and policy improvement. The “exploration” dataset Dµn [T] is collected on the fly under the current policy µn and is distinct from experience data, which are gathered under previous policies and stored in a replay buffer. The proposed framework (Algorithm 1) relies exclusively on Dµn [T] and does not use any experience data or a replay buffer. of Q-functions … view at source ↗
Figure 3
Figure 3. Gradient of the loss Lˆ µ[T](·) at Ω(j) , with t A j being the (Armijo) step-size. In general, the gradient is first projected onto the tangent space TΩ(j)MK and then retracted back to the manifold MK. In the present case, however, this projection is unnecessary because, as shown in Proposition 7, the computed gradient already lies in the tangent space. {Γ ∈ R Dz×Dz | Γ ⊺ = Γ} =: S Dz , ∀C ∈ S Dz ++. The most popula… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Control tasks considered in Section V. The dynamics of the systems given above are only used for simulation. 0 20 40 60 80 100 50 100 150 200 Number of trails = Index n of PI Steps to goal [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Inverted-pendulum dataset. Curve markers: Algorithm 1 (AffInv) with K = 5: , KLSPI [11]: , OBR [12]: , DQN [18]: , Dueling DDQN [53]: , PPO [54]: , EM-GMMRL [24]: . with T := (number of episodes) × (number of actions) = 20 × 70 = 1400 are collected. KLSPI [11] and OBR …
Figure 6
Figure 6. Figure 6: Mountain-car dataset. Curve markers: GMM-QFs (AffI) with K = 500: , others follow [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Effect of different K in Algorithm 1 for the setting of mountain car. Curve markers (for both AffI and BW) K = 20: , K = 50: , K = 200: . DQN [18]: as a baseline. The larger the K, the richer the hyperparameter space MK becomes and the faster the agent learns through t…
Figure 9
Figure 9. Figure 9: Performance of Algorithm 1, DQN [18], PPO [54] for acrobot dataset against cumulative number of gradients calculated. Markers: “tiny” DQN [57] (2 hidden layers of 16 neurons): , “tiny” PPO (Q-network of 2 16- neuron layers and policy network of 2 8-neuron layers): . Ot…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Hadamard-overparametrized sparse GMM Q-functions learn online off-policy policies that match deep RL with fewer parameters and geometrically interpretable components.

Reference graph

Works this paper leans on

59 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Bertsekas,Reinforcement Learning and Optimal Control

    D. Bertsekas,Reinforcement Learning and Optimal Control. Belmont, MA: Athena Scientific, 2019

    2019

  2. [2]

    R. S. Sutton and A. G. Barto,Reinforcement Learning: An Introduc- tion. Cambridge, MA: The MIT Press, 2018

    2018

  3. [3]

    Q-learning,

    C. Watkins and P. Dayan, “Q-learning,”Mach. Learn., vol. 8, pp. 279– 292, 1992

    1992

  4. [4]

    Convergence results for single-step on-policy reinforcement-learning algorithms,

    S. Singh, T. Jaakkola, M. L. Littman, and C. Szepesvári, “Convergence results for single-step on-policy reinforcement-learning algorithms,” Mach. Learn., vol. 38, no. 3, pp. 287–308, 2000

    2000

  5. [5]

    Kernel-based reinforcement learning,

    D. Ormoneit and ´S. Sen, “Kernel-based reinforcement learning,”Mach. Learn., vol. 49, pp. 161–178, 2002

    2002

  6. [6]

    Kernel-based reinforcement learning in average-cost problems,

    D. Ormoneit and P. Glynn, “Kernel-based reinforcement learning in average-cost problems,”IEEE Trans. Auto. Control, vol. 47, no. 10, pp. 1624–1636, Oct. 2002

    2002

  7. [7]

    Stochastic kernel temporal difference for reinforcement learning,

    J. Bae, L. S. Giraldo, P. Chhatbar, J. Francis, J. Sanchez, and J. Príncipe, “Stochastic kernel temporal difference for reinforcement learning,” inProc. IEEE Mach. Learn. Signal Process., 2011, pp. 1–6. DOI: 10.1109/MLSP.2011.6064634

    arXiv 2011

  8. [8]

    Learning to predict by the methods of temporal differ- ences,

    R. S. Sutton, “Learning to predict by the methods of temporal differ- ences,”Mach. Learn., vol. 3, no. 1, pp. 9–44, 1988.DOI: 10.1023/A: 1022633531479

    doi:10.1023/a: 1988

Show all 59 references
  1. [9]

    Least-squares policy iteration,

    M. G. Lagoudakis and R. Parr, “Least-squares policy iteration,”J. Mach. Learn. Res., vol. 4, pp. 1107–1149, Dec. 2003

    2003

  2. [10]

    Regularized policy iteration with nonparametric function spaces,

    A.-M. Farahmand, M. Ghavamzadeh, C. Szepesvári, and S. Mannor, “Regularized policy iteration with nonparametric function spaces,”J. Mach. Learn. Res., vol. 17, no. 1, pp. 4809–4874, 2016

    2016

  3. [11]

    Kernel-based least squares policy iteration for reinforcement learning,

    X. Xu, D. Hu, and X. Lu, “Kernel-based least squares policy iteration for reinforcement learning,”IEEE Trans. Neural Net., vol. 18, no. 4, pp. 973–992, 2007.DOI: 10.1109/TNN.2007.899161 13

    2007

  4. [12]

    Online Bellman residual and temporal difference algorithms with predictive error guarantees,

    W. Sun and J. A. Bagnell, “Online Bellman residual and temporal difference algorithms with predictive error guarantees,” inProc. Int. Joint Conf. Art. Intel., New York, NY , USA, 2016, pp. 4213–4217

    2016

  5. [13]

    Dynamic selection of p- norm in linear adaptive filtering via online kernel-based reinforcement learning,

    M. Vu, Y . Akiyama, and K. Slavakis, “Dynamic selection of p- norm in linear adaptive filtering via online kernel-based reinforcement learning,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process., Rhodes Island, Greece, 2023, pp. 1–5.DOI: 10.1109/ICASSP49357. 2023.10096825

    2023

  6. [14]

    Proximal Bellman mappings for rein- forcement learning and their application to robust adaptive filtering,

    Y . Akiyama and K. Slavakis, “Proximal Bellman mappings for rein- forcement learning and their application to robust adaptive filtering,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Seoul, Republic of Korea, 2024, pp. 5855–5859.DOI: 10.1109/ICASSP48485.2024. 10446701

    2024

  7. [15]

    Nonparametric Bellman map- pings for reinforcement learning: Application to robust adaptive filter- ing,

    Y . Akiyama, M. Vu, and K. Slavakis, “Nonparametric Bellman map- pings for reinforcement learning: Application to robust adaptive filter- ing,”IEEE Trans. Signal Process., vol. 72, pp. 5644–5658, 2024.DOI: 10.1109/TSP.2024.3505266

    2024

  8. [16]

    Theory of reproducing kernels,

    N. Aronszajn, “Theory of reproducing kernels,”Trans. American Math. Society, vol. 68, no. 3, pp. 337–404, 1950

    1950

  9. [17]

    Schölkopf and A

    B. Schölkopf and A. J. Smola,Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2002

    2002

  10. [18]

    Playing Atari with deep reinforce- ment learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller, “Playing Atari with deep reinforce- ment learning,”CoRR, vol. abs/1312.5602, 2013. arXiv: 1312.5602

    2013 arXiv

  11. [19]

    Deep reinforcement learning with double Q-learning,

    H. v. Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” inAAAI Conference on Artificial Intelligence, Phoenix, Arizona: AAAI Press, 2016, pp. 2094–2100

    2016

  12. [20]

    Reinforcement learning for robots using neural networks,

    L.-J. Lin, “Reinforcement learning for robots using neural networks,” Ph.D. dissertation, Carnegie Mellon University, USA, 1992

    1992

  13. [21]

    Reinforcement learning based on on-line EM algorithm,

    M. Sato and S. Ishii, “Reinforcement learning based on on-line EM algorithm,” inAdvance Neural Info. Process. Systems, vol. 11, Denver, CO, USA: MIT Press, 1998, pp. 1052–1058

    1998

  14. [22]

    Reinforcement learning with Gaussian processes,

    Y . Engel, S. Mannor, and R. Meir, “Reinforcement learning with Gaussian processes,” inProc. Int. Conf. Mach. Learn., Bonn, Germany: Association for Computing Machinery, 2005, pp. 201–208.DOI: 10. 1145/1102351.1102377

    2005

  15. [23]

    Reinforcement learning with a Gaussian mixture model,

    A. Agostini and E. Celaya, “Reinforcement learning with a Gaussian mixture model,” inProc. Int. Joint Conf. Neural Net., Barcelona, Spain, 2010, pp. 1–8.DOI: 10.1109/IJCNN.2010.5596306

    2010

  16. [24]

    Online reinforcement learning using a probability density estimation,

    A. Agostini and E. Celaya, “Online reinforcement learning using a probability density estimation,”Neural Comput., vol. 29, no. 1, pp. 220–246, Jan. 2017.DOI: 10.1162/NECO_a_00906

    2017 doi

  17. [25]

    Distributional deep reinforcement learning with a mixture of Gaussians,

    Y . Choi, K. Lee, and S. Oh, “Distributional deep reinforcement learning with a mixture of Gaussians,” inProc. IEEE Int. Conf. Robot. Auto., Montreal, QC, Canada, 2019, pp. 9791–9797.DOI: 10.1109/ICRA. 2019.8793505

    2019

  18. [26]

    Gaussian mixture models,

    D. Reynolds, “Gaussian mixture models,” inEncyclopedia of Biomet- rics. Boston, MA: Springer US, 2009, pp. 659–663.DOI: 10.1007/978- 0-387-73003-5_196

    2009 doi

  19. [27]

    McLachlan and D

    G. McLachlan and D. Peel,Finite Mixture Models. Wiley, 2000

    2000

  20. [28]

    Maximum likelihood from incomplete data via the EM algorithm,

    A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,”Journal of the Royal Statistical Society: Series B, vol. 39, pp. 1–38, 1977

    1977

  21. [29]

    Unsupervised learning of finite mixture models,

    M. A. T. Figueiredo and A. K. Jain, “Unsupervised learning of finite mixture models,”IEEE Trans. Pat. Ana. Mach. Intel., vol. 24, no. 3, pp. 381–396, 2002.DOI: 10.1109/34.990138

    2002 doi

  22. [30]

    Absil, R

    P.-A. Absil, R. Mahony, and R. Sepulchre,Optimization Algorithms on Matrix Manifolds. Princeton, NJ: Princeton University Press, 2008

    2008

  23. [31]

    Rie- mannian proximal policy optimization,

    S. Wang, B. Zhu, C. Li, M. Wu, J. Zhang, W. Chu, and Y . Qi, “Rie- mannian proximal policy optimization,”Computer and Information Science, vol. 13, no. 3, pp. 1–93, Aug. 2020

    2020

  24. [32]

    Policy gradient methods for reinforcement learning with function approximation,

    R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Advance Neural Info. Process. Systems, vol. 12, Denver, CO, USA: MIT Press, 1999, pp. 1057–1063

    1999

  25. [33]

    Approximation and radial-basis-function networks,

    J. Park and I. W. Sandberg, “Approximation and radial-basis-function networks,”Neural Comput., vol. 5, no. 2, pp. 305–316, 1993.DOI: 10.1162/neco.1993.5.2.305

    1993 doi

  26. [34]

    Riemannian Q-functions for policy iteration in reinforcement learning,

    M. Vu and K. Slavakis, “Riemannian Q-functions for policy iteration in reinforcement learning,” inEuropean Signal Process. Conf., Isola delle Femmine, Palermo, Italy, Sep. 8–12, 2025, pp. 1672–1676

    2025

  27. [35]

    H. H. Bauschke and P. L. Combettes,Convex Analysis and Monotone Operator Theory in Hilbert Spaces. New York, NY: Springer, 2011

    2011

  28. [36]

    Tight performance bounds on greedy policies based on imperfect value functions,

    R. J. Williams and L. C. Baird III, “Tight performance bounds on greedy policies based on imperfect value functions,” inTenth Yale Workshop on Adaptive and Learning Systems, New Haven, CT, USA: Center for Systems Science, Yale University, 1994

    1994

  29. [37]

    A correspondence between Bayesian estimation on stochastic processes and smoothing by splines,

    G. S. Kimeldorf and G. Wahba, “A correspondence between Bayesian estimation on stochastic processes and smoothing by splines,”The Annals of Mathematical Statistics, vol. 41, no. 2, pp. 495–502, 1970. DOI: 10.1214/aoms/1177697089

    1970

  30. [38]

    Universal approximation capability of EBF neural networks with arbitrary activation functions,

    T. Chen and H. Chen, “Universal approximation capability of EBF neural networks with arbitrary activation functions,”Circuits, Systems and Signal Processing, vol. 15, no. 5, pp. 671–683, 1996.DOI: 10. 1007/BF01188988

    1996

  31. [39]

    J. W. Robbin and D. A. Salamon,Introduction to Differential Geom- etry. Berlin: Springer, 2022

    2022

  32. [40]

    Williams,Probability with Martingales

    D. Williams,Probability with Martingales. Cambridge University Press, 1991

    1991

  33. [41]

    R. B. Ash and C. A. Doléans-Dade,Probability and Measure Theory, 2nd ed. Academic Press, 2000

    2000

  34. [42]

    Rudin,Real and Complex Analysis, 3rd ed

    W. Rudin,Real and Complex Analysis, 3rd ed. New York: McGraw- Hill, 1987

    1987

  35. [43]

    Wasserstein Riemannian geometry of Gaussian densities,

    L. Malagò, L. Montrucchio, and G. Pistone, “Wasserstein Riemannian geometry of Gaussian densities,”Information Geometry, vol. 1, no. 2, pp. 137–179, 2018.DOI: 10.1007/s41884-018-0014-4

    2018 doi

  36. [44]

    A sampling approach to finding Lyapunov functions for nonlinear discrete-time systems,

    R. Bobiti and M. Lazar, “A sampling approach to finding Lyapunov functions for nonlinear discrete-time systems,” inEuropean Control Conference (ECC), Aalborg, Denmark, 2016, pp. 561–566.DOI: 10. 1109/ECC.2016.7810344

    2016

  37. [45]

    Boumal,An Introduction to Optimization on Smooth Manifolds

    N. Boumal,An Introduction to Optimization on Smooth Manifolds. USA: Cambridge University Press, 2023

    2023

  38. [46]

    Early stopping—but when?

    L. Prechelt, “Early stopping—but when?” InNeural Networks: Tricks of the Trade: Second Edition. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 53–67.DOI: 10.1007/978-3-642-35289-8_5

    2012 doi

  39. [47]

    Learning near-optimal poli- cies with bellman-residual minimization based fitted policy iteration and a single sample path,

    A. Antos, C. Szepesvári, and R. Munos, “Learning near-optimal poli- cies with bellman-residual minimization based fitted policy iteration and a single sample path,” inLearning Theory, Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 574–588

    2006

  40. [48]

    Deterministic Bellman residual minimization,

    N. J. Ehsan Saleh, “Deterministic Bellman residual minimization,” in Advance Neural Info. Process. Systems, in 2019

    2019

  41. [49]

    Big Omicron and big Omega and big Theta,

    D. E. Knuth, “Big Omicron and big Omega and big Theta,” vol. 8, no. 2, 1976

    1976

  42. [50]

    Reinforcement learning in continuous time and space,

    K. Doya, “Reinforcement learning in continuous time and space,” Neural Comput., vol. 12, no. 1, pp. 219–245, 2000.DOI: 10.1162/ 089976600300015961

    2000

  43. [51]

    Efficient memory-based learning for robot control,

    A. W. Moore, “Efficient memory-based learning for robot control,” University of Cambridge, Tech. Rep., 1990

    1990

  44. [52]

    The swing up control problem for the acrobot,

    M. Spong, “The swing up control problem for the acrobot,”IEEE Control Systems Magazine, vol. 15, no. 1, pp. 49–55, 1995.DOI: 10. 1109/37.341864

    1995

  45. [53]

    Dueling network architectures for deep reinforcement learning,

    Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Freitas, “Dueling network architectures for deep reinforcement learning,” inProc. Int. Conf. Mach. Learn., New York, NY , USA, 2016, pp. 1995–2003

    2016

  46. [54]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”CoRR, vol. abs/1707.06347, 2017. arXiv: 1707.06347

    2017 arXiv

  47. [55]

    Julia: A fresh approach to numerical computing,

    J. Bezanson, A. Edelman, S. Karpinski, and V . B. Shah, “Julia: A fresh approach to numerical computing,”SIAM Review, vol. 59, no. 1, pp. 65–98, 2017.DOI: 10.1137/141000671

    2017 doi

  48. [56]

    Robust reinforcement learning using least squares policy iteration with provable performance guarantees,

    K. P. Badrinath and D. Kalathil, “Robust reinforcement learning using least squares policy iteration with provable performance guarantees,” in Proc. Int. Conf. Mach. Learn., vol. 139, PMLR, Jul. 2021, pp. 511–520

    2021

  49. [57]

    Geron,Tiny-dqn, https://github.com/ageron/tiny-dqn, 2017

    A. Geron,Tiny-dqn, https://github.com/ageron/tiny-dqn, 2017

    2017

  50. [58]

    J. B. Conway,A Course in Functional Analysis, 2nd ed. Springer, 1990

    1990

  51. [59]

    Density in approximation theory,

    A. Pinkus, “Density in approximation theory,”Surveys in Approxima- tion Theory, vol. 1, pp. 1–45, 2005. 1 APPENDIXA PROOF OFTHEOREM1 By the definition ofH pre Ck in (4a),Q k = PLk l=1 αklG(· | mkl,C k), for distinct{m kl}Lk l=1, that is,m kl ̸=m kl′, ∀(l, l′)∈ 1, Lk 2 withl < ...

    2005

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.