Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

Konstantinos Slavakis; Minh Vu

REVIEW 3 major objections 8 minor 57 references

Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

T0 review · 3 major / 8 minor · reviewed 2026-07-30 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sparse Gaussian-mixture Q-functions match deep RL with far fewer parameters, faster gains per transition, and geometric interpretability built in.

desk verdict Clean manifold packaging of sparse GMM critics with real interpretability upside, but the “beat deep RL” claim sits on thin and partly handicapped baselines. read the letter →

arxiv 2607.23474 v1 pith:FBB4MQK4 submitted 2026-07-26 cs.LG

Minh Vu , Konstantinos Slavakis This is my paper

classification cs.LG

keywords reinforcementlearningonlineGaussianmixturemodelRiemannianmanifoldsparsemodelingHadamardoverparametrizationexperiencereplayQ-functionapproximation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that Q-functions for online reinforcement learning can be represented as sparse mixtures of Gaussians whose means and covariances live on a Riemannian manifold, rather than as large neural nets or growing kernel models. By factoring each mixture weight into a product of auxiliary scalars (Hadamard overparametrization) and adding a smooth Frobenius penalty, the method starts from a large pool of components and lets online gradient descent drive uninformative weights to zero without non-smooth norms or a fixed pruning schedule. The same geometry that defines the parameter space also makes the surviving components readable: each center and covariance marks a region of state-action space the agent has learned to care about. On standard control tasks the sparse models match or beat deep Q-networks and PPO while using substantially fewer parameters, improving faster per observed transition, and holding up in low-parameter regimes where sparsified deep nets degrade. A sympathetic reader cares because the approach offers a lightweight, offline-to-online path that couples sample efficiency, parameter efficiency, and intrinsic interpretability in one object.

What carries the argument

S-GMM-QFs via Hadamard overparametrization: each mixture weight is written as a product of J auxiliary factors so a smooth sum-of-Frobenius regularizer can replace non-smooth ℓp penalties, enabling Riemannian gradient descent on the product manifold of Euclidean weight factors, means, and positive-definite covariance matrices inside an online policy-iteration loop with experience replay.

What would settle it

On the same lunar-lander and flappy-bird benchmarks, train S-GMM-QFs from a large initial pool and check whether they still reach or beat DQN/PPO reward curves at equal or lower effective parameter counts, and whether the surviving Gaussian centers still align with the learned policy’s state-visitation density; failure on either check falsifies the central claim.

Watch

Extended reading notes

Core claim

Sparse Gaussian-mixture-model Q-functions (S-GMM-QFs), trained by online off-policy policy iteration with Riemannian Adam on a product manifold and sparsified through Hadamard overparametrization, match or exceed deep RL baselines on standard continuous-state control tasks while using far fewer parameters, improving faster per observed transition, and remaining strong in low-parameter regimes where sparsified deep networks fail; interpretability follows because each surviving component’s mean and covariance explicitly encode its geometric role in state-action space.

Load-bearing premise

That the smooth product-of-factors regularizer, without explicit thresholding or a fixed sparsity schedule, will reliably drive useless mixture weights to practical zero under online Riemannian updates, and that dropping importance-sampling weights will not spoil those updates.

Editorial extensions

If this is right

Online agents can keep a fixed-size parametric Q-model that still adapts complexity by pruning mixture components rather than growing kernels or retraining large nets.
Interpretability of the value function becomes a direct readout of means and covariances, without post-hoc saliency or attribution tools.
In low-parameter budgets, sparse mixture models should retain performance where connection-pruned or dynamically sparse deep critics degrade.
Diversity-based replay and Hadamard sparsification can compound on delayed-reward tasks, while priority-only replay may suffice on dense-reward tasks.
The same Riemannian critic machinery extends naturally toward actor-critic continuous-action settings once a separate policy approximator is added.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If per-component covariance cost truly limits high-dimensional sensors, pairing S-GMM-QFs with a cheap online state encoder is the decisive next experiment, not further hyperparameter sweeps on moderate-D benchmarks.
The environment-dependent gap between priority and diversity replay suggests buffer choice should be treated as part of the model class, not a fixed preprocessing step.
Because sparsity emerges from optimization dynamics rather than a schedule, the method may transfer to other manifold-constrained mixture models outside RL where non-smooth penalties are currently blocked.
Failure modes in which many components collapse to the same region would look like geometric mode collapse and could be diagnosed directly from the means without inspecting gradients.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The actual novelty is the packaging: take their offline GMM-QFs, factor the mixture weights with Hadamard overparametrization so sparsity is smooth, run online off-policy PI with RAdam on the product manifold, and add a couple of replay variants that avoid IS weights. That combination is new relative to [35,36] and to the usual sparse-DQN literature.

What they do well is concrete. Proposition 2 and the appendix give usable Riemannian gradients; the retraction/parallel-transport story is standard but carefully applied. The interpretability figures are the strongest part of the paper—surviving means and covariances really do sit on the task-relevant state regions, which is more than you get from pruning a black-box net. Parameter-count and FLOP plots are honest about the cubic cost in state dimension, and they compare against prune/SET/RigL rather than only dense nets. Ten seeds and IQM-style reporting are fine for this venue class.

Soft spots, in proportion. The comparative claim is weaker than the abstract. Only two tasks; flappy bird is a custom 12-D variant with shaped rewards, and PPO essentially flatlines at every size—which is not how PPO behaves on standard Flappy setups. Calling that a “fundamental mismatch” with single-trajectory online learning is partly true and partly an evaluation choice that handicaps the baseline. Discount factor is missing from the hyperparameter table, which is a reproducibility miss. The geometric argument for dropping IS weights is overstated: IS scales the loss and thus the Riemannian gradient; it does not break retractions. So the buffer study runs under uncorrected sampling bias on a shaky premise. None of that sinks the method, but it does mean “match or exceed deep RL” should be read as “beats these particular deep baselines on these two tasks.”

Who it’s for: people already in function-approximation RL, manifold optimization, or interpretable critics—not a general RL audience. Math and citations look solid; self-cites to the offline precursors are appropriate. I’d send it to referees. I’d bring it to reading group only if someone is actively working on parametric critics or Riemannian RL. I would not cite it myself in the next year unless I needed a sparse geometric Q-model.

Referee Report

3 major / 8 minor

Summary. The manuscript introduces sparse Gaussian-mixture-model Q-functions (S-GMM-QFs) for online reinforcement learning. Mixture weights are overparametrized as a product of J Hadamard factors, turning a non-smooth ell_{2/J} quasi-norm sparsity penalty into a smooth Frobenius-norm regularizer compatible with Riemannian optimization on the product manifold of weight factors, means, and SPD covariances. Learning proceeds via one policy-iteration step per incoming transition, with a Bellman-residual objective, a Riemannian Adam (RAdam) update using parallel transport, and an experience replay buffer for which the authors propose two variants: a frequency-based fair priority decay (replacing PER importance-sampling weights, which the authors argue are incompatible with Riemannian retractions) and an online clustering buffer with within-cluster TD priorities. Experiments on Lunar Lander and a custom Flappy Bird variant report that S-GMM-QFs initialized with K=500 components collapse to a compact set of components, match or exceed DQN/PPO and their sparsified variants with far fewer parameters, improve faster per transition and per FLOP, and yield interpretable component geometry. The gradient derivation is correct, the experimental protocol is reasonably careful (10 seeds, IQM smoothing, baseline sweeps reported in the baseline's favor), but the evaluation is thin (two tasks, one custom with an unreported discount factor), the central comparative claim leans on baselines that behave ano

Significance. If the claims hold, the paper offers a genuinely useful combination for online RL under resource constraints: a fixed-size, interpretable function approximator whose sparse structure is learned rather than scheduled, with competitive sample efficiency at a fraction of deep-network parameter counts. Specific strengths worth crediting: the sparsification is achieved through a smooth objective with no explicit thresholding or sparsity schedule; the Riemannian gradients are derived (Proposition 2) and proved in Appendix A, and the derivations check out; the experimental protocol uses 10 seeds, IQM smoothing, sweeps over K and rho, and strong sparse deep RL baselines (pruning, SET, RigL) benchmarked per Graesser et al.; the FLOP-based analysis in Fig. 5 is an informative addition; and the buffer study in §7.5 reports negative as well as positive results honestly. These are real methodological assets. The main risks are the flawed IS/manifold argument in §6.1, an underspecified task setup (missing discount factor, custom environment), and a comparative claim that currently rests on two tasks with anomalously weak deep RL baselines.

major comments (3)

[§6.1] Section 6.1 (paragraph introducing L^IS): the claim that PER importance-sampling weights 'distort the manifold geometry, compromising Riemannian retractions on M(J)_K' is mathematically incorrect as stated. IS weights are scalar multipliers on per-sample squared residuals; the Riemannian gradient of the weighted loss is well-defined, and retractions (11)-(12), which act on tangent vectors independently of the objective, are untouched. The genuine issue is bias/variance of the resulting gradient estimator under a changed sampling distribution — a standard statistical tradeoff, not a manifold incompatibility. Since this claim motivates the fair-decay mechanism (a named contribution), it needs either a correct formal statement (e.g., interaction of the IS-weighted gradient variance with the scalar second moment sigma_n in (14b)) or empirical evidence: Fig. 11 disables IS across all buffer v
[§7.2, Figs. 4b and 7b; Table 1] The headline comparison 'match or exceed deep RL' rests on two tasks, one of which is a custom Flappy Bird variant (12-D observation, custom reward shaping), and the deep baselines behave anomalously there: PPO shows 'little observable improvement' at every model size (Figs. 4b, 7b) and DQN plateaus far below what published results on standard PLE/Gymnasium Flappy Bird achieve. PPO's failure is partly explained by the single-trajectory protocol (rollout length 2048, Table C.2), but that protocol is a choice of the evaluation, so PPO functions as a structurally handicapped comparator. The authors sweep DQN configurations in the baseline's favor, which is good practice, but with only two environments the blanket claim in the abstract is over-scoped. Request: (i) scope the claim explicitly to the strictly online, single-stream protocol; (ii) add at least one standard, unmodified benchmark (
[§4.2, §7.3 (Figs. 5–6), §7.4 (Fig. 9)] The computational-efficiency claim of Fig. 5 and the interpretability claim of Fig. 9 depend on how 'surviving' components are delimited, but the manuscript never states the criterion: Hadamard overparametrization with the smooth Frobenius regularizer (8) drives weights toward zero asymptotically, not to exact zeros at finite training time, and the paper explicitly contrasts itself with thresholding methods. If the FLOP trajectories in Fig. 5 assume hard exclusion of components, the exclusion rule (threshold? relative magnitude cut?) must be stated, since it is an additional algorithmic step not present in (10) or Algorithm 1; if no exclusion is performed, the FLOP savings attributed to 'annihilation' of components do not follow, because gradient and retraction costs scale with K regardless of weight magnitude. Similarly, the counts in Fig. 9 ('9 Gaussians', etc.) need a stated active-co

minor comments (8)

[Proposition 2(i)] Eq. (13a) omits the sum over t present in its derivation (Appendix A.1); as written it shows a single sample's contribution. Align the proposition with the appendix.
[§5.2, Eq. (14b)] The RAdam second moment sigma_n in (14b) is a scalar based on the gradient norm, unlike coordinate-wise Adam variance; this changes the effective preconditioning. Worth one remark, and a note on whether [43] uses this exact form.
[Notation] Both the discount factor and the PER exponent are denoted with alpha (alpha vs. alpha_PER); consider renaming one to avoid confusion, especially since the discount's value is missing (see major comment 2).
[Figure 6] Fig. 6 legend entries 'Dense-to-sparse DeepRL, 99%' and 'Sparse-training DeepRL, 99%' are not explained in the caption (presumably 99% sparsity); please clarify.
[Throughout] Numerous typographical/grammatical issues, e.g., 'smoothened' (Fig. 4 caption), 'PPO show little observable improvement' (§7.2), 'the sparse solutions are expect to lie' (§4.1), 'an online encoders' (§7.2). A careful proofread is advised.
[Figure 7] Fig. 7 notes 'visible large variance' for S-GMM-QFs on flappy bird but the figure reports only IQM; please add stratified bootstrap confidence intervals, which are standard companions to IQM reporting.
[§7.1] The omission of kernel/distributional baselines on lunar lander is justified by citing prior work on other tasks; a brief quantitative summary of those prior results (one table row) would make the omission self-contained.
[§7.1] No code or environment implementation is referenced; given the custom Flappy Bird variant, public release (or at least a full environment specification, including the discount factor) is important for reproducibility.

Circularity Check

1 steps flagged · score 1.0 of 10

Methods-and-benchmarks paper: objective and reported rewards are not algebraically identical; self-citations supply the dense GMM-QF class but do not force the online sparse results.

self citation load bearing [Section 3.1, Proposition 1; also C1–C2 and refs. [35, 36, 38]]
"It is shown in [36] that, each Ω specifies a single GMM-QFs in Q_K. ... The proof follows verbatim the proof of [36, Theorem 4(iii)]. ... This manuscript also constitutes a substantial extension of the short conference paper [38]."

The dense GMM-QF parameter space and universal-approximation property are imported from the authors’ own offline papers rather than re-derived. This is ordinary cumulative research, not a forced reduction: the online off-policy PI loop, Hadamard sparsification, experience-buffer designs, and empirical comparisons against DeepRL are independent content and are not entailed by those citations.

full rationale

The paper defines S-GMM-QFs via Hadamard factorization (Eqs. 8–10), minimizes a Bellman-residual plus smooth Frobenius regularizer by Riemannian RAdam on the product manifold, and reports cumulative reward on held-out evaluation episodes. That chain is not circular: the training loss is not identical to the test metric, and performance is measured against external DeepRL baselines (DQN, PPO, pruning, SET, RigL) on external control tasks. Self-citations to the authors’ offline GMM-QF papers [35, 36] and the short conference version [38] introduce the dense parametric class and the universal-approximation transfer (Prop. 1), but they do not algebraically force the online off-policy results, the sparsification dynamics, the buffer designs, or the comparative claims. Hadamard overparametrization is taken from external work [45]; Riemannian Adam and the affine-invariant metric are likewise external. No fitted parameter is renamed a prediction, no uniqueness theorem is imported to forbid alternatives, and no ansatz is smuggled in as a first-principles derivation. Score 1 reflects only the ordinary, non-load-bearing dependence on the authors’ prior dense GMM-QF definition.

Assumptions & free parameters 6 free parameters · 6 assumptions · 3 invented entities

The work sits on standard MDP/Bellman theory, Riemannian optimization on SPD manifolds, and the authors' prior GMM-QF function class, plus the external Hadamard-overparametrization lemma. Load-bearing modeling choices include representing Q as unconstrained-weight Gaussians per action, sparsifying only weights, dropping IS weights on the manifold, and a large set of hand-chosen optimizer and buffer hyperparameters. No new physical entities; the invented objects are the S-GMM-QF class and the fair-decay / clustering buffer mechanisms.

free parameters (6)

Number of Gaussian components K = 500 (sparse init); dense sweeps 20–500
Initial pool size; swept {20,50,100,500}; sparse runs start from K=500. Directly controls capacity and FLOPs.
Hadamard factors J = 3
Depth of weight factorization; fixes the implicit quasi-norm exponent p=2/J.
Sparsity regularization coefficient ρ = lunar {0.001–0.05}; flappy {1e-5–1e-2}
Strength of Frobenius penalty on Υ_j; task-specific grids, sharply different between environments.
RAdam learning rate γ and moments β1,β2 = γ=1e-3, β1=0.9, β2=0.999
Manifold optimizer stepsize and exponential averages; standard Adam-like defaults.
Replay buffer B, minibatch T, discount α = B=1e5, T=64 (α conventional in [0,1))
Standard RL hyperparameters controlling data reuse and return horizon.
PER exponent αPER; fair-decay Fs, λ; clustering C, η = αPER=0.6; Fs=20; λ=0.5; C=5; η=0.05
Hand-set buffer sampling knobs that change which transitions enter the BR loss.

assumptions (6)

domain assumption Classical discounted MDP Bellman operator is a contraction for α∈[0,1) and PI targets its fixed point via empirical Bellman residual.
Section 2; standard RL foundation for the loss (6)/(10).
standard math Parameter space MK / M_K^(J) is a Riemannian product manifold (Euclidean × SPD^K) with affine-invariant metric on covariances; retractions and parallel transport as in (11)–(14).
Sections 3.1 and 5; cites Absil et al. and Pennec affine-invariant metric.
domain assumption Hadamard overparametrization with R=∑||Υ_j||_F^2 yields solutions comparable to ||Ξ||_{2/J} for J>2 (Kolb et al. Lemma 10), inducing implicit dense-to-sparse bias under gradient descent.
Section 4.1; external lemma used to justify smooth surrogate instead of non-smooth ℓ_p.
standard math Union over K of GMM-QF classes is dense in L2(S×A) (universal approximation), inherited by S-GMM-QFs because factorization does not change the representable function class.
Proposition 1; proof deferred to prior paper [36].
ad hoc to paper Importance-sampling weights from PER distort Riemannian retractions and must be replaced by priority decay or clustering quotas.
Section 6.1; structural claim specific to this manifold setting, not standard PER practice.
domain assumption Online single-trajectory interaction with finite discrete actions and moderate state dimension is the operating regime; cubic SPD costs are acceptable.
Sections 3.2, 7.2, 8; explicitly limits scope away from pixels/LiDAR.

invented entities (3)

Sparse GMM Q-functions (S-GMM-QFs) via Hadamard weight factorization
purpose: Parametric Q-class that sparsifies mixture weights smoothly while keeping means/covariances on SPD geometry for online PI.
Defined in (9)–(10); main modeling contribution relative to dense GMM-QFs.
Frequency-based fair priority decay for manifold-safe PER
purpose: Rebalance over-sampled transitions without IS loss weights that break retractions.
Equation (16), Section 6.1.
Online clustering diversity replay with within-cluster TD priorities
purpose: Enforce equal sampling quotas across state-action clusters in a single-step stream.
Section 6.2; alternative to EDER/offline k-means.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning." pith.science (2026). https://pith.science/paper/FBB4MQK4

@misc{pith2026260723474,
  author       = {Pith},
  title        = {Pith review of: Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FBB4MQK4}},
  note         = {Machine review of arXiv:2607.23474}
}

read the original abstract

This paper develops an online, off-policy policy-iteration framework for reinforcement learning (RL), based on sparse Gaussian-mixture-model Q-functions (S-GMM-QFs). The framework reconciles streaming, non-stationary data with the Riemannian structure of the parameter space while handling distributional mismatch through experience replay. S-GMM-QFs are introduced via Hadamard overparametrization, enabling interpretable sparsification through smooth regularization that facilitates Riemannian-based optimization. Overparametrization allows the framework to adaptively identify meaningful components from a large initial pool, yielding sparse models where interpretability emerges naturally from geometry: each component's parameters (means and covariances) explicitly encode its geometric role in the ambient state-action space. These geometric roles are learned through online gradient descent on a smooth objective over a (Cartesian-product) Riemannian manifold. Numerical tests demonstrate that S-GMM-QFs match or exceed deep RL methods while using substantially fewer parameters and achieving faster improvement per observed transition. Notably, parameter efficiency and interpretability combine to maintain strong generalization in low-parameter regimes where sparsified deep RL approaches degrade.

Figures

Figures reproduced from arXiv: 2607.23474 by the authors.

**Figure 2.** Vanilla gradient-descent update on Riemannian manifold [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

**Figure 3.** RL benchmarks used in Section 7. lander at the top of the viewport subject to random initial force, and terminate upon landing, crashing, or leaving the observable screen. The Flappy Bird task requires controlling a bird to navigate an endless sequence of pipes under gravity, with the goal of passing through as many gaps as possible without collision or leaving the screen boundaries (see Figure 3b). The state space … view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Performance of dense models. Results are smoothened using moving interquartile mean, window of [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

**Figure 5.** Figure 5: Performance of dense models against cumulative FLOPs along training. S-GMM-QFs: [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

**Figure 6.** Figure 6: Change in number of learnable parameters of S-GMM-QFs and sparse DeepRL (DQN [25, 26], PPO [40]) along [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: Performance of models against parameter counts. Curve markers: Algorithm 1 (dense, [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

**Figure 8.** Figure 8: Distribution of Gaussian components in dense GMM-QFs on lunar lander benchmark, shared across all actions. [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

**Figure 9.** Figure 9: Distribution of Gaussian components in S-GMM-QFs on lunar lander benchmark, sparsified from initial [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]

**Figure 10.** Figure 10: State visitation density and distribution of Gaussian component centers under three conditions of Q-function on lunar [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]

**Figure 11.** Figure 11: Performance of dense GMM-QFs (K = 500) with different choices of experience buffer. Curve markers: uniform (baseline): , fair (proposed, Section 6.1): , PER [29]: , ranked PER [29]: , EDER [52]: , clustering (proposed, Section 6.2): , dense GMM-QFs of K = 100 (fair in…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 2 canonical work pages

[1]

Bertsekas, Reinforcement Learning and Optimal Control

D. Bertsekas, Reinforcement Learning and Optimal Control. Belmont, MA: Athena Scientific, 2019

2019
[2]

R. S. Sutton and A. G. Barto,Reinforcement Learning: An Introduction. Cambridge, MA: The MIT Press, 2018

2018
[3]

Q-learning,

C. Watkins and P. Dayan, “Q-learning,”Machine Learning, vol. 8, pp. 279–292, 1992

1992
[4]

Convergence results for single-step on-policy reinforcement- learning algorithms,

S. Singh, T. Jaakkola, M. L. Littman, and C. Szepesvári, “Convergence results for single-step on-policy reinforcement- learning algorithms,”Machine Learning, vol. 38, no. 3, pp. 287–308, 2000

2000
[5]

Kernel-based reinforcement learning,

D. Ormoneit and Ś. Sen, “Kernel-based reinforcement learning,”Machine Learning, vol. 49, pp. 161–178, 2002

2002
[6]

Kernel-based reinforcement learning in average-cost problems,

D. Ormoneit and P. Glynn, “Kernel-based reinforcement learning in average-cost problems,” IEEE Transactions on Automatic Control, vol. 47, no. 10, pp. 1624–1636, Oct. 2002

2002
[7]

Stochastic kernel temporal difference for reinforcement learning,

J. Bae, L. S. Giraldo, P. Chhatbar, J. Francis, J. Sanchez, and J. Príncipe, “Stochastic kernel temporal difference for reinforcement learning,” in Proceedings of IEEE International Worksop on Machine Learning for Signal Processing, 2011, pp. 1–6.doi: 10.1109/MLSP.2011.6064634

arXiv 2011
[8]

Learning to predict by the methods of temporal differences,

R. S. Sutton, “Learning to predict by the methods of temporal differences,”Machine Learning, vol. 3, no. 1, pp. 9–44,

Show all 57 references

[9]

Kernel-based least squares policy iteration for reinforcement learning,

X. Xu, D. Hu, and X. Lu, “Kernel-based least squares policy iteration for reinforcement learning,”IEEE Transactions on Neural Networks, vol. 18, no. 4, pp. 973–992, 2007.doi: 10.1109/TNN.2007.899161

2007
[10]

Least-squares policy iteration,

M. G. Lagoudakis and R. Parr, “Least-squares policy iteration,”Journal of Machine Learning Research, vol. 4, pp. 1107– 1149, Dec. 2003

2003
[11]

Regularized policy iteration with nonparametric function spaces,

A.-M. Farahmand, M. Ghavamzadeh, C. Szepesvári, and S. Mannor, “Regularized policy iteration with nonparametric function spaces,”Journal of Machine Learning Research, vol. 17, no. 1, pp. 4809–4874, 2016

2016
[12]

Online Bellman residual and temporal difference algorithms with predictive error guarantees,

W. Sun and J. A. Bagnell, “Online Bellman residual and temporal difference algorithms with predictive error guarantees,” in Proceedings of International Joint Conference on Artificial Intelligence, New York, NY, USA, 2016, pp. 4213–4217. 31 Table C.2: Hyperparameters of the De...

2016
[13]

Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learning,

M. Vu, Y. Akiyama, and K. Slavakis, “Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learning,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greece, 2023, pp. 1–5.doi: 1...

2023
[14]

Proximal Bellman mappings for reinforcement learning and their application to robust adaptive filtering,

Y. Akiyama and K. Slavakis, “Proximal Bellman mappings for reinforcement learning and their application to robust adaptive filtering,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, Seoul, Republic of Korea, 2024, pp. 5855–5859.doi: ...

2024
[15]

Nonparametric Bellman mappings for reinforcement learning: Application to robust adaptive filtering,

Y. Akiyama, M. Vu, and K. Slavakis, “Nonparametric Bellman mappings for reinforcement learning: Application to robust adaptive filtering,”IEEE Transactions on Signal Processing, vol. 72, pp. 5644–5658, 2024.doi: 10.1109/TSP. 2024.3505266

2024
[16]

Theory of reproducing kernels,

N. Aronszajn, “Theory of reproducing kernels,” Transactions of the American Mathematical Society, vol. 68, no. 3, pp. 337–404, 1950

1950
[17]

Schölkopf and A

B. Schölkopf and A. J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2002

2002
[18]

Reinforcement learning based on on-line EM algorithm,

M. Sato and S. Ishii, “Reinforcement learning based on on-line EM algorithm,” in Advances in Neural Information Processing Systems, vol. 11, Denver, CO, USA: MIT Press, 1998, pp. 1052–1058

1998
[19]

Online reinforcement learning using a probability density estimation,

A. Agostini and E. Celaya, “Online reinforcement learning using a probability density estimation,”Neural Computation, vol. 29, no. 1, pp. 220–246, Jan. 2017.doi: 10.1162/NECO_a_00906

2017 doi
[20]

Distributional deep reinforcement learning with a mixture of Gaussians,

Y. Choi, K. Lee, and S. Oh, “Distributional deep reinforcement learning with a mixture of Gaussians,” inProceedings of IEEE International Conference Robotics and Automation, Montreal, QC, Canada, 2019, pp. 9791–9797.doi: 10.1109/ ICRA.2019.8793505

2019
[21]

Reinforcement learning with Gaussian processes,

Y. Engel, S. Mannor, and R. Meir, “Reinforcement learning with Gaussian processes,” inProceedings of International Conference on Machine Learning, Bonn, Germany: Association for Computing Machinery, 2005, pp. 201–208. doi: 10.1145/1102351.1102377

2005
[22]

A distributional perspective on reinforcement learning,

M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” inProceedings of International Conference on Machine Learning, Sydney, NSW, Australia, 2017, pp. 449–458

2017
[23]

Distributional reinforcement learning with quantile regression,

W. Dabney, M. Rowland, M. Bellemare, and R. Munos, “Distributional reinforcement learning with quantile regression,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, Apr. 2018.doi: 10.1609/aaai.v32i1.11791

2018 doi
[24]

Gaussian mixture models,

D. Reynolds, “Gaussian mixture models,” in Encyclopedia of Biometrics, Boston, MA: Springer US, 2009, pp. 659–663. doi: 10.1007/978-0-387-73003-5_196

2009 doi
[25]

Playing Atari with deep reinforcement learning,

V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller, “Playing Atari with deep reinforcement learning,”CoRR, vol. abs/1312.5602, 2013. arXiv: 1312.5602

2013 arXiv
[26]

Deep reinforcement learning with double Q-learning,

H. v. Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” inProceedings of the AAAI Conference on Artificial Intelligence, Phoenix, Arizona: AAAI Press, 2016, pp. 2094–2100

2016
[27]

Dueling network architectures for deep reinforcement learning,

Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Freitas, “Dueling network architectures for deep reinforcement learning,” inProceedings of International Conference on Machine Learning, New York, NY, USA, 2016, pp. 1995–2003

2016
[28]

Reinforcement learning for robots using neural networks,

L.-J. Lin, “Reinforcement learning for robots using neural networks,” Ph.D. dissertation, Carnegie Mellon University, USA, 1992

1992
[29]

Prioritized experience replay,

T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” inProceedings of International Con- ference on Learning Representations, vol. 2016, 2016

2016
[30]

The state of sparse training in deep reinforcement learning,

L. Graesser, U. Evci, E. Elsen, and P. S. Castro, “The state of sparse training in deep reinforcement learning,” in Proceedings of International Conference on Machine Learning, vol. 162, PMLR, Jul. 2022, pp. 7766–7792

2022
[31]

EIE:Efficientinferenceengineoncompressed deep neural network,

S.Han,X.Liu,H.Mao,J.Pu,A.Pedram,M.A.Horowitz,andW.J.Dally,“EIE:Efficientinferenceengineoncompressed deep neural network,” inProceedings of International Symposium on Computer Architecture, Seoul, Republic of Korea: IEEE Press, 2016, pp. 243–254.doi: 10.1109/ISCA.2016.30. 33

2016 doi
[32]

Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science,

D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta, “Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science,”Nature Communications, vol. 9, 2017

2017
[33]

Dettmers and L

T. Dettmers and L. Zettlemoyer,Sparse networks from scratch: Faster training without losing performance, 2019. arXiv: 1907.04840 [cs.LG]

2019 arXiv
[34]

Rigging the lottery: Making all tickets winners,

U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen, “Rigging the lottery: Making all tickets winners,” inProceedings of International Conference on Machine Learning, vol. 119, PMLR, Jul. 2020, pp. 2943–2952

2020
[35]

Riemannian Q-functions for policy iteration in reinforcement learning,

M. Vu and K. Slavakis, “Riemannian Q-functions for policy iteration in reinforcement learning,” in European Signal Processing Conference (EUSIPCO), Isola delle Femmine, Palermo, Italy, Sep. 8–12, 2025, pp. 1672–1676

2025
[36]

Gaussian-mixture-model Q-functions for policy iteration in reinforcement learning,

M. Vu and K. Slavakis, “Gaussian-mixture-model Q-functions for policy iteration in reinforcement learning,” 2025. arXiv: 2512.18763 [cs.LG]

2025 arXiv
[37]

Towards continual reinforcement learning: A review and perspectives,

K. Khimya, R. Matthew, R. Irina, and P. Doina, “Towards continual reinforcement learning: A review and perspectives,” Journal of Artificial Intelligence Research, vol. 75, pp. 1401–1476, Dec. 2022.doi: 10.1613/jair.1.13673

2022 doi
[38]

Online reinforcement learning via sparse Gaussian mixture model Q-functions,

M. Vu and K. Slavakis, “Online reinforcement learning via sparse Gaussian mixture model Q-functions,” inEuropean Signal Processing Conference (EUSIPCO), to appear, Bruges, Belgium, Aug. 31–Sep. 4, 2026. arXiv: 2509.14585[cs.LG]

2026
[39]

H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces. New York, NY: Springer, 2011

2011
[40]

Proximal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”CoRR, vol. abs/1707.06347, 2017. arXiv: 1707.06347

2017 arXiv
[41]

J. W. Robbin and D. A. Salamon,Introduction to Differential Geometry. Berlin: Springer, 2022

2022
[42]

Absil, R

P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization Algorithms on Matrix Manifolds. Princeton, NJ: Princeton University Press, 2008

2008
[43]

Riemannian adaptive optimization methods,

G. Becigneul and O.-E. Ganea, “Riemannian adaptive optimization methods,” inProceedings of International Conference on Learning Representations, New Orleans, LA, USA, 2019

2019
[44]

Actor-critic algorithms,

V. Konda and J. Tsitsiklis, “Actor-critic algorithms,” in Advances in Neural Information Processing Systems, vol. 12, Denver, CO, USA: MIT Press, 1999, pp. 1008–1014

1999
[45]

Smoothing the edges: Smooth optimization for sparse regularization using Hadamard overparametrization,

C. Kolb, C. L. Müller, B. Bischl, and D. Rügamer, “Smoothing the edges: Smooth optimization for sparse regularization using Hadamard overparametrization,”Machine Learning, vol. 115, p. 87, 2026

2026
[46]

Loss landscapes and optimization in over-parameterized non-linear systems and neural networks,

C. Liu, L. Zhu, and M. Belkin, “Loss landscapes and optimization in over-parameterized non-linear systems and neural networks,” Applied and Computational Harmonic Analysis, vol. 59, pp. 85–116, 2022, Special Issue on Harmonic Analysis and Machine Learning.doi: https://doi.org/...

2022 doi
[47]

LASSO, fractional norm and structured sparse estimation using a Hadamard product parametrization,

P. D. Hoff, “LASSO, fractional norm and structured sparse estimation using a Hadamard product parametrization,” Computational Statistics & Data Analysis, vol. 115, pp. 186–198, 2017

2017
[48]

The tail-Hadamard product parametrization algorithm for compressed sensing,

G. Li, S. Li, D. Li, and C. Ma, “The tail-Hadamard product parametrization algorithm for compressed sensing,”Signal Processing, vol. 205, p. 108853, 2023

2023
[49]

Spred: Solving L1 penalty with SGD,

L. Ziyin and Z. Wang, “Spred: Solving L1 penalty with SGD,” inProceedings of International Conference on Machine Learning, PMLR, Honolulu, HI, USA, 2023, pp. 43407–43422

2023
[50]

Pennec, S

X. Pennec, S. Sommer, and T. Fletcher,Riemannian Geometric Statistics in Medical Image Analysis. San Diego, CA: Academic Press, 2019

2019
[51]

Hall, Lie Groups, Lie Algebras, and Representations: An Elementary Introduction(Graduate Texts in Mathematics)

B. Hall, Lie Groups, Lie Algebras, and Representations: An Elementary Introduction(Graduate Texts in Mathematics). New York, NY: Springer, 2003

2003
[52]

Efficient diversity-based experience replay for deep reinforcement learning,

K. Zhao, Y. Wang, Y. Chen, Y. Li, L. H. U, and X. Niu, “Efficient diversity-based experience replay for deep reinforcement learning,” inProceedings of International Joint Conference on Artificial Intelligence, ser. IJCAI ’25, Montreal, Canada,
[53]

Determinantal point processes for machine learning,

A. Kulesza and B. Taskar, “Determinantal point processes for machine learning,”Foundations and Trends in Machine Learning, vol. 5, no. 2–3, pp. 123–286, 2012

2012
[54]

Clustering experience replay for the effective exploitation in reinforcement learning,

M. Li, T. Huang, and W. Zhu, “Clustering experience replay for the effective exploitation in reinforcement learning,” Pattern Recognition, vol. 131, p. 108875, 2022.doi: https://doi.org/10.1016/j.patcog.2022.108875

2022
[55]

Explainable reinforcement learning: A survey and comparative review,

S. Milani, N. Topin, M. Veloso, and F. Fang, “Explainable reinforcement learning: A survey and comparative review,” ACM Computing Surveys, vol. 56, no. 7, pp. 1–36, 2024.doi: 10.1145/3616864. 34

2024 doi
[1988]

doi: 10.1023/A:1022633531479

doi
[2025]

doi: 10.24963/ijcai.2025/788

2025 doi

Pith tools

Reviewed July 30, 2026 · model on record in the stance chip above.

[1] [1]

Bertsekas, Reinforcement Learning and Optimal Control

D. Bertsekas, Reinforcement Learning and Optimal Control. Belmont, MA: Athena Scientific, 2019

2019

[2] [2]

R. S. Sutton and A. G. Barto,Reinforcement Learning: An Introduction. Cambridge, MA: The MIT Press, 2018

2018

[3] [3]

Q-learning,

C. Watkins and P. Dayan, “Q-learning,”Machine Learning, vol. 8, pp. 279–292, 1992

1992

[4] [4]

Convergence results for single-step on-policy reinforcement- learning algorithms,

S. Singh, T. Jaakkola, M. L. Littman, and C. Szepesvári, “Convergence results for single-step on-policy reinforcement- learning algorithms,”Machine Learning, vol. 38, no. 3, pp. 287–308, 2000

2000

[5] [5]

Kernel-based reinforcement learning,

D. Ormoneit and Ś. Sen, “Kernel-based reinforcement learning,”Machine Learning, vol. 49, pp. 161–178, 2002

2002

[6] [6]

Kernel-based reinforcement learning in average-cost problems,

D. Ormoneit and P. Glynn, “Kernel-based reinforcement learning in average-cost problems,” IEEE Transactions on Automatic Control, vol. 47, no. 10, pp. 1624–1636, Oct. 2002

2002

[7] [7]

Stochastic kernel temporal difference for reinforcement learning,

J. Bae, L. S. Giraldo, P. Chhatbar, J. Francis, J. Sanchez, and J. Príncipe, “Stochastic kernel temporal difference for reinforcement learning,” in Proceedings of IEEE International Worksop on Machine Learning for Signal Processing, 2011, pp. 1–6.doi: 10.1109/MLSP.2011.6064634

arXiv 2011

[8] [8]

Learning to predict by the methods of temporal differences,

R. S. Sutton, “Learning to predict by the methods of temporal differences,”Machine Learning, vol. 3, no. 1, pp. 9–44,