{"id":"a62d43bd-f64c-4bd0-af1f-323c8be3a91b","arxiv_id":"2607.23474","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Hadamard-overparametrized sparse GMM Q-functions learn online off-policy policies that match deep RL with fewer parameters and geometrically interpretable components.","lead":"The paper builds sparse Gaussian-mixture Q-functions for online reinforcement learning by factorizing mixture weights with a Hadamard product, so sparsity arises from smooth Riemannian optimization. The models match or beat deep RL on two control tasks with far fewer parameters and with mixture components that sit in readable regions of state space.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The headline \"match or exceed deep RL\" rests on two environments — one a nonstandard, custom Flappy Bird variant — where the DQN and especially PPO baselines underperform their published behavior, so the comparison may be measuring weak baselines rather than a genuine method advantage.","rationale":"I partially agree with the reader's weakest assumption: the IS/buffer story is indeed shaky, but for a sharper reason than the one given — the stated geometric justification for dropping IS weights is not valid, while the reader focused on whether the regularizer drives weights to near-zero. My read is that the more load-bearing concern sits one level higher: the comparative claim \"match or exceed deep RL\" depends on baselines that visibly underperform their known behavior on a custom, under-specified environment, with only two tasks and the discount factor unreported. Credit where due: the paper is unusually candid in places (it reports its own fair-buffer scheme failing on flappy bird, reports K=500 dense instability, gives full hyperparameters, FLOP accounting, per-seed aggregation, and best-of-sweep baseline selection), and the mathematics I checked (Appendix A derivations of (13a)–(13c), the affine-invariant metric gradient) is correct. That is why I do not move to REJECT: nothing indicates the S-GMM-QF results themselves are wrong, and the interpretability and parameter-efficiency claims are plausibly real. But the verdict should be CONDITIONAL with the condition re-centered: acceptance should hinge less on buffer/IS tightening (the reader's framing) and more on validating the comparison against independently tuned baselines on standard, fully specified environments. If that validation holds, the contribution is a solid accept; if baselines close the gap, the surviving contribution is the Hadamard sparsification machinery and its interpretability, not superiority over deep RL.","tokens_in":27384,"tokens_out":4557,"duration_ms":29726,"concrete_test":"Re-run the comparison on the paper's exact flappy bird environment plus 2–3 additional standard discrete-action benchmarks (e.g., CartPole, Acrobot, standard PLE FlappyBird) using an independently maintained DQN/PPO stack (e.g., Stable-Baselines3 with published tuned hyperparameters), with α and reward scaling reported. If tuned baselines reach their known-competitive plateaus and the S-GMM-QF gap closes or reverses, the headline downgrades to \"competitive on two custom tasks\"; if the ordering survives, the claim is materially strengthened. As a quick analytical check: implement the IS-weighted loss and verify numerically that RAdam updates remain well-defined — refuting or confirming §6.1's retraction-distortion claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim's comparative pillar (\"match or exceed DQN/PPO with fewer parameters, faster per transition\") is only as strong as the baselines it beats, and the baselines look anomalous. On flappy bird (Section 7.2, Fig. 4b), PPO shows \"little observable improvement\" over 10^6 transitions at every model size tested (Fig. 7b), and DQN plateaus far below S-GMM-QFs. This contradicts well-documented results: PPO and DQN both solve standard PLE/Gymnasium Flappy Bird variants reliably. The paper attributes PPO's failure to a \"fundamental mismatch\" with single-trajectory online training — but that mismatch is built into the evaluation protocol by construction, so PPO functions as a structurally handicapped comparator rather than a fair one. Second, the flappy bird task is itself nonstandard: a custom 12-D observation (three pipes' coordinates), custom reward shaping (0.1 alive, 1 per pipe, −0.5 ceiling), and — notably — the discount factor α is never reported anywhere in Table 1 or the text, so the effective task horizon is unreproducible. Third, the strongest DQN configuration is selected per benchmark post hoc from a sweep (\"best-performing... always reported in favor of the baseline\"), which is good practice, but with only two tasks total and kernel/distributional baselines omitted on lunar lander, the evidentiary base for a blanket \"match or exceed deep RL\" claim is thin. A secondary, internally flawed justification compounds this: Section 6.1's claim that PER importance-sampling weights \"distort the manifold geometry, compromising Riemannian retractions\" is mathematically unsupported — IS weights multiply the loss, hence scale the Riemannian gradient, which remains a valid tangent vector; retractions are unaffected. Dropping IS correction therefore injects un-corrected sampling bias into the BR fit on a false geometric premise, and the buffer study (Fig. 11) compares all variants under this bias. None of this invalidates the clean gradient derivations (Appendix","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The manuscript introduces sparse Gaussian-mixture-model Q-functions (S-GMM-QFs) for online reinforcement learning. Mixture weights are overparametrized as a product of J Hadamard factors, turning a non-smooth ell_{2/J} quasi-norm sparsity penalty into a smooth Frobenius-norm regularizer compatible with Riemannian optimization on the product manifold of weight factors, means, and SPD covariances. Learning proceeds via one policy-iteration step per incoming transition, with a Bellman-residual objective, a Riemannian Adam (RAdam) update using parallel transport, and an experience replay buffer for which the authors propose two variants: a frequency-based fair priority decay (replacing PER importance-sampling weights, which the authors argue are incompatible with Riemannian retractions) and an online clustering buffer with within-cluster TD priorities. Experiments on Lunar Lander and a custom Flappy Bird variant report that S-GMM-QFs initialized with K=500 components collapse to a compact set of components, match or exceed DQN/PPO and their sparsified variants with far fewer parameters, improve faster per transition and per FLOP, and yield interpretable component geometry. The gradient derivation is correct, the experimental protocol is reasonably careful (10 seeds, IQM smoothing, baseline sweeps reported in the baseline's favor), but the evaluation is thin (two tasks, one custom with an unreported discount factor), the central comparative claim leans on baselines that behave ano","tokens_in":27800,"tokens_out":4425,"duration_ms":76699,"significance":"If the claims hold, the paper offers a genuinely useful combination for online RL under resource constraints: a fixed-size, interpretable function approximator whose sparse structure is learned rather than scheduled, with competitive sample efficiency at a fraction of deep-network parameter counts. Specific strengths worth crediting: the sparsification is achieved through a smooth objective with no explicit thresholding or sparsity schedule; the Riemannian gradients are derived (Proposition 2) and proved in Appendix A, and the derivations check out; the experimental protocol uses 10 seeds, IQM smoothing, sweeps over K and rho, and strong sparse deep RL baselines (pruning, SET, RigL) benchmarked per Graesser et al.; the FLOP-based analysis in Fig. 5 is an informative addition; and the buffer study in §7.5 reports negative as well as positive results honestly. These are real methodological assets. The main risks are the flawed IS/manifold argument in §6.1, an underspecified task setup (missing discount factor, custom environment), and a comparative claim that currently rests on two tasks with anomalously weak deep RL baselines.","major_comments":[{"comment":"Section 6.1 (paragraph introducing L^IS): the claim that PER importance-sampling weights 'distort the manifold geometry, compromising Riemannian retractions on M(J)_K' is mathematically incorrect as stated. IS weights are scalar multipliers on per-sample squared residuals; the Riemannian gradient of the weighted loss is well-defined, and retractions (11)-(12), which act on tangent vectors independently of the objective, are untouched. The genuine issue is bias/variance of the resulting gradient estimator under a changed sampling distribution — a standard statistical tradeoff, not a manifold incompatibility. Since this claim motivates the fair-decay mechanism (a named contribution), it needs either a correct formal statement (e.g., interaction of the IS-weighted gradient variance with the scalar second moment sigma_n in (14b)) or empirical evidence: Fig. 11 disables IS across all buffer v","section":"§6.1"},{"comment":"The headline comparison 'match or exceed deep RL' rests on two tasks, one of which is a custom Flappy Bird variant (12-D observation, custom reward shaping), and the deep baselines behave anomalously there: PPO shows 'little observable improvement' at every model size (Figs. 4b, 7b) and DQN plateaus far below what published results on standard PLE/Gymnasium Flappy Bird achieve. PPO's failure is partly explained by the single-trajectory protocol (rollout length 2048, Table C.2), but that protocol is a choice of the evaluation, so PPO functions as a structurally handicapped comparator. The authors sweep DQN configurations in the baseline's favor, which is good practice, but with only two environments the blanket claim in the abstract is over-scoped. Request: (i) scope the claim explicitly to the strictly online, single-stream protocol; (ii) add at least one standard, unmodified benchmark (","section":"§7.2, Figs. 4b and 7b; Table 1"},{"comment":"The computational-efficiency claim of Fig. 5 and the interpretability claim of Fig. 9 depend on how 'surviving' components are delimited, but the manuscript never states the criterion: Hadamard overparametrization with the smooth Frobenius regularizer (8) drives weights toward zero asymptotically, not to exact zeros at finite training time, and the paper explicitly contrasts itself with thresholding methods. If the FLOP trajectories in Fig. 5 assume hard exclusion of components, the exclusion rule (threshold? relative magnitude cut?) must be stated, since it is an additional algorithmic step not present in (10) or Algorithm 1; if no exclusion is performed, the FLOP savings attributed to 'annihilation' of components do not follow, because gradient and retraction costs scale with K regardless of weight magnitude. Similarly, the counts in Fig. 9 ('9 Gaussians', etc.) need a stated active-co","section":"§4.2, §7.3 (Figs. 5–6), §7.4 (Fig. 9)"}],"minor_comments":[{"comment":"Eq. (13a) omits the sum over t present in its derivation (Appendix A.1); as written it shows a single sample's contribution. Align the proposition with the appendix.","section":"Proposition 2(i)"},{"comment":"The RAdam second moment sigma_n in (14b) is a scalar based on the gradient norm, unlike coordinate-wise Adam variance; this changes the effective preconditioning. Worth one remark, and a note on whether [43] uses this exact form.","section":"§5.2, Eq. (14b)"},{"comment":"Both the discount factor and the PER exponent are denoted with alpha (alpha vs. alpha_PER); consider renaming one to avoid confusion, especially since the discount's value is missing (see major comment 2).","section":"Notation"},{"comment":"Fig. 6 legend entries 'Dense-to-sparse DeepRL, 99%' and 'Sparse-training DeepRL, 99%' are not explained in the caption (presumably 99% sparsity); please clarify.","section":"Figure 6"},{"comment":"Numerous typographical/grammatical issues, e.g., 'smoothened' (Fig. 4 caption), 'PPO show little observable improvement' (§7.2), 'the sparse solutions are expect to lie' (§4.1), 'an online encoders' (§7.2). A careful proofread is advised.","section":"Throughout"},{"comment":"Fig. 7 notes 'visible large variance' for S-GMM-QFs on flappy bird but the figure reports only IQM; please add stratified bootstrap confidence intervals, which are standard companions to IQM reporting.","section":"Figure 7"},{"comment":"The omission of kernel/distributional baselines on lunar lander is justified by citing prior work on other tasks; a brief quantitative summary of those prior results (one table row) would make the omission self-contained.","section":"§7.1"},{"comment":"No code or environment implementation is referenced; given the custom Flappy Bird variant, public release (or at least a full environment specification, including the discount factor) is important for reproducibility.","section":"§7.1"}],"recommendation":"major_revision","confidential_remarks":"This is an extension of the authors' own line of work ([35, 36], and conference version [38]); the incremental delta over [38] (buffers, sparsification analysis, flappy bird) appears sufficient for a full paper, but the editor may wish to verify the overlap with the EUSIPCO 2026 version. The empirical base (two tasks, one custom; no code release) is below what the \"match or exceed deep RL\" framing would normally require; I would encourage the editor to ask for a public implementation alongside the requested revisions."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The actual novelty is the packaging: take their offline GMM-QFs, factor the mixture weights with Hadamard overparametrization so sparsity is smooth, run online off-policy PI with RAdam on the product manifold, and add a couple of replay variants that avoid IS weights. That combination is new relative to [35,36] and to the usual sparse-DQN literature.\n\nWhat they do well is concrete. Proposition 2 and the appendix give usable Riemannian gradients; the retraction/parallel-transport story is standard but carefully applied. The interpretability figures are the strongest part of the paper—surviving means and covariances really do sit on the task-relevant state regions, which is more than you get from pruning a black-box net. Parameter-count and FLOP plots are honest about the cubic cost in state dimension, and they compare against prune/SET/RigL rather than only dense nets. Ten seeds and IQM-style reporting are fine for this venue class.\n\nSoft spots, in proportion. The comparative claim is weaker than the abstract. Only two tasks; flappy bird is a custom 12-D variant with shaped rewards, and PPO essentially flatlines at every size—which is not how PPO behaves on standard Flappy setups. Calling that a “fundamental mismatch” with single-trajectory online learning is partly true and partly an evaluation choice that handicaps the baseline. Discount factor is missing from the hyperparameter table, which is a reproducibility miss. The geometric argument for dropping IS weights is overstated: IS scales the loss and thus the Riemannian gradient; it does not break retractions. So the buffer study runs under uncorrected sampling bias on a shaky premise. None of that sinks the method, but it does mean “match or exceed deep RL” should be read as “beats these particular deep baselines on these two tasks.”\n\nWho it’s for: people already in function-approximation RL, manifold optimization, or interpretable critics—not a general RL audience. Math and citations look solid; self-cites to the offline precursors are appropriate. I’d send it to referees. I’d bring it to reading group only if someone is actively working on parametric critics or Riemannian RL. I would not cite it myself in the next year unless I needed a sparse geometric Q-model.","headline":"Clean manifold packaging of sparse GMM critics with real interpretability upside, but the “beat deep RL” claim sits on thin and partly handicapped baselines.","tokens_in":28785,"tokens_out":558,"would_cite":false,"duration_ms":16681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Sparse Gaussian-mixture Q-functions match deep RL with far fewer parameters, faster gains per transition, and geometric interpretability built in.","keywords":["reinforcement learning","online learning","Gaussian mixture model","Riemannian manifold","sparse modeling","Hadamard overparametrization","experience replay","Q-function approximation"],"falsifier":"On the same lunar-lander and flappy-bird benchmarks, train S-GMM-QFs from a large initial pool and check whether they still reach or beat DQN/PPO reward curves at equal or lower effective parameter counts, and whether the surviving Gaussian centers still align with the learned policy’s state-visitation density; failure on either check falsifies the central claim.","tokens_in":28394,"feed_emoji":"🎯","tokens_out":1053,"duration_ms":31648,"temperature":0.7,"pith_summary":"This paper argues that Q-functions for online reinforcement learning can be represented as sparse mixtures of Gaussians whose means and covariances live on a Riemannian manifold, rather than as large neural nets or growing kernel models. By factoring each mixture weight into a product of auxiliary scalars (Hadamard overparametrization) and adding a smooth Frobenius penalty, the method starts from a large pool of components and lets online gradient descent drive uninformative weights to zero without non-smooth norms or a fixed pruning schedule. The same geometry that defines the parameter space also makes the surviving components readable: each center and covariance marks a region of state-action space the agent has learned to care about. On standard control tasks the sparse models match or beat deep Q-networks and PPO while using substantially fewer parameters, improving faster per observed transition, and holding up in low-parameter regimes where sparsified deep nets degrade. A sympathetic reader cares because the approach offers a lightweight, offline-to-online path that couples sample efficiency, parameter efficiency, and intrinsic interpretability in one object.","feed_headline":"Sparse mixture Q-functions match deep RL with fewer parameters","feed_subtitle":"Online Riemannian training prunes Gaussians that mark state regions the agent actually uses.","key_machinery":"S-GMM-QFs via Hadamard overparametrization: each mixture weight is written as a product of J auxiliary factors so a smooth sum-of-Frobenius regularizer can replace non-smooth ℓp penalties, enabling Riemannian gradient descent on the product manifold of Euclidean weight factors, means, and positive-definite covariance matrices inside an online policy-iteration loop with experience replay.","core_discovery":"Sparse Gaussian-mixture-model Q-functions (S-GMM-QFs), trained by online off-policy policy iteration with Riemannian Adam on a product manifold and sparsified through Hadamard overparametrization, match or exceed deep RL baselines on standard continuous-state control tasks while using far fewer parameters, improving faster per observed transition, and remaining strong in low-parameter regimes where sparsified deep networks fail; interpretability follows because each surviving component’s mean and covariance explicitly encode its geometric role in state-action space.","pith_inferences":["If per-component covariance cost truly limits high-dimensional sensors, pairing S-GMM-QFs with a cheap online state encoder is the decisive next experiment, not further hyperparameter sweeps on moderate-D benchmarks.","The environment-dependent gap between priority and diversity replay suggests buffer choice should be treated as part of the model class, not a fixed preprocessing step.","Because sparsity emerges from optimization dynamics rather than a schedule, the method may transfer to other manifold-constrained mixture models outside RL where non-smooth penalties are currently blocked.","Failure modes in which many components collapse to the same region would look like geometric mode collapse and could be diagnosed directly from the means without inspecting gradients."],"forward_implications":["Online agents can keep a fixed-size parametric Q-model that still adapts complexity by pruning mixture components rather than growing kernels or retraining large nets.","Interpretability of the value function becomes a direct readout of means and covariances, without post-hoc saliency or attribution tools.","In low-parameter budgets, sparse mixture models should retain performance where connection-pruned or dynamically sparse deep critics degrade.","Diversity-based replay and Hadamard sparsification can compound on delayed-reward tasks, while priority-only replay may suffice on dense-reward tasks.","The same Riemannian critic machinery extends naturally toward actor-critic continuous-action settings once a separate policy approximator is added."],"fun_headline_variants":["Hadamard-sparsified GMM Q-functions rival deep RL on fewer parameters","Online Riemannian S-GMM-QFs beat deep nets in low-parameter regimes","Sparse Gaussian mixtures match deep RL while encoding state geometry","Overparametrized GMM Q-functions prune to sparse, interpretable policies","Riemannian online training yields sparse Q-mixtures that outpace deep RL"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the smooth product-of-factors regularizer, without explicit thresholding or a fixed sparsity schedule, will reliably drive useless mixture weights to practical zero under online Riemannian updates, and that dropping importance-sampling weights will not spoil those updates.","fun_headline_variants_meta":{"raw":{"variants":["Hadamard-sparsified GMM Q-functions rival deep RL on fewer parameters","Online Riemannian S-GMM-QFs beat deep nets in low-parameter regimes","Sparse Gaussian mixtures match deep RL while encoding state geometry","Overparametrized GMM Q-functions prune to sparse, interpretable policies","Riemannian online training yields sparse Q-mixtures that outpace deep RL"]},"model":"grok-4.5","effort":"low","cost_usd":0.003487,"raw_usage":{"total_tokens":1177,"prompt_tokens":785,"num_sources_used":0,"completion_tokens":103,"cost_in_usd_ticks":34868000,"prompt_tokens_details":{"text_tokens":785,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":289,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":785,"tokens_out":103,"duration_ms":5665,"temperature":1.0,"reasoning_tokens":289,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T21:20:18.956115+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same lunar-lander and flappy-bird benchmarks, train S-GMM-QFs from a large initial pool and check whether they still reach or beat DQN/PPO reward curves at equal or lower effective parameter counts, and whether the surviving Gaussian centers still align with the learned policy’s state-visitation density; failure on either check falsifies the central claim.","supporting_citations":[],"review_version":1}