{"id":"ff66a224-be5c-49ed-afc8-284ed70379af","arxiv_id":"2508.21443","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces a modified geometric mean regularizer for multi-step Q-learning and claims it captures time-average growth, but the key theoretical and empirical supports are weak.","lead":"This paper proposes adding a regularizer based on the N-th root of a sliding-window reward sum to multi-step Q-learning, aiming to favor policies with high long-run performance on individual trajectories. The authors claim the regularizer estimates the time-average growth rate and improves robustness, but the theoretical grounding is flawed and the experiments lack statistical support.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MGM estimator in Eq. (8) is not a valid estimate of the time-average growth rate; it is the N-th root of an additive increment, which for GBM converges to exp(ḡ) or a spurious value, not to the log-growth rate ḡ of Prop. 1, severing the practical algorithm from the theoretical objective.","rationale":"The reader's weakest assumption correctly identifies the missing link between the modified geometric mean and the time-average growth rate. My analysis confirms and sharpens this: Eq. (8) is the N-th root of an additive N-step return, which is not mathematically connected to the logarithmic growth rate of a multiplicative process. For GBM, it asymptotically approaches exp(ḡ) rather than ḡ, so it is not even the same quantity. Additionally, the idealized G∞ in Definition 1 would be state- and action-independent under Assumption 1, making the regularizer incapable of influencing action selection; the only reason the algorithm appears to work is the finite-window heuristic, which is not covered by the paper's theory. These are load-bearing flaws affecting the central claim that the algorithm optimizes long-term performance via the time-average growth rate. The proposed concrete test directly settles whether the estimator tracks the claimed quantity. Thus the reader's REJECT verdict stands.","tokens_in":11355,"tokens_out":6456,"duration_ms":68787,"concrete_test":"Simulate a GBM with known drift μ=0.1 and volatility σ=0.2 (true log-growth rate ḡ=0.08). Generate R_t over a long horizon, compute R̂_t = R_{t+N} − R_t for N ∈ {10, 100, 1000} and Ĝ_t = sgn(R̂_t)|R̂_t|^{1/N}, averaged over many windows. If Ĝ_t converges to 0.08 (or to the Prop. 1 value), the estimator is validated; if it converges to e^{0.08}≈1.083 or to −1 for negative-drift GBM, the estimator is not measuring the time-average growth rate, and the theoretical claim in §IV-B2 fails. Also run the same check for a deterministic multiplicative process R_{t+1}=1.1 R_t to see Ĝ → 1.1, not ln 1.1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical bridge is the claim that the modified geometric mean Ĝ_t = sgn(R̂_t)|R̂_t|^{1/N} with R̂_t = Σ_{i=0}^{N-1} r_{t+i} estimates the time-average growth rate Ḡ∞ defined in Prop. 1 as Σ_i d_i^π(μ_i − σ_i^2/2). This is not supported. For a multiplicative GBM, the per-step increment is r_{t+i} = R_{t+i} − R_{t+i−1}, so R̂_t = R_{t+N} − R_t. The N-th root of this additive difference is not the logarithmic growth rate: if R_{t+N}/R_t ≈ e^{N ḡ}, then |R_{t+N} − R_t|^{1/N} → e^{ḡ} (for ḡ > 0), not ḡ, and for ḡ < 0 the expression tends to −1, not a negative growth rate. Thus Eq. (8) is not estimating the quantity that the Bellman operator in Definition 1 uses as G∞. Moreover, even in the idealized case, under Assumption 1 the time-average growth rate of an irreducible, aperiodic chain is almost surely independent of the initial state-action pair, so G∞(s,a) is constant across actions and cannot affect greedy action selection; the regularization term then cannot change the policy, contradicting the claimed mechanism. The empirical improvements therefore rest entirely on the finite-window heuristic, with no theoretical grounding. This is a load-bearing gap: if Eq. (8) is not an estimator of G∞, the practical algorithm does not optimize the stated objective (3).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reinforcement learning objective that adds a time-average growth-rate regularizer to the conventional expected discounted return. The authors define an N-step regularized Bellman optimality operator for an idealized known growth rate, introduce a 'modified geometric mean' (MGM) estimator over an N-sliding window to estimate this rate from observed rewards, and embed it in a multi-step Q-learning algorithm. They report improved median cumulative rewards on Lunar Lander and Cart-Pole compared with a multi-step Q-learning baseline for several values of the balance parameter λ.","tokens_in":11849,"tokens_out":7370,"duration_ms":82422,"significance":"The motivation is important: in non-ergodic reward dynamics, optimizing the ensemble mean can yield policies that perform poorly on individual trajectories, and a practical regularizer targeting long-run per-trajectory growth would be a useful contribution. The paper also ships code and evaluates on standard benchmarks, which is commendable. However, the central theoretical bridge—the claim that the MGM estimator (Eq. 8) estimates the time-average growth rate G∞ used in the Bellman operator—is not established, and the regularized Bellman operator is defined using the very quantity it is meant to compute. If these gaps cannot be repaired, the algorithm is an unexplained heuristic and the theoretical contribution is unsupported.","major_comments":[{"comment":"The MGM estimator does not estimate the time-average growth rate. With r_t = R_t - R_{t-1}, the N-window sum is R̂_t = R_{t+N} - R_t. The growth rate in Prop. 1 is asymptotically (1/N) log(R_{t+N}/R_t), whereas |R_{t+N} - R_t|^{1/N} is a different functional: for large R_t it behaves like R_t^{1/N}(e^{Ng}-1)^{1/N}, not g, and when R_{t+N} < R_t it returns a large negative value rather than a negative log-ratio. Thus Eq. (8) is not an estimator of G∞, and Algorithm 1 is not optimizing objective (3) in the stated sense.","section":"Sec. IV-B2, Eq. (8)"},{"comment":"The regularized Bellman optimality operator (TG)^N is defined using G∞, 'the time-average growth rate of the best policy'—the object that the operator is supposed to find. In the proof of Prop. 2(1), the authors prove a fixed point for T^π_G with a policy-dependent G^π_∞ and then substitute it into the optimality equation, treating G∞ as a fixed external parameter. The contraction proof likewise treats G∞ as independent of q. As stated, (TG)^N is not a well-defined operator on Q-functions, and the fixed-point and contraction claims do not establish convergence of the proposed regularized Q-learning algorithm.","section":"Definition 1 and Appendix B"},{"comment":"Under Assumption 1 (irreducible and aperiodic chain), the time-average growth rate of a fixed policy is almost surely independent of the initial state-action pair. Hence G^π_∞(s,a) is a constant across (s,a) for each π. In Definition 1, the regularizer (1-γ)λG∞(s,a) then adds the same value to every action at every state and cancels in the max over actions. The theoretical regularizer therefore cannot change the greedy policy in the idealized setting, undermining the claimed mechanism. Any empirical benefit must come from the finite-window heuristic, not from the stated objective.","section":"Sec. III and Definition 1"}],"minor_comments":[{"comment":"The relationship between the continuous-time Δt in Eq. (7) and the discrete window size N in Eq. (6) is never made precise; the paper moves between continuous SDE notation and discrete-time RL without a clear discretization.","section":"Sec. IV-B"},{"comment":"The sentence 'For λ = 1, this operator maps any q to a fixed point, G∞' is unclear: an operator maps a Q-function to a Q-function, not to a scalar. This should be rephrased or removed.","section":"Sec. IV-C1, after Definition 1"},{"comment":"Assumption 2 (existence of an ergodicity transformation) is stated as essential but is never used in any proof or derivation; the paper should either use it or state that it is only motivational.","section":"Sec. III and IV-A"},{"comment":"The experimental comparison reports only medians and density plots without error bars or statistical tests. The claim that the method 'outperforms conventional RL methods' would be strengthened by repeated-seed statistics and significance tests.","section":"Sec. V"}],"recommendation":"reject","confidential_remarks":"The core theoretical claims are not supportable as written: the estimator is disconnected from the growth-rate objective, the Bellman operator is circular, and the irreducibility assumption makes the time-average regularizer constant across actions. These are load-bearing issues, not presentation issues. A future version might salvage a heuristic algorithm, but that would require substantially reframing the paper's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: the paper combines a sensible motivation with a broken theoretical link. The core claim is that a modified geometric mean over an N-step window estimates the time-average growth rate, and that a regularized Bellman operator can optimize it. On reading the math, neither step holds together.\n\nWhat's genuinely new: the specific MGM regularizer embedded into multi-step Q-learning is not in the prior literature. The authors are honest about borrowing the GBM growth-rate result, and the algorithm is simple enough to reproduce — they even provide code. The empirical comparison against a λ=0 baseline shows consistent median improvements on two control tasks, which is something.\n\nThe problems start with Equation (8). The MGM is the N-th root of an additive window sum, but Proposition 1's growth rate is the logarithmic per-period growth of a multiplicative process. For positive growth, the N-th root of (R_{t+N} - R_t) converges to exp(ḡ), not ḡ; for negative growth it goes to -1. So the estimator is not estimating the quantity the theory defines. That severs the practical algorithm from the stated objective.\n\nThen there's the fixed-point issue. Definition 1 bakes in G∞, the optimal growth rate, as a parameter. That makes the optimality equation near-tautological; the proof of Proposition 2 substitutes Gπ and never establishes the fixed point for the actual definition. Under Assumption 1, the growth rate is independent of the initial (s,a) anyway, so even a perfectly known G∞ would be constant across actions and cannot change greedy selection. That undermines the whole mechanism.\n\nThe empirical section is also thin: no error bars, no seeds, and the λ sweep is done on the same test set. The gains are modest and on two basic benchmarks. So the paper doesn't meet the bar for an archival venue as it stands.\n\nThat said, the underlying question — how to optimize per-trajectory long-term performance under non-ergodic reward dynamics — is real and worth attention. The heuristic might work for reasons not captured by the theory. But the current manuscript's central claims don't survive scrutiny.\n\nRecommendation: send to peer review, because the topic is important and the flaws deserve a careful referee report. With the theory properly fixed or repositioned as a heuristic, there might be a useful paper here. For now, though, I'd expect rejection without major revision.","headline":"The paper's heuristic regularizer might help in some tasks, but the theoretical bridge it claims — MGM estimates the time-average growth rate — does not hold up, leaving the central objective unsupported.","tokens_in":12278,"tokens_out":3139,"would_cite":false,"duration_ms":34142,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement learning objective that mixes expected return with a sliding-window geometric mean can improve long-term per-trajectory performance.","keywords":["geometric mean","time-average growth rate","non-ergodic rewards","reinforcement learning","multi-step Q-learning","regularized MDP","sliding window","geometric Brownian motion"],"falsifier":"Run two policies on the multiplicative coin-toss process from the paper, chosen so their true time-average growth rates are rank-ordered and separated. For each policy, collect long reward trajectories, compute the MGM estimator on N-windows for several N, and check whether the estimator orders the policies in the same direction as the true growth rates. If the ordering flips for any N, or if the estimator does not converge to a quantity monotone in the growth rate, the regularizer is optimizing a proxy unrelated to its stated target.","tokens_in":11280,"feed_emoji":"📈","tokens_out":5398,"duration_ms":52878,"temperature":0.7,"pith_summary":"The paper argues that the standard RL objective—expected cumulative reward over many rollouts—can mislead when rewards are non-ergodic, because a policy that looks good on average may fail on most individual trajectories. To fix this, it proposes optimizing a weighted combination of the expected return and the time-average growth rate of a single trajectory, estimating the latter with a modified geometric mean over an N-step sliding window of rewards. The authors derive a regularized Bellman operator for this combined objective and prove that the idealized operator is a contraction whose fixed point is the optimal Q-function. In Lunar Lander and Cart-Pole simulations, the regularized multi-step Q-learning algorithm raises the median and mean cumulative reward compared with standard multi-step Q-learning. A reader would care because this is a drop-in modification to existing Q-learning that targets what an agent actually experiences over its lifetime, not just the ensemble average.","feed_headline":"Geometric-mean regularizer improves long-run RL returns","feed_subtitle":"Mixing expected reward with a sliding-window growth estimate beats multi-step Q-learning in two control benchmarks.","key_machinery":"The engine of the method is a modified geometric mean (MGM) over an N-sliding window of rewards, Ĝ_t = sgn(R̂_t)(|R̂_t|)^{1/N}, with R̂_t the sum of the next N rewards. It is meant to estimate the time-average growth rate of a trajectory without knowing the reward dynamics. This estimator is injected as a regularizer into the multi-step Q-learning target, weighted by λ, and the resulting objective is analyzed through an N-step regularized Bellman optimality operator (TG)^N. The operator, which replaces each per-step reward with (1−λ)r_t + (1−γ)λG∞, is shown to be a γ^N-contraction with fixed point the optimal regularized Q-function; this is what licenses the practical algorithm.","core_discovery":"The central claim is that the time-average growth rate—the long-run growth of rewards along one trajectory—can be estimated by a modified geometric mean of cumulative rewards over a sliding window, and that adding this estimate as a regularizer to the standard RL objective improves long-term policy performance. Concretely, the paper defines Ĝ_t = sgn(R̂_t)(|R̂_t|)^{1/N} with R̂_t = Σ_{i=0}^{N-1} r_{t+i}, embeds λĜ into the multi-step Q-learning target, and shows that under geometric Brownian motion reward increments a stationary Markov chain has a fixed growth rate ḡ = Σ_i d_i^π(μ_i − σ_i²/2). The idealized regularized Bellman operator is proved to be a γ^N-contraction with the optimal Q-","pith_inferences":["A testable extension is to replace the raw sliding-window sum with a discounted window, aligning the estimator's horizon with γ and making it more natural for off-policy corrections.","The same MGM regularizer could be applied to policy-gradient methods as a path-dependent intrinsic reward; the paper only demonstrates Q-learning.","The dynamic-awareness flag e suggests a curriculum—non-overlapping windows during exploration so estimates are consistent with one policy, then overlapping windows as the policy stabilizes—which is plausible but not analyzed.","Because the sign function handles negative rewards by preserving direction, the estimator's scale sensitivity in environments with large negative rewards is worth testing; a scale-invariant normalization may be needed."],"forward_implications":["If the estimator is faithful, setting λ=1 makes the objective directly maximize the invariant growth rate ḡ, connecting RL to ergodicity-theoretic results without requiring an explicit ergodicity transformation.","Because λ=0 recovers standard multi-step Q-learning, the regularizer can be added to existing multi-step return estimators by changing only the target computation.","The γ^N-contraction property means the idealized regularized operator has a unique fixed point and inherits the convergence behavior of standard multi-step Q-learning when the growth rate is known.","The sliding window makes the value update depend on the last N rewards, effectively turning the MDP into a path-dependent process, so the policy can react to recent reward trends rather than only the current state."],"supporting_citations":[{"why":"Provides the motivating non-ergodic reward examples and the ergodicity-transformation perspective the new objective approximates.","marker":"[8]"},{"why":"Supplies the regularized MDP framework into which the geometric mean is inserted as a regularization term.","marker":"[10]"},{"why":"Extends the regularized MDP theory and motivates the regularized modified policy iteration scheme used here.","marker":"[14]"},{"why":"Provides the multi-step Bellman optimality operator whose N-step contraction and update structure the paper adapts.","marker":"[17]"},{"why":"Supplies the multi-step Q-learning update rule that the regularizer is added to.","marker":"[18]"},{"why":"Gives the multi-step DQN family used as the conventional baseline in the Lunar Lander evaluation.","marker":"[19]"},{"why":"Establishes that ergodicity transformations exist for given reward dynamics, an assumption the paper relies on without constructing the transformation.","marker":"[24]"},{"why":"Supports the irreducible and aperiodic Markov chain assumption and the mixing-time stationarity used in the growth-rate derivation.","marker":"[29]"},{"why":"Provides the stable time-average growth rate for geometric Brownian motion used in Proposition 1.","marker":"[31]"}],"fun_headline_variants":["Sliding-window geometric mean boosts RL long-term returns","Time-average growth rate regularizer beats standard RL","RL that optimizes individual trajectory growth, not just average","Modified geometric mean regularizer improves policy performance","Beyond expected value: RL with growth-rate regularizer"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole algorithm rests on the assumption that taking the N-th root of a sliding-window sum of rewards actually measures how fast a trajectory grows over time; if that estimator does not track the growth rate, the regularizer optimizes the wrong quantity.","fun_headline_variants_meta":{"raw":{"variants":["Sliding-window geometric mean boosts RL long-term returns","Time-average growth rate regularizer beats standard RL","RL that optimizes individual trajectory growth, not just average","Modified geometric mean regularizer improves policy performance","Beyond expected value: RL with growth-rate regularizer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1382,"prompt_tokens":781,"completion_tokens":601,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":536}},"tokens_in":525,"tokens_out":601,"duration_ms":6927,"temperature":1.0,"reasoning_tokens":536,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:17:58.766287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run two policies on the multiplicative coin-toss process from the paper, chosen so their true time-average growth rates are rank-ordered and separated. For each policy, collect long reward trajectories, compute the MGM estimator on N-windows for several N, and check whether the estimator orders the policies in the same direction as the true growth rates. If the ordering flips for any N, or if the estimator does not converge to a quantity monotone in the growth rate, the regularizer is optimizing a proxy unrelated to its stated target.","supporting_citations":[{"cited_title":"Reinforcement learning with non-ergodic reward in- crements: robustness via ergodicity transformations,","cited_arxiv_id":null,"evidence_quote":"Provides the motivating non-ergodic reward examples and the ergodicity-transformation perspective the new objective approximates."},{"cited_title":"A theory of regularized Markov decision processes,","cited_arxiv_id":null,"evidence_quote":"Supplies the regularized MDP framework into which the geometric mean is inserted as a regularization term."},{"cited_title":"Leverage the average: an analysis of KL regularization in reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Extends the regularized MDP theory and motivates the regularized modified policy iteration scheme used here."},{"cited_title":"Multi-Bellman operator for convergence of $Q$-learning with linear function approximation","cited_arxiv_id":"2309.16819","evidence_quote":"Provides the multi-step Bellman optimality operator whose N-step contraction and update structure the paper adapts."},{"cited_title":"A novel multi-step q-learning method to improve data efficiency for deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-step Q-learning update rule that the regularizer is added to."},{"cited_title":"Rainbow: Combining improvements in deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Gives the multi-step DQN family used as the conventional baseline in the Lunar Lander evaluation."},{"cited_title":"The time interpretation of expected utility theory","cited_arxiv_id":"1801.03680","evidence_quote":"Establishes that ergodicity transformations exist for given reward dynamics, an assumption the paper relies on without constructing the transformation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the irreducible and aperiodic Markov chain assumption and the mixing-time stationarity used in the growth-rate derivation."},{"cited_title":"The ergodicity solution of the cooperation puzzle,","cited_arxiv_id":null,"evidence_quote":"Provides the stable time-average growth rate for geometric Brownian motion used in Proposition 1."}],"review_version":1}