{"id":"ba51e225-50be-421b-8ade-6224cc87aed2","arxiv_id":"2602.01460","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"REINFORCE's gradient-estimator noise-to-signal ratio is exactly computable for linear and polynomial systems, and it typically blows up as policies approach deterministic optima.","lead":"The paper derives exact formulas for how noisy the REINFORCE policy-gradient estimate is relative to the true gradient in finite-horizon linear and polynomial control problems, and shows this noise-to-signal ratio grows sharply near optimal policies. This supplies a concrete mathematical mechanism for why policy-gradient training can become unstable or collapse, and a benchmark for variance-reduction design.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exponential-NSR claim for unstable LQG is unsupported: Theorem 7 bounds variance only, and the gradient denominator may also grow with T.","rationale":"The exact Gaussian-moment derivations for LQG (Theorems 3 and 5) and the polynomial extension (Proposition 9) appear correct: the score identities, lifted-state decomposition, and Isserlis-based parity bookkeeping are standard and internally consistent. I found no algebraic error in the one-step or multi-step variance expansions. The reader's weakest-assumption focus on centered Gaussianity is valid but is a stated restriction, not a hidden flaw. The more consequential gap is the unsupported inference from exponential variance growth to exponential NSR growth in unstable systems. Since NSR is a ratio, the denominator's behavior is essential, and the paper never bounds ||∇J||^2 from above nor the variance from below in T. This directly affects the abstract's 'can grow exponentially in horizon' claim, which is a load-bearing part of the paper's narrative. The proposed numerical test can settle whether the claim is true; if it passes, the paper's conclusion is correct but still needs a rigorous proof of the denominator scaling. Because the central exact-computation theorems remain sound and the concern is about a derived implication, the verdict should stay CONDITIONAL rather than shift to REJECT: the authors should add the missing denominator analysis or soften the exponential-NSR claim, and release the experiment code/error bars.","tokens_in":34378,"tokens_out":19704,"duration_ms":197819,"concrete_test":"For a scalar LQG with unstable closed-loop F (e.g., A=1.2, B=1, Qs=Qa=1, K=0, Σ=0.1, Σ0=1, γ=1), evaluate the exact NSR from Theorem 5's formulas for T=5,10,...,30. Plot log NSR vs T, together with log Var_Fro(bG) and log ||∇J||^2. If NSR does not grow at least exponentially while variance does, the exponential-NSR claim fails; if it does grow exponentially, the conclusion is empirically supported but still requires an analytic bound on the denominator. Repeat with a neutrally stable F as a control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's abstract and contribution list assert that NSR grows exponentially with horizon for unstable closed-loop dynamics, but the supplied proof chain does not establish this. Theorem 7 upper-bounds Var_Fro(bG) and Theorem 8 bounds ||F_S||^2, yet NSR is Var_Fro(bG)/||∇J||^2. For an unstable closed-loop F, the cost, state covariance P_t, and cost-to-go Λ_t all grow exponentially in T, and the exact gradient formulas (15)-(16) depend on P_t and Λ_t; hence the denominator ||∇J||^2 can also grow exponentially. The variance bound in (17) contains a factor ||F_S||^2 times E[R^2], so it grows exponentially, but without a lower bound on the variance and an upper bound on the gradient this does not imply exponential NSR. The Θ analysis in §2.2 suppresses all T- and F-dependence, so it cannot justify the horizon-growth statement. This is a gap in a headline claim, not in the exact-moment theorems themselves, which appear sound under the stated centered-Gaussian assumptions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the variance and noise-to-signal ratio (NSR) of the REINFORCE policy-gradient estimator in finite-horizon control problems. For linear-quadratic systems with linear-Gaussian policies, it derives exact expressions for the mean and second moment of the estimator via Gaussian Isserlis/Wick identities and a lifted-state representation (Theorems 3 and 5), with mean-gradient formulas in Eqs. (15)-(16). It extends exact Gaussian-moment evaluation to polynomial dynamics with polynomial feedback (Proposition 9) and gives variance upper bounds for general nonlinear systems (Theorem 10). Numerical experiments on LQG, polynomial, and MLP-policy systems show NSR increasing near optima and occasional policy collapse. The abstract and contributions also claim that NSR grows exponentially with horizon for unstable closed-loop dynamics.","tokens_in":34608,"tokens_out":10133,"duration_ms":102309,"significance":"If the exact-moment theorems hold, this is a useful contribution to policy-gradient variance analysis: it provides non-asymptotic, parameter-free characterizations (no fitted constants) for LQG and polynomial systems, and it identifies the initial-state covariance/policy-covariance scaling. The explicit gradient formulas and the lifted decomposition are valuable. The practical claim that NSR is highly non-uniform and worsens near optima is supported by the exact formulas in the examples. However, the exponential-horizon NSR claim is not established by the supplied bounds, and this weakens the paper's headline.","major_comments":[{"comment":"The contribution list and Section 5 state that 'we prove the NSR can grow exponentially with the horizon when the closed-loop dynamics are unstable.' This is not supported by the proof chain. Theorem 7 gives upper bounds on Var_Fro(bG_K) and Var_Fro(bG_l); Theorem 8 bounds ||F_S||. NSR is Var_Fro(bG)/||∇J||^2. For ρ(F)>1, the true gradient in Eq. (15) contains P_t and Λ_{t+1}, both of which grow exponentially with T, so the denominator also grows exponentially. The Θ analysis in §2.2 absorbs all T- and F-dependence, so it cannot justify the horizon-growth statement. Figure 3 plots variance, not NSR. To retain the claim, the paper needs a lower bound on the variance and an upper bound on ||∇J||^2 in the same regime, or the claim should be downgraded to variance growth.","section":"§2.2, Theorems 7–8"},{"comment":"The scaling statement '∥E[bGK]∥^2_F, ∥E[bGℓ]∥^2_2 = Θ((σ^2+σ0^2)^2) if they are nonzero' is not correct for the log-std gradient: Eq. (16) gives ∇ℓJ = -2 diag(Σ Σ γ^t(...)), whose squared norm is Θ(σ^4), independent of σ0. The subsequent NSR bound uses a denominator that omits this separate σ^4 contribution. The α→∞ conclusion may survive, but the Θ justification as written is wrong and needs to be rewritten. Moreover, non-degeneracy conditions for the gradients being nonzero are not stated, and NSR is undefined at the deterministic optimum (Corollary 6).","section":"§2.2, Analysis of NSR after Theorem 7"},{"comment":"Figure 3 and the sentence 'This verifies the prediction of Theorem 7 and 8. Thus, in long-horizon tasks, closed-loop instability can dramatically worsen the NSR' conflate variance with NSR. The figure shows true variance growth with T; it does not show NSR. If the gradient also grows exponentially, the figure is consistent with NSR remaining bounded. The authors should either plot NSR for these systems using the exact Theorem 5 procedure, or change the interpretation to variance growth.","section":"§2.2, Figure 3"}],"minor_comments":[{"comment":"The theorem statement refers to equations in the proof for the exact decompositions. For a theorem claiming exact computation, these formulas should be restated as a standalone algorithm or proposition so the reader can verify them without reading the proof.","section":"Theorem 5"},{"comment":"The phrase 'where C is a constant' is unnecessarily vague; the proof gives C = 2T^2 sqrt(m(m+14)). State this explicitly.","section":"Theorem 10(ii)"},{"comment":"The proof bounds E[||bG||^2], not the variance itself. The inequality chain Var ≤ E[||bG||^2] is valid, but the theorem should make clear that it is a second-moment bound, which is why the derived NSR bound is only an upper bound.","section":"Theorem 7"},{"comment":"Notation is occasionally overloaded, e.g., the block covariance is written as Σ := I_T ⊗ Σ while Σ also denotes the per-step policy covariance. A bar or subscript would improve clarity. There are also minor typos such as 'F+S ,∈R^{nT×n}' in Theorem 4.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The exact-moment results appear sound and are a strong contribution. The main obstacle is the unsupported exponential-NSR claim; if scaled back to variance growth, the paper would be suitable. The §2.2 Θ analysis also contains a misstatement about the ℓ-gradient scaling. Please ask the authors to correct these before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is genuinely new: an exact finite-horizon computation of the REINFORCE estimator's variance and NSR for LQG and for polynomial systems with Gaussian policies, via lifting plus Gaussian moment evaluation. The derivations look careful and internally consistent; the parity bookkeeping in the multi-step decomposition checks out, and the mean-gradient formulas (15)-(16) are cleanly derived. This is a real advance over the prior O(σ^-2) bounds in Zhao et al. and the LQG upper bounds in Preiss et al. The polynomial extension and the nonlinear upper bound are sensible. Credit where due: the core moment machinery is serious and appears sound under the stated centered-Gaussian assumptions.\n\nThe soft spots are in the claims, not the core machinery. The stress-test note is right: the paper asserts in the abstract and contributions that NSR grows exponentially with horizon for unstable closed-loop systems, but Theorem 7 bounds variance, Theorem 8 bounds a state-map norm, and neither bounds the denominator ||∇J||^2. For unstable F, the exact gradient formulas themselves involve P_t and Λ_t that grow exponentially, so the ratio's behavior is not established. Figure 3 shows variance growth, not NSR growth. That is a gap in a headline claim, and the paper should either prove the NSR statement or reframe it as a variance-scaling result. Minor related issue: the Θ analysis in §2.2 is fine for the σ0/σ scaling but suppresses T and F dependence, so it cannot support the horizon claim either.\n\nThe other soft spots are proportionally mild. The empirical sections are illustrative: no code, no error bars, and the 'policy collapse' language outruns the evidence. The centered-Gaussian and nonzero-gradient assumptions are restrictive but stated, and the deterministic-optimum degeneracy is handled by defining NSR only where the gradient is nonzero—that's fair, though the 'blow-up near optimality' statements are about approach, not attainment.\n\nWho should read this: anyone working on policy-gradient variance, exploration-exploitation tradeoffs, or LQG policy optimization. It deserves a serious referee, but the referee should push on the horizon-scaling claim and request code and error bars. I'd recommend peer review with major revision, and I'd cite it for the exact moment evaluation even while flagging the overclaim.","headline":"The exact variance/NSR computation for LQG and polynomial systems is real and likely correct, but the abstract's exponential-in-horizon NSR claim is not proven by the supplied theorems.","tokens_in":35121,"tokens_out":2125,"would_cite":true,"duration_ms":26685,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The noise-to-signal ratio of the REINFORCE gradient estimator is exactly computable for LQG and polynomial systems, and it blows up as policies approach optimality.","keywords":["REINFORCE","policy gradient","noise-to-signal ratio","variance","linear-quadratic regulator","Gaussian policy","Isserlis theorem","exact Gaussian moments"],"falsifier":"Run the one-step double-integrator experiment with sigma_0 >> sigma and measure NSR by Monte Carlo for several ratios; Eq. (8) predicts linear growth in sigma_0^2/sigma^2, and any statistically significant deviation from the closed-form value would falsify the exact-moment machinery.","tokens_in":34261,"feed_emoji":"📈","tokens_out":7182,"duration_ms":69453,"temperature":0.7,"pith_summary":"The paper aims to show that the noise-to-signal ratio (NSR) of the REINFORCE policy-gradient estimator—the estimator's variance divided by the squared norm of the true gradient—is not a benign constant but varies sharply across policy parameters and typically worsens as training approaches an optimum. For finite-horizon linear-quadratic systems with Gaussian linear policies, and for polynomial systems with polynomial feedback, the paper proves the NSR can be computed exactly, either in closed form or through an explicit Gaussian-moment algorithm, with no approximation. For general nonlinear dynamics it gives upper bounds on the estimator variance. The exact formulas reveal scaling laws: the NSR grows as the policy becomes more deterministic and as the initial-state distribution broadens, and it can grow exponentially with horizon when the closed-loop dynamics are unstable. Because the optimal policy is deterministic, REINFORCE-style updates become increasingly noisy exactly where they need to converge, offering a concrete mechanism for training slowdowns, oscillations, and policy collapse.","feed_headline":"REINFORCE gradient noise blows up as a policy nears its optimum","feed_subtitle":"New exact formulas for LQG and polynomial systems show why stochastic gradient updates become unreliable near optimality.","key_machinery":"The load-bearing mechanism is exact Gaussian-moment evaluation: for centered Gaussian vectors, moments of products of quadratic forms are computed by Wick/Isserlis contractions, captured in the paper's IS_Omega shorthand (Lemma 2). For multi-step LQG, Theorem 4 lifts the T-step system into a single-step system with block lower-triangular matrices, so the return becomes one quadratic form in the initial state and one in the stacked exploration noise; parity bookkeeping (only even-degree terms survive in centered Gaussian expectations) reduces the variance expansion to sums of trace terms. For polynomial systems, Proposition 9 shows that the gradient estimator is itself a polynomial in Gaussia","core_discovery":"The paper's central claim is that the noise-to-signal ratio (NSR) of the REINFORCE estimator is exactly characterizable for two structured classes of control problems, and that this exactness exposes a non-uniform, often diverging NSR landscape. For LQG systems with Gaussian linear policies, Theorems 3 and 5 express the mean gradient and the estimator's second moment as explicit Gaussian-moment functions (Wick/Isserlis); for polynomial dynamics with polynomial feedback, Proposition 9 shows the estimator is a polynomial in Gaussian variables, so all needed moments are exactly computable. In both settings the NSR is obtained without Monte Carlo approximation, scaling inversely with policy cova","pith_inferences":["A testable extension the paper leaves implicit: because the lifted dynamics reduce multi-step LQG to one-step form, the same exact-moment construction should apply to REINFORCE with a state-dependent baseline, subtracting a function of state and changing only the polynomial degree of the estimator.","The NSR formulas amount to a deterministic diagnostic: monitoring the exact expression along a run could decide when to lower the learning rate or raise exploration covariance, a use the paper does not explore.","For neural policies, Theorem 10 suggests that penalizing the Jacobian norm of the policy mean (or otherwise limiting sensitivity) would directly shrink the variance bound; this is not tested here."],"forward_implications":["NSR diverges as the policy covariance goes to zero, so any REINFORCE-style update faces an exploration–exploitation tension near deterministic optimal policies.","For unstable closed-loop dynamics, estimator variance can grow exponentially with horizon, making long-horizon tasks with unstable controllers especially noisy.","The exact formulas allow computing NSR along an optimization path without Monte Carlo rollouts, turning a statistical diagnostic into a deterministic one.","The upper bounds for nonlinear systems show variance scales with the policy Jacobian, so policies with sharp state-to-action changes can have worse gradient estimates.","Gradient descent on the deterministic objective converges, while SGD and Adam with the same policy can oscillate or collapse once the NSR is large."],"fun_headline_variants":["Exact NSR math exposes unstable REINFORCE near optima","REINFORCE's noise-to-signal spikes at optimal policies","Non-uniform NSR: why REINFORCE becomes unreliable","Formula shows REINFORCE variance diverges near optimum","Exact NSR for LQG and polynomial systems shows blowup near optima"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The exact formulas require the initial state and exploration noise to be centered Gaussian variables; if either is non-Gaussian or has a nonzero mean, or if the true gradient vanishes, the exact NSR identities do not apply.","fun_headline_variants_meta":{"raw":{"variants":["Exact NSR math exposes unstable REINFORCE near optima","REINFORCE's noise-to-signal spikes at optimal policies","Non-uniform NSR: why REINFORCE becomes unreliable","Formula shows REINFORCE variance diverges near optimum","Exact NSR for LQG and polynomial systems shows blowup near optima"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000661,"raw_usage":{"total_tokens":2855,"prompt_tokens":739,"completion_tokens":2116,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":2024}},"tokens_in":483,"tokens_out":2116,"duration_ms":15610,"temperature":1.0,"reasoning_tokens":2024,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:38:45.981855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the one-step double-integrator experiment with sigma_0 >> sigma and measure NSR by Monte Carlo for several ratios; Eq. (8) predicts linear growth in sigma_0^2/sigma^2, and any statistically significant deviation from the closed-form value would falsify the exact-moment machinery.","supporting_citations":[],"review_version":1}