{"id":"bc87e36d-9fc3-4c9f-bfac-0589658eb45a","arxiv_id":"2412.05906","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper claims entropy regularization forces the optimal LQ feedback policy to be Gaussian and uses that to solve a mean-variance asset-liability problem, but the proof of the main theorem contains a correlation error that invalidates the general claim.","lead":"This paper derives an entropy-regularized optimal feedback policy for a discrete-time linear-quadratic control problem and applies it to a mean-variance asset-liability management model. The central theorem mishandles the correlation between the two noise sources, so the general result as stated does not follow.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 2 drops the noise-correlation ρ from the pure state-noise cross term, so the Riccati recursion in Theorem 1 and the closed forms in Theorem 2 are not established for ρ≠0.","rationale":"Two-step verification confirms the reader's weakest assumption. I recomputed the expectation: with noise vector ((C x + D u) w^x, \\bar C y w^y), the quadratic form's conditional expectation contains p11(Cx+Du)^2 + 2 p12 ρ (Cx+Du)\\bar C y + p22 \\bar C^2 y^2. The paper's Lemma 2 writes the corresponding state term as z' (C 0;0 \\bar C)' P (C 0;0 \\bar C) z, which has cross coefficient 2 p12 C \\bar C without ρ. Because the proof of Theorem 1 is a direct backward induction using this lemma for every step, the Riccati recursion (F_t, P_t) is wrong for ρ≠0 and C,\\bar C≠0. The closed forms in Theorem 2 are algebraic consequences of the same recursion and inherit the error. The Gaussian policy formula also depends on P through H_t except at the final step, so the optimal-control claim of the paper is not proven. The application sets C=0, in which case the erroneous term vanishes, so the mean-variance portion may be salvageable; that does not rescue the general theorem, which is the paper's central contribution. No machine-checked proof or independent numerical validation of the general case is provided, and the numerical section also uses the C=0 setting. Therefore the reader's REJECT verdict is appropriate and the concern is load-bearing.","tokens_in":25752,"tokens_out":8744,"duration_ms":74256,"concrete_test":"Independently re-derive Lemma 2 from the system dynamics (1) while keeping E[w^x_t w^y_t] = ρ. For u = 0, the coefficient of x_t y_t in E[z_{t+1}' P z_{t+1} | F_t] should be 2 p12 (A \\bar A + ρ C \\bar C); the paper's formula gives 2 p12 (A \\bar A + C \\bar C). A direct symbolic or numeric check with scalar A = \\bar A = C = \\bar C = 1, P = [[1,-1],[-1,1]], ρ = 0.5 and u = 0 will expose the mismatch. Then iterate one Riccati step from Q_T to see that P_{T-1} in Theorem 1 differs from the corrected value, confirming that the error propagates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Lemma 2 (Appendix A), the paper states that the conditional expectation E[z_{t+1}' P z_{t+1} | F_t] includes the term z' [(C 0; 0 \\bar C)' P (C 0; 0 \\bar C)] z, with P the symmetric matrix [[p11,p12],[p12,p22]]. Expanding the noise part of (1), ((C x + D u) w^x, \\bar C y w^y)', gives E[noise' P noise] = p11 (C x + D u)^2 + 2 p12 ρ (C x + D u) \\bar C y + p22 \\bar C^2 y^2. Thus the state-state cross term is 2 p12 ρ C \\bar C x y, not 2 p12 C \\bar C x y; the ρ is missing from the off-diagonal of (C 0;0 \\bar C)' P (C 0;0 \\bar C). This matrix defines F_t in the proof of Theorem 1, and hence P_t = F_t - H_t G_t^{-1} H_t' and the claimed value function (5) are wrong when ρ ≠ 0 and C, \\bar C ≠ 0. The control gain and covariance in (6) are derived from H_t and G_t, which are not the direct source of this error, but H_t depends recursively on the already-wrong P_{t+1}, so the optimal Gaussian policy is also not established. The financial application in Section 3 sets C = 0, which removes the missing term; Proposition 1 may survive, but Theorem 1 and Theorem 2 as stated are unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies an entropy-regularized discrete-time linear-quadratic (LQ) control problem with terminal cost, claiming that the optimal value function is quadratic plus log-determinant terms and that the optimal feedback policy is Gaussian. It then applies these results to a discrete-time mean-variance asset-liability management (MV-ALM) model, proposes a policy-iteration reinforcement learning algorithm, and reports simulations. The central evidence is Theorem 1 (general Gaussian optimality), Theorem 2 (closed-form value function for a special terminal matrix), Proposition 1 (the MV-ALM specialization), and Theorem 3 (policy improvement and convergence). The stress-test concern lands: Lemma 2 in Appendix A drops the correlation ρ from the state–state noise cross term, so Theorem 1 and Theorem 2 as stated for general C are not proved. The MV-ALM application sets C=0, which removes the missing term, but the paper's general theoretical claims are unsupported as written. The convergence proof of Theorem 3 is also too sketchy to be verifiable, and the stochastic gradient formulas in Section 4.2 are inconsistent with the parameterized objective.","tokens_in":26128,"tokens_out":11306,"duration_ms":110136,"significance":"If the general theorem were correct, the paper would provide a clean characterization of entropy-regularized discrete-time LQ control and a policy-iteration scheme with a finite-horizon convergence guarantee, with a relevant financial application to asset-liability management. The variational derivation leading to Gaussian policies is a genuine derivation rather than an assumption, and the explicit parameterization of the value function and policy is a useful modeling choice. However, because the main general theorem is invalid as stated, the significance of the paper is not currently realized. The C=0 MV-ALM case may be salvageable and could be a reasonable narrower contribution, but that is not what the paper claims to deliver.","major_comments":[{"comment":"The formula for E[z_{t+1}' P z_{t+1} | F_t] omits the correlation ρ in the state–state noise term. Expanding the noise part ((C x_t + D u_t) w^x_t, \\bar C y_t w^y_t)' P (...) gives p11 (C x_t + D u_t)^2 + 2 p12 ρ (C x_t + D u_t) \\bar C y_t + p22 \\bar C^2 y_t^2, so the off-diagonal entry of the state–state matrix should be ρ C \\bar C p12, not C \\bar C p12. This matrix is exactly F_t in the proof of Theorem 1 and enters the Riccati recursion P_t = F_t - H_t G_t^{-1} H_t'. Therefore the value function (5) and the Gaussian policy (6) are not established when ρ ≠ 0 and C \\bar C ≠ 0. The application in Section 3 sets C = 0, so Proposition 1 may survive, but Theorem 1 and Theorem 2 as stated are unsupported.","section":"Appendix A, Lemma 2"},{"comment":"The convergence proof is not self-contained. The proof introduces objects such as \\bar M_t^{t+1} and M_t^{t+1} without defining their recursions, and it asserts 'we can get' the iterated formulas and 'after deriving T - t iterations' convergence. It is not shown that the variance of the iterated policy coincides with λ/2 G_t^{-1} at j = T - t for every t, nor that the value-function terms telescope exactly as claimed. Thus Theorem 3's policy-improvement and convergence claim is not proved as written.","section":"Appendix E, Theorem 3"},{"comment":"The reported stochastic gradients are not gradients of L(θ). In the parameterization (16), P_{t,22} multiplies l_t^2, but the formulas for ∂L/∂θ3, ∂L/∂θ4, and ∂L/∂θ5 multiply derivatives of P_{t,22} by x_t^2 (and x_{t+1}^2). As written, Algorithm 1's update rules do not follow from the stated objective, so the numerical results do not validate the proposed algorithm as described.","section":"Section 4.2, Eqs. (21)-(23)"}],"minor_comments":[{"comment":"The proofs silently replace (C 0; 0 \\bar C) with (0 0; 0 \\bar C), thereby imposing C = 0; this is the MV-ALM specialization and should be stated explicitly rather than assumed without comment.","section":"Section 4, Lemma 1 and Theorem 3"},{"comment":"There are several typos and reference inconsistencies: 'Sharp Ratio' should be 'Sharpe Ratio'; 'Chang' in the figure captions should be 'Change'; Markowitz is dated 2952 instead of 1952; Wang and Zhou is cited as 2019 in the text but listed as 2020 in the bibliography; Sutton and Barto is cited as 2018 but listed as 1998.","section":"Throughout"},{"comment":"The parameter list includes B = 0.25 and \\bar B = 0.1, but the model uses A, B, C, \\bar A, \\bar C; the mapping from the simulation parameters to the model coefficients should be clarified, especially the roles of \\bar B and the stated excess return.","section":"Section 5"},{"comment":"The parameterization J^θ would be much easier to check if θ1,...,θ5 were defined explicitly in terms of the Proposition 1 quantities; as written the reader must reverse-engineer them from the closed-form value function.","section":"Eq. (16)"},{"comment":"The statement that 'the optimal feedback policy must be Gaussian type' should be qualified: this holds for the entropy-regularized exploratory problem, not for the original unregularized LQ problem.","section":"Abstract and Section 2"}],"recommendation":"reject","confidential_remarks":"The general theorem at the core of the paper is invalid as written because of the missing ρ in Lemma 2, and the convergence proof and algorithm gradients have further gaps. The MV-ALM application with C = 0 might be salvageable, but the manuscript as framed overclaims. If the authors were to resubmit a version restricted to the C = 0 setting with a corrected Lemma 2, a fully rewritten Theorem 3 proof, and corrected gradient formulas, it would need to be evaluated as a substantially new submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The reader's rejection is right, and the stress-test note lands cleanly. In Lemma 2, the pure state-noise cross term expands to p11(Cx+Du)^2 + 2p12 ρ (Cx+Du)\\bar C y + p22 \\bar C^2 y^2, so the off-diagonal of the state-state matrix should carry ρ. The paper writes (C 0;0 \\bar C)' P (C 0;0 \\bar C), which has C\\bar C p12, not ρ C\\bar C p12. That omission propagates into F_t, then P_t, then the value function in Theorem 1 and the closed forms in Theorem 2. So the general C result is not proved. The financial application sets C=0, which kills the offending term; Proposition 1 may well be correct.\n\nCredit where it's due: the liability extension is a real, if narrow, step beyond Wang-Zhou and Cui et al. The closed-form Riccati recursion for the two-state system is new, and the paper attempts a genuine policy iteration proof for the special case. The algorithm structure is sensible. The numerical section is weak — no code, no error bars, no baseline — and the listed parameters include C=0.2, which is inconsistent with the application where C=0. The convergence proof of Theorem 3 is sketchy; it asserts the policy after T−t iterations equals the optimum without a tight argument.\n\nIs the work salvageable? I think yes for the C=0 case. If the authors correct Lemma 2 or restrict the general theorem to the case where the missing term vanishes, the main application could stand. As it is, the central claim is unsupported. I'd send it to a referee because the error is specific and fixable, but the current version should not be accepted.\n\nFor a reading group: not really; the error is instructive but the rest is routine. I wouldn't cite it in the next year.","headline":"The paper has a promising special case, but a missing ρ in Lemma 2 sinks the general theorem.","tokens_in":26625,"tokens_out":5384,"would_cite":false,"duration_ms":47375,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["93E20","49L20","91G10","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Entropy-regularized discrete-time LQ control has a Gaussian optimal policy","keywords":["reinforcement learning","linear-quadratic control","entropy regularization","Gaussian policy","mean-variance asset-liability management","policy iteration","discrete-time stochastic control"],"falsifier":"For a scalar two-state version with $C$, $\\bar C$, and $\\rho$ all nonzero, evaluate $E[(x_{t+1},y_{t+1})' P (x_{t+1},y_{t+1}) \\mid F_t]$ directly and compare the coefficient of the $x_t y_t$ cross term with Lemma 2; a discrepancy in the $\\rho$ term would invalidate Theorem 1's value function and Gaussian policy.","tokens_in":25523,"feed_emoji":"📈","tokens_out":9345,"duration_ms":85171,"temperature":0.7,"pith_summary":"The paper studies a discrete-time linear-quadratic control problem, in a version where model parameters may be unknown and where the controller's reward is augmented by the entropy of the control distribution, so exploration carries an information cost. Its central claim is that in this exploratory problem the optimal feedback policy is Gaussian, with mean linear in the current state and covariance set by the temperature parameter, and that the optimal value function is quadratic in the state plus a deterministic entropy sum. The paper then transfers this result to a discrete-time mean-variance asset-liability management problem, proves a policy-improvement theorem for a policy-iteration scheme, and states that after $T-t$ iterations the scheme reaches the optimal Gaussian policy. A numerical simulation of monthly and daily rebalancing is presented as evidence that the learned returns, Lagrange multiplier, and Bellman error behave as predicted.","feed_headline":"Optimal exploratory LQ control is Gaussian","feed_subtitle":"Policy iteration reaches the optimal mean-variance asset-liability portfolio in T-t steps.","key_machinery":"The Bellman recursion for the exploratory value function is the engine: at each time step one minimizes over densities $\\pi_t(u)$ the expectation of the next value plus $\\lambda \\int \\pi_t \\ln \\pi_t$. The Gaussian density is the exact minimizer of the resulting quadratic-plus-entropy functional, turning the recursion into algebraic updates for the matrices $F_t,H_t,G_t$; the variance $(\\lambda/2)G_t^{-1}$ is the temperature-scaled inverse of the control-cost matrix. The policy-iteration proof reuses this same Gaussian minimization at every step, so each improvement is a closed-form update and convergence after $T-t$ steps follows from the finite horizon.","core_discovery":"Theorem 1 gives the optimal value function as $(x_t,y_t)' P_t (x_t,y_t) + (\\lambda/2) \\sum_{k=t}^{T-1} \\ln[(1/(\\pi\\lambda))^m |G_k|]$ and the optimal feedback density as $\\pi^*_t(u) = N(-G_t^{-1} H_t' (x_t,y_t)', (\\lambda/2) G_t^{-1})$, where $P_t$, $H_t$, and $G_t$ are built recursively from the system matrices and $P_T=Q_T$. Theorem 2 writes these objects in closed form for the terminal matrix $Q_T = \\begin{pmatrix}1&-1\\\\-1&1\\end{pmatrix}$. Theorem 3 asserts that the policy-iteration algorithm, starting from a Gaussian policy, produces non-increasing value functions and converges to the optimal Gaussian policy after $T-t$ updates. The intended contribution is that a financially meaningful mean-variance problem with liabilities, which normally requires a Lagrange multiplier and model parameters, can be solved by an RL procedure whose theoretical guarantees follow from the LQ analysis.","pith_inferences":["The Gaussian-optimality mechanism likely extends to a general symmetric terminal matrix $Q_T$ with the same recursion, although the paper only writes closed forms for the mean-variance terminal matrix.","A testable check is to run the same derivation in the $\\rho=0$ and $\\rho\\neq0$ cases; any mismatch in the noise cross term of Lemma 2 would shift the covariance of the optimal policy.","The same Bellman-plus-entropy machinery could be applied to other constrained portfolio problems, such as index tracking or liability management under a stochastic discount factor.","The paper explicitly leaves nonlinear systems and more general LQ costs to future work, so the Gaussian-policy result should be read as tied to linear dynamics and terminal-only costs."],"forward_implications":["The optimal exploration strategy in this discrete-time LQ model is to add Gaussian noise with covariance proportional to $\\lambda$; no other exploratory distribution improves the entropy-adjusted cost.","For the mean-variance asset-liability problem, the optimal allocation is affine in current wealth and liability, with coefficients expressible from the system parameters and the temperature.","The policy-iteration algorithm improves monotonically and terminates at the exact optimal policy after a number of updates equal to the remaining horizon.","The learned Lagrange multiplier and Bellman error are reported to converge in simulation, so an investor can approach the constrained mean-variance efficient frontier without knowing the model parameters in advance."],"supporting_citations":[{"why":"supplies the analytical multi-period mean-variance solution that the asset-liability model builds on.","marker":"[3]"},{"why":"derives the parameterized mean-variance portfolio solution with liabilities that the paper adapts to the RL setting.","marker":"[4]"},{"why":"provides the discrete-time mean-variance RL framework whose general assumptions this work extends to liabilities.","marker":"[1]"},{"why":"introduces the entropy-regularized reward for continuous-time mean-variance control, the source of the $\\lambda$-entropy term.","marker":"[8]"},{"why":"supplies the policy-evaluation and policy-improvement loop that Theorem 3 formalizes for convergence.","marker":"[7]"},{"why":"formulates continuous-time mean-variance selection as a stochastic LQ problem, motivating the LQ decomposition in Theorem 2.","marker":"[10]"}],"fun_headline_variants":["Gaussian policy is optimal in exploratory LQ control","RL solves mean-variance asset-liability with Gaussian policy","Optimal exploration in LQ is Gaussian, RL converges","Entropy-driven RL yields Gaussian optimal feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proofs rest on Lemma 2's conditional-expectation formula for the quadratic form of the next state; if that formula's treatment of the correlation $\\rho$ between the two noises is not exact, the value recursion and the Gaussian-optimality theorem do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Gaussian policy is optimal in exploratory LQ control","RL solves mean-variance asset-liability with Gaussian policy","Optimal exploration in LQ is Gaussian, RL converges","Entropy-driven RL yields Gaussian optimal feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1206,"prompt_tokens":838,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":454,"tokens_out":368,"duration_ms":4215,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:15:29.955205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a scalar two-state version with $C$, $\\bar C$, and $\\rho$ all nonzero, evaluate $E[(x_{t+1},y_{t+1})' P (x_{t+1},y_{t+1}) \\mid F_t]$ directly and compare the coefficient of the $x_t y_t$ cross term with Lemma 2; a discrepancy in the $\\rho$ term would invalidate Theorem 1's value function and Gaussian policy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the analytical multi-period mean-variance solution that the asset-liability model builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"derives the parameterized mean-variance portfolio solution with liabilities that the paper adapts to the RL setting."},{"cited_title":"Discrete-Time Mean-Variance Strategy Based on Reinforcement Learning","cited_arxiv_id":"2312.15385","evidence_quote":"provides the discrete-time mean-variance RL framework whose general assumptions this work extends to liabilities."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"introduces the entropy-regularized reward for continuous-time mean-variance control, the source of the $\\lambda$-entropy term."},{"cited_title":"Barto, A.G","cited_arxiv_id":null,"evidence_quote":"supplies the policy-evaluation and policy-improvement loop that Theorem 3 formalizes for convergence."},{"cited_title":"xt+1 yt+1 !′ P xt+1 yt+1 ! Ft # = xt yt !′","cited_arxiv_id":null,"evidence_quote":"formulates continuous-time mean-variance selection as a stochastic LQ problem, motivating the LQ decomposition in Theorem 2."}],"review_version":1}