{"id":"efd64334-c2f5-45f1-8615-a55232f8a623","arxiv_id":"2507.20580","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"PFDeePO modifies the DeePO adaptive LQR algorithm by pausing updates near equilibrium and applying random multiplicative gain scaling, removing the need for probing noise and eliminating state perturbations in simulation.","lead":"A modified version of an adaptive, data-driven controller is proposed that avoids adding probing noise to the control signal, which normally causes unwanted state jitter. The new method pauses controller updates near the equilibrium and randomly scales the control gain when the controller has converged, and the authors show in a simulation that this removes the jitter while keeping the system stable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's full-rank proof does not establish σ(Φ_i)>0, and the pre-convergence branch of Algorithm 2 can drive σ(Φ_i) toward zero before K_i converges, so PFDeePO's central claim is unproven.","rationale":"The reader's weakest_assumption is essentially the same: the pre-convergence phase of Algorithm 2 uses unexcited state-feedback data, so the DeePO update's well-posedness and progress are not guaranteed and σ(Φ) can degrade. My stress-test adds a precise reason the proof fails: Theorem 1's coefficient-wise inference is only valid if the sampled states span R^n, and that is never shown and is implausible for a stable open-loop plant. This is load-bearing because the paper's abstract and Theorem 1 promise that full rank is ensured; without it, Φ^{-1} in the update can become ill-conditioned and the gain may freeze at a non-optimal value. The missing convergence proof for the adaptive phase and the disconnected stability theorem (standard LQR gain vs. discounted DARE) reinforce the REJECT verdict. A simulation without the k = 15 disturbance is the natural decisive check, since the current experiment relies on that disturbance to keep the state away from equilibrium during learning.","tokens_in":11451,"tokens_out":9121,"duration_ms":107192,"concrete_test":"Re-run the paper's simulation with the same A, B, γ = 0.1, δ = 0.1, η = 1e-4, and v-interval [0.5, 1.5], but remove the injected disturbance at k = 15. Log σ(Φ_i) and ‖K_i − K*‖ at each step. If the pause rule triggers before K_i has converged (‖K_i − K*‖ not small), or if σ(Φ_i) decays below about 1e-6 during the first branch, then PFDeePO's full-rank and perturbation-free convergence claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that PFDeePO preserves full rank of Φ rests on Theorem 1, but the proof's key inference is invalid. From a row dependence ∑_i λ_i X0_i = ∑_j μ_j U0_j, substituting U0_{j,k} = ∑_i K^{(k)}_{j,i} x^{(k)}_i yields only (λ − K_k^T μ)^T x_k = 0 for each column k. The text then concludes λ_i = ∑_j μ_j K^{(k)}_{j,i} for all i,k, which requires the state vectors x_k to span R^n. No such spanning condition is stated or proven; for a stable open-loop plant the states typically decay and may lie close to a low-dimensional subspace, so a time-varying K_k does not preclude row dependence. Thus the contradiction in Theorem 1 does not follow. Independently, Lemma 4 shows that appending rank-deficient state-feedback data makes σ(Φ) tend to 0; Algorithm 2's first branch appends exactly such data whenever ‖ΔK‖ > δ and ‖x_i‖ > γ, and no lower bound on σ(Φ_i) is supplied. The pause rule does not fix this: for a stable plant, ‖x_i‖ ≤ γ can be reached with K_i still far from the LQR gain, freezing updates and stalling learning. The simulation masks this by injecting a disturbance at k = 15, and Theorem 2 further conflates the standard LQR DARE gain with the discounted-gain H from Eq. (31).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PFDeePO, a modified version of the data-enabled policy optimization (DeePO) algorithm for adaptive LQR, with the goal of avoiding the state perturbations caused by probing noise. The algorithm combines a pause rule that stops gain updates when the state is near equilibrium with a multiplicative random gain applied once the controller has approximately converged. The paper claims two main theoretical results: Theorem 1 states that PFDeePO preserves the full rank of the data matrix Φ, and Theorem 2 establishes exponential stability of the closed loop under the random scaling. The claims are supported by a numerical example comparing PFDeePO with DeePO with and without probing noise.","tokens_in":11858,"tokens_out":6407,"duration_ms":68873,"significance":"If the central claims were established, the contribution would be practically useful: it would let DeePO-style adaptive LQR run without persistent input excitation after convergence, reducing state chattering and actuator wear. The paper correctly identifies a real limitation of DeePO, namely that LQR state feedback cannot generate persistently exciting inputs, and the simulation qualitatively illustrates the intended benefit. However, the two theoretical pillars are not sound as written. The proof of Theorem 1 contains an invalid inference that requires an unstated spanning condition, Lemma 4 points to a mechanism by which the first branch of Algorithm 2 can drive σ(Φ) to zero, and Theorem 2 conflates the standard LQR DARE gain with a different gain defined by a modified DARE. The stability analysis also relies on an interval [v,v] whose existence is asserted but not constructively verified for the actual system data. The paper does not provide reproducible code or machine-checked proofs, and the simulation alone does not compensate for the gaps in the theory.","major_comments":[{"comment":"The inference from equations (23)-(24) to equation (25) is invalid. From (24) one obtains, for each k, the relation ∑_i (λ_i - ∑_j μ_j K_{j,i}^{(k)}) x_i^{(k)} = 0, which implies λ = (K^{(k)})^T μ only if the vectors x^{(k)} span R^n. No such spanning condition is stated or proven, and for a stable open-loop plant the states typically decay and may lie near a low-dimensional subspace. Moreover, the row dependence (23) involves all columns of X0 and U0, including the initial offline persistently exciting columns, whereas the substitution U0_{j,k} = ∑_i K_{j,i}^{(k)} x_i^{(k)} applies only to online columns generated by state feedback. The contradiction in Theorem 1 therefore does not follow.","section":"Theorem 1, Section III"},{"comment":"Lemma 4 shows that appending a rank-deficient block D2 drives σ(Φ) to zero. In Algorithm 2, whenever ∥ΔK∥ > δ and ∥x_i∥ > γ, the control is u_i = K_i x_i with no stochastic excitation, and the resulting column is appended to the dataset. By Lemma 3 these columns are rank-deficient, so σ(Φ_i) can be driven toward zero before the gain converges. Theorem 1 does not establish a positive lower bound on σ(Φ_i), and it cannot, because the first branch of Algorithm 2 is exactly the regime covered by Lemma 4. The pause rule does not fix this: it stops appending data only when ∥x_i∥ ≤ γ, not when ∥ΔK∥ > δ.","section":"Lemma 4 and Algorithm 2, Section III"},{"comment":"Theorem 2 defines the converged gain K through the modified DARE (31) with β ∈ (0,1), but the DeePO/LQR gain referenced in (26)-(27) solves the standard DARE, which corresponds to β = 1. Unless β = 1, these gains are different, so the key identity (37), on which the Lyapunov decrease (38) relies, does not hold for the actual DeePO gain. The statement that 'K is equal to a certainly equivalent LQR gain' is therefore inconsistent with (31) for β < 1. The existence and uniqueness of a positive definite solution H to the modified DARE (31) is also assumed without proof.","section":"Theorem 2, Equations (26)-(31)"},{"comment":"The instruction 'Find v and v as in Theorem 2 for Ki' is not implementable as stated. It requires the solution H of the modified DARE (31) and verification of the matrix inequality (28), but no data-driven procedure is given to compute H or to certify (28) from measured data. Remark 1's iterative narrowing is not a concrete check, since no estimate of H is supplied. The simulation states that the interval [0.5,1.5] 'satisfies (28)' without presenting the verification, so the stability guarantee is not actually demonstrated for the reported experiment.","section":"Algorithm 2, Remark 1, Section IV"},{"comment":"For the open-loop stable plant with initial gain K_t = 0, the state decays toward the equilibrium. If ∥x_i∥ ≤ γ is reached while ∥ΔK∥ > δ, the second if-else block freezes the gain K_{i+1} = K_i and stops appending data, so learning stalls permanently before the gain converges. The simulation avoids this by injecting a disturbance at sample k = 15, but Algorithm 2 contains no mechanism that guarantees the gain converges before the state enters the γ-ball. This is a load-bearing gap because the paper's central claim is that PFDeePO works without probing noise.","section":"Section IV, simulation setup"}],"minor_comments":[{"comment":"There are several language and typography issues: 'certainly equivalent' should be 'certainty-equivalent', 'Perturbations-free' should be 'Perturbation-free', and the author block contains a stray space in 'V .'.","section":"Throughout"},{"comment":"The notation in (15) is unclear: the bar notation for U0, X0, and X1 is introduced informally, and the relationship between Φ = (1/t)DD^T and the subsequent X1 = X1D^T/t is not defined cleanly. Please define all overlined quantities explicitly.","section":"Equation (15)"},{"comment":"The disturbance at sample k = 15 is described only as 'a uniform random value'; its dimension, range, and whether the reported results are from a single run or averaged over multiple runs should be specified.","section":"Section IV"},{"comment":"The curve labeled 'DeePO without e' is not covered by Algorithm 1, since Algorithm 1 requires a probing noise signal and Φ may become singular without it. The implementation of this benchmark should be described, particularly the matrix inversion step.","section":"Figure 2"},{"comment":"The closed-loop stability result is stated for the estimated matrices (Â,B̂), but the relationship between these estimates and the true system (1) is not discussed. The practical relevance of the bound (33) for the true plant is therefore unclear.","section":"Theorem 2, Equation (32)"}],"recommendation":"reject","confidential_remarks":"The paper's core idea is a reasonable heuristic, but the theoretical support is not close to being complete. The most serious issue is the combination of the invalid inference in Theorem 1 and the fact that Lemma 4 seems to describe the algorithm's own first branch; this is not a missing detail but a fundamental obstruction to the claimed full-rank guarantee. The inconsistency between the standard DARE gain and the modified DARE in Theorem 2 is similarly load-bearing. I would not recommend a revision unless the authors can either prove a positive lower bound on σ(Φ_i) under explicit excitation conditions or substantially change the algorithm, and can reconcile Theorem 2 with the actual LQR gain."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper identifies a real practical problem—the probing noise that DeePO needs for persistent excitation causes state chatter—and proposes a plausible heuristic to avoid it. But the central proof that the new algorithm preserves the full-rank condition does not hold up, and the stability theorem doesn't actually apply to the algorithm as implemented.\n\nWhat’s new: PFDeePO pauses gain updates when states are near equilibrium, and once the gain has (supposedly) converged it multiplies the control by a random scalar in [v,v] with mean one. That specific combination is new relative to DeePO. The authors also prove a stability bound for a fixed gain with random multiplicative scaling; that Lyapunov argument is standard and, on its own, works given the interval condition. The simulation clearly illustrates the problem they're trying to solve, and the writing is clear.\n\nThe soft spots are load-bearing. Theorem 1’s proof goes from the row-dependence equation to componentwise equalities that require the state vectors to span R^n; no such spanning assumption is stated or established. For a stable open-loop plant the states typically decay, so the inference fails. Worse, Lemma 4—which the paper itself states—shows that appending rank-deficient input/state data drives the minimum singular value of Φ to zero. The first branch of Algorithm 2 appends exactly that kind of data whenever the gain is still changing and the state is not too small, and no lower bound on σ(Φ) is supplied. The pause rule doesn't rescue this: for a stable plant, the state can cross the threshold γ while K_i is still far from optimal, freezing the adaptation. The simulation masks this by throwing in a disturbance at k=15.\n\nTheorem 2 has a separate conflation: the gain K is first defined via the standard LQR DARE, then the theorem redefines H as the solution of a modified discounted DARE and uses the same K formula. Those are different H's, so the stability result is not connected to the implemented algorithm. Finally, the interval [v,v] that makes Lemma 5 hold is asserted to exist, but no constructive check is given, and the paper admits parameter selection is future work.\n\nNet: the proposal is a reasonable heuristic, but the paper-as-written does not establish its main claims. I would not cite it, and I wouldn't bring it to a reading group except as an example of how easy it is to overclaim in adaptive data-driven control. Still, the underlying problem is real and the stability lemma could be repurposed. With a serious rewrite—either a correct full-rank proof or an honest framing as an empirical heuristic—the idea might be publishable. I'd let it go to review rather than desk-reject, because a referee's detailed comments could help the authors fix it.\n\nCheers.","headline":"A well-motivated heuristic for removing probing noise from DeePO, but the full-rank proof has a fatal gap and the stability theorem doesn't match the algorithm.","tokens_in":12352,"tokens_out":4170,"would_cite":false,"duration_ms":42486,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modified DeePO algorithm claims full-rank data and calm states without probing noise.","keywords":["data-enabled policy optimization","adaptive LQR","persistency of excitation","probing noise","state perturbations","data-driven control","multiplicative noise","minimum singular value"],"falsifier":"Run Algorithm 2 on the paper's four-state system with zero initial gain, no process noise, and no external disturbance; if the state norm drops below γ before ‖K_i − K*‖ approaches zero, the gain error or σ(Φ_i) will plateau, showing that the pause rule can stop learning before convergence.","tokens_in":11257,"feed_emoji":"⚙️","tokens_out":4903,"duration_ms":55995,"temperature":0.7,"pith_summary":"This paper targets a conflict at the heart of the data-enabled policy optimization (DeePO) algorithm for adaptive LQR control: DeePO needs persistently exciting inputs, but the linear state-feedback gains it produces cannot supply that excitation, so probing noise is normally added. That noise keeps the data informative but also rattles the states. The paper proposes Perturbation-Free DeePO (PFDeePO), which pauses gain updates when states are near equilibrium and applies a random multiplicative gain to the control signal only after the controller has converged. If the proposal works, adaptive LQR can keep learning from its own data without forever injecting dither noise, leaving states steady while still steering the gain toward the certainty-equivalence LQR solution.","feed_headline":"PFDeePO drops probing noise from adaptive LQR learning","feed_subtitle":"A pause rule plus safe random gain scaling keeps data informative and states calm.","key_machinery":"The central object is the data covariance Φ = (1/t)DD^T built from stacked input-state data; its full rank is exactly the persistency-of-excitation condition that lets DeePO invert Φ and update the gain. PFDeePO wraps this in two mechanisms: a pause rule keyed to ‖x_i‖ ≤ γ that prevents uninformative zero columns from entering the dataset, and a multiplicative random gain v_i on the control signal, triggered only when the gain update ‖ΔK‖ has fallen below δ. The stability guarantee rests on the interval [v,v] from Lemma 5 and the modified DARE (31), whose solution H yields the exponential bound ‖x_k‖ ≤ β^k $\\sqrt$(λmax(H)/λmin(H))‖x_0‖ for any v_k in the interval.","core_discovery":"The paper claims that PFDeePO resolves the DeePO excitation dilemma by separating two failure regimes. When states converge to the equilibrium, appending them to the data matrix drives the minimum singular value of the sample covariance Φ to zero; PFDeePO stops updating the gain in this regime. When the gain has already converged but states are still far from equilibrium, the algorithm multiplies the control signal by a random scalar v_i drawn from an interval [v,v] with v < 1 < v, ensuring newly appended columns keep the data matrix full rank. The paper proves in Theorem 1 that Φ_i stays full rank under Algorithm 2, and in Theorem 2 that any sequence v_i in that interval preserves exponential stability of the closed loop via a modified DARE and a quadratic Lyapunov function. Simulations on a four-state system show states remaining at equilibrium after a disturbance, while σ(Φ_i) stays bounded away from zero.","pith_inferences":["An immediate testable extension is to replace the threshold heuristics γ and δ with a direct monitor of σ(Φ_i), pausing only when the data actually lose informativeness; this could remove two hand-tuned parameters.","If a stable open-loop plant drives the state to equilibrium before the gain has converged, the pause rule can stall learning altogether; the paper does not analyze that pre-convergence stall, and adding a bounded external disturbance or retaining a few persistently exciting offline columns would be a natural workaround.","The interval [v,v] could be selected online from a data-driven estimate of B via the pseudoinverse formula in Remark 2, making the stability certificate adaptive rather than fixed, at the price of needing an accurate B estimate.","The same pause-and-scale idea may carry over to output-feedback or nonlinear extensions of DeePO, but those settings would need their own Lyapunov or observability assumptions, not just the LTI DARE argument."],"forward_implications":["Adaptive LQR can in principle run without continuous probing noise, eliminating the persistent dither that wears actuators and disturbs states.","The inverse Φ^{-1} in the DeePO update remains well-defined at every iteration because σ(Φ_i) is kept positive by the pause-and-scale logic.","When the random scaling is active, the closed loop stays exponentially stable for any sequence of gains in the allowed interval, so the added excitation does not sacrifice stability.","Near equilibrium, the gain update is frozen, so the algorithm stops spending control effort on data points that carry no new information.","The simulation comparison shows PFDeePO settling states after a disturbance while DeePO with probing noise keeps oscillating and increasing control effort."],"supporting_citations":[{"why":"Supplies the DeePO algorithm, the Φ-based policy parametrization, and the convergence-to-certainty-equivalence-LQR result that PFDeePO modifies.","marker":"[16]"},{"why":"Provides Willems' fundamental lemma and the definition of persistency of excitation, on which Lemma 1 and the full-rank requirement rest.","marker":"[1]"},{"why":"Supports the claim that the certainty-equivalence LQR gain is an efficient and valid optimal target for the data-driven update.","marker":"[4]"},{"why":"Supplies the H2-norm cost expression C(K) and the Lyapunov equation for the closed-loop state covariance used to define the LQR objective.","marker":"[19]"},{"why":"Earlier DeePO formulation that establishes the gradient-descent policy update procedure inherited by the modified algorithm.","marker":"[15]"},{"why":"Provides the Rouché–Capelli theorem used to connect full rank of D with the existence of a matrix G such that [K; I] = DG.","marker":"[23]"}],"fun_headline_variants":["PFDeePO eliminates probing noise, keeps LQR learning stable","Adaptive LQR without probing noise via pause and random gain","PFDeePO: pause near equilibrium, random gain for full rank","No probing noise needed: PFDeePO's two-pronged fix","PFDeePO resolves state perturbations without probing noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that in the noiseless pre-convergence phase, state-feedback data remain informative enough for the DeePO gradient update to make progress before the pause rule triggers; if a stable plant drives the state to equilibrium first, learning can stall before the optimal gain is reached.","fun_headline_variants_meta":{"raw":{"variants":["PFDeePO eliminates probing noise, keeps LQR learning stable","Adaptive LQR without probing noise via pause and random gain","PFDeePO: pause near equilibrium, random gain for full rank","No probing noise needed: PFDeePO's two-pronged fix","PFDeePO resolves state perturbations without probing noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1750,"prompt_tokens":980,"completion_tokens":770,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":680}},"tokens_in":596,"tokens_out":770,"duration_ms":6721,"temperature":1.0,"reasoning_tokens":680,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:41:58.845258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 2 on the paper's four-state system with zero initial gain, no process noise, and no external disturbance; if the state norm drops below γ before ‖K_i − K*‖ approaches zero, the gain error or σ(Φ_i) will plateau, showing that the pause rule can stop learning before convergence.","supporting_citations":[{"cited_title":"Data-enabled policy opti- mization for direct adaptive learning of the LQR,","cited_arxiv_id":null,"evidence_quote":"Supplies the DeePO algorithm, the Φ-based policy parametrization, and the convergence-to-certainty-equivalence-LQR result that PFDeePO modifies."},{"cited_title":"Certainty equivalence is efficient for linear quadratic control,","cited_arxiv_id":null,"evidence_quote":"Supports the claim that the certainty-equivalence LQR gain is an efficient and valid optimal target for the data-driven update."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the H2-norm cost expression C(K) and the Lyapunov equation for the closed-loop state covariance used to define the LQR objective."},{"cited_title":"Data-enabled policy optimization for the linear quadratic regulator,","cited_arxiv_id":null,"evidence_quote":"Earlier DeePO formulation that establishes the gradient-descent policy update procedure inherited by the modified algorithm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Rouché–Capelli theorem used to connect full rank of D with the existence of a matrix G such that [K; I] = DG."}],"review_version":2}