{"id":"2fee2cae-6696-44d0-b7cd-d3d3c947794a","arxiv_id":"2412.02758","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EXP-LQR tunes LQR feedback gains from finite-time cost measurements alone, using sinusoidal perturbations and averaging theory to converge near the optimal gain.","lead":"This paper presents EXP-LQR, an algorithm that learns the optimal linear feedback controller for an unknown system using only short, finite-time experiments that measure the cost of perturbed policies. It proves exponential convergence to a small neighborhood of the optimum and demonstrates the method on a simulated induction motor.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2 is proved only for an exact noiseless finite-time cost oracle; the practical claim for real finite-time experiments requires a noise-robustness argument that the paper does not provide.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing issue: the proof treats the finite-time truncated cost oracle as exact, with no measurement noise and with perfect ability to initialize experiments at the canonical basis vectors. I agree that this is the main gap between the theorem and the practical claim of data-driven LQR from real finite-time experiments. This concern is not a fatal flaw in the mathematical statement of Theorem 2, which is explicitly conditioned on the exact-oracle model, but it makes the advertised applicability conditional. I examined the proof for other possible objections: the final radius choice in Section V-C appears imprecise (rho should be chosen to satisfy sqrt(rho/psi) <= r, i.e., rho <= psi r^2, rather than simply rho < sqrt(r psi)), and the last step of the proof moves from V_lambda(z - J^delta_AV(K), K) to V_lambda(z - J(K), K) without explicitly bounding |J - J^delta_AV|. These are repairable given the freedom to choose T large and delta small, and they do not change the verdict. The genuinely load-bearing limitation is the absence of any noise-robustness analysis, which the paper itself does not claim to provide. Therefore I recommend keeping the reader's CONDITIONAL verdict unchanged.","tokens_in":21443,"tokens_out":24062,"duration_ms":272392,"concrete_test":"Re-run the Section VI DFIM experiment with the reported tuning (gamma = 1e-7, delta = 1e-2, T = 20, k_prd = 19) and replace the exact oracle by J_T^obs(K_k + delta D_k) = J_T(K_k + delta D_k) * (1 + sigma * eta_k), with eta_k i.i.d. standard normal per iteration, for sigma = 0, 1e-3, and 1e-2. Record (J(K_k) - J(K*))/J(K*) and sigma_max(A + B(K_k + delta D_k)) over 2e5 iterations. If the relative cost error floor rises significantly above the noiseless value, or if K_k + delta D_k leaves the stabilizing set in (13a), at sigma = 1e-3, then the advertised claim for real finite-time experiments is unsupported; the comparison sigma/delta = 1e-1 at sigma = 1e-3 shows the noise amplification is non-negligible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result, Theorem 2, is proved for the exact-oracle model in Eqs. (9)-(10): the oracle returns the precise truncated cost J_T(K_k + delta D_k), computed as the mean of n deterministic noiseless experiments initialized at the canonical basis vectors e_i. The most load-bearing assumption for the advertised 'real experiments' setting is that these measurements are exact. In a real experiment, state and input measurements carry noise, initial states cannot be set exactly to e_i, and the returned cost is corrupted. Writing the observed cost as J_T^obs(K_k + delta D_k) = J_T(K_k + delta D_k) + eps_k, the gain update (11b) becomes K_{k+1} = K_k - gamma * (2/delta) * (J_T(K_k + delta D_k) + eps_k - z_k) D_k, so the gradient estimate inherits an error term -2 gamma eps_k D_k / delta. The error is amplified by 1/delta, which is large because the proofs require delta small. Every error bound in Section V (Lemmas 1-3, the bound on p_{delta,T}, and the Lyapunov decrement leading to (64)) treats J_T as exact; there is no term for eps_k and no bound on the noise amplitude that would preserve the practical stability conclusions. Thus the convergence guarantee (13) does not cover the 'finite-time real experiments' claimed in the introduction, and the paper supplies no noise-robust version of Theorem 2. This does not show Theorem 2 is false under its stated exact-oracle assumptions, but it means the central practical claim is conditional on a noiseless oracle.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EXP-LQR, a data-driven algorithm for discrete-time LQR that replaces model knowledge with evaluations of a finite-horizon truncated cost. At each iteration the current gain is perturbed by deterministic sinusoidal dither matrices, the truncated cost of the perturbed gain is measured in n experiments initialized on the canonical basis, and a low-pass-filtered extremum-seeking update is applied. The main theoretical result, Theorem 2, states that under Assumptions 4-6 and for sufficiently small step size, small dither amplitude, and sufficiently long experiment horizon, the gain iterates remain stabilizing and converge exponentially to an arbitrarily small neighborhood of the optimal LQR gain. The proof uses a generic discrete-time averaging result (Theorem 1), a gradient approximation lemma (Lemma 2), and a Lyapunov analysis of the averaged system (Lemma 3). A numerical study on a doubly fed induction motor model compares EXP-LQR with a model-free policy gradient method.","tokens_in":21811,"tokens_out":13667,"duration_ms":140357,"significance":"If correct, EXP-LQR is a useful addition to the data-driven LQR literature: it requires only finite-time cost evaluations rather than knowledge of (A,B) or (Q,R), uses deterministic dithering and filtering, and provides non-asymptotic exponential convergence guarantees. The generic discrete-time averaging theorem (Theorem 1) is also potentially reusable. The paper is honest about the need for a stabilizing initial gain and about the difficulty of tuning parameters. However, the advertised 'real experiments' claim is currently supported only for an exact, noiseless oracle, and one key lemma is imported from a related paper. These points do not invalidate the exact-oracle theorem, but they do limit the practical reach of the paper as written.","major_comments":[{"comment":"The convergence guarantee is stated and proved only for the exact truncated-cost oracle in (9)-(10). If the measured cost is corrupted as J_T^obs(K_k+δD_k)=J_T(K_k+δD_k)+ε_k, the update (11b) becomes K_{k+1}=K_k - γ(2/δ)(J_T(K_k+δD_k)-z_k)D_k - γ(2/δ)ε_kD_k, so the error term is amplified by 1/δ. None of the bounds in Section V (Lemma 1, Eq. (44), Eq. (64)) contains ε_k or an admissible noise amplitude, and the theorem's conclusion (13) does not cover such perturbations. The abstract and introduction advertise 'real experiments'; as written, Theorem 2 covers only noiseless virtual or exactly repeatable experiments. Please either restrict the claims accordingly or provide a noise-robustness analysis with explicit noise bounds that preserve (13).","section":"§IV, Eqs. (9)-(11), Theorem 2"},{"comment":"Lemma 2 is not proved in this paper. Appendix C states that [56, Lemma 1] provides the same result and that the steps can be repeated after replacing global C^3 regularity by compactness of S. Since Lemma 2 is the key gradient-approximation bound that defines p_{δ,T} in (18b) and underpins Lemma 3, a load-bearing result is imported. The paper should either give a full proof, including the Taylor expansion and the uniform bound on the remainder, or explicitly state Theorem 2 as conditional on [56, Lemma 1] with the precise statement of that lemma included.","section":"Appendix C, Lemma 2"},{"comment":"After applying Theorem 1, the proof writes Vλ(z_k - J(K_k), K_k) ≤ c1 and Vλ(z_k - J(K_k), K_k) ≤ (1-γa)^k Vλ(z0 - J(K0), K0) + ρ. However, Lemma 3 and Assumption 3 are established for Vλ(z_k - J^δ_AV(K_k), K_k), where J^δ_AV is defined in (18a); the averaged filter's fixed point is J^δ_AV, not J. The displayed inequalities therefore do not follow from the preceding application of Theorem 1, and the derivation of (13b) must be redone with the correct Lyapunov function. This appears repairable by a change of notation plus a uniform approximation argument, but as written it is a load-bearing gap.","section":"§V-C, Eq. (23)"},{"comment":"The proof of Theorem 1 uses the fact that χ_k^AV ∈ Ω_{c0} for all k≥0, citing Assumption 3 in Eq. (4). Assumption 3 actually gives V(χ_k^AV) ≤ (1-γa)^k V(χ0) + ρAV, hence at most c0+ρAV, not c0. Consequently the step V(χ_k)≤c0+ϵ≤c1 requires an additional smallness condition on ρAV and ϵ (e.g., c0+ρAV+ϵ<c1), which is not stated. Since Theorem 1 is the mechanism for transferring averaged convergence to the original system, this gap should be fixed in the rewritten proof.","section":"Appendix A, Eq. (37)"}],"minor_comments":[{"comment":"The caption lists MFPG with γ=10^-9, while the text says MFPG is run with γ=10^-8 and otherwise with the same tuning as EXP-LQR; the correspondence between curves and step sizes should be clarified.","section":"§VI, Fig. 4"},{"comment":"The displayed bound uses 'min' where a bound on the norm requires a maximum; as written, the inequality is not a valid upper bound.","section":"Appendix D, around Eq. (56)"},{"comment":"The definition of J^δ_AV(K) uses τ from k+1 to k+kprd, but the quantity is independent of k because of periodicity; making the index range independent of k would improve clarity.","section":"§V-B, Eq. (18a)"},{"comment":"The averaged system state is denoted K^k_AV while K also denotes the set of stabilizing gains; this is a minor notational clash that makes some formulas harder to parse.","section":"§V, general notation"},{"comment":"The conclusions refer to an 'inductance motor'; the paper studies an induction motor.","section":"§VII"},{"comment":"In the experiment phase, 'Test x_{t+1}=A x_t + B u_t' should be 'Set' or 'Run', since no test is being performed.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the main idea is interesting. The exact-oracle theorem is plausible, but I would not recommend acceptance until the noise-robustness gap is addressed, Lemma 2 is either proved or precisely imported, and the Eq. (23) mismatch is repaired. The paper is candid about parameter tuning in Remark 2, which is a strength. No ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nQuick take: this is a real advance within data-driven LQR, and the main theorem looks correct under its stated assumptions. The method (EXP-LQR) combines deterministic sinusoidal dither, a low-pass filter, and finite-horizon truncated cost oracles; that exact combination is new, and it improves on the conference version that demanded exact infinite-horizon costs. The proof is a careful averaging-plus-Lyapunov analysis, with a generic discrete-time averaging theorem (Theorem 1) that is useful on its own. The simulation on a doubly fed induction motor shows the expected behavior and a comparison to MFPG.\n\nThe soft spots are in proportion to the claim. The biggest one is that Theorem 2 is proved only for an exact, noiseless oracle: the cost samples are assumed to be the precise truncated costs computed from deterministic experiments initialized at canonical basis vectors. The paper says \"real experiments\" but gives no noise-robustness argument. The stress-test note is right that measurement noise would enter the gradient estimate scaled by 1/δ, and the proofs have no ε_k term. This does not falsify the theorem, but it means the practical claim needs either a robustness analysis or a much more careful statement that real experiments are a motivational framing, not a covered case.\n\nTwo smaller issues. Lemma 2, the key gradient approximation bound, is not proved here; it is imported from the authors' earlier work. The reference is plausible and the extension to compact sets is natural, but a referee should verify the constant and the period conditions. And the numerical section has an inconsistency: the text says MFPG is run with the same tuning as EXP-LQR (which uses γ=10^-7), but the figure reports MFPG at γ=10^-8 and 10^-9, and EXP-LQR at γ=10^-8. The simulation also has no error bars. These are minor for the theory but should be fixed.\n\nOverall, the central argument holds up in the exact-oracle model, the writing is clear, and the literature is engaged honestly. This deserves a serious referee. I would send it to review, asking for a discussion of noise or an explicit limitation, and correction of the numerical details.\n\nBest,\n[Your name]","headline":"A solid averaging-theory proof for a genuinely new finite-experiment LQR scheme, but the convergence guarantee only covers the exact noiseless oracle, not real experiments.","tokens_in":22313,"tokens_out":2983,"would_cite":true,"duration_ms":29714,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["93B52","93C55","49N10","90C56"],"pacs":[],"model":"deepseek-v4-flash","headline":"Exponential LQR convergence from short experiments only","keywords":["data-driven LQR","extremum-seeking control","policy iteration","finite-horizon cost","averaging theory","discrete-time nonlinear systems","derivative-free optimization","model-free policy gradient"],"falsifier":"Run EXP-LQR on a known scalar or small-state LQR with exact finite-horizon cost oracle and compare $\\|K_k-K^\\star\\|$ against the claimed exponential envelope $a_0(1-\\gamma a)^k+r$ across many choices of $(\\gamma,\\delta,T)$; if the error never enters the $r$-ball along the predicted envelope for any admissible parameter choice, Theorem 2 is false. A complementary test is to add small adversarial noise to $J_T(K)$, a case the proof excludes, and observe whether convergence degrades; that would delineate the theorem's limit rather than falsify it.","tokens_in":1539,"feed_emoji":"⚙️","tokens_out":1798,"duration_ms":75912,"temperature":0.7,"pith_summary":"This paper proposes EXP-LQR, an iterative algorithm that solves Linear Quadratic Regulator problems using only finitely many short experiments on the system, each returning a truncated approximation of the infinite-horizon cost. The algorithm needs no knowledge of the system matrices or the cost matrices. The paper proves that EXP-LQR exponentially converges to an arbitrarily small neighborhood of the optimal state-feedback gain, while keeping every policy used in the experiments stabilizing. This matters because it gives a theoretically certified, derivative-free route to LQR optimality in settings where only finite-time data are available, such as simulators or real plants.","feed_headline":"Exponential LQR convergence from short experiments only","feed_subtitle":"Only finite-horizon cost data are needed—no system or cost matrices—and the optimal gain is reached provably.","key_machinery":"The engine is an extremum-seeking update combined with a deterministic sinusoidal dither matrix $D_k$ and a scalar low-pass filter $z_k$. At each iteration the current gain $K_k$ is perturbed to $K_k+\\delta D_k$, the truncated cost $J_T(K_k+\\delta D_k)$ is measured from experiments, and the filter tracks that cost while the gain is moved along the dither-normalized difference signal. The analysis uses a newly stated discrete-time averaging theorem (Theorem 1) to compare this time-varying system with its period-averaged version, whose dynamics are a perturbed policy gradient. A Lyapunov function $V_\\lambda(z,K)=\\frac{1}{2\\lambda}\\|z\\|^2+J(K)-J(K^\\star)$ combines the filter error with the cost gap, and gradient dominance of the LQR cost gives the exponential decay; the combination yields semi-global practical exponential stability.","core_discovery":"The central claim is Theorem 2: for any desired accuracy radius $r>0$ and any stabilizing initial gain $K_0$, there exist sufficiently small step size $\\gamma$, sufficiently small dither amplitude $\\delta$, and sufficiently large experiment horizon $T$ such that the EXP-LQR updates keep $K_k+\\delta D_k$ stabilizing at every step and satisfy $\\|K_k-K^\\star\\|\\le a_0(1-\\gamma a)^k+r$ for all $k$. In other words, the extremum-seeking policy iteration converges exponentially fast into any prescribed ball around the optimal LQR gain $K^\\star$, using only finite-horizon truncated cost measurements. The convergence statement is semi-global in the set of stabilizing gains and practical in the sense that the final accuracy can be made arbitrarily good, and the proof treats the algorithm as a discrete-time time-varying nonlinear system analyzed via averaging and a Lyapunov argument.","pith_inferences":["An extension the paper leaves implicit is a stochastic version: if the truncated-cost oracle is noisy, the averaging argument as written does not apply, but a stochastic-approximation variant with noise-robust proofs would be a natural next step.","The paper proves existence of suitable parameters but does not give explicit quantitative rates for how $\\gamma$, $\\delta$, and $T$ should be chosen; a concrete tuning rule would make the method easier to deploy.","Because only $n$ experiments are needed per iteration for an $n$-dimensional state, the per-iteration experiment count is independent of the input dimension $m$, which could make the method attractive for systems with many actuators."],"forward_implications":["A practitioner can run EXP-LQR on a simulator or real plant by prescribing $T$-step experiments, and Theorem 2 certifies exponential approach to the optimal gain once $\\gamma$, $\\delta$, and $T$ are chosen appropriately.","No identification of $(A,B)$ or $(Q,R)$ is needed; only the scalar value of the truncated LQR cost is used, so the method applies when cost matrices are known only through sensor data or user feedback.","Because the dither is deterministic and the averaged system is a policy-gradient update, the analysis connects extremum seeking directly with gradient-descent theory for LQR.","Every policy used during learning remains stabilizing, so the algorithm never drives the closed-loop system unstable while collecting data.","The standalone discrete-time averaging theorem can be reused for other time-varying algorithms whose averaged dynamics are stable."],"supporting_citations":[{"why":"Supplies the discrete-time averaging theory that Theorem 1 extends and uses.","marker":"[46]"},{"why":"Supplies the continuous-time averaging background referenced for the overall averaging approach.","marker":"[44]"},{"why":"Provides the LQR cost gradient expression, differentiability, gradient dominance, and compact level-set properties used in Lemma 3.","marker":"[24]"},{"why":"Provides the topological properties of the set of stabilizing gains needed to keep perturbed gains stabilizing and level sets compact.","marker":"[49]"},{"why":"Supplies the derivative-free gradient approximation lemma that Lemma 2 adapts to finite-horizon costs.","marker":"[56]"},{"why":"Provides the trigonometric orthogonality result used to construct dither matrices satisfying Assumption 6.","marker":"[53]"}],"fun_headline_variants":["Model-free LQR: exponential convergence from short trials","Short experiments only for provable exponential LQR convergence","Extremum-seeking policy iteration: exponential LQR from short experiments","Finite-horizon cost data: exponential LQR convergence"],"cache_read_input_tokens":24320,"weakest_assumption_plain":"The proof needs the oracle to return the exact truncated cost $J_T(K)$ as the mean of $n$ noise-free finite-time experiments starting from the coordinate basis vectors; if measurements carry noise or the initial states cannot be set exactly, the averaging argument as written no longer holds.","fun_headline_variants_meta":{"raw":{"variants":["Model-free LQR: exponential convergence from short trials","Short experiments only for provable exponential LQR convergence","Extremum-seeking policy iteration: exponential LQR from short experiments","Finite-horizon cost data: exponential LQR convergence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001257,"raw_usage":{"total_tokens":5128,"prompt_tokens":904,"completion_tokens":4224,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":4156}},"tokens_in":520,"tokens_out":4224,"duration_ms":31530,"temperature":1.0,"reasoning_tokens":4156,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:08:52.311323+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EXP-LQR on a known scalar or small-state LQR with exact finite-horizon cost oracle and compare $\\|K_k-K^\\star\\|$ against the claimed exponential envelope $a_0(1-\\gamma a)^k+r$ across many choices of $(\\gamma,\\delta,T)$; if the error never enters the $r$-ball along the predicted envelope for any admissible parameter choice, Theorem 2 is false. A complementary test is to add small adversarial noise to $J_T(K)$, a case the proof excludes, and observe whether convergence degrades; that would delineate the theorem's limit rather than falsify it.","supporting_citations":[{"cited_title":"Averaging analysis for discrete time and sampled data adaptive systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the discrete-time averaging theory that Theorem 1 extends and uses."},{"cited_title":"On topological properties of the set of stabilizing feedback gains,","cited_arxiv_id":null,"evidence_quote":"Provides the topological properties of the set of stabilizing gains needed to keep perturbed gains stabilizing and level sets compact."},{"cited_title":"Extremum seeking tracking for derivative-free distributed optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the derivative-free gradient approximation lemma that Lemma 2 adapts to finite-horizon costs."},{"cited_title":"Sines and cosines of angles in arithmetic progression,","cited_arxiv_id":null,"evidence_quote":"Provides the trigonometric orthogonality result used to construct dither matrices satisfying Assumption 6."}],"review_version":1}