{"id":"2abc54f7-9b85-459a-a865-963010ce1bc2","arxiv_id":"2505.09057","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A Thompson-sampling LQR algorithm that incorporates offline data from a similar unknown system achieves O~(sqrt(T/S)) Bayes regret when the systems are close.","lead":"This paper designs an online controller that uses pre-collected data from a different but similar system to reduce exploration in linear quadratic control. It proves a regret bound that improves with the amount of offline data when the two systems are close.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 3's R0=0 is unsupported: Eq. (7) samples an inflated least-squares perturbation, not the posterior of θ* under μ, so the Bayes regret decomposition omits a nonzero term and Theorem V.1 is not established.","rationale":"The reader's concern about S>T is legitimate and structural: Lemma 2 selects δ1 = δ/(16S), and the union bound over T only closes when S > T. This restricts the practical regime and, if S < T, the final high-probability event does not follow as written. However, I view the R0 issue as more load-bearing because it affects the proof even in the favorable S > T regime. Lemma 3 is the only step that removes the J(θ~t) − J(θ*) term from the regret decomposition; without it, the bound in Theorem V.1 has an additional term that the authors do not analyze. The sampling rule in Eq. (7) is a randomized least-squares estimate with an inflation factor β_t, not a draw from the posterior of θ* under the prior μ. The paper explicitly assumes μ is known only for analysis, and the algorithm never uses μ, so the cited posterior sampling lemma cannot apply. A degenerate point-mass prior is a valid instance of the paper's assumptions and makes the failure transparent: the posterior is a point mass, while the algorithm still samples a non-degenerate perturbation. This is an internal inconsistency in the proof, not merely a disagreement with common practice. The numerical experiments in Section VI provide useful intuition, but they do not test the Bayesian regret identity at the core of the proof. For these reasons, I would move the verdict from CONDITIONAL to REJECT unless the authors supply a correct treatment of R0 or modify the algorithm to sample from the true posterior.","tokens_in":13707,"tokens_out":20815,"duration_ms":227613,"concrete_test":"Fix μ = δ_{θ0} and θ_sim* = θ0 with Mδ = 0, choose S > T, and run the offline data collection. At t = 1, compute θhat_1 = θhat_sim and V_1 = U_S, then estimate E[J(S_Q(θhat_1 + β_1 V_1^{-1/2} η))] by Monte Carlo over η. Lemma 3 requires this expectation to equal J(θ0); with β_1 > 1 and θhat_1 ≠ θ0, a generic LQR instance will show a mismatch. As a second check, re-derive Theorem V.1 without Lemma 3, keeping the R0 term in the decomposition; if the extra term cannot be absorbed into the O~(sqrt(T/S)) bound, the theorem fails as stated.","verdict_should_be":"REJECT","load_bearing_attack":"Lemma 3 claims R0 = Σ_t E[J(θ~t) − J(θ*)] = 0 by the posterior sampling lemma [22, Lemma 1]. But the sampling distribution in Eq. (7) is not the posterior of θ* under the known prior μ. The algorithm never uses μ; it samples θ~t = S_Q(θhat_t + β_t(δ2) V_t^{-1/2} η_t) with β_t > 1, i.e., a confidence-inflated regularized least-squares perturbation. The posterior of θ* given F_t ∪ F_s would require a likelihood for the offline data D in terms of θ*, whereas D is generated from θ_sim* with only a norm bound Mδ on θ* − θ_sim*. Under the admissible prior μ = δ_{θ0}, the posterior is deterministic θ0, while Eq. (7) is non-degenerate, so the hypotheses of [22, Lemma 1] fail. For generic LQR instances, E[J(θ~t)] ≠ J(θ0), so R0 is not zero and the decomposition (11) is missing a term. This is independent of the S>T issue: even when S>T, the proof of Theorem V.1 is incomplete. The claimed O~(sqrt(T/S)) Bayes regret is therefore not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper considers online LQR control of an unknown linear system, when in addition to online state-action data, an offline trajectory of length S from a different but similar unknown system is available. The proposed TSOD-LQR algorithm forms a regularized least-squares estimate that combines online data with an offline estimate, and uses Thompson-sampling-style perturbations (Eq. 7) to select controllers. The central claim is a Bayesian regret bound of O~(sqrt(T/S)(log T + E[alpha_S(delta1) + sqrt(lambda_max(U_S)) M_delta])) under the assumption S>T (Theorem V.1), and a smaller regret when the dissimilarity M_delta is small. The paper also sketches an extension to multiple offline sources and presents numerical illustrations.","tokens_in":13985,"tokens_out":7490,"duration_ms":77603,"significance":"If the main result held, the paper would make a useful contribution: it gives a principled way to use offline data from a related system to reduce the regret of online LQR control, with an explicit dependence on the dissimilarity M_delta and offline length S. The attempt to characterize when offline data helps, rather than merely to demonstrate empirical gains, is valuable. However, the main theorem is not established as written because a key lemma in the regret decomposition is based on a misapplication of the posterior sampling lemma, and several central proof steps are only sketched with reference to previous work. The paper also contains a factual inconsistency between the stated theorem rate and the rate quoted in the conclusion.","major_comments":[{"comment":"The claim R0=0 is unsupported because the sampling distribution in Eq. (7) is not the posterior of theta* under the prior mu. The algorithm samples theta_tilde_t = S_Q(theta_hat_t + beta_t(delta2) V_t^{-1/2} eta_t), which is a confidence-inflated least-squares perturbation projected onto Q. The prior mu is known only to the analysis and is never used by the algorithm. The posterior sampling lemma [22, Lemma 1] requires the sampled parameter to be drawn from the posterior of theta* given the filtration; here the distribution is that of the random perturbation around the regularized least-squares estimate. For example, if mu is a point mass at theta0, the posterior is deterministic but Eq. (7) is non-degenerate, so E[J(theta_tilde_t)] != J(theta0) in general. Consequently, the regret decomposition (11) is missing a nonzero term R0, and the proof of Theorem V.1 is incomplete. This issue is independent of the S>T condition and affects the central regret bound.","section":"Section V, Lemma 3 and Eq. (11)"},{"comment":"The proof of Lemma 6, which bounds the term R3, is only an outline. The text states that the proof 'resembles that of [15, Lemma 1]' and then invokes [15, Propositions 7, 8, 9, 11] and Lemma 11, whose proof is also omitted. Since Lemma 6 is a load-bearing component of the final regret bound, the adaptation of these arguments to the present setting, where beta_t includes the offline terms alpha_S(delta1) and sqrt(lambda_max(U_S)) M_delta, must be shown explicitly. As written, the proof of Theorem V.1 cannot be fully verified from the manuscript.","section":"Section V, Lemma 6 and Appendix C"},{"comment":"The requirement S>T is structural, not an inconsequential technical convenience. In the proof of Lemma 2, selecting delta1 = delta/(16S) and delta2 = delta/(16T) gives P(hat E_T) >= 1 - T(delta1+delta2) = 1 - delta/16 - T delta/(16S). If S <= T, this lower bound can be as low as 1 - delta/8 - delta/16, which is less than the claimed 1 - delta/4, so the union bound over T events fails. The paper's statement that S>T is 'not an onerous assumption' is debatable; for real systems, offline trajectories longer than the entire online horizon may be unrealistic. The problem statement and abstract do not state this condition, and the paper does not provide any result for the practically relevant regime S <= T. The authors should either prove the bound for S <= T, state clearly that the result requires S>T in the abstract and introduction, or discuss the necessity of this assumption.","section":"Section V, Lemma 2 and Remark 1"}],"minor_comments":[{"comment":"The conclusion says the algorithm achieves O~(f(S,M_delta) sqrt(T)), which contradicts Theorem V.1 and the abstract, where the rate is sqrt(T/S). This inconsistency should be corrected.","section":"Section VIII"},{"comment":"The notation \\tilde{O} is used without a formal definition; please define it explicitly (the abstract and Section V use it with different implicit meanings).","section":"Abstract and Introduction"},{"comment":"The caption contains a typo, 'al algorithm', and the details of the two baseline algorithms ('does not utilize the offline data' and 'only utilizes the estimate theta_hat_sim') are not described in the text. Please specify the baselines so the simulation comparison is reproducible.","section":"Section VI, Figure 1 caption"},{"comment":"In the definition of e(θ), the term (x_{k+1} - θ^T z_s)(x_{k+1} - θ^T z_s)^T uses z_s instead of z_k; this appears to be a typographical error that should be fixed.","section":"Section VII"}],"recommendation":"major_revision","confidential_remarks":"The paper's main idea is plausible, but the proof has a load-bearing gap in Lemma 3 that cannot be patched by a simple rewriting; the authors need to either replace the posterior-sampling argument with a valid bound on the R0 term or modify the algorithm to sample from the true posterior. The reliance on outlined proofs deferring to Abeille-Lazaric is also substantial for a journal submission. The S>T assumption should be made prominent in the abstract and introduction, as it limits the advertised benefit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is genuinely new and worth engaging with: it extends Thompson sampling for LQR to incorporate an offline trajectory from a different unknown system, with a regret bound that scales like sqrt(T/S) and depends on a dissimilarity bound M_delta. That is a meaningful problem in sim-to-real transfer, and the algorithm (TSOD-LQR) is a natural construction: regularized least squares with the offline data folded into the precision and the estimate, plus an inflated confidence radius. The multi-source extension in Section VII is a nice bonus. Credit where due: the problem is well motivated, the related work is cited honestly, and the confidence-bound derivation in Theorem IV.1 follows a coherent chain.\n\nThe soft spots are real, and the most serious one is the stress-test concern. Lemma 3 claims R0 = 0 by invoking the posterior sampling lemma, but the algorithm never samples from the posterior. Equation (7) samples a truncated, confidence-inflated perturbation around the regularized estimate, with beta_t > 1 and no relationship to the prior mu. Under a Dirac prior, the posterior is deterministic, while the sampling is non-degenerate; under a general prior, the sampling distribution is still not the posterior. So the hypotheses of the posterior sampling lemma fail, R0 is not demonstrably zero, and the regret decomposition (11) is missing an unanalyzed term. This is a load-bearing gap: the claimed O~(sqrt(T/S)) bound is not established. It is also independent of the S>T issue. That said, the gap might be patchable—one could try to bound R0 directly using the confidence ellipsoid and Lipschitz properties of J, possibly with an extra term that still fits the rate—but that has not been done here.\n\nThe S>T assumption is also restrictive and is brushed off too quickly in Remark 1. Several lemmas (6, 9, 11) are only outlined or deferred to [15], so the paper is not fully self-contained. The numerical section is thin (10 runs) and there is no formal comparison against the no-offline baseline, despite the abstract claiming outperformance. The conclusion contains an error (O~(sqrt(T)) instead of sqrt(T/S)), which suggests a hasty final pass.\n\nThis is not a take-down; the idea deserves serious attention. But the proof as written has a genuine hole, and the authors should be asked to fix it, not just polish the presentation. I would send it to peer review because the problem is important and the core contribution is likely salvageable. I would not cite it in its current form until the gap is closed.","headline":"A plausible and novel transfer-learning framework for LQR, but the proof has a real gap: the sampling step is not posterior sampling, so Lemma 3's R0=0 does not follow and Theorem V.1 is not established as written.","tokens_in":758,"tokens_out":799,"would_cite":false,"duration_ms":55385,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Thompson-sampling LQR controller that warm-starts from an offline trajectory of a similar system achieves $\\tilde{\\mathcal{O}}(\\sqrt{T/S})$ Bayes regret when the two systems are close, and the guarantee degrades gracefully with a known…","keywords":["linear quadratic regulator","Thompson sampling","Bayes regret","offline data","warm start","sim-to-real gap","transfer learning","adaptive control"],"falsifier":"Concretely: simulate the identical-system case ($M_\\delta=0$), choose $S=T/2$, and set $\\delta_1=\\delta/(16S)$, $\\delta_2=\\delta/(16T)$ exactly as in Lemma 2. The proof's union bound then gives $\\Pr(\\hat{E}_T)\\ge 1 - T(\\delta_1+\\delta_2) \\approx 1 - \\delta/8 \\cdot (T/S)$, which is strictly worse than the claimed $1-\\delta/8$; if in repeated trials the regret violates the $\\tilde{\\mathcal{O}}(\\sqrt{T/S})$ rate whenever $S<T$, the $S>T$ premise is doing essential work.","tokens_in":13492,"feed_emoji":"📉","tokens_out":9209,"duration_ms":84733,"temperature":0.7,"pith_summary":"The paper asks whether an imperfect simulator can speed up online learning of a linear quadratic regulator when the real system's matrices are unknown. It proposes TSOD-LQR, a Thompson-sampling algorithm that initializes the online posterior with the offline trajectory's mean estimate and precision matrix, and adds a term penalizing a known bound $M_\\delta$ on the dissimilarity between simulator and real system. The main result is a Bayes-regret bound after $T$ steps of $\\tilde{\\mathcal{O}}(\\sqrt{T/S}(\\log T + E[\\alpha_S(\\delta_1)+\\sqrt{\\lambda_{\\max}(U_S)}M_\\delta]))$, where $S$ is the offline trajectory length, and when $M_\\delta$ is small this reduces to $\\tilde{\\mathcal{O}}(\\sqrt{T/S})$. A sympathetic reader should care because the bound shows offline data from a different unknown system, not just from the real system, can cut exploration cost, and the benefit grows with $S$.","feed_headline":"Simulator data trims online LQR regret to sqrt(T/S)","feed_subtitle":"Warm-starting Thompson sampling with an offline trajectory from a similar system provably cuts exploration cost.","key_machinery":"The engine is the augmented least-squares recursion $V_t = U_S + \\sum_{k=0}^{t-1} z_k z_k^{\\top}$, $\\hat{\\theta}_t = V_t^{-1}(\\sum_{k=0}^{t-1} z_k x_{k+1}^{\\top} + U_S \\hat{\\theta}_S^{\\mathrm{sim}})$, which converts the offline trajectory into a prior precision matrix $U_S$ and a prior mean $\\hat{\\theta}_S^{\\mathrm{sim}}$; this is what makes the offline data shrink the confidence ellipsoid at rate $1/\\sqrt{S}$. The Thompson-sampling radius $\\beta_t(\\delta_2) = n\\sqrt{2\\log(\\det(V_t)^{0.5}/\\det(U_S)^{0.5}\\delta_2)} + \\alpha_S(\\delta_1) + \\sqrt{\\lambda_{\\max}(U_S)}M_\\delta$ is the second ingredient: it honestly accounts for three sources of error, so the prior is not trusted too much when the simulator is far from the real system. The proof machinery is the standard Thompson-sampling regret decomposition into one-step cost difference, state-bound, and model-change terms, with the condition $S>T$ used to make the high-probability event $E_T$ hold via the union bound.","core_discovery":"The central claim is that a longer offline trajectory from a similar-but-not-identical linear system is a genuine resource for online LQR control, not just a warm-start heuristic. The algorithm forms $V_t = U_S + \\sum_{k=0}^{t-1} z_k z_k^{\\top}$ and $\\hat{\\theta}_t = V_t^{-1}(\\sum_{k=0}^{t-1} z_k x_{k+1}^{\\top} + U_S \\hat{\\theta}_S^{\\mathrm{sim}})$, so the offline data enter exactly as prior pseudo-observations with precision $U_S$. Thompson sampling then draws $\\tilde{\\theta}_t$ from a confidence ellipsoid around $\\hat{\\theta}_t$ whose radius $\\beta_t(\\delta_2)$ contains the online self-normalized noise, the offline estimation error $\\alpha_S(\\delta_1)$, and the dissimilarity penalty $\\sqrt{\\lambda_{\\max}(U_S)}M_\\delta$. Provided $S>T$, the paper proves, with probability at least $1-\\delta$, a Bayes regret bound $\\tilde{\\mathcal{O}}(\\sqrt{T/S}(\\log T + E[\\alpha_S(\\delta_1)+\\sqrt{\\lambda_{\\max}(U_S)}M_\\delta]))$, and in the identical-system case $M_\\delta=0$ it becomes $\\tilde{\\mathcal{O}}(\\sqrt{T/S})$.","pith_inferences":["Because the proof only needs Assumption 2 on the offline algorithm, the practical lever is to design the offline exploration policy to spread out the state-action pairs and grow the minimum eigenvalue of $U_S$, not merely to minimize the offline estimation error $\\alpha_S$; the regret bound rewards both, but $\\lambda_{\\min}(U_S)\\ge S/40$ enters the log-det terms.","The analysis is Bayesian and relies on the posterior-sampling lemma for $R_0=0$; a frequentist or adversarial version would need a different argument, and for large $M_\\delta$ the bound suggests the offline prior may actually hurt, so there should be a crossover point at which discarding the offline data is safer.","A direct testable extension: replace Thompson sampling with a certainty-equivalence or UCB sampler inside the same augmented-precision recursion; the concentration part of the proof would survive, but the regret decomposition terms $R_0$ and the model-change term would need reworking, so the $\\sqrt{T/S}$ rate may not be automatic."],"forward_implications":["With $M_\\delta=0$, Corollary 1 gives $\\tilde{\\mathcal{O}}(\\sqrt{T/S})$ regret, so an offline trajectory from the exact system can push the online learning cost toward logarithmic growth when $S$ is much larger than $T$.","If $M_\\delta$ is small but nonzero, the bound still improves over the no-offline baseline as long as $\\alpha_S(\\delta_1)+\\sqrt{\\lambda_{\\max}(U_S)}M_\\delta$ grows more slowly than $\\sqrt{T}$; the claimed speedup is real only when the dissimilarity penalty is dominated by the offline estimation benefit.","The algorithm works without knowing the prior distribution $\\mu$ of $\\theta_*$; the known-distribution assumption is used only in the analysis through the posterior-sampling lemma.","The framework extends to multiple offline sources: precision matrices and dissimilarity penalties add, giving a similar $\\tilde{\\mathcal{O}}(\\sqrt{T/S})$ bound with $S=\\sum_i S_i$ and $M_\\delta=\\max_i M_{\\delta,i}$.","Numerically, the paper reports that cumulative regret decreases as $S$ grows for fixed $M_\\delta$, and that using both the offline estimate and its uncertainty beats both ignoring the offline data and using only the offline point estimate."],"supporting_citations":[{"why":"Supplies the TSAC Thompson-sampling algorithm used to generate offline data in Theorem V.2 and the concentration properties ($\\alpha_S$, $\\lambda_{\\min}(U_S)\\ge S/40$) demanded by Assumption 2.","marker":"[2]"},{"why":"Establishes the Bayes-regret framework with known prior and the state-norm bound used in Lemma 9.","marker":"[3]"},{"why":"Provides one of the UCB-based LQR algorithms that satisfy the offline-data Assumption 2.","marker":"[4]"},{"why":"Provides the Thompson-sampling regret decomposition into $R_0$ through $R_3$ and the posterior-sampling and state-bound propositions on which Lemmas 5 through 8 rely.","marker":"[15]"},{"why":"Basis for the standard noise and filtration assumption (Assumption 1) used throughout the analysis.","marker":"[20]"},{"why":"Gives the posterior-sampling lemma that makes $R_0=0$ in Lemma 3.","marker":"[22]"},{"why":"Supplies the self-normalized martingale inequality (Corollary 1) and the elliptical-potential bound (Lemma 4) used to control the online noise terms.","marker":"[23]"}],"fun_headline_variants":["Similar-system offline data cuts online LQR regret to sqrt(T/S)","Offline data from similar systems trims LQR regret to sqrt(T/S)","Thompson sampling with similar-system offline data achieves sqrt(T/S) regret","Offline similar-system data provably lowers LQR regret to sqrt(T/S)","Use offline data from similar systems to cut LQR regret to sqrt(T/S)"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof's high-probability guarantee only goes through when the offline trajectory is longer than the online learning horizon ($S>T$), so the central claim is only established for problems where a simulator can supply that much data.","fun_headline_variants_meta":{"raw":{"variants":["Similar-system offline data cuts online LQR regret to sqrt(T/S)","Offline data from similar systems trims LQR regret to sqrt(T/S)","Thompson sampling with similar-system offline data achieves sqrt(T/S) regret","Offline similar-system data provably lowers LQR regret to sqrt(T/S)","Use offline data from similar systems to cut LQR regret to sqrt(T/S)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00087,"raw_usage":{"total_tokens":3827,"prompt_tokens":1063,"completion_tokens":2764,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":2667}},"tokens_in":679,"tokens_out":2764,"duration_ms":20319,"temperature":1.0,"reasoning_tokens":2667,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:44:56.876230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Concretely: simulate the identical-system case ($M_\\delta=0$), choose $S=T/2$, and set $\\delta_1=\\delta/(16S)$, $\\delta_2=\\delta/(16T)$ exactly as in Lemma 2. The proof's union bound then gives $\\Pr(\\hat{E}_T)\\ge 1 - T(\\delta_1+\\delta_2) \\approx 1 - \\delta/8 \\cdot (T/S)$, which is strictly worse than the claimed $1-\\delta/8$; if in repeated trials the regret violates the $\\tilde{\\mathcal{O}}(\\sqrt{T/S})$ rate whenever $S<T$, the $S>T$ premise is doing essential work.","supporting_citations":[{"cited_title":"Thompson sampling achieves ˜O √ Tregret in linear quadratic control,","cited_arxiv_id":null,"evidence_quote":"Supplies the TSAC Thompson-sampling algorithm used to generate offline data in Theorem V.2 and the concentration properties ($\\alpha_S$, $\\lambda_{\\min}(U_S)\\ge S/40$) demanded by Assumption 2."},{"cited_title":"Control of unknown linear systems with thompson sampling,","cited_arxiv_id":null,"evidence_quote":"Establishes the Bayes-regret framework with known prior and the state-norm bound used in Lemma 9."},{"cited_title":"Learning linear-quadratic regulators efficiently with only √ Tregret,","cited_arxiv_id":null,"evidence_quote":"Provides one of the UCB-based LQR algorithms that satisfy the offline-data Assumption 2."},{"cited_title":"Improved regret bounds for thompson sampling in linear quadratic control problems,","cited_arxiv_id":null,"evidence_quote":"Provides the Thompson-sampling regret decomposition into $R_0$ through $R_3$ and the posterior-sampling and state-bound propositions on which Lemmas 5 through 8 rely."}],"review_version":1}