{"id":"7b53c3a5-84b8-4a63-b2b2-a3c7d6b14a08","arxiv_id":"2606.12691","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Two-layer linear auto-regressive models recover Kalman filter state estimates up to similarity transformation when trained on data from partially observed linear dynamical systems.","lead":"The paper shows that two-layer linear auto-regressive models trained by empirical risk minimization on data from partially observed linear dynamical systems learn hidden representations matching Kalman filter state estimates up to similarity transformation. A smart generalist might read it to understand why basic sequence models can implicitly perform optimal state estimation in dynamical systems.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Benign landscape claim (all stationary points are strict saddles or global minima) is the load-bearing assumption needed to reach the Kalman-approximating solution via ERM.","rationale":"The reader's weakest assumption directly identifies the same point. The AR-to-Kalman approximation and finite-sample bounds are more routine once global optimality is granted; the landscape claim is what bridges training to the stated representation recovery. Because the abstract presents this as an insight rather than a standard result, it remains the highest-risk step.","tokens_in":1727,"tokens_out":374,"duration_ms":15030,"concrete_test":"Fix a controllable and observable 2-state LDS with identity observation matrix and Gaussian process/measurement noise; analytically compute the gradient of the two-layer squared multi-step prediction loss with respect to the first-layer weights and set it to zero; check whether any stationary point exists whose hidden representation is not a linear transformation of the true Kalman estimate (e.g., by solving the resulting polynomial system or numerically sampling critical points).","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that gradient-based ERM on the two-layer linear AR loss recovers parameters whose hidden representation matches the Kalman state estimate (up to similarity). This holds only if the global minimum of the non-convex loss is the desired Kalman-like solution and if training reaches it. The paper lists the benign-landscape property as one of its three main insights, but this property is the least secure link: standard two-layer linear models can admit spurious stationary points when the loss is a multi-step prediction error on partially observed LDS data. If any such point is a non-strict saddle or local minimum whose hidden layer does not span the observable subspace, the finite-sample latent-state-recovery guarantee does not apply to the output of practical optimization.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that two-layer linear auto-regressive models trained by empirical risk minimization on data from partially observed linear dynamical systems learn hidden representations that coincide, up to similarity transformation, with the state estimates of the optimal Kalman filter. This follows from three insights: (i) the Kalman filter is well-approximated by an AR model with bounded truncation error, (ii) the non-convex two-layer optimization landscape is benign (all stationary points are strict saddles or global minima), and (iii) finite-sample guarantees on prediction error, parameter estimation error, and latent-state recovery. Numerical simulations are provided in support.","tokens_in":1908,"tokens_out":638,"duration_ms":13290,"significance":"If correct, the result supplies a theoretical account of how simple AR models can recover latent states in sequential data without explicit dynamics knowledge, linking AR prediction to Kalman filtering. The finite-sample guarantees on latent-state recovery constitute a concrete strength. The benign-landscape property is presented as one of the three main insights but is the least secure link for the central claim.","major_comments":[{"comment":"§4 (Landscape Analysis), Theorem 4.1: the claim that all stationary points are strict saddles or global minima is load-bearing for the ERM-to-Kalman-recovery argument, yet the provided analysis does not explicitly rule out non-strict stationary points whose hidden-layer span fails to cover the observable subspace of the underlying LDS; such points would invalidate the subsequent latent-state recovery guarantee even if the global minimum is Kalman-like.","section":"§4, Theorem 4.1"},{"comment":"§3.1 (Approximation), Eq. (8): the truncation-error bound for the AR approximation to the Kalman filter is stated to be independent of horizon, but the derivation appears to rely on the stability of the LDS; it is unclear whether the same bound holds uniformly when the process noise or observation noise covariance matrices are only partially known, which is the regime in which the finite-sample results are applied.","section":"§3.1, Eq. (8)"}],"minor_comments":[{"comment":"Notation for the hidden representation h_t is introduced without an explicit statement of its dimension relative to the true state dimension; a short clarifying sentence would help readers track the similarity transformation.","section":"§2"},{"comment":"Figure 2 caption does not indicate whether the plotted trajectories are single runs or averages over multiple random seeds; adding error bars or stating the number of trials would improve clarity of the empirical support.","section":"Figure 2"},{"comment":"The finite-sample bound in Theorem 5.3 is stated in terms of the sample size n, but the dependence on the LDS parameters (e.g., the spectral radius) is only implicit; making the dependence explicit would aid comparison with related LDS identification results.","section":"Theorem 5.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. We address each major comment below, indicating planned revisions where appropriate.","responses":[{"response":"We appreciate the referee highlighting the need for explicitness on this point. The proof of Theorem 4.1 proceeds by showing that any stationary point of the two-layer objective must align with the observable subspace of the LDS (otherwise a descent direction exists in the hidden-layer weights, violating stationarity). However, to make this argument fully explicit and rule out degenerate non-strict stationary points, we will add a supporting lemma in the revision that directly connects the stationarity condition to full span of the observable subspace. This will strengthen the link to the latent-state recovery result without altering the theorem statement.","revision_made":"yes","referee_comment":"[§4, Theorem 4.1] §4 (Landscape Analysis), Theorem 4.1: the claim that all stationary points are strict saddles or global minima is load-bearing for the ERM-to-Kalman-recovery argument, yet the provided analysis does not explicitly rule out non-strict stationary points whose hidden-layer span fails to cover the observable subspace of the underlying LDS; such points would invalidate the subsequent latent-state recovery guarantee even if the global minimum is Kalman-like."},{"response":"The bound in Eq. (8) follows from the geometric series summation enabled by stability of the state-transition matrix (spectral radius strictly less than one), which is an assumption stated in Section 3 and used throughout the finite-sample analysis. The derivation depends on the system matrices and Kalman gain but holds for any fixed positive-definite noise covariances; it does not require the covariances to be known, only that they are bounded (which is implicit in the fixed but unknown LDS setting). The finite-sample results treat the entire LDS, including covariances, as fixed unknowns. We will insert a short remark after Eq. (8) clarifying uniformity over bounded covariances under the stability assumption.","revision_made":"partial","referee_comment":"[§3.1, Eq. (8)] §3.1 (Approximation), Eq. (8): the truncation-error bound for the AR approximation to the Kalman filter is stated to be independent of horizon, but the derivation appears to rely on the stability of the LDS; it is unclear whether the same bound holds uniformly when the process noise or observation noise covariance matrices are only partially known, which is the regime in which the finite-sample results are applied."}],"tokens_in":1420,"tokens_out":539,"duration_ms":21458,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that two-layer linear auto-regressive models trained by ERM on data from partially observed linear dynamical systems recover hidden representations that match Kalman filter state estimates up to similarity, along with finite-sample guarantees on prediction, parameters, and latent recovery.\n\nWhat stands out as new is the explicit three-part argument: first that a Kalman filter can be approximated by a truncated AR model with bounded error, second that the non-convex two-layer loss has a benign landscape where every stationary point is either a strict saddle or a global minimum, and third the finite-sample bounds that follow from those two facts. The numerical simulations are a small plus for checking the theory on simple cases.\n\nThe landscape claim is the soft spot. The abstract presents it as one of the three main insights, yet the stress-test note is right that standard two-layer linear models on multi-step prediction losses can admit stationary points that do not span the observable subspace. If any such point is not a strict saddle, then gradient-based training need not reach the Kalman-like solution and the latent-recovery guarantee does not apply to what practitioners actually get. Without the full derivations it is impossible to tell how tightly they controlled the truncation error or ruled out bad stationary points.\n\nThe work is aimed at theorists who care about representation learning in linear sequence models and connections to filtering. It is coherent on its own terms and engages the relevant literature, so it deserves a serious referee even though the landscape analysis will need close scrutiny. I would send it out for review.","headline":"The paper links two-layer linear AR models to Kalman state recovery on partially observed LDS via ERM with finite-sample bounds, but the benign landscape step is the unverified load-bearing piece.","tokens_in":2430,"tokens_out":389,"would_cite":false,"duration_ms":12296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Two-layer linear auto-regressive models recover Kalman filter state estimates when trained by empirical risk minimization on data from partially observed linear dynamical systems.","keywords":["auto-regressive models","Kalman filter","latent state estimation","linear dynamical systems","empirical risk minimization","state recovery","partially observed systems"],"falsifier":"A concrete counter-example in which a two-layer linear auto-regressive model trained to a stationary point on data from a partially observed linear dynamical system yields hidden representations that do not match Kalman filter state estimates up to any similarity transformation.","tokens_in":2642,"feed_emoji":"","tokens_out":671,"duration_ms":9579,"temperature":0.7,"pith_summary":"This paper establishes that two-layer linear auto-regressive models, when trained by empirical risk minimization on sequences from partially observed linear dynamical systems, learn hidden representations that coincide up to similarity transformation with the state estimates of the optimal Kalman filter. The models achieve this without explicit knowledge of the underlying dynamics or states. The result rests on three elements: the Kalman filter admits an auto-regressive approximation with bounded truncation error, the non-convex training landscape is benign with only global minima and strict saddles as stationary points, and finite-sample bounds control prediction error, parameter error, and state recovery error. A sympathetic reader would care because the finding supplies a concrete mechanism by which simple auto-regressive architectures implicitly perform classical optimal filtering on linear systems.","feed_headline":"AR models recover Kalman states without knowing dynamics","feed_subtitle":"Two-layer linear auto-regressive models trained on partially observed linear systems learn hidden representations that match optimal filter","key_machinery":"The two-layer linear auto-regressive model whose hidden layer is trained by empirical risk minimization to produce next-step predictions, thereby recovering latent states as a byproduct.","core_discovery":"When trained by empirical risk minimization on data from partially observed linear dynamical systems, two-layer linear auto-regressive models naturally learn to approximate Kalman filtering. In particular, the learned hidden representation coincides, up to a similarity transformation, with the state estimates produced by the optimal Kalman filter, even though the model has no explicit knowledge of the underlying dynamics or state. The result follows from establishing that the Kalman filter is well approximated by an auto-regressive model with bounded truncation error, that the two-layer optimization landscape is benign despite non-convexity, and that finite-sample guarantees hold for predict","pith_inferences":["The same benign-landscape argument may apply to deeper linear or mildly nonlinear auto-regressive architectures, suggesting a route to implicit filtering in higher-order systems.","Auto-regressive training could serve as a model-free alternative to explicit Kalman-filter design whenever the system is approximately linear.","The result raises the question of whether similar recovery guarantees exist for other classical estimators, such as particle filters, when the observation model is replaced by a learned auto-regressive layer."],"forward_implications":["Finite-sample guarantees hold simultaneously for prediction error, parameter estimation error, and latent state recovery.","The Kalman filter itself admits a bounded-error auto-regressive approximation.","All stationary points of the two-layer training problem are either strict saddles or global minima.","Numerical simulations confirm that the learned latent representations recover the Kalman state estimates."],"fun_headline_variants":["Two-layer AR models recover Kalman states from data","Linear AR nets learn to approximate Kalman filtering","AR models estimate latent states matching optimal filter","Two-layer linear models learn hidden Kalman representations"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The two-layer non-convex optimization landscape is benign, with every stationary point being either a strict saddle or a global minimum.","fun_headline_variants_meta":{"raw":{"variants":["Two-layer AR models recover Kalman states from data","Linear AR nets learn to approximate Kalman filtering","AR models estimate latent states matching optimal filter","Two-layer linear models learn hidden Kalman representations"]},"model":"grok-4.3","cost_usd":0.003991,"raw_usage":{"total_tokens":2046,"prompt_tokens":684,"num_sources_used":0,"completion_tokens":54,"cost_in_usd_ticks":39912000,"prompt_tokens_details":{"text_tokens":684,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1308,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":684,"tokens_out":54,"duration_ms":9823,"temperature":1.0,"reasoning_tokens":1308,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T09:56:43.066079+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A concrete counter-example in which a two-layer linear auto-regressive model trained to a stationary point on data from a partially observed linear dynamical system yields hidden representations that do not match Kalman filter state estimates up to any similarity transformation.","supporting_citations":[],"review_version":1}