REVIEW 2 major objections 3 minor
Two-Layer Linear Auto-Regressive Models Estimate Latent States
T0 review · 2 major / 3 minor · reviewed 2026-06-27 · grok-4.3
Pith's one-line read Two-layer linear auto-regressive models recover Kalman filter state estimates when trained by empirical risk minimization on data from partially observed linear dynamical systems.
desk verdict The paper links two-layer linear AR models to Kalman state recovery on partially observed LDS via ERM with finite-sample bounds, but the benign landscape step is the unverified load-bearing piece. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The two-layer linear auto-regressive model whose hidden layer is trained by empirical risk minimization to produce next-step predictions, thereby recovering latent states as a byproduct.
What would settle it
A concrete counter-example in which a two-layer linear auto-regressive model trained to a stationary point on data from a partially observed linear dynamical system yields hidden representations that do not match Kalman filter state estimates up to any similarity transformation.
Extended reading notes
Core claim
When trained by empirical risk minimization on data from partially observed linear dynamical systems, two-layer linear auto-regressive models naturally learn to approximate Kalman filtering. In particular, the learned hidden representation coincides, up to a similarity transformation, with the state estimates produced by the optimal Kalman filter, even though the model has no explicit knowledge of the underlying dynamics or state. The result follows from establishing that the Kalman filter is well approximated by an auto-regressive model with bounded truncation error, that the two-layer optimization landscape is benign despite non-convexity, and that finite-sample guarantees hold for predict
Load-bearing premise
The two-layer non-convex optimization landscape is benign, with every stationary point being either a strict saddle or a global minimum.
Editorial extensions
If this is right
- Finite-sample guarantees hold simultaneously for prediction error, parameter estimation error, and latent state recovery.
- The Kalman filter itself admits a bounded-error auto-regressive approximation.
- All stationary points of the two-layer training problem are either strict saddles or global minima.
- Numerical simulations confirm that the learned latent representations recover the Kalman state estimates.
Reading between the lines
- The same benign-landscape argument may apply to deeper linear or mildly nonlinear auto-regressive architectures, suggesting a route to implicit filtering in higher-order systems.
- Auto-regressive training could serve as a model-free alternative to explicit Kalman-filter design whenever the system is approximately linear.
- The result raises the question of whether similar recovery guarantees exist for other classical estimators, such as particle filters, when the observation model is replaced by a learned auto-regressive layer.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that two-layer linear auto-regressive models trained by empirical risk minimization on data from partially observed linear dynamical systems learn hidden representations that coincide, up to similarity transformation, with the state estimates of the optimal Kalman filter. This follows from three insights: (i) the Kalman filter is well-approximated by an AR model with bounded truncation error, (ii) the non-convex two-layer optimization landscape is benign (all stationary points are strict saddles or global minima), and (iii) finite-sample guarantees on prediction error, parameter estimation error, and latent-state recovery. Numerical simulations are provided in support.
Significance. If correct, the result supplies a theoretical account of how simple AR models can recover latent states in sequential data without explicit dynamics knowledge, linking AR prediction to Kalman filtering. The finite-sample guarantees on latent-state recovery constitute a concrete strength. The benign-landscape property is presented as one of the three main insights but is the least secure link for the central claim.
major comments (2)
- [§4, Theorem 4.1] §4 (Landscape Analysis), Theorem 4.1: the claim that all stationary points are strict saddles or global minima is load-bearing for the ERM-to-Kalman-recovery argument, yet the provided analysis does not explicitly rule out non-strict stationary points whose hidden-layer span fails to cover the observable subspace of the underlying LDS; such points would invalidate the subsequent latent-state recovery guarantee even if the global minimum is Kalman-like.
- [§3.1, Eq. (8)] §3.1 (Approximation), Eq. (8): the truncation-error bound for the AR approximation to the Kalman filter is stated to be independent of horizon, but the derivation appears to rely on the stability of the LDS; it is unclear whether the same bound holds uniformly when the process noise or observation noise covariance matrices are only partially known, which is the regime in which the finite-sample results are applied.
minor comments (3)
- [§2] Notation for the hidden representation h_t is introduced without an explicit statement of its dimension relative to the true state dimension; a short clarifying sentence would help readers track the similarity transformation.
- [Figure 2] Figure 2 caption does not indicate whether the plotted trajectories are single runs or averages over multiple random seeds; adding error bars or stating the number of trials would improve clarity of the empirical support.
- [Theorem 5.3] The finite-sample bound in Theorem 5.3 is stated in terms of the sample size n, but the dependence on the LDS parameters (e.g., the spectral radius) is only implicit; making the dependence explicit would aid comparison with related LDS identification results.
Simulated Author's Rebuttal
We thank the referee for the careful and constructive review. We address each major comment below, indicating planned revisions where appropriate.
read point-by-point responses
-
Referee: [§4, Theorem 4.1] §4 (Landscape Analysis), Theorem 4.1: the claim that all stationary points are strict saddles or global minima is load-bearing for the ERM-to-Kalman-recovery argument, yet the provided analysis does not explicitly rule out non-strict stationary points whose hidden-layer span fails to cover the observable subspace of the underlying LDS; such points would invalidate the subsequent latent-state recovery guarantee even if the global minimum is Kalman-like.
Authors: We appreciate the referee highlighting the need for explicitness on this point. The proof of Theorem 4.1 proceeds by showing that any stationary point of the two-layer objective must align with the observable subspace of the LDS (otherwise a descent direction exists in the hidden-layer weights, violating stationarity). However, to make this argument fully explicit and rule out degenerate non-strict stationary points, we will add a supporting lemma in the revision that directly connects the stationarity condition to full span of the observable subspace. This will strengthen the link to the latent-state recovery result without altering the theorem statement. revision: yes
-
Referee: [§3.1, Eq. (8)] §3.1 (Approximation), Eq. (8): the truncation-error bound for the AR approximation to the Kalman filter is stated to be independent of horizon, but the derivation appears to rely on the stability of the LDS; it is unclear whether the same bound holds uniformly when the process noise or observation noise covariance matrices are only partially known, which is the regime in which the finite-sample results are applied.
Authors: The bound in Eq. (8) follows from the geometric series summation enabled by stability of the state-transition matrix (spectral radius strictly less than one), which is an assumption stated in Section 3 and used throughout the finite-sample analysis. The derivation depends on the system matrices and Kalman gain but holds for any fixed positive-definite noise covariances; it does not require the covariances to be known, only that they are bounded (which is implicit in the fixed but unknown LDS setting). The finite-sample results treat the entire LDS, including covariances, as fixed unknowns. We will insert a short remark after Eq. (8) clarifying uniformity over bounded covariances under the stability assumption. revision: partial
Circularity Check
No significant circularity; derivation self-contained
full rationale
The paper establishes the Kalman approximation by AR models, the benign landscape property, and finite-sample guarantees as separate insights derived from properties of linear dynamical systems and empirical risk minimization. The central claim that the learned hidden representation coincides with Kalman estimates follows from these without any reduction by construction to fitted parameters, self-definitional equations, or load-bearing self-citations. No patterns matching the enumerated circularity kinds are exhibited in the abstract or described contributions.
Assumptions & free parameters
assumptions (2)
- domain assumption The underlying process is a partially observed linear dynamical system.
- domain assumption The two-layer linear auto-regressive model has a benign optimization landscape where all stationary points are strict saddles or global minima.
Cite this review
Pith. "Pith review of Two-Layer Linear Auto-Regressive Models Estimate Latent States." pith.science (2026). https://pith.science/paper/OAQGB7OG
@misc{pith2026260612691,
author = {Pith},
title = {Pith review of: Two-Layer Linear Auto-Regressive Models Estimate Latent States},
year = {2026},
howpublished = {\url{https://pith.science/paper/OAQGB7OG}},
note = {Machine review of arXiv:2606.12691}
}
read the original abstract
Auto-regressive models have emerged as powerful tools for sequential data, from language to video. Understanding how and why these models learn latent representations remains an open theoretical question. In this work, we demonstrate that when trained by empirical risk minimization on data from partially observed linear dynamical systems, two-layer linear auto-regressive models naturally learn to approximate Kalman filtering. In particular, we show that the learned hidden representation coincides, up to a similarity transformation, with the state estimates produced by the optimal (Kalman) filter, even though the model has no explicit knowledge of the underlying dynamics or state. The result follows from three main insights. First, we establish that the Kalman filter is well approximated by an auto-regressive model with bounded truncation error. Second, we show that despite non-convexity, the two-layer optimization landscape is benign, i.e., all stationary points are either strict saddles or global minima. Finally, as our main contributions, we provide finite-sample guarantees on prediction error, parameter estimation error, and latent state recovery. Numerical simulations support the theoretical results and demonstrate that the latent representations of auto-regressive models recover state estimates.
Figures
Reviewed June 27, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.