Pith. sign in

REVIEW 3 major objections 6 minor 17 references

Simultaneous Latent State Estimation and Latent Linear Dynamics Discovery from Image Observations

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A conditional normalizing flow supplies the observation likelihood so that a particle filter can jointly estimate latent states and latent linear dynamics from images alone.

desk verdict Honest but unfinished architecture sketch: the central training objective is not computable as written, and the experiments are far too weak to support the claims. read the letter →

arxiv 2501.01339 v2 pith:N3SREYPO submitted 2025-01-02 cs.LG

Nikita Kostin This is my paper · ORCID
classification cs.LG
keywords latentstateestimationnormalizingflowsparticlefilterlineardynamicsdiscoveryimageobservationsposteriorfilteringdistributionjointtraininghiddenMarkovmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a filtering method, NFPF, that treats image observations as outputs of a latent linear dynamical system. The method uses a conditional normalizing flow to compute the observation likelihood p(yt|xt), and trains the flow, the mean parametrization of its base distribution, and the linear dynamics matrices At and Bt jointly by maximizing the log-likelihood of the observed images. If this works, an agent could recover both a posterior distribution over latent states and the linear dynamics that generated them from raw images alone, with no manually specified observation model. The report includes a CartPole demonstration in which a bootstrap particle filter uses the learned likelihood to produce latent trajectories.

What carries the argument

The central object is the conditional normalizing flow used as an observation model: an invertible network gθ that maps the latent state x and the image y to a simple Gaussian variable ŷ, so that the observation density is evaluated exactly by the change-of-variables formula p(yt|xt) = p(ŷt|µϕ(xt), Σ)|det J_{$gθ^{{-1}}$}(yt)|. This identity is what makes the whole pipeline differentiable: gradients from the image likelihood flow back through the flow, through the mean parametrization, and through the linear dynamics matrices, enabling joint training of all parameters. In deployment, the resulting likelihood is plugged into the bootstrap particle filter weight update to obtain an approximate posterior filtering distribution.

What would settle it

On a synthetic image-sequence benchmark with known ground-truth latent states (for example, a pendulum rendered from its true angle and angular velocity), run the proposed joint training from several random initializations. If the learned latent trajectories do not track the true states and the learned matrix A does not predict the true next latent state better than a random linear model, then the claim that images alone yield latent state and linear dynamics is not supported.

Watch

Extended reading notes

Core claim

The central claim is that the observation likelihood p(yt|xt) of a hidden Markov model with latent linear dynamics can be learned from images by a conditional normalizing flow gθ whose base distribution is Gaussian with mean µϕ(xt) produced by a convolutional network. The dynamics are parametrized as xt = Aψ xt−1 + Bψ ut−1, where fψ outputs the matrices from the current latent state (and optionally the control). Maximizing log p(Y|X) over a batch, expanded via the change-of-variables formula, updates θ, ϕ, and ψ in one differentiable objective; the same likelihood then feeds the particle-filter weight update. The paper thus claims simultaneous latent state estimation and latent linear dynamics discovery from image observations alone, with stability constraints such as spectral radius less than one available for the learned matrices.

Load-bearing premise

The training objective maximizes log p(Y|X) after substituting the latent state xt by the deterministic dynamics recurrence, but the paper never explains how latent states are obtained from image observations during training, so the objective may not be computable for actual image data.

Editorial extensions

If this is right

  • From image observations alone, the method yields both an approximate posterior filtering distribution p(xt|y1:t) and time-dependent linear dynamics matrices At and Bt.
  • The learned observation likelihood can be inserted into any particle-filter weight update, not only the bootstrap variant used in the experiments.
  • Because the dynamics matrices are produced by a network fψ from the latent state, the formulation extends to state-dependent and control-dependent linear dynamics.
  • The paper identifies latent-space LQR control and the inclusion of controllability constraints as natural next steps if the learned matrices are to be used for control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The training objective in Eq. (9)-(11) treats the latent states xt as known deterministic nodes, yet the paper specifies no encoder or inference step for obtaining them from images during training; a variational or particle-based inference layer would make the objective well-posed and would be a natural completion of the method.
  • The paper's CartPole plots show two-particle filter trajectories against true latent states, but the report itself notes the evidence is thin; a stronger test would be a synthetic-image benchmark with known ground-truth latent states and quantitative trajectory error.
  • If the conditional flow were replaced by a dimension-changing flow, the method could drop the mean-parametrization CNN entirely and learn the observation model as a fully invertible encoder-decoder.
  • Comparing NFPF directly with SINDy-based autoencoders on the same images would clarify whether exact observation likelihoods change the discovered dynamics or only the posterior uncertainty estimates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a method called Normalizing Flows based Particle Filter (NFPF) for simultaneous latent state estimation and latent linear dynamics discovery from image observations. It reviews Kalman filtering, particle filtering, autoencoders, SINDy, and normalizing flows, then derives a training objective in Section 3.1 that is intended to jointly train a normalizing flow observation likelihood, a mean parametrization, and linear dynamics matrices. The experiments apply the method to CartPole image observations and show latent trajectories from a bootstrap particle filter with two particles, without quantitative evaluation or comparisons.

Significance. If the proposed training procedure were well defined, the idea of using a normalizing flow to supply an observation likelihood to a particle filter while jointly learning latent linear dynamics would address a real and useful problem. The paper also offers a readable survey of the relevant components and is unusually candid about its limitations. However, the central algorithmic claim is not supported: the training objective is written over unobserved latent states with no specified inference procedure, and the experiments do not provide quantitative evidence. The significance is therefore not established in the current form.

major comments (3)
  1. [Section 3.1, Eqs. (9)-(11)] The training objective in Eq. (9) is written as maximizing log p(Y|X) over a sequence of latent states X, but X is never observed and the paper specifies no encoder, variational posterior, importance-sampling distribution, or initial-state distribution p(x_k) to make this objective computable. Replacing x_t by A_psi x_{t-1} + B_psi u_{t-1} in Eq. (10) does not resolve the problem, because x_{t-1} is itself unobserved and no inference step is defined. Consequently, Eq. (11) is not actually computable from image observations alone, and the claimed joint training of g_theta, mu_phi, and the linear matrices A_t and B_t is not supported. This is the central gap in the paper.
  2. [Section 3.1, Eq. (11)] Even setting aside the missing inference procedure, the split of the minimization into separate terms over (phi, psi) and theta is algebraically incorrect. The first term contains y_hat_t = g_theta^{-1}(y_t), so it depends on theta through the inverse of the normalizing flow. The objective therefore cannot be minimized for phi and psi independently of theta as written; the claimed decoupling in Eq. (11) is false.
  3. [Section 4, Figures 4 and 5] The experimental evidence is insufficient to support the central claims. The evaluation uses only 10 training trajectories and a particle filter with 2 particles, reports no quantitative error metrics, no baselines, and no train/test split, and the plots are presented for visual inspection only. The Discussion itself lists these missing elements, including controllability restrictions, LQR evaluation, and better proposal distributions. Without a well-defined training procedure and quantitative validation, the empirical claims for latent state estimation and dynamics discovery are not established.
minor comments (6)
  1. [Section 2.1.1, Eq. (4)] The integral in Eq. (4) writes dx_{k-1} in the text; this should be dx_{t-1}.
  2. [Section 2.1.1] The statement that the Bayesian filter integrals 'can be computed analytically only when g(x_t) is a linear function' is imprecise, since the EKF is then described as a way to compute them; the EKF linearizes the observation function and yields approximate results, not analytical ones.
  3. [Section 4, Figures 4 and 5] Figure 5 is captioned as showing 100 dimensions while Section 4 states that the latent dimensionality was set to 4; the relation between these two figures should be clarified.
  4. [Section 3.1 and Figure 2] The initial latent state distribution p(x_k) and the covariance Sigma appear in the architecture figure but are not defined in the training objective; the paper should specify how these quantities are chosen or learned.
  5. [Section 2.3 and Section 4] The Methods section motivates comparison with SINDy, but no comparative experiments are reported; the paper should either include such a comparison or state that it is left to future work.
  6. [Abstract and Introduction] The text has a report-style tone, e.g., 'This report tries to conclude previous works'; the abstract and introduction should be revised to a journal-paper format.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the core equations are a standard conditional-normalizing-flow likelihood decomposition and particle-filter update; the paper's algebraic, inference, and validation flaws are correctness issues, not circular reductions.

full rationale

The load-bearing equations are not self-referential. Eq. (9) applies the normalizing-flow change-of-variables formula to write p(y_t|x_t) as p(ŷ_t|μ_φ(x_t),Σ)·|det J_{g_θ^{-1}}(y_t)|; this is a density decomposition, not a definition of the target in terms of itself. Eq. (10) substitutes the assumed linear latent recursion x_t = A_ψ x_{t-1}+B_ψ u_t into that likelihood, which is a model assumption rather than a circular reduction. Eq. (11)'s split into separate φ,ψ and θ minimizations is algebraically invalid because ŷ_t = g_θ^{-1}(y_t) appears in the first term, so θ is not decoupled; however, an incorrect separation of variables is a correctness defect, not a case of a prediction being equal to its input by construction. The experimental section is admittedly weak—"The resulting plots from the previous section might not convince the reader", "Only 10 trajectories were used to train NF observation likelihood", "there is no easy way to train NFPF using batches of observation rather than single images"—but these limitations undermine evidence quality and reproducibility, not circularity. All cited techniques (normalizing flows, conditional NF, particle filters) are external works; there is no load-bearing self-citation or imported uniqueness theorem. No step in the paper exhibits the required reduction of a claimed prediction to a fitted parameter or to a self-citation, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central claim rests on the ability to train an observation likelihood and dynamics jointly from images, but the most fragile ingredient is the unspecified treatment of unobserved latent states in the training objective. The remaining assumptions are standard domain assumptions for latent linear dynamical systems and normalizing flows. No new physical entities are introduced.

free parameters (5)
  • Latent state dimensionality = 4 (CartPole state dimension)
    Chosen by hand to match CartPole; this choice determines what the model can represent and is not learned or cross-validated. Section 4.
  • Observation likelihood covariance Sigma = not reported
    Constant covariance of the Gaussian base distribution in the conditional NF observation model; its value is not stated in the paper. Section 3.1.
  • Number of particles in evaluation filter = 2
    The qualitative plots use only 2 particles, far too few for reliable filtering, and no sensitivity analysis is given. Section 4.
  • Number of training trajectories = 10
    The discussion admits only 10 trajectories were used to train the NF observation likelihood due to compute. Section 5.
  • Neural network weights (theta, phi, psi) = not reported
    The flow, mean network, and dynamics network are fit to data, but no architecture details, weight counts, or checkpoints are given. Section 3.1.
assumptions (5)
  • domain assumption Latent dynamics are Markov and approximately linear; training ignores process noise and uses deterministic recurrence x_t = Aψ x_{t-1} + Bψ u.
    Eq (2) introduces Gaussian process noise, but Eqs (10)-(11) remove it during training; no evidence of linearity is given.
  • domain assumption A conditional normalizing flow with invertible same-dimensional transform can accurately represent p(y|x) for high-dimensional images.
    Section 3.1; no capacity or validation analysis is provided.
  • ad hoc to paper The training objective can be optimized over the latent states X without an inference model.
    Section 3.1; this is the unstated and structurally load-bearing assumption that makes Eq (11) appear computable.
  • ad hoc to paper Unit Frobenius norm on Aψ and Bψ ensures asymptotic stability.
    Section 4; the stated guarantee is mathematically too strong, since spectral radius can equal 1.
  • domain assumption CartPole dynamics are close to linear in a 4-dimensional latent coordinate system.
    Section 4; the paper assumes the true state space dimension (4) is an appropriate latent dimension but does not test this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simultaneous Latent State Estimation and Latent Linear Dynamics Discovery from Image Observations." pith.science (2026). https://pith.science/paper/N3SREYPO

@misc{pith2026250101339,
  author       = {Pith},
  title        = {Pith review of: Simultaneous Latent State Estimation and Latent Linear Dynamics Discovery from Image Observations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3SREYPO}},
  note         = {Machine review of arXiv:2501.01339}
}
read the original abstract

The problem of state estimation has a long history with many successful algorithms that allow analytical derivation or approximation of posterior filtering distribution given the noisy observations. This report tries to conclude previous works to resolve the problem of latent state estimation given image-based observations and also suggests a new solution to this problem.

Figures

Figures reproduced from arXiv: 2501.01339 by the authors.

Figure 1
Figure 1. Autoencoder architecture: here x(t) stands for high-dimensional observation, z(t) stands for latent state and ˆx(t) is reconstructed observation by the de￾coder ψ. The way to use AE architecture for approximating the observation model is straightforward: decoder ψ plays the role of the observation model. Training of AE requires only observations {yt} N t=1. The only question is how can we for￾mulate latent state xt … view at source ↗
Figure 2
Figure 2. Proposed model architecture: yt denotes image observation, yˆt de￾notes flow base distributed variables (Gaussian), xt denotes latent state, At denotes dynamics matrix, ut denotes control signal, Bt denotes control matrix, µ0 and Σ0 are mean and covariance of the first latent state. Linear dynamics matrices are obtained using dynamics network fψ, gθ denotes NF transforma￾tion, while µϕ stands for mean parametrizatio… view at source ↗
Figure 3
Figure 3. RGB observation of CartPole used in the experiments. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Particle filter with 2 particles NF observation likelihood against true [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: PF with 2 particles NF observation likelihood against true state evo [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 9 canonical work pages

  1. [1]

    A Non-Linear Kalman Filter for track parameters estimation in High Energy Physics

    Xiaocong Ai et al. A Non-Linear Kalman Filter for track parameters esti- mation in High Energy Physics. 2021. doi: 10.48550/ARXIV.2112.09470. url: https://arxiv.org/abs/2112.09470

    work page Pith review arXiv doi:10.48550/arxiv.2112.09470 2021

  2. [2]

    Stability of discrete dynamical systems

    Maria Barbarossa. Stability of discrete dynamical systems . 2011. url: http : / / m6auer . ma . tum . de / foswiki / pub / M6 / Lehrstuhl / MatBio1 _ 201011WS/discrete_dynamics.pdf

    work page 2011

  3. [3]

    Openai gym

    Greg Brockman et al. “Openai gym”. In: arXiv preprint arXiv:1606.01540 (2016)

    arXiv 2016

  4. [4]

    Discover- ing governing equations from data by sparse identification of nonlinear dynamical systems

    Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. “Discover- ing governing equations from data by sparse identification of nonlinear dynamical systems”. In: Proceedings of the National Academy of Sciences 113.15 (2016), pp. 3932–3937. doi: 10.1073/pnas.1517384113 . eprint: https : / / www . pnas . org / doi / pdf / 10 . 1073 / pnas . 1517384113. u...

    doi:10.1073/pnas.1517384113 2016

  5. [5]

    Data-driven discovery of coordinates and gov- erning equations

    Kathleen Champion et al. Data-driven discovery of coordinates and gov- erning equations. 2019. doi: 10.48550/ARXIV.1904.02107. url: https: //arxiv.org/abs/1904.02107

    arXiv doi:10.48550/arxiv.1904.02107 2019

  6. [6]

    Differentiable Particle Filters through Conditional Normalizing Flow

    Xiongjie Chen, Hao Wen, and Yunpeng Li. Differentiable Particle Fil- ters through Conditional Normalizing Flow . 2021. doi: 10.48550/ARXIV. 2107.00488. url: https://arxiv.org/abs/2107.00488

    work page Pith review arXiv doi:10.48550/arxiv.2107.00488 2021

  7. [7]

    Normalizing Flows Across Dimensions

    Edmond Cunningham et al. Normalizing Flows Across Dimensions . 2020. doi: 10 . 48550 / ARXIV . 2006 . 13070. url: https : / / arxiv . org / abs / 2006.13070

    arXiv doi:10.48550/arxiv.2006.13070 2020

  8. [8]

    Dorf and Robert H

    Richard C. Dorf and Robert H. Bishop. Modern Control Systems . 9th. USA: Prentice-Hall, Inc., 2000. isbn: 0130306606

    work page 2000

Show all 17 references
  1. [9]

    Optimal Gene Filtering for Single-Cell data (OGFSC)—a gene filtering algorithm for single-cell RNA-seq data

    Jie Hao et al. “Optimal Gene Filtering for Single-Cell data (OGFSC)—a gene filtering algorithm for single-cell RNA-seq data”. In: Bioinformat- ics 35.15 (Dec. 2018), pp. 2602–2609. issn: 1367-4803. doi: 10 . 1093 / bioinformatics/bty1016. eprint: https://academic.oup.com/bioin...

    2018 doi

  2. [10]

    A New Approach to Linear Filtering and Prediction Prob- lems

    R. E. Kalman. “A New Approach to Linear Filtering and Prediction Prob- lems”. In: Journal of Basic Engineering 82.1 (Mar. 1960), pp. 35–45. issn: 0021-9223. doi: 10.1115/1.3662552. eprint: https://asmedigitalcollection. asme.org/fluidsengineering/article- pdf/82/1/35/5518977/3...

    1960 doi

  3. [11]

    Funnels: Exact maximum likelihood with dimension- ality reduction

    Samuel Klein et al. Funnels: Exact maximum likelihood with dimension- ality reduction. 2021. doi: 10.48550/ARXIV.2112.08069 . url: https: //arxiv.org/abs/2112.08069. 10

    2021 doi arXiv

  4. [12]

    Playing Atari with Deep Reinforcement Learning

    Volodymyr Mnih et al. Playing Atari with Deep Reinforcement Learning

  5. [13]

    Normalizing Flows for Probabilistic Modeling and Inference

    George Papamakarios et al. “Normalizing Flows for Probabilistic Modeling and Inference”. In: (2019). doi: 10 . 48550 / ARXIV . 1912 . 02762. url: https://arxiv.org/abs/1912.02762

    2019 doi arXiv

  6. [14]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Adam Paszke et al. “PyTorch: An Imperative Style, High-Performance Deep Learning Library”. In: Advances in Neural Information Process- ing Systems 32 . Ed. by H. Wallach et al. Curran Associates, Inc., 2019, pp. 8024–8035. url: http://papers.neurips.cc/paper/9015-pytorch- an-i...

    2019

  7. [15]

    Cambridge University Press, 2013

    Simo S¨ arkk¨ a.Bayesian Filtering and Smoothing . Cambridge University Press, 2013

    2013

  8. [16]

    Learning Likelihoods with Conditional Normal- izing Flows

    Christina Winkler et al. Learning Likelihoods with Conditional Normal- izing Flows . 2019. doi: 10 . 48550 / ARXIV . 1912 . 00042. url: https : //arxiv.org/abs/1912.00042. 11

    2019 doi arXiv

  9. [2013]

    url: https://arxiv.org/abs/ 1312.5602

    doi: 10.48550/ARXIV.1312.5602. url: https://arxiv.org/abs/ 1312.5602

    doi arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.