{"id":"4eb5740a-b1f7-4473-97ba-cbdf2a85796c","arxiv_id":"2505.23569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new unsupervised model, RP-GSSM, learns Markovian Gaussian latent dynamics from time series via maximum likelihood without an observation decoder, and outperforms baselines on video tasks with distractors.","lead":"The paper introduces RP-GSSM, a probabilistic method that learns latent state dynamics from time series without training a decoder to reconstruct observations. The model estimates Gaussian latent states with Kalman smoothing, and reports better latent recovery than VAE and contrastive baselines, especially when video backgrounds are distracting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact-ML and exact-inference claims rest on an unchecked F≈pη approximation that also makes the Kalman E-step valid; if it fails, G is not the RPM likelihood and the empirical results may not reflect the stated model.","rationale":"The central claim has two parts: RP-GSSM is a valid probabilistic model learned by maximum likelihood, and it supports exact inference while outperforming alternatives. The derivations are careful and the experiments are broad, though no repository or commit hash is given despite the stated code release. The most load-bearing weakness is not that G fails to be a lower bound—Jensen's inequality guarantees it is—but that the exactness claims require the F≈pη approximation. This approximation is asserted via a large-data in-model limit and is never checked at finite N. It does essential work: it removes the qn-dependence from Γ̃tn, making the E-step ordinary Kalman smoothing, and it closes the Jensen gap so that G equals the original free energy. If F deviates from pη, the objective being maximized is a different functional and the reported Kalman qn is not the posterior of the stated model. I partly agree with the reader: the F≈pη assumption is indeed the weak point, but the sharper formulation is that G remains a valid bound even without it; what fails without it is the exact-ML and exact-inference interpretation and the precise E-step. The proposed diagnostic settles whether the assumption holds in the actual training regime and whether the empirical gains are attributable to the model as claimed. The verdict should therefore remain CONDITIONAL: neither accept as-is nor reject, pending this check.","tokens_in":17089,"tokens_out":14204,"duration_ms":141556,"concrete_test":"Train RP-GSSM on a small in-model linear-Gaussian dataset (e.g. D_Z=1, T=10, N=50) and, for each sequence, compute the exact RPM log-likelihood log p̃θ(x) by numerical integration over z, exploiting that f and F are Gaussian mixtures. Compare it with the optimized auxiliary free energy G and compute per-time gaps Δt = E_q log(pη/F) + log E_q(F/pη) and KL(qn || p̃θ(z|x)). If these are not near zero at convergence, G is not tight, the Kalman-smoothed qn is not the exact RPM posterior, and maximizing G is not maximizing the RPM likelihood. As a second check, rerun the main linear experiment with exact fixed-point handling of the qn-dependence in Γ̃tn; if reported R2 values change materially, the empirical claims hinge on the approximation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"G in Eq. (10) is a valid lower bound to the RPM free energy for any positive g, so the F≈pη assumption is not needed for validity. Its load-bearing role is tightness plus the E-step. Setting gtn=qn/pη makes Γtn=∫F qn/pη dz depend on qn, and the final loss in Appendix B.2 contains log Γ̃tn with qn natural parameters. The paper claims qn is computed exactly by standard Kalman smoothing because G has the form of a linear-Gaussian free energy; that is only correct if the qn-dependence of Γ̃tn is ignored, i.e. if F≈pη makes Γ≈1. Otherwise the E-step is a coupled fixed-point problem, and the coordinate-ascent claim is not established. Additionally, the true RPM posterior is proportional to pη(z)∏f/F, a ratio of a Gaussian to a mixture of Gaussians, so Kalman smoothing returns a variational qn, not an exact posterior. The paper asserts F→pη as a large-data in-model limit but reports no finite-sample diagnostic; without it, the central claims of maximum-likelihood learning and exact inference are unsupported, and the distractor-robustness results may be driven by the approximate objective rather than by the RPM structure itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the recognition-parametrized Gaussian state space model (RP-GSSM), an unsupervised method for learning latent dynamical structure from time series without an explicit generative decoder. The model combines a linear-Gaussian Markov prior over latents with arbitrarily nonlinear recognition factors that map observations to Gaussian distributions on latents, following the recognition-parametrized model (RPM) framework. The authors derive a free-energy objective, apply an interior variational bound with auxiliary factors, and assume that the empirical average of recognition factors F_phi_t(z_t) is approximately equal to the prior p_eta(z_t), which allows them to replace the intractable log F term and to claim that the E-step reduces to exact Kalman smoothing. Experiments on linear dynamical systems, pendulum and double-pendulum videos, and object tracking, with and without background distractors, show that RP-GSSM recovers latent variables more accurately than several VAE-based and contrastive baselines. The paper also demonstrates that an auxiliary generative model trained on RP-GSSM latents can ignore distractors and that future latent states can be predicted without auxiliary objectives.","tokens_in":17431,"tokens_out":10207,"duration_ms":99043,"significance":"If the central approximation is valid or adequately controlled, the RP-GSSM is a conceptually interesting contribution: it offers a principled probabilistic objective for learning latent dynamics without reconstruction, with tractable Gaussian inference via Kalman smoothing. The empirical results are broad, covering several dynamical systems and distractor conditions, and the paper compares against multiple strong baselines. The paper also ships useful details on implementation, hyperparameters, and runtime. However, the significance is conditional on an unverified approximation, F_phi_t(z_t) ≈ p_eta(z_t), which is load-bearing for both the maximum-likelihood claim and the exact-inference claim. The manuscript would benefit from either empirical validation of this approximation or a revised, more modest statement of what is being optimized and what inference is exact.","major_comments":[{"comment":"The approximation F_phi_t(z_t) ≈ p_eta(z_t) is load-bearing for the claim that the E-step is exact Kalman smoothing, but the paper does not verify it. The expression for log Gamma-tilde_tn in Appendix B.2 contains the natural parameters eta^n_q_t of the variational distribution q_n, so when F differs from p_eta, Gamma is not 1 and the objective is not the linear-Gaussian free energy with q-independent constants that the main text describes. In that case the 'E-step' would require solving a coupled fixed-point problem in q_n and Gamma-tilde, and the coordinate-ascent claim is not established. The paper asserts F → p_eta as a large-data in-model limit (citing Walker et al., 2023) but reports no finite-sample diagnostic of this convergence, such as a KL divergence or the size of log Gamma. Without such evidence, the central claim that the method performs maximum-likelihood learning with exact inference is unsupported.","section":"Section 3, after Eq. (6); Appendix B.2"},{"comment":"The claim of exact inference is overstated. Under the RPM joint, the posterior is proportional to p_eta(z) * product_t f_phi_t(z_t|x_t) / F_phi_t(z_t), which is in general a ratio of a Gaussian to a mixture of Gaussians and is not Gaussian. Standard Kalman smoothing returns the q_n that maximizes the auxiliary free energy G, which is a variational quantity, not the exact posterior. Exactness holds only when F_phi_t ≈ p_eta, the same unquantified assumption identified above. The paper should either provide evidence for this condition or revise the inference claim to describe variational inference within a Gaussian family that is exact only under the stated approximation.","section":"Abstract; Section 3, paragraphs on exact inference"},{"comment":"The parametrization p(z_1)=N(0,I) and transition covariance I-AA^T is motivated by Lemma 2.1, but the lemma only guarantees the existence of an equivalent model whose stationary distribution is N(0,I), not that the process starts in that distribution. The transformed initial distribution in Eq. (7) is N(Gm_1+c, GQ_1G^T), which is generally not N(0,I). Thus the paper's stationary-from-time-0 choice is an additional modeling restriction, and the statement that 'no generality is lost' (Section 3) is too strong for finite-horizon sequences. This does not invalidate the method, but the framing should be qualified.","section":"Section 3, Lemma 2.1 usage"}],"minor_comments":[{"comment":"The displayed formula for log Gamma-tilde_tn appears to be missing a logarithm on the sum over n' after '- log N'; as written, the expression would scale linearly rather than logarithmically with the number of training points N.","section":"Appendix B.2"},{"comment":"The sentence 'qn(zt) can be computed exactly via standard Kalman smoothing' is only true for the variational distribution under the F ≈ p_eta approximation; please make the conditional nature of this statement explicit in the main text.","section":"Section 3"},{"comment":"The axis label '( , )' appears to be a placeholder for the latent and observation dimensions and should be replaced with the actual variables (e.g., D_Z, D_X).","section":"Figure 2 caption"},{"comment":"The transition matrix is denoted B in the data-generating process but A elsewhere in the paper; please unify the notation to avoid confusion.","section":"Section 5.2"},{"comment":"The word 'unkown' in the CPC model license entry should be 'unknown'.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper builds on the RPM framework, which is self-cited from the same group, but the SSM-specific derivation and experiments are not entailed by RPM alone. The main concern is that the F ≈ p_eta approximation is inherited without independent verification, and it is load-bearing for both the maximum-likelihood interpretation and the exact-inference claim. I believe this is fixable within the manuscript's scope by adding diagnostics, tempering the claims, or solving the coupled E-step, so I recommend major revision rather than rejection. The novelty is sufficient for the venue if the approximation is addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as \"RPM applied to state space models.\" The core idea is clean: define the generative model implicitly through recognition factors, pair it with a linear-Gaussian latent prior, and derive an auxiliary free energy that lets you use Kalman smoothing. The empirical story is genuinely interesting: the model is the only one that decodes the pendulum and tracking latents in the presence of background distractors, and the data-efficiency result (angular velocity from 64 sequences) is a nice concrete win.\n\nWhat's new: the SSM-specific derivation, the f^Δ reparametrization, the closed-form Γ terms, and the distractor experiments. The derivations are mostly careful, and the paper is upfront about the linear-dynamics restriction. The bound in Eq. (6) is valid for any positive g, so the method is a legitimate variational approach even if the tightness story is incomplete.\n\nThe soft spot is the F≈pη assumption. The paper sets g=qn/pη based on a large-data in-model limit and asserts F→pη without any finite-sample diagnostic. That matters twice: first, the objective being maximized is not the likelihood of the RPM joint unless F is close to pη; second, Γtn in Eq. (11) depends on qn through its natural parameters, so \"standard Kalman smoothing\" is only an exact E-step if you ignore that dependence. The true RPM posterior is a ratio of Gaussians to a mixture of Gaussians, so Kalman smoothing returns a variational q, not an exact posterior. The abstract's claims of \"maximum likelihood\" and \"exact inference\" are therefore overstated; the paper should say \"a lower bound\" and \"approximate inference.\" This is not a fatal flaw for the empirical method, but it is a real gap in the theory.\n\nMinor issues: the main experimental figures lack error bars, and the promised code has no repository or commit hash. These are easy to fix.\n\nBottom line: the paper deserves a serious referee. With a softened title/abstract, a diagnostic for F vs pη, and error bars, it would be a solid venue contribution. I would not cite the ML/exact-inference claim as-is, but I would keep the paper on the reading list.","headline":"A promising extension of RPM to SSMs with strong distractor-robustness results, but the maximum-likelihood and exact-inference claims are undercut by the unchecked F≈pη approximation and a q-dependent Γ term in the E-step.","tokens_in":17907,"tokens_out":3388,"would_cite":false,"duration_ms":34055,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A recognition network and a linear-Gaussian prior suffice to learn latent dynamics from high-dimensional observations without any decoder or reconstruction loss.","keywords":["unsupervised learning","state space models","latent dynamics","recognition-parametrized models","Kalman smoothing","maximum likelihood","time series","video representation learning"],"falsifier":"Train an RP-GSSM on a synthetic video task and, during training, compute a standard distributional distance (for example, KL divergence) between the empirical mixture $F_{\\phi_t}(z_t)$ and the learned prior $p_\\eta(z_t)$; the maximum-likelihood account predicts the distance stays near zero whenever the auxiliary free energy rises. A second test: construct background distractors whose temporal autocorrelation matches that of the true latent signal; if the RP-GSSM then fails to separate signal from distractors, the robustness results are driven by the temporal-persistence assumption rather than by the likelihood structure.","tokens_in":16882,"feed_emoji":"🎥","tokens_out":8813,"duration_ms":78353,"temperature":0.7,"pith_summary":"The paper introduces the recognition-parametrized Gaussian state space model (RP-GSSM), an unsupervised method for time series that learns Markovian Gaussian latent variables from high-dimensional observations such as video. Its central claim is that a recognition network mapping observations to Gaussian beliefs, together with a linear-Gaussian latent prior, is sufficient to learn latent dynamics: no generative network that reconstructs observations is required. Because the latent prior is jointly Gaussian, the posterior over the whole sequence is available exactly by Kalman smoothing, and training is driven by a maximum-likelihood objective for a valid probabilistic model rather than by reconstruction or contrastive losses. The paper reports that this design makes learned latents focus on temporally persistent signal and ignore background distractors, outperforming generative and contrastive alternatives on pendulum, double-pendulum, and object-tracking video tasks both with and without distractions.","feed_headline":"Latent dynamics learned from video with no decoder","feed_subtitle":"Recognition network plus Gaussian prior outperforms baselines, even with distracting backgrounds.","key_machinery":"The load-bearing object is the recognition-parametrized joint distribution of a state space model: $p_\\eta(z_1)\\prod_t p_\\eta(z_{t+1}|z_t)\\prod_t f_{\\phi_t}(z_t|x_t)p_0(x_t)/F_{\\phi_t}(z_t)$, with Gaussian recognition factors and a linear-Gaussian prior constrained to stationary $N(0,I)$ so that the process remains stationary at every time step. The tractability mechanism is the auxiliary free energy $G$ obtained from an interior variational bound with auxiliary factors $g_{tn}(z_t)=q_n(z_t)/p_\\eta(z_t)$; this replaces the intractable expectation over the mixture $F_{\\phi_t}$ with closed-form Gaussian integrals. With that choice, $G$ has exactly the form of the free energy of a linear-Gaussian model whose 'likelihood' is the recognition factor, so the E-step is exact Kalman smoothing and the M-step is exact gradient ascent on $G$. The assumption that $F_{\\phi_t}(z_t)\\approx p_\\eta(z_t)$ is what makes the auxiliary choice valid and the bound tight.","core_discovery":"On its own terms, the paper's discovery is that a state space model can be fit by maximum likelihood without ever parametrizing the distribution of observations. The RP-GSSM builds a joint distribution over latents and observations by multiplying a Gaussian Markov prior with per-time recognition factors $f_{\\phi_t}(z_t|x_t)$ and normalizing each factor by its empirical average $F_{\\phi_t}(z_t)$ over the training set; the implied observational distribution is degenerate, but the posterior beliefs over latents are parametric and exact. The E-step is Kalman smoothing, and the M-step is gradient ascent on an auxiliary free energy whose variational gap is controlled by setting the auxiliary factors to $q_n(z_t)/p_\\eta(z_t)$. The paper's positive thesis is that decoders are not merely wasteful for latent-state tasks but can be harmful: reconstruction pressure pushes latents to encode irrelevant or distractor features, while an objective built purely from conditional-independence structure learns the temporally persistent state.","pith_inferences":["An untested consequence: the advantage over reconstruction-based methods should shrink when distractors have temporal statistics as persistent and structured as the true signal; this can be tested directly by varying background autocorrelation.","The same recognition-parametrization recipe should transfer to other graphical models in which conditional independences supply the learning signal, such as switching state space models or multimodal time series, wherever reconstruction is not the goal.","A direct probe of the likelihood claim is to monitor $F_{\\phi_t}$ against $p_\\eta$; if the gap stays large while $G$ increases, the model is optimizing a looser bound than the paper's maximum-likelihood framing suggests.","The small-data result on 64 pendulum sequences is the clearest place to stress-test the method: if the auxiliary approximation is only valid at large sample sizes, performance on tiny datasets may come from the variational gap rather than from the model's structure."],"forward_implications":["Because no decoder is learned, model capacity is spent on inference, so latent dynamics can be recovered from high-dimensional video without reconstructing pixels.","Posterior beliefs over full latent trajectories are exact Gaussian, giving downstream planning or filtering tasks calibrated uncertainty without sampling or auxiliary losses.","Latents learned this way capture temporally persistent signal while ignoring distractors with unreliable temporal statistics, including natural images and videos used as backgrounds.","Exact Gaussian inference scales more gracefully with latent dimension than sampling-based generative alternatives, as shown in higher-dimensional linear experiments.","If the central claim holds, the RP-GSSM can serve as a latent-dynamics foundation module: training an auxiliary generative model on its inferred latents gives reconstructions that filter out the distractors that other methods cannot remove."],"supporting_citations":[{"why":"Defines recognition-parametrized models and the interior variational bound that turns the intractable log-average term into closed-form integrals.","marker":"Walker et al. (2023)"},{"why":"Provides the stability and non-identifiability results used to reparametrize the latent process to stationary N(0,I) with transition covariance I-AA^T.","marker":"Buesing et al. (2012)"},{"why":"Supplies the Kalman smoothing procedure used as the exact E-step for the jointly Gaussian latent posterior.","marker":"Kalman (1960)"},{"why":"Establishes the EM algorithm that the paper uses to alternate between smoothing and parameter updates.","marker":"Dempster et al. (1977)"},{"why":"Gives the closed-form parameter estimation for linear-Gaussian state space models that motivates the tractable M-step structure.","marker":"Ghahramani and Hinton (1996)"},{"why":"Defines Contrastive Predictive Coding, the main contrastive baseline whose lack of a full probabilistic model the RP-GSSM is designed to address.","marker":"van den Oord et al. (2018)"},{"why":"Provides the structured VAE baseline and the self-supervised masking scheme used for comparison, which the RP-GSSM does not need.","marker":"Zhao and Linderman (2023)"},{"why":"Supplies the recurrent Kalman network baseline and the synthetic single-pendulum video dataset used in the experiments.","marker":"Becker et al. (2019)"},{"why":"Provides the RatInABox toolkit and simulated rat-running sequences used for the object-state tracking experiments.","marker":"George et al. (2024)"},{"why":"Supplies the DAVIS video benchmark whose clips are stitched together as temporally correlated background distractors.","marker":"Perazzi et al. (2016)"}],"fun_headline_variants":["ML learns latent dynamics without a decoder","Latent dynamics via ML, no reconstruction needed","No decoder, no reconstruction: just latent ML","Recognition net plus Gaussian prior wins without decoder"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, at every time step, the average of the recognition distributions over the training inputs closely matches the latent prior distribution; if that match fails during finite training, the objective being maximized is not the model's true likelihood, and both the maximum-likelihood claim and the reported distractor robustness could rest on that approximation.","fun_headline_variants_meta":{"raw":{"variants":["ML learns latent dynamics without a decoder","Latent dynamics via ML, no reconstruction needed","No decoder, no reconstruction: just latent ML","Recognition net plus Gaussian prior wins without decoder"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2286,"prompt_tokens":943,"completion_tokens":1343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1286}},"tokens_in":559,"tokens_out":1343,"duration_ms":12971,"temperature":1.0,"reasoning_tokens":1286,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:43:39.213340+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an RP-GSSM on a synthetic video task and, during training, compute a standard distributional distance (for example, KL divergence) between the empirical mixture $F_{\\phi_t}(z_t)$ and the learned prior $p_\\eta(z_t)$; the maximum-likelihood account predicts the distance stays near zero whenever the auxiliary free energy rises. A second test: construct background distractors whose temporal autocorrelation matches that of the true latent signal; if the RP-GSSM then fails to separate signal from distractors, the robustness results are driven by the temporal-persistence assumption rather than by the likelihood structure.","supporting_citations":[{"cited_title":"I., Soulat, H., Yu, C., and Sahani, M","cited_arxiv_id":null,"evidence_quote":"Defines recognition-parametrized models and the interior variational bound that turns the intractable log-average term into closed-form integrals."},{"cited_title":"H., and Sahani, M","cited_arxiv_id":null,"evidence_quote":"Provides the stability and non-identifiability results used to reparametrize the latent process to stationary N(0,I) with transition covariance I-AA^T."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Kalman smoothing procedure used as the exact E-step for the jointly Gaussian latent posterior."},{"cited_title":"and Hinton, G","cited_arxiv_id":null,"evidence_quote":"Gives the closed-form parameter estimation for linear-Gaussian state space models that motivates the tractable M-step structure."},{"cited_title":"and Linderman, S","cited_arxiv_id":null,"evidence_quote":"Provides the structured VAE baseline and the self-supervised masking scheme used for comparison, which the RP-GSSM does not need."},{"cited_title":"J., and Neumann, G","cited_arxiv_id":null,"evidence_quote":"Supplies the recurrent Kalman network baseline and the synthetic single-pendulum video dataset used in the experiments."},{"cited_title":"M., Rastogi, M., de Cothi, W., Clopath, C., Stachenfeld, K., and Barry, C","cited_arxiv_id":null,"evidence_quote":"Provides the RatInABox toolkit and simulated rat-running sequences used for the object-state tracking experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DAVIS video benchmark whose clips are stitched together as temporally correlated background distractors."}],"review_version":1}