pDANSE: Particle-based Data-driven Nonlinear State Estimation from Nonlinear Measurements
Pith reviewed 2026-05-18 03:12 UTC · model grok-4.3
The pith
Particle sampling from RNN priors lets data-driven nonlinear state estimation compete with full model-driven methods when dynamics are unknown.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
pDANSE computes second-order statistics of the state posterior for a model-free process by feeding sequential measurements into an RNN that outputs Gaussian prior parameters and then applying a reparameterization trick to generate particles that approximate the effect of nonlinear measurements; the resulting estimator supports both semi-supervised and unsupervised training and reaches performance comparable to model-driven methods that possess complete knowledge of the state transition model.
What carries the argument
Reparameterization trick-based particle sampling that estimates second-order posterior statistics directly from nonlinear measurements after an RNN supplies the Gaussian prior.
If this is right
- Nonlinear measurement functions such as cubic, half-wave rectification, Cartesian-to-spherical, and camera models can be handled without requiring closed-form posterior solutions.
- Both semi-supervised learning with partial labels and fully unsupervised learning are feasible depending on data availability.
- The method scales to higher-dimensional chaotic systems such as the Lorenz-96 while remaining computationally lighter than sequential Monte Carlo.
- State estimation performance remains competitive across multiple nonlinear measurement types without any explicit model of state evolution.
Where Pith is reading between the lines
- The approach could be tested on real sensor data from physical systems where only measurement sequences are recorded and no simulator is available.
- Replacing the RNN prior generator with other sequence models might improve robustness when measurement noise statistics change over time.
- Extending the particle count adaptively based on measurement nonlinearity could further close any remaining gap to model-driven accuracy.
Load-bearing premise
The RNN, given only sequential measurements, must output Gaussian parameters that adequately represent the current state distribution, and the modest particle set must yield sufficiently accurate posterior mean and variance.
What would settle it
On the stochastic Lorenz-63 system with half-wave rectified measurements, if pDANSE produces mean-squared state estimation error substantially larger than a model-driven Kalman filter that knows the true state transition model, the central performance claim would be refuted.
Figures
read the original abstract
We consider the problem of designing a data-driven nonlinear state estimation (DANSE) method that uses (noisy) nonlinear measurements of a process whose underlying state transition model (STM) is unknown. Such a process is referred to as a model-free process. A recurrent neural network (RNN) provides parameters of a Gaussian prior that characterize the state of the model-free process, using all previous measurements at a given time point. In the case of DANSE, the measurement system was linear, leading to a closed-form solution for the state posterior. However, the presence of a nonlinear measurement system renders a closed-form solution infeasible. Instead, the secondorder statistics of the state posterior are computed using the nonlinear measurements observed at the time point. We address the nonlinear measurements using a reparameterization trickbased particle sampling approach, and estimate the second-order statistics of the state posterior. The proposed method is referred to as particle-based DANSE (pDANSE). The RNN of pDANSE uses sequential measurements efficiently and avoids the use of computationally intensive sequential Monte-Carlo (SMC) and/or ancestral sampling. We describe the semi-supervised learning method for pDANSE, which transitions to unsupervised learning in the absence of labeled data. Using a stochastic Lorenz-63 system as a benchmark process, we experimentally demonstrate the state estimation performance for four nonlinear measurement systems. We explore cubic nonlinearity and a cameramodel nonlinearity where unsupervised learning is used; then we explore half-wave rectification nonlinearity and Cartesian-tospherical nonlinearity where semi-supervised learning is used. Additionally, we also show the performance of pDANSE for the stochastic Lorenz-96 system with a half-wave, rectified measurement system. The performance of state estimation is shown to be competitive vis-a-vis model-driven methods that have complete knowledge of the STM of the dynamical system.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes pDANSE, a data-driven approach for nonlinear state estimation in model-free processes (unknown state transition model) with nonlinear measurements. An RNN parameterizes a Gaussian prior over the state from sequential measurements; the reparameterization trick generates particles from this prior that are passed through the nonlinear measurement function to approximate the posterior mean and covariance. The method supports both semi-supervised and unsupervised training. Experiments on the stochastic Lorenz-63 system with four nonlinear measurement models (cubic, camera-model, half-wave rectification, Cartesian-to-spherical) and on Lorenz-96 with half-wave rectification claim performance competitive with model-driven filters that have full knowledge of the STM.
Significance. If the performance claims hold under rigorous quantitative validation, pDANSE would offer a practical, computationally lighter alternative to traditional nonlinear filters for settings where the dynamics are unknown but measurement data are available. The reparameterization-based approximation avoids full SMC while extending the earlier linear-measurement DANSE framework; this could be useful in signal-processing applications with compressive or many-to-one nonlinearities. The semi-to-unsupervised learning path is also a positive design choice.
major comments (3)
- [Abstract / Experiments] Abstract and experimental results: the central claim of competitive state-estimation performance versus model-driven methods that know the STM is unsupported by any numerical tables, RMSE values, error bars, or statistical comparisons in the provided description. Without these data for the four nonlinearities on Lorenz-63 and the Lorenz-96 case, the performance assertion cannot be evaluated.
- [Method / Experiments] Particle approximation for nonlinear measurements: the reparameterization-trick sampling is used to obtain second-order posterior statistics for compressive mappings (camera-model, Cartesian-to-spherical). No ablation on particle count, no variance estimates across random seeds, and no comparison against UKF sigma-point or full SMC alternatives are reported, which directly affects reliability of the moment estimates for these nonlinearities.
- [Learning Approach] Learning procedure: the transition between semi-supervised and unsupervised regimes is stated but lacks explicit loss-function definitions or validation metrics showing that the unsupervised camera-model case still yields reliable posterior statistics.
minor comments (2)
- [Method] Clarify the exact formulas used to compute posterior mean and covariance from the reparameterized particles after the nonlinear measurement function is applied.
- [Experiments] Ensure all reported figures include multiple-run statistics or confidence intervals to demonstrate variability.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below, providing clarifications from the full paper and indicating revisions where the manuscript will be strengthened.
read point-by-point responses
-
Referee: [Abstract / Experiments] Abstract and experimental results: the central claim of competitive state-estimation performance versus model-driven methods that know the STM is unsupported by any numerical tables, RMSE values, error bars, or statistical comparisons in the provided description. Without these data for the four nonlinearities on Lorenz-63 and the Lorenz-96 case, the performance assertion cannot be evaluated.
Authors: The full manuscript presents experimental results via figures that plot RMSE trajectories and average errors for pDANSE against model-driven baselines (EKF, UKF, PF) across the four Lorenz-63 nonlinearities and the Lorenz-96 case. These figures demonstrate competitive performance, but we agree that explicit numerical tables with mean RMSE, standard deviations over random seeds, and statistical comparisons would improve clarity and verifiability. We will add such a summary table in the revised manuscript. revision: yes
-
Referee: [Method / Experiments] Particle approximation for nonlinear measurements: the reparameterization-trick sampling is used to obtain second-order posterior statistics for compressive mappings (camera-model, Cartesian-to-spherical). No ablation on particle count, no variance estimates across random seeds, and no comparison against UKF sigma-point or full SMC alternatives are reported, which directly affects reliability of the moment estimates for these nonlinearities.
Authors: The reparameterization approach is chosen precisely to avoid the computational cost of full SMC while still enabling moment estimation for nonlinear measurements. The initial submission focused on overall method validation rather than extensive hyperparameter sweeps. We acknowledge the value of the requested analyses and will add an ablation on particle count (e.g., 50, 100, 500), report variance across seeds, and include a targeted comparison to UKF sigma-point methods for the applicable nonlinearities in the revised version. revision: yes
-
Referee: [Learning Approach] Learning procedure: the transition between semi-supervised and unsupervised regimes is stated but lacks explicit loss-function definitions or validation metrics showing that the unsupervised camera-model case still yields reliable posterior statistics.
Authors: The manuscript describes the semi-supervised loss (combining state and measurement reconstruction terms) and its reduction to an unsupervised form when state labels are unavailable. To address the concern, we will insert the explicit mathematical definitions of both loss functions in Section III and add validation metrics for the unsupervised camera-model case, such as measurement residual consistency checks and posterior covariance calibration plots, to confirm reliability of the estimated statistics. revision: yes
Circularity Check
No significant circularity in pDANSE derivation chain
full rationale
The paper proposes pDANSE as a constructive data-driven method: an RNN parameterizes a Gaussian prior from sequential nonlinear measurements, and a reparameterization trick generates modest particles to estimate posterior second-order statistics when closed-form solutions are unavailable. This is presented as a practical algorithmic combination rather than a first-principles derivation whose outputs reduce to its inputs by construction. Performance claims rest on experimental comparisons against model-driven filters on Lorenz-63/96 benchmarks, not on any self-definitional fit, renamed empirical pattern, or load-bearing self-citation chain. The central estimation procedure remains independently verifiable through the described training and sampling steps without collapsing to tautology.
Axiom & Free-Parameter Ledger
free parameters (1)
- RNN network weights and biases
axioms (1)
- domain assumption The hidden state of a model-free process can be adequately summarized by the parameters of a Gaussian distribution produced by an RNN from past measurements
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
reparameterization trick-based particle sampling approach to estimate the second-order statistics of the state posterior
-
IndisputableMonolith/Foundation/RealityFromDistinction.leanreality_from_one_distinction unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
RNN provides parameters of a Gaussian prior
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
-
[1]
A survey on state estimation techniques and challenges in smart distribution systems,
K. Dehghanpour, Z. Wang, J. Wang, Y . Yuan, and F. Bu, “A survey on state estimation techniques and challenges in smart distribution systems,” IEEE Transactions on Smart Grid, vol. 10, no. 2, pp. 2312–2322, 2019
work page 2019
-
[2]
Artificial Intelligence- Aided Kalman Filters: AI-Augmented Designs for Kalman-Type Algo- rithms,
N. Shlezinger, G. Revach, A. Ghosh, S. Chatterjee, S. Tang, T. Imbiriba, J. Dunik, O. Straka, P. Closas, and Y . C. Eldar, “Artificial Intelligence- Aided Kalman Filters: AI-Augmented Designs for Kalman-Type Algo- rithms,”IEEE Signal Processing Magazine, pp. 2–26, 2025
work page 2025
-
[3]
DANSE: Data-Driven Non-Linear State Estimation of Model-Free Process in Unsupervised Learning Setup,
A. Ghosh, A. Honor ´e, and S. Chatterjee, “DANSE: Data-Driven Non-Linear State Estimation of Model-Free Process in Unsupervised Learning Setup,”IEEE Transactions on Signal Processing, vol. 72, pp. 1824–1838, 2024
work page 2024
-
[4]
On the properties of neural machine translation: Encoder–decoder approaches,
K. Cho, B. van Merri ¨enboer, D. Bahdanau, and Y . Bengio, “On the properties of neural machine translation: Encoder–decoder approaches,” inProc. of8 th workshop, SSST-8, 2014, pp. 103–111
work page 2014
-
[5]
S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997
work page 1997
-
[6]
Unscented filtering and nonlinear estimation,
S.J. Julier and J.K. Uhlmann, “Unscented filtering and nonlinear estimation,”Proceedings of the IEEE, vol. 92, no. 3, pp. 401–422, 2004
work page 2004
-
[7]
The unscented Kalman filter for nonlinear estimation,
E.A. Wan and R. Van Der Merwe, “The unscented Kalman filter for nonlinear estimation,” inProceedings of the IEEE 2000 Adaptive Sys- tems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373). IEEE, 2000, pp. 153–158
work page 2000
-
[8]
I. Arasaratnam and S. Haykin, “Cubature Kalman filters,”IEEE Transactions on Automatic Control, vol. 54, no. 6, pp. 1254–1269, 2009
work page 2009
-
[9]
Novel approach to nonlinear/non-gaussian bayesian state estimation,
N. J. Gordon, D. J. Salmond, and A. FM. Smith, “Novel approach to nonlinear/non-gaussian bayesian state estimation,” inIEE proceedings F (radar and signal processing). IET, 1993, vol. 140, pp. 107–113
work page 1993
-
[10]
A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking,
M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, “A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking,” IEEE Transactions on Signal Processing, vol. 50, no. 2, pp. 174–188, 2002
work page 2002
-
[11]
S. S ¨arkk¨a and L. Svensson,Bayesian filtering and smoothing, vol. 17, Cambridge university press, 2023
work page 2023
- [12]
-
[13]
Self-supervised inference in state-space models,
D. Ruhe and P. Forr ´e, “Self-supervised inference in state-space models,” inICLR, 2021
work page 2021
-
[14]
Deep variational sequential monte carlo for high-dimensional observations,
W. L. van Nierop, N. Shlezinger, and R. J. van Sloun, “Deep variational sequential monte carlo for high-dimensional observations,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5
work page 2025
-
[15]
Differ- entiable particle filtering via entropy-regularized optimal transport,
A. Corenflos, J. Thornton, G. Deligiannidis, and A. Doucet, “Differ- entiable particle filtering via entropy-regularized optimal transport,” in International Conference on Machine Learning. PMLR, 2021, pp. 2100– 2111
work page 2021
-
[16]
An overview of differentiable particle filters for data-adaptive sequential bayesian inference,
X. Chen and Y . Li, “An overview of differentiable particle filters for data-adaptive sequential bayesian inference,” 2023
work page 2023
-
[17]
Differentiable particle filters: End-to-end learning with algorithmic priors,
R. Jonschkowski, D. Rastogi, and O. Brock, “Differentiable particle filters: End-to-end learning with algorithmic priors,”Robotics: Science and Systems XIV, 2018
work page 2018
-
[18]
Dynamical variational autoencoders: A comprehensive review,
L. Girin, S. Leglaive, X. Bie, J. Diard, T. Hueber, and X. Alameda- Pineda, “Dynamical variational autoencoders: A comprehensive review,” Foundations and Trends in Machine Learning, vol. 15, no. 1-2, pp. 1– 175, 2021
work page 2021
-
[19]
R.G. Krishnan, U. Shalit, and D. Sontag, “Deep Kalman filters,”arXiv preprint arXiv:1511.05121, 2015
work page internal anchor Pith review Pith/arXiv arXiv 2015
-
[20]
Structured inference networks for nonlinear state space models,
R. Krishnan, U. Shalit, and D. Sontag, “Structured inference networks for nonlinear state space models,” inProceedings of the AAAI Confer- ence on Artificial Intelligence, 2017, vol. 31
work page 2017
-
[21]
A disentangled recognition and nonlinear dynamics model for unsupervised learning,
M. Fraccaro, S. Kamronn, U. Paquet, and O. Winther, “A disentangled recognition and nonlinear dynamics model for unsupervised learning,” Advances in NeurIPS, vol. 30, 2017
work page 2017
-
[22]
GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models,
J. Ko and D. Fox, “GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models,”Autonomous Robots, vol. 27, pp. 75–90, 2009
work page 2009
-
[23]
Bayesian inference and learning in gaussian process state-space models with particle mcmc,
R. Frigola, F. Lindsten, T. B. Sch ¨on, and C. E. Rasmussen, “Bayesian inference and learning in gaussian process state-space models with particle mcmc,”Advances in neural information processing systems, vol. 26, 2013
work page 2013
-
[24]
Variational gaussian process state-space models,
R. Frigola, Y . Chen, and C. E. Rasmussen, “Variational gaussian process state-space models,”Advances in neural information processing systems, vol. 27, 2014
work page 2014
-
[25]
Computationally efficient bayesian learning of gaussian process state space models,
A. Svensson, A. Solin, S. S ¨arkk¨a, and T. B. Sch ¨on, “Computationally efficient bayesian learning of gaussian process state space models,” in Artificial Intelligence and Statistics. PMLR, 2016, pp. 213–221
work page 2016
-
[26]
Ancestor sampling for particle gibbs,
F. Lindsten, T. Sch ¨on, and M. Jordan, “Ancestor sampling for particle gibbs,”Advances in Neural Information Processing Systems, vol. 25, 2012
work page 2012
-
[27]
Particle gibbs with ancestor sampling,
F. Lindsten, M. I. Jordan, and T. B. Sch ¨on, “Particle gibbs with ancestor sampling,”The Journal of Machine Learning Research, vol. 15, no. 1, pp. 2145–2184, 2014
work page 2014
-
[28]
Unsupervised learned Kalman filtering,
G. Revach, N. Shlezinger, T. Locher, X. Ni, R.J.G. van Sloun, and Y .C. Eldar, “Unsupervised learned Kalman filtering,” in2022 30th European Signal Processing Conference (EUSIPCO). IEEE, 2022, pp. 1571–1575
work page 2022
-
[29]
KalmanNet: Neural network aided Kalman filtering for partially known dynamics,
G. Revach, N. Shlezinger, X. Ni, A. L. Escoriza, R.J.G. Van Sloun, and Y .C. Eldar, “KalmanNet: Neural network aided Kalman filtering for partially known dynamics,”IEEE Transactions on Signal Processing, vol. 70, pp. 1532–1547, 2022
work page 2022
-
[30]
Adaptive kalmannet: Data-driven kalman filter with fast adaptation,
X. Ni, G. Revach, and N. Shlezinger, “Adaptive kalmannet: Data-driven kalman filter with fast adaptation,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 5970–5974
work page 2024
-
[31]
A. Ghosh, Y . C. Eldar, and S. Chatterjee, “Particle-based data-driven nonlinear state estimation of model-free process from nonlinear mea- surements,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5
work page 2025
-
[32]
Deterministic nonperiodic flow,
E.N. Lorenz, “Deterministic nonperiodic flow,”Journal of atmospheric sciences, vol. 20, no. 2, pp. 130–141, 1963
work page 1963
-
[33]
Semi-Supervised Model-Free Bayesian State Estimation from Compressed Measurements
A Ghosh, Y . C. Eldar, and S. Chatterjee, “Semi-Supervised Model- Free Bayesian State Estimation from Compressed Measurements,”arXiv preprint arXiv:2407.07368, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[34]
C. M. Bishop and N. M. Nasrabadi,Pattern recognition and machine learning, vol. 4, Springer, 2006
work page 2006
-
[35]
Auto-Encoding Variational Bayes,
D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in 2nd International Conference on Learning Representations, ICLR, 2014
work page 2014
-
[36]
Accurately computing the log-sum-exp and softmax functions,
P. Blanchard, D. J. Higham, and N. J. Higham, “Accurately computing the log-sum-exp and softmax functions,”IMA Journal of Numerical Analysis, vol. 41, no. 4, pp. 2311–2330, 2021
work page 2021
-
[37]
Latent-kalmannet: Learned kalman filtering for tracking from high-dimensional signals,
I. Buchnik, G. Revach, D. Steger, R. J. Van Sloun, T. Routtenberg, and N. Shlezinger, “Latent-kalmannet: Learned kalman filtering for tracking from high-dimensional signals,”IEEE Transactions on Signal Processing, vol. 72, pp. 352–367, 2023
work page 2023
-
[38]
Survey of maneuvering target tracking: Iii. measurement models,
X. R. Li and V . P. Jilkov, “Survey of maneuvering target tracking: Iii. measurement models,” inSignal and Data Processing of Small Targets
- [39]
-
[40]
Combining generative and discriminative models for hybrid inference,
V . Garcia Satorras, Z. Akata, and M. Welling, “Combining generative and discriminative models for hybrid inference,”Advances in NeurIPS, vol. 32, 2019
work page 2019
-
[41]
PyTorch: An imperative style, high-performance deep learning library,
A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,”Advances in NeurIPS, vol. 32, 2019
work page 2019
-
[42]
Adam: A method for stochastic optimization,
D.P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in3rd International Conference on Learning Representations (ICLR), 2015
work page 2015
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.