REVIEW 4 major objections 5 minor 31 references
A single learned dynamics model inside an unscented Kalman filter can estimate the state of unseen container ships zero-shot, beating classical baselines when velocities are not measured.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A transformer dynamics model plugged into an unscented Kalman filter gives zero-shot state estimates on unseen simulated container ships and outperforms classical baselines when velocity sensors are missing.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Hybrid UKF + learned dynamics is a real step forward, but the sigma-point context shift is untested and the 'zero-shot foundation model' label oversells it. the 4 major comments →
Sailing Towards Zero-Shot State Estimation using Foundation Models Combined with a UKF
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central discovery is that the unscented transform's sigma points can double as the context window for a transformer dynamics model. At every filtering step, each sigma-point trajectory is fed through a decoder-only transformer to predict the next state, and the UKF then fuses these predictions with the known sensor model in the correction step. Because the transformer infers the dynamics in context, the same weights serve every ship in the family; because the sensor model is external and analytical, swapping a sensor—say, dropping velocity measurements—requires no new training data. The paper demonstrates this on a benchmark of 1000 container ship parameterizations, reporting that in the
What carries the argument
Sigma-point-as-context prediction inside a UKF. The transform carries sigma points forward through time; instead of a hand-derived dynamics function, a decoder-only transformer (a time-series forecasting architecture) reads the stored sigma-point trajectory plus control inputs and predicts the next state for each sigma point. The UKF's unscented transform propagates uncertainty through this learned predictor, and the analytically known sensor model performs the measurement correction—so the learned part only ever models dynamics, never sensors.
Load-bearing premise
Inside the filter, the transformer receives sigma-point estimates rather than the true state trajectories it was trained on, and the paper does not directly validate that these filtered trajectories stay close enough to the training distribution for the in-context inference to remain sound.
What would settle it
Run FM-UKF on a ship whose parameters lie at the edge of the training distribution while only position measurements are available, and compare the first 50-step error trajectory against the oracle filter: if the FM-UKF error diverges while the oracle stays bounded, the sigma-point context fails; equivalently, measure the shift between the sigma-point trajectories recorded during filtering and the true-state trajectories in the training set.
If this is right
- A single trained dynamics model can replace per-system system identification for a family of systems, since the filtering prediction step is supplied by an in-context learned transformer rather than a hand-derived dynamics function.
- New sensor configurations are handled without retraining: the sensor model enters only through the UKF correction, so sensor changes need no new trajectory data.
- The successful extension from a two-dimensional toy system to a 10-dimensional nonlinear container ship model suggests the approach scales to realistic nonlinear state spaces.
- Releasing the benchmark and dataset gives the community a standardized testbed—1000 parameterizations, 400k trajectories—for zero-shot state estimation research.
Where Pith is reading between the lines
- The same sigma-point-as-context trick could be transplanted to other ensemble filters (e.g., ensemble Kalman filters or particle filters), where the ensemble naturally provides the context sequence; this might trade some of the UKF's efficiency for robustness to multi-modal posteriors.
- Because the authors tuned process noise for the classical baselines and treated the FM-UKF as a lower bound, an obvious extension is to optimize the noise model for the learned predictor—or to have the transformer directly output a covariance—which could close the gap to the oracle and end-to-end methods.
- A direct test of the paper's coherence assumption would be to feed the transformer both true-state trajectories and filtered sigma-point trajectories from the same system and compare prediction errors; growing divergence would indicate distribution shift that the current evaluation only indirectly exposes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FM-UKF, a state estimator that combines a transformer-based dynamics foundation model with an unscented Kalman filter, using analytically known sensor models instead of learned ones. The authors argue that this hybrid approach retains the generalization of foundation-model dynamics across unseen systems while supporting new sensor configurations without retraining. They contribute an open-source benchmark of 400k container-ship trajectories from 1,000 parameter instances, and evaluate FM-UKF against an end-to-end transformer baseline and three classical UKF variants (Oracle, Base, CV). The main empirical claim, stated in Sec. VI-D, is that in the no-velocity sensor configuration (h2), FM-UKF with a kinematic integrator outperforms the classical baselines, while the end-to-end approach requires retraining per sensor configuration. The paper also reports that the end-to-end baseline, when trained on the same sensor configurations, closely matches the Oracle-UKF.
Significance. If the empirical claims hold, FM-UKF is a meaningful step toward zero-shot state estimation: it uses a learned dynamics prior that generalizes across system instances while leveraging known sensor models, which is a practical architecture for control applications where sensor models are available but system identification is costly. The benchmark itself is a contribution: it is open-sourced, includes a challenging 10-state nonlinear simulation with varied dynamics, and the evaluation uses a held-out split of 100 unseen ships. The paper also makes its code and data available, which supports reproducibility. However, the current evidence for the central claim is incomplete: the headline comparison in Table II lacks statistical uncertainty quantification, the sensor generalization advantage is not tested with a properly zero-shot end-to-end baseline, and a load-bearing assumption about sigma-point context is not validated. These issues need to be addressed before the claim 'FM-UKF outperforms classical baselines' can be considered established.
major comments (4)
- [§VI-C/Table II] The central empirical claim that FM-UKF outperforms CV-UKF and Base-UKF in h2 rests on Table II, which reports median MAE and average ranks over 5,000 trajectories. No confidence intervals, error bars, or significance tests are provided, and there are no multiple training seeds. In Fig. 3, the MAE distributions overlap substantially for several states (e.g., phi and psi in h2), so the rank differences in Table II (2.25 for FM-UKF vs 2.75 and 3.38 for Base-UKF and CV-UKF) may not be statistically meaningful. Please provide per-feature paired comparisons, bootstrap confidence intervals across trajectories and/or across the 100 held-out ships, and ideally a repeated-training analysis.
- [§VI-A, Fig. 3] The comparison against the end-to-end baseline does not support the paper's central differentiation. Sec. VI-A states that the End2End transformer 'was trained simultaneously on different sensor models, h1 and h2.' The caption to Fig. 3 likewise says the End2End approach matches the oracle 'when trained on identical sensor configurations.' But the paper's claimed advantage is that FM-UKF does not require retraining for new sensor configurations. To demonstrate this, the end-to-end baseline should be trained on h1 only and evaluated on h2, or otherwise the FM-UKF should be evaluated on a sensor configuration not seen during training by either method. The present design conflates dynamics generalization with sensor generalization and weakens the central claim.
- [§IV-A, Eq. (11)] The load-bearing assumption in Eq. (11) is that 'the sequence of each n-th sigma point trajectory forms a coherent and sufficient context for the transformer to infer the dynamics.' The transformer is trained on true state trajectories (Eq. (12), with small input noise), but at inference it receives sigma-point histories that are filter outputs and can deviate substantially from the true state path, especially for unobserved states such as sway velocity in h2. The paper provides no analysis or ablation quantifying this distribution shift. A direct test would be to feed the UKF's sigma-point histories into the transformer and compare against feeding the true state histories as context, or to evaluate FM-UKF under varying process-noise levels and sensor configurations to see if the learned dynamics become unreliable. Without such evidence, the robustness of the FM prediction step to sigma-
- [§VI-B] Filter performance depends critically on the process noise covariance Q, which is manually tuned for the classical baselines and then used unchanged for FM-UKF. The authors explicitly state this is a lower bound on FM-UKF performance, but they do not quantify the sensitivity of the ranking to Q. If the relative performance of FM-UKF versus the baselines changes substantially when Q is re-tuned or estimated online, the 'competitive accuracy, effort, and robustness trade-off' claimed in the abstract is not established. A Q-sweep or an adaptive Q method, applied across all filters, would make the comparison fairer and more informative.
minor comments (5)
- [Abstract/Introduction] Typo: 'we preserve th KF's interpretability' should be 'the KF's.'
- [§VI-D] Typo: 'not-sufficiently-tuned process nose' should be 'process noise.'
- [References] References [4] and [20] appear to be the same arXiv preprint (Bommasani et al., 'On the Opportunities and Risks of Foundation Models'). Please consolidate or disambiguate.
- [§IV-C] The description of the MLHP context says Lcontext is 'randomly varied between 64 and 84 during training,' but it is not specified how Lcontext is set at inference time. Since the transformer receives a length-192 trajectory, the masking rule in Eq. (12) must have an inference-time counterpart; please clarify.
- [§VI-A] The FM-UKF (no integrator) variant performs drastically worse for positions and yaw, and the authors add a first-principle kinematic integrator to recover performance. This should be discussed more prominently as a design choice: the final method uses a hand-derived kinematic model for position/angle integration, which tempers the 'zero-shot dynamics model' narrative. It would be helpful to state explicitly which states are predicted by the FM and which by the integrator.
Circularity Check
No significant circularity: the FM-UKF derivation is self-contained and empirical comparisons are not forced by construction.
full rationale
The paper's central claims are not circular by construction. The dynamics foundation model is trained on true state/control trajectories (Eq. 8 and the MLHP objective in Eq. 12), while the UKF integrates analytically known sensor models (Sec. V-C) using standard unscented transform equations (Eqs. 9-11). The transformer is evaluated on held-out ship parameter instances (100 unseen ships) not used in training, so the reported FM-UKF accuracy is an empirical generalization result, not a fitted quantity renamed as a prediction. The sensor models are not learned, and no parameter is fitted to the target evaluation metric. The only self-citation in the paper, Ref. [7], is a related-work citation about foundation models for dynamical systems and is not load-bearing for the proposed method or its evaluation. The sigma-point context assumption in Sec. IV-A is a validity assumption and a potential correctness risk (the transformer receives filtered sigma-point histories rather than true state histories), but this is not circularity: it is an empirical distribution-shift concern that could be tested, and the paper does not define the result in terms of that assumption. No equation reduces the claimed outcome to its inputs, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. Therefore the appropriate circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (4)
- Process noise covariance Q =
not reported numerically
- Transformer hyperparameters =
patch size p=2 (also stated as p=4), embedding dim 128, 8 decoder layers, 32 heads, dropout 0.01
- MLHP context length L_context =
randomly sampled between 64 and 84
- Huber loss threshold delta =
1
axioms (6)
- domain assumption The sensor model h_xi is known exactly and can be selected for each hardware configuration.
- ad hoc to paper The sequence of each sigma-point trajectory forms a coherent and sufficient context for the transformer to infer dynamics.
- domain assumption Test systems are drawn from the same 30% parameter-variation distribution as training systems.
- domain assumption Pink-noise control inputs provide sufficient excitation for state estimation.
- domain assumption The container ship model by Son and Nomoto, as implemented in the MSS toolbox, is a valid surrogate for real ship dynamics.
- standard math The unscented transform with the Bitzer base set correctly propagates Gaussian uncertainty through nonlinear maps.
Cite this review
Pith. "Pith review of Sailing Towards Zero-Shot State Estimation using Foundation Models Combined with a UKF." pith.science (2026). https://pith.science/paper/MMNTMPWB
@misc{pith2026250904213,
author = {Pith},
title = {Pith review of: Sailing Towards Zero-Shot State Estimation using Foundation Models Combined with a UKF},
year = {2026},
howpublished = {\url{https://pith.science/paper/MMNTMPWB}},
note = {Machine review of arXiv:2509.04213}
}
read the original abstract
State estimation in control and systems engineering traditionally requires extensive manual system identification or data-collection effort. However, transformer-based foundation models in other domains have reduced data requirements by leveraging pre-trained generalist models. Ultimately, developing zero-shot foundation models of system dynamics could drastically reduce manual deployment effort. While recent work shows that transformer-based end-to-end approaches can achieve zero-shot performance on unseen systems, they are limited to sensor models seen during training. We introduce the foundation model unscented Kalman filter (FM-UKF), which combines a transformer-based model of system dynamics with analytically known sensor models via an UKF, enabling generalization across varying dynamics without retraining for new sensor configurations. We evaluate FM-UKF on a new benchmark of container ship models with complex dynamics, demonstrating a competitive accuracy, effort, and robustness trade-off compared to classical methods with approximate system knowledge and to an end-to-end approach. The benchmark and dataset are open sourced to further support future research in zero-shot state estimation via foundation models.
Figures
Reference graph
Works this paper leans on
-
[1]
A new approach to linear filtering and prediction problems,
R. E. Kalman, “A new approach to linear filtering and prediction problems,” Journal of Basic Engineering , 1960
work page 1960
-
[2]
New extension of the Kalman filter to nonlinear systems,
S. J. Julier and J. K. Uhlmann, “New extension of the Kalman filter to nonlinear systems,” in Defense, Security, and Sensing , 1997
work page 1997
-
[3]
Neural extended Kalman filters for learning and predicting dynamics of structural systems,
W. Liu, Z. Lai, K. Bacsa, and E. Chatzi, “Neural extended Kalman filters for learning and predicting dynamics of structural systems,” Structural Health Monitoring , 2024
work page 2024
-
[5]
BERT: Pre- training of deep bidirectional transformers for language understand- ing,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understand- ing,” in NAACL, 2019
work page 2019
-
[6]
Language models are unsupervised multitask learners,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
-
[7]
On Foundation Models for Dynamical Systems from Purely Synthetic Data
M. Ziegler, A. F. Posada-Moreno, F. Solowjow, and S. Trimpe, “On foundation models for dynamical systems from purely synthetic data,” arXiv:2412.00395, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[8]
Zero-shot Imputation with Foundation Inference Models for Dynamical Systems
P. Seifner, K. Cvejoski, A. K ¨orner, and R. J. S ´anchez, “Foun- dational inference models for dynamical systems,” arXiv preprint arXiv:2402.07594, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[9]
FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model
Z. Song, J. Yuan, and H. Yang, “Fmint: Bridging human designed and data pretrained models for differential equation foundation model,” arXiv preprint arXiv:2404.14688 , 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[10]
In- context learning of state estimators,
R. Busetto, V . Breschi, M. Forgione, D. Piga, and S. Formentin, “In- context learning of state estimators,” IFAC-PapersOnLine, 2024
work page 2024
-
[11]
State of art on state estimation: Kalman filter driven by machine learning,
Y . Bai, B. Yan, C. Zhou, T. Su, and X. Jin, “State of art on state estimation: Kalman filter driven by machine learning,” Annual Reviews in Control, 2023
work page 2023
-
[12]
X. Gao, H. Luo, B. Ning, F. Zhao, L. Bao, Y . Gong, Y . Xiao, and J. Jiang, “RL-AKF: An adaptive Kalman filter navigation algorithm based on reinforcement learning for ground vehicles,” Remote Sensing, 2020
work page 2020
-
[13]
Kalmannet: Neural network aided Kalman filtering for partially known dynamics,
G. Revach, N. Shlezinger, X. Ni, A. L. Escoriza, R. J. G. van Sloun, and Y . C. Eldar, “Kalmannet: Neural network aided Kalman filtering for partially known dynamics,” IEEE Transactions on Signal Processing, 2022
work page 2022
-
[14]
Z. Cui, L. Kang, L. Li, L. Wang, and K. Wang, “A combined state-of-charge estimation method for lithium-ion battery using an improved BGRU network and UKF,” Energy, 2022
work page 2022
-
[15]
A Survey on In-context Learning,
Q. Dong et al., “A Survey on In-context Learning,” EMNLP, 2024
work page 2024
-
[16]
Can a transformer represent a Kalman filter?
G. Goel and P. Bartlett, “Can a transformer represent a Kalman filter?” arXiv:2312.06937, 2024
Pith/arXiv arXiv 2024
-
[17]
A decoder-only foundation model for time-series forecasting,
A. Das, W. Kong, R. Sen, and Y . Zhou, “A decoder-only foundation model for time-series forecasting,” in ICML, 2024
work page 2024
-
[18]
Timer: Generative pre-trained transformers are large time series models,
Y . Liu, H. Zhang, C. Li, X. Huang, J. Wang, and M. Long, “Timer: Generative pre-trained transformers are large time series models,” in ICML, 2024
work page 2024
-
[19]
Unified training of universal time series forecasting transformers,
G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo, “Unified training of universal time series forecasting transformers,” in ICML, 2024
2024
-
[20]
On the opportunities and risks of foundation models,
R. Bommasani, “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258 , 2021
Pith/arXiv arXiv 2021
-
[21]
E. A. Wan and R. Van Der Merwe, “The unscented Kalman filter,” in Kalman Filtering and Neural Networks , Wiley, 2001
work page 2001
-
[22]
Nonlinear filtering using random particles,
P. del Moral, “Nonlinear filtering using random particles,” Theory of Probability & Its Applications , 1996
work page 1996
-
[23]
J. K. Uhlmann, Dynamic Map Building and Localization: New Theoretical Foundations. University of Oxford, 1995
work page 1995
-
[24]
Bitzer, The UKF exposed: How it works, when it works and when it’s better to sample , Zenodo, 2016
S. Bitzer, The UKF exposed: How it works, when it works and when it’s better to sample , Zenodo, 2016. [Online]. Available: https: //github.com/sbitzer/UKF-exposed
work page 2016
-
[25]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017
2017
-
[26]
Robust estimation of a location parameter,
P. J. Huber, “Robust estimation of a location parameter,” The Annals of Mathematical Statistics , 1964
work page 1964
-
[27]
Why gradient clip- ping accelerates training: A theoretical justification for adaptivity,
J. Zhang, T. He, S. Sra, and A. Jadbabaie, “Why gradient clip- ping accelerates training: A theoretical justification for adaptivity,” arXiv:1905.11881, 2020
Pith/arXiv arXiv 1905
-
[28]
On the coupled motion of steering and rolling of a high speed container ship,
K. Son and K. Nomoto, “On the coupled motion of steering and rolling of a high speed container ship,” Journal of the Society of Naval Architects of Japan , 1981
work page 1981
-
[29]
T. I. Fossen, Cybergalactic/MSS, 2024. [Online]. Available: https://github.com/cybergalactic/MSS
work page 2024
-
[30]
Pink noise is all you need: Colored noise exploration in deep reinforcement learning,
O. Eberhard, J. Hollenstein, C. Pinneri, and G. Martius, “Pink noise is all you need: Colored noise exploration in deep reinforcement learning,” in ICLR, 2023
work page 2023
-
[31]
Weak in the NEES?: Auto-tuning Kalman filters with Bayesian optimization,
Z. Chen, C. Heckman, S. Julier, and N. Ahmed, “Weak in the NEES?: Auto-tuning Kalman filters with Bayesian optimization,” in FUSION, 2018
work page 2018
-
[32]
A mnemonic Kalman filter for non-linear systems with extensive temporal dependencies,
S. Jung, I. Schlangen, and A. Charlish, “A mnemonic Kalman filter for non-linear systems with extensive temporal dependencies,”IEEE Signal Processing Letters , 2020
work page 2020
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.