Pith. sign in

REVIEW 4 major objections 6 minor 19 references

Performance Evaluation of Deep Learning-Based State Estimation: A Comparative Study of KalmanNet

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A neural-network-aided Kalman filter, KalmanNet, is outperformed by an untuned classical IMM filter on real automotive radar data.

desk verdict Useful automotive-radar benchmark, but the safety-critical conclusion rests on a covariance KalmanNet never produces, and the IMM baseline is underspecified. read the letter →

arxiv 2411.16930 v1 pith:YL7P4UL5 submitted 2024-11-25 cs.RO cs.LG

Arian Mehrfard , Bharanidhar Duraisamy , Stefan Haag , Florian Geiss This is my paper · ORCID
classification cs.ROcs.LG
keywords stateestimationKalmanNetinteractingmultiplemodelfilterautomotiveradardeeplearningtrackingconsistencyADAS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to test whether KalmanNet, a neural-network-aided Kalman filter that claims to work with only partial knowledge of a system's dynamics and noise, actually beats a classical model-based filter on real automotive radar data. The authors train KalmanNet on the RadarScenes dataset and then compare it with an untuned interacting multiple model (IMM) filter on two driving scenarios, a figure-eight maneuver and a follow-drive. Their measurements show the IMM filter producing lower position, velocity, and acceleration errors, with the gap widening for velocity and acceleration, and KalmanNet's reported uncertainty failing the NEES and NIS consistency checks. If the comparison is accepted, it undercuts the broad claim that learned filters outperform model-based estimators in real-world conditions, while leaving room for KalmanNet in applications where model knowledge is scarce.

What carries the argument

The argument is carried by three components working together. KalmanNet's core mechanism is a recurrent gain predictor: three gated recurrent units stand in for the process-noise, state-covariance, and innovation-covariance matrices and output the Kalman gain directly, so the filter never propagates an explicit covariance. The reference mechanism is the interacting multiple model filter, which runs several Kalman filters under different motion models and mixes them with Markov-model probabilities; the paper labels this baseline 'untuned' for the test scenarios. The evaluation mechanism is filter-consistency testing: the normalized estimation error squared (NEES) and normalized innovation squared (NIS) compare reported covariances against actual residuals inside a 95% chi-square interval, which is what exposes KalmanNet's overconfidence.

What would settle it

A reader could settle the claim by re-running the comparison with the IMM configuration fully specified and repeated across a larger set of RadarScenes sequences: if the IMM no longer beats KalmanNet on velocity and acceleration when its parameters are chosen differently, or if KalmanNet's NIS falls inside the 95% interval after retraining on maneuvers such as the 8-drive, the paper's central conclusion would weaken.

Watch

Extended reading notes

Core claim

The paper's central claim is that KalmanNet has lower accuracy and lower precision than a linear IMM filter on both tested scenarios, directly contradicting the original KalmanNet claim that it can outperform classical Kalman filters using an approximate motion model and unknown noise statistics. In the 8-drive scenario, the IMM filter's velocity RMSE is less than half of KalmanNet's, and in the follow-drive it is about a third. The NEES and NIS results show that KalmanNet's estimated covariance is not consistent with its actual errors: the normalized innovation squared sits above the 95% confidence bound for most of the 8-drive, meaning the filter is too confident in its own estimates. The authors conclude that this combination of lower accuracy, volatile errors, and unreliable uncertainty estimates makes KalmanNet unsuited to safety-critical ADAS functions.

Load-bearing premise

The strongest conclusion depends on the IMM filter being a genuinely untuned, fair baseline; the paper never reports the IMM's model set, noise covariances, or Markov transition probabilities, so a reader cannot check whether these choices accidentally favor the classical filter.

Editorial extensions

If this is right

  • In automotive radar tracking, a standard IMM filter that was not tuned for the test scenarios beats KalmanNet on every state component in the two scenarios.
  • KalmanNet's covariance estimates are not trustworthy for fusion or safety decisions, because its NIS values are mostly above the 95% confidence bound.
  • The performance gap is largest in velocity and acceleration, so learned filters should not be assumed to handle dynamic maneuvers without dedicated validation.
  • KalmanNet can still serve as a filter in settings where the dynamical model is unknown and expert tuning is unavailable, though not for safety-critical use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the 8-drive scenario is far outside KalmanNet's training distribution of ordinary traffic, so the gap may be a distribution-shift failure rather than a fundamental limit of learned gains; training on such maneuvers is a natural next test.
  • A sharper comparison would hold the motion model constant and compare KalmanNet against an extended or unscented Kalman filter with only the same partial model knowledge, isolating whether the learned gain itself is the weakness.
  • The same NEES/NIS and covariance-volume checklist could be applied to newer variants, such as the multi-model KalmanNet extension, to see whether mixing multiple learned filters recovers consistency.
  • For deployment, the results imply that learned filters need a certification step in which the output covariance is validated against ground truth before use in sensor fusion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper evaluates the learned hybrid filter KalmanNet on real-world automotive radar data from the RadarScenes dataset, and compares it to an IMM filter on two additional driving scenarios (an 8-drive and a follow-drive). The authors report position, velocity, and acceleration errors, as well as NEES/NIS consistency metrics and a state-uncertainty comparison, and conclude that KalmanNet is less accurate, less precise, and less reliable than an untuned IMM filter, making it unsuitable for safety-critical ADAS applications. The claimed evidence is the systematic error gap in Tables II and III and the inconsistent uncertainty metrics in Figures 4 and 5.

Significance. If the results hold, this is a useful empirical counterweight to optimistic claims about learned state estimators, particularly because it uses a large real-world radar dataset and an open-loop evaluation protocol that isolates the filter's performance from data association. The paper also makes a clear attempt to assess not just accuracy but also statistical consistency. However, the reliability conclusion rests on a covariance analysis whose provenance for KalmanNet is never specified, and the IMM baseline is described only as 'untuned' without any parameterization. These issues undermine the reproducibility of the central comparison and the validity of the safety-critical claim, so the current version is not publishable without substantial revision.

major comments (4)
  1. [Section III-B and Section V (Eqs. (26)-(27), Fig. 5)] The NEES and NIS metrics in Eqs. (26) and (27) require the state covariance P and innovation covariance S, and Figure 4 explicitly plots the covariance volume for KalmanNet. Yet Section III-B states that KalmanNet 'operates without explicit knowledge of the state covariance matrix P or the innovation covariance matrix S'. The paper never explains how P and S were obtained for KalmanNet. If they are not produced by the KalmanNet forward pass, the consistency analysis and the state-uncertainty comparison are measuring a covariance that is not part of the evaluated architecture, which would invalidate the conclusion that KalmanNet has an 'unstable state covariance estimate' and is unsuitable for safety-critical systems. The authors must specify the source of P and S for KalmanNet, or clearly restrict the reliability claims to the accuracy metrics alone.
  2. [Section V (IMM baseline)] The IMM filter is introduced as 'an untuned IMM filter' but no parameterization is given. To evaluate fairness and reproduce the comparison, the manuscript must specify the number of models, the motion models (e.g., CV/CA/CT), the process and measurement noise covariances for each model, and the Markov transition probabilities. Without these, a reader cannot verify that the IMM is genuinely untuned or whether its parameters happen to align with the scenarios, and the central accuracy comparison (Tables II and III) is not reproducible.
  3. [Section IV-B (Observation matrix)] The state vector is defined as x = (x, y, ẋ, ẏ, ẍ, ÿ)ᵀ, a 6-dimensional state, but the observation matrix H is described as 'a 4 × 4 identity matrix'. A 4×4 matrix cannot map a 6-dimensional state to a 4-dimensional measurement (position and velocity). This is either a typographical error or a substantive modeling inconsistency that affects both the KalmanNet and IMM implementations. Please correct the dimension and provide the exact H used.
  4. [Section IV-C (Sequential update procedure)] The training and evaluation procedure feeds b radar points per timestep and invokes the update 'sequentially for each of the b input points'. It is unclear whether the state prediction is held fixed during these b updates, how the posterior is formed after multiple updates, and how the IMM baseline handles multiple measurements per timestep. Since the comparison is between two filters, the measurement integration scheme should be identical and clearly described; otherwise the error difference might reflect the integration procedure rather than the filtering algorithm.
minor comments (6)
  1. [Section III-B] The sentence 'KalmanNet operates without explicit knowledge of the state covariance matrix P or the innovation covariance matrix S' appears to contradict the earlier claim that the three GRUs represent Q, Pk|k−1, and Sk. Please reconcile these statements.
  2. [Section V (Eq. (24))] The MAE in Eq. (24) is defined as the sum of absolute errors in x and y, but Tables I–III report separate MAE values for x and y. Please clarify whether the reported values are per-component or the summed metric.
  3. [Figure 5] The notation for NEES and NIS is inconsistent: the text uses ϵk and ϵv,k, the figure uses 'NEES knet' and 'NIS knet', and the caption refers to 'NEES νk and NIS νv,k'. Standardize the notation.
  4. [Section V] The paper says the IMM 'was not fine-tuned for these scenarios', but without the IMM parameters (see major comment) this claim is not verifiable. Even with parameters, a more neutral phrasing would be 'trained without scenario-specific tuning'.
  5. [Section IV-A] The sensor setup is cited to reference [18] (an earlier work by two of the authors) instead of the original RadarScenes paper [17]; please cite the dataset paper directly for the sensor configuration.
  6. [Table I] The RMSE row in Table I appears to list a single value under the 'Position' column, but the MAE and σ rows are split by x and y. Please format consistently so that the column structure is clear.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central comparison is measured against external ground truth and an independent IMM baseline; the only self-citation is not load-bearing.

full rationale

This paper is an empirical evaluation rather than a derivation, so the main circularity patterns do not apply. KalmanNet is trained on real radar data (RadarScenes, Sec. IV) and its estimates are compared against ground-truth trajectories and against a separately implemented IMM filter (Sec. V, Tabs. II-III, Figs. 2-3). No fitted parameter of KalmanNet is reused as evidence for the accuracy conclusion: the RMSE/MAE numbers are computed directly from estimation errors relative to ground truth. The self-citation [18] (Haag and Duraisamy, two of the present authors) is used only for sensor-setup details in Sec. IV-A and does not carry the argument; the load-bearing comparison with the IMM does not depend on it. The only substantive weakness is a missing methodological detail rather than circularity: Sec. III-B states that 'KalmanNet operates without explicit knowledge of the state covariance matrix P or the innovation covariance matrix S,' yet Sec. V applies NEES/NIS (Eqs. 26-27) and Fig. 5 to KalmanNet without explaining how P and S were obtained for it. That omission weakens the consistency and safety-critical unfitness claim and should be corrected, but it is not a reduction of the conclusion to the inputs of the paper. The accuracy comparison stands on independent ground truth, and the paper is therefore not circular.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. The central claim rests on an empirical comparison; the main burden is on the unspecified baseline configuration and the unstated method for computing KalmanNet's covariance in the consistency analysis.

free parameters (4)
  • IMM model set and Markov transition probabilities = not reported
    The IMM baseline is described only as 'untuned' (Section V), but the number of models, the motion models, and the transition probabilities are not given, so the comparison cannot be reproduced or judged for fairness.
  • IMM process and measurement noise covariances = not reported
    The Kalman filters inside the IMM require Q and R; these are not specified, yet they determine the baseline's performance.
  • Loss weighting parameters wmin, wmax, dmin, dmax = wmin=0.4, wmax=1.0, dmin=20, dmax=120 (Section IV-C)
    These are hand-chosen and affect the trained KalmanNet's loss; no sensitivity analysis is provided.
  • KalmanNet training hyperparameters = not reported
    Learning rate, number of epochs, batch size, and hidden state sizes are not stated; the paper only says the Adam optimizer was used (Section IV-C).
assumptions (3)
  • standard math Standard Kalman filter equations and IMM algorithm
    Used as background for both filters.
  • domain assumption RadarScenes ground truth and OOD high-accuracy ground truth are accurate
    The evaluation treats the provided object tracks as ground truth (Section IV-A), which is an external data assumption.
  • domain assumption The two OOD scenarios are representative of the difficulty distribution
    Generalization to safety-critical suitability is drawn from two sequences (Section VI).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance Evaluation of Deep Learning-Based State Estimation: A Comparative Study of KalmanNet." pith.science (2026). https://pith.science/paper/YL7P4UL5

@misc{pith2026241116930,
  author       = {Pith},
  title        = {Pith review of: Performance Evaluation of Deep Learning-Based State Estimation: A Comparative Study of KalmanNet},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YL7P4UL5}},
  note         = {Machine review of arXiv:2411.16930}
}
read the original abstract

Kalman Filters (KF) are fundamental to real-time state estimation applications, including radar-based tracking systems used in modern driver assistance and safety technologies. In a linear dynamical system with Gaussian noise distributions the KF is the optimal estimator. However, real-world systems often deviate from these assumptions. This deviation combined with the success of deep learning across many disciplines has prompted the exploration of data driven approaches that leverage deep learning for filtering applications. These learned state estimators are often reported to outperform traditional model based systems. In this work, one prevalent model, KalmanNet, was selected and evaluated on automotive radar data to assess its performance under real-world conditions and compare it to an interacting multiple models (IMM) filter. The evaluation is based on raw and normalized errors as well as the state uncertainty. The results demonstrate that KalmanNet is outperformed by the IMM filter and indicate that while data-driven methods such as KalmanNet show promise, their current lack of reliability and robustness makes them unsuited for safety-critical applications.

Figures

Figures reproduced from arXiv: 2411.16930 by the authors.

Figure 1
Figure 1. Estimated position of OV from KalmanNet. Input was averaged over [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 4
Figure 4. Comparison of the state covariance volume between KalmanNet and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 2
Figure 2. RMSE comparison between KalmanNet and IMM on the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: NEES and NIS comparison between KalmanNet and IMM on the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 14 canonical work pages

  1. [1]

    Rudolph E. Kalman. A new approach to linear filtering and prediction problems. ASME - Journal of Basic Engineering , 1960

    work page 1960

  2. [2]

    Kalman and R.S

    R.E. Kalman and R.S. Bucy. New results in linear filtering and prediction theory. Journal of Basic Engineering , 1961

    work page 1961

  3. [3]

    Esti- mation with applications to tracking and navigation: theory algorithms and software

    Yaakov Bar-Shalom, X Rong Li, and Thiagalingam Kirubarajan. Esti- mation with applications to tracking and navigation: theory algorithms and software. John Wiley & Sons, 2004

    work page 2004

  4. [4]

    New extension of the kalman filter to nonlinear systems

    Simon J Julier and Jeffrey K Uhlmann. New extension of the kalman filter to nonlinear systems. In Signal processing, sensor fusion, and target recognition VI , volume 3068, pages 182–193. Spie, 1997

    work page 1997

  5. [5]

    Kalmannet: Neural network aided kalman filtering for partially known dynamics

    Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adria Lopez Escoriza, Ruud JG Van Sloun, and Yonina C Eldar. Kalmannet: Neural network aided kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing , 70:1532–1547, 2022

    work page 2022

  6. [6]

    Deep learning for computer vision: A brief review

    Athanasios V oulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience , 2018(1):7068349, 2018

    work page 2018

  7. [7]

    Recent trends in deep learning based natural language processing

    Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine , 13(3):55–75, 2018

    2018

  8. [8]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015

    2015

Show all 19 references
  1. [9]

    Empirical evaluation of gated recurrent neural net- works on sequence modeling

    Junyoung Chung. Empirical evaluation of gated recurrent neural net- works on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014

    2014 arXiv

  2. [10]

    Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network

    Alex Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network. Physica D: Nonlinear Phenomena, 404:132306, March 2020

    2020

  3. [11]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  4. [12]

    Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review

    Sibo Cheng, C ´esar Quilodr ´an-Casas, Said Ouala, Alban Farchi, Che Liu, Pierre Tandeo, Ronan Fablet, Didier Lucor, Bertrand Iooss, Julien Brajard, et al. Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review. IEEE/CAA Journal ...

    2023

  5. [13]

    Manzil Zaheer, Amr Ahmed, and Alexander J. Smola. Latent LSTM al- location: Joint clustering and non-linear dynamic modeling of sequence data. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceed...

    2017

  6. [14]

    Model-based deep learning: On the intersection of deep learning and optimization

    Nir Shlezinger, Yonina C Eldar, and Stephen P Boyd. Model-based deep learning: On the intersection of deep learning and optimization. IEEE Access, 10:115384–115398, 2022

    2022

  7. [15]

    Ushani, and Ryan M

    Nicholas Carlevaris-Bianco, Arash K. Ushani, and Ryan M. Eustice. University of Michigan North Campus long-term vision and lidar dataset. International Journal of Robotics Research , 35(9):1023–1035, 2015

    2015

  8. [16]

    Multi-model kalmannet for maneuvering target tracking

    Xuehan Han, Ling Ding, Cheng Peng, WenWen Zeng, Xin Zhang, Zheng Wen, and Le Zheng. Multi-model kalmannet for maneuvering target tracking. In IET International Radar Conference (IRC 2023) , volume 2023, pages 399–405. IET, 2023

    2023

  9. [17]

    RadarScenes: A Real- World Radar Point Cloud Data Set for Automotive Applications, March 2021

    Ole Schumann, Markus Hahn, Nicolas Scheiner, Fabio Weishaupt, Julius Tilly, J ¨urgen Dickmann, and Christian W ¨ohler. RadarScenes: A Real- World Radar Point Cloud Data Set for Automotive Applications, March 2021

    2021

  10. [18]

    Oafuser: Online adaptive extended object tracking and fusion using automotive radar detections

    Stefan Haag, Bharanidhar Duraisamy, Constantin Blessing, Reiner Marchthaler, Wolfgang Koch, Martin Fritzsche, and J ¨urgen Dickmann. Oafuser: Online adaptive extended object tracking and fusion using automotive radar detections. In 2020 IEEE International Conference on Multise...

    2020

  11. [19]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.