{"id":"8f7c5d75-4c1d-4dba-afa6-1665a5f4d3b2","arxiv_id":"2411.16930","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On automotive radar data, KalmanNet is outperformed by an untuned IMM filter in accuracy and consistency, contradicting the claim that learned filters beat model-based baselines.","lead":"This paper tests KalmanNet, a deep learning filter, on automotive radar data and compares it with a classic interacting multiple model (IMM) filter. The authors find KalmanNet produces larger and less consistent errors, and conclude it is not yet suitable for safety-critical driving applications.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KalmanNet's NEES/NIS and uncertainty analysis uses a covariance that the architecture does not produce and the paper never specifies; without this, the reliability conclusion is unsupported.","rationale":"The reader's weakest assumption focuses on the underspecified IMM baseline. That is a real reproducibility issue, but the more load-bearing flaw is the missing covariance for KalmanNet's consistency analysis. Section III-B states KalmanNet does not predict P or S, yet Section V and the Discussion rely on NEES and NIS for KalmanNet. Without knowing how P and S were computed, the NEES/NIS curves and the state-uncertainty comparison cannot be interpreted. The reliability conclusion—that KalmanNet is unsuitable for safety-critical systems—depends heavily on the 'inconsistent NIS' and 'unstable covariance' findings. If those are artifacts of an arbitrary covariance estimator, the central claim weakens to a narrow accuracy comparison on two sequences. This is not an ad hominem critique; it is a request for the methodological detail needed to reproduce the result. The proposed test—disclosing the covariance source and recomputing NIS with a sample-covariance baseline—would settle whether the inconsistency finding is robust. Because the paper is otherwise informative and the accuracy comparison is plausible, a conditional verdict remains appropriate; no change to the reader's verdict is needed.","tokens_in":10423,"tokens_out":4789,"duration_ms":47717,"concrete_test":"Ask the authors to state how the KalmanNet covariance P and innovation covariance S used in Figures 4 and 5 were obtained. If the answer is not 'outputs of the KalmanNet forward pass,' re-run the NIS computation using an innovation covariance estimated from the sample covariance of the innovations over the two test sequences. If the re-computed NIS falls inside the 95% chi-square acceptance interval, then the paper's inconsistency argument is an artifact of the unstated covariance choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central reliability claim—that KalmanNet lacks the robustness needed for safety-critical systems—rests on the NEES/NIS consistency analysis and the state-uncertainty comparison in Figures 4 and 5. Section III-B explicitly states that KalmanNet operates without explicit knowledge of the state covariance P or the innovation covariance S, learning only the Kalman gain via an RNN. Yet Equations (26) and (27) require P and S, and Figure 5 reports NEES and NIS for KalmanNet. The paper never explains how these matrices were obtained for KalmanNet. If they were not produced by the KalmanNet forward pass, the consistency metrics are measuring a covariance that is not part of the evaluated architecture, making the 'inconsistent NIS' and 'unstable covariance' conclusions ungrounded. This is distinct from the underspecified IMM baseline: even with a fully specified IMM, the reliability argument would still depend on this missing covariance. The accuracy comparison may survive, but the stronger claim about safety-critical unfitness does not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the learned hybrid filter KalmanNet on real-world automotive radar data from the RadarScenes dataset, and compares it to an IMM filter on two additional driving scenarios (an 8-drive and a follow-drive). The authors report position, velocity, and acceleration errors, as well as NEES/NIS consistency metrics and a state-uncertainty comparison, and conclude that KalmanNet is less accurate, less precise, and less reliable than an untuned IMM filter, making it unsuitable for safety-critical ADAS applications. The claimed evidence is the systematic error gap in Tables II and III and the inconsistent uncertainty metrics in Figures 4 and 5.","tokens_in":10611,"tokens_out":4063,"duration_ms":36985,"significance":"If the results hold, this is a useful empirical counterweight to optimistic claims about learned state estimators, particularly because it uses a large real-world radar dataset and an open-loop evaluation protocol that isolates the filter's performance from data association. The paper also makes a clear attempt to assess not just accuracy but also statistical consistency. However, the reliability conclusion rests on a covariance analysis whose provenance for KalmanNet is never specified, and the IMM baseline is described only as 'untuned' without any parameterization. These issues undermine the reproducibility of the central comparison and the validity of the safety-critical claim, so the current version is not publishable without substantial revision.","major_comments":[{"comment":"The NEES and NIS metrics in Eqs. (26) and (27) require the state covariance P and innovation covariance S, and Figure 4 explicitly plots the covariance volume for KalmanNet. Yet Section III-B states that KalmanNet 'operates without explicit knowledge of the state covariance matrix P or the innovation covariance matrix S'. The paper never explains how P and S were obtained for KalmanNet. If they are not produced by the KalmanNet forward pass, the consistency analysis and the state-uncertainty comparison are measuring a covariance that is not part of the evaluated architecture, which would invalidate the conclusion that KalmanNet has an 'unstable state covariance estimate' and is unsuitable for safety-critical systems. The authors must specify the source of P and S for KalmanNet, or clearly restrict the reliability claims to the accuracy metrics alone.","section":"Section III-B and Section V (Eqs. (26)-(27), Fig. 5)"},{"comment":"The IMM filter is introduced as 'an untuned IMM filter' but no parameterization is given. To evaluate fairness and reproduce the comparison, the manuscript must specify the number of models, the motion models (e.g., CV/CA/CT), the process and measurement noise covariances for each model, and the Markov transition probabilities. Without these, a reader cannot verify that the IMM is genuinely untuned or whether its parameters happen to align with the scenarios, and the central accuracy comparison (Tables II and III) is not reproducible.","section":"Section V (IMM baseline)"},{"comment":"The state vector is defined as x = (x, y, ẋ, ẏ, ẍ, ÿ)ᵀ, a 6-dimensional state, but the observation matrix H is described as 'a 4 × 4 identity matrix'. A 4×4 matrix cannot map a 6-dimensional state to a 4-dimensional measurement (position and velocity). This is either a typographical error or a substantive modeling inconsistency that affects both the KalmanNet and IMM implementations. Please correct the dimension and provide the exact H used.","section":"Section IV-B (Observation matrix)"},{"comment":"The training and evaluation procedure feeds b radar points per timestep and invokes the update 'sequentially for each of the b input points'. It is unclear whether the state prediction is held fixed during these b updates, how the posterior is formed after multiple updates, and how the IMM baseline handles multiple measurements per timestep. Since the comparison is between two filters, the measurement integration scheme should be identical and clearly described; otherwise the error difference might reflect the integration procedure rather than the filtering algorithm.","section":"Section IV-C (Sequential update procedure)"}],"minor_comments":[{"comment":"The sentence 'KalmanNet operates without explicit knowledge of the state covariance matrix P or the innovation covariance matrix S' appears to contradict the earlier claim that the three GRUs represent Q, Pk|k−1, and Sk. Please reconcile these statements.","section":"Section III-B"},{"comment":"The MAE in Eq. (24) is defined as the sum of absolute errors in x and y, but Tables I–III report separate MAE values for x and y. Please clarify whether the reported values are per-component or the summed metric.","section":"Section V (Eq. (24))"},{"comment":"The notation for NEES and NIS is inconsistent: the text uses ϵk and ϵv,k, the figure uses 'NEES knet' and 'NIS knet', and the caption refers to 'NEES νk and NIS νv,k'. Standardize the notation.","section":"Figure 5"},{"comment":"The paper says the IMM 'was not fine-tuned for these scenarios', but without the IMM parameters (see major comment) this claim is not verifiable. Even with parameters, a more neutral phrasing would be 'trained without scenario-specific tuning'.","section":"Section V"},{"comment":"The sensor setup is cited to reference [18] (an earlier work by two of the authors) instead of the original RadarScenes paper [17]; please cite the dataset paper directly for the sensor configuration.","section":"Section IV-A"},{"comment":"The RMSE row in Table I appears to list a single value under the 'Position' column, but the MAE and σ rows are split by x and y. Please format consistently so that the column structure is clear.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The strongest claim in the paper—that KalmanNet lacks the reliability needed for safety-critical systems—is not supported by the evidence as presented, because the NEES/NIS analysis depends on covariance matrices that KalmanNet supposedly does not produce. This is a fixable issue (either by explaining the source of P/S or by removing the consistency analysis), but it is central to the conclusion. The IMM baseline underspecification is also a serious reproducibility concern. I would not reject the paper outright, as the accuracy comparison on real-world data is valuable, but the revisions need to be substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. This is the first head-to-head of KalmanNet against an IMM on real automotive radar data, and the accuracy comparison is worth taking seriously: on the 8-drive and follow-drive scenarios, the IMM beats KalmanNet on nearly every RMSE/MAE component. The paper is honest about its scope and doesn't oversell KalmanNet. But the paper's central conclusion—that KalmanNet is unreliable for safety-critical systems—is built on the NEES/NIS analysis, and that analysis uses a covariance the architecture doesn't produce. Section III-B explicitly says KalmanNet operates without explicit knowledge of P or S; the RNN only outputs a gain. Equations (26)-(27) require P and S, and the paper never says where they come from for KalmanNet. Without that, Figure 5 is measuring something undefined. That is a load-bearing gap, not a minor omission. The IMM, meanwhile, is called 'untuned' but no model set, noise covariances, or transition probabilities are given, so the baseline's fairness can't be checked. Those two issues make the reliability conclusion unsupported as written. The accuracy finding may survive—KalmanNet's RMSE is worse even if you ignore covariance—but the strong 'unfitness for safety-critical' line goes further than two sequences and an underspecified comparison can support. The paper also has only two test sequences, no multi-run statistics. Good things: they use RadarScenes, a real public dataset; they isolate tracking from association; they report NEES/NIS even if execution is flawed; the training-loss distance weighting is a sensible domain adaptation. The self-citations are fine. The paper is clearly written and the comparison setup is easy to follow. Verdict: this deserves serious peer review because an authoritative negative evaluation of KalmanNet on automotive radar would be a useful community data point, but only after the authors specify the IMM configuration, explain exactly how P and S were obtained for KalmanNet (or drop the consistency analysis if they can't), and soften the scope of the safety-critical claim. I'd send it to a venue that cares about tracking filters, with heavy revision required.","headline":"Useful automotive-radar benchmark, but the safety-critical conclusion rests on a covariance KalmanNet never produces, and the IMM baseline is underspecified.","tokens_in":11166,"tokens_out":1546,"would_cite":false,"duration_ms":15507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural-network-aided Kalman filter, KalmanNet, is outperformed by an untuned classical IMM filter on real automotive radar data.","keywords":["state estimation","KalmanNet","interacting multiple model filter","automotive radar","Kalman filter","deep learning tracking","filter consistency","ADAS"],"falsifier":"A reader could settle the claim by re-running the comparison with the IMM configuration fully specified and repeated across a larger set of RadarScenes sequences: if the IMM no longer beats KalmanNet on velocity and acceleration when its parameters are chosen differently, or if KalmanNet's NIS falls inside the 95% interval after retraining on maneuvers such as the 8-drive, the paper's central conclusion would weaken.","tokens_in":10195,"feed_emoji":"📡","tokens_out":6993,"duration_ms":61002,"temperature":0.7,"pith_summary":"This paper sets out to test whether KalmanNet, a neural-network-aided Kalman filter that claims to work with only partial knowledge of a system's dynamics and noise, actually beats a classical model-based filter on real automotive radar data. The authors train KalmanNet on the RadarScenes dataset and then compare it with an untuned interacting multiple model (IMM) filter on two driving scenarios, a figure-eight maneuver and a follow-drive. Their measurements show the IMM filter producing lower position, velocity, and acceleration errors, with the gap widening for velocity and acceleration, and KalmanNet's reported uncertainty failing the NEES and NIS consistency checks. If the comparison is accepted, it undercuts the broad claim that learned filters outperform model-based estimators in real-world conditions, while leaving room for KalmanNet in applications where model knowledge is scarce.","feed_headline":"Learned radar filter loses to untuned classical tracker","feed_subtitle":"Velocity errors are two to three times the IMM's, and its uncertainty estimates fail consistency checks.","key_machinery":"The argument is carried by three components working together. KalmanNet's core mechanism is a recurrent gain predictor: three gated recurrent units stand in for the process-noise, state-covariance, and innovation-covariance matrices and output the Kalman gain directly, so the filter never propagates an explicit covariance. The reference mechanism is the interacting multiple model filter, which runs several Kalman filters under different motion models and mixes them with Markov-model probabilities; the paper labels this baseline 'untuned' for the test scenarios. The evaluation mechanism is filter-consistency testing: the normalized estimation error squared (NEES) and normalized innovation squared (NIS) compare reported covariances against actual residuals inside a 95% chi-square interval, which is what exposes KalmanNet's overconfidence.","core_discovery":"The paper's central claim is that KalmanNet has lower accuracy and lower precision than a linear IMM filter on both tested scenarios, directly contradicting the original KalmanNet claim that it can outperform classical Kalman filters using an approximate motion model and unknown noise statistics. In the 8-drive scenario, the IMM filter's velocity RMSE is less than half of KalmanNet's, and in the follow-drive it is about a third. The NEES and NIS results show that KalmanNet's estimated covariance is not consistent with its actual errors: the normalized innovation squared sits above the 95% confidence bound for most of the 8-drive, meaning the filter is too confident in its own estimates. The authors conclude that this combination of lower accuracy, volatile errors, and unreliable uncertainty estimates makes KalmanNet unsuited to safety-critical ADAS functions.","pith_inferences":["The paper leaves implicit that the 8-drive scenario is far outside KalmanNet's training distribution of ordinary traffic, so the gap may be a distribution-shift failure rather than a fundamental limit of learned gains; training on such maneuvers is a natural next test.","A sharper comparison would hold the motion model constant and compare KalmanNet against an extended or unscented Kalman filter with only the same partial model knowledge, isolating whether the learned gain itself is the weakness.","The same NEES/NIS and covariance-volume checklist could be applied to newer variants, such as the multi-model KalmanNet extension, to see whether mixing multiple learned filters recovers consistency.","For deployment, the results imply that learned filters need a certification step in which the output covariance is validated against ground truth before use in sensor fusion."],"forward_implications":["In automotive radar tracking, a standard IMM filter that was not tuned for the test scenarios beats KalmanNet on every state component in the two scenarios.","KalmanNet's covariance estimates are not trustworthy for fusion or safety decisions, because its NIS values are mostly above the 95% confidence bound.","The performance gap is largest in velocity and acceleration, so learned filters should not be assumed to handle dynamic maneuvers without dedicated validation.","KalmanNet can still serve as a filter in settings where the dynamical model is unknown and expert tuning is unavailable, though not for safety-critical use."],"supporting_citations":[{"why":"Defines the KalmanNet architecture and its reported advantage over model-based filters, which this paper tests.","marker":"[5]"},{"why":"Supplies the RadarScenes automotive radar dataset used for training and evaluation.","marker":"[17]"},{"why":"Provides the Kalman filter and IMM formulations that form the reference baseline.","marker":"[3]"},{"why":"Provides the NCLT odometry dataset used in the original KalmanNet real-world evaluation that this paper considers too thin to be a suitable benchmark.","marker":"[15]"}],"fun_headline_variants":["KalmanNet underperforms classical filter on real radar data","Neural filter's uncertainty estimates fail consistency checks","Deep learning tracker loses to classic Kalman method","Radar test: classical filter beats learned KalmanNet","KalmanNet's velocity errors double classical tracker's"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The strongest conclusion depends on the IMM filter being a genuinely untuned, fair baseline; the paper never reports the IMM's model set, noise covariances, or Markov transition probabilities, so a reader cannot check whether these choices accidentally favor the classical filter.","fun_headline_variants_meta":{"raw":{"variants":["KalmanNet underperforms classical filter on real radar data","Neural filter's uncertainty estimates fail consistency checks","Deep learning tracker loses to classic Kalman method","Radar test: classical filter beats learned KalmanNet","KalmanNet's velocity errors double classical tracker's"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000537,"raw_usage":{"total_tokens":2551,"prompt_tokens":888,"completion_tokens":1663,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1596}},"tokens_in":504,"tokens_out":1663,"duration_ms":11592,"temperature":1.0,"reasoning_tokens":1596,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:43:24.061716+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the claim by re-running the comparison with the IMM configuration fully specified and repeated across a larger set of RadarScenes sequences: if the IMM no longer beats KalmanNet on velocity and acceleration when its parameters are chosen differently, or if KalmanNet's NIS falls inside the 95% interval after retraining on maneuvers such as the 8-drive, the paper's central conclusion would weaken.","supporting_citations":[{"cited_title":"Kalmannet: Neural network aided kalman filtering for partially known dynamics","cited_arxiv_id":null,"evidence_quote":"Defines the KalmanNet architecture and its reported advantage over model-based filters, which this paper tests."},{"cited_title":"RadarScenes: A Real- World Radar Point Cloud Data Set for Automotive Applications, March 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the RadarScenes automotive radar dataset used for training and evaluation."},{"cited_title":"Esti- mation with applications to tracking and navigation: theory algorithms and software","cited_arxiv_id":null,"evidence_quote":"Provides the Kalman filter and IMM formulations that form the reference baseline."},{"cited_title":"Ushani, and Ryan M","cited_arxiv_id":null,"evidence_quote":"Provides the NCLT odometry dataset used in the original KalmanNet real-world evaluation that this paper considers too thin to be a suitable benchmark."}],"review_version":1}