{"id":"a4820814-3f94-4e4b-80bc-d580d1557b39","arxiv_id":"1908.11472","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A reproducible constant-velocity Kalman baseline and a model-based LSTM extension for NGSIM trajectory prediction, with explicitly defined RMSE, FDE, MNLL, and miss-rate metrics.","lead":"This paper provides a reproducible constant-velocity Kalman filter baseline and a recurrent-neural-network extension for single-vehicle trajectory prediction on the NGSIM highway dataset. It reports explicit error and likelihood metrics, and argues that simple kinematic models remain competitive when uncertainty is measured consistently.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NLL comparison to Deo & Trivedi [4] uses an unverified metric definition, undercutting the 'lower NLL' claim.","rationale":"The reader's weakest_assumption concerns the zero-mean white Gaussian acceleration noise. That assumption is central to the filter's derivation, but it is a standard modeling assumption; even if the true noise is non-Gaussian, the Gaussian NLL remains a proper scoring rule for the predictive distribution, and the paper's global covariance comparison in Fig. 1 provides a separate sanity check. The more immediately damaging issue is that the paper's quantitative superiority claim for NLL over published results is made without verifying that the published NLL was computed with the same definition, units, and averaging. The paper itself states that [4]'s NLL values are 'unclear' and that a different computation is 'a more likely hypothesis,' yet the abstract and introduction present the lower MNLL as an established result. If the metric definitions differ, the comparison is vacuous, and the claim that a simple Kalman filter matches or beats state-of-the-art covariance estimation loses its evidence. This is a correctness risk because it affects the central claim's external comparison, not just an internal modeling detail. The reader's second condition ('reproducing the exact NLL computation of [4] or retracting the direct lower-NLL comparison') identifies this same issue, so I partially agree; I would prioritize it over the Gaussian assumption as the single load-bearing concern.","tokens_in":10467,"tokens_out":4821,"duration_ms":43778,"concrete_test":"Run the open-source implementation of [4] (github.com/nachiket92/conv-social-pooling) or its NLL evaluation code on the exact NGSIM test set used in Table I (preprocessed with [4]'s published code), with error coordinates in meters and the same per-timestep averaging over sequences as Eq. (10). If the resulting MNLL at 1s is approximately 3.72, the discrepancy is a definition/units artifact and the 'lower NLL' claim must be withdrawn or qualified; if it is approximately 0.80, the comparison holds. Alternatively, ask the authors to provide the exact NLL formulation used in [4] and re-derive the number.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim that the constant-velocity Kalman filter achieves lower MNLL than published results for 'the same model' (abstract; Table I) is not supported because the comparison metric is not shown to be the same. Section II.B.1 states: 'The NLL values reported in articles [4], [8], [9] are unclear, they may have chosen a variation of the NLL definition and the dimensionality of the error and covariance may be meters or feet.' Later the paper concedes: 'It is unclear what the reported NLL values from [4] mean... a more likely hypothesis, that a different computation of the MNLL indicator has been used.' Yet Table I prints 'From [4] MNLL 3.72' at 1s vs 'ours 0.80' and the abstract claims superiority. No reproduction of [4]'s exact NLL computation is provided. Because the central contribution is a reproducible baseline, the NLL superiority claim is contingent on a metric-compatibility check. If [4]'s NLL is computed per-dimension, in feet, or with a different covariance matrix, the comparison is invalid and the conclusion that 'state of the art models' covariance predictions aren't much better' (Section I) loses its quantitative basis. The internal V-LSTM comparison in Table III is less exposed because the authors recomputed V-LSTM with a harmonized NLL, but the published-baseline comparison in Table I remains unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reproducible constant-velocity Kalman filter baseline for single-vehicle trajectory prediction on the NGSIM dataset, with explicit equations, a published implementation, and reported RMSE, FDE, MNLL, and miss-rate metrics. The authors learn the Kalman process and observation noise parameters by minimizing NLL on a training set, evaluate on the Deo-Trivedi test split, and compare with published constant-velocity and LSTM baselines. They further propose a multi-modal constant-velocity model obtained by sampling a learned exploration distribution over heading and velocity variations, and an LSTM-command Kalman extension that predicts accelerations as commands. The central claims are that the Kalman baseline matches published RMSE while achieving lower MNLL, that the multi-modal baseline reduces miss rate substantially, and that the LSTM-Kalman model improves accuracy and uncertainty estimation over a harmonized V-LSTM baseline.","tokens_in":10796,"tokens_out":7473,"duration_ms":68547,"significance":"If the claims are properly supported, the paper is a useful reproducibility contribution: it makes the baseline explicit, releases code, uses a standardized preprocessing pipeline, and carefully defines the NLL metric, which is often ambiguous in the trajectory prediction literature. The internal comparison in Table III, where the authors retrain the V-LSTM with a harmonized NLL definition, is a methodological strength. The multi-modal baseline with a miss-rate comparison is also potentially useful as a sanity check for more complex interaction-aware models. However, the external NLL comparison with Deo and Trivedi and the unclear data-split handling in the multi-modal section currently weaken the quantitative claims.","major_comments":[{"comment":"The central claim that the constant-velocity Kalman filter achieves a lower NLL than published results is not established. The authors state in Section II.B.1 that 'The NLL values reported in articles [4], [8], [9] are unclear' and later 'It is unclear what the reported NLL values from [4] mean', yet Table I prints a direct numeric comparison (From [4] MNLL 3.72 vs ours 0.80 at 1s) and the abstract repeats the superiority claim. Because the metric definition, units (metres vs feet), dimensionality (per-axis vs full bivariate), and covariance convention in [4] are not verified, the comparison is not meaningful as presented. Please either reproduce [4]'s exact NLL computation or remove the external NLL superiority claim and present the published values only as unverified references.","section":"Section II.B.1 and Table I"},{"comment":"The multi-modal parameters are selected on a subset of the data, but it is not stated whether this subset overlaps the test set. Section III.B says the exploration distribution is 'computed on a random subset of the NGSIM dataset' and that σθ and σαv were obtained 'with a simple grid search over σθ, and σαv using a fixed number of samples k = 6', with no indication that the subset and grid search are restricted to a training or validation split. If the test set is used for distribution estimation or parameter selection, the Table II results, including the miss-rate comparison with [4], are optimistically biased. Specify the exact split used for the exploration distribution and the grid search, and re-evaluate the chosen parameters on a held-out test set if needed.","section":"Section III.B and Fig. 2"}],"minor_comments":[{"comment":"The parameterization of the learned noise is not fully specified: equation (2) writes Q = E Q_a Q_a^T E^T, but the mapping from the listed arguments (ρ, σa, R, init) to Q_a and R is not given. Since reproducibility is a stated contribution, please make this mapping explicit.","section":"Section II.A"},{"comment":"The covariance assessment relies on a qualitative 'good match' between the empirical and predicted covariance ellipses. Consider adding a numerical agreement measure, such as the average ratio or Frobenius distance between the two covariance estimates, to make the evaluation less visual.","section":"Section II.B.2 and Fig. 1"},{"comment":"The multi-modal NLL (5.39 at 5s) is higher than the uni-modal NLL (4.44 at 5s) even though the multi-modal model is a strictly more expressive Gaussian mixture. The paper notes this and attributes it to the covariance coefficients, but a short discussion of why the likelihood does not improve would help readers interpret the baseline.","section":"Table II and Section III.B"},{"comment":"There are several small presentation issues: 'Therefor' should be 'Therefore' in the introduction, 'assessement' should be 'assessment' in Section II.B.2, and 'Pytorch' should be 'PyTorch'. In the provided manuscript text, the square-root symbol in equation (5) renders as 'vuu√' and should be corrected in the final version.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a useful reproducibility contribution, and the internal harmonized V-LSTM comparison is a strength. The main risk is that the abstract and Table I overclaim the NLL comparison with Deo and Trivedi despite the authors' own caveats, and the multi-modal parameter selection may leak test information. Both issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what you should know: this paper gives the trajectory prediction community a workable, reproducible constant-velocity Kalman baseline on NGSIM, with code and a clear NLL definition. That's genuinely useful. The headline claim that its MNLL beats published values for the same model, however, rests on a comparison the authors themselves say may be invalid. Fix that and it's a solid contribution.\n\nWhat's actually new: not the Kalman filter itself—that's textbook. The multi-modal extension adapts Schoeller et al.'s pedestrian work to vehicles, and the LSTM-Kalman command predictor is a variant of earlier hybrids. The most original pieces are the harmonized NLL metric and the way the LSTM is slotted into the Kalman prediction step to emit jerk commands. The internal comparison against a retrained V-LSTM is fair and shows modest gains.\n\nThe core method looks sound. The filter equations are explicit, parameters are learned on a training set and evaluated on a test set, and the code is linked. The RMSE/FDE values align with prior constant-velocity results, so the baseline itself is credible.\n\nSoft spots: the abstract and Table I claim lower MNLL than Deo & Trivedi [4], but the paper also states the NLL values in [4] are 'unclear' and may be computed differently, in different units, or with a different covariance. Printing 3.72 vs 0.80 as though they are comparable overreaches. The internal V-LSTM comparison avoids this because they recomputed the baseline, but the published-baseline comparison needs either a reproduction of the exact NLL computation or a retraction of the direct comparison. Also, the multi-modal grid search over sigma_theta and sigma_alpha_v should specify the split used; if the test set informed the parameter choice, those results are optimistic. Minor issues: the similarity metric needs a bit more justification, and the paper would benefit from reporting variance across seeds or subsamples given overlapping test sequences.\n\nOverall: this paper is for anyone building or evaluating trajectory prediction models who needs a trustworthy kinematic baseline and a common NLL metric. It deserves serious peer review, conditional on the NLL comparison being fixed and the multi-modal tuning protocol documented.\n\nRecommendation: send it to review with a request for revision on those two points.","headline":"Useful reproducible constant-velocity baseline on NGSIM, but the headline NLL comparison to prior published values is not yet supported.","tokens_in":11318,"tokens_out":2506,"would_cite":true,"duration_ms":24138,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A constant-velocity Kalman filter with learned noise outperforms published uncertainty estimates on the NGSIM dataset, reaching 0.75 m RMSE at 1 s and a mean NLL of 0.80.","keywords":["trajectory prediction","Kalman filter","NGSIM","baseline models","negative log-likelihood","LSTM","constant velocity model","multi-modal prediction"],"falsifier":"Compute the empirical residuals between constant-velocity predictions and observed positions on a held-out NGSIM subset, and compare their distribution and autocorrelation with the fitted Gaussian and with white noise. If the residuals show heavy tails, time-varying variance, or significant autocorrelation at short lags, the learned covariance and NLL baseline are misspecified.","tokens_in":10267,"feed_emoji":"🚗","tokens_out":5510,"duration_ms":46852,"temperature":0.7,"pith_summary":"This paper argues that a constant-velocity Kalman filter, with its noise parameters learned by minimizing negative log-likelihood, is a strong and reproducible baseline for freeway vehicle-trajectory prediction. On the NGSIM dataset it reports RMSE 0.75 m at 1 s and 6.70 m at 5 s, with a mean NLL of 0.80 at 1 s, lower than published values for the same model. It further claims that a multi-modal version of the filter cuts the 5 s miss rate to 30 percent, and that replacing the constant-velocity assumption with an LSTM command predictor improves both accuracy and uncertainty estimates over a retrained LSTM baseline. A careful reader would care because the paper supplies a transparent reference point for judging more complex trajectory models and their claimed covariances.","feed_headline":"Fitted Kalman filter undercuts published NLL on NGSIM","feed_subtitle":"A constant-velocity model with learned noise hits 0.75 m RMSE at 1 s and exposes inconsistent NLL reporting.","key_machinery":"The carrying object is a linear Kalman filter with state $(x, v_x, y, v_y)$, transition $A = \\mathrm{blkdiag}(A_x, A_y)$ with $A_x = A_y = \\begin{pmatrix}1 & dt \\\\ 0 & 1\\end{pmatrix}$, and acceleration noise entering through $Q = E Q_a E^T$, where $Q_a$ is a learned factorized matrix. The paper learns $Q_a$, the observation noise $R$, and the initial state by minimizing the time-averaged negative log-likelihood $\\mathrm{MNLL}(k) = \\frac{1}{N}\\sum_i \\mathrm{NLL}((x^i_k-\\hat{x}^i_k, y^i_k-\\hat{y}^i_k), P_k)$, with the NLL written explicitly as a bivariate Gaussian log-density. This same scaffold is extended in two ways: sampling a quantized exploration distribution over initial velocity to produce Gaussian-mixture modes, and replacing the constant-velocity step with an LSTM cell that outputs a jerk command and its variance at each prediction step.","core_discovery":"The paper's central claim is that a properly fitted kinematic model is a serious baseline, not a toy. On the NGSIM US-101 and I-80 datasets, using only relative positions, the constant-velocity Kalman filter with learned process and observation noise reaches RMSE 0.75 m at 1 s and 6.70 m at 5 s, and a mean negative log-likelihood of 0.80 at 1 s, much lower than values reported in earlier work for the same model. The paper attributes the discrepancy to inconsistent NLL definitions and shows that the average predicted error covariance matches the empirical error covariance well. It also shows that a six-mode multi-modal extension reaches a 5 s miss rate of 30 percent, and that an LSTM-commanded Kalman filter, which keeps the state interpretable as position, velocity, and acceleration, improves RMSE, FDE, and MNLL over a retrained LSTM encoder-decoder baseline.","pith_inferences":["If the NLL baseline is adopted as standard, several published trajectory models may need re-benchmarking: their reported uncertainties may be substantially worse than a constant-velocity filter's, not better.","The learned noise parameters are likely dataset-specific; a testable extension would fit the same filter on urban or intersection data and check whether MNLL and covariance matching degrade, indicating non-stationary driving dynamics.","The command-prediction architecture could be ablated to separate the contribution of the kinematic prior from the LSTM: replacing the Kalman prediction step with a pure LSTM output while keeping the same training loss would quantify how much inertia and state structure buy."],"forward_implications":["Any future trajectory-prediction method that reports NLL should compare against a fitted constant-velocity Kalman filter with an explicit NLL definition; otherwise lower RMSE alone does not demonstrate better uncertainty modeling.","Reported covariance matrices from more complex models can be assessed by the same ellipse-matching procedure; if their mean predicted covariance exceeds the empirical error covariance, the model is overconfident or underconfident in a quantifiable way.","Multi-modal baselines can be built without training by quantizing the initial-velocity exploration distribution; the resulting 30 percent miss rate at 5 s undercuts the need for hand-defined maneuver modes.","The LSTM-commanded Kalman filter shows that interpretable kinematic structure can be retained in a learned predictor, giving a route to constrain predictions by physical limits such as maximum acceleration."],"supporting_citations":[{"why":"Supplies the Kalman filtering and prediction equations that form the core of the baseline.","marker":"[1]"},{"why":"Motivates the claim that simple constant-velocity baselines can outperform more complex pedestrian motion prediction models.","marker":"[2]"},{"why":"Supplies the LSTM cell used in the command-prediction extension.","marker":"[3]"},{"why":"Provides the NGSIM test-set preprocessing code and the V-LSTM and social-pooling baselines against which the paper compares.","marker":"[4]"},{"why":"Represents another constant-velocity Kalman implementation with different results, motivating the need for explicit parameter fitting.","marker":"[5]"},{"why":"Contrasts a Kalman filter fed by precomputed neural-network accelerations with the paper's integrated command prediction.","marker":"[8]"},{"why":"Supplies the multi-modal constant-velocity sampling idea that the paper adapts to vehicle trajectories.","marker":"[11]"},{"why":"Contrasts an LSTM that replaces the state update with the paper's kinematic-constrained LSTM command predictor.","marker":"[12]"}],"fun_headline_variants":["Kalman baseline exposes NLL inconsistency in prior work","Simple kinematic model: reproducible NGSIM baselines","Kalman baseline: 0.75 m RMSE at 1 s on NGSIM","LSTM-Kalman hybrid improves NGSIM trajectory accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model assumes that, after fitting, future acceleration is zero-mean white Gaussian noise with fixed covariance, so the reported negative log-likelihood and covariance estimates are only meaningful if NGSIM trajectory variations are Gaussian and stationary.","fun_headline_variants_meta":{"raw":{"variants":["Kalman baseline exposes NLL inconsistency in prior work","Simple kinematic model: reproducible NGSIM baselines","Kalman baseline: 0.75 m RMSE at 1 s on NGSIM","LSTM-Kalman hybrid improves NGSIM trajectory accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001145,"raw_usage":{"total_tokens":4752,"prompt_tokens":947,"completion_tokens":3805,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":3731}},"tokens_in":563,"tokens_out":3805,"duration_ms":24589,"temperature":1.0,"reasoning_tokens":3731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:14:00.365715+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the empirical residuals between constant-velocity predictions and observed positions on a held-out NGSIM subset, and compare their distribution and autocorrelation with the fitted Gaussian and with white noise. If the residuals show heavy tails, time-varying variance, or significant autocorrelation at short lags, the learned covariance and NLL baseline are misspecified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Kalman filtering and prediction equations that form the core of the baseline."},{"cited_title":"Sch ¨oller, V","cited_arxiv_id":null,"evidence_quote":"Motivates the claim that simple constant-velocity baselines can outperform more complex pedestrian motion prediction models."},{"cited_title":"Hochreiter, and J","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM cell used in the command-prediction extension."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NGSIM test-set preprocessing code and the V-LSTM and social-pooling baselines against which the paper compares."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents another constant-velocity Kalman implementation with different results, motivating the need for explicit parameter fitting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contrasts a Kalman filter fed by precomputed neural-network accelerations with the paper's integrated command prediction."},{"cited_title":"Sch ¨oller, V","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-modal constant-velocity sampling idea that the paper adapts to vehicle trajectories."},{"cited_title":"Coskun, F","cited_arxiv_id":null,"evidence_quote":"Contrasts an LSTM that replaces the state update with the paper's kinematic-constrained LSTM command predictor."}],"review_version":1}