REVIEW 3 major objections 5 minor 7 references
Stable at Any Speed: Speed-Driven Multi-Object Tracking with Learnable Kalman Filtering
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that adapting a Kalman filter's noise covariances to the ego vehicle's own speed stabilizes multi-object tracking and sets the top reported HOTA among vision-based methods on KITTI 2D MOT.
desk verdict Speed-conditioned Kalman filtering is a new idea, but the paper never isolates speed from scale and losses, so the headline mechanism is unproven even though the full system works. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Speed-Guided Learnable Kalman Filter (SG-LKF), a standard Kalman filter whose covariance parameters are produced on the fly by MotionScaleNet (MSNet), a decoupled token-mixing and channel-mixing MLP with inputs $N_0=[v, w, h]$ (plus object length $l$ in the 3D version). MSNet outputs positive semi-definite values for $P$, $Q$, and $R$, so the filter reacts to speed rather than running with fixed noise. The second mechanism is a self-supervised trajectory consistency loss, combined with a semantic consistency loss computed from DINOv2 box embeddings and a position consistency loss based on CIoU, which together enforce temporal smoothness, appearance coherence, and spatial alignment during training; none of these require a re-identification network at inference.
What would settle it
Train SG-LKF with the speed input $v$ replaced by a constant while keeping MSNet and all three losses identical and rerun the KITTI and nuScenes benchmarks; if HOTA and AMOTA do not drop materially, the reported gains are not caused by speed conditioning. A second check is to split sequences by the angle between the ego-velocity vector and the target's bearing at equal speeds; if the gain disappears when the vehicle moves across the target's line of sight rather than toward it, the scalar-speed premise is incomplete.
Extended reading notes
Core claim
The discovery is that the three covariance matrices that control a Kalman filter for multi-object tracking — the predicted-state covariance $P$, the process noise $Q$, and the observation noise $R$ — can be generated on each frame by a tiny MLP, MotionScaleNet (MSNet), whose only inputs are ego-vehicle speed $v$ and the tracked object's width and height (plus length in 3D). With those covariances adapted per frame, the filter widens its uncertainty when ego speed would inflate prediction error and tightens it when the object is stable, and the paper shows higher HOTA, DetA, and AssA at every ego speed from 0 to 60 km/h compared with BoT-SORT. The same module transplanted into VirConvTrack and SimpleTrack raises their 3D tracking scores, which the paper takes as evidence that speed-conditioned covariance generation is a general mechanism rather than a dataset-specific trick.
Load-bearing premise
The method assumes that one scalar number — the vehicle's own speed, combined with the tracked object's box size — is enough to describe the extra uncertainty that the vehicle's motion injects into the tracker, so the learned adjustments transfer to any speed, direction, and driving geometry.
Editorial extensions
If this is right
- Tracking-by-detection systems can absorb ego-vehicle speed without extra sensors, since speed is already available on the vehicle and the learnable covariance module adds only 78.2K parameters.
- Speed conditioning should flatten the performance-versus-speed curve: the paper's comparison shows BoT-SORT's HOTA and AssA falling sharply as ego speed rises, while SG-LKF's decline is much slower.
- The mechanism transfers across modalities: the same parameter-generation idea works for 2D image-plane tracking and for 3D LiDAR-based tracking on KITTI and nuScenes.
- Because the Kalman equations are unchanged, the module can be inserted into other KF-based trackers; the reported 2.2% AMOTA gain over SimpleTrack on nuScenes is a direct demonstration.
- The training-only use of DINOv2 semantic features means the association gains do not require running a re-identification network at inference time.
Reading between the lines
- Beyond the paper: the scalar-speed premise suggests a concrete refinement — conditioning on the ego velocity vector (speed plus heading) and the target's bearing or depth should capture the geometry of ego-motion noise more fully; this is testable by comparing gains when the vehicle drives toward versus across a target.
- Beyond the paper: if the mechanism is the cause of the gains, then any tracker that already outputs object scale can adopt the same noise-parameterization pattern, making the contribution a reusable design rather than a dataset-specific fix.
- Beyond the paper: the reported experiments do not isolate the speed input from the new losses, so an ablation with speed held constant while keeping MSNet and the trajectory loss would reveal how much of the 2.37 HOTA gain is specifically due to speed conditioning.
- Beyond the paper: the authors' limitation statement means the 'stable at any speed' claim is only verified up to about 60 km/h on KITTI and 35 km/h on nuScenes; highway-speed data above 100 km/h is the natural stress test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SG-LKF, a tracking-by-detection MOT framework in which the Kalman filter's process noise Q, observation noise R, and posterior covariance P are predicted by a lightweight MLP (MSNet) from ego-vehicle speed and object box size, together with self-supervised trajectory, semantic, and positional consistency losses. The method is evaluated on KITTI 2D MOT (79.59% HOTA, first among vision-based methods), KITTI 3D MOT (82.03% HOTA), and nuScenes 3D MOT (69.00% AMOTA, +2.2% over SimpleTrack). The paper also reports speed-robustness experiments and speed-stratified comparisons against BoT-SORT and SimpleTrack.
Significance. The reported benchmark numbers are valuable: they come from official test servers, the component ablation in Table 1 is internally coherent, and the MSNet is extremely lightweight (78.2K parameters), making the approach practically attractive if the speed-conditioning mechanism is real. The paper also demonstrates integration into two existing KF-based trackers, which strengthens the generality claim. However, the central causal claim that ego-vehicle speed is the driver of the gains is not isolated by any experiment that removes the speed input while keeping the learnable filter and losses. Given that the title, abstract, and method definition all attribute the gains to speed conditioning, this missing control is a load-bearing gap. The manuscript is a solid systems contribution with official numbers, but the mechanism named in the title is the least-tested part of the system.
major comments (3)
- [§4, Eqs. (6)-(7); Table 2] The causal role of ego speed is never isolated. There is no ablation that keeps MSNet and all losses but removes or constantizes the speed input. Table 2 is the closest to a control, and it undercuts the speed-attribution claim: with v replaced by v·N(0,1), KITTI 2D HOTA is 78.84, only 0.75 below the clean-speed value of 79.59 and still 4.37 above the BoT-SORT baseline. Since MSNet still receives (w,h) and the trajectory/semantic/position losses remain active in that condition, most of the gain is apparently not attributable to the specific speed value. The authors should add a direct control (e.g., constant speed, zero speed, or shuffled speed) with all other components unchanged and report HOTA/DetA/AssA. Without such a control, the 'speed-driven' mechanism in the title and abstract is unsupported.
- [§4, Eq. (7)] The update stage defines P_t|t = MSNet(v_t, w_z, h_z), which replaces the standard Kalman posterior covariance update P_t|t = (I - K_t H_t) P_t|t-1 with a learned mapping. This is a structural deviation from the Kalman recursion, not merely a learned tuning of fixed parameters, and the paper provides no justification or analysis of its effect. The P/Q/R ablation mentioned in Appendix B is not included in the main text, so the reader cannot see whether the learned P_t|t is essential or even beneficial. Please either justify this design choice theoretically, or add an ablation that compares it against the standard Riccati update while keeping the same learned Q and R.
- [§4, 'N0=[v,w,h]'; Discussion and Limitations] The modeling assumption that scalar ego speed plus box scale is sufficient conditioning information for ego-motion-induced noise is untested. Physically, the effect of ego motion on a target's image-plane or BEV uncertainty depends on the angle between the ego velocity vector and the object's bearing, object depth, camera pose, and the object's own motion, none of which are inputs. The authors' own limitation section concedes that effectiveness above 100 km/h and under irregular motion is unverified. Figure 5 aggravates the concern: SG-LKF-3D improves over SimpleTrack at 0 km/h, where the speed input carries no signal, indicating that non-speed mechanisms (losses, learned P/R, or the 3D association changes) contribute substantially. The authors should test the sufficiency assumption directly, for example by holding speed constant across different bearings/depths or by including velocity direction as an additional input and showing whether gains change.
minor comments (5)
- [Table 1] The checkmark columns are not fully self-explanatory. Please label the row conditions explicitly (e.g., list the active components for each row) so the reader does not have to infer which combination corresponds to 76.84, 79.49, and 79.22.
- [§4, Eq. (8)] The displayed equation for L_TCL is malformed in the text; the formula is split mid-sentence with symbols appearing before the 'defined as follows' clause. Please rewrite the definition cleanly.
- [§4, Loss Function Design] The phrase 'Mamba-inspired temporal aggregates' is vague: define the aggregation operation or cite the specific construction used for f_i^k and c_i^k.
- [Table 2] The notation 'v (1 ± 5%·N(0,1))' and 'v·N(0,1)' is ambiguous. Clarify whether the noise is multiplicative, additive, or scaled, and how each row was generated.
- [Figure 2] Figure 2 is extremely dense and contains rendering artifacts (the '��' characters) in the provided version. Clean the figure and enlarge the key components so the architecture diagram is readable.
Circularity Check
No significant circularity: SG-LKF's reported gains come from held-out benchmark evaluation; the speed-attribution claim is under-isolated but not circular by construction.
full rationale
The paper's core mechanism is a learned mapping: Eqs. 6-7 define Q_t, R_t, and P_t|t as outputs of MSNet conditioned on ego speed and box scale, with the network trained by trajectory, semantic, and position losses and then evaluated on KITTI and nuScenes test servers. No parameter is fitted to the headline HOTA/AMOTA metric and then renamed as a prediction, and no uniqueness theorem or load-bearing author-overlap citation chain forces the speed-conditioning choice. The self-citations present, such as SimpleTrack and ImmortalTracker, are used as comparative baselines or integration targets and are externally established methods, so they do not constitute circular support. The closest concern is experimental rather than definitional: Table 1 ablates SG-LKF plus all losses as one block against BoT-SORT, and Table 2 shows that replacing v with v·N(0,1) still yields 78.84 HOTA versus 79.59 with clean speed, so the specific causal contribution of speed information is not isolated. The limitation section also concedes that effectiveness above 100 km/h and under irregular motion is unverified. These are validity and generalization limitations, not reductions of the derivation to its inputs, and therefore do not raise the circularity score.
Assumptions & free parameters
free parameters (4)
- MSNet weights (78.2K parameters) =
Trained on KITTI/nuScenes train splits; final weights not released
- Loss weights alpha, beta, lambda, gamma =
alpha and beta learnable, final values not reported; lambda and gamma unspecified
- Speed-driven max-age linear scaling coefficients =
Not specified
- Inherited BoT-SORT/SimpleTrack hyperparameters =
Inherited unchanged
assumptions (5)
- domain assumption Constant-velocity linear-Gaussian motion model with learned noise is an adequate motion model for MOT
- ad hoc to paper Scalar ego speed v plus object scale is a sufficient conditioning signal for speed-induced noise variation
- domain assumption Ego speed is reliably available at inference from vehicle sensors
- domain assumption DINOv2 embeddings of bounding boxes are stable and discriminative enough across frames for the consistency losses
- domain assumption Official KITTI and nuScenes test-server metrics are correct and comparable across methods with different detectors
Cite this review
Pith. "Pith review of Stable at Any Speed: Speed-Driven Multi-Object Tracking with Learnable Kalman Filtering." pith.science (2026). https://pith.science/paper/JSXVEOOQ
@misc{pith2026250800358,
author = {Pith},
title = {Pith review of: Stable at Any Speed: Speed-Driven Multi-Object Tracking with Learnable Kalman Filtering},
year = {2026},
howpublished = {\url{https://pith.science/paper/JSXVEOOQ}},
note = {Machine review of arXiv:2508.00358}
}
read the original abstract
Multi-object tracking (MOT) enables autonomous vehicles to continuously perceive dynamic objects, supplying essential temporal cues for prediction, behavior understanding, and safe planning. However, conventional tracking-by-detection methods typically rely on static coordinate transformations based on ego-vehicle poses, disregarding ego-vehicle speed-induced variations in observation noise and reference frame changes, which degrades tracking stability and accuracy in dynamic, high-speed scenarios. In this paper, we investigate the critical role of ego-vehicle speed in MOT and propose a Speed-Guided Learnable Kalman Filter (SG-LKF) that dynamically adapts uncertainty modeling to ego-vehicle speed, significantly improving stability and accuracy in highly dynamic scenarios. Central to SG-LKF is MotionScaleNet (MSNet), a decoupled token-mixing and channel-mixing MLP that adaptively predicts key parameters of SG-LKF. To enhance inter-frame association and trajectory continuity, we introduce a self-supervised trajectory consistency loss jointly optimized with semantic and positional constraints. Extensive experiments show that SG-LKF ranks first among all vision-based methods on KITTI 2D MOT with 79.59% HOTA, delivers strong results on KITTI 3D MOT with 82.03% HOTA, and outperforms SimpleTrack by 2.2% AMOTA on nuScenes 3D MOT.
Reference graph
Works this paper leans on
-
[3]
In Proceedings of the IEEE/CVF interna- tional conference on computer vision , 9784–9794
3dmotformer: Graph transformer for online 3d multi- object tracking. In Proceedings of the IEEE/CVF interna- tional conference on computer vision , 9784–9794. Du, Y .; Zhao, Z.; Song, Y .; Zhao, Y .; Su, F.; Gong, T.; and Meng, H. 2023. Strongsort: Make deepsort great again. IEEE Transactions on Multimedia, 25: 8725–8737. Fu, S.; Yang, Q.; Mo, Q.; Yan, J....
arXiv 2023
-
[5]
arXiv preprint arXiv:2405.11536
RobMOT: Robust 3D Multi-Object Tracking by Ob- servational Noise and State Estimation Drift Mitigation on LiDAR PointCloud. arXiv preprint arXiv:2405.11536. Oquab, M.; Darcet, T.; Moutakanni, T.; V o, H.; Szafraniec, M.; Khalidov, V .; Fernandez, P.; Haziza, D.; Massa, F.; El- Nouby, A.; et al. 2023. Dinov2: Learning robust visual fea- tures without super...
arXiv 2023
-
[7]
International Journal of Computer Vision, 1–26
APPTracker+: Displacement Uncertainty for Occlu- sion Handling in Low-Frame-Rate Multiple Object Track- ing. International Journal of Computer Vision, 1–26. Zhou, X.; Koltun, V .; and Kr¨ahenb¨uhl, P. 2020. Tracking ob- jects as points. In European conference on computer vision, 474–490. Springer. Zhu, Z.; Nie, J.; Wu, H.; He, Z.; and Gao, M. 2022. MSA- M...
work page 2020
-
[2020]
In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , 11621–11631
nuscenes: A multimodal dataset for autonomous driv- ing. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , 11621–11631. Cao, J.; Pang, J.; Weng, X.; Khirodkar, R.; and Kitani, K
-
[2023]
In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , 9686– 9696
Observation-centric sort: Rethinking sort for robust multi-object tracking. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , 9686– 9696. Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020. End-to-end object detection with transformers. In European conference on computer vi- sion, ...
arXiv 2020
-
[2024]
Ego-Motion Aware Target Prediction Module for Robust Multi-Object Tracking
Ego-motion aware target prediction module for robust multi-object tracking. arXiv preprint arXiv:2404.03110. Mao, C.; Tan, C.; Liu, H.; Hu, J.; and Zheng, M. 2023. Stereo3DMOT: Stereo Vision Based 3D Multi-object Track- ing with Multimodal ReID. In Chinese Conference on Pat- tern Recognition and Computer Vision (PRCV) , 495–507. Springer. Miah, M.; Bilode...
work page Pith review arXiv 2023
-
[2025]
S3MOT: Monocular 3D Object Tracking with Selective State Space Model
S3MOT: Monocular 3D Object Tracking with Selec- tive State Space Model. arXiv preprint arXiv:2504.18068. Zhang, Y .; Sun, P.; Jiang, Y .; Yu, D.; Weng, F.; Yuan, Z.; Luo, P.; Liu, W.; and Wang, X. 2022. Bytetrack: Multi- object tracking by associating every detection box. In Eu- ropean conference on computer vision, 1–21. Springer. Zhang, Y .; Wang, C.; W...
work page Pith review arXiv 2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.