REVIEW 3 major objections 5 minor 33 references
Training Trajectory Predictors Without Ground-Truth Data
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that training a trajectory predictor on DynoSAM's smooth estimates, rather than on dataset ground truth, both eliminates the need for annotated trajectories and improves prediction accuracy and temporal consistency in the…
desk verdict The idea is worth a look but the evaluation is circular: the headline claim of beating ground-truth-trained models is not supported by the experiments as designed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is DynoSAM, a dynamic SLAM framework that uses factor-graph inference to jointly estimate camera pose and each detected object's world-centric SE(3) motion from RGB-D frames, then projects those estimates to 2D for the predictor. The predictor is Trajectron++, a graph-structured recurrent neural network with a conditional variational autoencoder that outputs future positions from the estimated state histories. The third piece is the Absolute Consistency Error (ACE), defined as the $\ell^2$ distance between two predictions of the same future position made at consecutive time steps; it is introduced because ADE and FDE do not capture the temporal stability that matters for planning.
What would settle it
Evaluate the DynoSAM-trained and ground-truth-trained models against an independent motion reference—for example, high-rate GPS/IMU position logs, hand-annotated object positions, or the output of a second, independent SLAM system—and compute the same ADE, FDE, and ACE. If the ground-truth-trained model matches the independent reference better while the DynoSAM-trained model only matches DynoSAM's own future trajectory, the paper's central comparison would not survive.
Extended reading notes
Core claim
The central claim, stated in the paper's own terms, is that replacing ground-truth trajectories with the smooth estimates produced by DynoSAM—2D position, heading, and velocity recovered from world-centric SE(3) motion—as both the training signal and the comparison target yields a Trajectron++ model that outperforms models trained on raw ground truth and on ground truth smoothed by an Extended Kalman Filter. On the KITTI sequences with enough object tracks, average ADE drops from 1.11 m (GT) and 1.00 m (GT+EKF) to 0.76 m, average FDE drops from 2.90 m and 2.73 m to 2.15 m, and the proposed ACE consistency metric drops from 0.88 and 0.73 to 0.51; on nuScenes sequence 61 the numbers are 0.92 m, 2.58 m, and 0.30. The paper explains the gain by the smoothness of the estimated data: raw ground truth shows unnatural oscillations in position, velocity, and heading, while DynoSAM's factor-graph smoothing produces physically plausible motion, and smoother inputs lead to more accurate and more stable predictions.
Load-bearing premise
The central assumption is that a predictor's accuracy can be judged by how well it reproduces the future trajectory produced by the same estimator that supplied its training inputs, rather than by comparison with independent observations of where objects actually went.
Editorial extensions
If this is right
- A trajectory prediction model can be trained from raw RGB-D sensor data end-to-end, with the dynamic SLAM module supplying all state labels, so annotated ground-truth trajectories become unnecessary.
- Because the same estimation module generates inputs for every environment, cross-dataset differences in preprocessing and annotation style stop propagating into the predictor; the paper demonstrates transfer to nuScenes sequence 61 after training only on KITTI data.
- The quality of the input state estimate directly controls prediction quality, so estimation and prediction should be treated as one pipeline rather than as separate modules.
- A temporal-consistency metric such as ACE can expose prediction instability that average and final displacement errors hide, giving planners a more direct signal about how erratic a model's outputs will be.
- Even a small amount of training data suffices when the input is smooth and consistent, since the paper trains from scratch on the limited eligible KITTI tracks and still outperforms ground-truth-trained models.
Reading between the lines
- Editorial inference: the evaluation compares predictions against trajectories generated by the same estimator used to produce the training inputs, so the reported gains may partly measure how well the model imitates DynoSAM rather than how well it predicts true future motion; an independent reference would be needed to separate the two effects.
- Editorial inference: if input smoothness is the active mechanism, then the result should generalize beyond SLAM—any state estimator or smoothing filter, applied to the same raw measurements, could improve predictor training, and this could be tested by holding the predictor fixed and varying only the filtering level.
- Editorial inference: the paper's framing suggests a closed-loop architecture in which prediction errors are fed back to refine the object-motion estimates; that joint training direction is not explored here but is a natural consequence of the stated interdependence.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a pipeline in which DynoSAM dynamic SLAM estimates of object position, heading, and velocity are used as inputs to train and evaluate a Trajectron++ trajectory prediction model, thereby removing the need for ground-truth trajectory annotations. The authors compare prediction performance for models trained and tested on DynoSAM estimates ('Ours'), dataset ground truth ('GT'), and EKF-smoothed ground truth ('GT+EKF') on five KITTI sequences and one nuScenes mini sequence, reporting ADE, FDE, and a new ACE metric. The central claim is that training on DynoSAM estimates outperforms ground-truth-based training and enables robust predictions across different environments.
Significance. If the central claim were established, the work would be significant for robotics and autonomous driving: it would decouple trajectory prediction training from manual annotation and connect motion estimation and prediction in a single pipeline. The paper also contributes the ACE temporal-consistency metric and an explicit experimental protocol. However, the current evaluation is self-referential: all models are scored against futures produced by the same method that generated their training inputs, so the reported gains do not demonstrate superiority in forecasting true future motion. The small dataset and use of a calibrated scale factor for nuScenes further limit the generality of the stated claims. As presented, the evidence does not support the abstract and the Table III caption.
major comments (3)
- [Section IV-C-2, Table III, Eqs. (3)-(4)] The evaluation protocol is circular. The text states that "the trajectory predictions at each step are compared with the actual trajectories generated by the same method used as input." Consequently, for the 'Ours' rows, ADE and FDE measure error against DynoSAM-estimated future positions, whereas for the 'GT' rows they measure error against dataset bounding-box centers, and for the 'GT+EKF' rows against EKF-smoothed positions. Because the targets differ across rows, the numbers in Table III do not compare like with like; a model that perfectly imitates DynoSAM would score low ADE/FDE without predicting true future motion. To support the abstract and the Table III caption claim, all models must be evaluated against the same ground-truth future trajectories (e.g., the KITTI annotations), in addition to any distribution-specific consistency metrics.
- [Section IV-A, Table I] The experimental basis is very small: only 15 eligible objects across five KITTI sequences, and one nuScenes mini object. No error bars, confidence intervals, or repeated training runs are reported, so the consistent-looking improvements in Table III could reflect random variation. Aggregate error bars or per-object results are needed before claiming robust improvements.
- [Section IV-A, Table III] The nuScenes transfer experiment uses a free scaling factor: "we align the data by applying a uniform scaling factor to both positional coordinates and velocity estimates, calibrating them to match the velocity profile of Object 2 in KITTI 00." Because this factor is chosen from the target dataset and only one nuScenes object is evaluated (with GT and GT+EKF rows absent), the result on sequence 61 does not demonstrate generalization to a new environment. Reporting the scale factor's value and an evaluation without calibration, or with a held-out calibration, would be necessary to support the cross-environment claim.
minor comments (5)
- [Section III-C] The text "1.5 sprediction horizon" should read "1.5 s prediction horizon".
- [Figure 5] The caption says "DynoSAM, avg: 0.4", but the curve is for the Trajectron++ model trained on DynoSAM data; label it as "Ours" for consistency with Table III.
- [Section IV-A] The implementation details of the EKF used for GT+EKF (state space, noise parameters) are not provided, which is important because the GT+EKF results depend on those tuning choices.
- [References] Several references have malformed entries: reference [1] places the venue before the title, and reference [32] splits the venue across fields; the reference list should be cleaned up.
- [Table III caption] The caption states "Models trained using data from our estimation method outperform ground truth and smoothed ground truth data models"; this is the claim being tested, so the caption should be rephrased neutrally.
Circularity Check
Table III superiority is forced: the Ours model is scored against DynoSAM's own future estimates, so ADE/FDE measure self-imitation, not prediction accuracy.
-
self definitional
[Section IV, opening comparison paragraph; ADE/FDE Eqs. (3)-(4); Table III]
"The trajectory predictions at each step are compared with the actual trajectories generated by the same method used as input (e.g. estimated by DynoSAM, GT or GT+EKF)."
For the Ours condition, Y_GT in Eqs. (3) and (4) is not dataset ground truth but DynoSAM's own smoothed, SE(3)-regularized future estimate, because the reference trajectory is 'generated by the same method used as input.' Thus low ADE/FDE for Ours only shows that Trajectron++ reproduces DynoSAM's future outputs. The GT and GT+EKF rows are scored against raw and filtered KITTI annotations respectively, so the three rows use different targets. Comparing error magnitudes across different references makes the headline claim 'models trained using data from our estimation method outperform ground truth...' true by construction rather than as evidence of superior forecasting of true motion.
-
fitted input called prediction
[Section IV.C.2, 'Trajectory Prediction' and Table III caption]
"For each Estimated, GT, and GT+EKF case, we trained the model using data from all other available KITTI sequences before performing inference on a target sequence. ... We compare our predictions to the future trajectory of an object, which is derived using the same method employed to obtain the data for both training and prediction."
The model is trained on DynoSAM estimates and its prediction target is also 'derived using the same method employed to obtain the data for both training and prediction.' No independent ground-truth trajectory is used for the Ours evaluation. The reported ADE/FDE gains are therefore statistically forced: a model trained to regress DynoSAM's outputs will naturally have small error against DynoSAM's outputs, and DynoSAM trajectories are smoother than the raw KITTI bounding-box centers used as the GT target. The Table III caption's 'outperform ground truth' claim compares errors against mismatched references and is not a prediction-accuracy result.
1 more flagged steps
-
other
[Section IV.B.2, Eq. (5) and ACE description; Table III]
"ACE is the l2 distance between the last and second to last steps in the prediction horizon, that is T and T − 1, made at two consecutive time steps: ACE = |Y T pred(s) − Y T −1 pred(s+1)| ... The ACE metric quantifies temporal consistency, revealing deviations that may indicate uncertainty or instability."
ACE compares two predictions made by the same model at consecutive time steps for the same future point; it is an internal self-consistency measure by definition. It contains no reference to any true future state. Placing ACE in Table III as a 'lower is better' performance metric and using it in the claim that the Estimated model 'significantly outperforms' GT models treats self-consistency as if it were accuracy. Low ACE is easier to achieve when the DynoSAM input/target is smooth and does not validate that predictions match reality.
full rationale
The paper's central claim is that training Trajectron++ on DynoSAM estimates 'outperform[s] ground truth and smoothed ground truth data models' (Table III) and enables effective training without ground-truth data. The evaluation protocol makes this claim circular: Section IV states that predictions are compared with 'the actual trajectories generated by the same method used as input,' and Section IV.C.2 reiterates that the future trajectory used as reference is 'derived using the same method employed to obtain the data for both training and prediction.' For the Ours row, the ADE/FDE reference is DynoSAM's own future estimate, not an independent ground-truth trajectory, while the GT and GT+EKF rows use different references. Cross-row error comparisons are therefore not comparable, and the reported superiority is forced by construction. The ACE metric (Eq. 5) is likewise defined over the model's own consecutive predictions and measures self-consistency, not predictive accuracy. Some independent content exists: Table II reports RPE against KITTI ground truth, and the pipeline could support training without labels. However, the headline claim of outperforming ground-truth models is substantiated only by a self-referential evaluation, so the paper receives a high circularity score (8).
Assumptions & free parameters
free parameters (1)
- nuScenes scale factor =
not stated (calibrated to match velocity profile of Object 2, KITTI 00)
assumptions (4)
- domain assumption DynoSAM's estimated trajectories are accurate enough to serve as training labels for trajectory prediction.
- domain assumption The CVAE future encoding in Trajectron++ can use estimated future trajectories as the target 'ground-truth' trajectory without degrading learning.
- domain assumption The nuScenes stream can be treated as a fixed 20 Hz signal despite hardware-triggered variation between 10 and 20 Hz.
- domain assumption Objects move in a 2D plane, so projecting DynoSAM's SE(3) motion estimates to 2D is lossless for prediction.
Cite this review
Pith. "Pith review of Training Trajectory Predictors Without Ground-Truth Data." pith.science (2026). https://pith.science/paper/EJJSDHVM
@misc{pith2026250208957,
author = {Pith},
title = {Pith review of: Training Trajectory Predictors Without Ground-Truth Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/EJJSDHVM}},
note = {Machine review of arXiv:2502.08957}
}
read the original abstract
This paper presents a framework capable of accurately and smoothly estimating position, heading, and velocity. Using this high-quality input, we propose a system based on Trajectron++, able to consistently generate precise trajectory predictions. Unlike conventional models that require ground-truth data for training, our approach eliminates this dependency. Our analysis demonstrates that poor quality input leads to noisy and unreliable predictions, which can be detrimental to navigation modules. We evaluate both input data quality and model output to illustrate the impact of input noise. Furthermore, we show that our estimation system enables effective training of trajectory prediction models even with limited data, producing robust predictions across different environments. Accurate estimations are crucial for deploying trajectory prediction models in real-world scenarios, and our system ensures meaningful and reliable results across various application contexts.
Figures
Reference graph
Works this paper leans on
-
[1]
T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, ” Proc. of the European Conf. on Computer Vision (ECCV) “Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. 683–700 Springer, 2020
work page 2020
-
[2]
Motion planning in dynamic environments using context-aware human trajectory prediction,
M. N. Finean, L. Petrovi ´c, W. Merkt, I. Markovi ´c, and I. Havoutis, “Motion planning in dynamic environments using context-aware human trajectory prediction,” 104450 166 J. of Robotics and Autonomous Systems, 2023
work page 2023
-
[3]
J. Guo, U. Kurup, and M. Shah, “Is it safe to drive? an overview of factors, metrics, and datasets for driveability assessment in autonomous driving,” 3135–3151 8 21 IEEE Transactions on Intelligent Transporta- tion Systems , 2019
work page 2019
-
[4]
A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook,
M. Liu, E. Yurtsever, J. Fossaert, X. Zhou, W. Zimmer, Y . Cui, B. L. Zagar, and A. C. Knoll, “A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook,” IEEE Transactions on Intelligent V ehicles, 2024
work page 2024
-
[5]
A survey of autonomous driving: Common practices and emerging technologies,
E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” 58443–58469 8 IEEE access , 2020
work page 2020
-
[6]
Trajectory data mining: an overview,
Y . Zheng, “Trajectory data mining: an overview,” 1–41 3 6 ACM Transactions on Intelligent Systems and Technology (TIST) , 2015
work page 2015
-
[7]
Dynosam: Open-source smoothing and mapping framework for dynamic slam,
J. Morris, Y . Wang, M. Kliniewski, and V . Ila, “Dynosam: Open-source smoothing and mapping framework for dynamic slam,” arXiv preprint arXiv:2501.11893, 2025
arXiv 2025
-
[8]
Multimotion Visual Odometry (MVO),
K. M. Judd and J. D. Gammell, “Multimotion Visual Odometry (MVO),” Intl. J. of Robotics Research , 2024
work page 2024
Show all 33 references
-
[9]
Gonzalez, E
M. Gonzalez, E. Marchand, A. Kacete, and J. Royan, ” Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) “Twist- slam++: Fusing multiple modalities for accurate dynamic semantic slam. 9126–9132 IEEE, 2023
2023
-
[10]
Nicholson, M
L. Nicholson, M. Milford, and N. S ¨underhauf, ” Proc. of the IEEE/CVF Intl. Conf. Computer Vision and Pattern Recognition “QuadricSLAM: Dual Quadrics as SLAM Landmarks, 313–314 2018
2018
-
[11]
Dy- naquadric: Dynamic quadric slam for quadric initialization, mapping, and tracking,
R. Tian, Y . Zhang, L. Yang, J. Zhang, S. Coleman, and D. Kerr, “Dy- naquadric: Dynamic quadric slam for quadric initialization, mapping, and tracking,” 17234-17246 11 25 IEEE Transactions on Intelligent Transportation Systems, 2024
2024
-
[12]
Morris, Y
J. Morris, Y . Wang, and V . Ila, ” Proc. of the IEEE Intl. Conf. on Robotics and Automation (ICRA) “The importance of coordinate frames in dynamic slam, 2024
2024
-
[13]
Twistslam: Constrained slam in dynamic environment,
M. Gonzalez, E. Marchand, A. Kacete, and J. Royan, “Twistslam: Constrained slam in dynamic environment,” 6846–6853 3 7 IEEE Robotics and Automation Letters , 2022
2022
-
[14]
A survey on trajectory-prediction methods for autonomous driving,
Y . Huang, J. Du, Z. Yang, Z. Zhou, L. Zhang, and H. Chen, “A survey on trajectory-prediction methods for autonomous driving,” 652–674 3 7 IEEE Transactions on Intelligent V ehicles , 2022
2022
-
[15]
Review of pedestrian trajectory prediction methods: Comparing deep learning and knowledge-based approaches,
R. Korbmacher and A. Tordeux, “Review of pedestrian trajectory prediction methods: Comparing deep learning and knowledge-based approaches,” 24126–24144 12 23 IEEE Transactions on Intelligent Transportation Systems, 2022
2022
-
[16]
Exploring tra- jectory prediction through machine learning methods,
C. Wang, L. Ma, R. Li, T. S. Durrani, and H. Zhang, “Exploring tra- jectory prediction through machine learning methods,” 101441–101452 7 IEEE Access , 2019
2019
-
[17]
Deep learning-based vehicle behavior prediction for autonomous driving applications: A review,
S. Mozaffari, O. Y . Al-Jarrah, M. Dianati, P. Jennings, and A. Mouzaki- tis, “Deep learning-based vehicle behavior prediction for autonomous driving applications: A review,” 33–47 1 23 IEEE Transactions on Intelligent Transportation Systems , 2020
2020
-
[18]
J. Yue, D. Manocha, and H. Wang, ” Proc. of the European Conf. on Computer Vision (ECCV) “Human trajectory prediction via neural social physics. 376–394 Springer, 2022
2022
-
[19]
Gupta, J
A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, ” Proc. of the IEEE Intl. Conf. Computer Vision and Pattern Recognition “Social gan: Socially acceptable trajectories with generative adversarial networks, 2255–2264 2018
2018
-
[20]
N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. Torr, and M. Chandraker, ” Proc. of the IEEE Intl. Conf. Computer Vision and Pattern Recognition “Desire: Distant future prediction in dynamic scenes with interacting agents, 336–345 2017
2017
-
[21]
Sch ¨afer, K
M. Sch ¨afer, K. Zhao, and A. Kummert, ” IEEE Intelligence V ehicles Symp. (IV) “Caspnet++: Joint multi-agent motion prediction. 1294– 1301 IEEE, 2024
2024
-
[22]
L. Feng, M. Bahari, K. M. B. Amor, ´E. Zablocki, M. Cord, and A. Alahi, ” Proc. of the European Conf. on Computer Vision (ECCV) “Unitraj: A unified framework for scalable vehicle trajectory prediction. 106–123 Springer, 2024
2024
-
[23]
Ivanovic, J
B. Ivanovic, J. Harrison, and M. Pavone, ” Proc. of the IEEE Intl. Conf. on Robotics and Automation (ICRA) “Expanding the deployment envelope of behavior prediction via adaptive meta-learning, 2023
2023
-
[24]
Sce- narionet: Open-source platform for large-scale traffic scenario simulation and modeling,
Q. Li, Z. M. Peng, L. Feng, Z. Liu, C. Duan, W. Mo, and B. Zhou, “Sce- narionet: Open-source platform for large-scale traffic scenario simulation and modeling,” 36 Conf. on Neural Information Processing Systems (NIPS), 2024
2024
-
[25]
borglab/gtsam,
F. Dellaert and GTSAM Contributors, “borglab/gtsam,” May 2022 https://github.com/borglab/gtsam
2022
-
[26]
Ultralytics YOLO,
G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLO,” Jan. 2023 https://github.com/ultralytics/ultralytics
2023
-
[27]
Teed and J
Z. Teed and J. Deng, ” Proc. of the European Conf. on Computer Vision (ECCV) “Raft: Recurrent all-pairs field transforms for optical flow. 402– 419 Springer, 2020
2020
-
[28]
Zhang, M
J. Zhang, M. Henein, R. Mahony, and V . Ila, ” Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) “Robust Ego and Object 6-DoF Motion Estimation and Tracking, 5017–5023 2020
2020
-
[29]
Zhang, P
Y . Zhang, P. Sun, Y . Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, ” Proc. of the European Conf. on Computer Vision (ECCV) “Bytetrack: Multi-object tracking by associating every detection box. 1–21 Springer, 2022
2022
-
[30]
VDO-SLAM: A Visual Dynamic Object-aware SLAM System,
J. Zhang, M. Henein, R. Mahony, and V . Ila, “VDO-SLAM: A Visual Dynamic Object-aware SLAM System,” arXiv preprint arXiv:2005.11052, 2020
2005 arXiv
-
[31]
Better unicycle models,
S. LaValle, “Better unicycle models,” 743–743 Planning Algorithms , 2006
2006
-
[32]
Dynamo-depth: fixing unsupervised depth estimation for dynamical scenes,
Y . Sun and B. Hariharan, “Dynamo-depth: fixing unsupervised depth estimation for dynamical scenes,” 36 Conf. on Neural Information Processing Systems (NIPS) , 2024
2024
-
[33]
Sturm, N
J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, ” Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) “A benchmark for the evaluation of rgb-d slam systems. IEEE, 2012
2012
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.