REVIEW 2 major objections 2 minor 41 references
Towards Metric-Agnostic Trajectory Forecasting
T0 review · 2 major / 2 minor · reviewed 2026-07-02 · grok-4.3
Pith's one-line read Training trajectory forecasters on metric-agnostic probabilistic objectives and applying metric-specific policies downstream achieves state-of-the-art results across all Waymo metrics.
desk verdict The paper reframes trajectory forecasting as NLL training plus metric-specific TraDiE policies, claims SOTA on all Waymo metrics with DONUT-NLL, but the abstract supplies no evidence that the post-hoc policies avoid mode loss or selection bias on conflicting metrics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Trajectory Distribution Evaluation (TraDiE) policies, which are metric-specific mappings from a predictive distribution over trajectories to the K trajectories and associated confidences required by evaluation metrics.
What would settle it
A direct comparison in which the original DONUT model is retrained from scratch on each individual Waymo metric and still outperforms DONUT-NLL plus TraDiE policies on those same metrics would falsify the advantage of the metric-agnostic approach.
Extended reading notes
Core claim
The authors argue that because benchmark metrics encourage incompatible behaviors, models should be trained to produce accurate predictive distributions rather than to optimize any one metric directly. They introduce TraDiE policies as the mechanism for extracting the required K trajectories from such a distribution in a metric-specific way. When the DONUT architecture is retrained with an NLL objective to produce these distributions, the combination of the learned distribution and the TraDiE policies yields state-of-the-art performance on all metrics of the Waymo motion prediction benchmark.
Load-bearing premise
That metric-specific TraDiE policies applied after NLL training on the predictive distribution can recover or exceed the performance of models trained directly on the target metrics without introducing selection bias or information loss.
Editorial extensions
If this is right
- Forecasting models can be trained once and then evaluated on multiple conflicting metrics without retraining.
- The quality of the predictive distribution becomes the primary training target rather than any single scalar metric.
- Downstream tasks that require different trajectory selections can reuse the same trained model by swapping the extraction policy.
- Performance gains on one metric no longer come at the expense of others.
Reading between the lines
- Such an approach could reduce the computational cost of developing new forecasting systems by eliminating the need for metric-specific retraining cycles.
- If the predictive distribution is sufficiently accurate, the same model could support entirely new metrics that are introduced after training.
- Extending TraDiE policies to other datasets like Argoverse 2 might reveal whether the metric-agnostic advantage holds beyond Waymo.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes shifting trajectory forecasting from metric-specific training objectives to a metric-agnostic paradigm: models are trained with negative log-likelihood (NLL) on the predictive distribution, after which metric-specific Trajectory Distribution Evaluation (TraDiE) policies map the distribution to the K trajectories and confidences required by benchmarks. The authors introduce DONUT-NLL (an NLL-adapted version of DONUT) and claim that, when paired with these policies, it achieves state-of-the-art results on every metric of the Waymo Open Motion Dataset.
Significance. If the central empirical claim holds, the work would be significant because it decouples training from benchmark metrics that encourage mutually inconsistent behaviors, potentially allowing a single model to serve multiple downstream tasks. The introduction of TraDiE policies is a concrete, reusable contribution; the paper also receives credit for adapting an existing strong architecture (DONUT) to a probabilistic objective rather than inventing an entirely new model.
major comments (2)
- [Abstract and §4 (Experiments)] Abstract and §4 (Experiments): the headline claim that DONUT-NLL + TraDiE reaches SOTA on all Waymo metrics is load-bearing yet unsupported by any reported baselines, K values, metric scores, or ablation showing that the NLL distribution supplies sufficient modes for every TraDiE policy simultaneously.
- [§3 (TraDiE policies)] §3 (TraDiE policies): the policies are defined to extract K trajectories from an NLL-trained distribution, but no analysis, bound, or ablation addresses whether selection or sampling steps introduce bias or discard probability mass when metrics conflict, which directly undermines the claim that post-hoc policies can recover or exceed direct metric training.
minor comments (2)
- [§3] Notation for the mapping performed by TraDiE policies would benefit from an explicit equation relating the predictive distribution p(y|x) to the final set of K trajectories and scores.
- [§4] The manuscript should clarify whether the reported Waymo numbers use the official test server or a validation split, and whether any hyper-parameters of the TraDiE policies were tuned on the target metrics.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback. We address the two major comments below and will revise the manuscript to strengthen the empirical presentation and analysis.
read point-by-point responses
-
Referee: [Abstract and §4 (Experiments)] Abstract and §4 (Experiments): the headline claim that DONUT-NLL + TraDiE reaches SOTA on all Waymo metrics is load-bearing yet unsupported by any reported baselines, K values, metric scores, or ablation showing that the NLL distribution supplies sufficient modes for every TraDiE policy simultaneously.
Authors: We agree the claim requires explicit support. While the abstract summarizes the outcome, §4 does not tabulate the full baseline comparisons, K values, or per-metric scores. In revision we will insert a results table in §4 reporting DONUT-NLL + each TraDiE policy against prior SOTA on all Waymo metrics (K=6), plus an ablation confirming the NLL distribution supplies adequate modes for simultaneous policy use. revision: yes
-
Referee: [§3 (TraDiE policies)] §3 (TraDiE policies): the policies are defined to extract K trajectories from an NLL-trained distribution, but no analysis, bound, or ablation addresses whether selection or sampling steps introduce bias or discard probability mass when metrics conflict, which directly undermines the claim that post-hoc policies can recover or exceed direct metric training.
Authors: The policies are constructed to draw from the full learned distribution, but we acknowledge the absence of quantitative checks on mass retention or bias under metric conflict. In revision we will add an ablation in §3 (or §4) that measures retained probability mass for each policy, compares against direct metric training, and reports any observed bias, thereby supporting the post-hoc paradigm. revision: yes
Circularity Check
No significant circularity; derivation is self-contained
full rationale
The paper trains models using a standard negative log-likelihood objective on the predictive distribution (metric-agnostic), then applies newly defined TraDiE policies as a post-hoc mapping from that distribution to the K trajectories required by each benchmark metric. The SOTA claim on Waymo is an empirical result from evaluating this two-stage process on an external dataset, with no reduction of the reported performance to a fitted parameter renamed as prediction, no self-definitional loop in the equations, and no load-bearing self-citation that collapses the central argument to prior unverified work by the same authors. The derivation chain remains independent of the target metrics during training.
Assumptions & free parameters
assumptions (1)
- domain assumption Negative log-likelihood is an appropriate objective for learning multimodal trajectory distributions
invented entities (1)
-
TraDiE policies
Cite this review
Pith. "Pith review of Towards Metric-Agnostic Trajectory Forecasting." pith.science (2026). https://pith.science/paper/UZXOQHRT
@misc{pith2026260701133,
author = {Pith},
title = {Pith review of: Towards Metric-Agnostic Trajectory Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/UZXOQHRT}},
note = {Machine review of arXiv:2607.01133}
}
abstract
Accurate trajectory forecasting of surrounding traffic participants is a core capability for autonomous driving, enabling vehicles to anticipate behavior and plan safe maneuvers. We observe that current state-of-the-art forecasting models on Argoverse 2 and the Waymo Open Motion Dataset tailor their training objectives to the different benchmark metrics. Because these metrics encourage conflicting behavior, we propose a paradigm change for trajectory forecasting: training models with metric-agnostic probabilistic objectives and treating metric optimization as a downstream task applied to the predictive distribution. Concretely, we introduce Trajectory Distribution Evaluation (TraDiE) policies, metric-specific policies that map a predictive distribution to the set of $K$ trajectories and confidences required by trajectory forecasting metrics. We evaluate this framework by introducing DONUT-NLL, which adapts the training objective of the state-of-the-art trajectory forecasting model DONUT to directly optimize the predictive distribution. Using our policies, DONUT-NLL achieves state-of-the-art results on all metrics of the Waymo motion prediction benchmark.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Chai, Y., Sapp, B., Bansal, M., Anguelov, D.: MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction. In: CoRL (2020)
work page 2020
-
[2]
Cui,H.,Radosavljevic,V.,Chou,F.C.,Lin,T.H.,Nguyen,T.,Huang,T.K.,Schnei- der, J., Djuric, N.: Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks. In: ICRA (2019)
work page 2019
-
[3]
Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Pradhan, S., Chai, Y., Sapp, B., Qi, C.R., Zhou, Y., Yang, Z., Chouard, A., Sun, P., Ngiam, J., Vasudevan, V., McCauley, A., Shlens, J., Anguelov, D.: Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset. In: ICCV (2021)
work page 2021
-
[4]
Everingham,M.,VanGool,L.,Williams,C.K.,Winn,J.,Zisserman,A.:Thepascal visual object classes (voc) challenge. IJCV (2010)
work page 2010
-
[5]
Gao, J., Sun, C., Zhao, H., Shen, Y., Anguelov, D., Li, C., Schmid, C.: Vector- Net: Encoding HD Maps and Agent Dynamics from Vectorized Representation. In: CVPR (2020)
work page 2020
-
[6]
Gilles, T., Sabatini, S., Tsishkou, D., Stanciulescu, B., Moutarde, F.: HOME: Heatmap Output for future Motion Estimation. In: ITSC (2021)
work page 2021
-
[7]
Gilles, T., Sabatini, S., Tsishkou, D., Stanciulescu, B., Moutarde, F.: GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation. In: ICRA (2022)
work page 2022
-
[8]
Gilles, T., Sabatini, S., Tsishkou, D., Stanciulescu, B., Moutarde, F.: THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling. In: ICLR (2022)
work page 2022
Show all 41 references
-
[9]
In: Hochbaum, D.S
Hochbaum, D.S.: Approximating Covering and Packing Problems: Set Cover, Ver- tex Cover, Independent Set, and Related Problems. In: Hochbaum, D.S. (ed.) Ap- proximation Algorithms for NP-hard Problems, chap. 3, pp. 94–143. PWS Pub- lishing Company (1997)
1997
-
[10]
In: CVPR (2019)
Hong, J., Sapp, B., Philbin, J.: Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions. In: CVPR (2019)
2019
-
[11]
In: ICML (2025)
Huang, X., Wolff, E.M., Vernaza, P., Phan-Minh, T., Chen, H., Hayden, D.S., Edmonds, M., Pierce, B., Chen, X., Jacob, P.E., Chen, X., Tairbekov, C., Agarwal, P., Gao, T., Chai, Y., Srinivasa, S.: DriveGPT: Scaling Autoregressive Behavior Models for Driving. In: ICML (2025)
2025
-
[12]
In: ICLR (2015)
Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. In: ICLR (2015)
2015
-
[13]
In: ICLR (2014)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: ICLR (2014)
2014
-
[14]
In: ICCV (2025)
Knoche, M., de Geus, D., Leibe, B.: DONUT: A Decoder-Only Model for Trajec- tory Prediction. In: ICCV (2025)
2025
-
[15]
In: ICLR (2024)
Lan, Z., Jiang, Y., Mu, Y., Chen, C., Li, S.E.: SEPT: Towards Efficient Scene Representation Learning for Motion Prediction. In: ICLR (2024)
2024
-
[16]
In: CVPR (2017)
Lee, N., Choi, W., Vernaza, P., Choy, C.B., Torr, P.H., Chandraker, M.: Desire: Distant future prediction in dynamic scenes with interacting agents. In: CVPR (2017)
2017
-
[17]
In: ECCV (2020)
Liang, M., Yang, B., Hu, R., Chen, Y., Liao, R., Feng, S., Urtasun, R.: Learning Lane Graph Representations for Motion Forecasting. In: ECCV (2020)
2020
-
[18]
In: AAAI (2024)
Lin, L., Lin, X., Lin, T., Huang, L., Xiong, R., Wang, Y.: EDA: Evolving and Distinct Anchors for Multimodal Motion Prediction. In: AAAI (2024)
2024
-
[19]
In: CVPR (2021)
Liu, Y., Zhang, J., Fang, L., Jiang, Q., Zhou, B.: Multimodal Motion Prediction with Stacked Transformers. In: CVPR (2021)
2021
-
[20]
In: CVPR (2020) Towards Metric-Agnostic Trajectory Forecasting 17
Mohamed, A., Qian, K., Elhoseiny, M., Claudel, C.: Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Pre- diction. In: CVPR (2020) Towards Metric-Agnostic Trajectory Forecasting 17
2020
-
[21]
In: ICRA (2023)
Nayakanti, N., Al-Rfou, R., Zhou, A., Goel, K., Refaat, K.S., Sapp, B.: Wayformer: Motion Forecasting via Simple & Efficient Attention Networks. In: ICRA (2023)
2023
-
[22]
In: CVPR (2020)
Phan-Minh, T., Grigore, E.C., Boulton, F.A., Beijbom, O., Wolff, E.M.: CoverNet: Multimodal Behavior Prediction using Trajectory Sets. In: CVPR (2020)
2020
-
[23]
In: NeurIPS (2024)
Ruan, H., Yu, H., Yang, W., Fan, S., Nie, Z.: Learning Cooperative Trajectory Representations for Motion Forecasting. In: NeurIPS (2024)
2024
-
[24]
In: ECCV (2020)
Salzmann, T., Ivanovic, B., Chakravarty, P., Pavone, M.: Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In: ECCV (2020)
2020
-
[25]
In: ICCV (2023)
Seff,A.,Cera,B.,Chen,D.,Ng,M.,Zhou,A.,Nayakanti,N.,Refaat,K.S.,Al-Rfou, R., Sapp, B.: MotionLM: Multi-Agent Motion Forecasting as Language Modeling. In: ICCV (2023)
2023
-
[26]
Shi, C., Shi, S., Jiang, L.: MTR v3: 1st Place Solution for 2024 Waymo Open Dataset Challenge - Motion Prediction (2024)
2024
-
[27]
In: NeurIPS (2022)
Shi, S., Jiang, L., Dai, D., Schiele, B.: Motion Transformer with Global Intention Localization and Local Movement Refinement. In: NeurIPS (2022)
2022
-
[28]
Shi, S., Jiang, L., Dai, D., Schiele, B.: MTR-A: 1st Place Solution for 2022 Waymo Open Dataset Challenge - Motion Prediction (2022)
2022
-
[29]
IEEE TPAMI (2024)
Shi, S., Jiang, L., Dai, D., Schiele, B.: MTR++: Multi-Agent Motion Prediction With Symmetric Scene Modeling and Guided Intention Querying. IEEE TPAMI (2024)
2024
-
[30]
In: NeurIPS (2025)
Song, N., Jiang, J., Li, J., Zhu, X., Zhang, L.: UniMotion: A Unified Motion Frame- work for Simulation, Prediction and Planning. In: NeurIPS (2025)
2025
-
[31]
In: ICRA (2025)
Sun, J., Li, J., Liu, T., Yuan, C., Sun, S., Huang, Z., Wong, A., Tee, K.P., Ang, M.H.: RMP-YOLO: A Robust Motion Predictor for Partially Observable Scenarios even if You Only Look Once. In: ICRA (2025)
2025
-
[32]
RAL (2025)
Sun, J., Yue, X., Li, J., Shen, T., Yuan, C., Sun, S., Guo, S., Zhou, Q., Ang Jr, M.H.: IMPACT: Behavioral Intention-Aware Multimodal Trajectory Prediction With Adaptive Context Trimming. RAL (2025)
2025
-
[33]
In: ICRA (2025)
Wang, M., Ren, X., Jin, R., Li, M., Zhang, X., Yu, C., Wang, M., Yang, W.: FutureNet-LOF: Joint Trajectory Prediction and Lane Occupancy Field Prediction with Future Context Encoding. In: ICRA (2025)
2025
-
[34]
In: NeurIPS Datasets and Benchmarks (2021)
Wilson, B., Qi, W., Agarwal, T., Lambert, J., Singh, J., Khandelwal, S., Pan, B., Kumar, R., Hartnett, A., Pontes, J.K., Ramanan, D., Carr, P., Hays, J.: Argov- erse 2: Next Generation Datasets for Self-Driving Perception and Forecasting. In: NeurIPS Datasets and Benchmarks (2021)
2021
-
[35]
In: IROS (2025)
Yan, Q., Zhang, B., Zhang, Y., Yang, D., White, J., Chen, D., Liu, J., Liu, L., Zhuang, B., Shi, S., Liao, R.: Trajflow: Multi-modal motion prediction via flow matching. In: IROS (2025)
2025
-
[36]
In: IROS (2021)
Zeng, W., Liang, M., Liao, R., Urtasun, R.: Lanercnn: Distributed representations for graph-centric motion forecasting. In: IROS (2021)
2021
-
[37]
arXiv preprint arXiv:2508.11492 (2025)
Zhang, B., Song, N., Gao, B., Zhang, L.: Relative Position Matters: Trajectory Pre- diction and Planning with Polar Representation. arXiv preprint arXiv:2508.11492 (2025)
2025
-
[38]
In: NeurIPS (2024)
Zhang, B., Song, N., Zhang, L.: DeMo: Decoupling Motion Forecasting into Direc- tional Intentions and Dynamic States. In: NeurIPS (2024)
2024
-
[39]
In: CVPR (2023)
Zhou, Z., Wang, J., Li, Y.H., Huang, Y.K.: Query-Centric Trajectory Prediction. In: CVPR (2023)
2023
-
[40]
In: CVPR (2022) 18 M
Zhou, Z., Ye, L., Wang, J., Wu, K., Lu, K.: HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction. In: CVPR (2022) 18 M. Knoche et al
2022
-
[41]
−1 2 x−µ σ 2# . (22) The generalized Gaussian uses a locationµ∈R, a scaleα >0, and a shape parameterβ >0, and is given by GG(x|µ, α, β) = β 2αΓ(1/β) exp
Zhou, Z., Zhou, H., Hu, H., Wen, Z., Wang, J., Li, Y.H., Huang, Y.K.: ModeSeq: Taming Sparse Multimodal Motion Prediction with Sequential Mode Modeling. In: CVPR (2025) Towards Metric-Agnostic Trajectory Forecasting 19 Towards Metric-Agnostic Trajectory Forecasting Supplementa...
2025
Reviewed July 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.