Pith. sign in

REVIEW 4 major objections 5 minor 12 references

Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that training on a larger dataset does not automatically make trajectory prediction models more robust to out-of-distribution data, and that the smallest model with polynomial representation and homogeneous augmentation…

desk verdict Honest and useful cross-dataset study, but the headline OoD conclusion is confounded by the authors' own homogenization protocol, which makes the Waymo task easier than Argoverse. read the letter →

arxiv 2501.15842 v1 pith:53RGTONY submitted 2025-01-27 cs.LG cs.AIstat.ML

Yue Yao , Daniel Goehring , Joerg Reichardt This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords trajectorypredictionout-of-distributiongeneralizationautonomousdrivinginductivebiasdataaugmentationpolynomialrepresentationArgoverse2WaymoOpenMotion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to test whether trajectory prediction models that score well on in-distribution benchmarks also generalize to data from a different motion dataset. The authors train three models with different inductive biases on Argoverse 2 and Waymo Open Motion in both directions, using a homogenization protocol to align history length, prediction horizon, map elements, and focal-agent selection. They find that the smallest model, EP-Q, which represents trajectories as polynomials and treats focal and non-focal agents identically during training, shows the best generalization when trained on the smaller Argoverse 2 dataset and tested on the larger Waymo dataset. In the reverse direction, training on the larger Waymo dataset makes all models generalize poorly to Argoverse 2, so more training data did not deliver the expected robustness gain. The paper concludes that out-of-distribution testing should be a standard complement to in-distribution benchmarks.

What carries the argument

The argument is carried by three design mechanisms: polynomial data representation, in which trajectories are expressed as coefficients of low-degree polynomials rather than raw point sequences, imposing an inductive bias that restricts trajectory shapes; homogeneous augmentation, in which focal and non-focal agents are treated identically, each in its own coordinate frame, with equal loss weighting; and the dataset homogenization protocol that aligns history length (5 s), prediction horizon (4.1 s), map elements, and focal-agent selection across Argoverse 2 and Waymo Open Motion so cross-dataset errors can be compared. The small EP-Q model combines the first two mechanisms, and the protocol defines what counts as a fair out-of-distribution test.

What would settle it

Train all models on Waymo's original 1.1-second history task instead of the homogenized 5-second history and test on Argoverse 2; if the large out-of-distribution error increase persists, the claim that more data fails to improve robustness is supported, but if it shrinks, the reported generalization failure is an artifact of task simplification.

Watch

Extended reading notes

Core claim

The central claim is that out-of-distribution robustness of trajectory predictors is governed by model design and dataset properties, not simply by training set size. With models that are competitive in in-distribution performance, the model with the strongest inductive bias, EP-Q, which uses low-degree polynomial input and output representations plus homogeneous augmentation, achieves the smallest error increase when moving from Argoverse 2 to Waymo Open Motion. In the opposite setting, all models, including EP-Q, show substantially larger error increases when trained on the larger Waymo dataset and tested on the smaller Argoverse 2 dataset, contradicting the expectation that more training data improves out-of-distribution generalization. The paper attributes this asymmetry to two dataset properties: the homogenized Waymo prediction task becomes simpler when its history is extended to 5 seconds, and Argoverse 2 has higher trajectory noise than Waymo. It argues that out-of-distribution evaluation should complement standard in-distribution benchmarks.

Load-bearing premise

The comparison is a fair test of robustness only if the homogenization protocol makes the prediction task equally hard in both datasets, and the paper's own appendix suggests the Waymo task becomes easier when its history is extended to 5 seconds.

Editorial extensions

If this is right

  • Out-of-distribution error should be reported alongside in-distribution leaderboard scores, because the model that tops an in-distribution leaderboard may not be the most robust across datasets.
  • Model capacity alone does not determine robustness: the smallest model with the strongest inductive bias can generalize better than much larger sequence-based models.
  • Increasing training data volume does not automatically improve cross-dataset robustness; the difficulty and noise level of the prediction task matter just as much.
  • Benchmark homogenization choices such as history length are part of the distribution, and extending Waymo's history to 5 seconds appears to make its prediction task easier, which can mask or exaggerate robustness differences.
  • For autonomous driving, out-of-distribution robustness should be a design goal in itself, motivating polynomial representations and homogeneous augmentation in future models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the homogenization protocol is the real driver, then a fair cross-dataset test would need to match task difficulty, not just data format; the paper's own appendix shows the 5-second-history Waymo task deviates less from constant velocity, so the reported asymmetry may be partly an artifact of task simplification.
  • The noise-level hypothesis is testable: injecting synthetic noise into Waymo trajectories or denoising Argoverse 2 data should shift the robustness ranking if noise, rather than task complexity, is the causal factor.
  • A practical implication the authors leave implicit is that fleet operators should not choose a trajectory model by in-distribution benchmark rank alone, because out-of-distribution behavior can invert the ranking.
  • If this pattern holds more broadly, benchmark suites should include deliberately matched out-of-distribution splits with controlled difficulty and noise, rather than relying on whole-dataset swaps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a cross-dataset out-of-distribution (OoD) evaluation of three trajectory prediction models (FMAE, QCNet, and the authors' EP variants) trained on Argoverse 2 (A2) and Waymo Open Motion (WO) and tested on the other dataset. Using a homogenization protocol to align histories, horizons, map elements, and focal-agent selection, the authors find that when models are trained on the smaller A2 dataset and tested on WO, the polynomial-representation model EP-Q with homogeneous augmentation generalizes best. In the reverse setting, training on the larger WO dataset and testing on A2, all models degrade substantially; EP-Q still shows the smallest degradation. The paper interprets this as evidence that more training data does not automatically improve OoD robustness and argues that OoD evaluation should complement standard in-distribution benchmarks.

Significance. If the finding is robust, this is a useful empirical contribution: it applies an explicit cross-dataset protocol to three modern architectures, compares representation and augmentation strategies, and introduces delta metrics for OoD robustness. The paper is transparent about its protocol and includes a concrete complexity analysis in Appendix A. It also states two alternative explanations for the main result, which is a strength. However, the central WO-trained-to-A2 conclusion is not yet causally identified because the homogenization protocol changes prediction-task difficulty between datasets, and the paper's own Appendix A documents exactly such a difference. The absence of repeated seeds and the unequal treatment of QCNet further weaken the quantitative claims. The contribution is defensible as a preliminary empirical study, but the headline conclusion needs additional controls.

major comments (4)
  1. [Section 3, Table 2, Appendix A] The central claim that training on the larger WO dataset leads to poor OoD generalization on A2 is confounded by task complexity. The homogenization protocol extends WO history from 1.1s to 5s and caps the prediction horizon at 4.1s (Table 2). Appendix A shows via Eq. (1) and Figure 3 that the 5s-history WO prediction task has a markedly narrower distribution of deviations from constant velocity than the 1.1s-history task; Section 4.3.3 itself states that this 'potentially moves the challenging behaviors into the historical data.' Since A2 retains its native 5s history and its focal agents are selected for harder maneuvers, models trained on the easier WO task would be expected to have larger errors on A2 even with ideal OoD robustness. The conclusion that 'more data does not automatically improve OoD robustness' therefore requires a complexity-matched control, such as training also on 1.1s-history WO or reporting a constant-velocity baseline on both test sets. Without such a control, the headline finding is not yet supported.
  2. [Section 4.1, Table 3, Figure 2] All results appear to come from a single training run, with no error bars, repeated seeds, or statistical significance tests. Some in-distribution differences are very small (e.g., WO minADE1: QCNet 0.820 vs EP-Q 0.821), and the OoD deltas in Figure 2 are derived from these single runs. The ranking of EP-Q as the most robust model, and the claim that all models generalize poorly, need to be quantified against run-to-run variance; otherwise the main comparisons may reflect optimization noise rather than model properties.
  3. [Section 4.1, Section 4.3.1] QCNet was trained with truncated scene context (up to 50 agents and 80 map elements) because of GPU memory limits, while FMAE and the EP models were not truncated. This unequal treatment directly affects the cross-model comparison: Section 4.3.1 attributes QCNet's worse OoD robustness under homogeneous augmentation to this limited scenario complexity. Since the paper's conclusions compare augmentation strategies and data representations across models, QCNet's numbers are not on equal footing with the other models. The authors should either run QCNet with full context on appropriate hardware or explicitly state that all comparisons involving QCNet are upper bounds on its true performance.
  4. [Section 4.3.3] The paper states that Appendix A supports the complexity hypothesis, but Appendix A only demonstrates that longer history narrows the distribution of constant-velocity deviations; it does not connect this distributional difference to the observed model error increases on A2. A direct test, such as comparing model errors on complexity-matched subsets of A2 and WO, would be needed to validate the proposed explanation.
minor comments (5)
  1. [Section 3] The sentence describing focal-agent selection in WO says the list is unordered, so choosing the first fully observed non-ego agent corresponds to random sampling; this detail is useful, but it also means the selected WO agent is not necessarily the most challenging agent, which compounds the task-complexity issue and should be listed as an additional limitation.
  2. [Table 3] The label 'w/o pre-train' is ambiguous: the text should clarify whether FMAE was otherwise pre-trained (e.g., on a different dataset) and what the official FMAE configuration is, since pretraining status can materially affect both ID and OoD performance.
  3. [Section 4.1] Notation is inconsistent: the text introduces minADEK and minFDEK with K as a subscript, while Table 3 and Figure 2 use minADE1/minADE6 without subscripting. Please unify the notation across the manuscript.
  4. [Figure 2] The transparent bars representing the OoD error increase are difficult to distinguish from the solid ID bars, especially in grayscale. Consider using distinct hatching or a clearer legend so the delta values are visually separable.
  5. [Section 4.3.3] The phrase 'We present our experimental results to support this hypothesis in Appendix A' is misleading: Appendix A characterizes the datasets but does not present model-behavior results. Please rephrase to avoid implying causal evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cross-dataset OoD results are new empirical measurements against external benchmarks; the cited prior protocol is a methodological input, not a derived prediction.

full rationale

This paper is an empirical cross-dataset generalization study, not a derivation. Its core comparisons are new experiments: models are trained on WO and evaluated on A2, and vice versa, using the same homogenized task definition, and benchmarked against two external open-source models (FMAE and QCNet). No parameter is fitted to the OoD test split, and no reported quantity is defined in terms of the claim it supports. The prior-work citations ([3], [11]) supply the EP model architecture, the homogenization protocol, and a noise analysis; these are methodological inputs, not results that the paper 'predicts'. The closest candidate for circularity is the protocol's history extension (Table 2), which Appendix A shows makes the WO prediction task simpler, and the paper itself acknowledges this: 'This adjustment potentially moves the challenging behaviors into the historical data, effectively making the prediction task less complex for WO.' However, this is an explicitly stated confound and interpretation, not a derivation step: the OoD error numbers are measured, not constructed. A complexity mismatch would be a validity threat to the conclusion that more data does not improve OoD robustness, and belongs under correctness risk, not circularity. The self-citations are not load-bearing in the sense required for circularity: they identify model components and protocol choices that are then tested empirically against external benchmarks on public datasets.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The empirical claims rely on protocol choices and assumptions about dataset comparability rather than a formal derivation. The main hand-chosen constants are the homogenization settings and the QCNet scene limits; the paper introduces no new physical or conceptual entities. The EP model is an architecture from the authors' prior work, evaluated here, not an invented entity.

free parameters (4)
  • prediction horizon = 4.1 s
    Chosen because WO recordings are 9.1s long; all models tested with this horizon, constraining the OoD comparison.
  • history length = 5 s
    Inherited from A2 and imposed on WO, whose native history is 1.1s; this may change task difficulty (Appendix A) and confound OoD transfer.
  • QCNet scene truncation = 50 agents; 80 map elements
    Required to fit QCNet into the available GPU memory; only QCNet is truncated, so OoD comparisons involving QCNet are potentially unfair.
  • focal agent selection in WO = first fully observed, non-ego agent
    WO has up to 8 focal agents; picking the first listed is effectively random and may not match A2's curated focal agent distribution.
assumptions (5)
  • domain assumption The homogenized A2 and WO datasets form a valid OoD pair.
    Section 3 assumes that after unifying history length, horizon, map elements, and focal agent count, error increases reflect model robustness rather than protocol mismatch.
  • domain assumption Original hyperparameters remain appropriate under the homogenized protocol.
    Section 4.1 trains all models on WO using the original hyperparameters from their source papers.
  • domain assumption minADE/minFDE and their differences capture OoD robustness.
    The paper introduces Delta metrics as differences of ID and OoD displacement errors, but these inherit the limitations of the base metrics and do not directly measure distribution shift.
  • domain assumption Deviation from constant velocity is a valid proxy for prediction task complexity.
    Appendix A uses this proxy to argue WO becomes less complex with a 5s history, but this is not proven to capture all sources of prediction difficulty.
  • domain assumption Noise-level differences between A2 and WO from prior work apply to these experiments.
    Section 4.3.3 invokes lower noise in WO vs A2 from the authors' prior study [11] to explain asymmetric transfer, without direct verification in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness." pith.science (2026). https://pith.science/paper/53RGTONY

@misc{pith2026250115842,
  author       = {Pith},
  title        = {Pith review of: Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/53RGTONY}},
  note         = {Machine review of arXiv:2501.15842}
}
read the original abstract

We study the Out-of-Distribution (OoD) generalization ability of three SotA trajectory prediction models with comparable In-Distribution (ID) performance but different model designs. We investigate the influence of inductive bias, size of training data and data augmentation strategy by training the models on Argoverse 2 (A2) and testing on Waymo Open Motion (WO) and vice versa. We find that the smallest model with highest inductive bias exhibits the best OoD generalization across different augmentation strategies when trained on the smaller A2 dataset and tested on the large WO dataset. In the converse setting, training all models on the larger WO dataset and testing on the smaller A2 dataset, we find that all models generalize poorly, even though the model with the highest inductive bias still exhibits the best generalization ability. We discuss possible reasons for this surprising finding and draw conclusions about the design and test of trajectory prediction models and benchmarks.

Figures

Figures reproduced from arXiv: 2501.15842 by the authors.

Figure 1
Figure 1. The two augmentation strategies for non-focal agent data employed in bench [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The OoD testing results of FMAE, QCNet, EP and their variants. We indicate [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Kernel density plot of normalized longitudinal and lateral distances with 1.1- [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

  1. [1]

    Argoverse 2: Next generation datasets for self-driving perception and forecasting,

    B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Ku- mar, A. Hartnett, J. K. Pontes, et al. , “Argoverse 2: Next generation datasets for self-driving perception and forecasting,” in Thirty-fifth Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track (Round 2) , 2021

    work page 2021

  2. [2]

    Large scale interactive motion forecasting for autono- mous driving: The waymo open motion dataset,

    S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, et al. , “Large scale interactive motion forecasting for autono- mous driving: The waymo open motion dataset,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 9710–9719, 2021

    work page 2021

  3. [3]

    Improving out-of- distribution generalization of trajectory prediction for autonomous driving via poly- nomial representations,

    Y. Yao, S. Yan, D. Goehring, W. Burgard, and J. Reichardt, “Improving out-of- distribution generalization of trajectory prediction for autonomous driving via poly- nomial representations,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 488–495, IEEE, 2024

    work page 2024

  4. [4]

    Deep learning scaling is predictable, empirically,

    J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou, “Deep learning scaling is predictable, empirically,” CoRR, vol. abs/1712.00409, 2017

    arXiv 2017

  5. [5]

    FMAE-noAug

    (FMAE) and QCNet [6], with nearly 1900k and 7600k parameters, respectively. As summarized in Table 1, both models employ sequence-based representation but employ different strategies in dealing with non-focal agents: Heterogeneous Augmentation: FMAE follows the prediction competition protocol and prioritizes focal agent prediction. Thus, agent history and...

    work page

  6. [6]

    Forecast-mae: Self-supervised pre-training for motion forecasting with masked autoencoders,

    J. Cheng, X. Mei, and M. Liu, “Forecast-mae: Self-supervised pre-training for motion forecasting with masked autoencoders,” in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , pp. 8679–8689, 2023

    work page 2023

  7. [7]

    Query-centric trajectory predicti- on,

    Z. Zhou, J. Wang, Y.-H. Li, and Y.-K. Huang, “Query-centric trajectory predicti- on,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17863–17873, 2023

    work page 2023

  8. [8]

    Plop: Probabilistic polynomial objects trajectory planning for autonomous driving,

    T. Buhet, E. Wirbel, A. Bursuc, and X. Perrotton, “Plop: Probabilistic polynomial objects trajectory planning for autonomous driving,” CoRR, vol. abs/2003.08744, 2020

    arXiv 2003

Show all 12 references
  1. [9]

    Temporally-continuous probabilistic prediction using polynomial trajectory para- meterization,

    Z. Su, C. Wang, H. Cui, N. Djuric, C. Vallespi-Gonzalez, and D. Bradley, “Temporally-continuous probabilistic prediction using polynomial trajectory para- meterization,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 3837–3843, IEEE, 2021

    2021

  2. [10]

    Trajectories as markov-states for long term traffic scene prediction,

    J. Reichardt, “Trajectories as markov-states for long term traffic scene prediction,” in 14-th UniDAS F AS-Workshop, p. 14, 2022

    2022

  3. [11]

    Unitraj: A unified framework for scalable vehicle trajectory prediction,

    L. Feng, M. Bahari, K. M. B. Amor, ´E. Zablocki, M. Cord, and A. Alahi, “Unitraj: A unified framework for scalable vehicle trajectory prediction,” in European Conference on Computer Vision , pp. 106–123, Springer, 2025

    2025

  4. [12]

    An empirical bayes analysis of object tra- jectory representation models,

    Y. Yao, D. Goehring, and J. Reichardt, “An empirical bayes analysis of object tra- jectory representation models,” in IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pp. 902–909, IEEE, 2023. Appendix A: Complexity of Prediction Task We analyze ...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.