Pith. sign in

REVIEW 4 major objections 4 minor 20 references

Beyond Features: How Dataset Design Influences Multi-Agent Trajectory Prediction Performance

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that adding richer lane, stop-line, speed-limit, Frenet, and agent-geometry features to a top trajectory prediction model produces no measurable gain, and that dataset builders should invest in geographic and scenario…

desk verdict A useful new dataset and some genuinely informative transfer experiments, but the headline feature-sufficiency claim is underdetermined and needs more statistical and architectural support. read the letter →

arxiv 2507.05098 v1 pith:EJAJNYUT submitted 2025-07-07 cs.RO cs.AIcs.LG

Tobias Demmler , Jakob Häringer , Andreas Tamke , Thao Dang , Alexander Hegai , Lars Mikelsons This is my paper · ORCID
classification cs.ROcs.AIcs.LG
keywords multi-agenttrajectorypredictiondatasetdesignfeatureselectioncross-datasettransfergeographicdiversityautonomousdrivingQCNetArgoverse2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that dataset design decisions, not just model architecture, determine how well multi-agent trajectory prediction performs and transfers. It introduces the L4 Motion Forecasting dataset, built from two years of German and US driving recordings and formatted to match the Argoverse 2 benchmark, and evaluates the QCNet model on it. The central finding is that adding supplementary map and agent features—extra lane segments, speed limits, stop and yield lines, Frenet attributes, distances to road objects, and finer geometry—produces no measurable improvement over the baseline feature set on any reported metric. The authors conclude that modern architectures do not need extensive feature sets and that dataset builders should put their effort into geographic and scenario diversity. Supporting experiments show that pretraining on a larger US-centric dataset and fine-tuning on L4 preserves most target performance, and that a multi-country training set helps each individual region.

What carries the argument

The load-bearing mechanism is the controlled feature-ablation setup. Because the L4 dataset is constructed in the Argoverse 2 schema, the same public QCNet implementation can be trained on baseline features without modification and on expanded features with only small changes, isolating feature content while architecture, splits, and evaluation are fixed. The evaluation itself uses the Argoverse 2 benchmark metrics—b-minFDE, minADE6, minFDE6, MR6, minADE1, minFDE1, MR1—averaged over all scored agents, so the feature comparison is measured on the same scale as public leaderboards.

What would settle it

Train the same QCNet configuration on the L4 splits with a feature-blind control (e.g., constant speed limits and constant stop-line distances) and compare with the full-feature model: identical metrics would show the model ignored the extra inputs, while a difference would show the features are usable and the null result needs re-examination. Stronger still, rerun the same six feature configurations on a second architecture with an explicit lane-graph encoder; any repeatable improvement beyond run-to-run noise on the Argoverse 2 metrics would overturn the paper's general conclusion.

Watch

Extended reading notes

Core claim

On the L4 dataset, the paper compares six feature configurations—Baseline, Enhanced Lanes, Enhanced Stop Lines, Enhanced Agents, Full Integration, and Focused Integration—all trained with the same QCNet model and evaluated with the Argoverse 2 competition metrics. The numbers in Table II move only by a few hundredths of a meter (b-minFDE stays at 1.29–1.30 m and minADE6 at 0.47 m across all variants), which the authors read as no measurable benefit from the added information. Cross-dataset experiments then show that a model trained only on Argoverse 2 loses more when tested on L4 than an L4-trained model loses when tested on Argoverse 2, with L4's wider speed distribution and roundabouts cited as the reason. Splitting L4 by country, a Stuttgart-trained model beats a Sunnyvale-trained model even on the Sunnyvale test set, while the model trained on both countries is best on both. The paper concludes that richer features are unnecessary for current architectures and that geographic diversity is the more valuable dataset property.

Load-bearing premise

The load-bearing premise is that QCNet actually consumes the added features—the paper reports only that small modifications are needed, without validating the feature encoder—and that one model's non-reaction means modern architectures in general do not need richer features.

Editorial extensions

If this is right

  • The limited feature sets of public datasets suffice for state-of-the-art trajectory predictors, so dataset creators can redirect annotation effort toward more scenarios rather than more attributes per scenario.
  • Pretraining on a large, geographically different dataset and fine-tuning on the target dataset recovers most of the target-domain accuracy while retaining knowledge of the source domain.
  • Geographic diversity in training data improves performance on every region tested, even when one region dominates the data, so adding a smaller dataset from another country is a worthwhile investment.
  • Domain shift between datasets is asymmetric: a model trained on the broader, higher-speed L4 data transfers to Argoverse 2 better than an Argoverse 2-trained model transfers to L4.
  • Roundabouts and high-speed rural driving, underrepresented in US-centric benchmarks, are the kinds of scenarios that most expose cross-dataset transfer failures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the added features would help simpler architectures; its null result is specific to QCNet, so a useful extension is repeating the same feature ablation with a non-transformer baseline or a model with an explicit lane-graph encoder.
  • The averaged competition metrics may mask safety-critical differences; counting violations of stop/yield rules at roundabout entries or unsignalized intersections would be a sharper test of whether stop-line information matters in edge cases.
  • If the finding generalizes, the expensive parts of HD-map annotation (speed limits, stop lines, lane-subtype labels) would be low-value for trajectory prediction, and dataset cost could be cut by recording more diverse routes instead of annotating the same routes more finely.
  • One can make the claim falsifiable in an inexpensive way by adding a dropout or noise injection on the extra features: if the model's predictions are unchanged, the features are not being used at all.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces the L4 Motion Forecasting dataset, recorded with a sensor-equipped test vehicle in Stuttgart (Germany) and Sunnyvale (USA), and structured to be compatible with Argoverse 2. Using QCNet as the evaluation model, the authors report three main findings: (i) adding enriched map and agent features (e.g., speed limits, stop/yield lines, Frenet attributes, geometric details) yields no measurable improvement over baseline features; (ii) cross-dataset transfer between L4 and Argoverse 2 shows asymmetric degradation, with pre-training on external data and fine-tuning on the target set mitigating the domain shift; and (iii) training on the combined multi-country dataset outperforms country-specific training. The paper concludes that dataset designers should prioritize geographic and scenario diversity over feature quantity.

Significance. If the headline results were rigorously established, they would be practically important for autonomous-driving dataset design: they would imply that the feature sets provided by public benchmarks are sufficient for state-of-the-art trajectory predictors and that multi-country diversity matters more than annotation richness. The paper's strengths include the creation of a new multi-country dataset with an Argoverse-2-compatible structure, the explicit manipulation of feature subsets, and the cross-dataset and cross-country transfer experiments. However, the current manuscript does not supply the statistical evidence needed to support the central negative feature result, the feature-encoding pathway is not audited, and the geographic-diversity conclusion is confounded with data volume. The dataset and code are not released, further limiting verification. The paper acknowledges the core ambiguity in Section IV-A but does not resolve it before drawing conclusions in Section V.

major comments (4)
  1. [Section IV-A, Table II] The 'no measurable improvement' claim is not statistically grounded. All metric differences across feature sets are in the 0.01-0.02 range (e.g., minFDE1 2.39 vs. 2.37; minADE1 0.99 vs. 1.01), but no training seeds, variance estimates, confidence intervals, or significance tests are reported. Without these, the observed differences could easily be training noise, and the claim should either be supported by multiple-seed statistics or rephrased as 'no improvement was observed in this single run.'
  2. [Section IV-A and Section V] The feature-encoding pathway for the added L4 features is never specified. The paper only says that 'small modifications are required' to feed the extra features into QCNet, and neither the code nor the dataset is released. The paper itself acknowledges the ambiguity in Section IV-A ('Either, the network is already powerful enough to infer the additional features from the limited baseline features or it is not capable of leveraging the additional features'), yet Section V concludes the first reading as established fact. A positive control is needed—for example, a feature whose effect can be independently verified—to show the added channels are actually consumed by the model; without it, the null result may be an artifact of input channels that the customized encoder never exposes to the network in a usable way.
  3. [Section IV-C, Table IV] The conclusion that geographic diversity improves generalization is confounded with dataset size. The full L4 training set is larger than either country subset, and the Stuttgart subset is larger than the Sunnyvale subset, so the superior performance of the full- and Stuttgart-trained models could reflect more training data rather than diversity. The statement that 'a more diverse dataset spanning multiple countries' is the cause requires matched-size experiments, for example comparing a Stuttgart-only model trained on the same number of scenarios as the full set or a downsampled multi-country set.
  4. [Section V and Abstract] The paper generalizes from a single architecture, QCNet, to 'modern architectures' and to dataset-design recommendations. Since the additional-feature null result may depend on how QCNet's encoders handle extra inputs, this generalization is not supported by the experiments. At minimum, the conclusions should be restricted to QCNet, or a second architecture should be evaluated to test whether the null result is model-specific.
minor comments (4)
  1. [Section III-C2] The 150 ms temporal-discontinuity threshold and the 5 m stationary-trajectory cutoff are presented without justification or sensitivity analysis; since these thresholds shape the dataset statistics, a brief discussion would improve the manuscript.
  2. [Section III-C1] The quality-control description does not state how tracking errors or inconsistent agent IDs were identified; some details on the filtering procedure would improve reproducibility.
  3. [Reference list] Reference [17] appears to have a garbled author list ('Kong, and D. Global') and an incomplete title; please correct it.
  4. [Section IV-B] The claim that the L4 dataset 'poses greater inherent challenges' is inferred from lower metric values; this should be framed as a hypothesis, since evaluation difficulty can be affected by dataset composition, label quality, and scenario mix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is an empirical ablation study whose claims rest on measured metrics, not on definitions or fitted-prediction loops.

full rationale

This paper is an empirical measurement and ablation study, not a derivation with fitted constants or a prediction that reduces to its inputs. The central null result (Section IV-A, Table II) is obtained by training QCNet on explicit feature subsets and reporting benchmark metrics; no parameter is fitted to the outcome and then renamed as a prediction. The paper itself acknowledges the main internal-validity ambiguity in Section IV-A: 'Either, the network is already powerful enough to infer the additional features from the limited baseline features or it is not capable of leveraging the additional features.' That underdetermination is a validity threat, not circularity. The final interpretive sentence, 'This validates the chosen feature sets of other public datasets,' generalizes from a single model, but the baseline feature set is not defined in terms of the outcome metric, so no definitional reduction or equation-level equivalence is present. The self-citations [14] and [18] concern prior motion-prediction architectures and are not load-bearing for the dataset-design conclusions. No uniqueness theorem, no ansatz smuggled via citation, and no fitted-input-called-prediction pattern appears. Therefore the derivation chain is self-contained for the purposes of the circularity analysis, and the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claims rest on four unproved premises: QCNet as the single representative model, correct integration of the added features into QCNet, accuracy of the proprietary dataset annotations, and acceptance of the Argoverse 2 metrics. The three dataset construction thresholds (150 ms segmentation, 5 m stationary cutoff, 100 m map radius) are hand-chosen but only shape dataset composition, not any scientific constant. No new physical or conceptual entities are introduced.

free parameters (3)
  • temporal discontinuity threshold = 150 ms
    Used to segment continuous recordings into 11-second scenarios; hand-chosen and affects scenario composition.
  • stationary trajectory cutoff = 5 m
    Used to classify maneuvers as stationary in Section III-D; hand-chosen and affects the reported maneuver distribution.
  • HD map extraction radius = 100 m
    Matches Argoverse 2's map size; hand-chosen and affects the map context available to the model.
assumptions (4)
  • domain assumption QCNet is a valid state-of-the-art baseline and its public implementation faithfully reproduces its leaderboard behavior.
    Section IV relies on QCNet as the single evaluation model; the paper inherits the model's known performance on Argoverse 2 but does not reproduce leaderboard numbers.
  • ad hoc to paper The additional L4 features were correctly integrated into QCNet's input representation.
    Section IV-A states 'only small modifications are required' and 'Focused Integration' deliberately omits derivable features, but no verification of feature uptake is provided. This is the load-bearing premise behind the null result.
  • domain assumption The L4 dataset's labels, HD maps, and tracking are accurate enough for metric comparisons.
    Section III describes quality control but does not provide annotation accuracy or inter-rater checks.
  • domain assumption minADE, minFDE, and MR are accepted proxies for trajectory prediction quality.
    Section IV uses Argoverse 2 metrics; the paper does not discuss their limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Features: How Dataset Design Influences Multi-Agent Trajectory Prediction Performance." pith.science (2026). https://pith.science/paper/EJAJNYUT

@misc{pith2026250705098,
  author       = {Pith},
  title        = {Pith review of: Beyond Features: How Dataset Design Influences Multi-Agent Trajectory Prediction Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJAJNYUT}},
  note         = {Machine review of arXiv:2507.05098}
}
read the original abstract

Accurate trajectory prediction is critical for safe autonomous navigation, yet the impact of dataset design on model performance remains understudied. This work systematically examines how feature selection, cross-dataset transfer, and geographic diversity influence trajectory prediction accuracy in multi-agent settings. We evaluate a state-of-the-art model using our novel L4 Motion Forecasting dataset based on our own data recordings in Germany and the US. This includes enhanced map and agent features. We compare our dataset to the US-centric Argoverse 2 benchmark. First, we find that incorporating supplementary map and agent features unique to our dataset, yields no measurable improvement over baseline features, demonstrating that modern architectures do not need extensive feature sets for optimal performance. The limited features of public datasets are sufficient to capture convoluted interactions without added complexity. Second, we perform cross-dataset experiments to evaluate how effective domain knowledge can be transferred between datasets. Third, we group our dataset by country and check the knowledge transfer between different driving cultures.

Figures

Figures reproduced from arXiv: 2507.05098 by the authors.

Figure 1
Figure 1. Test vehicle used to record data trajectory prediction and which features do not contribute to more detailed predictions. Additionally, all of these public datasets are most likely less detailed than the proprietary version used internally by these companies. Based on a selection of our own data recorded by our test vehicles (see [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Distribution of object types in raw data. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Comparison of the type distribution across scored [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (7 more)
Figure 2
Figure 2. Figure 2: Recording locations for the L4 Motion Forecasting [PITH_FULL_IMAGE:figures/full_fig_p003_2.png]
Figure 5
Figure 5. Figure 5: Exact distributions in dataset splits for a) regions and [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: Trajectory length (m) for scored and focal tracks with [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]
Figure 10
Figure 10. Figure 10: Number of scored and focal tracks per scenario. [PITH_FULL_IMAGE:figures/full_fig_p005_10.png]
Figure 11
Figure 11. Figure 11: Comparison of the environments in our dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_11.png]
Figure 13
Figure 13. Figure 13: Comparison between models trained on L4 and AV2 [PITH_FULL_IMAGE:figures/full_fig_p007_13.png]
Figure 12
Figure 12. Figure 12: Comparison between models trained on different L4 [PITH_FULL_IMAGE:figures/full_fig_p007_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

  1. [1]

    Argoverse 2: Next generation datasets for self-driving perception and forecasting,

    B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays, “Argoverse 2: Next generation datasets for self-driving perception and forecasting,” in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks) , 2021

    work page 2021

  2. [2]

    Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset,

    S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y . Chai, B. Sapp, C. R. Qi, Y . Zhou, Z. Yang, A. Chouard, P. Sun, J. Ngiam, V . Vasudevan, A. McCauley, J. Shlens, and D. Anguelov, “Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset,” in Proceedings of the IEEE/CVF International Conference on Co...

    2021

  3. [3]

    One thousand and one hours: Self-driving motion prediction dataset,

    J. L. Houston, G. C. A. Zuidhof, L. Bergamini, Y . Ye, A. Jain, S. Omari, V . I. Iglovikov, and P. Ondruska, “One thousand and one hours: Self-driving motion prediction dataset,” in Conference on Robot Learning, 2020

    work page 2020

  4. [4]

    Argoverse: 3D Tracking and Forecasting With Rich Maps,

    M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays, “Argoverse: 3D Tracking and Forecasting With Rich Maps,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2019, pp. 8740–8749

    work page 2019

  5. [5]

    Shifts: A dataset of real distributional shift across multiple large-scale tasks,

    A. Malinin, N. Band, Y . Gal, M. Gales, A. Ganshin, G. Chesnokov, A. Noskov, A. Ploskonosov, L. Prokhorenkova, I. Provilkov, V . Raina, V . Raina, D. Roginskiy, M. Shmatova, P. Tigas, and B. Yangel, “Shifts: A dataset of real distributional shift across multiple large-scale tasks,” in Proceedings of the Neural Information Processing Systems Track on Datas...

    work page 2021

  6. [6]

    Query-Centric Trajectory Prediction,

    Z. Zhou, J. Wang, Y . Li, and Y . Huang, “Query-Centric Trajectory Prediction,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023, pp. 17 863–17 873

    work page 2023

  7. [7]

    Argoverse 2: Motion Forecasting Competition,

    “Argoverse 2: Motion Forecasting Competition,” https://eval.ai/web/challenges/challenge-page/1719/overview, accessed on 28.04.2025

    work page 2025

  8. [8]

    Vision meets robotics: The KITTI dataset,

    A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” The International Journal of Robotics Research , vol. 32, no. 11, p. 1231–1237, 2013

    work page 2013

Show all 20 references
  1. [9]

    Planet dump retrieved from https://planet.osm.org ,

    OpenStreetMap contributors, “Planet dump retrieved from https://planet.osm.org ,” https://www.openstreetmap.org , 2025

    2025

  2. [10]

    Womd-lidar: Raw sensor dataset benchmark for motion forecasting,

    K. Chen, R. Ge, H. Qiu, R. Ai-Rfou, C. R. Qi, X. Zhou, Z. Yang, S. Et- tinger, P. Sun, Z. Leng, M. Mustafa, I. Bogun, W. Wang, M. Tan, and D. Anguelov, “Womd-lidar: Raw sensor dataset benchmark for motion forecasting,” in Proceedings of the IEEE International Conference on Rob...

    2024

  3. [11]

    Vectornet: Encoding hd maps and agent dynamics from vectorized representation,

    J. Gao, C. Sun, H. Zhao, Y . Shen, D. Anguelov, C. Li, and C. Schmid, “Vectornet: Encoding hd maps and agent dynamics from vectorized representation,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 11 522–11 530, 2020

    2020

  4. [12]

    Learning lane graph representations for motion forecasting,

    M. Liang, B. Yang, R. Hu, Y . Chen, R. Liao, S. Feng, and R. Urta- sun, “Learning lane graph representations for motion forecasting,” in ECCV, 2020

    2020

  5. [13]

    TNT: Target- driven trajectory prediction,

    H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y . Shen, Y . Shen, Y . Chai, C. Schmid, C. Li, and D. Anguelov, “TNT: Target- driven trajectory prediction,” in Conference on Robot Learning , 2020

    2020

  6. [14]

    Towards Consistent and Explainable Motion Prediction using Heterogeneous Graph Attention,

    T. Demmler, A. Tamke, T. Dang, K. Haug, and L. Mikelsons, “Towards Consistent and Explainable Motion Prediction using Heterogeneous Graph Attention,” in Proceedings of the 2024 IEEE Intelligent V ehicles Symposium (IV) , 2024, pp. 168–175

    2024

  7. [15]

    Motion Transformer with Global Intention Localization and Local Movement Refinement,

    S. Shi, L. Jiang, D. Dai, and B. Schiele, “Motion Transformer with Global Intention Localization and Local Movement Refinement,” in Advances in Neural Information Processing Systems , vol. 35, 2022, pp. 6531–6543

    2022

  8. [16]

    MTR-A: 1st place solution for 2022 waymo open dataset challenge–motion prediction,

    ——, “MTR-A: 1st place solution for 2022 waymo open dataset challenge–motion prediction,” arXiv preprint arXiv:2209.10033, 2022

    2022 arXiv

  9. [17]

    Mtr v3: 1st place solution for 2024 waymo open dataset challenge - motion prediction

    C. Shi, S. Shi, L. Jiang, Kong, and D. Global, “Mtr v3: 1st place solution for 2024 waymo open dataset challenge - motion prediction.” [Online]. Available: https://api.semanticscholar.org/CorpusID:270523399

    2024

  10. [18]

    Dynamic intent queries for motion transformer-based trajectory prediction,

    T. Demmler, L. Hartung, A. Tamke, T. Dang, A. Hegai, K. Haug, and L. Mikelsons, “Dynamic intent queries for motion transformer-based trajectory prediction,” in Proceedings of the 2025 IEEE Intelligent V ehicles Symposium (IV), 2025

    2025

  11. [19]

    QCNet: An Elegant, Performant, And Scalable Framework For Marginal/Joint Multi-Agent Trajectory Prediction,

    Z. Zhou, “QCNet: An Elegant, Performant, And Scalable Framework For Marginal/Joint Multi-Agent Trajectory Prediction,” https://github.com/ZikangZhou/QCNet, accessed on 28.04.2025

    2025

  12. [20]

    Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting,

    B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays, “Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting,” arXiv:2301.00493, 2023

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.