REVIEW 3 major objections 3 minor 12 references
Risk-Based Filtering of Valuable Driving Situations in the Waymo Open Motion Dataset
T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A probabilistic collision-risk filter, applied to every road-user pair and triple in the Waymo Open Motion Dataset, retrieves millions of driving situations that existing metrics ignore, and the paper argues this yields a complementary…
desk verdict A useful open dataset artifact, but the 'valuable' claim is tautological and the second-order definition is ambiguous; needs major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the probabilistic risk model of the paper's reference [4]: each road user's future position is a Gaussian with uncertainty that grows over time, collision probability between two road users is the integral of the product of their Gaussians, and a survival function $S(s;t)=\exp\{-\int_t^{t+s}(\tau_0^{-1}+P_{\mathrm{coll}}(s;t)/\Delta t)\,d\tau\}$ weights the collision probability over the prediction horizon. The integrated, weighted probability gives the risk $R_i(t)$, and the threshold $R_{\mathrm{valuable}}=10^{-9}$ decides what counts as valuable. Around this risk computation the paper builds an interaction graph whose nodes are road users and whose directed edges carry risk values; Algorithm 1 retrieves first-order situations by thresholding all pairs, and Algorithm 2 retrieves second-order situations by thresholding all triples. A Gaussian-mixture refinement for curved paths and the exclusion of stationary or short-path vehicles are auxiliary adjustments meant to keep the retrieved set meaningful.
What would settle it
Take a random sample of situations from the Waymo dataset, annotate whether a near-collision or strong evasive maneuver actually occurs, and compare the precision of the risk-filtered set ($R\geq10^{-9}$) with the precision of a random sample and of the TTP-flagged set; if the risk-filtered set is not substantially richer in such events, the central claim that the filter selects valuable situations is falsified.
Extended reading notes
Core claim
The central claim is that a survival-function-based collision risk measure, computed from Gaussian uncertainty predictions of all road-user pairs, can serve as a unified filter for valuable driving situations in the Waymo Open Motion Dataset. A first-order situation is a pair whose predicted risk exceeds $R_{\mathrm{valuable}}=10^{-9}$, and a second-order situation is a triple in which two constituent pairs both exceed this threshold. The paper reports about 4.4 million valuable first-order and 3.3 million valuable second-order situations, with car-car and car-car-car interactions dominating. Comparison with Kalman difficulty and TTP shows off-diagonal agreement of only 1–4%, meaning the risk model flags largely different situations, which the authors interpret as evidence that it extracts complex, complementary interactions that enrich automated-vehicle testing. In the authors' framing, risk-based filtering recovers a distinct, large set of high-interaction scenarios that existing prediction-error-based and rule-based metrics do not surface.
Load-bearing premise
The whole method assumes that 'valuable' can be equated with high predicted collision risk at the chosen threshold $R_{\mathrm{valuable}}=10^{-9}$; if that equivalence is wrong, the filter just selects a different arbitrary subset rather than valuable situations.
Editorial extensions
If this is right
- The retrieved valuable situations are largely complementary to Kalman difficulty and TTP, so combining all three filters should yield a wider and more diverse set of test scenarios for automated-vehicle software.
- Because the filter uses generic risk computation rather than dataset-specific rules, it can be transferred to other motion datasets, which the paper names as future work.
- The first-order/second-order classification allows targeted extraction of both direct conflicts and chain interactions, such as a lane-change maneuver by one car forcing a following car to act on the ego vehicle.
- The open-source release of the computed risk values and situation IDs lets other researchers reproduce the filtered set and use it as a common testbed.
Reading between the lines
- The threshold $R_{\mathrm{valuable}}=10^{-9}$ is chosen without a sensitivity analysis; a natural next step is to vary this threshold and check whether the 1–4% complementarity with Kalman difficulty and TTP is stable or an artifact of one cutoff.
- The post-hoc exclusion of stationary and short-path vehicles could remove legitimate low-speed interactions; retaining them and labeling them separately would show whether the filtering loses valuable cases.
- The risk-graph representation naturally extends beyond second-order chains: instead of stopping at triples, one could follow directed edges with $R\geq R_{\mathrm{valuable}}$ to retrieve chains of arbitrary length from the same graph.
- The paper mentions interactivity scores based on conditional behavior prediction but does not compare with them; such a comparison would reveal whether risk-based filtering captures a genuinely different phenomenon from behavior-change-based interaction metrics.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a risk-based filtering method for the Waymo Open Motion Dataset that labels driving situations as valuable when a probabilistic collision-risk model exceeds a threshold. It distinguishes first-order situations (ego paired with one other road user) from second-order situations (risk propagating through an intermediary vehicle), and compares the retrieved sets with two baselines, Kalman difficulty and Tracks-To-Predict (TTP). The paper reports that the risk-based filter retrieves millions of situations, that most are complementary to the baselines (off-diagonal entries of 1-4% in confusion matrices), and that the risk data and situation IDs are released publicly.
Significance. If the central claims are substantiated, the contribution is practically useful: a scalable, unified filter for extracting high-risk interaction scenarios from large naturalistic driving datasets would support trajectory-prediction training and automated-vehicle testing. The paper ships an explicit algorithmic description, applies the method to the full Waymo dataset (67.8 million road users), and makes the risk values and filtered situation IDs publicly available, which is a concrete reproducibility asset. The comparison with Kalman difficulty and TTP is informative and honestly acknowledges that TTP cannot be recomputed. However, the significance is currently limited by the fact that 'valuable' is defined by the model itself, and by a definitional ambiguity in the second-order retrieval, so the empirical results support complementarity but not the value claim.
major comments (3)
- [Section II.C and Algorithm 2] The definition of a second-order situation is internally contradictory and load-bearing. Algorithm 2 (lines 3-4) computes Rfirst for {Vego, Vfirst} and Rsecond for {Vfirst, Vsecond}, i.e., a chain through the intermediary vehicle. Section II.C, however, states that 'we compute the risk Rfirst for the first vehicle pair {Vego, Vfirst}, and the risk Rsecond for the second vehicle pair {Vego, Vsecond}', i.e., two independent links from the ego vehicle. The Introduction's example (front red car affects the intermediate red car, which then affects the ego) supports the chain interpretation, but the manuscript does not state which definition was actually implemented. Since second-order situations are a central contribution and the released code is not inspected in the paper, this ambiguity must be resolved before the second-order results can be interpreted.
- [Section II and Section IV.A] The claim that the filter 'selects valuable driving situations' is circular as presented. Section II defines a situation as valuable 'when road users are likely to encounter high collision risks', and the only operationalization is the authors' own risk model with the threshold Rvaluable in Table I. The conclusion that the retrieved situations are valuable is therefore true by construction and is not supported by any external benchmark, such as expert annotations, near-crash events, or downstream prediction performance. The confusion-matrix analysis in Section IV.A legitimately shows that the risk model selects a different set of situations than Kalman difficulty or TTP, but it does not show that either set is 'better' or 'more valuable'. Please either reframe the contribution as 'high-risk situations according to the model' or add an external validation of value.
- [Section III.B and Section III.C] The quantitative results are not shown to be robust to the chosen thresholds and post-hoc exclusions. The threshold Rvaluable = 10^-9 is presented in Table I without a sensitivity analysis, and Section III.C introduces ad hoc exclusions of vehicles with near-zero speeds or short paths and a Gaussian-mixture refinement. The reported counts (e.g., 4.4 million first-order and 3.3 million second-order situations in Fig. 4, and the 1.5 million situations in Section IV.A) can be expected to change substantially with these choices. A sensitivity study over Rvaluable and a clear statement of the exclusion criteria, or a justification of these values from prior data, are needed before the quantitative claims can be considered stable.
minor comments (3)
- [Abstract and Section V] The abstract states that the risk data is 'made open-source', while Section V says the risk values and situation IDs are 'available upon request' and gives a GitHub link. Please clarify whether the data is directly downloadable or must be requested, and provide a direct stable link to the repository.
- [Throughout] There are several typographical errors, including 'seperates', 'aditionally', 'biycles', 'defintions', and 'sitations'. A careful proofreading pass is needed.
- [Section IV.A] The sentence 'Driving data can repeat within the Waymo dataset, and the same vehicle may be flagged as valuable by TTP in one instance but not in another' is vague; please specify whether the comparison unit is vehicle-track instances and whether deduplication was attempted.
Circularity Check
The 'valuable' label is defined by the same risk threshold used to filter, so the headline selection claim is tautological; the baseline comparisons remain independent empirical content.
-
self definitional
[Section II (opening paragraph); Section III.B, Table I; Section IV.C]
"We define a situation as valuable when road users are likely to encounter high collision risks ... Finally, the threshold of the risk value for detecting valuable driving situations with the risk-based filtering approach is set to Rvaluable = 10−9 ... In valuable first-order driving situations, the risk between pairs of road users exceeds Rvaluable = 10−9 as the threshold."
The paper defines 'valuable driving situation' as one with high collision risk, and then uses the same risk model's threshold Rvaluable to retrieve situations (Algorithms 1 and 2 retrieve exactly the pairs/triples whose risk exceeds Rvaluable). The abstract and Section IV conclude that 'our approach effectively selects valuable driving situations.' That conclusion is true by construction: the set of 'valuable' situations is identical to the set selected by the threshold on the model that defines 'valuable.' No external or independent benchmark of value is used; the baseline comparisons only establish that different metrics select different subsets, not that the risk-selected subset is more valuable for automated vehicle testing.
full rationale
The only load-bearing circularity is in the semantics of 'valuable.' The paper equates value with high collision risk as computed by its own risk model, sets a threshold for 'valuable situations,' and then reports that the filter selects such situations. This is a tautological restatement rather than an empirically derived result. However, there is independent, non-circular content in the paper: the actual application to the Waymo dataset, the classification into first- and second-order situations, and the confusion-matrix comparisons showing complementarity with Kalman difficulty and TTP (off-diagonal entries 1–4%). Those claims are self-contained against the dataset and do not reduce to the definition. The risk model from [4] involves overlapping authorship, but it is used as a published parameterized model applied to new data, not as a self-citation invoked to forbid alternatives, so that self-citation is not load-bearing circularity. The discrepancy between Algorithm 2 (Rsecond for {Vfirst, Vsecond}) and Section II.C (Rsecond for {Vego, Vsecond}) is a reproducibility/correctness issue, not a circularity argument, and is therefore not counted in the score. Overall, the central 'valuable' claim reduces to the definition by construction, but the paper's more specific empirical findings remain meaningful; score 6 reflects this partial circularity.
Assumptions & free parameters
free parameters (5)
- Rvaluable threshold =
10^-9
- sigma_car_max and sigma_cyc_max and sigma_ped_max =
15 m, 3.3 m, 1.5 m
- tau_0_inv =
0.56 1/s
- smax =
8 s
- dt =
0.25 s
assumptions (4)
- domain assumption Valuable driving situation = high collision risk under the model.
- domain assumption The probabilistic risk model of Puphal et al. [4] correctly estimates future collision risk for all road users in Waymo.
- domain assumption Constant velocity prediction along the driving path is a sufficient motion model for risk estimation.
- standard math Gaussian overlap integral computes collision probability correctly.
Cite this review
Pith. "Pith review of Risk-Based Filtering of Valuable Driving Situations in the Waymo Open Motion Dataset." pith.science (2026). https://pith.science/paper/HJQ4ACFM
@misc{pith2026250623433,
author = {Pith},
title = {Pith review of: Risk-Based Filtering of Valuable Driving Situations in the Waymo Open Motion Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/HJQ4ACFM}},
note = {Machine review of arXiv:2506.23433}
}
read the original abstract
Improving automated vehicle software requires driving data rich in valuable road user interactions. In this paper, we propose a risk-based filtering approach that helps identify such valuable driving situations from large datasets. Specifically, we use a probabilistic risk model to detect high-risk situations. Our method stands out by considering a) first-order situations (where one vehicle directly influences another and induces risk) and b) second-order situations (where influence propagates through an intermediary vehicle). In experiments, we show that our approach effectively selects valuable driving situations in the Waymo Open Motion Dataset. Compared to the two baseline interaction metrics of Kalman difficulty and Tracks-To-Predict (TTP), our filtering approach identifies complex and complementary situations, enriching the quality in automated vehicle testing. The risk data is made open-source: https://github.com/HRI-EU/RiskBasedFiltering.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[4]
Probabilistic Uncertainty-Aware Risk Spot Detector for Naturalistic Driving,
T. Puphal, M. Probst, and J. Eggert, “Probabilistic Uncertainty-Aware Risk Spot Detector for Naturalistic Driving,” IEEE Transactions on Intelligent Vehicles (T-IV), 2019
work page 2019
-
[1]
Large Scale Interactive Motion Fore- casting for Autonomous Driving: The Waymo Open Motion Dataset,
S. Ettinger, S. Cheng, and et al., “Large Scale Interactive Motion Fore- casting for Autonomous Driving: The Waymo Open Motion Dataset,” in IEEE/CVF International Conference on Computer Vision (CVPR), 2021
work page 2021
-
[2]
nuScenes: A Multimodal Dataset for Autonomous Driving,
H. Caesar, V . Bankiti, A. H. Lang, and et al., “nuScenes: A Multimodal Dataset for Autonomous Driving,” in IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , 2020
work page 2020
-
[3]
Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting,
B. Wilson, W. Qi, T. Agarwal, and et al., “Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting,” in Conference on Neural Information Processing Systems (NeurIPS) , 2021
work page 2021
-
[5]
UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction,
L. Feng, M. Bahari, K. M. B. Amor, and et al., “UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction,” in European Conference on Computer Vision (ECCV) , 2024
work page 2024
-
[6]
Identifying Driver Interactions via Conditional Behavior Prediction,
E. Tolstaya, R. Mahjourian, C. Downey, and et al., “Identifying Driver Interactions via Conditional Behavior Prediction,” in IEEE International Conference on Robotics and Automation (ICRA) , 2021
work page 2021
-
[7]
Analyzing Multiagent Interactions in Traffic Scenes via Topological Braids,
C. Mavrogiannis, J. DeCastro, and S. Srinivasa, “Analyzing Multiagent Interactions in Traffic Scenes via Topological Braids,” in IEEE Interna- tional Conference on Robotics and Automation (ICRA) , 2022
work page 2022
-
[8]
S. Klingelschmitt and et al., “Probabilistic Situation Assessment Frame- work for Multiple, Interacting Traffic Participants in Generic Traffic Scenes,” in IEEE Intelligent Vehicles Symposium (IV) , 2016
work page 2016
Show all 12 references
-
[9]
On a Formal Model of Safe and Scalable Self-driving Cars,
S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a Formal Model of Safe and Scalable Self-driving Cars,” arXiv:1708.06374, 2017
2017 arXiv
-
[10]
Non-Gaussian Risk Bounded Trajectory Optimization for Stochastic Nonlinear Systems in Uncertain Environments,
W. Han, A. Jasour, and B. Williams, “Non-Gaussian Risk Bounded Trajectory Optimization for Stochastic Nonlinear Systems in Uncertain Environments,” in IEEE International Conference on Robotics and Automation (ICRA), 2022
2022
-
[11]
Continuous Risk Measures for ADAS and AD,
J. Eggert and T. Puphal, “Continuous Risk Measures for ADAS and AD,” Future Active Safety Technology towards Zero-Traffic-Accidents (FAST-zero), 2017
2017
-
[12]
INTERACTION Dataset: An INTER- national, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps,
W. Zhan, L. Sun, and et al., “INTERACTION Dataset: An INTER- national, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps,” arXiv:1910.03088, 2019
1910 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.