{"id":"c8ee573d-7d3d-4be5-8691-46c32c6f744d","arxiv_id":"2506.23433","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A risk-based filter classifies first- and second-order high-risk interactions in the Waymo dataset, yielding situations complementary to the Kalman difficulty and TTP baselines.","lead":"Applying a probabilistic collision-risk model to the Waymo Open Motion Dataset, this paper flags road-user pairs and triples whose predicted overlap exceeds a threshold, and releases the risk scores online. It is a candidate tool for curating automated-driving test data, but because the paper defines 'valuable' as high risk, its core evaluation is partly circular.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2 and Section II.C specify different pairs for Rsecond, so the definition of a second-order situation is ambiguous and the core claim is not yet reproducible.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the equivalence between 'valuable' and the risk-model threshold is a substantial weakness. However, the most load-bearing concern I find is the internal inconsistency between Algorithm 2 and Section II.C over which pair defines Rsecond. This ambiguity directly affects the reproducibility and meaning of the second-order situations, which are a core contribution. The reader's rationale did mention this inconsistency, but it was not identified as the weakest assumption. A single check on the released code would settle it, so the appropriate verdict remains CONDITIONAL pending that verification.","tokens_in":7789,"tokens_out":6581,"duration_ms":63130,"concrete_test":"Download the released repository at https://github.com/HRI-EU/RiskBasedFiltering and inspect the code that generates second-order situations. Determine whether Rsecond is computed for {Vfirst, Vsecond} (Algorithm 2) or {Vego, Vsecond} (Section II.C). Alternatively, sample a set of released second-order scenario IDs and check the interaction graphs: for a chain, the high-risk edge connects Vfirst and Vsecond; for a hub, the high-risk edges connect Vego to both. If the code follows Algorithm 2, the inconsistency is a prose typo and the second-order claim stands; if it follows the prose, the second-order situations are mislabeled and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is the retrieval of second-order situations, defined as risk propagating through an intermediary vehicle. However, Algorithm 2 and Section II.C give contradictory definitions of this retrieval. Algorithm 2 (lines 3-4) computes Rfirst for {Vego, Vfirst} and Rsecond for {Vfirst, Vsecond}; Section II.C states 'we compute the risk Rfirst for the first vehicle pair {Vego, Vfirst}, and the risk Rsecond for the second vehicle pair {Vego, Vsecond}.' These are different pairs. The Introduction's example (front red car affects the intermediate red car, which then affects the ego) supports the chain interpretation in Algorithm 2, so the prose is likely a typo. But the paper does not state which definition was implemented, and the GitHub code is not inspected. If the implementation followed the prose, the retrieved 'second-order' situations are not chains through an intermediary but hubs where the ego has two independent high-risk links; the propagation claim and the confusion-matrix results for second-order situations would then be misleading. This ambiguity is load-bearing because second-order situations are a main contribution, and it is verifiable from the released data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a risk-based filtering method for the Waymo Open Motion Dataset that labels driving situations as valuable when a probabilistic collision-risk model exceeds a threshold. It distinguishes first-order situations (ego paired with one other road user) from second-order situations (risk propagating through an intermediary vehicle), and compares the retrieved sets with two baselines, Kalman difficulty and Tracks-To-Predict (TTP). The paper reports that the risk-based filter retrieves millions of situations, that most are complementary to the baselines (off-diagonal entries of 1-4% in confusion matrices), and that the risk data and situation IDs are released publicly.","tokens_in":8033,"tokens_out":3279,"duration_ms":35452,"significance":"If the central claims are substantiated, the contribution is practically useful: a scalable, unified filter for extracting high-risk interaction scenarios from large naturalistic driving datasets would support trajectory-prediction training and automated-vehicle testing. The paper ships an explicit algorithmic description, applies the method to the full Waymo dataset (67.8 million road users), and makes the risk values and filtered situation IDs publicly available, which is a concrete reproducibility asset. The comparison with Kalman difficulty and TTP is informative and honestly acknowledges that TTP cannot be recomputed. However, the significance is currently limited by the fact that 'valuable' is defined by the model itself, and by a definitional ambiguity in the second-order retrieval, so the empirical results support complementarity but not the value claim.","major_comments":[{"comment":"The definition of a second-order situation is internally contradictory and load-bearing. Algorithm 2 (lines 3-4) computes Rfirst for {Vego, Vfirst} and Rsecond for {Vfirst, Vsecond}, i.e., a chain through the intermediary vehicle. Section II.C, however, states that 'we compute the risk Rfirst for the first vehicle pair {Vego, Vfirst}, and the risk Rsecond for the second vehicle pair {Vego, Vsecond}', i.e., two independent links from the ego vehicle. The Introduction's example (front red car affects the intermediate red car, which then affects the ego) supports the chain interpretation, but the manuscript does not state which definition was actually implemented. Since second-order situations are a central contribution and the released code is not inspected in the paper, this ambiguity must be resolved before the second-order results can be interpreted.","section":"Section II.C and Algorithm 2"},{"comment":"The claim that the filter 'selects valuable driving situations' is circular as presented. Section II defines a situation as valuable 'when road users are likely to encounter high collision risks', and the only operationalization is the authors' own risk model with the threshold Rvaluable in Table I. The conclusion that the retrieved situations are valuable is therefore true by construction and is not supported by any external benchmark, such as expert annotations, near-crash events, or downstream prediction performance. The confusion-matrix analysis in Section IV.A legitimately shows that the risk model selects a different set of situations than Kalman difficulty or TTP, but it does not show that either set is 'better' or 'more valuable'. Please either reframe the contribution as 'high-risk situations according to the model' or add an external validation of value.","section":"Section II and Section IV.A"},{"comment":"The quantitative results are not shown to be robust to the chosen thresholds and post-hoc exclusions. The threshold Rvaluable = 10^-9 is presented in Table I without a sensitivity analysis, and Section III.C introduces ad hoc exclusions of vehicles with near-zero speeds or short paths and a Gaussian-mixture refinement. The reported counts (e.g., 4.4 million first-order and 3.3 million second-order situations in Fig. 4, and the 1.5 million situations in Section IV.A) can be expected to change substantially with these choices. A sensitivity study over Rvaluable and a clear statement of the exclusion criteria, or a justification of these values from prior data, are needed before the quantitative claims can be considered stable.","section":"Section III.B and Section III.C"}],"minor_comments":[{"comment":"The abstract states that the risk data is 'made open-source', while Section V says the risk values and situation IDs are 'available upon request' and gives a GitHub link. Please clarify whether the data is directly downloadable or must be requested, and provide a direct stable link to the repository.","section":"Abstract and Section V"},{"comment":"There are several typographical errors, including 'seperates', 'aditionally', 'biycles', 'defintions', and 'sitations'. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The sentence 'Driving data can repeat within the Waymo dataset, and the same vehicle may be flagged as valuable by TTP in one instance but not in another' is vague; please specify whether the comparison unit is vehicle-track instances and whether deduplication was attempted.","section":"Section IV.A"}],"recommendation":"major_revision","confidential_remarks":"The second-order definitional mismatch is the most urgent issue: before acceptance, the authors should verify against their released code which variant was implemented. If the implementation followed the prose rather than Algorithm 2, the 'risk propagation through an intermediary' claim in the abstract and introduction should be withdrawn or the experiments rerun. The circularity of 'valuable' is also likely to attract criticism from reviewers; a title and abstract that say 'high-risk situations' instead of 'valuable driving situations' would be more defensible unless external validation is added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a straightforward application of a known risk model to the Waymo Open Motion Dataset, plus a new categorization scheme and an open release of the risk scores. The dataset artifact is real and potentially useful. But the central claim—that the filter selects 'valuable' driving situations—is tautological as stated, and there's a load-bearing ambiguity about what a second-order situation is.\n\nWhat's actually new: the risk values and first/second-order labels for the full Waymo dataset, released openly. That's a useful resource for the AV testing community. The paper is honest that the risk model itself is prior work [4], and the pipeline (uncertainty modeling, overlap probability, survival-function integration, thresholding) is described clearly. The comparison with Kalman difficulty and TTP shows the sets are largely disjoint (off-diagonal 1-4%), which is a useful observation about how different interaction metrics pick different things.\n\nThe soft spots are significant. The definition of 'valuable' is circular: Section II defines a valuable situation as one where road users likely encounter high collision risks, and that risk is computed by the same model with an arbitrary threshold (Rvaluable = 1e-9). So the conclusion that the filter selects valuable situations is tautological. The baselines show different sets, not better ones; there's no external benchmark or downstream evidence (e.g., showing that a predictor trained on these situations improves). The post-hoc exclusion of stationary and short-path vehicles (Section III.C) is disclosed, but it could bias the retrieved set.\n\nThe bigger problem is the mismatch between Algorithm 2 and Section II.C. Algorithm 2 computes Rfirst for {Vego, Vfirst} and Rsecond for {Vfirst, Vsecond}—a chain through an intermediate vehicle. The prose says Rsecond is for {Vego, Vsecond}—the ego and the second vehicle directly. The Introduction and Figure 1 support the chain interpretation, so the prose is likely a typo, but the paper never says which was implemented, and the released code wasn't inspected. Since second-order situations are a main contribution, this ambiguity undermines reproducibility.\n\nThe paper deserves a serious referee, but only after the authors resolve the second-order definition and preferably add some external validation of 'value.' The open risk data is worth having regardless.\n\nRecommendation: send to peer review with major revision requested; the core claim is not yet proven.","headline":"A useful open dataset artifact, but the 'valuable' claim is tautological and the second-order definition is ambiguous; needs major revision.","tokens_in":8515,"tokens_out":4216,"would_cite":true,"duration_ms":37172,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A probabilistic collision-risk filter, applied to every road-user pair and triple in the Waymo Open Motion Dataset, retrieves millions of driving situations that existing metrics ignore, and the paper argues this yields a complementary…","keywords":["risk-based filtering","driving situation retrieval","Waymo Open Motion Dataset","first-order situations","second-order situations","collision risk model","Kalman difficulty","Tracks-To-Predict"],"falsifier":"Take a random sample of situations from the Waymo dataset, annotate whether a near-collision or strong evasive maneuver actually occurs, and compare the precision of the risk-filtered set ($R\\geq10^{-9}$) with the precision of a random sample and of the TTP-flagged set; if the risk-filtered set is not substantially richer in such events, the central claim that the filter selects valuable situations is falsified.","tokens_in":7612,"feed_emoji":"🚗","tokens_out":5918,"duration_ms":61705,"temperature":0.7,"pith_summary":"The paper tries to establish that large driving datasets can be screened for valuable test scenarios by a probabilistic collision-risk model, instead of handcrafted rules. It defines a situation as valuable when road users are likely to encounter high collision risk, then separates first-order situations (a pair of road users in direct risk) from second-order situations (risk propagating through an intermediary road user). Applied to the Waymo dataset, this filter retrieves millions of situations whose overlap with Kalman difficulty and TTP is only 1–4%. A sympathetic reader would care because the approach is rule-free, portable to other datasets, and produces a large open set of retrieved situations for automated-vehicle testing.","feed_headline":"Risk filter finds 4.4M driving situations other metrics miss","feed_subtitle":"A probability-based risk model retrieves first- and second-order interactions that Kalman difficulty and TTP don't flag.","key_machinery":"The load-bearing mechanism is the probabilistic risk model of the paper's reference [4]: each road user's future position is a Gaussian with uncertainty that grows over time, collision probability between two road users is the integral of the product of their Gaussians, and a survival function $S(s;t)=\\exp\\{-\\int_t^{t+s}(\\tau_0^{-1}+P_{\\mathrm{coll}}(s;t)/\\Delta t)\\,d\\tau\\}$ weights the collision probability over the prediction horizon. The integrated, weighted probability gives the risk $R_i(t)$, and the threshold $R_{\\mathrm{valuable}}=10^{-9}$ decides what counts as valuable. Around this risk computation the paper builds an interaction graph whose nodes are road users and whose directed edges carry risk values; Algorithm 1 retrieves first-order situations by thresholding all pairs, and Algorithm 2 retrieves second-order situations by thresholding all triples. A Gaussian-mixture refinement for curved paths and the exclusion of stationary or short-path vehicles are auxiliary adjustments meant to keep the retrieved set meaningful.","core_discovery":"The central claim is that a survival-function-based collision risk measure, computed from Gaussian uncertainty predictions of all road-user pairs, can serve as a unified filter for valuable driving situations in the Waymo Open Motion Dataset. A first-order situation is a pair whose predicted risk exceeds $R_{\\mathrm{valuable}}=10^{-9}$, and a second-order situation is a triple in which two constituent pairs both exceed this threshold. The paper reports about 4.4 million valuable first-order and 3.3 million valuable second-order situations, with car-car and car-car-car interactions dominating. Comparison with Kalman difficulty and TTP shows off-diagonal agreement of only 1–4%, meaning the risk model flags largely different situations, which the authors interpret as evidence that it extracts complex, complementary interactions that enrich automated-vehicle testing. In the authors' framing, risk-based filtering recovers a distinct, large set of high-interaction scenarios that existing prediction-error-based and rule-based metrics do not surface.","pith_inferences":["The threshold $R_{\\mathrm{valuable}}=10^{-9}$ is chosen without a sensitivity analysis; a natural next step is to vary this threshold and check whether the 1–4% complementarity with Kalman difficulty and TTP is stable or an artifact of one cutoff.","The post-hoc exclusion of stationary and short-path vehicles could remove legitimate low-speed interactions; retaining them and labeling them separately would show whether the filtering loses valuable cases.","The risk-graph representation naturally extends beyond second-order chains: instead of stopping at triples, one could follow directed edges with $R\\geq R_{\\mathrm{valuable}}$ to retrieve chains of arbitrary length from the same graph.","The paper mentions interactivity scores based on conditional behavior prediction but does not compare with them; such a comparison would reveal whether risk-based filtering captures a genuinely different phenomenon from behavior-change-based interaction metrics."],"forward_implications":["The retrieved valuable situations are largely complementary to Kalman difficulty and TTP, so combining all three filters should yield a wider and more diverse set of test scenarios for automated-vehicle software.","Because the filter uses generic risk computation rather than dataset-specific rules, it can be transferred to other motion datasets, which the paper names as future work.","The first-order/second-order classification allows targeted extraction of both direct conflicts and chain interactions, such as a lane-change maneuver by one car forcing a following car to act on the ego vehicle.","The open-source release of the computed risk values and situation IDs lets other researchers reproduce the filtered set and use it as a common testbed."],"supporting_citations":[{"why":"Supplies the probabilistic risk model (Gaussian uncertainty overlap and survival-function integration) on which the entire filtering pipeline is built.","marker":"[4]"},{"why":"Provides the Waymo Open Motion Dataset and the TTP field used as a baseline; all statistical results are computed over this data.","marker":"[1]"},{"why":"Defines Kalman difficulty (minFDE between a linear prediction and ground truth), one of the two baselines the paper compares against.","marker":"[5]"}],"fun_headline_variants":["Risk filter surfaces 4.4M driving situations others miss","Probabilistic risk model flags unseen high-interaction scenarios","Beyond Kalman and TTP: risk filter finds 4.4M scenarios","Risk-based filtering enriches automated vehicle testing data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes that 'valuable' can be equated with high predicted collision risk at the chosen threshold $R_{\\mathrm{valuable}}=10^{-9}$; if that equivalence is wrong, the filter just selects a different arbitrary subset rather than valuable situations.","fun_headline_variants_meta":{"raw":{"variants":["Risk filter surfaces 4.4M driving situations others miss","Probabilistic risk model flags unseen high-interaction scenarios","Beyond Kalman and TTP: risk filter finds 4.4M scenarios","Risk-based filtering enriches automated vehicle testing data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1226,"prompt_tokens":887,"completion_tokens":339,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":268}},"tokens_in":503,"tokens_out":339,"duration_ms":4042,"temperature":1.0,"reasoning_tokens":268,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:42:35.745441+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of situations from the Waymo dataset, annotate whether a near-collision or strong evasive maneuver actually occurs, and compare the precision of the risk-filtered set ($R\\geq10^{-9}$) with the precision of a random sample and of the TTP-flagged set; if the risk-filtered set is not substantially richer in such events, the central claim that the filter selects valuable situations is falsified.","supporting_citations":[{"cited_title":"Probabilistic Uncertainty-Aware Risk Spot Detector for Naturalistic Driving,","cited_arxiv_id":null,"evidence_quote":"Supplies the probabilistic risk model (Gaussian uncertainty overlap and survival-function integration) on which the entire filtering pipeline is built."},{"cited_title":"Large Scale Interactive Motion Fore- casting for Autonomous Driving: The Waymo Open Motion Dataset,","cited_arxiv_id":null,"evidence_quote":"Provides the Waymo Open Motion Dataset and the TTP field used as a baseline; all statistical results are computed over this data."},{"cited_title":"UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction,","cited_arxiv_id":null,"evidence_quote":"Defines Kalman difficulty (minFDE between a linear prediction and ground truth), one of the two baselines the paper compares against."}],"review_version":1}