{"id":"b80d9ce3-557f-4887-b5df-1cdd5070d922","arxiv_id":"2607.07103","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"K-Risk curates 31,398 high-risk driving events from 20 trajectory datasets with multi-layered semantic and LLM-generated annotations validated via closed-loop simulation.","lead":"This paper builds K-Risk, a dataset of 31,398 high-risk driving events curated from 20 existing trajectory datasets, each annotated with structured descriptions and LLM-generated risk analyses. It provides a standardized resource for training and evaluating autonomous driving agents on rare, safety-critical scenarios.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Closed-loop validation on 262 events may reflect prompt overfitting rather than generalizable decision improvement","rationale":"The reader correctly identified that the closed-loop validation may reflect prompt overfitting and that the evaluation sample is small. This is the most load-bearing concern because the paper's abstract and contributions explicitly claim 'verifiable decisions' and 'decision supervision' as distinguishing features versus prior datasets (Table 1). Without held-out evaluation, the verifiability claim is circular: the simulator validates decisions on events that were used to refine the prompts that produced those decisions. The reader also flagged underspecified DRF parameters, which is a legitimate secondary concern — Appendix A gives the mathematical formulation but the actual parameter values (p, t_la, m, c, k_1, k_2) and per-source threshold calibrations are not reported, making full reproducibility assessment difficult. However, the DRF calibration issue affects dataset quality assessment, while the closed-loop evaluation issue directly undermines the paper's strongest differentiating claim. The dataset itself — 31,398 events from 20 sources with synchronized trajectory/metadata/text triples — is a genuine and useful contribution regardless of the validation gap. The code and data are publicly released, which supports reproducibility. The CONDITIONAL verdict is appropriate: the dataset contribution warrants publication, but the claims about verifiable decision supervision should either be softened to match the evidence (descriptive validation on a fixed set) or supported with held-out evaluation. I agree with the reader's assessment on all major points.","tokens_in":24090,"tokens_out":618,"duration_ms":97668,"concrete_test":"Select a held-out set of 100+ extreme-risk events not used in any of the three refinement trials. Run the Trial-3 prompt on these unseen events and report the collision rate. If it exceeds ~3.5% (i.e., closer to Trial 1 than Trial 3), the improvement is overfitting rather than generalizable reasoning.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's claim of 'verifiable decision supervision' rests on the closed-loop validation in Table 2, where collision rate drops from 4.58% to 1.91% across three trials. However, the same 262 extreme-risk scenarios are used across all three trials, with each trial's failures fed back as reflection prompts for the next. This is iterative prompt refinement on the evaluation set itself — there is no held-out test set. The 58.3% relative reduction could simply be the LLM memorizing corrections for specific scenarios rather than learning generalizable safety reasoning. The paper does not report performance on any unseen events, so the claim that K-Risk 'provides verifiable safety signals' for 'developing risk-aware autonomous driving agents' is not substantiated by the evidence presented. The dataset contribution (31,398 curated events with multi-layer annotations) stands on its own merits, but the stronger claim about decision supervision quality is unsupported by the experimental design.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"The manuscript presents K-Risk, a knowledge-augmented dataset of 31,398 high-risk driving scenarios curated from 20 public trajectory datasets across Europe, China, and the United States. Each event is released as a synchronized triple (CSV trajectory, JSON metadata, text annotation) containing structured scenario descriptions, abnormal-behavior notifications, and, for a representative subset, LLM-generated causal risk analyses and action recommendations. A closed-loop validation framework iteratively refines LLM-generated decisions through a collision-free simulator. The dataset addresses a genuine gap: existing driving datasets rarely combine event-level risk labels, semantic annotations, and verifiable safety signals. The curation pipeline (DRF filtering, hard-maneuver detection, TTC/trajectory-conflict prediction) is clearly described, and the release of code and standardized file formats is commendable.","tokens_in":24017,"tokens_out":2157,"duration_ms":101413,"significance":"The dataset contribution is substantial and timely. Aggregating 20 heterogeneous sources into a unified event-level format with multi-layer annotations fills a real need for training and evaluating risk-aware autonomous driving agents. The synchronized triple format (CSV/JSON/text) is well-designed for bridging trajectory-level and language-level supervision. The release of processing code, benchmark protocols, and preference pairs from closed-loop trials adds practical value. The five-action schema and the alignment with standard LLM post-training stages (CPT, SFT, RLHF, RLVR) make the dataset immediately usable for the community.","major_comments":[{"comment":"Table 2 and surrounding text (Technical Validation, 'Closed-loop usability of the extreme subset'): The three-trial closed-loop validation uses the same 262 scenarios across all trials, with each trial's failures fed back as reflection prompts for the next. There is no held-out test set. The reported 58.3% relative reduction in collision rate (4.58% → 1.91%) therefore cannot distinguish generalizable safety improvement from scenario-specific prompt memorization. The abstract's claim of 'verifiable decision supervision' and 'verifiable decisions' rests on this result. The dataset contribution stands independently, but the decision-supervision claim needs either (a) a held-out evaluation on unseen events, or (b) a clear reframing stating that the closed-loop loop produces training data (preference pairs) rather than demonstrating generalizable decision quality. As written, the text implies","section":null},{"comment":"Appendix A: The DRF formulation depends on six parameters (p, t_la, m, c, k_1, k_2) whose numerical values are never specified. The text states 'the DRF threshold is set per source' (Annotation Protocol) and that only the top 10% of frames are retained, but without the parameter values the DRF computation is not reproducible from the mathematical description alone. The code release may contain these values, but the manuscript should either state them or explicitly direct the reader to the specific code file and configuration where they are defined.","section":null},{"comment":"Annotation Protocol, third filter, and Stage 5 of Appendix C: The closed-loop simulator uses constant-acceleration and constant-steering assumptions to roll trajectories forward for two seconds. While this model is cited as well-validated for short-term motion prediction (Refs 34, 35), it is also used to evaluate discrete action recommendations that may include lane changes (actions 2 and 3 in the five-action schema). A lane change executed under constant-steering is physically inconsistent. The manuscript should discuss whether this simplification systematically biases the evaluation of lateral actions, or at minimum state it as a known limitation.","section":null}],"minor_comments":[{"comment":"The manuscript mentions 'domain experts' reviewed LLM-generated annotations (Annotation Protocol, last paragraph) but does not specify how many experts, their qualifications, inter-rater agreement, or the fraction of annotations revised. A brief statement would strengthen the validation.","section":null},{"comment":"Figure 1 contains a placeholder citation mark '?,' in the Background & Summary text ('systems that are at once safer and more interpretable?, 4'). This should be corrected.","section":null},{"comment":"The event distribution across sources is highly uneven (highD: 8,807; FreewayB: 5,304; vs. aggregated AV sources: 2,570). While the text acknowledges this, it would help to state the number of events per AV source individually, since the 14 AV sources collectively contribute fewer events than a single HV source.","section":null},{"comment":"Table 1: The 'Post-training stages' column for K-Risk lists 'CPT + SFT + RLHF + RLVR,' which is aspirational rather than demonstrated. The paper shows SFT data and preference pairs, but CPT and RLVR are only suggested in Usage Notes. Consider clarifying which stages are supported by released data versus proposed as future work.","section":null},{"comment":"Reference 33 (FHWA surrogate safety measures) includes the note 'placeholder pending verification' in the reference list. This should be resolved before publication.","section":null}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a solid contribution that I would support for publication. The main concern is that the closed-loop validation is oversold relative to what the experimental design can support. The same 262 events used for iterative refinement cannot serve as evidence of generalizable decision improvement. The authors could easily address this by either running a held-out evaluation or by carefully reframing the closed-loop results as producing training data rather than demonstrating decision quality. The DRF parameter values are a straightforward omission that should be fixable from the released code. I would not view any of these issues as fundamental; they are addressable within one revision cycle."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"K-Risk is a dataset paper that aggregates 20 trajectory sources into 31,398 high-risk driving events, each released as a synchronized triple: CSV trajectories, JSON metadata, and natural-language annotations. The core dataset contribution is real and useful. The team does the unglamorous work of unifying heterogeneous sources across three continents, applying a consistent risk-extraction pipeline (driver risk field, hard-maneuver thresholds, TTC filtering, trajectory-conflict prediction), and shipping the result in a standardized format with code. That alone has value for the community — safety-critical edge cases are genuinely scarce in existing driving datasets, and a curated, multi-source collection with this level of structure is a legitimate resource. The three-phase event window (lead-in, peak-risk, resolution) is a thoughtful design choice that supports temporal reasoning. The file schema is documented in detail, and the pipeline is reproducible from the released code. Credit where it's due: this is a well-organized data release, not a hand-wavy position paper. The soft spots are real but mostly contained. The DRF model is adopted from SafeDrive (a co-author's prior work), and the parameters p, tla, m, c, k1, k2 are never specified numerically. That makes the core extraction step hard to fully audit — though the pipeline is reproducible from code, so a motivated reader could dig them out. The closed-loop validation is the weakest part. The stress-test concern lands: the same 262 extreme scenarios are used across all three trials, with failures fed back as reflection prompts. The collision rate drops from 4.58% to 1.91%, but there's no held-out set, so this could be prompt overfitting rather than generalizable improvement. The paper doesn't claim it's generalizable per se — it frames the trials as producing preference pairs for downstream training — but the language around 'verifiable safety signals' oversells what the experiment actually shows. The constant-acceleration simulator is also a crude proxy for real vehicle dynamics. None of this undermines the dataset itself. The 31K events with structured annotations stand on their own. The LLM annotation layer is a useful scaffold even if its quality isn't independently validated at scale. This paper is for researchers building risk-aware driving agents who need curated edge-case data with language supervision. It deserves a serious referee — the dataset contribution is solid enough to warrant careful review, and the validation gaps are addressable in revision.","headline":"Dataset of 31K high-risk driving events with LLM annotations — solid resource, but closed-loop validation is in-sample and underspecified","tokens_in":24916,"tokens_out":576,"would_cite":true,"duration_ms":106411,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"31,398 high-risk driving events paired with LLM reasoning and simulator checks","keywords":[],"falsifier":"If a substantial fraction of the 31,398 curated events, when reviewed by independent traffic-safety experts, turn out to be routine driving situations that do not require evasive or defensive maneuvers — or conversely, if a large number of genuine high-risk events in the source datasets are missed by the three-stage filter — then the extraction pipeline does not accurately identify safety-critical driving, and the dataset's value for training risk-aware agents would be undermined.","tokens_in":24206,"feed_emoji":"🚗","tokens_out":1245,"duration_ms":154197,"temperature":0.7,"pith_summary":"The paper introduces K-Risk, a dataset built to address a gap in autonomous driving research: existing driving datasets are dominated by routine, uneventful driving, while the rare safety-critical events that matter most for safe deployment are almost absent. The authors aggregate 20 trajectory datasets from Europe, China, and the United States, then apply a three-stage filtering pipeline — a driver risk field scorer, calibrated hard-maneuver detectors, and a two-second trajectory-conflict predictor — to extract 31,398 high-risk events, including 1,036 extreme near-collision cases. Each event is stored as a synchronized triple: raw trajectory data in CSV, structured metadata in JSON, and natural language annotations in text. The language layer includes rule-based scenario descriptions and abnormal-behavior alerts for every event, and for a representative subset, an LLM generates causal risk analyses and discrete action recommendations from a five-option schema (keep, turn left, turn right, accelerate, decelerate). These recommendations are validated in a collision-free simulator with iterative reflection, producing verified safety signals. The paper's central claim is that by combining multi-dimensional physical risk modeling, interpretable language supervision, and closed-loop decision verification in a single standardized format, K-Risk provides the missing foundation for training and evaluating risk-aware autonomous driving agents that can reason about why a situation is dangerous and what to do about it.","feed_headline":"31,398 high-risk driving events paired with LLM reasoning and simulator checks","feed_subtitle":"A new dataset bridges raw trajectory data, semantic risk analysis, and verified decisions for autonomous driving safety","key_machinery":"The annotation protocol has three physical filters: (1) a Driver Risk Field (DRF) that scores each frame by ego-vehicle dynamics, surrounding-vehicle configuration, and spatial proximity, retaining only the top 10% per source; (2) a hard-maneuver detector using ±3 m/s² acceleration thresholds and dataset-specific lateral-speed thresholds within a 0.7-second window; (3) a two-second trajectory-conflict predictor under constant-acceleration and constant-steering assumptions. Events are graded moderate (76.8%), high (19.7%), or extreme (3.5%). The semantic layer adds rule-based scenario descriptions and abnormal-behavior notifications for all events, plus LLM-generated causal risk analyses and ","core_discovery":"The paper demonstrates that it is feasible to construct a large-scale, multi-source dataset of safety-critical driving events where each event carries not just trajectory data but also structured semantic annotations and verifiable decision labels. The key mechanism is a layered pipeline: first, a driver risk field filter retains only the top 10% of frames by perceived danger; second, calibrated behavioral thresholds detect hard maneuvers that measurably reduce time-to-collision; third, a two-second trajectory-conflict predictor flags imminent collisions. On top of this physical filtering, an LLM annotation layer converts the numerical records into natural language risk analyses and action推荐","pith_inferences":["If the DRF thresholds and ±3 m/s² hard-maneuver cutoffs are not equally valid across all 20 source datasets — which span different road geometries, traffic cultures, and vehicle types — then the curated events may systematically over- or under-represent certain risk categories, biasing downstream models trained on K-Risk.","The constant-acceleration, constant-steering trajectory prediction used in the conflict detector is a simplification that may miss conflicts arising from non-linear driver behavior, particularly in complex intersection or roundabout scenarios where vehicles follow curved paths.","The 5% human-driver collision rate on the extreme subset, compared to the LLM's 1.91% after three trials, could be interpreted as evidence that LLM-based reasoning outperforms human drivers in these scenarios — but the comparison may be unfair because the human drivers operated without the structured risk reminders and eight-agent situational context provided to the LLM."],"forward_implications":["K-Risk's synchronized trajectory-metadata-language format could become a standard template for other safety-critical domains where rare events must be paired with interpretable reasoning, such as aviation near-misses or medical emergency response.","The closed-loop validation approach — where LLM recommendations are tested in simulation and failure cases are fed back as reflection — could be generalized as a training methodology for any decision-making agent where verifiable outcomes exist.","The trial-1 vs trial-3 preference pairs generated through iterative reflection provide a ready-made dataset for preference optimization methods without additional human labeling, which could lower the cost of aligning language models to safety-critical domains.","The multi-source aggregation across three continents demonstrates that heterogeneous trajectory data can be unified under a common risk framework, suggesting that similar unification could be attempted for other fragmented driving data collections."],"fun_headline_variants":["K-Risk dataset links trajectory data with LLM risk annotations for autonomous driving","Risk-filtered driving dataset adds LLM reasoning and simulator-verified decisions","Pipeline extracts high-risk driving events with multi-layer risk detection and LLM annotat","31,398 high-risk driving scenarios with structured language supervision and safety checks","Knowledge-augmented dataset pairs rare driving events with causal LLM risk analysis"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The annotation protocol assumes that a single set of calibrated thresholds — DRF top-10%, ±3 m/s² acceleration, TTC below 5 seconds — correctly identifies safety-critical events across all 20 heterogeneous source datasets spanning different countries, road types, and vehicle behaviors. If these thresholds are miscalibrated for specific driving cultures or road geometries, the curated events may not represent genuine high-risk scenarios.","fun_headline_variants_meta":{"raw":{"variants":["K-Risk dataset links trajectory data with LLM risk annotations for autonomous driving","Risk-filtered driving dataset adds LLM reasoning and simulator-verified decisions","Pipeline extracts high-risk driving events with multi-layer risk detection and LLM annotations","31,398 high-risk driving scenarios with structured language supervision and safety checks","Knowledge-augmented dataset pairs rare driving events with causal LLM risk analysis"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":694,"prompt_tokens":595,"completion_tokens":99,"prompt_tokens_details":null},"tokens_in":595,"tokens_out":99,"duration_ms":61289,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T20:01:24.464412+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a substantial fraction of the 31,398 curated events, when reviewed by independent traffic-safety experts, turn out to be routine driving situations that do not require evasive or defensive maneuvers — or conversely, if a large number of genuine high-risk events in the source datasets are missed by the three-stage filter — then the extraction pipeline does not accurately identify safety-critical driving, and the dataset's value for training risk-aware agents would be undermined.","supporting_citations":[],"review_version":1}