{"id":"326c8f3c-c1d6-4c4c-b433-4fe833d217de","arxiv_id":"2507.15874","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An LLM-based pipeline classifies braking scenarios from Argoverse 2 and retrieves known and out-of-distribution scenarios better than rule-based baselines.","lead":"This paper describes a pipeline that converts vehicle sensor data into text and uses a large language model to explain why a car braked and to label the scenario. It reports better retrieval of known and novel braking scenarios than a tuned rule-based filter on the Argoverse 2 dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KeyIdent's risk-tag gate removes anticipatory braking before the LLM sees it, so the reported ID/OOD gains do not cover the full 'why braking' question.","rationale":"The reader identified the same load-bearing assumption in KeyIdent: a causal relation between a guest object and ego braking is only admitted if the guest is accompanied by LOW/HIGH collision risk or trajectory-overlap risk. The paper's own failure cases make this concrete, showing that right-ped and approach-stop scenarios often involve early, gentle braking with no detected risk. This is the weakest point of the central claim because it affects every downstream metric: the LLM never sees the filtered objects, the scenario database omits their descriptions, and both the ID retrieval comparison and the OOD retrieval tables are computed on the surviving subset. The concern is not that the method is wrong on what it evaluates; it is that the evaluation set is biased toward critical, risk-detectable braking, so the claim of general 'why braking' reasoning is overbroad. This is fully consistent with the reader's conditional verdict. I do not see a reason to move the verdict to ACCEPT or REJECT: the paper is internally coherent, the failure analysis is honest, and the limitation is addressable by reporting per-category KeyIdent recall and by evaluating with an unfiltered baseline. The recommended verdict remains CONDITIONAL, which is the same as the reader's verdict, so verdict_should_be is UNCHANGED.","tokens_in":10004,"tokens_out":5289,"duration_ms":65666,"concrete_test":"Using the paper's own annotation set (Table III), compute per-category KeyIdent recall for all 727 annotations: for each annotated log/guest pair, check whether the guest survives the Sec. III-C rule. In particular, measure survival for the 91 approach-stop and 27 right-ped instances; the paper's Fig. 5b/c suggests it will be well below the reported 0.86 overall. Then recompute Table VI's P@10/R@50 with filtered-out instances counted as retrieval misses. If R@50 for approach stop drops materially below 0.31, the zero-shot OOD claim does not extend to anticipatory braking, and the paper should scope its conclusion accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The KeyIdent module (Sec. III-C) admits only objects with LOW or HIGH CollisionTag or TrajOverlapTag while the ego is DECELERATING or STANDING STILL. TrajOverlapTag is computed from future bounding-box overlaps within 1.5/3.0 s (Sec. III-B), so an object that triggers only gentle anticipatory braking — where the ego slows early and stops safely short — typically has neither risk tag. The paper's own failure analysis (Sec. V-E, Fig. 5b/c) confirms this for approach-stop and right-ped scenarios. As a result, the scenario database contains only risk-tagged ego-object pairs, and Table VI's OOD retrieval metrics are computed on that filtered subset. KeyIdent recall is reported only for four ID categories (Table IV), not for approach stop, lead brake, or ped crossing, so the filter's impact on the OOD claim is not quantified. Consequently, the central claims that the method 'outperforms rule-based baseline' and 'demonstrates zero-shot generalization on OOD unseen scenarios' are established only for braking events whose cause is detectable by conventional collision or trajectory-overlap risk. Since the motivating question is 'Why does a vehicle brake?', the claim is narrower than stated. This is an acknowledged limitation rather than an internal inconsistency, but it is load-bearing for the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based framework for extracting and reasoning about the causes of ego-vehicle braking events from BUS data. The pipeline first applies rule-based activity and interaction tagging (including a newly introduced TrajOverlapTag), then uses a KeyIdent module to filter candidate interacting objects, converts the selected ego-object pairs into structured natural-language descriptions, and asks an LLM to produce a rephrased description, a scenario category, and an explanation. A dual-path retrieval module supports both category-based search for known scenarios and embedding-based similarity search for out-of-distribution (OOD) scenarios. The authors curate 727 scenario annotations from 700 Argoverse 2 logs and report that their approach outperforms the rule-based baseline of Guo et al. [3] (overall F1 0.52 vs 0.33) and shows 'zero-shot generalization' on three OOD categories not present in the prompt.","tokens_in":10295,"tokens_out":5275,"duration_ms":62346,"significance":"If the claims hold, the paper makes a useful applied contribution: it shows how low-level time-series vehicle data can be converted into natural-language scenario descriptions that an LLM can classify and retrieve, without fine-tuning and without relying on HD maps. The dual-path retrieval design is sensible, and the introduction of TrajOverlapTag is a reasonable extension of risk tagging that uses actual future trajectories. The paper also contributes a curated scenario annotation set for Argoverse 2. The main claims are not circular in a derivation sense, and the OOD retrieval path is genuinely zero-shot in that no OOD labels are used in the prompt. However, the scope of the central claim is narrower than stated because the KeyIdent filter removes exactly the anticipatory-braking cases that the motivating question 'Why braking?' most naturally includes, and several evaluation details are missing. The paper is therefore of interest to the scenario-mining community, but the headline claims need revision and additional evidence.","major_comments":[{"comment":"The KeyIdent gate defined in Section III-C admits only objects with LOW or HIGH CollisionTag or TrajOverlapTag while the ego is DECELERATING or STANDING STILL. The failure analysis in Section V-E and Figures 5b/5c explicitly states that approach-stop and right-pedestrian scenarios often involve only gentle, anticipatory braking with no detected collision or trajectory-overlap risk, so those objects are filtered out before the LLM sees them. Consequently, the scenario database and the OOD retrieval metrics in Table VI are computed on the filtered subset only, and Table IV does not report KeyIdent recall for approach-stop, lead-brake, or pedestrian-crossing categories. The abstract and conclusion therefore overstate the scope of the contribution: the reported gains apply to braking events whose cause is detectable by conventional collision or overlap risk, not to all braking events behind the motivating question 'Why braking?'. I ask the authors to quantify KeyIdent's coverage for the OOD categories and to qualify the generalization claim accordingly.","section":"Section III-C and V-E"},{"comment":"The baseline comparison is not fully specified. Section V-B states only that the baseline retrieval filter was 'carefully adjusted' to maximize its F1 score, without giving the exact thresholds, the search procedure, or the per-category tuning details. Since the baseline is a rule-based filter with many free thresholds, and since some categories have very few examples (e.g., 5 left-oppo annotations in Table III), the headline overall F1 improvement (0.52 vs 0.33) cannot be distinguished from test-set tuning or sampling noise. Please report the exact baseline rule set, the optimization criterion, and per-category counts, and add confidence intervals or a significance test over annotation or bootstrap samples.","section":"Section V-B"},{"comment":"The in-distribution evaluation is a closed-set classification task in which the LLM prompt contains the same scenario-category names and definitions that the human annotators used to label the data, and Table V evaluates only four of the seven categories. This makes the ID result largely a test of label matching rather than of semantic scenario understanding. I recommend including 'unknown_but_relevant' and 'not_relevant' in the reported confusion matrix, holding out one or more categories from the prompt to test category-level generalization, or otherwise demonstrating that the LLM is not simply exploiting the provided definitions.","section":"Sections III-D and V-B"},{"comment":"The OOD retrieval evaluation is under-specified. The query descriptions in Table VI are short natural-language statements that are very close to the names of the target categories, and the retrieved embeddings are computed from LLM-generated rephrased descriptions of the same logs. The paper does not state whether these query texts were excluded from any development process, nor does it compare the embedding-based retrieval path against a rule-based or TF-IDF baseline. Without such a comparison, the P@10 and R@50 values in Table VI do not by themselves support the claim of 'strong zero-shot generalization.' Please add a baseline for the retrieval path and a discussion of query-robustness.","section":"Section V-C"},{"comment":"The evaluation relies entirely on the authors' curated annotations over 700 Argoverse 2 logs, but no annotation protocol, inter-annotator agreement measure, or label-quality statistics are reported. Inconsistent or subjective labels would directly affect all precision, recall, and F1 numbers in Tables IV-VI. Please add a short annotation-quality subsection, for example a second annotator on a random sample with Cohen's kappa or a similar agreement metric.","section":"Section IV-A"}],"minor_comments":[{"comment":"The figure contains the misspelled word 'Explaination'; it should be 'Explanation.'","section":"Figure 2"},{"comment":"The paper does not state whether the 'Overall' precision, recall, and F1 values are macro-averages or micro-averages over the four categories; please define the aggregation and, if macro, report the category weights or the per-category support.","section":"Table V"},{"comment":"The deterministic sampling settings (temperature = 0, top_p = 1.0, top_k = 1) are reported for the best model, but it is not stated whether the same settings were applied to GPT-4.1 and Qwen3-32B; please clarify.","section":"Section IV-B"},{"comment":"The acronym 'BUS' is used without expansion; please spell out 'body control module / vehicle bus system' at first use and define what signals are meant.","section":"Abstract and Section I"},{"comment":"Please state whether the embedding-based retrieval was evaluated only on the filtered subset produced by KeyIdent or on the full annotated set; this is closely related to the first major comment and should be made explicit in the table caption.","section":"Section V-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the core pipeline is plausible, but the headline claim of zero-shot OOD generalization is currently broader than the evidence. The KeyIdent filtering issue is not merely a presentation problem: it changes what population the reported metrics describe. The revision should either modify the filter to preserve anticipatory-braking cases or reframe the claims to state that the method handles braking events with detectable risk tags. I would also urge the editor to require the exact baseline configuration and annotation-quality metrics before considering the paper for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. Core idea is solid: translate BUS data into structured text, let an LLM reason about braking causes, and offer dual-path retrieval (exact category plus embedding similarity). The new TrajOverlapTag, which uses actual future trajectories instead of predicted TTC, is a genuinely useful addition for protective braking. The curated Argoverse 2 annotations (700 logs, 9 categories) are valuable even if not yet released.\n\nThe main experiment supports the headline: overall F1 0.52 vs 0.33 against the rule-based baseline, with consistent gains across all four ID categories. The failure analysis is honest, and the three-LLM comparison is a nice touch.\n\nNow the soft spots. The stress-test concern is real and it lands. KeyIdent only admits objects with LOW/HIGH CollisionTag or TrajOverlapTag while the ego is decelerating or standing still. The paper's own Figures 5b/5c show that right-ped and approach-stop often involve gentle anticipatory braking with no detected risk, so those objects are filtered out before the LLM sees them. That means the ID/OOD results are computed on a risk-filtered subset, and the zero-shot generalization claim is narrower than \"why braking\" suggests. The paper acknowledges this in V-E, but it is a load-bearing caveat.\n\nOther issues: ID categories are listed in the prompt, so the LLM is doing closed-set classification with the same labels used in annotations, which limits the strength of the ID result. The 700-log selection is under-described; no annotation-reliability metrics or error bars; baseline tuning is \"carefully adjusted\" with no specifics. No code or annotations are released, which hurts reproducibility. OOD retrieval numbers are modest, especially approach stop (P@10 0.30, R@50 0.31), and that is on the filtered set.\n\nNone of this is fatal. The engineering is coherent, the writing is clear, and the failure analysis is candid. It deserves a serious referee. I would ask the authors to release code and annotations, report annotation agreement, add error bars, and either broaden KeyIdent to cover anticipatory braking or explicitly scope the claims to risk-triggered braking.\n\nRecommendation: send to peer review. It is a respectable contribution to ADAS scenario mining.","headline":"A useful LLM-based braking scenario extractor with a genuinely new overlap tag, but the KeyIdent filter silently drops gentle anticipatory braking, so the zero-shot OOD claim is narrower than the title suggests.","tokens_in":10830,"tokens_out":2025,"would_cite":false,"duration_ms":22159,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that converting vehicle bus signals into structured natural-language scenario descriptions lets an LLM classify the cause of braking events, outperforming a rule-based baseline on known categories and retrieving unseen…","keywords":["autonomous driving","ADAS","scenario extraction","large language model","braking events","scenario-based validation","out-of-distribution retrieval","zero-shot generalization"],"falsifier":"Take the annotated right-ped and approach-stop logs from the authors' own dataset, filter them through the KeyIdent rule, and compute recall among those with gentle braking and no risk tags; if that recall is near zero, the zero-shot OOD claim holds only for risk-flagged events, not for the full set of annotated braking logs. A second check is to run retrieval with a query for a truly novel category (e.g., 'ego brakes for a cyclist swerving from behind a parked truck') and measure whether any correct matches appear in the top 50.","tokens_in":9818,"feed_emoji":"🚗","tokens_out":6834,"duration_ms":63657,"temperature":0.7,"pith_summary":"The paper tries to answer a practical question for automated-driving validation: when a logged vehicle brakes, which surrounding object caused it, and what scenario does that interaction belong to? Its proposal is to convert low-level vehicle bus signals into compact natural-language descriptions of each ego-vehicle–object interaction and let a large language model rephrase, classify, and explain the event. The authors claim this beats a hand-tuned rule-based filter on four known scenario categories and, through embedding-based similarity search, retrieves three previously unseen categories in a zero-shot manner. That matters because braking events are abundant in real-world driving data but rule filters are brittle in urban settings, so a language-based pipeline could make corner-case mining more scalable and more flexible.","feed_headline":"LLM pipeline explains why cars brake, beats rule filters","feed_subtitle":"Bus signals become descriptions that retrieve known and unseen braking events at higher F1 than rule filters.","key_machinery":"The load-bearing mechanism is a two-stage translation. First, a rule-based module assigns activity tags (longitudinal, lateral, position, collision, distance) and a newly defined TrajOverlapTag, which aggregates future bounding boxes over 1.5 and 3 second windows to flag overlap risk. A KeyIdent step keeps only ego–object pairs where the ego is decelerating or stationary and the object carries LOW or HIGH collision or overlap risk. Those surviving pairs are compressed from timestamp tables into duration-based text segments, fed to the LLM with a role prompt and a fixed closed set of scenario categories plus an 'unknown but relevant' option; the LLM returns a rephrased description, a category, and an explanation. The rephrased description is embedded, and retrieval runs two paths: exact match on the LLM-assigned category, or cosine similarity against a free-form query.","core_discovery":"The central claim is that an LLM, given structured text built from rule-based activity tags, can recover the cause of ego braking better than the rule-based filter alone. The paper reports overall F1 of 0.52 for the LLM pipeline versus 0.33 for the tuned rule baseline, with recall rising from 0.66 to 0.78 on the four known categories, and top-50 recall between 0.31 and 0.67 on three out-of-distribution categories found by embedding similarity. The authors also identify a boundary: because the KeyIdent filter only forwards objects that already show collision or trajectory-overlap risk while the ego decelerates or stands still, gentle anticipatory braking cases such as approaching a stopped vehicle or yielding to a pedestrian are often filtered out before the LLM sees them.","pith_inferences":["A natural next experiment is to relax the KeyIdent risk condition to include shrinking distance or predicted time-to-arrival, then measure whether right-ped and approach-stop recall rises without flooding the LLM with irrelevant objects.","The stored scenario embeddings could be clustered to discover a data-driven taxonomy of braking reasons, which the paper does not attempt.","Because the pipeline uses bus signals and object tracks rather than HD maps, it may transfer across cities or sensor configurations, but the paper does not test that transfer.","The top-10 precision of 0.30-0.40 on embedding retrieval suggests that in practice a human-in-the-loop review would be needed before the retrieved OOD scenarios are trusted for validation."],"forward_implications":["Scenario catalogs for ADAS validation could be built by typing a natural-language description rather than tuning filter thresholds.","Unseen scenario types can be surfaced from raw logs without retraining, as long as their descriptions resemble the stored text embeddings.","The LLM's explanation field gives engineers a human-readable reason for each retrieved braking event, which rule-based tags do not provide.","The reported coverage is limited to braking with detected risk: anticipatory or gentle braking without a risk tag will be missed, so the method should be seen as a critical-braking miner, not a universal braking-cause oracle."],"supporting_citations":[{"why":"Supplies the rule-based tagging rules and the baseline retrieval filter that the LLM pipeline must beat.","marker":"[3]"},{"why":"Provides the Argoverse 2 Sensor Dataset logs that the authors annotate and evaluate on.","marker":"[6]"},{"why":"Supplies the Savitzky-Golay smoother that denoises bus signals while preserving sudden braking transitions.","marker":"[16]"},{"why":"One of the three LLMs compared; its GPT-4 family is the reference for the strongest non-reasoning baseline.","marker":"[18]"},{"why":"The Gemini-2.5-Flash model that achieved the best reported F1 and is the primary reported result.","marker":"[19]"},{"why":"The Qwen3-32B model that underperformed due to token window limits, bounding the comparison.","marker":"[20]"},{"why":"Provides the text embedding model used for similarity-based retrieval of scenario descriptions.","marker":"[21]"}],"fun_headline_variants":["LLM decodes brake events, outscores rule filters on F1","Why that brake? LLM names scenario better than rule heuristics","LLM reads brake telemetry, names cause, tops rule filters","Braking event explainer: LLM beats rule filters on known and unknown","LLM turns brake signals into scenarios, beats rule filters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that any object that causes the ego to brake will already show a detected collision or trajectory-overlap risk while the ego is decelerating or standing still, so objects that trigger only gentle anticipatory braking are excluded before the language model can examine them.","fun_headline_variants_meta":{"raw":{"variants":["LLM decodes brake events, outscores rule filters on F1","Why that brake? LLM names scenario better than rule heuristics","LLM reads brake telemetry, names cause, tops rule filters","Braking event explainer: LLM beats rule filters on known and unknown","LLM turns brake signals into scenarios, beats rule filters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000904,"raw_usage":{"total_tokens":3867,"prompt_tokens":898,"completion_tokens":2969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2877}},"tokens_in":514,"tokens_out":2969,"duration_ms":23905,"temperature":1.0,"reasoning_tokens":2877,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:35:51.104408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the annotated right-ped and approach-stop logs from the authors' own dataset, filter them through the KeyIdent rule, and compute recall among those with gentle braking and no risk tags; if that recall is near zero, the zero-shot OOD claim holds only for risk-flagged events, not for the full set of annotated braking logs. A second check is to run retrieval with a query for a truly novel category (e.g., 'ego brakes for a cyclist swerving from behind a parked truck') and measure whether any correct matches appear in the top 50.","supporting_citations":[{"cited_title":"Scenario extraction from a large real-world dataset for the assessment of automated vehicles,","cited_arxiv_id":null,"evidence_quote":"Supplies the rule-based tagging rules and the baseline retrieval filter that the LLM pipeline must beat."},{"cited_title":"What is a savitzky-golay filter?[lecture notes],","cited_arxiv_id":null,"evidence_quote":"Supplies the Savitzky-Golay smoother that denoises bus signals while preserving sudden braking transitions."}],"review_version":1}