{"id":"efa462a7-2bf7-4359-b8a0-8cc5642d670d","arxiv_id":"2502.02145","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM with ego-centric prompts detects collisions and generates adversarial driving scenarios more reliably than Cartesian prompts, though validation of generation is limited.","lead":"This paper tests whether large language models can judge driving scenarios for safety and then rewrite vehicle paths to force collisions. The ego-centric prompting approach detected 96 of 100 collision cases correctly, suggesting LLM-based tools could reduce manual scenario testing effort.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The generation module's reported success is self-scored by the same LLM risk prompt used to select the trajectory; a single case study cannot carry the central generation claim.","rationale":"I concur with the reader's verdict. The reader's weakest assumption—that generation success is measured by the same LLM prompt used for selection—is the most load-bearing concern. I examined whether the evaluation module itself has a more serious flaw: the balanced 100/100 test set makes the reported F1 prevalence-dependent, and no confidence intervals are given. That is a real reporting limitation, but it does not undermine the relative comparison of Promptcart versus Promptego or the main evaluation claim. Another possible concern is that handcrafted TTC/MDC thresholds are used to create safety-inference ground truth; this is acceptable because the paper claims reduced dependence, not elimination. The generation result, however, is the only component that directly supports the 'generate safety-critical scenarios' part of the central claim, and its sole quantitative evidence is self-referential. This warrants a CONDITIONAL rather than ACCEPT verdict. I would not move to REJECT because the evaluation results are concrete and reproducible in principle, and the generation module could plausibly be validated; the missing objective validation is exactly what should be required. Thus the reader's verdict should remain unchanged, with the generation validation as the condition.","tokens_in":11116,"tokens_out":3501,"duration_ms":36765,"concrete_test":"Run the generation module on 50 randomly selected safe scenarios from the 4,700. For each generated scenario, re-simulate with the same Frenetix motion planner and compute objective safety metrics (collision flag, minimum TTC, minimum MDC) from the actual modified trajectories. Compare the distribution of these objective metrics against (a) the original safe scenarios and (b) a control condition in which ego-attacker selection is random or rule-based and trajectories are perturbed with random or optimization-based modifications, while keeping the LLM scoring prompt only for candidate selection. If the LLM-generated scenarios do not show a statistically significant reduction in min-TTC/MDC or increase in collision rate over the control, the generation claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim has two components: evaluation and generation. The evaluation component (Sec. IV-B/C) is reasonably supported by stratified 100/100 tests across three LLMs. The load-bearing weakness is in generation (Sec. IV-D). The generation module (Sec. III-C.2) uses the safety-critical metrics prompt to (i) assign risk scores to identify the ego-attacker, (ii) score multiple candidate adversarial trajectories under Self-Consistency, and (iii) select the candidate with the lowest risk score. The paper then reports that selected candidate's risk score decreasing over time (Fig. 8) as evidence that a safety-critical scenario was synthesized. This is circular: the same LLM prompt is both the selection criterion and the evaluation metric, and no independent objective check is reported. The only additional evidence is a single visualized case study (BEL Antwerp-1_14_T-1, Fig. 9). There is no re-simulation of the modified trajectory with the Frenetix planner, no recomputation of TTC/MDC from the modified geometry, no collision check, no distribution over multiple generated scenarios, and no baseline such as random trajectory perturbations or a rule-based attacker selection. Without such validation, a low LLM-assigned risk score on a generated trajectory does not establish that the scenario is actually safety-critical or realistic; it may only reflect the prompt's bias toward modified trajectories. Because the abstract and conclusion explicitly tout the generation module as a contribution, this unsupported link is load-bearing for the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an LLM-based framework with two modules: an evaluation module that converts structured CommonRoad scenario data into natural-language descriptions via Cartesian or ego-centric parsers and prompts (with contextual prompting, chain-of-thought, and in-context learning), and a generation module that identifies an ego-attacker and adversarially modifies its trajectory to create safety-critical scenarios. The evaluation is tested on 200 randomly selected scenarios from a 6000-scenario simulation dataset; ego-centric prompts reach F1 0.87 for collision evaluation and 0.83 for safety inference with Gemini-1.5Pro, with ablations and comparisons across three LLMs. The generation module is validated by one case study and a plot of the selected trajectory's risk score over time. The paper argues that LLMs with domain-informed prompting can reduce dependence on handcrafted safety metrics.","tokens_in":11386,"tokens_out":5222,"duration_ms":54100,"significance":"If the evaluation results hold up, the ego-centric prompt design is a useful contribution: the ablations show clear gains from CP, CoT, and ICL, the comparison across three LLMs is informative, and the open-source release supports reproducibility. The collision-evaluation claim is not circular because the labels are external simulator collisions. The generation direction is novel—using an LLM to select an ego-attacker and synthesize adversarial trajectories—but the evidence for it is currently much weaker than for evaluation. The absence of baselines and uncertainty quantification for the evaluation, and the self-scored generation validation, mean the central two-part claim is only partially established.","major_comments":[{"comment":"The central claim that the generation module 'synthesizes realistic, safety-critical scenarios' is supported only by the selected candidate's LLM-assigned risk score decreasing over time (Fig. 8) and by a single visualized case study (Fig. 9, BEL Antwerp-1_14_T-1). Because the same safety-critical metrics prompt is used both to score candidate trajectories under Self-Consistency and to produce the reported post-generation risk scores, the selection criterion and the evaluation metric coincide; a low risk score on a modified trajectory may reflect prompt bias rather than genuine safety-criticality. The authors should re-simulate the generated trajectories with the Frenetix planner for a distribution of generated scenarios, recompute TTC/MDC and collision outcomes from the modified geometry, and compare against a random trajectory-perturbation or rule-based attacker baseline.","section":"§III-C.2 and §IV-D"},{"comment":"The evaluation use cases report F1 scores without any comparison to the TTC/MDC threshold classifiers from which the ground-truth labels are derived (Section IV-A), and the prompts explicitly embed the same kinds of thresholds (Section III-B.2). A simple threshold-based baseline is therefore the natural lower bound for these tasks, and its omission leaves unclear how much the LLM adds over handcrafted metrics. Please add such baselines for both collision evaluation and safety inference, and report the threshold values used.","section":"§IV-B and §IV-C, Tables I–IV"},{"comment":"The abstract's claim of 'reducing dependence on handcrafted metrics' is not reflected in the design details: TTC/MDC thresholds are handcrafted in the prompts, the same thresholds generate the ground truth for safety inference, and the generation module's ego-attacker selection depends on unspecified risk-score thresholds and a spatial search radius. Since these free parameters directly affect the reported F1 scores and the generation outcomes, they should be stated explicitly, and a sensitivity analysis should be provided for at least the main thresholds.","section":"§IV-A and §III-C.2"}],"minor_comments":[{"comment":"For GPT-4o and DeepSeek-V3, the confusion-matrix entries sum to 204 rather than 200 for both prompt types; please correct or explain the discrepancy.","section":"Table II"},{"comment":"The mapping from TTC/MDC values to the 0–5 risk score is not defined; please state the thresholds and describe how the plotted risk scores are computed.","section":"Fig. 8"},{"comment":"The text contains a likely typo: 'within 30 min front' should presumably read 'within 30 m in front'; other minor spacing errors (e.g., 'outcome,i.e.') should also be corrected.","section":"§IV-D"},{"comment":"The safety-inference results are point estimates from a single 100/100 scenario split without confidence intervals; given LLM API stochasticity, reporting intervals over repeated splits or samples would make the cross-model differences in Tables II and IV interpretable.","section":"§IV-C"}],"recommendation":"major_revision","confidential_remarks":"The generation module's self-scored validation is the main risk: if the authors can re-simulate generated trajectories and report independent collision/TTC/MDC statistics over multiple scenarios, the paper's central claim would become defensible. The evaluation-only contribution might be publishable with a threshold baseline, but as submitted the two-part claim is unevenly supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the evaluation half of this paper is solid and worth reading; the generation half is not yet supported by the evidence. The ego-centric prompt finding is the real contribution.\n\nWhat's actually new: the paper is the first I know of to use LLMs for direct safety-criticality assessment of structured driving scenarios, rather than for realism or driving-style evaluation. The Promptego vs. Promptcart comparison is clean and consistent across three LLMs: the ego-centric representation lifts collision-evaluation F1 from 0.21 to 0.87 on Gemini-1.5Pro, and similar gains hold for GPT-4o and DeepSeek-V3. The ablation shows each prompt technique (CP, CoT, ICL) adds a measurable increment. The safety-inference task, using a 1 s history, is a genuine predictive setup and the 0.83 F1 is respectable. The authors also release code and scenarios, which is more than many papers in this area do.\n\nWhere it gets soft: the evaluation experiments lack a trivial baseline. A simple TTC-threshold classifier on the same 200 scenarios would tell you how much value the LLM actually adds over a handcrafted metric. Since the ground truth is simulator collisions, this is not a circularity problem, just a missing comparison that limits the strength of the \"reducing dependence on handcrafted metrics\" claim.\n\nThe bigger problem is the generation module. The pipeline identifies an ego-attacker, generates candidate trajectories, scores them with the same safety-critical metrics prompt, and then reports the selected trajectory's risk drop as evidence of success. That is selection and evaluation with the same instrument. There is no re-simulation of the modified trajectory, no recomputation of TTC/MDC from the new geometry, no collision check, no distribution over generated scenarios, and no baseline like random perturbation or rule-based attacker selection. The single visualized case study cannot carry that load. The abstract and conclusion explicitly tout generation as a contribution, so this gap is load-bearing, not cosmetic.\n\nMinor but worth noting: the point estimates lack confidence intervals, and LLM API variability means the exact numbers could shift on re-run. I did not independently run the code, so I am taking the reported results on faith, but the release makes verification possible.\n\nBottom line: this paper deserves a serious referee. The evaluation results are a real empirical step forward for LLM-based scenario assessment, and the generation weakness is fixable with additional experiments. If I were the editor, I would send it to review and ask for baselines, confidence intervals, and an independent check of the generated scenarios (re-simulation or metric recomputation) before publication.","headline":"The evaluation half is solid and the ego-prompt finding is the real contribution; the generation half leans on a self-scored LLM prompt and a single case study, so that claim is not yet supported.","tokens_in":11920,"tokens_out":1626,"would_cite":true,"duration_ms":17428,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an LLM prompted with ego-centric driving descriptions can evaluate collision and safety-critical scenarios and adversarially edit an agent's trajectory to turn a safe scene into a collision.","keywords":["Autonomous Driving","Large Language Models","Scenario-based Testing","Safety-critical Scenario Evaluation","Safety-critical Scenario Generation","Prompt Engineering","Ego-centric Representation","Adversarial Scenario Generation"],"falsifier":"Re-run every generated trajectory through the paper's 2D simulator with its motion planner and compare outcomes against the original safe runs; if the LLM-selected trajectories do not cause a significantly higher rate of collisions, near-misses, or metric values below critical thresholds, the generation claim fails. In parallel, run the collision evaluator on a fresh held-out set of simulated scenarios and check its verdicts against simulator ground truth.","tokens_in":10924,"feed_emoji":"🚗","tokens_out":9061,"duration_ms":83442,"temperature":0.7,"pith_summary":"The paper is trying to establish that a general-purpose large language model, prompted with structured driving data described from the ego vehicle's viewpoint, can replace parts of the handcrafted metric pipeline in scenario-based testing. Its evaluation module turns 2D simulation scenarios into natural language and decides whether a collision occurs or whether the scene will become safety-critical; the ego-centric prompt reaches an F1 of 0.87 for collision evaluation and 0.83 for safety inference with one model, and similar scores with two other models. Its generation module identifies the most risk-inducing agent, rewrites that agent's trajectory, and reports a much lower risk score, turning a previously safe scenario into a collision in the shown case study. If this holds, test engineers could scale virtual safety testing by asking an LLM to flag and create critical cases instead of pre-defining every metric and reading the output by hand.","feed_headline":"Ego-centric prompts lift LLM collision detection to 0.87 F1","feed_subtitle":"The same ego-view prompts reach 0.83 F1 for predicting risky scenes and power adversarial generation.","key_machinery":"The load-bearing mechanism is the ego-centric parser and prompt template, Promptego, contrasted with the Cartesian Promptcart. The parser re-expresses every agent's state in longitudinal and lateral coordinates relative to the ego vehicle, adding motion phrases such as 'approaching the EV longitudinally from the front', so the LLM receives a driver's-perspective description rather than absolute coordinates. The prompt stacks Contextual Prompting, which defines safety metrics such as time-to-collision and minimum distance to collision, with Chain-of-Thought reasoning and In-Context Learning examples. The generation module adds Self-Consistency, sampling several adversarial trajectories and selecting the one with the lowest LLM-assigned risk score.","core_discovery":"The authors claim that LLMs can act as both evaluator and generator of safety-critical driving scenarios. They compare two prompt families, Cartesian and ego-centric, and find the ego-centric representation is consistently superior: for Gemini-1.5Pro, F1 rises from 0.21 to 0.87 in collision evaluation and from 0.62 to 0.83 in safety inference, with DeepSeek-V3 reaching 0.90 and 0.87 on the same tasks. The generator uses a safety-metric prompt to score each obstacle, identifies an ego-attacker, creates multiple candidate trajectories through self-consistency, and keeps the candidate with the lowest risk score, producing a case where the ego-attacker's risk score falls over time and ends in collision. The paper concludes that this reduces dependence on handcrafted metrics while remaining controllable.","pith_inferences":["Beyond the paper, the same ego-centric relabeling could extend to vision-language models operating on camera views or rendered scenes, since the gain seems to come from spatial framing rather than from text generation itself.","Beyond the paper, the generation module should be validated by re-simulating the generated trajectories and checking collisions or metric thresholds; the current evidence is one case study whose outcome is scored by the same prompt that selected it.","Beyond the paper, a natural next test is to feed generated scenarios to a motion planner and measure how frequently it fails, converting scenario quality into a downstream safety statistic.","Beyond the paper, the method reduces rather than eliminates handcrafted metrics, because metric thresholds still appear as prompt context and as the ground truth used to label the evaluation data."],"forward_implications":["Scenario evaluators can flag collisions and predict imminent risk directly from structured log data, without a human re-reading metric traces.","The ego-centric framing, not the particular model, drives most of the gain, since all three tested LLMs perform far better with Promptego than with Promptcart.","Adding Contextual Prompting, Chain-of-Thought, and In-Context Learning together improves the ego-prompt collision F1 from 0.70 to 0.87.","The generation module offers a controllable way to produce targeted critical scenarios by editing one risk-inducing agent, complementing data-driven and knowledge-based generators.","The safety inference output can serve as an online supplementary risk signal for motion planners, although the paper notes the current response time is seconds rather than real time."],"supporting_citations":[{"why":"Defines the TTC and MDC safety metrics that the prompts use as contextual domain knowledge and that the dataset uses as ground truth.","marker":"[4]"},{"why":"Motivates adapting pretrained LLMs to domain tasks through prompting rather than fine-tuning.","marker":"[7]"},{"why":"Supplies the 2D simulator and standardized scenario representation from which the structured data and 6000 simulated scenarios come.","marker":"[29]"},{"why":"Provides the lanelet map representation used to describe road networks in the structured scenario data and parsers.","marker":"[30]"},{"why":"Provides the motion planner whose simulations generate the collision and non-collision trajectories used as ground truth.","marker":"[31]"},{"why":"One of the three pretrained LLMs compared in the evaluation experiments.","marker":"[32]"},{"why":"Supports the structured output format that keeps LLM responses parseable and reduces hallucination.","marker":"[33]"},{"why":"One of the three pretrained LLMs compared, and the model used in the main ablation tables.","marker":"[34]"},{"why":"One of the three pretrained LLMs compared in the evaluation experiments.","marker":"[35]"}],"fun_headline_variants":["LLM ego-view prompts hit 0.87 F1 for crash detection","Ego-centric LLM prompts lift collision detection F1 to 0.87","LLM safety: ego-prompting beats Cartesian for crash scenes","Adversarial LLM generator crafts safety-critical driving scenes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generation module's reported success is measured by the same LLM risk-score prompt that selected the trajectory, and the only validation is a single visualized case study, so the claim of realistic safety-critical generation assumes that self-scored risk drops correspond to genuinely critical driving situations.","fun_headline_variants_meta":{"raw":{"variants":["LLM ego-view prompts hit 0.87 F1 for crash detection","Ego-centric LLM prompts lift collision detection F1 to 0.87","LLM safety: ego-prompting beats Cartesian for crash scenes","Adversarial LLM generator crafts safety-critical driving scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1512,"prompt_tokens":937,"completion_tokens":575,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":497}},"tokens_in":553,"tokens_out":575,"duration_ms":6189,"temperature":1.0,"reasoning_tokens":497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:08:29.982300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run every generated trajectory through the paper's 2D simulator with its motion planner and compare outcomes against the original safe runs; if the LLM-selected trajectories do not cause a significantly higher rate of collisions, near-misses, or metric values below critical thresholds, the generation claim fails. In parallel, run the collision evaluator on a fresh held-out set of simulated scenarios and check its verdicts against simulator ground truth.","supporting_citations":[{"cited_title":"Simulation-based identification of critical scenarios for cooperative and automated vehicles,","cited_arxiv_id":null,"evidence_quote":"Defines the TTC and MDC safety metrics that the prompts use as contextual domain knowledge and that the dataset uses as ground truth."},{"cited_title":"Lanelets: Efficient map representation for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Provides the lanelet map representation used to describe road networks in the structured scenario data and parsers."},{"cited_title":"Frenetix: A high-performance and modular motion planning framework for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Provides the motion planner whose simulations generate the collision and non-collision trajectories used as ground truth."}],"review_version":1}