REVIEW 3 major objections 6 minor 28 references
Chat2Scenic turns regulatory text into executable driving scenarios at a 76.42% compilation success rate, roughly 2.5x the best prior method, by generating scripts component by component with retrieval-augmented prompting.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 02:12 UTC pith:QIF6GVWF
load-bearing objection Useful framework with a genuinely novel iterative component-wise RAG design, but the headline CSR/FA are best-on-benchmark numbers and a data-leakage check is needed before trusting them. the 3 major comments →
Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central discovery, on its own terms, is that component-wise iterative generation, combined with retrieval-augmented in-context learning, resolves the trade-off between compilation reliability and generalizability in DSL-based scenario generation. Rather than assembling pre-existing snippets (which compiles but does not generalize) or generating a full script at once (which generalizes but rarely compiles), the framework parses a query into a logical structure, generates each component in dependency order, and feeds the accumulated code back as context for later components. A dual retriever supplies semantically similar code snippets and documentation chunks to ground each generat
What carries the argument
The load-bearing mechanism is a logical structure schema S = {G, R, E, O, T} that decomposes any scenario description into global configuration (map, weather, vehicles) and scenario components (spatial relations, ego behavior, objects, restrictions). The generation module produces code component by component in a fixed dependency order, passing each generated component's code into the prompt for the next one, so compatibility is maintained incrementally. A dual-retriever RAG module fetches the most relevant code snippets (by semantic similarity) and documentation chunks (by hybrid BM25 plus embedding search fused with Reciprocal Rank Fusion) for each component description. These retrieved co
Load-bearing premise
The reported 76.42% compilation success and 58.17% framework accuracy come from the single best configuration (C11) chosen among 12 ablations on the same 123-scenario benchmark it is then evaluated against; the load-bearing premise is that this selection does not materially overfit the benchmark and that the human layer-scoring is a stable measure of quality.
What would settle it
Run the final C11 configuration and the same human scoring protocol on a fresh set of regulatory descriptions that were not part of the 123-scenario benchmark. If the compilation success rate falls substantially below 76.4% (for example, below 50%), the headline numbers are at least partly artifacts of benchmark-driven configuration selection.
If this is right
- Regulatory descriptions that previously required manual translation into simulation code can be converted to executable scenarios with a majority success rate, reducing the cost of building diverse test suites.
- The component-wise structure means a single failed component can be regenerated or corrected without discarding the whole scenario, easing debugging and human-in-the-loop refinement.
- The proposed benchmark (123 scenarios) and the layer-based human scoring protocol give the research community a shared, repeatable way to compare scenario-generation methods.
- Because the framework relies on prompting rather than fine-tuning, it can be applied to different LLMs, but the large gap between proprietary and open models suggests that model capability is currently the binding constraint.
Where Pith is reading between the lines
- The same component-wise generation idea likely transfers to other scenario DSLs (e.g., OpenScenario) and other simulators, so the contribution may generalize beyond the specific language evaluated here.
- The paper selects the best of 12 ablation configurations on the same benchmark it uses for the final comparison; an independent holdout set would clarify whether the 76.42% CSR reflects true generalizability or benchmark-specific tuning.
- The human layer-scoring protocol is expensive and subjective; a future automated metric (e.g., a learned verifier that checks alignment against the description) could make the benchmark cheaper and more reproducible.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Chat2Scenic, an iterative retrieval-augmented generation framework that converts natural-language regulation-style scenario descriptions into executable Scenic programs. The framework parses a description into logical components (global configuration, spatial relations, ego, objects, restrictions), retrieves code snippets and documentation through a dual RAG architecture, and generates components iteratively using contextual prompting, chain-of-thought, few-shot in-context learning, and retrieval-augmented in-context learning. The authors introduce a 123-scenario benchmark drawn from CARLA Leaderboard, NHTSA, and UN regulations, with metrics comprising CSR, response time, token usage, and human-scored layer alignment, plus FA = CSR × SQ. Using Gemini-3-Flash with configuration C11, they report CSR 76.42%, SQ 76.12%, FA 58.17%, outperforming ChatScene (30.08% CSR) and NL2Scenic (16.26% CSR) on the same benchmark.
Significance. If the reported results are unbiased, the contribution is significant: it addresses a real bottleneck in DSL-based scenario generation, introduces a reusable regulation-grounded benchmark, and provides an open-source implementation. Strengths include the public code release, the use of simulator compilation as an objective filter for CSR, and the component-wise iterative design, which is well motivated by the failure modes of full-script generation. The ablation over prompting techniques and the multi-backbone evaluation are also useful. However, the empirical claim currently depends on unaddressed evaluation-hygiene issues: possible overlap between the retrieval database and benchmark scenarios, selection of the best configuration on the same benchmark, and unmeasured human-evaluator variance. These issues are fixable, but the headline numbers cannot be taken at face value until they are resolved.
major comments (3)
- [§II-B.1 and §III-A.1] The RAG code database is built from "official Scenic sources" (§II-B.1), while the benchmark deliberately includes 24 CARLA Leaderboard scenarios (§III-A.1). The paper never states that the source files or scenario descriptions corresponding to the 123 benchmark items were excluded from the database. Since CARLA Leaderboard scenarios are official Scenic examples and §III-B.2 notes that ChatScene's database is built from CARLA Leaderboard scenarios, the retriever could return code that is effectively the ground-truth solution for a subset of test items, inflating both CSR and SQ. Please report a decontamination analysis, e.g., remove any snippet whose source scenario appears in the benchmark and re-run, or show retrieval outputs for benchmark queries and verify that no ground-truth snippet is returned.
- [§III-B.1, Table III and Table IV] Configuration C11 was selected as the best of 12 ablation configurations evaluated on the complete 123-scenario benchmark (Table III) and then used as the final result in Table IV. The headline 76.42% CSR and 58.17% FA are therefore selected-on-test estimates, not unbiased predictive estimates for new regulation descriptions. This also makes the comparison to ChatScene and NL2Scenic unfair to the extent that those baselines were not configuration-selected on this benchmark. Please either split the benchmark into development and held-out test sets, use nested cross-validation, or at minimum report the distribution over configurations and the selection rule, together with confidence intervals.
- [§III-A.3] FA is defined as CSR × SQ, where SQ is the average of human layer-alignment scores. The manuscript reports no detail on the human evaluation: number of evaluators, whether they were blind to the framework or method, whether the same evaluator scored all systems, or inter-rater reliability. Since the headline FA depends directly on these subjective scores, unmeasured evaluator variance could change the comparison. Please provide the evaluation protocol and agreement statistics (e.g., Cohen's kappa or per-layer variance), and ideally report SQ with confidence intervals.
minor comments (6)
- [Table I] The table header for RAG-ICL and the distinction between the 'Assemble' and 'Generation' rows are difficult to follow; consider simplifying the column layout or adding a footnote.
- [§I-C] The contribution text says '120+ scenario descriptions' while the benchmark has 123; please be consistent.
- [§II-C.1] SettingsDetector is described as using 'rule-based keyword matching patterns and example-based reasoning,' but the rules and examples are not provided. This limits reproducibility; please include them in an appendix or supplementary material.
- [Table III] For configurations with very low CSR (e.g., C2, C3), the layer-alignment averages are based on a handful of compiled scenarios and contain many dashes. Please clarify how SQ is computed in these cases and consider reporting the number of compiled scenarios.
- [§III-A.3] Response time and token usage are reported as averages without variance. Adding standard deviations or confidence intervals would help assess stability across scenarios.
- [Abstract and §I-A] The claim 'first iterative retrieval-augmented framework' is strong given that prior work such as Talk2Traffic and Rubavicius already combine interaction and retrieval. Please soften the claim or clarify the specific novelty.
Circularity Check
No significant circularity: the reported results are empirical benchmark measurements with independent SOTA baselines, and no derivation step reduces to its own inputs.
full rationale
Chat2Scenic is an empirical systems paper; its central claims are benchmark measurements, not a mathematical derivation. The only composite metric, FA = CSR × SQ, is an explicit definition rather than a quantity that is derived from or equivalent to the inputs, and its components are measured independently: CSR by executing Scenic programs in CARLA, and SQ by human layer-wise evaluation. The RAG code database is stated to come from official Scenic sources, while the benchmark is drawn from CARLA Leaderboard, NHTSA, and UN regulations; no quoted passage establishes that benchmark scenarios were already present in the retrieval database. The comparison against ChatScene and NL2Scenic on the same benchmark provides external grounding for the framework's relative improvement. The self-citation [28] is used only as an explanatory aside for why Gemini-3-Flash outperforms Gemini-3-Pro and is not load-bearing for the headline CSR/FA numbers. Selecting configuration C11 on the same benchmark used for final reporting is a legitimate model-selection/generalization-risk concern, but it is not a circularity in the derivation-chain sense: no fitted parameter is renamed as a prediction. Similarly, the absence of an explicit de-duplication statement is a potential data-leakage risk, not a demonstrated circular step. Under the rule that circularity requires a quotable reduction (Eq. X = Eq. Y by construction, or a fitted input presented as a prediction), no such reduction is present.
Axiom & Free-Parameter Ledger
free parameters (5)
- Code retriever top-K =
3
- SettingsDetector confidence threshold =
0.6
- Default global settings =
Town05, ClearNoon, Lincoln mkz
- Prompting configuration C11 =
CP+CoT+ICL+CodeICL (DocICL off)
- Human Scenario Quality scoring =
Average over six layers
axioms (6)
- domain assumption Pre-trained LLMs, prompted with CP/CoT/ICL/RAG-ICL, can translate component descriptions into compilable Scenic code.
- ad hoc to paper Concatenation of individually generated Scenic components yields a valid, executable Scenic program.
- domain assumption The official Scenic code snippet and documentation database covers the component types needed for all 123 benchmark scenarios.
- domain assumption Human layer-based scoring is a valid measure of semantic fidelity.
- domain assumption Benchmark scenario texts from CARLA, NHTSA, and UN regulations are complete and unambiguous enough to serve as ground truth.
- domain assumption The same prompting strategy is appropriate for all LLMs without per-model tuning.
read the original abstract
Validating autonomous driving systems requires diverse, regulation-compliant test scenarios. In simulation-based testing, scenarios are defined as executable scripts. Yet automatically generating such scripts from regulatory descriptions remains an open challenge, and existing approaches face fundamental trade-offs. Retrieval-assemble methods achieve reasonable compilation rates but lack scalability, whereas retrieval-based full-script generation suffers from low compilation success rates. We present Chat2Scenic, the first iterative retrieval-augmented framework to generate scenario scripts in Domain Specific Language (DSL). Specifically, Chat2Scenic provides a chatbot interface that supports interactive scenario refinement and integrates Retrieval-augmented Generation (RAG) to ground scenario generation in regulatory knowledge and DSL syntax. Furthermore, we propose an open benchmark for scenario generation comprising 123 scenarios from various regulations, including NHTSA and United Nations Vehicle Regulations, as well as other sources. Extensive evaluation with State-of-the-Art (SOTA) Large Language Models (LLMs) demonstrates that Chat2Scenic achieves 76.42% Compilation Success Rate (CSR) and 58.17% Framework Accuracy (FA), outperforming existing methods (Retrieval Assemble with 30.08% CSR, 11.03% FA and Retrieval full script generation with 16.26% CSR, 10.86% FA). To facilitate future research, we release our code as open source at https://github.com/TUM-AVS/chat2scenic.
Figures
Reference graph
Works this paper leans on
-
[1]
A new taxonomy for automated driving: Structuring applications based on their operational design domain, level of automation and automation readiness,
J. Betz, M. Lutwitzi, and S. Peters, “A new taxonomy for automated driving: Structuring applications based on their operational design domain, level of automation and automation readiness,” in2024 IEEE Intelligent Vehicles Symposium (IV), 2024
2024
-
[2]
B ¨aumler and E
M. B ¨aumler and E. Cuvillier,Generating Test Scenarios for Assessing Automated Driving Systems Using Scenario Fusion, ser. Schriftenreihe des Lehrstuhls Kraftfahrzeugtechnik. Cuvillier Verlag, 2024
2024
-
[3]
Critical scenario identification for realistic testing of autonomous driving systems,
Q. Song, K. Tan, P. Runeson, and S. Persson, “Critical scenario identification for realistic testing of autonomous driving systems,” Software Quality Journal, 2023
2023
-
[4]
Target: Automated scenario generation from traffic rules for testing autonomous vehicles,
Y . Deng, J. Yao, Z. Tu, X. Zheng, M. Zhanget al., “Target: Automated scenario generation from traffic rules for testing autonomous vehicles,” arXiv preprint arXiv:2305.06018, 2023
Pith/arXiv arXiv 2023
-
[5]
Chatscene: Knowledge-enabled safety- critical scenario generation for autonomous vehicles,
J. Zhang, C. Xu, and B. Li, “Chatscene: Knowledge-enabled safety- critical scenario generation for autonomous vehicles,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024
2024
-
[6]
Text2scenario: Text- driven scenario generation for autonomous driving test,
X. Cai, X. Bai, Z. Cui, D. Xie, D. Fuet al., “Text2scenario: Text- driven scenario generation for autonomous driving test,”Automotive Innovation, 2026
2026
-
[7]
H. Tian, X. Han, Y . Zhou, G. Wu, A. Guoet al., “Lmm- enhanced safety-critical scenario generation for autonomous driv- ing system testing from non-accident traffic videos,”arXiv preprint arXiv:2406.10857, 2024
Pith/arXiv arXiv 2024
-
[8]
Legend: A top- down approach to scenario generation of autonomous driving systems assisted by large language models,
S. Tang, Z. Zhang, J. Zhou, L. Lei, Y . Zhouet al., “Legend: A top- down approach to scenario generation of autonomous driving systems assisted by large language models,” in39th IEEE/ACM International Conference on Automated Software Engineering, 2024
2024
-
[9]
Scenicnl: generating proba- bilistic scenario programs from natural language,
K. Elmaaroufi, D. Shanker, A. Cismaru, M. Vazquez-Chanlatte, A. Sangiovanni-Vincentelliet al., “Scenicnl: generating proba- bilistic scenario programs from natural language,”arXiv preprint arXiv:2405.03709, 2024
Pith/arXiv arXiv 2024
-
[10]
B.-K. Ruan, H.-T. Tsui, Y .-H. Li, and H.-H. Shuai, “Traffic scene generation from natural language description for autonomous vehicles with large language model,”arXiv preprint arXiv:2409.09575, 2024
arXiv 2024
-
[11]
R. Rubavicius, A. V . Miceli-Barone, A. Lascarides, and S. Ramamoor- thy, “Conversational code generation: a case study of designing a dia- logue system for generating driving scenarios for testing autonomous vehicles,”arXiv preprint arXiv:2410.09829, 2024
Pith/arXiv arXiv 2024
-
[12]
P. Bauerfeind, A. Salarpour, D. Fernandez, P. MohajerAnsari, J. Reschkeet al., “David vs. goliath: A comparative study of different- sized llms for code generation in the domain of automotive scenario generation,”arXiv preprint arXiv:2510.14115, 2025
arXiv 2025
-
[13]
A survey on safety- critical driving scenario generation—a methodological perspective,
W. Ding, C. Xu, M. Arief, H. Lin, B. Liet al., “A survey on safety- critical driving scenario generation—a methodological perspective,” IEEE Transactions on Intelligent Transportation Systems, 2023
2023
-
[14]
1001 ways of scenario generation for testing of self-driving cars: A survey,
B. Sch ¨utt, J. Ransiek, T. Braun, and E. Sax, “1001 ways of scenario generation for testing of self-driving cars: A survey,” in2023 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2023
2023
-
[15]
Foundation models in autonomous driving: A survey on scenario generation and scenario analysis,
Y . Gao, M. Piccinini, Y . Zhang, D. Wang, K. Molleret al., “Foundation models in autonomous driving: A survey on scenario generation and scenario analysis,”IEEE Open Journal of Intelligent Transportation Systems, 2026
2026
-
[16]
Carla: An open urban driving simulator,
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “Carla: An open urban driving simulator,” inConference on robot learning. PMLR, 2017
2017
-
[17]
Microscopic traffic simulation using sumo,
P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y .-P. Fl¨otter¨od et al., “Microscopic traffic simulation using sumo,” inThe 21st IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018
2018
-
[18]
Chatsumo: Large language model for automating traffic scenario generation in simulation of urban mobility,
S. Li, T. Azfar, and R. Ke, “Chatsumo: Large language model for automating traffic scenario generation in simulation of urban mobility,” IEEE Transactions on Intelligent Vehicles, 2024
2024
-
[19]
Multimodal large language model driven scenario testing for autonomous vehicles,
Q. Lu, X. Wang, Y . Jiang, G. Zhao, M. Maet al., “Multimodal large language model driven scenario testing for autonomous vehicles,” arXiv preprint arXiv:2409.06450, 2024
Pith/arXiv arXiv 2024
-
[20]
Generating traffic scenarios via in-context learning to learn better motion planner,
A. Aiersilan, “Generating traffic scenarios via in-context learning to learn better motion planner,” inAAAI Conference on Artificial Intelligence, 2025
2025
-
[21]
Sovar: Build generalizable scenarios from accident reports for autonomous driving testing,
A. Guo, Y . Zhou, H. Tian, C. Fang, Y . Sunet al., “Sovar: Build generalizable scenarios from accident reports for autonomous driving testing,” inProceedings of the 39th IEEE/ACM International Confer- ence on Automated Software Engineering, 2024
2024
-
[22]
Q. Shi, Q. Meng, H. Cheng, Q. Xu, and J. Wang, “Linguasim: Interactive multi-vehicle testing scenario generation via natural lan- guage instruction based on large language models,”arXiv preprint arXiv:2510.08046, 2025
arXiv 2025
-
[23]
Realistic corner case generation for autonomous vehicles with multimodal large language model,
Q. Lu, M. Ma, X. Dai, X. Wang, and S. Feng, “Realistic corner case generation for autonomous vehicles with multimodal large language model,”arXiv preprint arXiv:2412.00243, 2024
Pith/arXiv arXiv 2024
-
[24]
Automatic generation method for autonomous driving simulation scenarios based on large language model,
X. Zhou, Y . Huang, J. Zhang, J. Shao, D. Panet al., “Automatic generation method for autonomous driving simulation scenarios based on large language model,” inInternational Conference on Artificial Intelligence and Autonomous Transportation. Springer, 2024
2024
-
[25]
Scenic: a language for scenario specification and scene generation,
D. J. Fremont, T. Dreossi, S. Ghosh, X. Yue, A. L. Sangiovanni- Vincentelliet al., “Scenic: a language for scenario specification and scene generation,” in40th ACM SIGPLAN conference on programming language design and implementation, 2019
2019
-
[26]
From words to collisions: Llm-guided evaluation and adversarial generation of safety-critical driving scenarios,
Y . Gao, M. Piccinini, K. Moller, A. Alanwar, and J. Betz, “From words to collisions: Llm-guided evaluation and adversarial generation of safety-critical driving scenarios,” in2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2025
2025
-
[27]
6-layer model for a structured description and categorization of urban traffic and environment,
M. Scholtes, L. Westhofen, L. R. Turner, K. Lotto, M. Schuldeset al., “6-layer model for a structured description and categorization of urban traffic and environment,”IEEE Access, 2021
2021
-
[28]
Nurisk: A visual question answering dataset for agent-level risk assessment in autonomous driving,
Y . Gao, M. Piccinini, R. Brusnicki, Y . Zhang, and J. Betz, “Nurisk: A visual question answering dataset for agent-level risk assessment in autonomous driving,” in2026 IEEE International Conference on Robotics and Automation (ICRA), 2026
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.