{"id":"7213b43c-9054-43a2-9ac3-61132e6f89a1","arxiv_id":"2607.14387","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Chat2Scenic generates executable Scenic driving-scenario scripts from regulation-style text with 76.4% compilation success, using iterative component-wise generation with retrieval-augmented prompting.","lead":"Chat2Scenic turns natural-language traffic descriptions into executable Scenic simulation scripts using an iterative, retrieval-augmented LLM pipeline. On a new 123-scenario benchmark it reports a 76% compilation success rate, roughly 2.5x the best prior method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RAG code database may contain snippets for benchmark scenarios (e.g., CARLA Leaderboard), inflating the headline CSR/FA; no de-duplication is reported.","rationale":"The reader identified configuration selection on the same benchmark as the weakest assumption. I agree that is a real limitation, but the more specific and potentially more damaging issue is the lack of documented separation between the RAG code database and the benchmark. Since the code database is built from official Scenic sources and the benchmark contains CARLA Leaderboard scenarios, there is a concrete mechanism by which the retrieval step could return near-complete solutions. This can be settled by a cheap audit of the released repository, so I would keep the reader's CONDITIONAL verdict rather than reject. The human-scoring variance is secondary because CSR is an objective executable check and the largest gap over baselines is in CSR.","tokens_in":11301,"tokens_out":7551,"duration_ms":71071,"concrete_test":"Inspect the released code database against the benchmark specification. For each of the 123 scenarios, run the same embedding-based top-3 code retrieval and check whether any returned snippet originates from a scenario source file (e.g., a CARLA Leaderboard scenario) that is also in the benchmark, or whether the scenario description is semantically near-identical to a database entry. Then delete all overlapping entries and rerun Chat2Scenic (C11) on the complete benchmark. If CSR/FA drops materially (e.g., >5 points), the headline numbers are contaminated by leakage; if no overlap is found, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Chat2Scenic achieves 76.42% CSR and 58.17% FA on a 123-scenario benchmark and thereby outperforms ChatScene and NL2Scenic. For this claim to be a valid comparison, the RAG code database used by Chat2Scenic must not contain the answers for the benchmark scenarios. Section II-B.1 says the code snippet database is built from official Scenic sources and decomposed into component-level units; the benchmark (Section III-A.1) deliberately includes 24 CARLA Leaderboard scenarios. The paper never states that the benchmark scenarios or their source files were excluded from the database. If any of the 123 scenarios (especially the CARLA Leaderboard subset) appear in the official Scenic examples used to build the database, the code retriever can return code that is effectively the ground-truth solution for those test items, inflating CSR/FA relative to genuinely unseen regulation descriptions. This is not merely a hypothetical: the same CARLA Leaderboard sources are also used to build ChatScene's database, and the paper's own comparison shows retrieval-assemble is strongest on those scenarios. The final configuration C11 was also selected on the same 123 scenarios (Table III) before being reported in Table IV, so the headline is at minimum a best-on-benchmark number. The database-overlap issue is the more decisive and checkable threat.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Chat2Scenic, an iterative retrieval-augmented generation framework that converts natural-language regulation-style scenario descriptions into executable Scenic programs. The framework parses a description into logical components (global configuration, spatial relations, ego, objects, restrictions), retrieves code snippets and documentation through a dual RAG architecture, and generates components iteratively using contextual prompting, chain-of-thought, few-shot in-context learning, and retrieval-augmented in-context learning. The authors introduce a 123-scenario benchmark drawn from CARLA Leaderboard, NHTSA, and UN regulations, with metrics comprising CSR, response time, token usage, and human-scored layer alignment, plus FA = CSR × SQ. Using Gemini-3-Flash with configuration C11, they report CSR 76.42%, SQ 76.12%, FA 58.17%, outperforming ChatScene (30.08% CSR) and NL2Scenic (16.26% CSR) on the same benchmark.","tokens_in":11588,"tokens_out":4820,"duration_ms":47196,"significance":"If the reported results are unbiased, the contribution is significant: it addresses a real bottleneck in DSL-based scenario generation, introduces a reusable regulation-grounded benchmark, and provides an open-source implementation. Strengths include the public code release, the use of simulator compilation as an objective filter for CSR, and the component-wise iterative design, which is well motivated by the failure modes of full-script generation. The ablation over prompting techniques and the multi-backbone evaluation are also useful. However, the empirical claim currently depends on unaddressed evaluation-hygiene issues: possible overlap between the retrieval database and benchmark scenarios, selection of the best configuration on the same benchmark, and unmeasured human-evaluator variance. These issues are fixable, but the headline numbers cannot be taken at face value until they are resolved.","major_comments":[{"comment":"The RAG code database is built from \"official Scenic sources\" (§II-B.1), while the benchmark deliberately includes 24 CARLA Leaderboard scenarios (§III-A.1). The paper never states that the source files or scenario descriptions corresponding to the 123 benchmark items were excluded from the database. Since CARLA Leaderboard scenarios are official Scenic examples and §III-B.2 notes that ChatScene's database is built from CARLA Leaderboard scenarios, the retriever could return code that is effectively the ground-truth solution for a subset of test items, inflating both CSR and SQ. Please report a decontamination analysis, e.g., remove any snippet whose source scenario appears in the benchmark and re-run, or show retrieval outputs for benchmark queries and verify that no ground-truth snippet is returned.","section":"§II-B.1 and §III-A.1"},{"comment":"Configuration C11 was selected as the best of 12 ablation configurations evaluated on the complete 123-scenario benchmark (Table III) and then used as the final result in Table IV. The headline 76.42% CSR and 58.17% FA are therefore selected-on-test estimates, not unbiased predictive estimates for new regulation descriptions. This also makes the comparison to ChatScene and NL2Scenic unfair to the extent that those baselines were not configuration-selected on this benchmark. Please either split the benchmark into development and held-out test sets, use nested cross-validation, or at minimum report the distribution over configurations and the selection rule, together with confidence intervals.","section":"§III-B.1, Table III and Table IV"},{"comment":"FA is defined as CSR × SQ, where SQ is the average of human layer-alignment scores. The manuscript reports no detail on the human evaluation: number of evaluators, whether they were blind to the framework or method, whether the same evaluator scored all systems, or inter-rater reliability. Since the headline FA depends directly on these subjective scores, unmeasured evaluator variance could change the comparison. Please provide the evaluation protocol and agreement statistics (e.g., Cohen's kappa or per-layer variance), and ideally report SQ with confidence intervals.","section":"§III-A.3"}],"minor_comments":[{"comment":"The table header for RAG-ICL and the distinction between the 'Assemble' and 'Generation' rows are difficult to follow; consider simplifying the column layout or adding a footnote.","section":"Table I"},{"comment":"The contribution text says '120+ scenario descriptions' while the benchmark has 123; please be consistent.","section":"§I-C"},{"comment":"SettingsDetector is described as using 'rule-based keyword matching patterns and example-based reasoning,' but the rules and examples are not provided. This limits reproducibility; please include them in an appendix or supplementary material.","section":"§II-C.1"},{"comment":"For configurations with very low CSR (e.g., C2, C3), the layer-alignment averages are based on a handful of compiled scenarios and contain many dashes. Please clarify how SQ is computed in these cases and consider reporting the number of compiled scenarios.","section":"Table III"},{"comment":"Response time and token usage are reported as averages without variance. Adding standard deviations or confidence intervals would help assess stability across scenarios.","section":"§III-A.3"},{"comment":"The claim 'first iterative retrieval-augmented framework' is strong given that prior work such as Talk2Traffic and Rubavicius already combine interaction and retrieval. Please soften the claim or clarify the specific novelty.","section":"Abstract and §I-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is timely and the framework is well motivated. The main risk is evaluation contamination: the possible overlap between the retrieval database and the benchmark, and the selection of the best configuration on the test benchmark, could materially lower the reported numbers if corrected. I see no reason to doubt the authors' good faith; these are standard evaluation-hardening concerns. If the authors can provide a decontamination analysis and a proper selection protocol, the paper would be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth engaging with, but the abstract oversells the numbers. The core idea — iterative, dependency-ordered, component-wise generation of Scenic code with a dual code-plus-documentation retriever and interactive refinement — is genuinely new and clearly better than the retrieval-assemble and full-script baselines they compare against. The 123-scenario regulation-grounded benchmark (NHTSA, UN R152/R157/R171, CARLA Leaderboard) is a concrete artifact, the human layer-alignment scoring is a reasonable attempt at semantic evaluation, and the code and benchmark are open source. That is real, reproducible evidence.\n\nThe soft spots are, in order of severity. First, the ablation table tests 12 configurations, and the reported final result uses the best one (C11) selected on the same 123 scenarios. That makes 76.42% CSR and 58.17% FA a best-on-benchmark estimate, not a predictive one. A held-out split or cross-validation would fix this, and without it the comparison to ChatScene and NL2Scenic is not apples-to-apples. Second, the stress-test about RAG leakage is legitimate. The code snippet database is built from official Scenic sources, and the benchmark includes 24 CARLA Leaderboard scenarios. The paper never states that those scenarios or their source files were excluded from the retrieval corpus. If the code retriever can return ground-truth snippets for any benchmark item, the headline CSR/FA are inflated. This is checkable by inspecting the released database, but the burden is on the authors to rule it out. Third, the closest prior conversational RAG method (Rubavicius et al.) is described in the related work but omitted from the baseline table. Since the paper claims \"first iterative RAG framework,\" that omission is conspicuous. Fourth, human-evaluator reliability is undocumented: how many annotators, what agreement? With FA = CSR × SQ, evaluator variance directly affects the headline metric.\n\nThe central hypothesis — iterative component-wise generation beats retrieval-assemble and full-script generation — probably holds. The magnitude of the advantage and the specific numbers are uncertain. This deserves a serious referee, and the issues are fixable with a cleaner evaluation and a de-duplication statement. I'd send it to peer review rather than desk reject, and I'd bring it to a reading group to discuss evaluation methodology in LLM-generated DSLs.","headline":"Useful framework with a genuinely novel iterative component-wise RAG design, but the headline CSR/FA are best-on-benchmark numbers and a data-leakage check is needed before trusting them.","tokens_in":12106,"tokens_out":2532,"would_cite":true,"duration_ms":25658,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chat2Scenic turns regulatory text into executable driving scenarios at a 76.42% compilation success rate, roughly 2.5x the best prior method, by generating scripts component by component with retrieval-augmented prompting.","keywords":["autonomous driving","scenario generation","retrieval-augmented generation","large language models","Scenic","component-wise generation","compilation success rate","regulatory test scenarios"],"falsifier":"Run the final C11 configuration and the same human scoring protocol on a fresh set of regulatory descriptions that were not part of the 123-scenario benchmark. If the compilation success rate falls substantially below 76.4% (for example, below 50%), the headline numbers are at least partly artifacts of benchmark-driven configuration selection.","tokens_in":11146,"feed_emoji":"🚗","tokens_out":8056,"duration_ms":68700,"temperature":0.7,"pith_summary":"The paper proposes an iterative, retrieval-augmented framework, Chat2Scenic, to generate executable driving-scenario scripts from natural-language regulatory descriptions. The central claim is that breaking each scenario into logical components (global settings, spatial relations, ego behavior, objects, restrictions) and generating them one at a time, with retrieval of code snippets and documentation at each step, makes the generation reliable enough for simulation-based testing. On a new benchmark of 123 scenarios drawn from U.S. and international vehicle-safety regulations and public leaderboard tasks, the best configuration achieves a 76.42% compilation success rate and 58.17% framework accuracy, compared with 30.08% and 16.26% CSR for prior retrieval-assemble and full-script generation methods. If this result is robust, it would let engineers turn regulatory text into simulation-ready tests with far less manual authoring.","feed_headline":"76% of text-described driving scenarios now compile to simulation code","feed_subtitle":"A component-wise RAG pipeline beats whole-script and snippet-assembly methods on regulation-style scenarios.","key_machinery":"The load-bearing mechanism is a logical structure schema S = {G, R, E, O, T} that decomposes any scenario description into global configuration (map, weather, vehicles) and scenario components (spatial relations, ego behavior, objects, restrictions). The generation module produces code component by component in a fixed dependency order, passing each generated component's code into the prompt for the next one, so compatibility is maintained incrementally. A dual-retriever RAG module fetches the most relevant code snippets (by semantic similarity) and documentation chunks (by hybrid BM25 plus embedding search fused with Reciprocal Rank Fusion) for each component description. These retrieved co","core_discovery":"The paper's central discovery, on its own terms, is that component-wise iterative generation, combined with retrieval-augmented in-context learning, resolves the trade-off between compilation reliability and generalizability in DSL-based scenario generation. Rather than assembling pre-existing snippets (which compiles but does not generalize) or generating a full script at once (which generalizes but rarely compiles), the framework parses a query into a logical structure, generates each component in dependency order, and feeds the accumulated code back as context for later components. A dual retriever supplies semantically similar code snippets and documentation chunks to ground each generat","pith_inferences":["The same component-wise generation idea likely transfers to other scenario DSLs (e.g., OpenScenario) and other simulators, so the contribution may generalize beyond the specific language evaluated here.","The paper selects the best of 12 ablation configurations on the same benchmark it uses for the final comparison; an independent holdout set would clarify whether the 76.42% CSR reflects true generalizability or benchmark-specific tuning.","The human layer-scoring protocol is expensive and subjective; a future automated metric (e.g., a learned verifier that checks alignment against the description) could make the benchmark cheaper and more reproducible."],"forward_implications":["Regulatory descriptions that previously required manual translation into simulation code can be converted to executable scenarios with a majority success rate, reducing the cost of building diverse test suites.","The component-wise structure means a single failed component can be regenerated or corrected without discarding the whole scenario, easing debugging and human-in-the-loop refinement.","The proposed benchmark (123 scenarios) and the layer-based human scoring protocol give the research community a shared, repeatable way to compare scenario-generation methods.","Because the framework relies on prompting rather than fine-tuning, it can be applied to different LLMs, but the large gap between proprietary and open models suggests that model capability is currently the binding constraint."],"fun_headline_variants":["Iterative RAG lifts driving scenario script compile rate to 76%","Chat2Scenic: RAG pipeline doubles DSL scenario compilation success","Component-wise RAG beats snippet and full-script methods for driving tests","New benchmark: 123 regulation-based scenarios for autonomous driving","RAG with iterative refinement solves DSL scenario generation bottleneck"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported 76.42% compilation success and 58.17% framework accuracy come from the single best configuration (C11) chosen among 12 ablations on the same 123-scenario benchmark it is then evaluated against; the load-bearing premise is that this selection does not materially overfit the benchmark and that the human layer-scoring is a stable measure of quality.","fun_headline_variants_meta":{"raw":{"variants":["Iterative RAG lifts driving scenario script compile rate to 76%","Chat2Scenic: RAG pipeline doubles DSL scenario compilation success","Component-wise RAG beats snippet and full-script methods for driving tests","New benchmark: 123 regulation-based scenarios for autonomous driving","RAG with iterative refinement solves DSL scenario generation bottleneck"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000324,"raw_usage":{"total_tokens":1674,"prompt_tokens":781,"completion_tokens":893,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":805}},"tokens_in":525,"tokens_out":893,"duration_ms":7060,"temperature":1.0,"reasoning_tokens":805,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T02:12:52.747530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the final C11 configuration and the same human scoring protocol on a fresh set of regulatory descriptions that were not part of the 123-scenario benchmark. If the compilation success rate falls substantially below 76.4% (for example, below 50%), the headline numbers are at least partly artifacts of benchmark-driven configuration selection.","supporting_citations":[],"review_version":1}