{"id":"bc4cc648-eb23-4065-af8e-07d58bba6bc6","arxiv_id":"2412.18086","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"AutoSceneGen uses LLM in-context learning to generate CARLA traffic scenarios and trains trajectory predictors on the synthetic data, but its claimed improvements are inconsistent and unsupported.","lead":"AutoSceneGen turns typed descriptions of traffic scenes into scripts for the CARLA simulator using a large language model, then uses the generated synthetic trajectories to train prediction networks. The paper reports mixed results, with some benchmarks improving and others getting worse, and provides no code or data to reproduce the claims.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is confounded: Table 2 pits 17,919 synthetic frames against 94 real training examples, and the paper never fixes the test distribution; Table 3's NGSIM result directly contradicts the abstract.","rationale":"After reading the paper in good faith, I agree with the reader's rejection. The framework itself is a plausible demonstration of LLM-driven scenario generation in CARLA, and the qualitative scenario generation pipeline is coherent. However, the paper's central value claim is empirical: synthetic data improves motion planners. That claim is load-bearing and unsupported. The most serious problem is not merely that CARLA data differs from real data—that would be acceptable if the sim-to-real gap were measured and the comparison were fair—but that the paper's only head-to-head table (Table 2) is confounded by training-set size and by an ambiguous test distribution. A 17,919-frame synthetic set compared with a 94-example real set cannot isolate the effect of the generation method. Table 3 is the paper's own evidence that synthetic-only data can be far worse than real data (NGSIM ADE 7.98 vs 0.88) and that combining can hurt (VIRAT/ActEV). These internal contradictions make the abstract's significant outperform claim untenable. A matched-size, fixed-test-set re-evaluation would settle the question. The reader's weakest assumption about sim-to-real generalization is related and important, but I would sharpen it further: the comparison is not even well-defined as reported because the test distribution of the Ours row is not specified. This is why I mark agreement as partial rather than full. I would not alter the reader's REJECT verdict; no change needed.","tokens_in":12519,"tokens_out":5193,"duration_ms":50769,"concrete_test":"Re-run the Table 2 comparison with a fixed evaluation protocol: use the ApolloScapes test split for all rows. Train TrafficPredict on (a) ApolloScapes train, (b) a random subset of AutoSceneGen matched to the ApolloScapes training set in number of clips and total frames, and (c) their union. Report mean +/- std over at least 5 random seeds with a pre-specified epoch-selection rule. If row (b) does not beat row (a) beyond the noise, or if row (c) is not better than (a), the headline claim is unsupported. If the paper's 'Ours' row was evaluated on AutoSceneGen's own split, re-run row (b) on the real ApolloScapes test set; a large performance drop would confirm the sim-to-real gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's claim that training with AutoSceneGen data 'significantly outperform[s] those trained solely on real-world data' rests entirely on Table 2, but that table does not provide a controlled comparison. The AutoSceneGen training set is 17,919 frames while the ApolloScapes training set contains only 94 examples, so even if both rows are evaluated on the same ApolloScapes test split, the improvement could be an effect of training-set volume or of CARLA's simpler distribution rather than of the generation method. Worse, the evaluation protocol is ambiguous: the text says 'we tested exclusively with our own data,' but also says 'replacing only the train set' of the target dataset. If the 'Ours' row is tested on AutoSceneGen data, the result is a synthetic-to-synthetic evaluation and says nothing about generalization to real driving. If it is tested on ApolloScapes, the paper still provides no error bars, seeds, or statistical tests, and Table 3 undermines the claim: synthetic-only Pishgu on NGSIM has ADE 7.98 vs 0.88 for real data, and adding synthetic data to VIRAT/ActEV increases ADE from 14.11 to 15.32. The central claim therefore lacks a valid experimental foundation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AutoSceneGen, a framework that uses LLM in-context learning to translate text descriptions of traffic scenarios into simulator-executable scripts, and then uses CARLA to collect synthetic trajectory data. The authors train existing trajectory predictors (TrafficPredict, TraPHic, Pishgu) on synthetic data, real-world data, and combinations thereof, reporting ADE/FDE/TAE on ApolloScapes, NGSIM, ETH/UCY, and VIRAT/ActEV. The abstract claims that motion planners trained on AutoSceneGen data 'significantly outperform those trained solely on real-world data.' The central evidence is Tables 2 and 3, with Table 2 showing large improvements on ApolloScapes and Table 3 showing mixed, often worse, results on other benchmarks.","tokens_in":12745,"tokens_out":4676,"duration_ms":43313,"significance":"If the central claim were supported, the framework would offer a cost-effective complement to real-world data collection for safety-critical scenario generation. The ICL-based pipeline is clearly described, modular, and released with source code, and the filter/validator design for simulator compatibility is a practical strength. However, the experimental evaluation does not establish the claimed benefit: the headline comparison in Table 2 is confounded by training-set size and ambiguous test protocol, and Table 3 directly contradicts the abstract for NGSIM and VIRAT/ActEV. The potential significance is therefore not realized in the present submission.","major_comments":[{"comment":"The abstract's claim that 'motion planners trained with our data significantly outperform those trained solely on real-world data' is directly contradicted by Table 3. On NGSIM, training only on AutoSceneGen data yields ADE 7.98 and FDE 15.43, versus 0.88 and 1.96 for NGSIM real data, roughly a nine-fold degradation. On VIRAT/ActEV, synthetic-only training worsens ADE to 16.05 from 14.11, and combining synthetic with real data increases ADE to 15.32, again worse than real-only. The text acknowledges the NGSIM replacement is worse but still concludes that generated data 'outperforms in most cases'; this is not supported by the table. The paper must either remove the broad claim or provide evidence beyond the single ApolloScapes setting.","section":"Evaluations, Table 3"},{"comment":"The Table 2 comparison is not controlled. The caption states that the ApolloScapes official training set contains 94 examples, while AutoSceneGen generated 17,919 frames. Training on 17,919 synthetic frames versus 94 real frames confounds the effect of the generation method with training-set volume and distribution shift. Additionally, the text says both 'we tested exclusively with our own data' and 'replacing only the train set'; it is never specified whether the 'Ours' row is evaluated on AutoSceneGen test data or on the ApolloScapes test split. If the former, this is a synthetic-to-synthetic evaluation with no sim-to-real transfer; if the latter, a matched training-size baseline and explicit test protocol are required.","section":"Evaluations, Table 2"},{"comment":"No error bars, random seeds, or statistical significance tests are reported for any metric. The conclusions rest on a single run per condition, despite the known variance in trajectory prediction training and the relatively small synthetic corpus (41 executed scenarios from 125 configurations). Furthermore, the paper does not report the success rate of the ICL-to-script generation, the number of invalid configurations, or the filtering/validation failure rates, so the claimed 'effectiveness of our data generation method' is not directly measured. These omissions make it impossible to determine whether any improvement is attributable to AutoSceneGen rather than to the simulator's simpler traffic distribution.","section":"Evaluations, Tables 2 and 3"}],"minor_comments":[{"comment":"The model name is spelled 'Pihgu' in Table 3 but 'Pishgu' in the text and references; please unify.","section":"Table 3"},{"comment":"The row labels 'A.S. train-set + Ours' and 'Ours' are ambiguous about whether evaluation is on ApolloScapes test data or on AutoSceneGen data; clarify in the caption and text.","section":"Table 2"},{"comment":"The AutoSceneGen column reports frames (×10^3) as 0.05, implying 50 frames, which is inconsistent with the 17,919 examples and the 1,848k/15,400k frame counts mentioned in the Evaluations section; reconcile these numbers.","section":"Table 1"},{"comment":"The citation 'TARGET (Deng et al. 2023)' has no corresponding entry in the References list.","section":"Related Work"},{"comment":"The text refers to 'ChatGPT-3'; the correct model name is GPT-3.","section":"In-Context Learning"},{"comment":"The caption mentions 'all metrics' but only explicitly describes FDE-vehicle and ADE-vehicle; please label each subplot and describe what is shown.","section":"Figure 3"},{"comment":"The limitations section does not discuss the fact that synthetic-only training failed on NGSIM or that combining synthetic data worsened VIRAT/ActEV results; this should be acknowledged given the paper's central claim.","section":"Limitations"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim is contradicted by its own Table 3, and the Table 2 comparison is confounded by training-set size and ambiguous evaluation protocol. The paper would need a substantially redesigned evaluation or a major reframing of its claims to be salvageable, so I recommend rejection. The authors' framework design may have merit, but the current evidence does not support the stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the short version: the pipeline is fine, but the paper's central claim is not supported by its own numbers. The LLM-with-in-context-learning-to-CARLA idea is already in the works this paper cites (Li et al. 2024b, Deng et al. 2023); the genuine additions are a filter and validator for simulator API compatibility. That is legitimate, useful engineering, and the architecture is described clearly in Algorithms 1 and 2. The limitations section is also honest about simulator and LLM dependency, which I appreciate. Table 1 gives a useful quick comparison of dataset acquisition costs.\n\nThe problem is the evaluation. The abstract says training on AutoSceneGen data significantly outperforms real-data-only training, but Table 3 shows synthetic-only NGSIM ADE of 7.98 versus 0.88 on real NGSIM, and mixing synthetic data into VIRAT/ActEV makes errors worse (15.32 vs 14.11). The text acknowledges the VIRAT point and claims a 'tendency to outperform,' but the numbers do not show that tendency on those benchmarks. Table 2, the main evidence for the abstract's claim, is not a controlled comparison: AutoSceneGen has 17,919 frames while ApolloScapes has only 94 training examples. If the 'Ours' row is evaluated on AutoSceneGen data, that is synthetic-to-synthetic; if it is evaluated on ApolloScapes, the improvement could come from training-set volume or from CARLA's simpler distribution. The text is ambiguous about which protocol was used, and there are no error bars, seeds, or statistical tests anywhere in the paper. The epoch selections in Figure 3 also look post hoc. On top of that, the title says motion planning but the experiments are trajectory prediction, which is a different task.\n\nDespite the code URL, no code, data, or prompts are actually included in the submission, so nothing is reproducible. The filter/validator idea could be a useful data point for someone building an LLM-based scenario generation pipeline, but the paper's conclusion should not be cited. If the authors release the code/data and run controlled experiments with matched training sizes, multiple seeds, fixed test sets, and statistical tests, it could become a reasonable workshop or short-paper contribution. As submitted, this is a desk reject: the central claim is contradicted by the paper's own table, and the remaining evidence is too weak to justify referee time.","headline":"The framework engineering is reasonable and clearly described, but the central empirical claim is contradicted by the paper's own Table 3, and Table 2 is a confounded comparison.","tokens_in":13298,"tokens_out":3807,"would_cite":false,"duration_ms":38080,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes AutoSceneGen, a pipeline that uses an LLM's in-context learning to turn text descriptions into executable simulator scripts, producing rare safety-critical traffic scenarios cheaply and improving motion-planning…","keywords":["traffic scenario generation","in-context learning","large language models","motion planning","autonomous driving","synthetic training data","trajectory prediction","CARLA simulator"],"falsifier":"Train the same three models on AutoSceneGen data alone and on the corresponding real training set, then evaluate on the real test split: if the synthetic-only model's ADE/FDE is not close to the real-only model's, or if adding synthetic frames to real training raises error on any benchmark, the central claim fails. A direct check is to compare the empirical distributions of speed, acceleration, lane offset, and interaction frequency between AutoSceneGen frames and ApolloScapes or NGSIM frames.","tokens_in":12287,"feed_emoji":"🚗","tokens_out":9664,"duration_ms":84468,"temperature":0.7,"pith_summary":"The paper sets out to show that a large language model, given a few examples embedded in its prompt, can turn short text descriptions of traffic situations into executable simulator scripts, generating abundant safety-critical driving scenarios without manual annotation. This matters because curated real-world driving datasets are expensive to collect and rarely cover corner cases, leaving motion planners untested against uncommon events. The authors report that trajectory-prediction models trained on the resulting CARLA data, alone or mixed with real data, achieve lower average and final displacement errors than models trained on real data alone in several comparisons. The proposed pipeline, AutoSceneGen, combines a description filter, an API validator, and in-context learning exemplars to keep LLM output compatible with the simulator.","feed_headline":"LLM-scripted scenarios outperform curated data for training planners","feed_subtitle":"AutoSceneGen turns text prompts into CARLA scripts, generating rare safety-critical scenarios for training motion planners.","key_machinery":"The central object is the scenario script, a text-and-code representation of traffic logic that the CARLA simulator can execute. The machinery is AutoSceneGen's pipeline: a filter that replaces simulator-incompatible words, in-context learning exemplars that teach the LLM the simulator's API, a validator that repairs or removes unsupported API calls, and the CARLA simulator itself, which turns scripts into simulated trajectories. In-context learning is the ability of a language model to follow a new task from examples placed in the prompt at inference time; this is what lets a general LLM emit executable simulator code for rare scenarios without training or fine-tuning a generative model.","core_discovery":"The central claim is that scenario logic can be captured as a script, and an LLM with in-context learning can generate that script from a natural-language description well enough for a physics-based simulator to produce diverse, safety-relevant traffic. Using GPT-4, the authors generated 125 scenario descriptions and collected thousands of trajectory frames, including vehicles and pedestrians, across CARLA maps. They then trained three trajectory-prediction models on the synthetic data, on real benchmarks such as ApolloScapes and NGSIM, and on both combined. In the comparisons they report, the synthetic data, especially when combined with real data, lowers displacement errors, which they take as evidence that LLM-generated scenarios are useful for training better motion planners.","pith_inferences":["The same pipeline could be pointed inward: feeding dashcam footage through a vision-language model to produce the scenario description would close the loop from real-world observations to synthetic re-enactment, which the paper does not implement.","A distributional audit of generated versus real trajectories, comparing speed, acceleration, and lane-offset histograms, would make the sim-to-real transfer claim directly testable rather than inferred from final prediction errors.","Because the framework separates description from simulator, its diversity ceiling is set partly by the simulator's built-in stochasticity; sweeping one description across maps, weathers, and seed traffic would isolate how much diversity comes from the LLM script versus the simulator.","The reported results suggest the generated data acts most safely as a complement to real data, so an obvious extension is a per-scenario-type rule for deciding when to add synthetic frames and when to keep real ones."],"forward_implications":["Trajectory-prediction models can be trained on data generated from text descriptions, reducing reliance on hand-curated real-world datasets for routine scenarios.","Rare and safety-critical events, such as wrong-way drivers or vehicles with open doors, can be produced at scale, filling gaps in long-tail scenario coverage.","Mixing generated data with real data can lower prediction errors relative to real data alone, indicating that the synthetic distribution is complementary rather than redundant.","Because scenario descriptions are user-specified, dataset heterogeneity can be controlled at generation time without post-hoc annotation.","The framework is modular: swapping the LLM or the simulator changes what scenarios can be produced without retraining the generation model."],"supporting_citations":[{"why":"Supplies CARLA, the simulator that executes the generated scripts and produces all synthetic trajectory data.","marker":"Dosovitskiy et al. 2017"},{"why":"Provides GPT-4, the large language model whose in-context learning produces the scenario configuration scripts.","marker":"Achiam et al. 2023"},{"why":"Establishes that language models can learn tasks from a few prompt examples, the basis for the in-context learning step.","marker":"Mann et al. 2020"},{"why":"Shows which components of demonstrations drive in-context learning, motivating the construction of exemplars for the simulator API.","marker":"Min et al. 2022"},{"why":"Supplies TrafficPredict and the ApolloScapes baseline that AutoSceneGen data replaces and augments.","marker":"Ma et al. 2019"},{"why":"Supplies TraPHic, the second trajectory-prediction method used for evaluating the generated data.","marker":"Chandra et al. 2019"},{"why":"Supplies Pishgu, the third prediction model used for the NGSIM, ETH/UCY, and VIRAT/ActEV comparisons.","marker":"Alinezhad Noghre et al. 2023"},{"why":"Provides the ApolloScapes real-world dataset that serves as the primary comparison and augmentation target.","marker":"Huang et al. 2018"},{"why":"Provides the NGSIM real-world vehicle trajectory dataset used in the replacement and augmentation experiments of Table 3.","marker":"Kovvali, Alexiadis, and Zhang PE 2007"}],"fun_headline_variants":["Text prompts script rare traffic scenes for better planners","LLM-crafted scenarios sharpen motion planner accuracy","AutoSceneGen: LLM turns text into driving test cases","In-context learning generates diverse traffic for training","Synthetic scenarios from text improve planner performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that CARLA trajectories written from LLM scripts are similar enough to real road traffic that training on them transfers to real test sets; if the synthetic distribution is simpler or shifted, lower prediction errors on synthetic-heavy training are an artifact, not a real improvement.","fun_headline_variants_meta":{"raw":{"variants":["Text prompts script rare traffic scenes for better planners","LLM-crafted scenarios sharpen motion planner accuracy","AutoSceneGen: LLM turns text into driving test cases","In-context learning generates diverse traffic for training","Synthetic scenarios from text improve planner performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1403,"prompt_tokens":946,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":562,"tokens_out":457,"duration_ms":5180,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:02:17.909255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same three models on AutoSceneGen data alone and on the corresponding real training set, then evaluate on the real test split: if the synthetic-only model's ADE/FDE is not close to the real-only model's, or if adding synthetic frames to real training raises error on any benchmark, the central claim fails. A direct check is to compare the empirical distributions of speed, acceleration, lane offset, and interaction frequency between AutoSceneGen frames and ApolloScapes or NGSIM frames.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies TraPHic, the second trajectory-prediction method used for evaluating the generated data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Pishgu, the third prediction model used for the NGSIM, ETH/UCY, and VIRAT/ActEV comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ApolloScapes real-world dataset that serves as the primary comparison and augmentation target."},{"cited_title":"G.; Alexiadis, V.; and Zhang PE, L","cited_arxiv_id":null,"evidence_quote":"Provides the NGSIM real-world vehicle trajectory dataset used in the replacement and augmentation experiments of Table 3."}],"review_version":1}