{"id":"f37c15e0-195b-4350-afdf-66176ac30f46","arxiv_id":"2505.11247","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A system that turns natural-language queries into guidance losses for latent diffusion models, generating controllable adversarial driving scenarios with a 40.75% ego-collision rate on nuScenes.","lead":"LD-Scene combines large language models with latent diffusion models so that a user can describe an adversarial driving situation in plain English and get a realistic simulation where a vehicle collides with the ego car. The method reports higher collision rates than prior adversarial scenario generators on the nuScenes dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-of-10 selection by the LLM-generated guidance loss makes the 40.75% Adv-Ego collision rate an artifact of the evaluation protocol unless baselines receive the same selection procedure.","rationale":"The reader's weakest_assumption identifies the same issue, and I agree it is the central uncertainty. The paper otherwise has reasonable independent support: the framework is a clean integration of a frozen Strive VAE with a latent U-Net, LLM guidance via CoT code generation plus debugger, and ablations showing each component contributes. The debugger ablation (success rate 69.4% to 95.0%) is concrete and plausibly useful, though it measures code execution, not scenario quality. The controllability studies show monotone TTC/acceleration trends, which is encouraging. However, none of these demonstrate that the headline 40.75% collision rate is a property of the underlying generative model rather than the best-of-10 guidance-loss selection. There is no code release, no error bars, and CTG++, the closest LLM-based baseline, is only discussed qualitatively. These reinforce the conditional verdict rather than overturning it: the concern is addressable by a controlled re-evaluation, and the method may still be state-of-the-art under a fair protocol. Therefore I keep the reader's CONDITIONAL verdict.","tokens_in":19467,"tokens_out":4236,"duration_ms":45954,"concrete_test":"Re-run the Table 1 protocol under two matched conditions: (a) LD-Scene with number of test samples set to 1 (no guidance-loss selection); (b) all baselines allowed the same best-of-10 sampling with the same feasibility filter and the same selection rule. Report the Adv-Ego collision rate for each condition, plus the rejection rate of the feasibility filter. If LD-Scene's advantage shrinks to within a few points of Safe-Sim/AdvSim under (a), or if the baselines match 40% under (b), the headline SOTA claim is an artifact of the selection protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — state-of-the-art adversarial scenario generation — rests on Table 1, specifically LD-Scene's Adv-Ego Collision Rate of 40.75%. Section 4.3 states that at inference the method generates 10 candidate future trajectories and selects the one that minimizes the guidance loss while satisfying physical feasibility constraints (cited to the authors' own Peng et al., 2025). Since the LLM-generated guidance loss is explicitly designed to encourage collision between the adversarial and ego vehicle, selecting the candidate with the lowest guidance loss will mechanically raise the measured collision rate even if the underlying conditional distribution is no more adversarial than the baselines. The three diffusion baselines (DiffScene, Safe-Sim) are not described as receiving this best-of-10 selection, and AdvSim/Strive are optimization methods with a different search protocol; the reported 40.75% vs 27.81% margin may therefore reflect protocol asymmetry rather than model quality. The feasibility filter is also unspecified and self-cited, so it is impossible to know how many candidates are rejected or how the filter interacts with the selection objective. This is the load-bearing weak point for the SOTA claim; without controlling the selection protocol, the headline comparison is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LD-Scene, a framework that combines a latent diffusion model for multi-agent trajectory generation with an LLM-based guidance module. The LLM translates a natural-language user query into an adversarial guidance loss function through chain-of-thought code generation and an automated code debugger. The guidance loss is used to perturb the denoising process so that a designated adversarial vehicle collides with an ego vehicle controlled by a rule-based planner. The authors evaluate on the nuScenes dataset in closed-loop simulation, reporting that LD-Scene achieves a 40.75% Adv-Ego collision rate, outperforming AdvSim, Strive, DiffScene, and Safe-Sim (Table 1). They also report ablation studies on the guidance components and the debugger, a controllability study of adversarial level and behavior, and an analysis of key parameters. The central claim is that LD-Scene achieves state-of-the-art controllable generation of realistic, diverse, and effective adversarial safety-critical driving scenarios.","tokens_in":19735,"tokens_out":3418,"duration_ms":33703,"significance":"If the claims are substantiated, the paper makes a useful contribution: it offers a user-friendly, natural-language interface for adversarial scenario generation, which is currently a gap in the diffusion-based trajectory generation literature. The combination of LLM-generated guidance code with a latent diffusion model is a sensible and timely integration, and the closed-loop evaluation against several representatives (optimization-based and diffusion-based) is relevant. The debugger module with quantitative success-rate improvements is a practical strength. However, the headline performance claim rests on an evaluation protocol that is not yet convincingly fair, so the significance of the empirical advance is currently uncertain. The paper's underlying idea is sound and the empirical claims are in principle testable, which makes the weaknesses fixable rather than fatal.","major_comments":[{"comment":"The headline Adv-Ego Collision Rate of 40.75% is obtained after a best-of-10 selection step: the paper states that the final trajectory is selected as the one that minimizes the guidance loss while satisfying physical feasibility constraints. Since the guidance loss is explicitly designed to encourage collision between the adversarial and ego vehicle, this selection protocol mechanically raises the measured collision rate even if the underlying conditional distribution is no more adversarial than that of the baselines. The three diffusion baselines (DiffScene, Safe-Sim) are not reported to receive the same selection protocol, and the optimization-based baselines (AdvSim, Strive) follow a different search procedure. The reported margin over Safe-Sim (40.75% vs 27.81%) may therefore reflect protocol asymmetry rather than model quality. To substantiate the SOTA claim, the authors should either apply the same best-of-10 selection to the diffusion baselines, or report the Adv-Ego collision rate without selection, or both; they should also report the mean (not best) collision rate over the 10 candidates.","section":"§4.3, Table 1"},{"comment":"The 'physical feasibility constraints' from Peng et al. (2025) are load-bearing for the selection procedure, but they are not defined anywhere in this manuscript. The reader cannot tell how many of the 10 candidate trajectories are rejected, how the filter interacts with the guidance-loss selection, or whether the same filter is applied to the baselines. Given that the feasibility filter is self-cited to the authors' own prior work, this is a reproducibility gap that directly affects the validity of Table 1. The constraints should be stated explicitly in the paper (or in an appendix), and the authors should report the rejection rate of the candidate trajectories.","section":"§4.3, Eq. (6)-(7)"},{"comment":"All metrics in Table 1 are reported as single point estimates without error bars, confidence intervals, or significance tests, and the number of evaluated scenarios is not stated. Some gaps between methods are small (e.g., Other Offroad is 17.95% for LD-Scene vs 16.64% for Strive), so the claimed 'significant advantages' are not supported by the reported data. The authors should report the number of scenes, the standard deviation or confidence intervals across scenes or random seeds, and perform pairwise significance tests (e.g., bootstrap or paired tests) for at least the headline Adv-Ego Collision Rate.","section":"§4.4, Table 1"},{"comment":"Fig. 5(a) reports that with the debugger, GPT-4o achieves a 95.0% success rate in generating executable guidance functions, meaning 5% of user queries fail to produce valid code. The paper does not state whether scenarios generated from failed code are included in the evaluation of Table 1, or whether the 40.75% collision rate is conditioned on successful code generation. If failures are excluded, the evaluation is only over successful queries, which biases the collision rate upward and differs from the baselines, which use predefined guidance functions with no generation failure. The authors should either include the failures in the denominator or clearly condition the reported metrics on successful code generation and discuss the resulting bias.","section":"§4.5.2, Fig. 5(a), §4.3"}],"minor_comments":[{"comment":"In Eq. (5), the notation Σg is used without defining the covariance matrix or how the gradient g is scaled; the relation between Σ and the diffusions steps is unclear. Please define all symbols.","section":"§3.4, Eq. (5)"},{"comment":"The text under Fig. 8 refers to 'Adv Offroad collision rate', which conflates an offroad metric with a collision rate; the y-axis label should be simply 'Adv Offroad Rate'.","section":"§4.7, Fig. 8 caption"},{"comment":"There is a typo in the Reasoning Prompt: 'guidance loss funtion class' should be 'guidance loss function class'.","section":"Fig. 2(c)"},{"comment":"The AdvAcceleration metric is defined only as 'acceleration magnitude' without specifying whether it is peak, mean, or per-scenario averaged; please clarify the exact computation.","section":"§4.2"},{"comment":"The adversarial vehicle selection strategy is described as 'closest to the ego vehicle in the initial state' and is said to follow Strive; however, Strive changes the adversarial vehicle dynamically, while the paper says it is fixed. Please clarify the divergence from Strive and its potential effect on the comparison.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The self-citation to Peng et al. (2025) for the load-bearing feasibility filter is a concern: if that prior work is not readily available or the constraints are not standard, the acceptance of the headline result depends on an unchecked external artifact. The authors should make the filter specification fully transparent in this paper. Additionally, the evaluation protocol asymmetry between LD-Scene and the baselines is the core issue; I would recommend the editor require the authors to either re-run baselines with the same selection protocol or justify why the protocol is inherent to their method and not applicable to baselines. The paper is within scope for the journal and the central idea is promising, but the empirical support is not yet sufficient for the SOTA claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper does something genuinely new, and the debugger ablation is a nice piece of engineering. But the 40.75% Adv-Ego collision rate that carries the SOTA claim is not a fair comparison as reported, because LD-Scene picks the best of 10 candidate trajectories using the exact guidance loss that rewards collision, while the diffusion baselines are not described as getting the same treatment. That asymmetry alone could account for a meaningful chunk of the gap over Safe-Sim.\n\nWhat is actually new: combining a Strive-style latent VAE with a diffusion denoiser and an LLM module that writes the adversarial guidance loss via chain-of-thought, then iteratively debugs it. That gives users natural-language control over the adversarial behavior and intensity, which is a step beyond hand-crafted objectives. The system is evaluated in closed loop with a rule-based planner, and the ablations show the guidance components each contribute and that the debugger raises code execution success from 69.4% to 95.0%. That is real, reproducible evidence of the system's internal value.\n\nWhere the paper is soft: the selection protocol is not applied to the baselines. AdvSim and Strive are optimization methods, so they may implicitly search over trajectories, but DiffScene and Safe-Sim are diffusion samplers that presumably return one sample per scene; giving only LD-Scene ten tries with a loss that directly targets the evaluation metric is not a like-for-like comparison. The physical feasibility filter is cited to the authors' own prior arXiv preprint and is not described, so we cannot tell how many candidates are rejected or how that interacts with selection. There are also no error bars or significance tests, and the closest related method, CTG++, is discussed but never compared. The controllability study shows the \"Strong\" level yields a lower collision rate than \"Medium\" (39.33% vs 40.75%), which the authors attribute to timing issues; this is plausible but further suggests the headline numbers are noisy.\n\nEven with these issues, the core idea survives. The LLM-as-guidance-writer plus debugger is a reasonable direction, and the closed-loop evaluation is appropriate. I would send this to peer review with a request that the authors clarify the selection protocol, apply the same protocol to all baselines, and report variance across seeds. The paper deserves referee time; the SOTA claim just needs better controls before it is taken at face value.","headline":"LD-Scene is a useful integration of LLM-generated guidance with latent diffusion, but the headline SOTA claim rests on a best-of-10 selection protocol that baselines did not receive, so the margin is not yet trustworthy.","tokens_in":20217,"tokens_out":2182,"would_cite":true,"duration_ms":24180,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By letting a large language model write the attack objective as code, LD-Scene guides a latent diffusion model to generate realistic, natural-language-controllable crash scenarios, reaching a 40.75% adversarial-ego collision rate in…","keywords":["large language models","diffusion models","traffic simulation","safety-critical scenario generation","autonomous driving","adversarial scenario generation","chain-of-thought","latent diffusion"],"falsifier":"Run LD-Scene with a single candidate trajectory per non-ego agent, removing the best-of-10 guidance-loss-plus-feasibility selection, and measure the Adv-Ego collision rate on the nuScenes validation split; if the rate falls toward the 15-28% band of the baselines, the reported advantage comes from the selection protocol rather than the learned guidance. Conversely, grant the same best-of-10 selection to AdvSim, Strive, DiffScene, and Safe-Sim and check whether the 40.75% gap persists.","tokens_in":19277,"feed_emoji":"🚗","tokens_out":8726,"duration_ms":72778,"temperature":0.7,"pith_summary":"LD-Scene claims that a user who can describe a dangerous driving event in plain language should be able to obtain a realistic, closed-loop traffic simulation in which a chosen background vehicle attacks the ego vehicle in exactly that way. The paper's central result is that pairing a latent diffusion model, which has learned realistic driving behavior from the nuScenes dataset, with an LLM that writes the adversarial objective as Python code lifts the adversarial-ego collision rate to 40.75%, well above the 15-28% range achieved by prior generators. This matters because safety-critical scenarios are rare in real driving data, and current generators either require expert-designed objectives or retrained classifiers for each new attack type. The paper also shows that the user can tune the attack's intensity and specify the maneuver, such as a high-speed overtake or a sharp turn. If the claim holds, LD-Scene turns adversarial scenario generation into a natural-language prompting task accessible to non-experts.","feed_headline":"LLM-guided simulator crashes the test car 40.75% of the time","feed_subtitle":"Describe an attack in plain language, and LD-Scene writes the loss code that steers realistic crash scenarios.","key_machinery":"The load-bearing object is the LLM guidance module: a zero-shot Chain-of-Thought prompt that decomposes a user query into three steps — classify the adversarial level (weak, medium, or strong), assign loss weights from predefined ranges, and complete a code template that produces a guidance loss class — followed by an LLM-based debugger that runs unit tests on the generated code and iteratively repairs it using the traceback. This guidance loss is injected into a latent diffusion model at inference time by decoding the predicted clean latent into trajectories, computing the gradient of the objective with respect to the latent, and shifting the denoising update; the gradient is applied separately to the adversarial vehicle's latent and to the non-adversarial vehicles' latents, with separate realism losses keeping background agents plausible. A pretrained graph-based VAE provides the latent space and the autoregressive kinematic decoder, while DDIM accelerates sampling. The code debugger is what makes the pipeline reliable: it lifts the success rate of generated guidance functions from 69.4% to 95.0% for the model used in the main experiments.","core_discovery":"LD-Scene's central claim is that an LLM-based guidance generation module, consisting of a Chain-of-Thought code generator and an iterative code debugger, can translate a natural-language user query into a differentiable adversarial loss function, and that this loss can steer the denoising process of a latent diffusion model to produce adversarial safety-critical scenarios that are both realistic and precisely controllable. Evaluated in closed-loop simulation with a rule-based planner on the nuScenes validation split, the method achieves a 40.75% adversarial-ego collision rate, a 12.52% adversarial offroad rate, and a 229.40-second average simulation time, which the paper reports as state-of-the-art relative to AdvSim, Strive, DiffScene, and Safe-Sim. The paper further claims that generation is controllable along two axes: the adversarial level, inferred from intensity descriptors in the query and mapped to loss-weight ranges, and the adversarial behavior, determined by the structure of the generated loss function.","pith_inferences":["The best-of-10 candidate selection described in the inference protocol means the reported 40.75% collision rate measures the best of ten guided samples, not the average sample; a single-sample evaluation would likely report a lower rate, and giving the same selection protocol to the baselines is the cleanest way to test whether the advantage is intrinsic to the guidance.","The guidance template could be extended beyond collision events to other safety-critical conditions such as near-misses with specified margins, traffic-rule violations that do not involve contact, or sensor-level stressors, since the LLM only needs to generate loss code within the provided template.","The framework's cost scales with LLM token usage for code generation and debugging; as cheaper code-capable models improve, the per-scenario cost should fall without changing the diffusion backbone.","A stricter test of controllability would measure whether the requested collision type or maneuver is actually achieved in the closed-loop rollout, not just whether the generated loss function contains the corresponding code structure."],"forward_implications":["A user can specify an adversarial event in plain language and receive a closed-loop simulation in which the specified behavior occurs, without writing any objective function or retraining a classifier.","The adversarial intensity of the generated scenarios scales with the query wording: weak queries yield longer time-to-collision and lower accelerations, while strong queries produce shorter time-to-collision and higher accelerations.","Because the adversarial vehicle remains fixed during the scenario and the ego planner reacts through a closed loop, generated scenarios can expose planner vulnerabilities such as yield-versus-pass decisions at junctions and ramps.","The debugger reduces dependence on the specific LLM: all evaluated models improve their guidance-code success rates, making the framework's reliability less sensitive to LLM choice."],"supporting_citations":[{"why":"Supplies the pretrained graph-based VAE traffic prior and the latent-space adversarial optimization paradigm that LD-Scene builds on.","marker":"(Rempe et al., 2022)"},{"why":"Provides the latent diffusion formulation that motivates diffusing in a compact latent space instead of raw trajectory space.","marker":"(Rombach et al., 2022)"},{"why":"Defines the forward and reverse diffusion processes and the simplified noise-prediction objective used to train the denoising network.","marker":"(Ho et al., 2020)"},{"why":"Supplies the U-Net denoising architecture and the gradient-based guidance rule that steers sampling toward an objective.","marker":"(Janner et al., 2022)"},{"why":"The language-guided diffusion baseline whose code-generation instability motivates the debugger, and the direct predecessor for LLM-written guidance.","marker":"(Zhong et al., 2023a)"},{"why":"The strongest diffusion-based adversarial baseline, whose TTC-augmented safety guidance sets the comparison point for adversarial realism.","marker":"(Chang et al., 2024)"},{"why":"The classic trajectory-space adversarial perturbation baseline that LD-Scene must outperform.","marker":"(Wang et al., 2021)"},{"why":"Provides the inference-time selection of the best of ten candidate trajectories under guidance loss and physical-feasibility constraints.","marker":"(Peng et al., 2025)"},{"why":"DDIM non-Markovian sampling that makes the 20-step inference cheap enough for closed-loop use.","marker":"(Song et al., 2020)"},{"why":"The rule-based lane-graph planner used as the reacting ego vehicle in the closed-loop evaluation.","marker":"(Montemerlo et al., 2008)"}],"fun_headline_variants":["LLM-written attack code crashes AVs 40.75% of the time","Plain-language crash queries steer diffusion to 40.75% hits","Describe a crash, get a crash: LD-Scene's LLM does it","User says 'crash', LLM writes loss, sim obeys","Diffusion + LLM = controllable car crashes on demand"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 40.75% collision rate rests on the inference protocol described in Section 4.3: ten candidate trajectories per agent are generated and the one minimizing guidance loss while passing a physical-feasibility filter is kept, and the baselines are not given the same selection advantage.","fun_headline_variants_meta":{"raw":{"variants":["LLM-written attack code crashes AVs 40.75% of the time","Plain-language crash queries steer diffusion to 40.75% hits","Describe a crash, get a crash: LD-Scene's LLM does it","User says 'crash', LLM writes loss, sim obeys","Diffusion + LLM = controllable car crashes on demand"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1699,"prompt_tokens":980,"completion_tokens":719,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":596,"tokens_out":719,"duration_ms":7000,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:54:46.189005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LD-Scene with a single candidate trajectory per non-ego agent, removing the best-of-10 guidance-loss-plus-feasibility selection, and measure the Adv-Ego collision rate on the nuScenes validation split; if the rate falls toward the 15-28% band of the baselines, the reported advantage comes from the selection protocol rather than the learned guidance. Conversely, grant the same best-of-10 selection to AdvSim, Strive, DiffScene, and Safe-Sim and check whether the 40.75% gap persists.","supporting_citations":[],"review_version":1}