{"id":"9ac7e9e2-0992-4531-a557-ef8a7403b42d","arxiv_id":"2412.02574","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"AVASTRA uses deep reinforcement learning over ADS internal states and environment states to generate realistic collision scenarios, outperforming DeepCollision by 30-115%.","lead":"The paper presents AVASTRA, a reinforcement-learning system that automatically configures weather, traffic, and road conditions in a driving simulator to create dangerous situations for an autonomous vehicle under test. It reports that AVASTRA produces 30-115% more collision scenarios than the previous best method, which could make simulation-based safety testing more effective.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Collision-counting exclusions in Sec. 5.2.3 are never quantified per method; if they remove more DeepCollision events than AVASTRA events, the 30–115% advantage is a measurement artifact.","rationale":"The Reader's weakest assumption identifies exactly the load-bearing risk: the Sec. 5.2.3 collision-counting conventions can be applied unevenly across methods. I agree with that assessment. The paper's own ablation shows that heuristic constraints halve the number of subjective collisions (Table 7), which is indirect evidence that the exclusion policy interacts with the method being evaluated. Since no per-method exclusion counts are provided, the 30–115% headline margin cannot be separated from the counting rule. This is the single most load-bearing concern because it directly affects the quantitative claim in the abstract and conclusion, not just the interpretation. The requested fix is feasible: expose raw collision logs and recompute under transparent counting policies. I do not see evidence of dishonesty or internal contradiction; conditional acceptance with these reporting requirements remains the right outcome, which matches the Reader's verdict. I therefore do not change the verdict, but I would make the per-method exclusion audit a hard condition.","tokens_in":24976,"tokens_out":5604,"duration_ms":64184,"concrete_test":"Instrument the collision evaluator to log, per episode and method, the raw collision event stream with timestamps and obstacle IDs. Recompute Table 5 under three counting policies: (a) all raw collision events; (b) only first-collision deduplication, no subjective exclusions; (c) the full Sec. 5.2.3 conventions, tagging each excluded event with a deterministic reason. If the AVASTRA-to-DeepCollision ratio under policy (b) is materially below the reported 1.30–2.15 range, the counting convention, not the method, drives the result. Also report the standard deviation and 95% confidence interval over the five seeds for each cell; without these, the Road 4 30% advantage cannot be distinguished from seed-to-seed noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AVASTRA generates 30–115% more collision scenarios than DeepCollision. Table 5 counts only collisions surviving the Sec. 5.2.3 conventions: first-collision deduplication and exclusion of 'subjective collisions' (sudden obstacles in the safety zone, and side/rear-end collisions caused by NPC vehicles or pedestrians). The paper never reports how many raw collision events were removed per method per road, nor how the 'beyond ADS control' judgment was operationalized. This matters because AVASTRA's heuristic constraints were explicitly designed to reduce subjective collisions: Table 7 shows 2.2 subjective collisions with constraints versus 4.4 without them on the same road. DeepCollision, by the authors' own characterization, does not enforce realism, so it may produce a larger share of exactly the excluded collision types. If so, the reported 30–115% improvement could reflect the counting rule selectively deleting DeepCollision's successes rather than AVASTRA finding more valid critical scenarios. The paper also reports only means over five seeds and 15 episodes, with no variance or significance test, so the Road 4 gap (10.4 vs 13.4) may be within noise. This is not an internal inconsistency, but the headline advantage is currently unverifiable from the data presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AVASTRA, a reinforcement-learning approach for generating critical driving scenarios that stress-test an autonomous driving system (ADS) in the LGSVL simulator with Apollo 7.0. AVASTRA represents the environment through both external factors (weather, time of day, traffic, road condition) and internal ADS states (localization error, perception error, control parameters), uses a 45-action space to configure those factors, and applies heuristic constraints to keep scenarios realistic. The authors compare AVASTRA against DeepCollision and Random Search on four road configurations, reporting that AVASTRA generates 30% to 115% more collisions than DeepCollision and up to 275% more than Random Search. They also conduct component, parameter, and sensitivity analyses. The core claim is that combining internal and external state awareness with realism constraints improves the discovery of safety-critical scenarios.","tokens_in":25326,"tokens_out":4408,"duration_ms":44472,"significance":"If the claimed advantage is real, AVASTRA is a useful contribution to simulation-based ADS testing: it is grounded in actual simulator collisions rather than a reward proxy, it uses a publicly available implementation, and its ablations give insight into the value of internal states, action groups, and heuristic constraints. The four-road evaluation across two maps and the concrete scenario examples are strengths. However, the headline result is not yet verifiable from the reported data because the collision-counting conventions are not quantified per method and no variance or significance information is provided. The tuning of distance constants and RL hyperparameters on the evaluation roads also makes the comparison vulnerable to overfitting. These issues are addressable with additional reporting and re-analysis rather than being fundamental flaws.","major_comments":[{"comment":"The #Collisions metric counts only first collisions and excludes 'subjective collisions,' but the paper reports neither the raw collision counts nor the number of excluded events per method per road, and it does not operationalize the 'beyond ADS control' or 'safety zone' judgments. This matters because Table 7 shows that AVASTRA's constraints reduce subjective collisions from 4.4 to 2.2, while DeepCollision, by the authors' own characterization, does not enforce such realism and may therefore produce a larger share of the excluded categories. Without per-method, per-road exclusion counts and a repeatable exclusion rule, the 30–115% improvement in Table 5 could be a measurement artifact rather than a genuine difference in scenario-generation ability.","section":"Sec. 5.2.3 and Table 5"},{"comment":"All headline results are reported as means over five seeds and 15 episodes with no error bars, confidence intervals, or significance tests. The Road 4 gap (10.4 vs 13.4 collisions) is especially likely to fall within seed-to-seed noise, so the claim that AVASTRA 'consistently outperforms' across all roads is not statistically supported. Please report per-seed distributions or standard deviations and, where appropriate, a paired significance test, or justify why such statistics are unnecessary.","section":"Sec. 5.2.2 and Table 5"},{"comment":"The distance constants fd, fv, fs and the weights α, β, γ in Eq. (7) are described as 'empirically determined via various experiments,' and the RL hyperparameters OTP, Rcol, and θ_epsilon are then selected through the experiments in Figs. 7–9, without specifying which road(s) those parameter studies used. If the same four roads were used for both tuning and the final comparison, the reported advantage over the baselines may partly reflect tuning to those specific test roads. Please state the road(s) used in the parameter analyses and assess the sensitivity of the main comparison to the chosen values.","section":"Sec. 3.2, Eq. (7), and Sec. 6.3"}],"minor_comments":[{"comment":"The subfigure captions are out of order: (c) labels Road 4 and (d) labels Road 3. Please reorder the panels or captions so that the road numbering is consistent.","section":"Fig. 3"},{"comment":"There are several typos, including 'Apolo' for 'Apollo' and 'migrate' for 'mitigate.' In Sec. 6.1, 'versa vise' should be 'vice versa.'","section":"Sec. 6.5"},{"comment":"The discussion of TTC is cursory: the paper states that lower TTC is better, then reports AVASTRA with higher TTC than DeepCollision on Roads 1 and 3 without explaining why a method that generates more collisions can take longer to trigger them. A brief explanation would help readers interpret the TTC metric when collision counts differ.","section":"Sec. 6.1"},{"comment":"The paragraph on weather pattern constraints says the authors 'do not consider weather constraints in this paper' and defer them to future work. This is a clear statement of a limitation and should be acknowledged in the threats-to-validity section as well.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software engineering or testing venue, and the authors have made a genuine effort to evaluate on a real simulator with an industrial ADS. The main blocking issue is the unquantified collision-counting convention combined with the absence of any statistical support for the headline improvement; both are fixable by additional reporting and re-analysis. I do not see grounds for rejection, but I would not accept the paper in its current form because the central quantitative claim is not verifiable from the presented data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, AVASTRA is a real step beyond DeepCollision: it puts ADS-internal state (localization, perception, control errors) into the RL state, adds a constrained action catalogue, and shows consistent collision gains across four roads. Second, don't quote the 30–115% number without checking the counting rules—the paper excludes \"subjective collisions\" but never reports how many raw events were removed per method. If those exclusions hit DeepCollision harder, the advantage shrinks.\n\nWhat's actually new: the internal-state representation is the main contribution. Equations 1–6 give concrete error signals for localization and perception, and the ablation in Fig. 6 shows full state beats either alone by 15–65%. The heuristic constraints on spawn distance (Eq. 7) also have a clear effect: Table 7 shows constraints double collisions and halve subjective collisions (4.4 to 2.2). That is a real, reproducible result, and the authors ship code and data.\n\nSoft spots, in order of importance. First, the counting convention. Section 5.2.3 excludes sudden obstacles in the safety zone and side/rear-end NPC collisions; Table 7 shows constraints reduce subjective collisions, and DeepCollision has no equivalent mechanism, so it plausibly produces more of exactly the excluded class. Without per-method, per-road counts, the headline comparison is a measurement artifact in waiting. Second, no error bars or significance tests anywhere. Five seeds × 15 episodes is enough to report variance, and without it Road 4's 10.4 vs 13.4 could be noise. Third, tuning: OTP, Rcol, θϵ and the Eq. 7 distance constants were selected using the same test roads, which inflates the absolute numbers even if it doesn't invalidate the relative comparison.\n\nThe paper is honest about its own limitations—the lane-structure failure in Fig. 4 is acknowledged, as are external validity threats. It is not circular: the reward is based on ProC but the metric is actual simulator collisions. The core idea is sound and the empirical direction is consistent. It deserves a serious referee, but the referee should push for per-method exclusion tallies, variance reporting, and ideally a held-out road for parameter selection.\n\nWho it's for: anyone in ADS validation or search-based testing; it's a useful baseline for future work, not a paradigm shift.","headline":"AVASTRA is a solid extension of DeepCollision with a genuine internal-state contribution, but the headline collision advantage is undercut by unquantified exclusion rules and missing variance.","tokens_in":25801,"tokens_out":1713,"would_cite":true,"duration_ms":16790,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AVASTRA claims that a reinforcement-learning agent which represents both the internal states of the ADS (localization, perception, control) and external environmental states generates 30–115% more collision scenarios than DeepCollision…","keywords":["autonomous driving systems","critical scenario generation","reinforcement learning","collision generation","simulation-based testing","Double Deep Q-Network","internal and external states","heuristic constraints"],"falsifier":"Compare both methods on the same four roads with a single pre-registered collision-counting rule that reports raw collision events before any exclusion of subjective collisions, and also report the number of excluded events per method; if DeepCollision's raw or post-filter counts are not below AVASTRA's by the same margins, the claimed 30–115% advantage is a measurement artifact.","tokens_in":24786,"feed_emoji":"🚗","tokens_out":9042,"duration_ms":74135,"temperature":0.7,"pith_summary":"Testing autonomous driving systems on real roads is expensive and risky, so the industry relies on simulators that can be configured into dangerous scenarios. This paper tries to establish that a reinforcement-learning agent generates more of those safety-critical scenarios when it is given a holistic view of the situation: both the internal state of the ADS (localization error, perception error, and control signals) and the external environment (weather, time of day, traffic flow, road condition). The authors build AVASTRA, which trains a Double Deep-Q Network to set weather, time, and the placement and behavior of NPC vehicles and pedestrians, subject to heuristic realism constraints. Across four road configurations in the LGSVL simulator with Apollo 7.0, they report 30–115% more collisions than the prior RL method DeepCollision and up to 275% more than random search. If these numbers hold, simulation-based testing can uncover more ADS failure modes before deployment.","feed_headline":"Testing agent that watches car and road finds 30–115% more crashes","feed_subtitle":"Simulation tests expose more ADS failures before real-world driving; it beats DeepCollision by 30–115%.","key_machinery":"The machinery is a 19-variable state representation (seven external plus twelve internal variables) feeding a Double Deep-Q Network (a deep Q-learning variant with separate selection and evaluation networks to reduce overestimation) whose 45 actions configure the simulator. The external variables follow the JAMA safety-evaluation framework's perception disturbance factors; the internal variables turn ADS component health into numbers by measuring localization deviation, perception discrepancies for nearby objects, and actual control signals, with prediction and planning represented indirectly through their inputs and outputs. The action space is kept tractable and realistic by three heuristic constraints: chronological time progression, distance-dependent spawn positions for NPC vehicles and pedestrians (a weighted function of distance mode, vehicle size, and speed), and behavior restrictions for pedestrians. The reward is a collision probability $ProC$ in $[0,1]$, computed from longitudinal and lateral safety distances versus current distance, with $R_{col}$ for actual collisions. This combination is what the paper credits for learning to place the AV in dangerous but plausible situations.","core_discovery":"On its own terms, the paper claims that configuring a driving simulator can be learned as a Markov decision process whose state is a 19-variable tuple: seven external state variables (weather condition, time of day, nearest traffic light, number of obstacles, distance to nearest obstacle, speed of nearest obstacle, road condition) and twelve internal state variables (localization distance and angle deviation, perception position/direction/size/velocity differences, and control values throttle, steerRate, steerTarget, accRate, brakePercentage, speed). The RL agent acts by choosing among 45 actions that change the time of day, set a weather phenomenon and intensity, or spawn an NPC vehicle or pedestrian with a specific behavior, distance, speed, and lane. AVASTRA trains a DDQN with prioritized experience replay, rewarding actions that raise a collision probability computed from longitudinal and lateral safety distances. In the evaluation, AVASTRA's #Collisions are 12.0, 18.9, 11.6, and 13.4 on the four roads, versus DeepCollision's 7.6, 12.6, 5.4, and 10.4 and Random Search's 3.2, 5.4, 5.2, and 7.0, which the paper reports as 30–115% and up to 275% improvements. The paper also claims the full state space beats internal-only or external-only variants, that all three action groups contribute, that heuristic constraints double the collision count and halve subjective collisions, and that training on all four roads transfers best to a fixed test road.","pith_inferences":["The reported advantage may depend on the Section 5.2.3 collision-counting conventions; re-running DeepCollision with the same raw-event counting and the same subjective-collision exclusion pipeline could shrink or widen the 30–115% range.","Because the state tuple and collision-probability equations are simulator-agnostic, the same design could be dropped into CARLA or AirSim; the main unknowns are the action-space mapping and whether those simulators expose comparable internal ADS errors.","A testable extension is to keep the state and action spaces fixed but swap DDQN for an on-policy or actor-critic algorithm; if gains persist, the state design rather than the specific RL algorithm is doing the work.","The authors' own median example (Fig. 4) suggests adding lane-structure awareness, such as medians and road boundaries, to the state or action constraints could remove false-risk configurations and push the collision yield higher on roads with physical separation."],"forward_implications":["If AVASTRA's central claim is correct, ADS testing can move more of the search for failure-inducing scenarios from expensive, risky on-road testing into simulation.","The 30–115% margin over DeepCollision implies that adding internal ADS states (localization, perception, control) to the RL observation is a genuine lever, not just extra bookkeeping.","The heuristic constraints' effect—doubling collisions and halving subjective collisions—suggests that realism filters can improve both yield and relevance rather than merely suppress candidates.","The RQ4 result that comprehensive-road training beats same-road and cross-road training implies scenario generators should be trained on diverse maps even when the target road is fixed.","The mixed TTC results imply the method trades some time-to-collision speed for a higher volume of collisions; scenario-generation efficiency has at least two dimensions."],"supporting_citations":[{"why":"LGSVL simulator, the high-fidelity testbed in which AVASTRA generates scenarios and whose configurable parameters define the action space.","marker":"[4]"},{"why":"DeepCollision, the state-of-the-art RL baseline AVASTRA extends and compares against; also the source of the collision-probability measurement idea.","marker":"[8]"},{"why":"JAMA Automated Driving Safety Evaluation Framework, whose perception disturbance factors structure the external states and whose safety requirements motivate the heuristic constraints.","marker":"[10]"},{"why":"Cheng et al.'s longitudinal collision avoidance (Berkeley algorithm), which supplies the safety-distance default values (deceleration -6 m/s², Rmin 5 m) used in the collision-probability and reward computation.","marker":"[21]"},{"why":"MOSAT, a search-based scenario generator that motivates the scalability criticism and provides one of the map/road choices (San Francisco) reused in evaluation.","marker":"[6]"},{"why":"AV-Fuzzer, another search-based baseline focused on NPC maneuvers, which AVASTRA contrasts with its broader environmental parameter coverage.","marker":"[7]"},{"why":"Defines Apollo's five ADS modules (localization, perception, prediction, planning, control), which structure AVASTRA's internal-state design.","marker":"[15]"},{"why":"Weighted double Q-learning, the foundation for using DDQN to avoid overestimated Q-values in the stochastic driving environment.","marker":"[19]"}],"fun_headline_variants":["RL agent crafts driving tests that find 30–115% more crashes","Simulated crash hunting with RL: up to 275% better than random","AVASTRA: RL-generated scenarios expose 30–115% more collisions","Driving sims get harder: RL bait finds more ADS failures","Auto-test your self-driving car with RL-crafted danger"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline improvement depends on the rules it uses to count collisions—excluding 'subjective' collisions that an ADS cannot avoid and counting only the first collision in a repeated sequence—and those rules must be applied identically to AVASTRA and both baselines for the 30–115% and 275% numbers to be a fair comparison.","fun_headline_variants_meta":{"raw":{"variants":["RL agent crafts driving tests that find 30–115% more crashes","Simulated crash hunting with RL: up to 275% better than random","AVASTRA: RL-generated scenarios expose 30–115% more collisions","Driving sims get harder: RL bait finds more ADS failures","Auto-test your self-driving car with RL-crafted danger"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000499,"raw_usage":{"total_tokens":2553,"prompt_tokens":1167,"completion_tokens":1386,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":783,"completion_tokens_details":{"reasoning_tokens":1292}},"tokens_in":783,"tokens_out":1386,"duration_ms":10139,"temperature":1.0,"reasoning_tokens":1292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:17:46.037003+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare both methods on the same four roads with a single pre-registered collision-counting rule that reports raw collision events before any exclusion of subjective collisions, and also report the number of excluded events per method; if DeepCollision's raw or post-filter counts are not below AVASTRA's by the same margins, the claimed 30–115% advantage is a measurement artifact.","supporting_citations":[{"cited_title":"Lgsvl simulator: A high fidelity simulator for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"LGSVL simulator, the high-fidelity testbed in which AVASTRA generates scenarios and whose configurable parameters define the action space."},{"cited_title":"Learning configurations of operating environment of au- tonomous vehicles to maximize their collisions,","cited_arxiv_id":null,"evidence_quote":"DeepCollision, the state-of-the-art RL baseline AVASTRA extends and compares against; also the source of the collision-probability measurement idea."},{"cited_title":"Auto- mated Driving Safety Evaluation Framework Ver 3.0,","cited_arxiv_id":null,"evidence_quote":"JAMA Automated Driving Safety Evaluation Framework, whose perception disturbance factors structure the external states and whose safety requirements motivate the heuristic constraints."},{"cited_title":"Longitudinal collision avoidance and lateral stability adaptive control system based on mpc of autonomous vehicles,","cited_arxiv_id":null,"evidence_quote":"Cheng et al.'s longitudinal collision avoidance (Berkeley algorithm), which supplies the safety-distance default values (deceleration -6 m/s², Rmin 5 m) used in the collision-probability and reward computation."},{"cited_title":"Mosat: finding safety violations of autonomous driving systems using multi-objective genetic algorithm,","cited_arxiv_id":null,"evidence_quote":"MOSAT, a search-based scenario generator that motivates the scalability criticism and provides one of the map/road choices (San Francisco) reused in evaluation."},{"cited_title":"Av-fuzzer: Finding safety violations in autonomous driving systems,","cited_arxiv_id":null,"evidence_quote":"AV-Fuzzer, another search-based baseline focused on NPC maneuvers, which AVASTRA contrasts with its broader environmental parameter coverage."},{"cited_title":"A first look at the integration of machine learning models in complex autonomous driving systems: a case study on apollo,","cited_arxiv_id":null,"evidence_quote":"Defines Apollo's five ADS modules (localization, perception, prediction, planning, control), which structure AVASTRA's internal-state design."},{"cited_title":"Weighted double q- learning,","cited_arxiv_id":null,"evidence_quote":"Weighted double Q-learning, the foundation for using DDQN to avoid overestimated Q-values in the stochastic driving environment."}],"review_version":1}