{"id":"1ca0ed3a-96d0-40e6-89e9-640c8860984d","arxiv_id":"2507.22769","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A Bayesian optimization framework finds critical scenarios for an MPC motion planner using one to two orders of magnitude fewer simulations than full-factorial testing.","lead":"This paper uses Bayesian optimization to pick which driving scenarios to simulate, hunting for dangerous situations faster than testing every combination. It shows the approach finds off-road and controller-failure cases in a simple highway planner with far fewer simulation runs than an exhaustive grid.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The speed-up numbers are produced by a stopping rule that already knows the baseline critical clusters, so the 'orders of magnitude fewer simulations' claim is a hitting-time for known targets, not a de novo discovery cost.","rationale":"The reader's CONDITIONAL verdict is appropriate, and this stress-test reinforces it by sharpening the concern: the stopping criterion is not merely a cluster-validity assumption but an oracle termination rule that invalidates the simulation counts as discovery costs. The paper does acknowledge the upper bound of 150 simulations for 3-DoF, but the reported means (often <100) come from early stopping on known clusters; without baseline knowledge, a user would need a convergence-based or fixed-budget rule, which could require more runs to achieve the same completeness. The proposed concrete test directly measures whether the framework retains its efficiency under a realistic stopping rule. Credit is due for the co-simulation setup, the explicit comparison of acquisition functions, and the hybrid-metric idea, which are useful contributions; the concern is not that BO cannot find failures, but that the headline quantitative claim is not yet supported for de novo discovery. The random/LHS baseline issue also matters and is noted by the reader, but the oracle stopping criterion is the more load-bearing flaw because it affects the validity of every reported count, not just the baseline comparison.","tokens_in":15262,"tokens_out":9167,"duration_ms":118344,"concrete_test":"Rerun Simulation Sets B (3-DoF Thompson + continuous metric) and F (6-DoF hybrid) with the baseline cluster labels withheld, using a de novo stopping rule such as: stop when 20 consecutive Thompson draws yield no increase in the observed maximum criticality metric, or when the acquisition function maximum over the domain falls below a threshold. Count how many of the 10 runs per set discover both baseline critical clusters (off-road and solver-failure) within the reported mean budget. If any run misses the solver-failure cluster or requires more simulations than the oracle-stopped runs, the 'orders of magnitude' speed-up is contingent on knowing the answer in advance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV states: 'The iteration of the Bayesian Optimization framework simulations terminate when at least one instance in each of the critical clusters identified in the baseline simulations are identified.' Thus the reported simulation counts (Figures 6 and 8) are measured with an oracle stopping rule that uses the exhaustive baseline's DBSCAN clusters as termination targets. In a real validation campaign, these clusters are unknown without first running the very exhaustive DoE the method is meant to replace. The central claim 'identifies hazardous situations ... using orders of magnitude fewer simulations' (Abstract) is therefore not demonstrated as a discovery cost; it is the cost of hitting a pre-specified set of targets. The framework as described provides no practical stopping criterion for unknown critical regions, and without one the reported mean/median counts are not achievable in deployment. This is independent of whether the DBSCAN clusters are complete or correct, which the reader already flagged.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a Bayesian optimization (BO) framework for discovering safety-critical scenarios during virtual validation of an autonomous driving function. The logical scenario parameters (first three, then six) are searched by a Gaussian-process surrogate and an acquisition function (Thompson Sampling or probability of improvement) against a criticality metric: maximum lateral displacement or MPC solver status. The case study is an MPC-based motion planner in a two-lane highway with a preceding vehicle; the baseline is a full-factorial discretization (1,331 runs in 3D and 19,800 runs in 6D), and DBSCAN clustering of the baseline results defines the critical clusters. BO runs are stopped when at least one point in each baseline-defined critical cluster has been found, and the paper reports mean/median counts below 100 in 3D and around 100-200 in 6D, claiming one to two orders of magnitude fewer simulations than brute-force DoE.","tokens_in":15512,"tokens_out":6927,"duration_ms":84498,"significance":"Accelerated critical-scenario search is an important problem, and the paper offers a transparent case study with two failure modes, repeated random-seed experiments, and a clearly described co-simulation setup. The central quantitative claim, however, is not established by the current evaluation: the stopping rule uses the baseline cluster labels as targets, no comparison to random sampling or Latin hypercube sampling at equal budget is provided, and the success criterion of one point per cluster is weak. If the authors reframe the claim as targeted re-identification of known critical clusters and add appropriate comparator methods, the work would be a useful empirical contribution. The paper does not provide code or data, which limits reproducibility.","major_comments":[{"comment":"The paper states: 'The iteration of the Bayesian Optimization framework simulations terminate when at least one instance in each of the critical clusters identified in the baseline simulations are identified.' Because the target clusters are produced by the exhaustive baseline, the simulation counts in Figures 6 and 8 are hitting times for a known target set, not the cost of discovering unknown critical regions. In a real validation campaign the clusters are unknown without first running the baseline that the method is meant to replace. The abstract's claim that the framework 'identifies hazardous situations ... using orders of magnitude fewer simulations' is therefore not demonstrated as a de novo discovery cost. Please either add a deployable stopping rule and evaluate de novo discovery, or explicitly reframe the results as targeted search for known clusters and adjust the title and abstract claims accordingly.","section":"Section IV, stopping criterion"},{"comment":"No comparison is made against random sampling, Latin hypercube sampling, or any other model-free search at the same simulation budget. This matters because the off-road cluster occupies 56 of 1,331 points (~4%) in 3D, and in 6D the two critical clusters together occupy 6,894 of 19,800 points (~35%). At budgets of 50-200 simulations, random or LHS sampling would be expected to hit these clusters by chance a substantial fraction of the time. The reported one-to-two-order speed-up relative to the full-factorial grid does not establish that BO itself is responsible for the efficiency. Same-budget random/LHS detection rates should be reported, and the BO advantage over these baselines should be quantified.","section":"Section V, Figures 6-8"},{"comment":"The success criterion is defined as finding at least one instance in each baseline-derived critical cluster. This is weak because it does not measure coverage of the critical regions, and the clusters themselves depend on the arbitrarily chosen baseline discretization and on the DBSCAN hyperparameters (eps, min_samples), which are not reported. A single point in a cluster could be an artifact of the grid or of the clustering parameters, and critical regions missed by the baseline grid are invisible to the evaluation. Please report the DBSCAN hyperparameters, provide a sensitivity analysis of the clusters, and use a stronger discovery metric such as the volume fraction of each cluster that is covered, or require multiple points per cluster.","section":"Section V-A, DBSCAN clustering and success criterion"},{"comment":"The statistical reporting is not complete enough to support the scalability claims. The 'hybrid strategy' of Set F, said to alternate between the two criticality metrics, is never defined (no schedule, no equations, no pseudocode). Figure 8's caption labels the categories as A and B although the text discusses Sets E and F, and the text states that a 472-simulation outlier in Set E is omitted from the figure while the reported mean is affected by it. Furthermore, in the 3D case, sets C and D that failed to find both clusters in some runs have their means excluded (Section V-A), which biases the acquisition-function comparison. Please report success rates, include all runs (including failures and outliers), and give an exact definition of the hybrid metric alternation.","section":"Section V-B and Figure 8"}],"minor_comments":[{"comment":"The claim that this is the first work applying such a framework to a developed ADF is not supported by the cited related work, since references [34]-[36] already apply BO or scenario-exploration methods to automated-driving test-case search; please temper the novelty claim or explain the precise distinction.","section":"Abstract and I-D.2"},{"comment":"The sentence 'Different combinations of the acquisition functions and Hence this paper also benchmarks for the stopping criteria...' is ungrammatical and should be revised.","section":"Section IV, paragraph after Table I"},{"comment":"Table I has an extra bracket in the x_0,ego range ('[-0.5,0.5]]'), and Table II's caption reverses the mapping of Clusters 1 and 3 to off-road events and a priori solver failure relative to the description in Section V-A.","section":"Table I and Table II"},{"comment":"The caption uses 'A' and 'B' for the two simulation sets, while the text refers to Sets E and F; the caption should be corrected for consistency.","section":"Figure 8 caption"},{"comment":"The probability-of-improvement formula does not specify the behavior when the posterior variance sigma(x) is zero; standard implementations add a small jitter, which should be stated.","section":"Equation (7)"},{"comment":"The GP surrogate is trained on the discrete solver-status metric (values such as 0 and 4) using a Gaussian likelihood and a squared-exponential kernel; the authors should justify this treatment of a discrete response or use a surrogate model better suited to classification outputs.","section":"Section III-B and Sections V-A/V-B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's framing exceeds what the evidence supports: the main efficiency numbers are produced with an oracle stopping rule that presumes knowledge of the baseline clusters, and there is no comparison to simple random or LHS search. I would ask the editor to require a clear re-framing or additional experiments before publication. The novelty claim in I-D.2 is probably overclaimed given the cited literature, and the absence of code/data and exact cluster parameters weakens reproducibility. The case study itself is useful and the underlying BO methodology is sound; the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate case study, but the headline 'orders of magnitude fewer simulations' is not backed by the evidence as written. The two soft spots are the oracle stopping rule and the lack of a random/LHS baseline. Both are fixable.\n\nWhat's actually new: most of the machinery is textbook BO, as the authors acknowledge. The value is in the benchmark: two acquisition functions (Thompson Sampling vs Probability of Improvement) crossed with two criticality metrics (continuous lateral displacement vs discrete solver status), including a hybrid strategy that alternates metrics. That comparison is useful and not something I've seen in the cited BO-for-AV papers. The 3D/6D scalability study on their own MPC planner is also concrete, and the co-simulation setup (Simulink + acados + CarMaker) is non-trivial. They report failure cases honestly, e.g., PI fails to find both clusters in 5 of 10 runs, and they note the single-metric strategy in 6D had an outlier run of 472 simulations.\n\nWhere it gets soft. The stopping criterion in Section IV terminates when one instance in each of the baseline-identified DBSCAN clusters is found. That means the reported simulation counts are hitting times for known targets, not discovery costs for unknown critical regions. In a real validation campaign, you don't know these clusters without running the exhaustive DoE you're trying to avoid. The abstract's claim is therefore stronger than what is demonstrated. This is not a fatal flaw for a case study, but it needs to be said plainly.\n\nThe bigger practical problem is the baseline. The paper only compares against full-factorial DoE. In the 6D case, the critical clusters cover about 35% of the space (6,174 off-road + 720 solver-failure out of 19,800), and the smaller solver-failure cluster is about 3.6% of the space. Random or LHS sampling at the same budget would likely find both clusters with high probability within a few hundred runs. Without that comparison, the speed-up over 'brute-force' is a low bar. The authors should add a random/LHS baseline at equal total budget.\n\nI also wish the DBSCAN parameters and the alternation schedule for the hybrid strategy were specified in more detail, but those are minor.\n\nBottom line: the paper deserves a serious referee. It's a well-written, honest engineering study with a useful empirical comparison. But I'd send it back for major revision: add a random baseline, replace or supplement the oracle stopping rule with a practical one, and temper the abstract. If the authors do that, the result will be a solid contribution to scenario-based validation.","headline":"The case study is honest and the benchmark is useful, but the 'orders of magnitude' claim rests on an oracle stopping rule and a strawman baseline; it merits peer review with major revision.","tokens_in":16013,"tokens_out":4961,"would_cite":true,"duration_ms":56809,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bayesian optimization finds all critical driving scenarios with one to two orders of magnitude fewer simulations.","keywords":["Bayesian optimization","autonomous driving validation","critical scenario discovery","Gaussian process surrogate","Thompson sampling","motion planning","simulation-based testing"],"falsifier":"Construct a parameter space where a genuine critical region lies between the grid lines of the baseline discretization, so the baseline and its DBSCAN clusters do not contain it; run the BO framework with the paper's stopping rule. If the loop terminates having hit every baseline-identified cluster but never the hidden region, the completeness claim fails.","tokens_in":15090,"feed_emoji":"🚗","tokens_out":7267,"duration_ms":79155,"temperature":0.7,"pith_summary":"The paper is trying to establish that exhaustive simulation is not necessary to locate the dangerous scenarios of an autonomous driving function. It casts critical-scenario discovery as the maximization of a safety-violation metric over continuous scenario parameters and solves that maximization with Bayesian Optimization, letting a Gaussian-process surrogate choose which parameter sets to simulate next. Tested on a model-predictive motion planner in a two-lane highway operational design domain, the framework recovers every critical region that an exhaustive baseline found, including off-road events and solver infeasibility, using one to two orders of magnitude fewer simulations. The point of the work is that validation budgets could be concentrated on the parameter regions most likely to fail, shortening development and homologation loops for autonomous driving functions.","feed_headline":"Critical driving scenarios found with 1-2 orders fewer simulations","feed_subtitle":"An MPC motion planner's off-road and solver-failure regions are recovered using a fraction of the brute-force simulation budget.","key_machinery":"The load-bearing mechanism is the Bayesian Optimization loop: a Gaussian process with a squared exponential kernel models the map from scenario parameters to a criticality metric; an acquisition function then selects the next parameter set, the co-simulation evaluates the metric, and the surrogate is refit. Thompson Sampling is the acquisition function that naturally balances exploration and exploitation, and it is compared with Probability of Improvement. Critical regions are defined by DBSCAN clustering of the exhaustive baseline results, and the loop stops when at least one simulation has landed in every cluster that the baseline identified, subject to an upper budget one order of magnitude below the baseline count. The two metrics used are the maximum lateral displacement of the ego vehicle (continuous) and the MPC solver's infeasibility status (discrete), which correspond to the two failure modes: off-road excursions and solver failure.","core_discovery":"The paper claims that critical-scenario discovery for an autonomous driving function can be treated as a black-box optimization problem: find the scenario parameters $x$ that maximize a criticality metric $f(x)$ computed by one simulation run. A Bayesian Optimization loop with a Gaussian process surrogate and an acquisition function recovers all critical regions found by exhaustive combinatorial testing while using a fraction of the simulation budget. In the 3-parameter case, 15 seed simulations plus acquisition iterations found both critical clusters in under 100 simulations on average, against a 1,331-run full-factorial baseline; in the 6-parameter case, 30 seed simulations plus iterations matched clusters found in a 19,800-run baseline. The paper further claims that the workflow is scenario-agnostic, so any logical scenario with continuous parameters inside an operational design domain can be plugged into the same loop, and that the hybrid strategy of alternating the criticality metric used to train the surrogate is what makes the higher-dimensional search reliable.","pith_inferences":["An implication the authors leave implicit is that the stopping rule presupposes the baseline-identified clusters are the complete set of critical regions; in a real development setting with no exhaustive baseline, the loop would need an alternative stopping rule such as a budget cap or a surrogate-convergence criterion.","A natural stress test would be to run the same BO loop on the same motion planner with a critical region deliberately placed between the grid lines of the baseline discretization; the reported speed-up is defined relative to the grid, not to an oracle.","The hybrid metric strategy suggests a general recipe: when failure modes are heterogeneous, no single scalar objective captures them all, so alternating or multi-objective acquisition may be necessary.","A direct extension is to multi-agent urban scenarios by treating all agents' initial poses and behaviors as the parameter vector and using a collision or near-miss metric as the objective; the machinery would carry over intact."],"forward_implications":["A validation campaign can substitute a BO-guided search for a full-factorial sweep and still claim to have seen the same critical regions, at a fraction of the simulation cost.","Continuous criticality metrics, like maximum lateral displacement, should guide the surrogate when possible; discrete metrics alone rely on exploration and can miss clusters under exploitative acquisition functions.","Acquisition functions that balance exploration and exploitation, such as Thompson Sampling, are more reliable for multi-cluster discovery than a greedy Probability of Improvement.","In higher-dimensional parameter spaces, alternating the metric used to train the surrogate is the paper's recommended fix for reliably finding small, hard-to-locate failure clusters.","The framework's scenario-agnostic formulation means the same loop can be re-applied to other logical scenarios and other driving functions without changing the optimization machinery."],"supporting_citations":[{"why":"Supplies the rule-defined adaptive MPC motion planner that the framework is validated on.","marker":"[17]"},{"why":"Provides the rule of thumb that initial seed samples are two to five times the parameter dimension, used to size the training set.","marker":"[40]"},{"why":"DBSCAN defines the critical clusters on baseline results, which act as the ground truth and stopping condition.","marker":"[46]"},{"why":"Surveys criticality metrics and motivates the choice of direct safety indicators over TTC or PET.","marker":"[38]"},{"why":"Supplies the optimal-control solver whose runtime and a priori infeasibility constitute one of the two failure modes.","marker":"[44]"},{"why":"Supplies the previous validation context and the unchanged controller weights used in the case study.","marker":"[45]"},{"why":"Earlier work identifying test cases for automated driving with Bayesian optimization, which this framework extends.","marker":"[34]"},{"why":"Prior study of Bayesian optimization with different surrogate models for worst-case failure search, the baseline this work builds on.","marker":"[36]"}],"fun_headline_variants":["BO finds critical driving scenarios with orders of magnitude fewer sims","Critical AV scenario search: 100x fewer simulations with BO","Find safety-critical driving cases with 1/100th the simulations","Bayesian optimization finds critical scenarios with 100x fewer simulations","Cut AV validation simulations by orders of magnitude via Bayesian optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the DBSCAN clusters found in the exhaustive baseline are the complete and correct set of critical regions, and that finding one simulation in each cluster is enough to prove all critical scenarios were discovered.","fun_headline_variants_meta":{"raw":{"variants":["BO finds critical driving scenarios with orders of magnitude fewer sims","Critical AV scenario search: 100x fewer simulations with BO","Find safety-critical driving cases with 1/100th the simulations","Bayesian optimization finds critical scenarios with 100x fewer simulations","Cut AV validation simulations by orders of magnitude via Bayesian optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002185,"raw_usage":{"total_tokens":8443,"prompt_tokens":901,"completion_tokens":7542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":7456}},"tokens_in":517,"tokens_out":7542,"duration_ms":54525,"temperature":1.0,"reasoning_tokens":7456,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:18:04.773580+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a parameter space where a genuine critical region lies between the grid lines of the baseline discretization, so the baseline and its DBSCAN clusters do not contain it; run the BO framework with the paper's stopping rule. If the loop terminates having hit every baseline-identified cluster but never the hidden region, the completeness claim fails.","supporting_citations":[{"cited_title":"A rule-defined adaptive mpc based motion planner for autonomous driving applications,","cited_arxiv_id":null,"evidence_quote":"Supplies the rule-defined adaptive MPC motion planner that the framework is validated on."},{"cited_title":"A density-based al- gorithm for discovering clusters in large spatial databases with noise,","cited_arxiv_id":null,"evidence_quote":"DBSCAN defines the critical clusters on baseline results, which act as the ground truth and stopping condition."},{"cited_title":"Criticality metrics for automated driving: A review and suitability analysis of the state of the art,","cited_arxiv_id":null,"evidence_quote":"Surveys criticality metrics and motivates the choice of direct safety indicators over TTC or PET."},{"cited_title":"acados – a modular open-source framework for fast embedded optimal control,","cited_arxiv_id":null,"evidence_quote":"Supplies the optimal-control solver whose runtime and a priori infeasibility constitute one of the two failure modes."},{"cited_title":"Virtual validation and uncertainty quantification of an adaptive model predictive controller-based motion planner for autonomous driving systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the previous validation context and the unchanged controller weights used in the case study."},{"cited_title":"A bayesian optimization method for finding the worst-case scenarios of autonomous vehicles,","cited_arxiv_id":null,"evidence_quote":"Prior study of Bayesian optimization with different surrogate models for worst-case failure search, the baseline this work builds on."}],"review_version":1}