Pith. sign in

REVIEW 4 major objections 5 minor 67 references

EVITA generates diverse multi-vehicle interactions in simulation, exposing safety failures that single-vehicle tests miss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:29 UTC pith:CQBU7LRI

load-bearing objection Solid multi-simulator test-generation paper with a real contribution, but the headline interaction advantage is partly by construction; the independent collision/violation results carry the load. the 4 major comments →

arxiv 2607.12452 v2 pith:CQBU7LRI submitted 2026-07-14 cs.SE

Automatic Testing of Interacting Autonomous Vehicles

Fabio Cavaleri , Alessio Gambi , Paolo Arcaini , Dejan Ni\v{c}kovi\'c , Mauro Pezz\`e This is my paper
classification cs.SE
keywords autonomous vehiclesinteraction testingscenario-based testingsearch-based testingmulti-objective optimizationevolutionary algorithmtrajectory replanningsafety-critical scenario generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

EVITA is an automated test generator for autonomous vehicles that abandons the common single-ego-vehicle setup: instead of testing one AV against scripted traffic, it runs several self-driving vehicles in the same simulated scene and actively searches for scenarios in which those vehicles interact in many different ways. The paper's claim is that maximizing the number and variety of such interactions, while keeping the number of vehicles small, uncovers collisions and traffic-rule violations that existing approaches overlook. The authors evaluate EVITA across two different autonomous-driving systems and six road networks, and report that it consistently produces more interactions, more distinct interaction types, more collisions, and more traffic violations than the state-of-the-art multi-vehicle baseline, with simpler scenarios. If true, this gives the AV-testing community a cheap, computable objective—interaction coverage—that points test generation at the emergent, hard-to-predict failures that matter most before deployment.

Core claim

The central claim is that safety-relevant multi-AV behavior can be detected by a lightweight operational definition of an interaction: an AV is interacting at time t if it significantly replans its planned trajectory (C1) and that replanning coincides with a nearby scenario element such as another vehicle, a traffic light, or a road sign (C2). EVITA monitors consecutive planned trajectories, flags replannings by a cumulative Euclidean-distance threshold, and records them on a two-dimensional interaction map that classifies the direction and magnitude of each trajectory change. Three fitness functions—interaction count, interaction diversity, and number of vehicles—guide an evolutionary multi

What carries the argument

The mechanism that carries the argument is the interaction definition, Conditions C1 and C2: a significant change between an AV's consecutive planned trajectories (measured by cumulative Euclidean distance against a threshold) that occurs while a scenario element—another vehicle, traffic light, or road sign—is nearby. On top of this, interaction maps turn each interaction into a cell on a grid whose axes are lateral and longitudinal trajectory change; this grid both classifies interactions into eight direction types and provides the diversity fitness that steers the search. EVITA wraps these concepts in an evolutionary multi-objective loop with selection, crossover, and mutation that generat

Load-bearing premise

The load-bearing premise is that a significant trajectory replanning that happens near another vehicle, traffic light, or road sign is a genuine safety-relevant interaction, and that maximizing the number and variety of such replannings raises the chance of revealing real safety failures; Section 7.1 explicitly concedes that a replanning may not actually be caused by the nearby element, so false positives could weaken that link.

What would settle it

Use a planner that logs the true cause of each replanning, and run EVITA on scenarios where the nearby-element check is deliberately disabled; if the generated scenarios still trigger many interactions but almost no collisions or traffic violations, the replanning proxy would be shown to be disconnected from safety outcomes. A simpler check: place scenario elements far away but keep the proxy threshold intact and see whether collision and violation rates disappear.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Safety testing can be steered by interaction coverage rather than by known failure types, so previously unknown emergent behaviors have a chance to surface.
  • The same interaction objective works across different planners and road types (highway, suburban, urban), suggesting it is not tied to one simulator or vehicle stack.
  • Smaller scenarios—fewer vehicles—suffice to trigger diverse interactions, which makes failures easier to reproduce, debug, and fix.
  • A concrete coverage metric for multi-AV interaction is now available: the interaction map, analogous to a coverage table for traditional software testing.
  • Because the approach does not depend on controlling pedestrians or pre-programmed NPCs, it can be applied to fleets of independent AVs where no single vehicle is scripted in advance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same replanning-based interaction detector could be applied to logged real-world driving data, testing whether interaction diversity correlates with safety incidents outside simulation; the paper does not report such a study.
  • Editorial inference: the interaction map's grid resolution is a free parameter, and the paper does not analyze its sensitivity; coarser or finer grids would change the diversity fitness and could alter the search's behavior.
  • Editorial inference: the reported gain over the baseline is measured with the baseline's own failure oracles; a stronger test would compare against random scenario generation to isolate how much of the benefit comes from interaction-guided search rather than from the richer scenario representation itself.
  • Editorial inference: the authors' observation that most interactions cluster around direction changes rather than speed changes suggests the proxy inherits the planner's behavioral bias; different planners might yield different interaction distributions, so the headline results may not extend unchanged to all AV stacks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents EVITA, a search-based approach for generating test scenarios that involve multiple interacting autonomous vehicles (AVs). EVITA defines an "interaction" as a trajectory replanning (Condition C1) spatially co-occurring with a scenario element such as another vehicle, traffic light, or road sign (Condition C2). It then uses NSGA-II to optimize three objectives: maximize the number of interactions, maximize the diversity of interaction types, and minimize the number of AVs in each scenario. The approach is evaluated in two simulation environments: FrenetiX on five highway/suburban road networks and Baidu Apollo on one urban network, comparing against DoppelTest as the only open-source multi-AV baseline. Experiments use 10 repetitions, Mann-Whitney U tests, and A12 effect sizes. The paper reports that EVITA outperforms DoppelTest on total/unique interactions in most FrenetiX settings, on collisions and traffic violations, and on scenario simplicity, while showing a small statistical disadvantage on unique interactions in the Apollo urban setting.

Significance. If the central claims hold, EVITA would be a useful contribution to scenario-based testing for multi-AV systems, an area where most prior work targets single AVs. The paper provides a replication package, follows recommended statistical practice, and evaluates on two substantially different AV stacks and multiple road networks. The notion of interaction diversity as a search objective is interesting and under-explored. However, the empirical evidence is weakened by the tight coupling between the fitness functions and the RQ1 metrics, and by the lack of validation of the interaction proxy. These issues are load-bearing for the headline claim and need to be addressed before the contribution can be fully assessed.

major comments (4)
  1. [§4.6.1, §4.6.2, §5.2.1, Eqs. (3)–(4)] The RQ1 metrics are essentially the same as the optimized objectives. fit_int (Eq. 3) is exactly TotalInts, and fit_div (Eq. 4) is computed from the same interaction-count vectors cts that are aggregated into UniqueInts via the interaction map. Thus, the RQ1 comparison mainly measures whether EVITA's NSGA-II search can maximize its own fitness functions, and the superiority over DoppelTest partly reflects a difference in objectives. The comparison with DoppelTest is still meaningful because DoppelTest optimizes different objectives, but the abstract's claim that EVITA 'triggers a higher variety' should be supported by an independent measure not used in the search, or by a comparison with a random-search baseline to quantify the optimization gain.
  2. [§3, Conditions C1/C2; §7.1] The interaction detector checks spatial/temporal co-occurrence, not causation. The threats section candidly states that 'a change in a planned trajectory does not necessarily originate from an interaction with such scenario elements' and dismisses the impact based only on 'sample inspection,' with no quantitative precision/recall analysis. This is load-bearing because the fitness functions and RQ1 metrics are computed from this detector, and because RQ2/RQ3 results are interpreted as consequences of interaction diversity. Please provide a validation of the detector on labeled or manually inspected scenarios, or at least a systematic error analysis reporting false-positive rates. Without this, the claimed link between interaction diversity and safety-critical behavior remains unsupported.
  3. [§6.1, Fig. 9b, Answer to RQ1] In the Apollo urban setting, UniqueInts shows a 'small statistical disadvantage' for EVITA relative to DoppelTest. This directly contradicts the abstract/introduction claim of consistently higher variety. The explanation offered in the text and in RQ4 — that DoppelTest achieves this with more complex scenarios — is post hoc and not backed by a controlled analysis. The authors should either temper the general claim or design an experiment that matches scenario complexity across approaches to test this explanation.
  4. [§5.3.3, §8] The evaluation compares against a single baseline, DoppelTest, which the authors extended to work with FrenetiX. While this may be the only open-source multi-AV approach, the paper's title and abstract claim superiority over 'state-of-the-art approaches.' A comparison against an interaction-agnostic multi-objective search (e.g., NSGA-II optimizing only oracle-based objectives such as collisions and violations) and against random scenario generation would help isolate the contribution of the interaction fitness. Given that DoppelTest is already extended, adding such baselines seems feasible and would materially strengthen the conclusions.
minor comments (5)
  1. [§3.1, Eq. (1)] The equation appears to have a typographical issue in the brackets: the Euclidean norm is written with an extra opening square bracket. Please fix the formatting.
  2. [Table 1, §3.1] The threshold Th_ED introduced in Section 3.1 is not listed among the hyperparameters in Table 1, despite being a free parameter of the interaction detector. Please add it with its default value. Also, max_emergency_slowdown appears in Table 1 but is not described in the text.
  3. [§5.2.2] The definition of RIV partitions says '11 partitions (10 partitions capture values between 0 and 100Km/h, and one partition captures the edge case > 100Km/h).' Please clarify the interval boundaries (e.g., whether the last partition is >=100 km/h or >100 km/h) and similarly for RIA.
  4. [§6.2] Minor language issue: 'The data confirm the superiority of the EVITA' should be 'of EVITA' without the definite article.
  5. [§7.1] The statement that 'the search algorithm is robust and works well with approximate metrics' is not supported by any data in the paper. A sensitivity analysis varying the interaction-detector parameters or a comparison with precise but slower detectors would be useful.

Circularity Check

2 steps flagged

RQ1 interaction metrics mirror EVITA's own fitness functions: TotalInts is literally Eq. 3, and UniqueInts is derived from the same interaction-count vectors that fitdiv optimizes, so the headline advantage over DoppelTest is partly by construction.

specific steps
  1. fitted input called prediction [Section 4.6.1 (Eq. 3) and Section 5.2.1 (TotalInts), used in Section 6.1 (RQ1)]
    "fitint(s)=∑_{av∈AVs} |{t_i∈{t_1,...,t_n}: isInteraction(av,t_i)}| ... We compute TotalInts as per Equation 3"

    TotalInts, the primary RQ1 outcome used to claim EVITA 'outperforms DoppelTest', is the same formula as the fitness function fitint that EVITA's NSGA-II search maximizes. DoppelTest optimizes test oracles, not interaction count, so EVITA's advantage in TotalInts is a direct result of optimizing the measured quantity rather than independent evidence. Section 7.1 itself admits the detector can produce false positives ('a change in a planned trajectory does not necessarily originate from an interaction'), and the same detector feeds both the fitness and the metric, so the measured superiority may reflect optimizing a noisy replanning proxy.

  2. self definitional [Section 4.6.2 (Eq. 4) and Section 5.2.1 (UniqueInts), used in Section 6.1 (RQ1)]
    "fitdiv(s)=min_{s_i∈{s_1,...,s_k}} ∥c_{t_s}−c_{t_{s_i}}∥_2 ... we compute UniqueInts using the interaction map described in Section 3.3 ... number of cells in the interaction map that are covered by at least one interaction"

    Both fitdiv and UniqueInts are defined from the same interaction-count/cell-map representation: fitdiv records cts=[#BL,#L,...,#R,#FR] and maximizes pairwise distance in that vector space, while UniqueInts counts how many cells of the same map are covered. The diversity selection pressure and the diversity metric are therefore not independent; the RQ1 'higher variety of interactions' result is substantially implemented by the same bookkeeping used to evaluate it. The equivalence is not exact (min-distance vs. cardinality), making this a partial rather than total circularity.

full rationale

The central circularity is evaluative: EVITA's search explicitly maximizes Eqs. 3 and 4, and RQ1 then reports TotalInts and UniqueInts computed from those same interaction definitions. TotalInts is exactly the interaction fitness (Eq. 3), and UniqueInts is a readout of the same cell-count vectors that fitdiv spreads out. The paper's own construction-validity section (7.1) concedes false positives without quantitative precision/recall, so the headline interaction advantage could be inflated by routine replanning events. That is load-bearing because the abstract and introduction claim the interaction trigger itself 'improves the likelihood to reveal safety-critical behaviors'. However, the paper is not wholly circular: RQ2/RQ3 rely on external oracles (collision bounding-box checks; traffic-light, stop-sign, and speeding violations) that are not EVITA's fitness functions, and DoppelTest is an external open-source baseline, so the safety-critical part of the claim has independent content. No load-bearing self-citation chain was found: self-citations (e.g., Riccio et al. for feature maps, Gambi et al. for TIAV) are contextual, not used to force the interaction definition. Given one exact metric=fitness reduction and one closely related reduction, with independent oracle-based results, score 6 (partial circularity) is appropriate.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 1 invented entities

The paper's central empirical claims rest on hand-chosen thresholds and an unvalidated interaction proxy; the free parameters are search/tuning choices rather than fitted constants, but they shape every reported metric. The main invented construct is the interaction definition itself, which has no external falsifiable handle.

free parameters (8)
  • Th_ED trajectory-replanning threshold
    Chosen threshold in Eq. 1 (Section 3.1) that decides whether a planned-trajectory change counts as an interaction; no value reported, directly gates RQ1/RQ2 metrics.
  • Interaction map grid: 10 bins per axis plus extra bucket; lateral ±3.70 m, longitudinal ±4.5 m
    Hand-set binning (Section 3.3) determines UniqueInts and fit_div; changing grid size changes reported interaction diversity.
  • initial_population_size = 10
    Table 1; NSGA-II search setting chosen for both simulators.
  • min_AVs / max_AVs = 2;8 (FrenetiX), 2;5 (Apollo)
    Table 1; bounds on scenario complexity affect RQ4 and the search space.
  • min/max green light duration = 50–150 s (Apollo)
    Table 1; traffic-light timing is mutated and affects urban interaction and violation results.
  • generation budget = 3 CPU-hours (FrenetiX), 24 CPU-hours (Apollo)
    Table 1; unequal budgets across simulators constrain how many scenarios each approach can generate.
  • max_time_step = 200 (FrenetiX), 300 (Apollo)
    Table 1; scenario timeout affects whether long interactions and collisions are observed.
  • Mutation probabilities = 1/|AVs|, 1/3, 1/2
    Section 4.4.1; hand-chosen probabilities governing evolution, not varied in a sensitivity analysis.
axioms (5)
  • ad hoc to paper Trajectory replanning near a scenario element (C1+C2) is a valid and safety-relevant interaction.
    Section 3; introduced by the paper; no external benchmark; Section 7.1 admits false positives.
  • domain assumption CommonRoad/FrenetiX and Baidu Apollo simulations faithfully represent real-world multi-AV emergent behavior.
    Sections 2 and 5.3.2; all empirical conclusions are about simulated planners, not physical roads.
  • ad hoc to paper More diverse interactions cause more diverse collisions.
    Remark 2 (Section 6.2) asserts this from correlation; no causal mechanism or controlled experiment is provided.
  • domain assumption NSGA-II with the given operators and budget is an adequate optimizer for scenario generation.
    Section 4.1; typical for search-based testing, but no comparison against other optimizers or budget sensitivity.
  • standard math Euclidean cumulative distance is a valid significance measure for trajectory changes.
    Eq. 1, Section 3.1; standard metric, but the threshold choice remains arbitrary.
invented entities (1)
  • Formal "interaction" (C1+C2) and interaction maps no independent evidence
    purpose: Define and quantify AV interactions for both fitness and evaluation (Sections 3–4).
    A new measurement construct introduced by the paper; no external validation beyond the authors' sample inspection and correlation with collisions in the same experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 22200 in / 14382 out tokens · 135672 ms · 2026-08-02T06:29:33.673021+00:00 · methodology

0 comments
read the original abstract

Autonomous vehicles (AVs) must be thoroughly tested to meet high safety standards and avoid endangering both AV passengers and road users. Scenario-based testing implements driving scenarios in virtual simulation environments as a cost-effective alternative to field testing. Common scenario-based testing approaches set the environment and the surrounding traffic and test a single AV. Recent studies show that the approaches that test single AVs miss critical behaviors that emerge from interactions among multiple AVs. Effective approaches to test scenarios that emerge from n-way interactions must address the combinatorial explosion that the presence of multiple AVs further exacerbates. In this paper, we propose EVITA, an approach that leverages multi-objective optimization to generate scenarios that trigger multiple and diverse AVs interactions, while minimizing the complexity of the generated scenarios, to effectively test multiple interacting AVs and reveal safety-critical scenarios that current approaches overlook. The experimental results that we discuss in this paper confirm that EVITA triggers a higher variety of AVs interactions than state-of-the-art approaches, thus improving the likelihood to reveal safety-critical behaviors.

Figures

Figures reproduced from arXiv: 2607.12452 by Alessio Gambi, Dejan Ni\v{c}kovi\'c, Fabio Cavaleri, Mauro Pezz\`e, Paolo Arcaini.

Figure 1
Figure 1. Figure 1: Motivating example: Testing two vehicles with the same initial and final positions, and trajectories [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example of a Multi-AV CommonRoad highway scenario [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Identifying trajectory replanning in FrenetiX [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The interaction maps from the trajectory replanning of the orange vehicle in Figure [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Benchmark road networks for test subject one (FrenetiX) [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Benchmark road network for test subject two (Baidu Apollo) [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: RQ1 – Interactions found by EVITA and DoppelTest with FrenetiX experimental setting Figure 9a and Figure 9b report the distribution of TotalInts and UniqueInts obtained with EVITA and DoppelTest across the urban road network used in the experimental setting with Baidu Apollo [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: RQ1 – Interactions maps with FrenetiX experimental setting [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: RQ1 – Interactions found by EVITA and DoppelTest with Baidu Apollo experimental setting Remark 1. The interaction maps in [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: RQ1 – Interactions maps with Baidu Apollo experimental setting [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: RQ2 – Collisions found by EVITA and DoppelTest with FrenetiX experimental setting 6.2 RQ2 – Does EVITA generate scenarios with multiple and different types of collisions? RQ2 evaluates the ability to generate critical scenarios and compares EVITA to DoppelTest in terms of both the number and types of triggered collisions. Figure 11a reports the total number of EVITA and DoppelTest scenarios that lead to c… view at source ↗
Figure 12
Figure 12. Figure 12: RQ3 – Number of detected violations with the Baidu Apollo experimental setting [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: RQ4 – Number of vehicles in the generated scenarios with the FrenetiX experimental setting [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: RQ4 – Number of vehicles in the generated scenarios with the Baidu Apollo experimental setting [PITH_FULL_IMAGE:figures/full_fig_p022_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: RQ5 – Generation statistics with the FrenetiX experimental setting [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: RQ5 – Generation statistics with the Baidu Apollo experimental setting [PITH_FULL_IMAGE:figures/full_fig_p023_16.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 5 canonical work pages

  1. [1]

    Self-driving car blocking road ‘delayed patient care’, San Francisco officials say

    2023. Self-driving car blocking road ‘delayed patient care’, San Francisco officials say. https://www.theguardian.com/us- news/2023/sep/05/san-francisco-cruise-robotaxi-death-ambulance

    2023

  2. [2]

    16,000 test licenses for autonomous vehicles issued in China

    2024. 16,000 test licenses for autonomous vehicles issued in China. https://english.www.gov.cn/news/202408/27/ content_WS66cd745ac6d0868f4e8ea485.html

    2024

  3. [3]

    Autonomous Vehicle Testing Permit Holders

    2024. Autonomous Vehicle Testing Permit Holders. https://www.dmv.ca.gov/portal/vehicle-industry-services/ autonomous-vehicles/autonomous-vehicle-testing-permit-holders/

    2024

  4. [4]

    CommonRoad scenarios

    2024. CommonRoad scenarios. https://commonroad.in.tum.de/scenarios

    2024

  5. [5]

    Cruise recalls its robotaxis after passenger injured in crash

    2024. Cruise recalls its robotaxis after passenger injured in crash. https://edition.cnn.com/2022/09/01/business/cruise- robotaxi-recall/index.html

    2024

  6. [6]

    Driverless Cruise robotaxis stop working simultaneously, blocking San Francisco street

    2024. Driverless Cruise robotaxis stop working simultaneously, blocking San Francisco street. https://www.theverge. com/2022/7/1/23191045/cruise-robotaxis-driverless-roadblock-san-francisco

    2024

  7. [7]

    Over 1,400 self-driving vehicles are now in testing by 80+ companies across the US

    2024. Over 1,400 self-driving vehicles are now in testing by 80+ companies across the US. https://techcrunch.com/ 2019/06/11/over-1400-self-driving-vehicles-are-now-in-testing-by-80-companies-across-the-u-s/

    2024

  8. [8]

    Houssam Abbas, Georgios Fainekos, Sriram Sankaranarayanan, Franjo Ivancic, and Aarti Gupta. 2013. Probabilistic Temporal Logic Falsification of Cyber-Physical Systems.ACM Trans. Embed. Comput. Syst.12, 2s (2013), 95:1–95:30. doi:10.1145/2465787.2465797

    arXiv 2013

  9. [9]

    Saleh Albeaik, Alexandre Bayen, Maria Teresa Chiri, Xiaoqian Gong, Amaury Hayat, Nicolas Kardous, Alexander Keimer, Sean T McQuade, Benedetto Piccoli, and Yiling You. 2022. Limitations and improvements of the intelligent driver model (IDM).SIAM journal on Applied Dynamical Systems21, 3 (2022), 1862–1892

    2022

  10. [10]

    Matthias Althoff, Markus Koschi, and Stefanie Manzinger. 2017. CommonRoad: Composable benchmarks for motion planning on roads. In2017 IEEE Intelligent Vehicles Symposium (IV). 719–726. doi:10.1109/IVS.2017.7995802

    arXiv 2017

  11. [11]

    Matthias Althoff, Stefan Urban, and Markus Koschi. 2018. Automatic Conversion of Road Networks from OpenDRIVE to Lanelets. In2018 IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI), Singpapore, Singapore, July 31 - Aug. 2, 2018. IEEE, 157–162. doi:10.1109/SOLI.2018.8476801 26 Cavaleri et al

    arXiv 2018

  12. [12]

    Paolo Arcaini, Xiao-Yi Zhang, and Fuyuki Ishikawa. 2021. Targeting Patterns of Driving Characteristics in Testing Autonomous Driving Systems. In2021 IEEE 14th International Conference on Software Testing, Validation and Verification (ICST). 295–305. doi:10.1109/ICST49551.2021.00042

    arXiv 2021

  13. [13]

    Andrea Arcuri and Lionel Briand. 2011. A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering. InProceedings of the 33rd International Conference on Software Engineering(Waikiki, Honolulu, HI, USA)(ICSE ’11). Association for Computing Machinery, New York, NY, USA, 1–10. doi:10.1145/1985793.1985795

    arXiv 2011

  14. [14]

    Baidu Apollo team. 2025. Apollo: Open Source Autonomous Driving. https://github.com/ApolloAuto/apollo. Last access: June, 2026

    2025

  15. [15]

    Kockelman

    Prateek Bansal and Kara M. Kockelman. 2017. Forecasting Americans’ long-term adoption of connected and au- tonomous vehicle technologies.Transportation Research Part A: Policy and Practice95 (2017), 49–63. doi:10.1016/j.tra. 2016.10.013

    work page doi:10.1016/j.tra 2017

  16. [16]

    Briand, and Thomas Stifter

    Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2016. Testing Advanced Driver Assistance Systems Using Multi-objective Search and Neural Networks. InProceedings of the 31st IEEE/ACM International Con- ference on Automated Software Engineering(Singapore, Singapore)(ASE 2016). ACM, New York, NY, USA, 63–74. doi:10.1145/2970276.2970311

    arXiv 2016

  17. [17]

    Briand, and Thomas Stifter

    Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing Vision-based Control Systems Using Learnable Evolutionary Algorithms. InProceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden)(ICSE ’18). ACM, New York, NY, USA, 1016–1026. doi:10.1145/3180155.3180160

    arXiv 2018

  18. [18]

    Briand, and Thomas Stifter

    Raja Ben Abdessalem, Annibale Panichella, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing Au- tonomous Cars for Feature Interaction Failures Using Many-objective Search. InProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering(Montpellier, France)(ASE 2018). ACM, New York, NY, USA, 143–154. doi:10.114...

    arXiv 2018

  19. [19]

    Matteo Biagiola and Stefan Klikovits. 2024. SBFT Tool Competition 2024 - Cyber-Physical Systems Track. InProceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing(Lisbon, Portugal)(SBFT ’24). Association for Computing Machinery, New York, NY, USA, 33–36. doi:10.1145/3643659.3643932

    arXiv 2024

  20. [20]

    Matteo Biagiola, Stefan Klikovits, Jarkko Peltomäki, and Vincenzo Riccio. 2023. SBFT Tool Competition 2023 - Cyber- Physical Systems Track. In2023 IEEE/ACM International Workshop on Search-Based and Fuzz Testing (SBFT). 45–48. doi:10.1109/SBFT59156.2023.00010

    arXiv 2023

  21. [22]

    Alessandro Calò, Paolo Arcaini, Shaukat Ali, Florian Hauer, and Fuyuki Ishikawa. 2020. Generating Avoidable Collision Scenarios for Testing Autonomous Driving Systems. In2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). 375–386. doi:10.1109/ICST46399.2020.00045

    arXiv 2020

  22. [23]

    Automatic Testing of Interacting Autonomous Vehicles

    Fabio Cavaleri, Alessio Gambi, Paolo Arcaini, Dejan Ničković, and Mauro Pezzè. 2026. Repository for the paper “Automatic Testing of Interacting Autonomous Vehicles”. https://github.com/fabiocavaleri/EVITA

    2026

  23. [24]

    Mingfei Cheng, Yuan Zhou, and Xiaofei Xie. 2023. BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (Seattle, WA, USA)(ISSTA 2023). Association for Computing Machinery, New York, NY, USA, 488–500. doi:10.1145/ 3597926.3598072

    arXiv 2023

  24. [25]

    Marcus Vinicius Leal de Carvalho, Roberto Simoni, Leopoldo Rideki Yoshioka, João Francisco Justo Filho, and Bruno Mamoru Kato Kawakami. 2025. A Performance Evaluation of Open Source Autonomous Driving Frameworks: Case Studies of Apollo and Autoware.IEEE Access13 (2025), 44294–44316. doi:10.1109/ACCESS.2025.3549649

    arXiv 2025

  25. [26]

    Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. A. M. T. Meyarivan. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II.IEEE Transactions on Evolutionary Computation6, 2 (2002), 182–197. doi:10.1109/4235. 996017

    work page doi:10.1109/4235 2002

  26. [27]

    Oussama Derbel, Tamas Peter, Hossni Zebiri, Benjamin Mourllion, and Michel Basset. 2012. Modified Intelligent Driver Model.Periodica Polytechnica Transportation Engineering40, 2 (2012), 53–60. doi:10.3311/pp.tr.2012-2.02

    work page doi:10.3311/pp.tr.2012-2.02 2012

  27. [28]

    Alessio Gambi, Paolo Arcaini, and Dejan Ničković. 2025. Generation of Critical Interactive Scenarios for Trajectory Planning. In2025 IEEE Intelligent Vehicles Symposium (IV). 1950–1955. doi:10.1109/IV64158.2025.11097787

    arXiv 2025

  28. [29]

    Alessio Gambi, Gunel Jahangirova, Vincenzo Riccio, and Fiorella Zampetti. 2022. SBST Tool Competition 2022. In15th IEEE/ACM International Workshop on Search-Based Software Testing, SBST@ICSE 2022, Pittsburgh, PA, USA, May 9, 2022. IEEE, 25–32. doi:10.1145/3526072.3527538

    arXiv 2022

  29. [30]

    Alessio Gambi, Marc Müller, and Gordon Fraser. 2019. Automatically testing self-driving cars with search-based procedural content generation. InProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019, Dongmei Zhang and Anders Møller (Eds.). ACM, 318–328. doi:10.1145/3293882...

    arXiv 2019

  30. [31]

    Maximilian Geisslinger, Phillip Karle, Johannes Betz, and Markus Lienkamp. 2021. Watch-and-Learn-Net: Self- supervised Online Learning for Probabilistic Vehicle Trajectory Prediction. In2021 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2021, Melbourne, Australia, October 17-20, 2021. IEEE, 869–875. doi:10.1109/ SMC52423.2021.9659079

    arXiv 2021

  31. [32]

    Luca Giamattei, Antonio Guerriero, Roberto Pietrantuono, and Stefano Russo. 2024. Causality-driven Testing of Autonomous Driving Systems.ACM Trans. Softw. Eng. Methodol.33, 3, Article 74 (mar 2024), 35 pages. doi:10.1145/ 3635709

    2024

  32. [33]

    Grace Eliza Goodwin. 2025. Waymo robotaxis stop in the streets during San Francisco power outage. https://www. bbc.com/news/articles/c36zdxl41jro

    2025

  33. [34]

    Yuqi Huai, Yuntianyi Chen, Sumaya Almanee, Tuan Ngo, Xiang Liao, Ziwen Wan, Qi Alfred Chen, and Joshua Garcia

  34. [35]

    WuLing Huang, Kunfeng Wang, Yisheng Lv, and FengHua Zhu. 2016. Autonomous vehicles testing methods review. In2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC)(Rio de Janeiro, Brazil). IEEE Press, 163–168. doi:10.1109/ITSC.2016.7795548

    arXiv 2016

  35. [36]

    Dmytro Humeniuk, Foutse Khomh, and Giuliano Antoniol. 2023. AmbieGen: A search-based framework for au- tonomous systems testing.Science of Computer Programming230 (2023), 102990. doi:10.1016/j.scico.2023.102990

    arXiv 2023

  36. [37]

    Nidhi Kalra and Susan M. Paddock. 2016.Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?RAND Corporation

    2016

  37. [38]

    Marc Kaufeld, Rainer Trauth, and Johannes Betz. 2024. Investigating Driving Interactions: A Robust Multi-Agent Simulation Framework for Autonomous Vehicles. In2024 IEEE Intelligent Vehicles Symposium (IV). 803–810. doi:10. 1109/IV55156.2024.10588423

    arXiv 2024

  38. [39]

    Tanmay Khandait, Deyun Lyu, Paolo Arcaini, Georgios Fainekos, Federico Formica, Sauvik Gon, Abdelrahman Hekal, Atanu Kundu, Claudio Menghi, Giulia Pedrielli, Rajarshi Ray, Quinn Thibeault, Masaki Waga, and Zhenya Zhang

  39. [40]

    Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong. 2017. Robust Statistical Methods for Empirical Software Engineering.Empirical Softw. Engg.22, 2 (apr 2017), 579–630. doi:10.1007/s10664-016-9437-5

    doi:10.1007/s10664-016-9437-5 2017

  40. [41]

    Philip Koopman and Michael Wagner. 2016. Challenges in Autonomous Vehicle Testing and Validation.SAE International journal of Transportation Safety4, 1 (apr 2016), 15–24. doi:10.4271/2016-01-0128

    doi:10.4271/2016-01-0128 2016

  41. [42]

    Guanpeng Li, Yiran Li, Saurabh Jha, Timothy Tsai, Michael Sullivan, Siva Kumar Sastry Hari, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2020. AV-FUZZER: Finding Safety Violations in Autonomous Driving Systems. In2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). 25–36. doi:10.1109/ISSRE5003.2020.00012

    arXiv 2020

  42. [43]

    Yuanfei Lin, Michael Ratzel, and Matthias Althoff. 2023. Automatic Traffic Scenario Conversion from OpenSCENARIO to CommonRoad. In25th IEEE International Conference on Intelligent Transportation Systems, ITSC 2022, Macau, China, October 8-12, 2022. IEEE, 4941–4946. doi:10.1109/ITSC57777.2023.10422422

    arXiv 2023

  43. [45]

    Chengjie Lu, Yize Shi, Huihui Zhang, Man Zhang, Tiexin Wang, Tao Yue, and Shaukat Ali. 2023. Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their Collisions.IEEE Transactions on Software Engineering49, 1 (2023), 384–402. doi:10.1109/TSE.2022.3150788

    arXiv 2023

  44. [46]

    Yixing Luo, Xiao-Yi Zhang, Paolo Arcaini, Zhi Jin, Haiyan Zhao, Fuyuki Ishikawa, Rongxin Wu, and Tao Xie. 2021. Targeting Requirements Violations of Autonomous Driving Systems by Dynamic Evolutionary Search. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 279–291. doi:10.1109/ASE51524.2021. 9678883

    arXiv 2021

  45. [47]

    Gerkey, Chris Lalancette, and William Woodall

    Steven Macenski, Tully Foote, Brian P. Gerkey, Chris Lalancette, and William Woodall. 2022. Robot Operating System 2: Design, architecture, and uses in the wild.Sci. Robotics7, 66 (2022). doi:10.1126/SCIROBOTICS.ABM6074

    doi:10.1126/scirobotics.abm6074 2022

  46. [48]

    Sakr, Tamer Mostafa, and Khaled El-Bahnasy

    Nehal Magdy, Mahmoud A. Sakr, Tamer Mostafa, and Khaled El-Bahnasy. 2015. Review on trajectory similarity measures. In2015 IEEE Seventh International Conference on Intelligent Computing and Information Systems (ICICIS). 613–619. doi:10.1109/IntelCIS.2015.7397286

    arXiv 2015

  47. [49]

    Jiri Opletal. 2025. China’s massive ADAS test: 36 cars, 15 hazard scenarios, 216 crashes. https://carnewschina.com/ 2025/07/24/chinas-massive-adas-test-36-cars-15-hazard-scenarios-216-crashes/. 28 Cavaleri et al

    2025

  48. [50]

    Qi Pan, Tiexin Wang, Jianwei Ma, Paolo Arcaini, and Tao Yue. 2026. Simulation-based Safety Assessment of Vehicle Characteristics Variations in Autonomous Driving Systems.ACM Trans. Softw. Eng. Methodol.35, 3, Article 81 (Feb. 2026), 30 pages. doi:10.1145/3743673

    doi:10.1145/3743673 2026

  49. [51]

    2009.Causality

    Judea Pearl. 2009.Causality. Cambridge university press

    2009

  50. [52]

    Fabian Poggenhans, Jan-Hendrik Pauls, Johannes Janosovits, Stefan Orf, Maximilian Naumann, Florian Kuhnt, and Matthias Mayr. 2018. Lanelet2: A high-definition map framework for the future of automated driving. In21st International Conference on Intelligent Transportation Systems, ITSC 2018, Maui, HI, USA, November 4-7, 2018, Wei-Bin Zhang, Alexandre M. Ba...

    arXiv 2018

  51. [53]

    Morgan Quigley, Brian Gerkey, Ken Conley, Josh Faust, Tully Foote, Jeremy Leibs, Eric Berger, Rob Wheeler, and Andrew Y. Ng. 2009. ROS: an open-source Robot Operating System. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA) Workshop on Open Source Robotics

    2009

  52. [54]

    Franz Scheuer, Alessio Gambi, and Paolo Arcaini. 2023. STRETCH: Generating Challenging Scenarios for Testing Collision Avoidance Systems. In2023 IEEE Intelligent Vehicles Symposium (IV). 1–6. doi:10.1109/IV55152.2023.10186634

    arXiv 2023

  53. [55]

    Hans-Peter Schöner. 2018. Simulation in development and testing of autonomous vehicles. In18. Internationales Stuttgarter Symposium, Michael Bargende, Hans-Christian Reuss, and Jochen Wiedemann (Eds.). Springer Fachmedien Wiesbaden, Wiesbaden, 1083–1095

    2018

  54. [56]

    Software Aurora Lab. 2023. DoppelTest: Replication Package. https://github.com/Software-Aurora-Lab/DoppelTest. Last access: June, 2026

    2023

  55. [57]

    Shuncheng Tang, Zhenya Zhang, Yi Zhang, Jixiang Zhou, Yan Guo, Shuang Liu, Shengjian Guo, Yan-Fu Li, Lei Ma, Yinxing Xue, and Yang Liu. 2023. A Survey on Automated Driving System Testing: Landscapes and Trends.ACM Trans. Softw. Eng. Methodol.32, 5, Article 124 (jul 2023), 62 pages. doi:10.1145/3579642

    doi:10.1145/3579642 2023

  56. [58]

    Shuncheng Tang, Zhenya Zhang, Jixiang Zhou, Lei Lei, Yuan Zhou, and Yinxing Xue. 2024. LeGEND: A Top-Down Approach to Scenario Generation of Autonomous Driving Systems Assisted by Large Language Models. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacramento, CA, USA)(ASE ’24). Association for Computing Mac...

    arXiv 2024

  57. [59]

    Haoxiang Tian, Yan Jiang, Guoquan Wu, Jiren Yan, Jun Wei, Wei Chen, Shuo Li, and Dan Ye. 2022. MOSAT: finding safety violations of autonomous driving systems using multi-objective genetic algorithm. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore)(E...

    arXiv 2022

  58. [60]

    Haoxiang Tian, Guoquan Wu, Jiren Yan, Yan Jiang, Jun Wei, Wei Chen, Shuo Li, and Dan Ye. 2023. Generating Critical Test Scenarios for Autonomous Driving Systems via Influential Behavior Patterns. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering(Rochester, MI, USA)(ASE ’22). Association for Computing Machinery, ...

    arXiv 2023

  59. [61]

    Rainer Trauth, Korbinian Moller, Gerald Würsching, and Johannes Betz. 2024. FRENETIX: A High-Performance and Modular Motion Planning Framework for Autonomous Driving.IEEE Access(2024), 1–1. doi:10.1109/ACCESS.2024. 3436835

    doi:10.1109/access.2024 2024

  60. [62]

    Martin Treiber, Ansgar Hennecke, and Dirk Helbing. 2000. Congested traffic states in empirical observations and microscopic simulations.Phys. Rev. E62 (Aug 2000), 1805–1824. Issue 2. doi:10.1103/PhysRevE.62.1805

    doi:10.1103/physreve.62.1805 2000

  61. [63]

    Ziyu Wang, Jing Ma, and Edmund M-K Lai. 2024. A Survey of Scenario Generation for Automated Vehicle Testing and Validation.Future Internet16 (2024). doi:10.3390/fi16120480

    work page doi:10.3390/fi16120480 2024

  62. [64]

    WEF. 2025. Autonomous Vehicles: Timeline and Roadmap Ahead. https://www.weforum.org/publications/autonomous- vehicles-timeline-and-roadmap-ahead/

    2025

  63. [65]

    Moritz Werling, Julius Ziegler, Sören Kammel, and Sebastian Thrun. 2010. Optimal trajectory generation for dynamic street scenarios in a Frenét Frame. InIEEE International Conference on Robotics and Automation, ICRA 2010, Anchorage, Alaska, USA, 3-7 May 2010. IEEE, 987–993. doi:10.1109/ROBOT.2010.5509799

    arXiv 2010

  64. [66]

    Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, and Paolo Tonella. 2021. DeepHyperion: exploring the feature space of deep learning-based systems through illumination search. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis(Virtual, Denmark)(ISSTA 2021). Association for Computing Machinery, New York, NY, U...

    arXiv 2021

  65. [67]

    Tahereh Zohdinasab, Vincenzo Riccio, and Paolo Tonella. 2024. Focused Test Generation for Autonomous Driving Systems.ACM Trans. Softw. Eng. Methodol.33, 6, Article 152 (June 2024), 32 pages. doi:10.1145/3664605

    work page doi:10.1145/3664605 2024

  66. [2023]

    InProceedings of the 45th International Conference on Software Engineering(Melbourne, Victoria, Australia)(ICSE ’23)

    Doppelgänger Test Generation for Revealing Bugs in Autonomous Driving Software. InProceedings of the 45th International Conference on Software Engineering(Melbourne, Victoria, Australia)(ICSE ’23). IEEE Press, 2591–2603. doi:10.1109/ICSE48619.2023.00216

    arXiv 2023

  67. [2025]

    InProceedings of 12th Int

    ARCH-COMP25 Category Report: Falsification. InProceedings of 12th Int. Workshop on Applied Verification for Continuous and Hybrid Systems (EPiC Series in Computing, Vol. 108), Goran Frehse and Matthias Althoff (Eds.). EasyChair, 169–189. doi:10.29007/dgnn

    doi:10.29007/dgnn