Pith. sign in

REVIEW 5 major objections 5 minor 114 references

Using Cooperative Co-evolutionary Search to Generate Metamorphic Test Cases for Autonomous Driving Systems

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining metamorphic relations with a two-population cooperative search lets CoCoMEGA find 83 percent more distinct unsafe-driving scenarios than a standard genetic algorithm and 87 percent more than random search under the same…

desk verdict A careful and novel integration of MT and CCEA for ADS testing, with a replication package and mostly honest reporting, though the 'consistent' superiority claim needs softening. read the letter →

arxiv 2412.03843 v3 pith:OQZJAU2Q submitted 2024-12-05 cs.SE cs.NE

Hossein Yousefizadeh , Shenghui Gu , Lionel C. Briand , Ali Nasr This is my paper · ORCID
classification cs.SEcs.NE
keywords autonomousdrivingsystemssystem-leveltestingmetamorphicsearch-basedcooperativeco-evolutionaryalgorithmsautomatedtestoracle
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the central obstacle to testing autonomous-driving systems in simulation—knowing what 'bad' behavior looks like without a complete specification—can be handled by using metamorphic relations as partial test oracles and searching for violations of those relations. Its proposed method, CoCoMEGA, evolves a population of driving scenarios and a population of metamorphic perturbations in parallel, scoring each pairing by how strongly the system under test breaks a prescribed behavioral relation, such as slowing down by at least 20 percent when a pedestrian appears or keeping the steering angle fixed when fog density changes. Within a fixed simulation budget on the CARLA simulator and the InterFuser ADS, CoCoMEGA reports 83 percent more distinct violating scenarios than a standard genetic algorithm and 87 percent more than random search, along with larger relative gains in covering the defined relations. If these results hold, automated simulator-based testing can catch subtle undesirable behaviors that do not yet cause collisions, and do so more cheaply than unguided search.

What carries the argument

The machinery has three parts. First, the metamorphic relation (MR) acts as the oracle: an input relation (for example, a pedestrian appears in front of the ego vehicle) paired with an output relation (speed must drop by at least 20 percent, or steering must stay within one degree), and a violation is a deviation from that relation. Second, the fitness signal is the extent of violation $E(s, q)$: dynamic time warping with a Sakoe-Chiba constraint aligns the time series of source and follow-up scenarios, matched pairs inside a critical interval are retained, and the average deviation $dif\!f_{or}$ over those pairs quantifies severity. Third, a cooperative co-evolutionary algorithm maintains two populations—scenarios and perturbations—that are bred separately with custom crossover and mutation operators but assessed jointly, with fitness clearing (using a dynamically computed clearing radius) and pure-diversity-based archive updates keeping the solutions spread across the search space.

What would settle it

Replay the archived MR-violating scenarios in an independent setting (a different simulator or a closed-course test) and have new raters score a random sample that includes the mild violations, not only the 50 most severe ones; if the mild violations are mostly judged harmless and severity scores do not track standard proximity metrics such as time-to-collision, the oracle calibration—not the search—is the limiting factor.

Watch

Extended reading notes

Core claim

CoCoMEGA's central claim is that test generation for an ADS can be reformulated as a search over scenario–perturbation pairs that violate metamorphic relations. The paper formalizes this as finding a diverse set of pairs $(s, q)$ with violation extent $E(s, q) > 0$, where $E(s, q)$ is computed from simulator time series: dynamic time warping aligns the source and follow-up trajectories, aligned pairs inside a critical interval are kept, and the average deviation from the output relation (invariance, increasing, or decreasing) gives the severity. Because the input space is high-dimensional and each simulator run is expensive, the method splits the search into two co-evolving populations—source scenarios and perturbations made of metamorphic transformations—that collaborate to form complete test cases. Individual fitness is the best joint severity an individual achieves with members of the other population, and two diversity mechanisms, fitness clearing with a dynamic niche radius and archive selection by a pure-diversity metric, keep the returned set spread out. Evaluated on CARLA with InterFuser, the paper reports that this design finds more distinct, severe, and behaviorally diverse MR violations than a standard genetic algorithm or random search, and that the diversity mechanisms alone account for a 91 percent gain in distinct violations.

Load-bearing premise

Everything depends on the metamorphic relations being faithful proxies for real risk: the output thresholds (a 20 percent speed-reduction and a one-degree steering change) were set by literature review plus a pilot study, so if those thresholds do not track genuine safety concerns, the search objective and the effectiveness metrics are measuring violations of a calibrated proxy, not unsafe driving.

Editorial extensions

If this is right

  • With the same simulation budget used by the baselines, testers can obtain roughly twice as many distinct severe MR-violating scenarios, so expensive simulator time is converted into more failure-revealing tests.
  • The defined speed-reduction relations can be covered almost completely (100 percent at the lowest fitness threshold, 56 percent even at the strictest threshold tested), so a single search session exercises most of the oracle set rather than one relation at a time.
  • Because MRs need not be perfect or complete to guide the search, the method transfers to systems where no full behavioral specification exists, provided a domain expert can state relations like 'steering should not change when weather changes'.
  • Removing the diversity mechanisms collapses the gain (91 percent fewer distinct violations), so preserving spread across scenarios and perturbations is what makes the co-evolutionary search useful rather than merely guided.
  • Since the scenario representation is tied to what a simulator exposes and compatible with standard formats, the framework can be ported to other OpenScenario-compatible simulation platforms without changing the search core.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct calibration test would correlate the 20 percent speed-reduction threshold with established proximity metrics such as time-to-collision in the same scenarios; if near-threshold violations never coincide with metric deterioration, the oracle is measuring a proxy rather than risk.
  • The scenario–perturbation split mirrors a general divide in cyber-physical testing between ambient context and imposed disturbances, so the same two-population scheme may speed up failure search for other autonomous systems with expensive simulators.
  • The paper's AUC advantage over baselines widens at high fitness thresholds, which suggests the method's real strength is concentrating a limited budget on severe violations while mild anomalies remain better found by random spread.
  • Because the archive accumulates labeled (scenario, perturbation) pairs with measured violation extents, it doubles as training data for the surrogate models the paper names as future work, which could cut the dominant two-minute-per-run simulation cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces CoCoMEGA, a framework that combines metamorphic testing (MT) with cooperative co-evolutionary search to generate diverse, MR-violating test scenarios for autonomous driving systems. The approach maintains separate populations of source scenarios and metamorphic perturbations, evaluates complete solutions using a joint fitness function based on DTW-aligned time series, and employs fitness clearing and diversity-aware archiving. The empirical evaluation in CARLA with the InterFuser ADS compares CoCoMEGA against random search (RS) and a standard genetic algorithm (SGA) across three MR groups (GP1, GP2, GP3), measuring distinct solutions, solution diversity, MR coverage, and search-budget efficiency. The paper reports that CoCoMEGA finds 83% more distinct MR-violating solutions than SGA and 87% more than RS, with AUC efficiency gains of 46% and 32% respectively for DS, and 90% and 86% for MRC. An ablation study on the diversity mechanisms and expert ratings of 50 severe cases are also presented.

Significance. If the results hold, CoCoMEGA is a relevant contribution to search-based testing of ADSs, addressing the oracle problem via MT and leveraging co-evolution to improve search efficiency. The paper has notable strengths: 10 repeated runs with 95% confidence intervals for most experiments, an ablation isolating the diversity mechanisms, a publicly available replication package, and a clear reporting of the MR sources. However, the central claim that CoCoMEGA 'consistently outperforms' the baselines is qualified by the paper's own data at low fitness thresholds combined with high distance thresholds, and the lack of a simulator-noise baseline weakens the evidence that the detected MR violations correspond to genuine behavioral anomalies. The expert validation is limited to a selected subset of the most severe cases, so the safety relevance of the broader solution set remains unconfirmed. These issues are correctable and do not undermine the fundamental approach, but they need to be addressed before the claims can be accepted as stated.

major comments (5)
  1. [§V-C2 and Fig. 3; §V-D2 and Fig. 6] The abstract and Section VII state that CoCoMEGA 'consistently outperforms' the baseline methods, but the paper's own results contain explicit exceptions. In Section V-C2, the authors acknowledge that at low fitness thresholds (θf ≤ 0.8) combined with high distance thresholds (θd ≥ 1.2), SGA and RS obtain higher DS values than CoCoMEGA. Similarly, in the RQ2 results (Fig. 6), at θf = 0.5 and θd = 1.5, RS outperforms CoCoMEGA. The 'consistently' claim is therefore not supported by the presented evidence. I recommend revising the wording to qualify the claim (e.g., 'consistently outperforms for severe violations, θf ≥ 1.0') and discussing these exceptions in the conclusion rather than only in the middle of the results section.
  2. [§V-B2 and §VI-C1] The MR output thresholds (e.g., 20% speed reduction for GP1, 1-degree steering invariance for GP2/GP3) are stated to be 'derived from the established literature and refined through a pilot experiment,' but no control experiment is reported to establish a noise floor for the fitness function E(s, q) in Eq. (2). If CARLA or InterFuser exhibits run-to-run nondeterminism, E(s, q) could be non-zero even for identical scenario-perturbation pairs or for no-op perturbations, and thresholds calibrated only on a pilot set might be below the simulator's noise floor. This would affect both the search objective (maximizing E) and the effectiveness metrics (counting solutions with E > θf). To address this load-bearing concern, I request a supplementary analysis that (i) replays a sample of scenario-perturbation pairs multiple times under identical conditions and (ii) evaluates no-op perturbations, reporting the distribution of E(s, q). The thresholds should then be justified as exceeding this noise baseline.
  3. [§VI-A] The expert feedback in Section VI-A is based on only the 50 most severe violations found by CoCoMEGA, selected across its 10 executions. This selection is not a random sample nor does it include violations generated by the baseline methods. Consequently, the ratings cannot support the claim that CoCoMEGA's advantage over RS and SGA reflects genuinely more safety-relevant violations; the experts did not compare methods or rate the full distribution of outputs. I recommend either rating a random sample of CoCoMEGA's solutions covering the full range of fitness values, or rating matched samples from CoCoMEGA, SGA, and RS with the experts blind to the generating method. This would provide the missing evidence that the measured MR violations are practically meaningful and that the reported improvements are not artifacts of a calibrated proxy.
  4. [§V-B3] The experimental comparison is affected by an asymmetry in hyperparameter tuning. For SGA and RS, the paper uses 'widely recommended' values (mutation 0.2, crossover 0.8, tournament size 3), while for CoCoMEGA the hyperparameters (e.g., maximum archive size 3) were tuned via a pilot experiment because 'there are no suggested values for CCEAs.' Since the research questions compare CoCoMEGA with SGA to isolate the effect of cooperative co-evolution, the tuned CoCoMEGA configuration against an untuned SGA configuration confounds the algorithmic choice with the hyperparameter tuning effort. I suggest tuning SGA (or at least adding a tuned-SGA variant) or providing evidence that the chosen SGA parameters are strong for this problem, so that the reported 83% improvement in DS is a fair comparison.
  5. [§V-C2 (GP2 results)] The results for GP2 show that none of the three methods found any MR violation for the environmental invariance relations (MR6, MR7). This means the reported effectiveness and efficiency findings are based entirely on GP1 (speed reduction) and GP3 (actor invariance). The paper mentions this result in passing but the conclusions do not temper the generality claims. Since a substantial portion of the MRs (6 of 13) produced zero violations, the claim of 'broader exploration of the test space' is only supported for the decreasing-relation family, not for invariance relations. I recommend explicitly stating in the discussion and conclusion that the approach was effective for the decreasing and actor-invariance MR groups, while the environmental-invariance relations were not violated by the subject ADS, and discussing what this implies for the general applicability of the method.
minor comments (5)
  1. [Throughout] The name of the ADS is spelled inconsistently as 'INTERFUSER' in the abstract and elsewhere, 'InterFuser' in Section V-B1, and 'INTER FUSER' in Section V-B1 and the references. Please unify the spelling.
  2. [Tables II, III, and IV] The MRC and CMR values are presented in a single cell (e.g., '100.0 ± 0.0 (3.0 ± 0.3)'). The caption indicates the parentheses contain CMR, but this is easy to miss, especially when scanning the tables. I suggest splitting these into two clear columns for readability.
  3. [Eq. (2)] In Definition 3, the notation Tci is introduced in the text but the equation uses Tci without an explicit definition of the subscript symbol; please place the definition of Tci directly under the equation to improve clarity.
  4. [Section I (Contributions)] The first bullet states 'We proposed CoCoMEGA, the first automated testing method that combines MT and CCEA.' Since establishing novelty is difficult and the related work section does not provide an exhaustive search over combinations of these techniques, I recommend softening this to 'to the best of our knowledge' to avoid overclaiming.
  5. [Section V-D2 and Fig. 8] The p-values reported (e.g., p < 10^-31) are extremely small for experiments with 10 runs per method and integer-valued DS counts, suggesting possible over-dispersion or that the effect size should be reported in addition to the p-value. I recommend reporting effect sizes (e.g., Cliff's delta or Vargha-Delaney A) alongside the significance tests to give the reader a sense of magnitude.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CoCoMEGA's effectiveness claims rest on an external comparison against RS and SGA under a shared MR oracle, with thresholds pilot-calibrated before the main study and expert validation providing an independent check.

full rationale

The paper's central claim is that CoCoMEGA finds more severe and diverse MR-violating test cases than SGA and RS. The search objective E(s,q) in Eq. (2) and the effectiveness metrics (DS, MRC) both count MR violations, but this is ordinary oracle-based evaluation rather than circular derivation: the metric is the objective by design, and the external comparison against two baselines on the same oracle is what carries the claim. The MR thresholds are explicitly stated in Section V-B2 to be 'derived from the established literature and refined through a pilot experiment conducted prior to the main study,' so they are not fitted to the main study's data, and the same thresholds are applied to all methods, making the comparison fair rather than forced. The paper also explicitly disclaims the MR compilation as a contribution, stating in Section I that 'this compilation of MRs is simply used to support our experimental evaluation on a rich and diverse set of MRs that were defined independently from our study,' which removes the main self-definitional risk. The few self-citations involving co-author Briand (e.g., [19], [26], [66], [105]) appear in related work or for standard design choices, and the CCEA foundations are cited from external sources such as [23] and [61], so no load-bearing claim reduces to a self-citation. The limitation acknowledged in Section VI-C1 that 'an inaccurate threshold in an output relation could either fail to detect a true violation or detect one where there is not' is an internal validity threat, not a circular step; similarly, the absence of a no-perturbation noise baseline is a correctness risk about whether thresholds exceed simulator noise, but it does not make the reported comparison equivalent to its inputs by construction. The expert validation in Section VI-A covers only the 50 most severe cases, which limits generalization, but it is an independent external check rather than a circular derivation. Overall, the derivation chain is self-contained with respect to the stated effectiveness comparison.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method rests on calibrated MR thresholds, unreported algorithmic constants, and a simulator/model pairing; these are the main costs the reader pays beyond the authors' code.

free parameters (6)
  • MR output thresholds (speed reduction 20%, steering invariance 1 degree, t1/t2 ranges) = 20%; 1 degree; t1 and t2 not reported
    Defined from literature and 'refined through a pilot experiment' (Section V-B2). These thresholds define what counts as an MR violation, so they directly shape the search objective and all effectiveness metrics.
  • CoCoMEGA hyperparameters (population size 7, tournament size 3, crossover 0.8, mutation 0.2, max archive size 3) = 7; 3; 0.8; 0.2; 3
    Chosen from literature or pilot experiments (Section V-B3). No sensitivity analysis is provided, so the reported advantage could be specific to these settings.
  • Niche capacity kappa in fitness clearing = not reported
    Used in Algorithm 3 and Definition 6; the paper does not state the value used in the experiments.
  • Evaluation thresholds theta_f and theta_d in DS/SD metrics = theta_f in {0.3,0.5,0.8,1.0,1.3,1.5}; theta_d in [0.0,1.7]
    Used to define distinct solutions and diversity (Section V-C1). The range is justified by observed distributions, but the choice affects the reported magnitudes of improvement.
  • DTW Sakoe-Chiba warping window and critical interval parameters = not specified
    The violation extent depends on the warping window and critical interval definition (Section IV-B), but exact windows are not given.
  • Mutation probabilities (eta for add/remove objects) = not reported
    Scenario mutation uses a configurable hyperbolic distribution with base probability eta (Section IV-D2), but eta is not reported.
assumptions (5)
  • domain assumption CARLA and Interfuser faithfully represent real ADS behavior in the evaluated scenarios.
    The paper draws general conclusions about ADS testing from one simulator and one ADS, acknowledging this threat in Section VI-C2.
  • domain assumption The MRs from previous literature are valid checks of unsafe behavior for this system.
    MRs are the only oracle; if they are wrong, violations are meaningless. The paper relies on literature and pilot calibration (Section V-B2), with expert feedback only on 50 severe cases.
  • standard math DTW with Sakoe-Chiba constraint provides a meaningful comparison of desynchronized ego-vehicle time series.
    Used to define the extent of violation (Definition 2); assumes the optimal alignment reflects real behavioral differences and that the critical interval filter removes false violations.
  • ad hoc to paper The dynamic clearing radius formula sigma(n) = HD_max(n)/(2N(n)) maintains useful diversity without manual tuning.
    This is a heuristic borrowed from Tan et al. [84] and adapted; its behavior in this high-dimensional scenario space is not characterized.
  • standard math Fitness clearing and Pure Diversity archives preserve a representative spread of solutions.
    These mechanisms are standard but assume the heterogeneous distance function (Eq. 4) captures meaningful differences between scenarios and perturbations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using Cooperative Co-evolutionary Search to Generate Metamorphic Test Cases for Autonomous Driving Systems." pith.science (2026). https://pith.science/paper/OQZJAU2Q

@misc{pith2026241203843,
  author       = {Pith},
  title        = {Pith review of: Using Cooperative Co-evolutionary Search to Generate Metamorphic Test Cases for Autonomous Driving Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQZJAU2Q}},
  note         = {Machine review of arXiv:2412.03843}
}
read the original abstract

Autonomous Driving Systems (ADSs) rely on Deep Neural Networks, allowing vehicles to navigate complex, open environments. However, the unpredictability of these scenarios highlights the need for rigorous system-level testing to ensure safety, a task usually performed with a simulator in the loop. Though one important goal of such testing is to detect safety violations, there are many undesirable system behaviors, that may not immediately lead to violations, that testing should also be focusing on, thus detecting more subtle problems and enabling a finer-grained analysis. This paper introduces Cooperative Co-evolutionary MEtamorphic test Generator for Autonomous systems (CoCoMEGA), a novel automated testing framework aimed at advancing system-level safety assessments of ADSs. CoCoMEGA combines Metamorphic Testing (MT) with a search-based approach utilizing Cooperative Co-Evolutionary Algorithms (CCEA) to efficiently generate a diverse set of test cases. CoCoMEGA emphasizes the identification of test scenarios that present undesirable system behavior, that may eventually lead to safety violations, captured by Metamorphic Relations (MRs). When evaluated within the CARLA simulation environment on the Interfuser ADS, CoCoMEGA consistently outperforms baseline methods, demonstrating enhanced effectiveness and efficiency in generating severe, diverse MR violations and achieving broader exploration of the test space. These results underscore CoCoMEGA as a promising, more scalable solution to the inherent challenges in ADS testing with a simulator in the loop. Future research directions may include extending the approach to additional simulation platforms, applying it to other complex systems, and exploring methods for further improving testing efficiency such as surrogate modeling.

Figures

Figures reproduced from arXiv: 2412.03843 by the authors.

Figure 1
Figure 1. The scenario domain model for the case study. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Overview of CoCoMEGA. Algorithm 3: assessFitness Input: Population of scenarios Ps Population of perturbations Pq Archive of scenarios Xs Archive of perturbations Xq Archive of complete solutions X Niche capacity κ Output: Updated population of scenarios Ps Updated population of perturbations Pq Updated archive of complete solutions X 1 Set of complete solutions CS ← collaborate(Ps, Pq, Xs, Xq); 2 foreach Complete s… view at source ↗
Figure 3
Figure 3. Distinct Solutions (DS) vs. distance threshold (θd) across different fitness thresholds (θf ). A higher DS indicates more distinct violating solutions discovered by the method. Each curve plots the average DS at different θd settings, under a specific fitness threshold θf . This figure reveals how each method balances the quantity and diversity of solutions as θf varies. on applied fitness thresholds θf and distance… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Average Pairwise Distance (APD) vs. fitness threshold (θf ) across different distance thresholds (θd). APD quantifies how spread out the solutions are. A higher APD indicates greater diversity among the found solutions. Each subplot corresponds to a certain distance th…
Figure 5
Figure 5. Figure 5: Evolution of Average Pairwise Distance (APD) across generations. Higher APD values imply that solutions are more spread out in the search space, indicating broad exploration. Subplots differ by the chosen fitness threshold (θf ) and distance threshold (θd). Each curve …
Figure 6
Figure 6. Figure 6: Distinct Solutions (DS) vs. search budget. A higher DS indicates more distinct violating solutions discovered by the method. Each subplot shows DS at incremental percentages of the total simulation budget. Different curves represent different methods, compared in disti…
Figure 7
Figure 7. Figure 7: Metamorphic Relation Coverage (MRC) vs. search budget. MRC indicates the percentage of defined MRs that are violated by at least one discovered solution. Higher MRC means the method finds solutions that violate more of the predefined MRs, indicating thorough behavioral…
Figure 8
Figure 8. Figure 8: Comparison of computational efficiency (execution time) for [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Comparison of CoCoMEGA with and without diversity mechanisms: Distinct Solutions (DS) vs. distance threshold (θd) across different fitness thresholds (θf ). A higher DS indicates more distinct violating solutions discovered by the method. Each curve plots the average D…
Figure 10
Figure 10. Figure 10: Comparison of CoCoMEGA with and without diversity mechanisms: Average Pairwise Distance (APD) vs. fitness threshold (θf ) across different distance thresholds (θd). APD quantifies how spread out the solutions are. A higher APD indicates greater diversity among the fou…
Figure 11
Figure 11. Figure 11: Combined histograms of expert ratings for the 50 test cases: Left [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Distinct Solutions (DS) vs. distance threshold (θd) across different fitness thresholds (θf ). A higher DS indicates more distinct violating solutions discovered by the method. Each curve plots the average DS at different θd settings, under a specific fitness threshol…
Figure 13
Figure 13. Figure 13: Average Pairwise Distance (APD) vs. fitness threshold (θf ) across different distance thresholds (θd). APD quantifies how spread out the solutions are. A higher APD indicates greater diversity among the found solutions. Each subplot corresponds to a certain distance t…
Figure 14
Figure 14. Figure 14: Evolution of Average Pairwise Distance (APD) across generations. Higher APD values imply that solutions are more spread out in the search space, indicating broad exploration. Subplots differ by the chosen fitness threshold (θf ) and distance threshold (θd). Each line …
Figure 15
Figure 15. Figure 15: Distinct Solutions (DS) vs. search budget. A higher DS indicates more distinct violating solutions discovered by the method. Each subplot shows DS at incremental percentages of the total simulation budget. Different curves represent different methods, compared in dist…
Figure 16
Figure 16. Figure 16: Metamorphic Relation Coverage (MRC) vs. search budget. MRC indicates the percentage of defined MRs that are violated by at least one discovered solution. Higher MRC means the method finds solutions that violate more of the predefined MRs, indicating thorough behaviora…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

114 extracted references · 18 canonical work pages

  1. [1]

    Coverage based testing for v&v and safety assurance of self-driving autonomous vehicles: A systematic literature review,

    Z. Tahir and R. Alexander, “Coverage based testing for v&v and safety assurance of self-driving autonomous vehicles: A systematic literature review,” in 2020 IEEE International Conference On Artificial Intelligence Testing , ser. AITest ’20. Institute of Electrical and Electronics Engineers (IEEE), Aug. 2020, pp. 23–30. [Online]. Available: http://dx.doi....

    arXiv 2020

  2. [2]

    DeepTest: automated testing of deep-neural-network-driven autonomous cars,

    Y . Tian, K. Pei, S. Jana, and B. Ray, “DeepTest: automated testing of deep-neural-network-driven autonomous cars,” in Proceedings of the 40th International Conference on Software Engineering , ser. ICSE ’18. New York, NY , USA: Association for Computing Machinery, May 2018, pp. 303–314. [Online]. Available: http: //dx.doi.org/10.1145/3180155.3180220

    arXiv 2018

  3. [3]

    End to end learning for self-driving cars,

    M. Bojarski, “End to end learning for self-driving cars,” arXiv preprint arXiv:1604.07316, 2016

    arXiv 2016

  4. [5]

    Safety in wearable robotic exoskeletons: Design, control, and testing guidelines,

    A. Nasr, K. Inkol, and J. McPhee, “Safety in wearable robotic exoskeletons: Design, control, and testing guidelines,” Journal of Mechanisms and Robotics , pp. 1–13, Oct. 2024. [Online]. Available: https://doi.org/10.1115/1.4066900

    work page doi:10.1115/1.4066900 2024

  5. [6]

    Hands off the wheel in autonomous vehicles?: A systems perspective on over a million miles of field data,

    S. S. Banerjee, S. Jha, J. Cyriac, Z. T. Kalbarczyk, and R. K. Iyer, “Hands off the wheel in autonomous vehicles?: A systems perspective on over a million miles of field data,” in 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, ser. DSN ’18. Institute of Electrical and Electronics Engineers (IEEE), Jun. 2018, pp. 58...

    arXiv 2018

  6. [7]

    [Online]

    List of self-driving car fatalities. [Online]. Available: https: //en.wikipedia.org/wiki/Self-driving car#Incidents

  7. [8]

    (2022) 24 self-driving car statistics & facts

    Carsurance. (2022) 24 self-driving car statistics & facts. [Online]. Available: https://carsurance.net/insights/self-driving-car-statistics/

    2022

  8. [9]

    A comprehensive study of autonomous vehicle bugs,

    J. Garcia, Y . Feng, J. Shen, S. Almanee, Y . Xia, and A. Q. Chen, “A comprehensive study of autonomous vehicle bugs,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE ’20. New York, NY , USA: Association for Computing Machinery, Jun. 2020, pp. 385–396. [Online]. Available: http://dx.doi.org/10.1145/3377811.3380397

    arXiv 2020

Show all 114 references
  1. [10]

    J. D. Musa, Software Reliability Engineering: More Reliable Software Faster and Cheaper. Authorhouse, 2004

    2004

  2. [11]

    A survey on automated driving system testing: Landscapes and trends,

    S. Tang, Z. Zhang, Y . Zhang, J. Zhou, Y . Guo, S. Liu, S. Guo, Y .-F. Li, L. Ma, Y . Xue, and Y . Liu, “A survey on automated driving system testing: Landscapes and trends,” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 5, pp. 1–62, Jul. 2023. [Onlin...

    2023 doi

  3. [12]

    Evaluating decision optimality of autonomous driving via metamorphic testing,

    M. Cheng, Y . Zhou, X. Xie, J. Wang, G. Meng, and K. Yang, “Evaluating decision optimality of autonomous driving via metamorphic testing,” 2024. [Online]. Available: https://arxiv.org/abs/2402.18393

    2024 arXiv

  4. [14]

    Rapidly-exploring random trees for testing automated vehicles,

    C. E. Tuncali and G. Fainekos, “Rapidly-exploring random trees for testing automated vehicles,” in 2019 IEEE Intelligent Transportation Systems Conference , ser. ITSC ’19. Institute of Electrical and Electronics Engineers (IEEE), Oct. 2019, pp. 661–666. [Online]. Available: ht...

    2019

  5. [15]

    Traffic conflict analysis using vehicle tracking system/digital vcr and proposal of a new conflict indicator,

    H. Wakabayashi, Y . Takahashi, S. Niimi, and K. Renge, “Traffic conflict analysis using vehicle tracking system/digital vcr and proposal of a new conflict indicator,” INFRASTRUCTURE PLANNING REVIEW, vol. 20, pp. 949–956, 2003. [Online]. Available: http: //dx.doi.org/10.2208/jo...

    2003 doi

  6. [16]

    A flexible method for criticality assessment in driver assistance systems,

    A. Tamke, T. Dang, and G. Breuel, “A flexible method for criticality assessment in driver assistance systems,” in 2011 IEEE Intelligent Vehicles Symposium, ser. IV ’11. Institute of Electrical and Electronics Engineers (IEEE), Jun. 2011, pp. 697–702. [Online]. Available: http:...

    2011

  7. [17]

    A roadmap for simulation-based testing of autonomous cyber-physical systems: Challenges and future direction,

    C. Birchler, S. Khatiri, P. Rani, T. Kehrer, and S. Panichella, “A roadmap for simulation-based testing of autonomous cyber-physical systems: Challenges and future direction,” 2024. [Online]. Available: https://arxiv.org/abs/2405.01064

    2024 arXiv

  8. [18]

    Challenges in autonomous vehicle testing and validation,

    P. Koopman and M. Wagner, “Challenges in autonomous vehicle testing and validation,” SAE International Journal of Transportation Safety, vol. 4, no. 1, pp. 15–24, Apr. 2016. [Online]. Available: http://www.jstor.org/stable/26167741

    2016

  9. [19]

    Efficient online testing for dnn-enabled systems using surrogate-assisted and many-objective optimization,

    F. U. Haq, D. Shin, and L. Briand, “Efficient online testing for dnn-enabled systems using surrogate-assisted and many-objective optimization,” in Proceedings of the 44th International Conference on Software Engineering , ser. ICSE ’22. New York, NY , USA: Association for Comp...

    2022

  10. [20]

    Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles,

    Z. Zhong, G. Kaiser, and B. Ray, “Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles,” IEEE Transactions on Software Engineering , vol. 49, no. 4, pp. 1860–1875, Apr. 2023. [Online]. Available: http: //dx.doi.org/10.1109/TSE.2022.3195640

    2023

  11. [21]

    software verification and validation of safe autonomous cars: A systematic literature review,

    N. Rajabli, F. Flammini, R. Nardone, and V . Vittorini, “software verification and validation of safe autonomous cars: A systematic literature review,” IEEE Access, vol. 9, pp. 4797–4819, 2021. [Online]. Available: http://dx.doi.org/10.1109/ACCESS.2020.3048047

    2021

  12. [22]

    Survey on scenario-based safety assessment of automated vehicles,

    R. Stefan, T. Ponn, D. Ludwig, B. Schick, and F. Diermeyer, “Survey on scenario-based safety assessment of automated vehicles,” IEEE Access, vol. 8, pp. 87 456–87 477, May 2020. [Online]. Available: http://dx.doi.org/10.1109/ACCESS.2020.2993730

    2020

  13. [23]

    A cooperative coevolutionary approach to function optimization,

    M. A. Potter and K. A. Jong, “A cooperative coevolutionary approach to function optimization,” in Parallel Problem Solving from Nature — PPSN III. Berlin, Heidelberg: Springer Berlin Heidelberg, 1994, pp. 249–257. [Online]. Available: http://dx.doi.org/10.1007/3-540-58484-6 269

    1994 doi

  14. [24]

    ISO 21448:2022, road vehicles–safety of the intended functionality

    ISO. ISO 21448:2022, road vehicles–safety of the intended functionality. [Online]. Available: https://www.iso.org/standard/77490.html

    2022

  15. [25]

    for Standardization of Automation and M

    A. for Standardization of Automation and M. S. (ASAM). ASAM OpenSCENARIO. [Online]. Available: https://www.asam.net/standards/ detail/openscenario/v200/

  16. [26]

    Testing advanced driver assistance systems using multi-objective search and neural networks,

    R. B. Abdessalem, S. Nejati, L. C. Briand, and T. Stifter, “Testing advanced driver assistance systems using multi-objective search and neural networks,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’16. New York, NY...

    2016

  17. [28]

    Compositional falsification of cyber-physical systems with machine learning components,

    T. Dreossi, A. Donz ´e, and S. A. Seshia, “Compositional falsification of cyber-physical systems with machine learning components,” Journal of Automated Reasoning , vol. 63, no. 4, pp. 1031–1053, Jan. 2019. [Online]. Available: http://dx.doi.org/10.1007/s10817-018-09509-5

    2019 doi

  18. [29]

    A V-FUZZER: Finding safety violations in autonomous driving systems,

    G. Li, Y . Li, S. Jha, T. Tsai, M. Sullivan, S. K. S. Hari, Z. Kalbarczyk, and R. Iyer, “A V-FUZZER: Finding safety violations in autonomous driving systems,” in 2020 IEEE 31st International Symposium on Software Reliability Engineering, ser. ISSRE ’20. Institute of Electrical...

    2020

  19. [30]

    Baidu. Apollo. [Online]. Available: https://en.apollo.auto/ apollo-self-driving IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021 28

    2021

  20. [31]

    Fitness function templates for testing automated and autonomous driving systems in intersection scenarios,

    N. Kolb, F. Hauer, and A. Pretschner, “Fitness function templates for testing automated and autonomous driving systems in intersection scenarios,” in 2021 IEEE International Intelligent Transportation Systems Conference , ser. ITSC ’21. Institute of Electrical and Electronics ...

    2021

  21. [32]

    Fitness functions for testing automated and autonomous driving systems,

    F. Hauer, A. Pretschner, and B. Holzm ¨uller, “Fitness functions for testing automated and autonomous driving systems,” in Computer Safety, Reliability, and Security , ser. SAFECOMP ’19. Springer International Publishing, Aug. 2019, pp. 69–84. [Online]. Available: http://dx.do...

    2019 doi

  22. [33]

    Targeting requirements violations of autonomous driving systems by dynamic evolutionary search,

    Y . Luo, X.-Y . Zhang, P. Arcaini, Z. Jin, H. Zhao, F. Ishikawa, R. Wu, and T. Xie, “Targeting requirements violations of autonomous driving systems by dynamic evolutionary search,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’21....

    2021

  23. [34]

    Cost-effective simulation-based test selection in self-driving cars software,

    C. Birchler, N. Ganz, S. Khatiri, A. Gambi, and S. Panichella, “Cost-effective simulation-based test selection in self-driving cars software,” Science of Computer Programming, vol. 226, p. 102926, Mar

  24. [35]

    Single and multi-objective test cases prioritization for self-driving cars in virtual environments,

    C. Birchler, S. Khatiri, P. Derakhshanfar, S. Panichella, and A. Panichella, “Single and multi-objective test cases prioritization for self-driving cars in virtual environments,” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 2, pp. 1–30, Apr

  25. [36]

    Model-based exploration of the frontier of behaviours for deep learning system testing,

    V . Riccio and P. Tonella, “Model-based exploration of the frontier of behaviours for deep learning system testing,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FS...

    2020

  26. [37]

    Available: http://dx.doi.org/10.1145/3533818

    [Online]. Available: http://dx.doi.org/10.1145/3533818

    doi

  27. [38]

    Efficient and effective feature space exploration for testing deep learning systems,

    ——, “Efficient and effective feature space exploration for testing deep learning systems,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 2, pp. 1–38, Mar. 2023. [Online]. Available: http://dx.doi.org/10.1145/3544792

    2023 doi

  28. [39]

    DeepHyperion: Exploring the feature space of deep learning-based systems through illumination search,

    T. Zohdinasab, V . Riccio, A. Gambi, and P. Tonella, “DeepHyperion: Exploring the feature space of deep learning-based systems through illumination search,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA ’21. New Yor...

    2021

  29. [40]

    Analysis of road representations in search-based testing of autonomous driving systems,

    E. Castellano, A. Cetinkaya, and P. Arcaini, “Analysis of road representations in search-based testing of autonomous driving systems,” in 2021 IEEE 21st International Conference on Software Quality, Reliability and Security , ser. QRS ’21. Institute of Electrical and Electroni...

    2021

  30. [41]

    Illuminating search spaces by mapping elites,

    J. Mouret and J. Clune, “Illuminating search spaces by mapping elites,” CoRR, vol. abs/1504.04909, 2015. [Online]. Available: http://arxiv.org/abs/1504.04909

    2015 arXiv

  31. [42]

    Eagle strategy with local search for scenario based validation of autonomous vehicles,

    Q. Goss and M. I. Akba s ¸, “Eagle strategy with local search for scenario based validation of autonomous vehicles,” in 2022 International Conference on Connected Vehicle and Expo , ser. ICCVE ’22. Institute of Electrical and Electronics Engineers (IEEE), Mar. 2022, pp. 1–6. [...

    2022

  32. [43]

    Automatically testing self-driving cars with search-based procedural content generation,

    A. Gambi, M. Mueller, and G. Fraser, “Automatically testing self-driving cars with search-based procedural content generation,” in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA ’19, vol. 4. New York, NY , USA: Associa...

    2019

  33. [44]

    Scenario optimisation and sensitivity analysis for safe automated driving using gaussian processes,

    F. Batsch, A. Daneshkhah, V . Palade, and M. Cheah, “Scenario optimisation and sensitivity analysis for safe automated driving using gaussian processes,” Applied Sciences, vol. 11, no. 2, p. 775, Jan. 2021. [Online]. Available: http://dx.doi.org/10.3390/app11020775

    2021 doi

  34. [45]

    Rapid generation of challenging simulation scenarios for autonomous vehicles based on adversarial test,

    X. Zheng, H. Liang, B. Yu, B. Li, S. Wang, and Z. Chen, “Rapid generation of challenging simulation scenarios for autonomous vehicles based on adversarial test,” in 2020 IEEE International Conference on Mechatronics and Automation, ser. ICMA ’20. Institute of Electrical and El...

    2020

  35. [47]

    Testing of autonomous vehicles using surrogate models and stochastic optimization,

    H. Beglerovic, M. Stolz, and M. Horn, “Testing of autonomous vehicles using surrogate models and stochastic optimization,” in 2017 IEEE 20th International Conference on Intelligent Transportation Systems , ser. ITSC ’17. Institute of Electrical and Electronics Engineers (IEEE)...

    2017

  36. [48]

    Metamorphic testing: A new approach for generating next test cases,

    T. Y . Chen, S. Cheung, and S. Yiu, “Metamorphic testing: A new approach for generating next test cases,” CoRR, vol. abs/2002.12543, Feb. 2020. [Online]. Available: https://arxiv.org/abs/2002.12543

    2002 arXiv

  37. [49]

    Metamorphic testing: A review of challenges and opportunities,

    T. Y . Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys , vol. 51, no. 1, pp. 1–27, Jan

  38. [50]

    MetaSem: metamorphic testing based on semantic information of autonomous driving scenes,

    Z. Yang, S. Huang, T. Bai, Y . Yao, Y . Wang, C. Zheng, and C. Xia, “MetaSem: metamorphic testing based on semantic information of autonomous driving scenes,” Software Testing, Verification and Reliability, vol. 34, no. 5, p. e1878, May 2024. [Online]. Available: http://dx.doi...

    2024 doi

  39. [51]

    Metamorphic testing of driverless cars,

    Z. Q. Zhou and L. Sun, “Metamorphic testing of driverless cars,” Communications of the ACM , vol. 62, no. 3, pp. 61–67, 2019

    2019

  40. [52]

    MetaLiDAR: Automated metamorphic testing of lidar-based autonomous driving systems,

    Z. Yang, S. Huang, C. Zheng, X. Wang, Y . Wang, and C. Xia, “MetaLiDAR: Automated metamorphic testing of lidar-based autonomous driving systems,” Journal of Software: Evolution and Process, vol. 36, no. 7, p. e2644, Dec. 2023. [Online]. Available: http://dx.doi.org/10.1002/smr.2644

    2023 doi

  41. [53]

    Metamorphic testing for autonomous driving systems in fog based on quantitative measurement,

    Y . Pan, H. Ao, and Y . Fan, “Metamorphic testing for autonomous driving systems in fog based on quantitative measurement,” in 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion , ser. QRS-C ’21. Institute of Electrical and Electron...

    2021

  42. [54]

    Metamorphic fuzz testing of autonomous vehicles,

    J. C. Han and Z. Q. Zhou, “Metamorphic fuzz testing of autonomous vehicles,” in Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops , ser. ICSEW ’20. New York, NY , USA: Association for Computing Machinery, Jun. 2020, pp. 380–385. [Onlin...

    2020

  43. [55]

    DeepRoad: Gan-based metamorphic testing and input validation framework for autonomous driving systems,

    M. Zhang, Y . Zhang, L. Zhang, C. Liu, and S. Khurshid, “DeepRoad: Gan-based metamorphic testing and input validation framework for autonomous driving systems,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ser. ASE ’18. New Yo...

    2018

  44. [56]

    Adaptive stress testing for autonomous vehicles,

    M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer, “Adaptive stress testing for autonomous vehicles,” in 2018 IEEE Intelligent Vehicles Symposium , ser. IV ’18. Institute of Electrical and Electronics Engineers (IEEE), Jun. 2018, pp. 1–7. [Online]. Available: http://dx.doi.o...

    2018

  45. [57]

    Adaptive stress testing with reward augmentation for autonomous vehicle validation,

    A. Corso, P. Du, K. R. Driggs-Campbell, and M. J. Kochenderfer, “Adaptive stress testing with reward augmentation for autonomous vehicle validation,” in 2019 IEEE Intelligent Transportation Systems Conference, ser. ITSC ’19. Institute of Electrical and Electronics Engineers (I...

    2019

  46. [58]

    Metamorphic relation generation: State of the art and visions for future research,

    R. Li, H. Liu, P.-L. Poon, D. Towey, C.-A. Sun, Z. Zheng, Z. Q. Zhou, and T. Y . Chen, “Metamorphic relation generation: State of the art and visions for future research,” 2024. [Online]. Available: https://arxiv.org/abs/2406.05397

    2024 arXiv

  47. [59]

    Automatic generation of critical test cases for the development of highly automated driving functions,

    D. Baumann, R. Pfeffer, and E. Sax, “Automatic generation of critical test cases for the development of highly automated driving functions,” in 2021 IEEE 93rd Vehicular Technology Conference, ser. VTC ’21 Spring. Institute of Electrical and Electronics Engineers (IEEE), Apr. 2...

    2021

  48. [60]

    Efficient statistical validation with edge cases to evaluate highly automated vehicles,

    D. Karunakaran, S. Worrall, and E. Nebot, “Efficient statistical validation with edge cases to evaluate highly automated vehicles,” in 2020 IEEE 23rd International Conference on Intelligent IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021 29 Transportati...

    2020

  49. [62]

    Available: http://arxiv.org/abs/1708.06374

    [Online]. Available: http://arxiv.org/abs/1708.06374

    arXiv

  50. [63]

    Archive-based cooperative coevolutionary algorithms,

    L. Panait, S. Luke, and J. F. Harrison, “Archive-based cooperative coevolutionary algorithms,” in Proceedings of the 8th annual conference on Genetic and evolutionary computation , ser. GECCO ’06. New York, NY , USA: Association for Computing Machinery, Jul. 2006, pp. 345–

    2006

  51. [64]

    A declarative metamorphic testing framework for autonomous driving,

    Y . Deng, X. Zheng, T. Zhang, H. Liu, G. Lou, M. Kim, and T. Y . Chen, “A declarative metamorphic testing framework for autonomous driving,” IEEE Transactions on Software Engineering , vol. 49, no. 4, pp. 1964–1982, Apr. 2023. [Online]. Available: http://dx.doi.org/10.1109/TSE...

    1964

  52. [65]

    A survey on cooperative co-evolutionary algorithms,

    X. Ma, X. Li, Q. Zhang, K. Tang, Z. Liang, W. Xie, and Z. Zhu, “A survey on cooperative co-evolutionary algorithms,” IEEE Transactions on Evolutionary Computation , vol. 23, no. 3, pp. 421–441, Jun. 2019. [Online]. Available: http://dx.doi.org/10.1109/TEVC.2018.2868770

    2019

  53. [66]

    Large scale evolutionary optimization using cooperative coevolution,

    Z. Yang, K. Tang, and X. Yao, “Large scale evolutionary optimization using cooperative coevolution,” Information Sciences , vol. 178, no. 15, pp. 2985–2999, Aug. 2008. [Online]. Available: http://dx.doi.org/10.1016/j.ins.2008.02.017

    2008 doi

  54. [67]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning , ser. Proceedings of Machine Learning Research, vol. 78. PMLR, Nov. 2017, pp. 1–16. [Online]. Available: htt...

    2017

  55. [68]

    Scenario-based test reduction and prioritization for multi-module autonomous driving systems,

    Y . Deng, X. Zheng, M. Zhang, G. Lou, and T. Zhang, “Scenario-based test reduction and prioritization for multi-module autonomous driving systems,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engine...

    2022

  56. [69]

    Using dynamic time warping to find patterns in time series,

    D. J. Berndt and J. Clifford, “Using dynamic time warping to find patterns in time series,” in Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining , ser. AAAIWS ’94, vol. 10. Association for the Advancement of Artificial Intelligence (AAAI), ...

  57. [70]

    Can offline testing of deep neural networks replace their online testing?: A case study of automated driving systems,

    F. U. Haq, D. Shin, S. Nejati, and L. Briand, “Can offline testing of deep neural networks replace their online testing?: A case study of automated driving systems,” Empirical Software Engineering, vol. 26, no. 90, Jul. 2021. [Online]. Available: http://dx.doi.org/10.1007/s106...

    2021 doi

  58. [71]

    Identifying the hazard boundary of ml-enabled autonomous systems using cooperative coevolutionary search,

    S. Sharifi, D. Shin, L. C. Briand, and N. Aschbacher, “Identifying the hazard boundary of ml-enabled autonomous systems using cooperative coevolutionary search,” IEEE Transactions on Software Engineering , vol. 49, no. 12, pp. 5120–5138, Dec. 2023. [Online]. Available: http://...

    2023

  59. [72]

    Berlin, Heidelberg: Springer Berlin Heidelberg, 1998, pp

    ——, Evolutionary computing in multi-agent environments: Operators . Berlin, Heidelberg: Springer Berlin Heidelberg, 1998, pp. 43–52. [Online]. Available: http://dx.doi.org/10.1007/BFb0040758

    1998 doi

  60. [73]

    An empirical analysis of collaboration methods in cooperative coevolutionary algorithms,

    R. P. Wiegand, W. C. Liles, and K. A. D. Jong, “An empirical analysis of collaboration methods in cooperative coevolutionary algorithms,” in Proceedings of the genetic and evolutionary computation conference, ser. GECCO ’01, vol. 2611. Morgan Kaufmann, Jul. 2001, pp. 1235–1245

    2001

  61. [74]

    On the role of population size and niche radius in fitness sharing,

    A. D. Cioppa, C. D. Stefano, and A. Marcelli, “On the role of population size and niche radius in fitness sharing,” IEEE Transactions on Evolutionary Computation , vol. 8, no. 6, pp. 580–592, Dec. 2004. [Online]. Available: http://dx.doi.org/10.1109/TEVC.2004.837341

    2004

  62. [75]

    Dynamic programming algorithm optimization for spoken word recognition,

    H. Sakoe and S. Chiba, “Dynamic programming algorithm optimization for spoken word recognition,” IEEE Transactions on Acoustics, Speech, and Signal Processing , vol. 26, no. 1, pp. 43–49, Feb. 1978. [Online]. Available: http://dx.doi.org/10.1109/TASSP.1978.1163055

    1978

  63. [76]

    Evolutionary computation in multi-agent environments: Partners,

    L. Bull, “Evolutionary computation in multi-agent environments: Partners,” in Proceedings of the 7th International Conference on Genetic Algorithms. Morgan Kaufmann, Jul. 1997, pp. 370–377

    1997

  64. [77]

    Fitness sharing and niching methods revisited,

    B. Sareni and L. Krahenbuhl, “Fitness sharing and niching methods revisited,” IEEE Transactions on Evolutionary Computation , vol. 2, no. 3, pp. 97–106, Sep. 1998. [Online]. Available: http://dx.doi.org/10.1109/4235.735432

    1998

  65. [78]

    Selforganization of matter and the evolution of biological macromolecules,

    M. Eigen, “Selforganization of matter and the evolution of biological macromolecules,” Die Naturwissenschaften, vol. 58, no. 10, pp. 465–523, Oct. 1971. [Online]. Available: http://dx.doi.org/10.1007/BF00623322

    1971 doi

  66. [79]

    Horn and D

    J. Horn and D. E. Goldberg, A timing analysis of convergence to fitness sharing equilibrium . Berlin, Heidelberg: Springer Science and Business Media LLC, 1998, pp. 23–33. [Online]. Available: http://dx.doi.org/10.1007/BFb0056846

    1998 doi

  67. [80]

    Genetic algorithms with sharing for multimodal function optimization,

    D. E. Goldberg and J. Richardson, “Genetic algorithms with sharing for multimodal function optimization,” in Genetic algorithms and their applications: Proceedings of the Second International Conference on Genetic Algorithms , vol. 4149. Hillsdale, NJ: Lawrence Erlbaum, 1987

    1987

  68. [81]

    A clearing procedure as a niching method for genetic algorithms,

    A. Petrowski, “A clearing procedure as a niching method for genetic algorithms,” in Proceedings of IEEE International Conference on Evolutionary Computation , ser. ICEC ’96. Institute of Electrical and Electronics Engineers (IEEE), May 1996, pp. 798–803. [Online]. Available: h...

    1996

  69. [82]

    Improved heterogeneous distance functions,

    D. R. Wilson and T. R. Martinez, “Improved heterogeneous distance functions,” Journal of Artificial Intelligence Research , vol. 6, pp. 1–34, Jan. 1997. [Online]. Available: http://dx.doi.org/10.1613/jair.346

    1997 doi

  70. [83]

    A comprehensive survey of evolutionary-based multiobjective optimization techniques,

    C. A. C. Coello, “A comprehensive survey of evolutionary-based multiobjective optimization techniques,” Knowledge and Information Systems, vol. 1, no. 3, pp. 269–308, Aug. 1999. [Online]. Available: http://dx.doi.org/10.1007/BF03325101

    1999 doi

  71. [84]

    An evolutionary algorithm with advanced goal and priority specification for multi-objective optimization,

    K. C. Tan, E. F. Khor, T. H. Lee, and R. Sathikannan, “An evolutionary algorithm with advanced goal and priority specification for multi-objective optimization,” Journal of Artificial Intelligence Research, vol. 18, pp. 183–215, Feb. 2003. [Online]. Available: http://dx.doi.or...

    2003 doi

  72. [85]

    Population size and genetic drift in fitness sharing,

    S. W. Mahfoud, “Population size and genetic drift in fitness sharing,” in Foundations of Genetic Algorithms . Elsevier BV , 1995, vol. 3, pp. 185–223. [Online]. Available: http://dx.doi.org/10.1016/ B978-1-55860-356-1.50014-5

    1995

  73. [86]

    Analysis of the clearing diversity- preserving mechanism,

    E. C. Osuna and D. Sudholt, “Analysis of the clearing diversity- preserving mechanism,” in Proceedings of the 14th ACM/SIGEVO Conference on Foundations of Genetic Algorithms, ser. FOGA ’17. New York, NY , USA: Association for Computing Machinery, Jan. 2017, pp. 55–63. [Online]...

    2017

  74. [87]

    Fault tolerant design using single and multicriteria genetic algorithm optimization,

    J. R. Schott, “Fault tolerant design using single and multicriteria genetic algorithm optimization,” Master’s thesis, Massachusetts Institute of Technology, 1995

    1995

  75. [88]

    On measuring multiobjective evolutionary algorithm performance,

    D. V . Veldhuizen and G. Lamont, “On measuring multiobjective evolutionary algorithm performance,” in Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512) , vol. 1. Institute of Electrical and Electronics Engineers (IEEE), Jul. 2000, pp. 204–2...

  76. [89]

    Comparison of multiobjective evolutionary algorithms: Empirical results,

    E. Zitzler, K. Deb, and L. Thiele, “Comparison of multiobjective evolutionary algorithms: Empirical results,” Evolutionary Computation, vol. 8, no. 2, pp. 173–195, Jun. 2000. [Online]. Available: http://dx.doi.org/10.1162/106365600568202

    doi

  77. [90]

    Luke, Essentials of Metaheuristics , 2nd ed

    S. Luke, Essentials of Metaheuristics , 2nd ed. Lulu, 2013, available for free at http://cs.gmu.edu/ ∼sean/book/metaheuristics/

    2013

  78. [91]

    Quality evaluation of solution sets in multiobjective optimisation: A survey,

    M. Li and X. Yao, “Quality evaluation of solution sets in multiobjective optimisation: A survey,” ACM Computing Surveys , vol. 52, no. 2, pp. 1–38, Mar. 2019. [Online]. Available: http://dx.doi.org/10.1145/3300148

    2019 doi

  79. [92]

    Performance metric ensemble for multiobjective evolutionary algorithms,

    G. G. Yen and Z. He, “Performance metric ensemble for multiobjective evolutionary algorithms,” IEEE Transactions on Evolutionary Computation , vol. 18, no. 1, pp. 131–144, Feb. 2014. [Online]. Available: http://dx.doi.org/10.1109/TEVC.2013.2240687

    2014

  80. [94]

    Simulated binary crossover for continuous search space,

    K. Deb and R. B. Agrawal, “Simulated binary crossover for continuous search space,” Complex systems, vol. 9, no. 2, pp. 115–148, Apr. 1995

    1995

  81. [95]

    Diversity assessment in many- objective optimization,

    H. Wang, Y . Jin, and X. Yao, “Diversity assessment in many- objective optimization,” IEEE Transactions on Cybernetics , vol. 47, no. 6, pp. 1510–1522, Jun. 2017. [Online]. Available: http: //dx.doi.org/10.1109/TCYB.2016.2550502

    2017

  82. [96]

    Diversity assessment of multi-objective evolutionary algorithms: Performance metric and benchmark problems [research frontier],

    Y . Tian, R. Cheng, X. Zhang, M. Li, and Y . Jin, “Diversity assessment of multi-objective evolutionary algorithms: Performance metric and benchmark problems [research frontier],” IEEE Computational Intelligence Magazine, vol. 14, no. 3, pp. 61–74, Aug. 2019. [Online]. Availab...

    2019

  83. [97]

    EvoSuite: On the challenges of test case generation in the real world,

    G. Fraser and A. Arcuri, “EvoSuite: On the challenges of test case generation in the real world,” in 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation . Institute of Electrical and Electronics Engineers (IEEE), Mar. 2013, pp. 362–369. [O...

    2013 doi

  84. [98]

    Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,

    A. Panichella, F. M. Kifetew, and P. Tonella, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” IEEE Transactions on Software Engineering, vol. 44, no. 2, pp. 122–158, Feb. 2018. [Online]. Available: http://dx.doi....

    2018

  85. [99]

    J. H. Holland, Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT press, 1992

    1992

  86. [100]

    EvoSuite: automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “EvoSuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering , ser. ESEC/FSE ’11. New York, NY , USA: Association for ...

    2011

  87. [101]

    Reformulating branch coverage as a many-objective optimization problem,

    A. Panichella, F. M. Kifetew, and P. Tonella, “Reformulating branch coverage as a many-objective optimization problem,” in 2015 IEEE 8th International Conference on Software Testing, IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING, VOL. 14, NO. 8, AUGUST 2021 30 Verification and Va...

    2015

  88. [102]

    CARLA autonomous driving leaderboard

    CARLA. CARLA autonomous driving leaderboard. [Online]. Available: https://leaderboard.carla.org/leaderboard/

  89. [103]

    BMT: Behavior driven development- based metamorphic testing for autonomous driving models,

    Y . Deng, G. Lou, X. Zheng, T. Zhang, M. Kim, H. Liu, C. Wang, and T. Y . Chen, “BMT: Behavior driven development- based metamorphic testing for autonomous driving models,” in 2021 IEEE/ACM 6th International Workshop on Metamorphic Testing, ser. MET ’21. Institute of Electrica...

    2021

  90. [104]

    Metamorphic relations for testing automated and autonomous driving systems and simulation platforms,

    M. Iqbal, “Metamorphic relations for testing automated and autonomous driving systems and simulation platforms,” Ph.D. dissertation, School of Computing and Information Technology, University of Wollongong, Feb. 2024, doctor of Philosophy thesis. [Online]. Available: https://r...

    2024

  91. [105]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning , ser. Proceedings of Machine Learning Research, S. Levine, V . Vanhoucke, and K. Goldberg, Eds., vol. 78. PM...

    2017

  92. [106]

    Safety-enhanced autonomous driving using interpretable sensor fusion transformer,

    H. Shao, L. Wang, R. Chen, H. Li, and Y . Liu, “Safety-enhanced autonomous driving using interpretable sensor fusion transformer,” in Proceedings of The 6th Conference on Robot Learning , ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., ...

    2023

  93. [107]

    Whitley, Next Generation Genetic Algorithms: A User’s Guide and Tutorial

    D. Whitley, Next Generation Genetic Algorithms: A User’s Guide and Tutorial. Springer International Publishing, Sep. 2018, pp. 245–274. [Online]. Available: http://dx.doi.org/10.1007/978-3-319-91086-4 8

    2018 doi

  94. [108]

    Experimental analysis of the tournament size on genetic algorithms,

    Y . Lavinas, C. Aranha, T. Sakurai, and M. Ladeira, “Experimental analysis of the tournament size on genetic algorithms,” in 2018 IEEE International Conference on Systems, Man, and Cybernetics, ser. SMC ’18. Institute of Electrical and Electronics Engineers (IEEE), Oct. 2018, ...

    2018

  95. [109]

    Integrated crossover rules in real coded genetic algorithms,

    P. Kaelo and M. Ali, “Integrated crossover rules in real coded genetic algorithms,” European Journal of Operational Research , vol. 176, no. 1, pp. 60–76, Jan. 2007. [Online]. Available: http://dx.doi.org/10.1016/j.ejor.2005.07.025

    2007 doi

  96. [110]

    Automated repair of feature interaction failures in automated driving systems,

    R. B. Abdessalem, A. Panichella, S. Nejati, L. C. Briand, and T. Stifter, “Automated repair of feature interaction failures in automated driving systems,” in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA ’20. New York...

    2020

  97. [111]

    A survey on handling computationally expensive multiobjective optimization problems with evolutionary algorithms,

    T. Chugh, K. Sindhya, J. Hakanen, and K. Miettinen, “A survey on handling computationally expensive multiobjective optimization problems with evolutionary algorithms,” Soft Computing , vol. 23, no. 9, pp. 3137–3166, Dec. 2017. [Online]. Available: http: //dx.doi.org/10.1007/s0...

    2017 doi

  98. [112]

    Comparing results of 31 algorithms from the black-box optimization benchmarking bbob-2009,

    N. Hansen, A. Auger, R. Ros, S. Finck, and P. Po ˇs´ık, “Comparing results of 31 algorithms from the black-box optimization benchmarking bbob-2009,” in Proceedings of the 12th Annual Conference Companion on Genetic and Evolutionary Computation, ser. GECCO ’10. New York, NY , U...

    2009

  99. [113]

    A technique for the measurement of attitudes

    R. Likert, “A technique for the measurement of attitudes.” Archives of psychology, vol. 22, no. 140, p. 55, 1932

    1932

  100. [114]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. C. Ohlsson, B. Regnell, and A. Wessl ´en, Experimentation in Software Engineering . Berlin, Heidelberg: Springer Science and Business Media LLC, Sep. 2024. [Online]. Available: http://dx.doi.org/10.1007/978-3-662-69306-3 APPENDIX EXPERIMENT...

    2024 doi

  101. [115]

    Genetic programming: principles and applications,

    S. Sette and L. Boullart, “Genetic programming: principles and applications,” Engineering Applications of Artificial Intelligence , vol. 14, no. 6, pp. 727–736, Dec. 2001. [Online]. Available: http://dx.doi.org/10.1016/S0952-1976(02)00013-1

    2001 doi

  102. [116]

    Mirjalili, Genetic Algorithm

    S. Mirjalili, Genetic Algorithm . Springer International Publishing, Jun. 2018, pp. 43–55. [Online]. Available: http://dx.doi.org/10.1007/ 978-3-319-93025-1 4

    2018

  103. [352]

    Available: http://dx.doi.org/10.1145/1143997.1144060

    [Online]. Available: http://dx.doi.org/10.1145/1143997.1144060

  104. [1696]

    Available: https://doi.org/10.1145/1830761.1830790

    [Online]. Available: https://doi.org/10.1145/1830761.1830790

  105. [2018]

    Available: https://doi.org/10.1145/3143561

    [Online]. Available: https://doi.org/10.1145/3143561

    doi

  106. [2023]

    Available: http://dx.doi.org/10.1016/j.scico.2023.102926

    [Online]. Available: http://dx.doi.org/10.1016/j.scico.2023.102926

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.