{"id":"2d92d878-d77e-4c22-af3e-edfc76f5e43d","arxiv_id":"2412.03843","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CoCoMEGA combines metamorphic testing with cooperative co-evolutionary search and finds more severe, diverse autonomous-driving safety violations in simulation than random search and a standard genetic algorithm.","lead":"This paper presents CoCoMEGA, a testing framework that combines metamorphic testing with cooperative co-evolutionary search to automatically generate driving scenarios that expose unsafe or unusual behavior in autonomous driving systems. In CARLA simulations of the Interfuser system, CoCoMEGA found more severe and more diverse unsafe scenarios than random search and a standard genetic algorithm.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The effectiveness claim rests on MR thresholds calibrated by pilot without a no-perturbation noise baseline; if thresholds are near the simulator's noise floor, the reported gains may be artifacts.","rationale":"The reader's verdict is CONDITIONAL, and the weakest assumption they identified is precisely the validity of the MR thresholds as faithful safety oracles. My stress-test agrees: the central claim that CoCoMEGA generates more severe and diverse MR violations is only meaningful if the MRs and their thresholds separate true safety-relevant behavior changes from artifacts. The paper provides a replication package, ablations, and expert feedback as independent support, which I credit, but the expert rating covers only 50 hand-picked severe cases and does not establish that the full set of detected violations (or the thresholds used in the search objective) is free from noise or miscalibration. My proposed control experiment directly tests the noise-floor hypothesis, and the threshold sensitivity analysis tests whether the reported advantage is robust to the specific threshold values that were pilot-tuned. If the control shows negligible null violation rates and the advantage persists across thresholds, the concern is resolved; if not, the CONDITIONAL verdict should be reconsidered toward REJECT or UNVERDICTED. Since the evidence for the concern is not yet conclusive and the required checks are straightforward, keeping the verdict as CONDITIONAL is appropriate, and my analysis does not change that.","tokens_in":45598,"tokens_out":5384,"duration_ms":117738,"concrete_test":"Run a null-distribution control experiment: (1) Sample a set of source scenarios from the search space; (2) for each, construct the empty perturbation (all no-ops) and simulate both source and follow-up multiple times (e.g., 10 repeats) under identical random seeds/conditions; (3) compute E(s,q) with the same DTW and critical-interval pipeline. If a non-negligible fraction (e.g., >5%) of these null runs exceeds the MR thresholds (20% speed reduction, 1 degree steering), the thresholds are below the simulation's noise floor and the reported violation counts are unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CoCoMEGA finds significantly more severe and diverse MR-violating test cases than SGA and RS. All effectiveness metrics (DS, MRC, AUC) count solutions whose joint fitness E(s,q) exceeds user-set thresholds θf, where E is defined in Eq. (2) and the thresholds (e.g., 20% speed reduction, 1-degree steering invariance) are stated in Section V-B2 to be 'derived from the established literature and refined through a pilot experiment.' The paper does not report a control experiment measuring E(s,q) when the perturbation q is a no-op or when the same scenario is replayed under identical conditions. If CARLA/InterFuser exhibits run-to-run variability (the paper does not state that the simulator is deterministic or that a fixed seed is used), then E(s,q) may be non-zero even without a real behavioral change, and thresholds calibrated only to a pilot set could be below the noise floor. In that case, both the search objective (maximizing E) and the effectiveness metric (counting E>θf) would be measuring simulation noise rather than unsafe behavior. The expert validation in Section VI-A covers only the 50 most severe cases selected by CoCoMEGA and does not rate a random sample or compare baselines, so it cannot confirm that CoCoMEGA's advantage reflects genuinely more safety-relevant violations. The paper itself acknowledges in Section VI-C1 that an inaccurate threshold 'could either fail to detect a true violation or detect one where there is not,' making this the load-bearing assumption for the entire empirical evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoCoMEGA, a framework that combines metamorphic testing (MT) with cooperative co-evolutionary search to generate diverse, MR-violating test scenarios for autonomous driving systems. The approach maintains separate populations of source scenarios and metamorphic perturbations, evaluates complete solutions using a joint fitness function based on DTW-aligned time series, and employs fitness clearing and diversity-aware archiving. The empirical evaluation in CARLA with the InterFuser ADS compares CoCoMEGA against random search (RS) and a standard genetic algorithm (SGA) across three MR groups (GP1, GP2, GP3), measuring distinct solutions, solution diversity, MR coverage, and search-budget efficiency. The paper reports that CoCoMEGA finds 83% more distinct MR-violating solutions than SGA and 87% more than RS, with AUC efficiency gains of 46% and 32% respectively for DS, and 90% and 86% for MRC. An ablation study on the diversity mechanisms and expert ratings of 50 severe cases are also presented.","tokens_in":45998,"tokens_out":4807,"duration_ms":60604,"significance":"If the results hold, CoCoMEGA is a relevant contribution to search-based testing of ADSs, addressing the oracle problem via MT and leveraging co-evolution to improve search efficiency. The paper has notable strengths: 10 repeated runs with 95% confidence intervals for most experiments, an ablation isolating the diversity mechanisms, a publicly available replication package, and a clear reporting of the MR sources. However, the central claim that CoCoMEGA 'consistently outperforms' the baselines is qualified by the paper's own data at low fitness thresholds combined with high distance thresholds, and the lack of a simulator-noise baseline weakens the evidence that the detected MR violations correspond to genuine behavioral anomalies. The expert validation is limited to a selected subset of the most severe cases, so the safety relevance of the broader solution set remains unconfirmed. These issues are correctable and do not undermine the fundamental approach, but they need to be addressed before the claims can be accepted as stated.","major_comments":[{"comment":"The abstract and Section VII state that CoCoMEGA 'consistently outperforms' the baseline methods, but the paper's own results contain explicit exceptions. In Section V-C2, the authors acknowledge that at low fitness thresholds (θf ≤ 0.8) combined with high distance thresholds (θd ≥ 1.2), SGA and RS obtain higher DS values than CoCoMEGA. Similarly, in the RQ2 results (Fig. 6), at θf = 0.5 and θd = 1.5, RS outperforms CoCoMEGA. The 'consistently' claim is therefore not supported by the presented evidence. I recommend revising the wording to qualify the claim (e.g., 'consistently outperforms for severe violations, θf ≥ 1.0') and discussing these exceptions in the conclusion rather than only in the middle of the results section.","section":"§V-C2 and Fig. 3; §V-D2 and Fig. 6"},{"comment":"The MR output thresholds (e.g., 20% speed reduction for GP1, 1-degree steering invariance for GP2/GP3) are stated to be 'derived from the established literature and refined through a pilot experiment,' but no control experiment is reported to establish a noise floor for the fitness function E(s, q) in Eq. (2). If CARLA or InterFuser exhibits run-to-run nondeterminism, E(s, q) could be non-zero even for identical scenario-perturbation pairs or for no-op perturbations, and thresholds calibrated only on a pilot set might be below the simulator's noise floor. This would affect both the search objective (maximizing E) and the effectiveness metrics (counting solutions with E > θf). To address this load-bearing concern, I request a supplementary analysis that (i) replays a sample of scenario-perturbation pairs multiple times under identical conditions and (ii) evaluates no-op perturbations, reporting the distribution of E(s, q). The thresholds should then be justified as exceeding this noise baseline.","section":"§V-B2 and §VI-C1"},{"comment":"The expert feedback in Section VI-A is based on only the 50 most severe violations found by CoCoMEGA, selected across its 10 executions. This selection is not a random sample nor does it include violations generated by the baseline methods. Consequently, the ratings cannot support the claim that CoCoMEGA's advantage over RS and SGA reflects genuinely more safety-relevant violations; the experts did not compare methods or rate the full distribution of outputs. I recommend either rating a random sample of CoCoMEGA's solutions covering the full range of fitness values, or rating matched samples from CoCoMEGA, SGA, and RS with the experts blind to the generating method. This would provide the missing evidence that the measured MR violations are practically meaningful and that the reported improvements are not artifacts of a calibrated proxy.","section":"§VI-A"},{"comment":"The experimental comparison is affected by an asymmetry in hyperparameter tuning. For SGA and RS, the paper uses 'widely recommended' values (mutation 0.2, crossover 0.8, tournament size 3), while for CoCoMEGA the hyperparameters (e.g., maximum archive size 3) were tuned via a pilot experiment because 'there are no suggested values for CCEAs.' Since the research questions compare CoCoMEGA with SGA to isolate the effect of cooperative co-evolution, the tuned CoCoMEGA configuration against an untuned SGA configuration confounds the algorithmic choice with the hyperparameter tuning effort. I suggest tuning SGA (or at least adding a tuned-SGA variant) or providing evidence that the chosen SGA parameters are strong for this problem, so that the reported 83% improvement in DS is a fair comparison.","section":"§V-B3"},{"comment":"The results for GP2 show that none of the three methods found any MR violation for the environmental invariance relations (MR6, MR7). This means the reported effectiveness and efficiency findings are based entirely on GP1 (speed reduction) and GP3 (actor invariance). The paper mentions this result in passing but the conclusions do not temper the generality claims. Since a substantial portion of the MRs (6 of 13) produced zero violations, the claim of 'broader exploration of the test space' is only supported for the decreasing-relation family, not for invariance relations. I recommend explicitly stating in the discussion and conclusion that the approach was effective for the decreasing and actor-invariance MR groups, while the environmental-invariance relations were not violated by the subject ADS, and discussing what this implies for the general applicability of the method.","section":"§V-C2 (GP2 results)"}],"minor_comments":[{"comment":"The name of the ADS is spelled inconsistently as 'INTERFUSER' in the abstract and elsewhere, 'InterFuser' in Section V-B1, and 'INTER FUSER' in Section V-B1 and the references. Please unify the spelling.","section":"Throughout"},{"comment":"The MRC and CMR values are presented in a single cell (e.g., '100.0 ± 0.0 (3.0 ± 0.3)'). The caption indicates the parentheses contain CMR, but this is easy to miss, especially when scanning the tables. I suggest splitting these into two clear columns for readability.","section":"Tables II, III, and IV"},{"comment":"In Definition 3, the notation Tci is introduced in the text but the equation uses Tci without an explicit definition of the subscript symbol; please place the definition of Tci directly under the equation to improve clarity.","section":"Eq. (2)"},{"comment":"The first bullet states 'We proposed CoCoMEGA, the first automated testing method that combines MT and CCEA.' Since establishing novelty is difficult and the related work section does not provide an exhaustive search over combinations of these techniques, I recommend softening this to 'to the best of our knowledge' to avoid overclaiming.","section":"Section I (Contributions)"},{"comment":"The p-values reported (e.g., p < 10^-31) are extremely small for experiments with 10 runs per method and integer-valued DS counts, suggesting possible over-dispersion or that the effect size should be reported in addition to the p-value. I recommend reporting effect sizes (e.g., Cliff's delta or Vargha-Delaney A) alongside the significance tests to give the reader a sense of magnitude.","section":"Section V-D2 and Fig. 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within the scope of TSE and addresses a timely problem. The main concern for the editor is that the central claim of 'consistent' superiority is overstated relative to the data, and the missing noise-floor control could be seen by reviewers as a fundamental validity threat. The replication package is a strong plus, and the expert validation is a good attempt but needs a more rigorous sampling design. If the authors can address the noise-floor issue and qualify the claims, the paper would be a solid contribution. I would not recommend rejection, as the core ideas are sensible and the empirical evaluation is generally careful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a solid, useful paper that deserves a serious referee, with one central claim that needs to be toned down. The new thing here is the combination of metamorphic testing with cooperative co-evolution for system-level ADS testing. Both pieces are established, but the integration is non-trivial: scenarios and perturbations evolve in separate populations, collaborate through a joint fitness, and the archive and fitness-clearing mechanisms preserve diversity. The authors ship a replication package, run 10 repetitions per method with confidence intervals, and include an ablation showing the diversity mechanisms matter. That is real evidence and it earns credit.\n\nThe soft spots are real but not fatal. The word 'consistently' in the abstract overstates the data. In their own Figure 3, at low fitness thresholds and high distance thresholds, RS and SGA sometimes find more distinct solutions. They acknowledge this but then fall back on severity. The deeper issue is oracle calibration: the output thresholds (20% speed reduction, 1-degree steering invariance) came from literature and pilot tuning, with no no-perturbation control. If CARLA or InterFuser has run-to-run variability, some measured 'violations' could be noise. The paper's own threat-to-validity discussion admits an inaccurate threshold can create false positives. Given how much depends on those thresholds, a no-op baseline should have been reported. The expert assessment only covers the 50 most severe cases selected by CoCoMEGA, so it validates the top of the distribution, not the full set.\n\nAlso, the baseline set is weak. SGA and random search are reasonable sanity checks, but the paper does not compare against modern search-based testing methods for ADSs (e.g., AV-Fuzzer, DeepHyperion, illumination search), which limits the practical significance claim. These are addressable; the method itself is clearly described and the empirical design is careful.\n\nI would send this to peer review. A referee should ask for a no-op control, a softened consistency claim, and ideally one stronger baseline. The replication package makes these requests actionable. The paper is worth reading for anyone working on scenario-based validation of ADSs.\n\nRecommendation: engage with it.","headline":"A careful and novel integration of MT and CCEA for ADS testing, with a replication package and mostly honest reporting, though the 'consistent' superiority claim needs softening.","tokens_in":46518,"tokens_out":2670,"would_cite":true,"duration_ms":27617,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining metamorphic relations with a two-population cooperative search lets CoCoMEGA find 83 percent more distinct unsafe-driving scenarios than a standard genetic algorithm and 87 percent more than random search under the same…","keywords":["autonomous driving systems","system-level testing","metamorphic testing","search-based testing","cooperative co-evolutionary algorithms","automated testing","test oracle"],"falsifier":"Replay the archived MR-violating scenarios in an independent setting (a different simulator or a closed-course test) and have new raters score a random sample that includes the mild violations, not only the 50 most severe ones; if the mild violations are mostly judged harmless and severity scores do not track standard proximity metrics such as time-to-collision, the oracle calibration—not the search—is the limiting factor.","tokens_in":45409,"feed_emoji":"🚗","tokens_out":10625,"duration_ms":95255,"temperature":0.7,"pith_summary":"This paper claims that the central obstacle to testing autonomous-driving systems in simulation—knowing what 'bad' behavior looks like without a complete specification—can be handled by using metamorphic relations as partial test oracles and searching for violations of those relations. Its proposed method, CoCoMEGA, evolves a population of driving scenarios and a population of metamorphic perturbations in parallel, scoring each pairing by how strongly the system under test breaks a prescribed behavioral relation, such as slowing down by at least 20 percent when a pedestrian appears or keeping the steering angle fixed when fog density changes. Within a fixed simulation budget on the CARLA simulator and the InterFuser ADS, CoCoMEGA reports 83 percent more distinct violating scenarios than a standard genetic algorithm and 87 percent more than random search, along with larger relative gains in covering the defined relations. If these results hold, automated simulator-based testing can catch subtle undesirable behaviors that do not yet cause collisions, and do so more cheaply than unguided search.","feed_headline":"83% more unsafe driving tests found by co-evolutionary search","feed_subtitle":"Two co-evolving populations hunting metamorphic-relation violations beat genetic and random baselines in CARLA.","key_machinery":"The machinery has three parts. First, the metamorphic relation (MR) acts as the oracle: an input relation (for example, a pedestrian appears in front of the ego vehicle) paired with an output relation (speed must drop by at least 20 percent, or steering must stay within one degree), and a violation is a deviation from that relation. Second, the fitness signal is the extent of violation $E(s, q)$: dynamic time warping with a Sakoe-Chiba constraint aligns the time series of source and follow-up scenarios, matched pairs inside a critical interval are retained, and the average deviation $dif\\!f_{or}$ over those pairs quantifies severity. Third, a cooperative co-evolutionary algorithm maintains two populations—scenarios and perturbations—that are bred separately with custom crossover and mutation operators but assessed jointly, with fitness clearing (using a dynamically computed clearing radius) and pure-diversity-based archive updates keeping the solutions spread across the search space.","core_discovery":"CoCoMEGA's central claim is that test generation for an ADS can be reformulated as a search over scenario–perturbation pairs that violate metamorphic relations. The paper formalizes this as finding a diverse set of pairs $(s, q)$ with violation extent $E(s, q) > 0$, where $E(s, q)$ is computed from simulator time series: dynamic time warping aligns the source and follow-up trajectories, aligned pairs inside a critical interval are kept, and the average deviation from the output relation (invariance, increasing, or decreasing) gives the severity. Because the input space is high-dimensional and each simulator run is expensive, the method splits the search into two co-evolving populations—source scenarios and perturbations made of metamorphic transformations—that collaborate to form complete test cases. Individual fitness is the best joint severity an individual achieves with members of the other population, and two diversity mechanisms, fitness clearing with a dynamic niche radius and archive selection by a pure-diversity metric, keep the returned set spread out. Evaluated on CARLA with InterFuser, the paper reports that this design finds more distinct, severe, and behaviorally diverse MR violations than a standard genetic algorithm or random search, and that the diversity mechanisms alone account for a 91 percent gain in distinct violations.","pith_inferences":["A direct calibration test would correlate the 20 percent speed-reduction threshold with established proximity metrics such as time-to-collision in the same scenarios; if near-threshold violations never coincide with metric deterioration, the oracle is measuring a proxy rather than risk.","The scenario–perturbation split mirrors a general divide in cyber-physical testing between ambient context and imposed disturbances, so the same two-population scheme may speed up failure search for other autonomous systems with expensive simulators.","The paper's AUC advantage over baselines widens at high fitness thresholds, which suggests the method's real strength is concentrating a limited budget on severe violations while mild anomalies remain better found by random spread.","Because the archive accumulates labeled (scenario, perturbation) pairs with measured violation extents, it doubles as training data for the surrogate models the paper names as future work, which could cut the dominant two-minute-per-run simulation cost."],"forward_implications":["With the same simulation budget used by the baselines, testers can obtain roughly twice as many distinct severe MR-violating scenarios, so expensive simulator time is converted into more failure-revealing tests.","The defined speed-reduction relations can be covered almost completely (100 percent at the lowest fitness threshold, 56 percent even at the strictest threshold tested), so a single search session exercises most of the oracle set rather than one relation at a time.","Because MRs need not be perfect or complete to guide the search, the method transfers to systems where no full behavioral specification exists, provided a domain expert can state relations like 'steering should not change when weather changes'.","Removing the diversity mechanisms collapses the gain (91 percent fewer distinct violations), so preserving spread across scenarios and perturbations is what makes the co-evolutionary search useful rather than merely guided.","Since the scenario representation is tied to what a simulator exposes and compatible with standard formats, the framework can be ported to other OpenScenario-compatible simulation platforms without changing the search core."],"supporting_citations":[{"why":"Supplies the cooperative co-evolutionary algorithm that CoCoMEGA uses to split the search into scenario and perturbation populations.","marker":"[23]"},{"why":"Supplies the formal definition of metamorphic relations and the MT background that turns MR violations into test oracles.","marker":"[47]"},{"why":"Source of the speed-reduction metamorphic relations (MR1, MR2) that define group GP1's expected behavior.","marker":"[64]"},{"why":"Source of the weather-invariance metamorphic relation (MR7, steering consistency across synthesized weather) used in group GP2.","marker":"[52]"},{"why":"Source of the actor-invariance metamorphic relations (MR8–MR13) that define group GP3's steering-consistency expectations.","marker":"[104]"},{"why":"Supplies the fitness-clearing niche procedure used to keep each population diverse during evolution.","marker":"[76]"},{"why":"Supplies the dynamic-fitness-sharing idea that CoCoMEGA adapts into its dynamic clearing radius, removing a tuned parameter.","marker":"[84]"},{"why":"The CARLA simulator in which every scenario is executed and every violation metric is measured.","marker":"[67]"},{"why":"The InterFuser ADS under test; the empirical claims are about this system inside CARLA.","marker":"[101]"}],"fun_headline_variants":["Co-evolution finds 83% more dangerous scenarios","Cooperative search hunts down hidden driving flaws","Two-population evolution uncovers more unsafe drives","Metamorphic testing boosted by co-evolution","Co-evolutionary search reveals subtle driving failures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything depends on the metamorphic relations being faithful proxies for real risk: the output thresholds (a 20 percent speed-reduction and a one-degree steering change) were set by literature review plus a pilot study, so if those thresholds do not track genuine safety concerns, the search objective and the effectiveness metrics are measuring violations of a calibrated proxy, not unsafe driving.","fun_headline_variants_meta":{"raw":{"variants":["Co-evolution finds 83% more dangerous scenarios","Cooperative search hunts down hidden driving flaws","Two-population evolution uncovers more unsafe drives","Metamorphic testing boosted by co-evolution","Co-evolutionary search reveals subtle driving failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1391,"prompt_tokens":1098,"completion_tokens":293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":714,"tokens_out":293,"duration_ms":4044,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:00:40.561364+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replay the archived MR-violating scenarios in an independent setting (a different simulator or a closed-course test) and have new raters score a random sample that includes the mild violations, not only the 50 most severe ones; if the mild violations are mostly judged harmless and severity scores do not track standard proximity metrics such as time-to-collision, the oracle calibration—not the search—is the limiting factor.","supporting_citations":[{"cited_title":"A cooperative coevolutionary approach to function optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the cooperative co-evolutionary algorithm that CoCoMEGA uses to split the search into scenario and perturbation populations."},{"cited_title":"An evolutionary algorithm with advanced goal and priority specification for multi-objective optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the dynamic-fitness-sharing idea that CoCoMEGA adapts into its dynamic clearing radius, removing a tuned parameter."}],"review_version":1}