{"id":"8de11e88-91d0-4d29-a004-88804fd40a18","arxiv_id":"2506.08688","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Causal-Fuzzer uses causal graphs of scene, action, and violation relationships to guide simulation fuzzing, and reports finding more diverse violations and better testing sufficiency than three baselines on Apollo.","lead":"This paper describes Causal-Fuzzer, a simulator-based testing tool that learns cause-and-effect links between the driving scene, the car's actions, and crashes, then uses those links to create more varied dangerous scenarios. It reports finding more distinct violations than existing fuzzers on Apollo and LGSVL, though without code or detailed statistics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LiNGAM causal discovery is run on binary occupancy/action/violation indicators, violating the continuous non-Gaussian noise and i.i.d. assumptions that make it identifiable; if the discovered edges are spurious, the diversity feedback and mutation priorities lose causal meaning.","rationale":"Reader's weakest assumption pinpointed the LiNGAM-on-binary-data mismatch, and that is exactly the most load-bearing vulnerability. The entire 'causality-aware' machinery--feedback thresholds in Section III-B3 and the adaptive mutation probabilities in Algorithm 2--consumes the causal strength matrix W from a discovery step that is not identifiable under the given inputs. A misspecified discovery step can produce arbitrary DAGs that pass global Markov-fit tests while being causally wrong. This undermines both the internal logic (mutation no longer targets causally influential NPCs) and the external metrics (SAC/SAVC count spurious edges). The statistical power issues (two runs per cell) and the arithmetic inconsistency in the 134.09% claim are real but secondary: they affect confidence intervals and presentation, not the core conceptual validity. If the synthetic-calibration check succeeds, the LiNGAM concern is largely retired and the remaining issues are ordinary reproducibility and statistics concerns, consistent with the reader's CONDITIONAL verdict. If it fails, the conditional acceptance should be reconsidered. Therefore the verdict remains unchanged.","tokens_in":22040,"tokens_out":6557,"duration_ms":82071,"concrete_test":"Run a synthetic calibration using the paper's exact vectorization and LiNGAM pipeline: plant known causal edges in a simulator or script (e.g., NPC presence in a specific sector causes a left-turn action, which causes an ego-induced collision), generate 200 traces, compute the recovered adjacency matrix, and measure edge precision/recall against the planted graph. If precision or recall is not significantly above chance (e.g., F1 < 0.6), the discovered causal graphs are not trustworthy and the evaluation cannot support the causality claims; if recovery is high, the LiNGAM concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of Causal-Fuzzer is a scene-action-violation causal graph discovered per scenario by LiNGAM (Section III-B2, Eq. 5). LiNGAM requires each variable to be a linear function of its parents plus independent continuous non-Gaussian noise; identifiability of the graph and of the causal strengths W relies on that assumption. However, the inputs X_sav are 32 binary sector-occupancy indicators, 5 binary action indicators, and 2 binary violation indicators. A binary variable cannot be generated by a linear SEM with continuous noise, so the model is misspecified and the recovered W is not identifiable from the data. The data also come from a single execution trace (time series), violating the i.i.d. sample assumption, and violation indicators are almost all zero in non-collision scenarios, making edges into violation variables especially fragile. The RQ1 validation (Table I) uses DoWhy's permutation/independence tests and refutation tests; these only check statistical compatibility of some DAG with the data, not whether the specific edges correspond to real causal mechanisms, and they do not compare against any ground truth. Consequently, the SAC and SAVC edge counts (the reported 'testing sufficiency' and 'violation diversity') may simply count spurious edges, and the per-NPC causal effects used in Eq. 12 and Algorithm 2 may be noise. Since both feedback and mutation are driven by these graphs, the reported improvements over baselines could reflect the extra exploration from the feedback loop rather than genuine causal structure. Algorithm 1 also contains typos (Gs'saf, F P) and never adds violated mutants to SAVC, making the diversity bookkeeping ambiguous.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Causal-Fuzzer, a fuzzing technique for simulation-based testing of autonomous driving systems. It vectorizes scenario observations into binary scene-sector occupancy indicators, binary ego-action indicators, and binary violation indicators; runs LiNGAM causal discovery on each executed scenario to build a scene-action-violation causal graph; and uses that graph to define two feedback metrics (testing sufficiency via scene-to-action edges and violation diversity via scene/action-to-violation edges) and an adaptive mutation strategy that mutates NPCs with higher estimated causal effects. The authors evaluate Causal-Fuzzer on Apollo 7.0 with the LGSVL simulator across four functional scenarios, comparing against random testing, AV-Fuzzer, and DoppelTest. They report higher violation counts, higher SAC/SAVC values, and earlier first-failure detection, and they include an ablation study to isolate the contributions of the causal feedback and causal mutation components.","tokens_in":22386,"tokens_out":6849,"duration_ms":73793,"significance":"If the discovered causal graphs are valid, this is a genuinely useful direction: it ties together input-scene diversity, action diversity, and violation diversity in a single causal structure, and it uses causal strengths to prioritize mutations in an interpretable way. The evaluation on an industry-grade ADS with a high-fidelity simulator, the inclusion of three baselines, and the ablation study are strengths. The central weakness is that the causal discovery step is not validated against any ground truth, and the same causal graphs are both the object being discovered and the basis for the reported diversity and sufficiency metrics. The empirical claims therefore currently rest on an unverified modeling assumption, and the headline quantitative claims contain internal inconsistencies.","major_comments":[{"comment":"LiNGAM is used despite the data violating its identifiability assumptions. Equation (5) assumes each variable is a linear function of its parents plus independent non-Gaussian continuous noise, and the standard estimation assumes i.i.d. samples. The input matrix X_sav, however, is composed of 32 binary occupancy indicators, 5 binary action indicators, and 2 binary violation indicators, collected from a single execution trace over time. Binary variables cannot be generated by a linear SEM with continuous additive noise, and temporal samples are not independent. As a result, the weighted adjacency matrix W and the edge orientations used in Sections III-B3 and III-C1 are not identifiable from the stated assumptions. The authors need to either justify the application of LiNGAM to this data, use a causal discovery method whose assumptions match discrete/mixed, serially dependent observations, or demonstrate via synthetic ground-truth experiments that the recovered graphs are still reliable.","section":"Section III-B2, Eq. (5)"},{"comment":"The DoWhy-based validation does not establish that the discovered edges correspond to true causal mechanisms. The permutation and independence tests check statistical compatibility of a DAG with the observed data; they do not compare the recovered edge set against a known ground-truth causal graph. Moreover, the paper reports mean p-values of 0.001 as evidence that the graphs accurately represent the data, but without stating the null hypothesis, a p-value below 0.05 cannot be interpreted as confirming compatibility. The KL divergence and MSE values also lack any baseline or reference model. A meaningful validation would use scenarios with known causal structure, such as hand-constructed situations where the true cause of a collision is known, and measure edge-level precision/recall of the discovered graph.","section":"Section IV-B, Table I"},{"comment":"The headline numbers are internally inconsistent. In Table II, the average SAVC of Causal-Fuzzer is 12.9, and the best baseline average SAVC is 6.8 (AV-Fuzzer), not 5.5 (DoppelTest). The claimed 134.09% violation-diversity increase therefore compares against the wrong baseline; the correct increase relative to the best baseline is 89.7%. In addition, the abstract states that testing sufficiency is 12.9 versus 6.8, but the testing sufficiency metric defined in the paper is SAC, whose averages are 183.3 versus 150.1. The abstract's 98.4 versus 42.4 values refer to violation count, not SAVC. These metric labels and percentages need to be corrected and recomputed consistently.","section":"Section IV-C, Table II"},{"comment":"All effectiveness and efficiency claims rest on only two 10-hour runs per scenario type. No variance, confidence intervals, or statistical significance tests are reported, and no information is given about random-seed variation or initialization diversity. With just two runs, differences such as Scenario 1 SAVC of 11.0 versus 7.0 are within plausible simulator noise, so the statement that Causal-Fuzzer 'significantly outperforms' the baselines is not supported by the experimental design. The authors should report distributions over a larger number of independent runs and apply appropriate statistical tests or effect-size measures.","section":"Section IV-C, Tables II and III"},{"comment":"The SAC and SAVC metrics are defined from the same causal graphs that Causal-Fuzzer is rewarded to expand, so the reported improvements in these metrics partly measure the algorithm's success at optimizing its own objective. This is a form of circularity: a fuzzer that is explicitly rewarded for creating new scene-action or scene-action-violation edge combinations will naturally report more such combinations, even if the edges are spurious. The absolute violation count is a more independent signal, but the diversity and sufficiency claims need an external notion of coverage or a ground-truth set of causal relations in the simulator to avoid circularity. A concrete test would be to fix a set of semantically meaningful scenario categories with known causes and measure whether the discovered edges align with those causes.","section":"Section III-B3 and Section IV-C"}],"minor_comments":[{"comment":"In Line 1, the set 'SV C' appears to be a typo for 'SAV C', and in Line 17 the expression 'Distance(Gs′ saf, F P)' appears to be a typo for 'Distance(Gs′ sav, SAVC)'.","section":"Algorithm 1"},{"comment":"The text says thresholds 'θts and θcd', but the violation-diversity threshold was introduced as 'θvd'; use consistent notation.","section":"Section III-B3"},{"comment":"The sentence 'We evaluated Causal-Fuzzer on an industry-grade ADS Apollo, with a high-fidelity.' is incomplete; the simulator name should appear in the sentence or the sentence should be finished.","section":"Abstract"},{"comment":"The definition of the weights w_i should clarify the direction convention, since W in Eq. (5) is defined with wij representing the strength from xj to xi; the sum 'Pp j=1 wij' needs to be written unambiguously.","section":"Equation (12)"},{"comment":"The units of SAC and SAVC should be stated explicitly in the table captions, and the half-integer average values resulting from two runs should be explained or replaced with per-run values.","section":"Tables II and III"}],"recommendation":"major_revision","confidential_remarks":"The central idea of using causal graphs to unify scene, action, and violation diversity is attractive and fits the journal's scope, but the current manuscript's quantitative claims are not defensible without a valid causal discovery step and corrected statistics. I would encourage the editor to invite a major revision that addresses the LiNGAM assumption violation, provides ground-truth validation, fixes the metric inconsistencies, and reports more than two runs per condition. If the causal discovery issue cannot be resolved, the paper's core claims would not survive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, but read it skeptically. The core construction is new: Causal-Fuzzer runs causal discovery on vectorized scenes, actions, and violations, uses the discovered scene-action edge counts as a testing-sufficiency feedback, uses scene-action-violation edge combinations as violation diversity, and weights NPC mutations by estimated causal effects. No one has quite done that in ADS testing. The qualitative examples are persuasive, and the ablation suggests both the feedback and the mutation strategy contribute to the extra violations.\n\nThe evaluation does not yet support the headline. Table II shows average SAVC of 12.9 for Causal-Fuzzer against 6.8 for the best baseline; that is an 89.7% increase, not the claimed 134.09%. The 134.09% only works if you compare with DoppelTest's 5.5. Also, only two runs per scenario, no variance, no significance tests; \"significantly outperforms\" is doing a lot of work. No code or data released, so the numbers aren't checkable.\n\nThe bigger risk is the causal discovery. LiNGAM assumes each variable is a linear function of its parents with independent non-Gaussian continuous noise. Here the inputs are binary sector-occupancy indicators, binary action indicators, and binary violation indicators, sampled from a single execution trace. That is a misspecified model. The DoWhy validation in RQ1 checks statistical compatibility of some DAG with the data, not whether the specific edges are real mechanisms. If the edges are spurious, the diversity feedback and causal-effect-based mutation priorities are just a fancy way to randomize. The absolute violation count (98.4 vs. 42.4) is an independent signal, so the method might work as a heuristic diversity objective even if the causal interpretation is wrong; but the paper's claims of interpretability and causality rest on unexamined assumptions.\n\nThere are also bookkeeping problems in Algorithm 1: violated mutants are never added to SAVC, and lines 14-18 reference undefined sets (Gs'saf, F P, SV C). That makes the reported metrics ambiguous.\n\nWho is it for? Researchers in simulation-based ADS testing. The central idea is worth discussing and can inspire follow-up work, but this version needs major revision before I'd trust the numbers. I would send it to peer review because the idea is important enough and the evaluation runs on a real ADS, but I'd expect reviewers to demand released artifacts, more runs, a corrected claim, and a serious treatment of the causal discovery assumptions.","headline":"Genuinely novel idea for diversity-aware ADS fuzzing, but the evaluation is too thin and the LiNGAM-on-binary-data step is a load-bearing risk.","tokens_in":22958,"tokens_out":3501,"would_cite":false,"duration_ms":39870,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A causal-graph fuzzer finds 56 extra Apollo violations with 134% more violation diversity.","keywords":["autonomous driving systems","simulation-based testing","fuzzing","causal graph","causal discovery","LiNGAM","violation diversity","testing sufficiency"],"falsifier":"Rerun Causal-Fuzzer on the same four scenarios and time budgets with causal-effect mutation weights replaced by uniform random weights while keeping the same causal-graph feedback and thresholds; if the reported 22.06% gain in testing sufficiency and 134.09% gain in violation diversity shrink to statistical noise, the causal machinery is not load-bearing.","tokens_in":21790,"feed_emoji":"🚗","tokens_out":6690,"duration_ms":78211,"temperature":0.7,"pith_summary":"Causal-Fuzzer is a fuzzing method for simulation-based testing of autonomous driving systems that measures testing diversity through causal relationships instead of treating input scenarios, ego-vehicle actions, and safety violations as independent metrics. The paper claims that by learning a scene-to-action-to-violation causal graph from each executed scenario, a fuzzer can reward test cases that activate new causal edges, and by weighting mutations toward NPCs with higher causal effect on ego behavior, it can find more distinct failures earlier. On Baidu Apollo with the LGSVL simulator, the reported average results are 98.4 discovered violations versus 42.4 for the best baseline, 22.06% higher testing sufficiency, 134.09% higher violation diversity, and 32.1 scenarios explored before the first failure versus 71.5. The significance of the claim is that comprehensive ADS safety testing should be judged by coverage of causal relationships, not by counts of scenarios, actions, or violations alone.","feed_headline":"Causal fuzzer finds 56 extra Apollo safety violations","feed_subtitle":"Modeling scene-action-violation causal edges lifts diversity 134% and halves scenarios before first failure.","key_machinery":"The scene-action-violation causal graph is the central object: a weighted DAG whose nodes are binary scene occupancy indicators (32 radial-annular sectors around the ego vehicle), five binary ego-action indicators (accelerate, decelerate, left turn, right turn, maintain), and two binary violation indicators (ego-caused or NPC-caused collision). The graph is estimated with LiNGAM, a linear non-Gaussian acyclic causal discovery method, from the vectorized scenario observation; binary adjacency matrices of the estimated graphs supply the cosine-distance feedback for testing sufficiency and violation diversity, and the outgoing causal strengths supply the per-NPC average causal effects that drive adaptive mutation. It is the mechanism that turns raw scenario traces into a searchable coverage space.","core_discovery":"The central discovery claimed is that the interdependence between scenes, ADS motion commands, and violations can be encoded as a directed acyclic causal graph, and the edge combinations of that graph define two useful coverage objectives: scene-to-action edges measure testing sufficiency, while (scene, action)-to-violation edges measure violation diversity. Causal-Fuzzer discovers such a graph per executed scenario from vectorized binary observations, converts the causal strength matrix to a binary adjacency matrix, and uses the minimum cosine distance to previously seen graphs as novelty feedback. It then estimates the average causal effect of each NPC vehicle on ego actions and violations, and mutates NPCs with probability proportional to that effect, with an epsilon-greedy escape to random mutation. The paper reports that this combination finds more violations, more causally distinct violations, and better scene-action edge coverage than random testing, AV-Fuzzer, and DoppelTest.","pith_inferences":["Because scene occupancy is binary and actions are five discrete indicators, the causal graph cannot distinguish quantitative risk levels within the same edge combination; graded encodings or continuous causal models would be a natural extension.","The mutation weighting depends entirely on the discovered graph's outgoing causal strengths, so the method's practical value is bounded by the quality of causal discovery on the vectorized data; replacing LiNGAM with a discrete or nonlinear causal discovery method is the first robustness check.","The 32.1-versus-71.5 speed-up to first failure suggests the largest efficiency gain comes from early causal-effect weighting, while the diversity gains may accumulate later from the feedback thresholds; testing this temporal decomposition would be a direct follow-up experiment."],"forward_implications":["Testing coverage for ADSs should be reported as activated causal edges, not merely as counts of inputs, actions, or violations, because identical motions can be safe or unsafe depending on scene context.","Prioritizing mutations by NPC causal effect reduces the number of scenarios needed to reach the first safety violation, so the same time budget yields more distinct failures.","The 134.09% increase in violation diversity means the fuzzer finds failures across more semantic categories, such as ego-accelerated collisions versus NPC-induced collisions, not just more instances of one collision type.","With correct causal graphs, the same feedback and mutation machinery generalizes beyond the four tested scenarios to any scenario expressible in the vectorized scene-action-violation representation."],"supporting_citations":[{"why":"Supplies the LiNGAM causal discovery method that turns vectorized scenario observations into the scene-action-violation directed acyclic graph.","marker":"[41]"},{"why":"AV-Fuzzer is the genetic-algorithm-plus-local-fuzzer baseline that Causal-Fuzzer is compared against for violation discovery and efficiency.","marker":"[17]"},{"why":"DoppelTest is the multi-instance ADS baseline used for comparison, and its oracle handling is adapted in the experiments.","marker":"[6]"},{"why":"LGSVL is the high-fidelity simulator used to execute Apollo scenarios and collect the scenario observations.","marker":"[8]"},{"why":"Behavior-diversity-guided testing work that supplies the waypoint mutation scheme and action-diversity notion this paper builds on.","marker":"[5]"},{"why":"Baidu Apollo is the multi-module autonomous driving system under test in the evaluation.","marker":"[25]"}],"fun_headline_variants":["Causal-Fuzzer finds 56 new Apollo safety violations via causal graphs","Causal graph fuzzing boosts Apollo violation diversity by 134%","Causal-Fuzzer maps causal edges to find Apollo safety bugs faster","Causality-aware fuzzer uncovers diverse Apollo safety failures","Causal fuzzer cuts scenarios to first Apollo critical case in half"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LiNGAM causal discovery, which assumes linear structural equations with independent non-Gaussian continuous noise and no hidden confounders, yields correct causal graphs when applied to binary occupancy, action, and violation indicators; if the graphs are wrong, both the diversity feedback and the mutation weights lose their meaning.","fun_headline_variants_meta":{"raw":{"variants":["Causal-Fuzzer finds 56 new Apollo safety violations via causal graphs","Causal graph fuzzing boosts Apollo violation diversity by 134%","Causal-Fuzzer maps causal edges to find Apollo safety bugs faster","Causality-aware fuzzer uncovers diverse Apollo safety failures","Causal fuzzer cuts scenarios to first Apollo critical case in half"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4472,"prompt_tokens":1021,"completion_tokens":3451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":3355}},"tokens_in":637,"tokens_out":3451,"duration_ms":27414,"temperature":1.0,"reasoning_tokens":3355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:04:25.415516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun Causal-Fuzzer on the same four scenarios and time budgets with causal-effect mutation weights replaced by uniform random weights while keeping the same causal-graph feedback and thresholds; if the reported 22.06% gain in testing sufficiency and 134.09% gain in violation diversity shrink to statistical noise, the causal machinery is not load-bearing.","supporting_citations":[{"cited_title":"A linear non-gaussian acyclic model for causal discovery","cited_arxiv_id":null,"evidence_quote":"Supplies the LiNGAM causal discovery method that turns vectorized scenario observations into the scene-action-violation directed acyclic graph."},{"cited_title":"Av-fuzzer: Finding safety violations in autonomous driving systems,","cited_arxiv_id":null,"evidence_quote":"AV-Fuzzer is the genetic-algorithm-plus-local-fuzzer baseline that Causal-Fuzzer is compared against for violation discovery and efficiency."},{"cited_title":"Doppelg ¨anger test generation for revealing bugs in autonomous driving software,","cited_arxiv_id":null,"evidence_quote":"DoppelTest is the multi-instance ADS baseline used for comparison, and its oracle handling is adapted in the experiments."},{"cited_title":"Lgsvl simulator: A high fidelity simulator for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"LGSVL is the high-fidelity simulator used to execute Apollo scenarios and collect the scenario observations."},{"cited_title":"Apollo: Open source autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Baidu Apollo is the multi-module autonomous driving system under test in the evaluation."}],"review_version":1}