{"id":"82eab90f-fb26-4601-87f6-ce1ef72ba794","arxiv_id":"2502.08504","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MoDitector generates collision scenarios that are caused by errors in a user-specified ADS module, reporting 55.3, 75.3, 71.7, and 14.3 module-induced critical scenarios for perception, prediction, planning, and control, versus at most 18.3, 25.3, 34.3, and 7.0 for baselines.","lead":"MoDitector is a new testing method that generates driving scenarios tailored to expose failures in specific modules of an autonomous driving system, such as perception or planning. Its value is in identifying which module caused a collision, which could help developers debug and improve autonomous driving systems more efficiently.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MICS counts hinge on a 0.5 s pre-collision error window that quietly drops Definition 3.1's all-scenes requirement; extending the window may reclassify many scenarios and deflate the headline numbers.","rationale":"The paper is a genuine contribution: it defines a new testing objective (MICS), designs a concrete module-directed search pipeline, and reports large gains over adapted baselines with an ablation. The reader's CONDITIONAL verdict is appropriate. My stress-test focus is the oracle's temporal scope rather than the specific threshold values the reader emphasized. The MICS definition is strict—no non-target errors in any scene—but Algorithm 2 and Eq. 7 only inspect the final 0.5 s before collision. This is an internal inconsistency between the formal definition and the implementation, and it directly threatens the root-cause attribution that the paper's central claim rests on. The repair-rate experiment cannot rescue it: replacing a target module's output with perfect information can prevent a collision even when another module erred earlier, and immediate braking makes the planning/control repair rates trivially high. A single computational check—extending the detection window to the full scenario and recounting—would settle whether this concern lands. If few scenarios are reclassified, the concern is resolved and the central claim stands as conditional; if many are reclassified, the headline numbers are inflated and the root-cause claim is not supported. I therefore keep the reader's CONDITIONAL verdict unchanged rather than escalating to rejection, because the needed evidence is obtainable from data the authors already have.","tokens_in":20804,"tokens_out":7163,"duration_ms":83646,"concrete_test":"Re-run the MICS classification for all scenarios reported in Table 3 (including the 216.7 total) with Eq. 7's detection window Δt increased from 0.5 s to the full scenario duration, keeping all thresholds fixed. Count how many scenarios are reclassified as non-MICS because a non-target module has a nonzero filtered error before the 0.5 s window; if the reclassification rate is non-negligible (e.g., >10%), the headline counts are an artifact of the window choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MoDitector generates MICSs—failures caused by exactly one target module—depends on Definition 3.1 being enforced. It is not fully enforced. Definition 3.1(b) requires a target-module error at some scene and 3.1(c) requires every non-target module to be error-free in every scene, but Eq. 7 in Section 4.2.3 checks module errors only inside the detection window [T−Δt,T] with Δt=0.5 s. Any non-target-module error that occurs earlier—say a perception mis-detection at T−1 s that steers the planner into a bad lane, followed by error-free modules during the final half second—is invisible to the oracle, so the scenario is counted as a MICS even though the root cause is not the target module. The RQ2 repair experiment does not close this gap: replacing the target module with perfect outputs can remove a collision even when an earlier non-target error set up the dangerous state, and for planning/control the 'repair' is immediate braking (Section 5.2.1), which trivially avoids any collision. Because every count in Table 3, and the abstract's 216.7-vs-79.0 headline, passes through this filtered oracle, the window-size choice is load-bearing. The hand-set thresholds λ_M in Eq. 7 are a related but secondary sensitivity: λ_Mplan=0 means a single intersecting waypoint flags planning, and λ_Mctrl=0.05 is in unscaled position/velocity units, with no sensitivity analysis reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoDitector, a fuzzing framework for module-based autonomous driving systems that generates \"Module-Induced Critical Scenarios\" (MICS), i.e., collision scenarios attributed to exactly one target module. It formalizes MICS in Definition 3.1, designs module-specific oracle metrics for perception, prediction, planning, and control (Eqs. 2-6), filters module errors through a detection window (Eq. 7), introduces a module-directed feedback score (Eqs. 8-9), and implements adaptive seed selection and mutation (Algorithm 3). The evaluation on Pylot/CARLA across four NHTSA-derived scenarios compares MoDitector with Random, AVFuzzer, and BehAVExplor (Table 3), reports repair-rate evidence for oracle fidelity (Table 4), and includes ablation results for the feedback and seed-generation components (Table 5).","tokens_in":21148,"tokens_out":6015,"duration_ms":62575,"significance":"The problem addressed is important: existing ADS testing methods typically report system-level failures without localizing the responsible module, and the paper's framing as root-cause-aware testing is a useful step forward. The paper makes concrete contributions in the form of module-specific oracle design, module-directed feedback, adaptive scenario generation, and a comparative empirical study with ablations. If the module-error oracles were properly validated, the method would be a practical tool for module-level debugging of ADSs. However, the central claim depends on the fidelity of the proxy metrics and on the detection-window interpretation of Definition 3.1, and the current evidence is not yet convincing on those load-bearing points.","major_comments":[{"comment":"The implemented oracle does not enforce Definition 3.1 over the full scenario. Definition 3.1(c) requires every non-target module to be error-free in every scene, and Definition 3.1(b) only requires a target-module error somewhere in the scenario. Algorithm 2, however, evaluates errors only inside the window [T−Δt, T] with Δt = 0.5 s. A scenario in which perception erred at T−1.0 s and then recovered, while planning subsequently steered into a collision at T, would be classified as a planning- or control-MICS even though a non-target module was faulty during the scene sequence. Since every count in Table 3 and the abstract's 216.7-vs-79.0 headline passes through this filtered oracle, the window-size choice is load-bearing. Please either enforce Definition 3.1 over the full trace or provide direct evidence that errors before the window cannot influence the final collision, e.g., a sensitivity analysis over Δt and a manual audit of how scenarios are re-classified when the window is extended.","section":"§4.2.3, Eq. (7), and Definition 3.1"},{"comment":"The repair-rate experiment does not validate the oracle's root-cause attribution. For planning and control, the \"ground truth\" used in the repair test is immediate braking (Section 5.2.1); immediate braking trivially avoids any collision regardless of which module caused it, so the 100% repair rates for planning and control carry no evidential weight. For perception and prediction, replacing only the target module's outputs with ground truth can still leave a collision if another module misbehaves later, and the observed 89% repair rates do not rule out the possibility that an earlier error in a non-target module set up the dangerous state. The only direct test of oracle fidelity should be on scenarios where exactly one module is faulty by construction, e.g., via controlled fault injection into one module while all other modules are verified error-free against ground truth over the entire trace, or via human-labeled module-error annotations.","section":"§5.2.1, Table 4"},{"comment":"The module-error metrics are proxies with hand-set thresholds, and the paper reports no sensitivity analysis. The planning oracle in Eq. (5) compares the planned trajectory against predicted object positions, which are themselves outputs of perception and prediction; if prediction is wrong, the planned trajectory can be flagged as unsafe even though planning is correct given its inputs. The control oracle in Eq. (6) mixes position and velocity deviations in unscaled units, and λ_Mctrl = 0.05 has no stated unit or independent basis; λ_Mplan = 0 means a single intersecting waypoint flags a planning error. The tolerance thresholds in Section 5 are said to be set according to the preliminary study, but no derivation or sensitivity study is given. Because the same oracle is used both to filter MICS and to guide the feedback score, all effectiveness results are potentially sensitive to these choices. Please report a sensitivity analysis over λ_M and Δt, and justify the thresholds from an independent standard or from measured distributions of module errors.","section":"§4.2.2, Eqs. (5)-(6), and §4.2.3"},{"comment":"The contribution claim that MoDitector 'generates' more MICS than the baselines is entangled with the oracle that counts them. All methods are scored by MoDitector's module-specific oracle, so baseline counts are reduced not only by how baselines generate collisions but also by how the oracle classifies those collisions. This is not a problem per se, but it means the headline comparison inherits every limitation of the oracle. In particular, if the oracle under- or over-counts errors for certain modules (as the concerns above suggest), the per-module differences in Table 3 may reflect oracle artifacts rather than genuine differences in the generated scenarios. The paper should either justify that the oracle is module-faithful for all four modules or present a secondary analysis that does not rely solely on the proposed oracle.","section":"§4.2.2 and §5.1, Table 3"}],"minor_comments":[{"comment":"The text says the perception error is measured by comparing bounding boxes between 'Simulator observation Y(s) and ADS observations Y(s)'; the second Y(s) should presumably be A(s), the ADS observation.","section":"§4.2.2, Perception metric, Eq. (2)"},{"comment":"The return value of ModuleSpecificOracle is described with inconsistent notation: the text says it returns 'module errors δA and safety-critical distance δA,' but the algorithm and Section 4.2.4 show the safety-critical distance as δ_safe. Please unify the notation.","section":"Algorithm 1, line 5"},{"comment":"The abstract reports 216.7 MICS in total, while summing Table 3 gives 216.6; also, 'the best-performing baseline detects only 79.0' refers to BehAVExplor's total across modules, whereas the best baseline per module differs (AVFuzzer for prediction). Please state this explicitly to avoid confusion.","section":"Abstract and Table 3"},{"comment":"The efficiency comparison reports mean processing times without variance, scenario counts, or statistical testing; given that only three runs are averaged, please add the range or standard deviation and state the number of executed scenarios per run.","section":"§5.4, Table 6"},{"comment":"AVFuzzer and BehAVExplor were originally evaluated with Apollo/LGSVL and are adapted here to Pylot/CARLA; the paper should state what configuration parameters were preserved or re-tuned in the adaptation, since this affects the fairness of the comparison.","section":"§5, Baselines"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the problem is timely, but the validation is not yet at journal strength. The strongest issue is the mismatch between Definition 3.1 (all scenes) and the implemented detection window, compounded by a repair test that cannot confirm root causes. The authors should be encouraged to add a full-trace oracle check or a Δt sensitivity analysis, an independent validation of the module-level oracles (e.g., fault injection or manual labeling), and a sensitivity study over the hand-set thresholds. Releasing the code and scenario corpus would also substantially increase the credibility of the comparative evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: genuinely new problem formulation and a working pipeline, but the headline MICS counts are built on an oracle that only checks the final 0.5 seconds before a collision, while the formal definition demands error-free non-target modules across all scenes. That gap is load-bearing, not cosmetic.\n\nWhat's actually new and good: the MICS definition reframes ADS testing as root-cause-attributable failure generation; the module-specific error metrics (Eqs. 2-6) are a sensible first cut; the module-directed feedback and adaptive seed selection are reasonable and the ablation (Table 5) shows they contribute. The repair-rate experiment is an honest attempt to validate the oracle, even if flawed. Writing is clear, related work is well positioned.\n\nSoft spots, in proportion:\n1. Window mismatch. Equation 7 checks only [T−Δt,T] with Δt=0.5s, but Definition 3.1(c) requires every non-target module to be error-free in every scene. An earlier perception miss at T−1s that causes the planner to commit to a bad lane is invisible; the scenario is then labeled a MICS for the wrong module. No sensitivity analysis on Δt. Every count in Table 3 passes through this filter.\n2. The four thresholds λ_M are hand-set with no sensitivity analysis. λ_Mplan=0 means one intersecting waypoint flags planning as error; λ_Mctrl=0.05 is in unscaled position/velocity units.\n3. The repair validation for planning and control replaces the module with immediate braking. That trivially avoids any collision, so the 100% repair rates don't demonstrate oracle correctness.\n4. Baselines are adapted from Apollo/LGSVL to Pylot/CARLA, but adaptation is undocumented and no code is shipped, so the comparison is not reproducible.\n5. The abstract's 'best-performing baseline detects only 79.0' aggregates over different baselines per module; no single baseline gets 79.0. That is misleading.\n\nThese issues don't sink the core idea, but they soften the quantitative claims. A reader in ADS testing will get value from the problem formulation and the ablation, less from the absolute numbers.\n\nRecommendation: send to peer review, not desk reject. Require code release, sensitivity analysis on Δt and λ, and a more convincing repair experiment (e.g., a safe planner instead of an emergency brake). I'd be skeptical of the current numbers until then, but the direction is worth engaging.","headline":"Genuinely new module-directed ADS testing objective, but the MICS oracle only checks the last 0.5s while the formal definition requires all scenes, making headline counts window-dependent.","tokens_in":21716,"tokens_out":4180,"would_cite":false,"duration_ms":34110,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoDitector is a search-based testing method that claims to generate collision scenarios whose root cause is a single module of an autonomous driving system, by directing the search with module-specific oracles and feedback.","keywords":["module-induced critical scenarios","autonomous driving system testing","search-based fuzzing","root cause analysis","module-specific oracles","scenario generation","simulation-based testing"],"falsifier":"Take a random sample of scenarios MoDitector labels as MICS for perception or prediction, correct the target module's outputs using independent ground truth (e.g., human-annotated bounding boxes or an offline high-precision detector), and check whether the reported collision actually disappears; if the repair rate falls substantially below the reported 89–93%, the oracle is over- or under-attributing root cause. A second check is to vary the thresholds in Eq. 7 slightly and count how many 'MICS' reclassify—high sensitivity would show the headline counts are an artifact of the chosen thresholds.","tokens_in":20559,"feed_emoji":"🚗","tokens_out":6836,"duration_ms":60547,"temperature":0.7,"pith_summary":"MoDitector is a search-based testing method for module-based autonomous driving systems that aims to generate collision scenarios whose root cause is one specific module—perception, prediction, planning, or control—rather than just any system failure. The paper defines such scenarios as Module-Induced Critical Scenarios (MICS) and argues that existing black-box fuzzers, which optimize only for collisions or behavior diversity, produce few of them and distribute them unevenly across modules. MoDitector adds three components to the usual fuzzing loop: per-module oracles that decide whether each module is erroring, a feedback score that rewards scenarios where only the targeted module is at fault and the vehicle is close to a collision, and adaptive seed selection and mutation that push the search toward such scenarios. Across four representative driving scenarios on a modular ADS platform, the method reports 216.7 MICSs total, versus 79.0 for the strongest baseline. If correct, this gives developers a testing tool that not only finds crashes but attributes each one to a debuggable module.","feed_headline":"216.7 crash scenarios traced to one specific ADS module","feed_subtitle":"MoDitector's module-directed fuzzing finds far more single-module-caused collisions than prior black-box methods.","key_machinery":"The load-bearing object is the Module-Induced Critical Scenario (MICS), defined as a collision in which, within the detection window before the crash, exactly one module shows error while all others do not. Three mechanisms carry the argument: (1) module-specific oracles—metric formulas for perception (weighted IoU), prediction (maximum deviation from perception-biased trajectories), planning (collisions between planned and predicted points), and control (deviation from planned path)—that produce a per-module error signal; (2) module-specific feedback, a scalar score equal to the target module's filtered error minus the average of other modules' filtered errors minus the safety-critical distance, so that the search is pushed toward scenarios where only the target module errs and the vehicle nearly collides; and (3) adaptive seed generation, which selects seeds with probability proportional to feedback and switches between fine-grained and coarse-grained mutation depending on how close a seed is to a MICS. These components turn the definition of MICS into a continuous optimization target.","core_discovery":"The paper's central claim is that module-induced critical scenarios—system failures in which exactly one named module (perception, prediction, planning, or control) is responsible—can be generated efficiently by a fuzzer that uses per-module oracles, a feedback term that maximizes error in the target module while minimizing error in others, and adaptive seed selection and mutation. The key reported result is that MoDitector finds 216.7 MICSs across four scenarios and four modules, whereas the best baseline finds 79.0, and that the detected scenarios survive a repair test: when the target module's output is replaced by ground truth, the collision disappears in 89 to 100 percent of cases. This is claimed to be the first work to connect module-level root causes to system-level failures in ADS testing.","pith_inferences":["Beyond the paper, the single-module restriction is a design choice rather than a necessity: the same oracle and feedback machinery could be extended to multi-module root causes, a direction the paper mentions but does not explore.","The perception-biased trajectory trick used to isolate prediction errors might transfer to other settings where upstream and downstream error sources must be separated without extra instrumentation.","Because the oracles rely on ground-truth simulator observations, the method should transfer most readily to any modular ADS that exposes internal module outputs; a natural test is to rerun the same four scenarios and oracle formulas on a different modular platform and compare the yield."],"forward_implications":["If the central claim holds, ADS developers can use MoDitector to obtain a per-module breakdown of safety weaknesses from simulation, rather than studying collisions whose cause is ambiguous.","Because MICSs are defined to be single-module failures, each discovered scenario is directly actionable: it suggests a targeted fix in one module, such as retraining a detector or adjusting a planner's cost function.","The reported overhead beyond the simulator is small (about 1.9% of processing time), so the technique can be deployed in large-scale simulation campaigns.","The hierarchy of detected MICS counts (prediction > planning > perception > control) gives a concrete robustness ranking of the tested ADS's modules, and the repair-rate experiment provides a way to validate any such ranking."],"supporting_citations":[{"why":"Supplies the modular ADS under test, including perception, prediction, planning, and control modules.","marker":"[20]"},{"why":"Provides the high-fidelity driving simulator and the ground-truth observations used by the module oracles.","marker":"[13]"},{"why":"AVFuzzer, the collision-focused baseline the method is compared against.","marker":"[31]"},{"why":"BehAVExplor, the behavior-diversity baseline that yields the strongest overall comparison numbers.","marker":"[8]"},{"why":"NHTSA pre-crash typology, the source of the four representative driving scenarios used in the evaluation.","marker":"[35]"},{"why":"Weighted IoU metric on which the perception module oracle formula is based.","marker":"[19]"}],"fun_headline_variants":["Root-cause fuzzing pinpoints 216.7 ADS crashes to one module","MoDitector links each collision to perception, planning, or control","216.7 crashes blamed on a single ADS module by new fuzzer","Module-directed testing catches 3x more single-module crashes","New fuzzer identifies which ADS module caused each crash"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline depends on the assumption that the hand-picked module error metrics and their thresholds (0.5 for perception, 0.1 for prediction, 0 for planning, 0.05 for control) correctly separate a module that is working from a module that is failing; if these proxies misjudge, scenarios get wrongly classified as MICS and the comparison to baselines measures the wrong thing.","fun_headline_variants_meta":{"raw":{"variants":["Root-cause fuzzing pinpoints 216.7 ADS crashes to one module","MoDitector links each collision to perception, planning, or control","216.7 crashes blamed on a single ADS module by new fuzzer","Module-directed testing catches 3x more single-module crashes","New fuzzer identifies which ADS module caused each crash"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000722,"raw_usage":{"total_tokens":3243,"prompt_tokens":950,"completion_tokens":2293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":2200}},"tokens_in":566,"tokens_out":2293,"duration_ms":15290,"temperature":1.0,"reasoning_tokens":2200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:49:08.395448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of scenarios MoDitector labels as MICS for perception or prediction, correct the target module's outputs using independent ground truth (e.g., human-annotated bounding boxes or an offline high-precision detector), and check whether the reported collision actually disappears; if the repair rate falls substantially below the reported 89–93%, the oracle is over- or under-attributing root cause. A second check is to vary the thresholds in Eq. 7 slightly and count how many 'MICS' reclassify—high sensitivity would show the headline counts are an artifact of the chosen thresholds.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the modular ADS under test, including perception, prediction, planning, and control modules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the high-fidelity driving simulator and the ground-truth observations used by the module oracles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AVFuzzer, the collision-focused baseline the method is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BehAVExplor, the behavior-diversity baseline that yields the strongest overall comparison numbers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NHTSA pre-crash typology, the source of the four representative driving scenarios used in the evaluation."}],"review_version":1}