{"id":"387d776d-004d-47da-b23b-11d7c6ac4431","arxiv_id":"2502.09982","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"In the first ICST SDC testing competition, the ITS4SDC tool outperformed four other tools and a random baseline on precision and cost-effectiveness for selecting failing simulation test cases.","lead":"This paper reports the first ICST tool competition on regression test selection for simulation-based testing of self-driving cars, comparing five tools on the SensoDat benchmark against a random baseline. It finds ITS4SDC has the best fault-to-selection and time-to-fault ratios, making it the competition winner, while several tools select fewer tests than random.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Time-to-fault advantage of ITS4SDC over DRVN is within sampling noise; a single 80/20 split and N=36 with no significance testing do not support the claim that ITS4SDC is the best-performing tool.","rationale":"The reader's CONDITIONAL verdict correctly identifies the lack of statistical analysis as a key weakness, and the proposed concrete test would directly address that concern. However, the reader's stated weakest_assumption focuses on oracle validity and split representativeness, whereas I see the most load-bearing issue as the statistical robustness of the headline ranking, specifically the time-to-fault metric. The paper is otherwise a well-described competition report with public artifacts, Docker images, and a reasonable baseline, which are genuine strengths. The claim that ITS4SDC has the best fault-to-selection precision is plausible, but the broad conclusion 'best-performing tool' overreaches the evidence because the time-to-fault advantage is within sampling noise and no repeated-split analysis is provided. Since the reader already conditioned the verdict on additional analysis, no change in verdict is needed; the concrete test would settle whether the ranking survives proper statistical scrutiny.","tokens_in":6456,"tokens_out":5670,"duration_ms":60584,"concrete_test":"Obtain the per-collection metric values for ITS4SDC and DRVN from the Zenodo artifact. For both fault-to-selection and simulation time-to-fault, compute the paired differences across the 36 collections, then run a paired t-test and a 10,000-sample bootstrap to estimate 95% confidence intervals for the mean difference. If the confidence interval for time-to-fault includes 0, the claim should be weakened to 'best fault-to-selection precision' rather than 'best-performing tool.' If feasible, also re-run the competition with 10 different random 80/20 splits, or bootstrap resample the 36 collections, to check whether ITS4SDC remains the top-ranked tool under resampling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section V ('the best-performing tool is ITS4SDC') rests on two mean ratios: fault-to-selection (0.8048 vs DRVN's 0.7424) and simulation time-to-fault (65.39 vs DRVN's 72.96), computed over N=36 collections. The paper reports only means and standard deviations, with no confidence intervals, significance tests, or repeated-split analysis. For the time-to-fault ratio, the observed difference of 7.57 is small relative to the reported standard deviations (33.46 for ITS4SDC, 40.18 for DRVN). A rough two-sample comparison gives a standard error of approximately 8.7, implying t ≈ 0.87, which is far from significant. Thus the 'best time-to-fault ratio' finding is not statistically established. Because the conclusion explicitly singles out the time-to-fault metric (Finding 4) and uses it to declare an overall winner, this fragility is load-bearing. Additionally, Section II-F describes a single 80/20 split per collection; with such high between-collection variance, the ranking could easily change under a different split. The paper does not report any robustness check, so the stability of the ranking is unknown. The fault-to-selection difference may be more robust, but the broad 'best-performing' claim depends on both metrics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the first ICST Tool Competition on self-driving car testing, held with five participating tools (DRVN, DETOUR, ITS4SDC, Graph Selector, CertiFail) plus a random baseline. The competition platform provides a gRPC/Protocol Buffers interface, Docker packaging, and a public GitHub repository; evaluation is performed on 36 pre-executed SensoDat test collections using regression-test-selection metrics such as selection count, initialization time, selection time, simulation time-to-fault ratio, fault-to-selection ratio, and curvature diversity. The paper's central result, stated in Section V, is that ITS4SDC is the best-performing tool, based on the best mean fault-to-selection ratio (0.8048) and best mean simulation time-to-fault ratio (65.39) over N=36 collections.","tokens_in":6649,"tokens_out":3606,"duration_ms":40478,"significance":"If the empirical result is statistically supported, the paper provides a useful, reproducible evaluation infrastructure for SDC regression test selection: the Docker/gRPC interface, the public benchmark artifacts, and the comparison of five independently submitted tools are concrete contributions that lower the entry barrier for future competition participants. The paper ships the evaluation data on Zenodo and the platform on GitHub, which is a genuine strength. However, the headline conclusion rests on means without significance testing, robustness checks, or repeated runs, and the paper's second research question (RQ2) is not actually answered. With appropriate statistical strengthening and a more carefully scoped conclusion, the paper would be a valuable competition report for the SDC testing community.","major_comments":[{"comment":"The claim that ITS4SDC is the best-performing tool is not statistically established for the simulation time-to-fault ratio. From Table III, ITS4SDC has mean 65.39 (std 33.46) and DRVN has mean 72.96 (std 40.18) over N=36 collections. The raw difference is 7.57, while a rough two-sample standard error is sqrt((33.46^2 + 40.18^2)/36) = 8.71, giving t ≈ 0.87, which is far from significant. No confidence intervals, paired tests, or adjustment for multiple comparisons are reported anywhere in Section III or Section II-G, so Finding 4, and therefore the overall winner declaration in Section V, is fragile. I recommend adding paired significance tests or bootstrap confidence intervals for all metrics, and rewording the conclusion to say that ITS4SDC achieved the best observed mean values rather than that it is definitively best-performing.","section":"Section III (Findings 4–5) and Section V"},{"comment":"The evaluation uses a single 80/20 split per collection and no repeated executions of the tools, although several tools are likely stochastic. The reported standard deviations therefore capture between-collection variance only, not run-to-run variability or split sensitivity. With only 36 collections and high variance in the time-to-fault metric, the ranking could change under a different split or seed. The paper should report repeated splits (or bootstrap resampling over collections) and, where applicable, repeated runs of each tool, to demonstrate that the observed ranking, especially the ITS4SDC versus DRVN comparison, is stable.","section":"Section II-F and Section II-G"},{"comment":"The paper poses RQ2 as 'What are the features leveraged by the top-performing tools?' but never answers it with evidence. Section IV speculates that the tools use 'most likely models, which predict the test outcomes' and that DETOUR and DRVN have 'rather complex' models, but no feature analysis, ablation, or inspection of the submitted tools is provided. Either the RQ should be answered with concrete measurements (e.g., type of model, input features, selection heuristic reported by each participant) or RQ2 should be removed or explicitly deferred to the individual tool papers.","section":"Section I (RQ2) and Section IV"}],"minor_comments":[{"comment":"There are several typos: 'approache' should be 'approach', 'SDCS' should be 'SDCs', and Figure 1's caption says 'Intercation' instead of 'Interaction'.","section":"Abstract and Section I"},{"comment":"The column headers 'time to fault ration' and 'fault to selection ration' should be spelled 'ratio'.","section":"Table III"},{"comment":"The generator names are inconsistent: 'frenetic v', 'freneticV', and 'FreneticV' are used in different rows, which makes the table harder to read and could confuse the mapping to the three generators described in Section II-C.","section":"Table II"},{"comment":"Reference [10] lists the fourth author as 'R. S.'; this appears to be an incomplete or corrupted author name and should be corrected.","section":"References"},{"comment":"The definition of curvature diversity says 'return its mean value' without specifying whether the mean is taken over road segments of a single test case or over the selected test cases in a collection; please clarify.","section":"Section II-D"},{"comment":"The procedure does not state what information the initialization data contains (e.g., whether failure labels are provided for the 80% training portion). This is important for interpreting the results and should be stated explicitly.","section":"Section II-F"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competition report rather than a research contribution, and its main value is the public infrastructure and benchmark plus the comparison data. The statistical weakness in the headline comparison is the key issue: with the numbers in Table III, the time-to-fault advantage of ITS4SDC over DRVN is within sampling noise, so the 'best-performing tool' claim needs to be substantially re-scoped or supported with additional analysis. The authors are also the organizers evaluating their own platform and benchmarks, which is normal for a competition report but worth keeping in mind when assessing the neutrality of the framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a genuinely useful competition report. It is the first SDC-specific edition of the ICST tool competition, and the authors did the organizational work right: public GitHub repo, Docker images for the five tools, Zenodo archive of results, gRPC interface, and a random baseline. The 36 SensoDat collections are a reasonable testbed, and the paper is transparent about the simple metrics it uses. That transparency is a real strength.\n\nWhat is actually new is the empirical comparison itself. The finding that ITS4SDC achieves a mean fault-to-selection ratio of 0.80 versus the random baseline's 0.38 is the kind of result the community wants. The authors are also candid that most tools select far fewer tests than random, which suggests learned predictors are doing something.\n\nThe soft spots are real. The headline conclusion, 'the best-performing tool is ITS4SDC,' rests on two means with no significance testing, no confidence intervals, and no repeated runs. The stress-test note is right: the time-to-fault difference between ITS4SDC (65.39) and DRVN (72.96) is trivial relative to the standard deviations (33.46 and 40.18). A rough two-sample comparison gives t ≈ 0.87. That difference is not statistically established, and Finding 4 should not be presented as a meaningful advantage. The fault-to-selection difference (0.8048 vs 0.7424) looks more substantial—rough t ≈ 2.5—so the paper does have one robust result. But the 'best' claim depends on both metrics, so it needs to be softened.\n\nThe single 80/20 split per collection is another concern. With between-collection variance this high, a different split could change the ranking. The paper should report repeated splits or at least show that the precision result is stable. The time-to-fault metric is also not protected against differences in test length, and RQ2 is answered by speculation rather than by analyzing the submitted tools' actual features.\n\nNone of this kills the paper. It is a competition report, not a definitive scientific study, and the artifacts alone justify publication. Who is this for? Researchers working on SDC regression testing and participants in tool competitions. They will use the platform and benchmark. It deserves a serious referee, but the authors should add basic significance testing, report confidence intervals, and revise the conclusion to match the evidence.\n\nRecommendation: send to peer review with a request for statistical analysis and a more careful claim.","headline":"Useful SDC tool-competition report with real artifacts, but the 'best tool' claim overreaches: the time-to-fault difference is within sampling noise; only the precision result is likely real.","tokens_in":7261,"tokens_out":2775,"would_cite":true,"duration_ms":26983,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In the first tool competition on self-driving-car testing, the ITS4SDC tool proved best at selecting fault-revealing simulation tests.","keywords":["self-driving car testing","test selection","regression testing","simulation-based testing","tool competition","SensoDat","fault-to-selection ratio","cost-effectiveness"],"falsifier":"Re-run the identical protocol but hold out entire SensoDat collections instead of splitting each collection 80/20, and compare ITS4SDC with the random baseline on the untouched suite; if the winner's 0.805 fault-to-selection ratio collapses toward the baseline's 0.379, the in-collection split is what carried the result.","tokens_in":6183,"feed_emoji":"🚗","tokens_out":6475,"duration_ms":57960,"temperature":0.7,"pith_summary":"This paper reports the first tool competition on regression test selection for simulation-based testing of self-driving cars. Five submitted tools and a random-selection baseline picked test cases from 36 pre-executed SensoDat collections, and the results were scored with two cost-effectiveness metrics. The paper's central claim is that ITS4SDC is the best-performing tool: it achieves the best mean fault-to-selection ratio (0.805) and the best mean simulation time-to-fault ratio (65.39). A sympathetic reader should care because the finding suggests that a prediction-based selector can find more failing simulation tests per unit of simulation time than random selection or the other entrants.","feed_headline":"ITS4SDC tops first self-driving-car test selection competition","feed_subtitle":"Across 36 SensoDat suites it scored 0.80 fault-to-selection ratio, more than double the random baseline's 0.38.","key_machinery":"The engine of the comparison is the competition's evaluation protocol: a containerized interface through which each tool receives an initialization set and then must select test cases from a suite, with all interactions logged by an automated evaluator. Every tool is run on the same 36 SensoDat collections, each treated as an independent test suite; within each suite, 80% of the tests are given for initialization (training) and the remaining 20% are the selection targets. SensoDat is a repository of pre-executed simulation test cases for self-driving cars with pass/fail labels, generated by three different test generators. The cost-effectiveness metrics that carry the ranking are the fault-to-selection ratio (precision of the selection) and the simulation time-to-fault ratio (execution cost per revealed fault), and the paper's findings report the averaged statistics over the 36 runs.","core_discovery":"According to the competition's results, ITS4SDC outperforms both the random baseline and the other four tools on the two cost-effectiveness metrics defined in the study. Across the 36 SensoDat test suites, ITS4SDC selected on average 67.8 test cases per suite and achieved a mean fault-to-selection ratio of 0.8048, meaning about four of every five selected tests failed; the random baseline selected 88.2 tests and achieved 0.3790. ITS4SDC also achieved the best mean simulation time-to-fault ratio of 65.39 seconds of simulation per fault, versus 156.23 for random selection and 72.96 for the next best tool, DRVN Tool. The authors conclude from these numbers that ITS4SDC is the best-performing tool in this first edition.","pith_inferences":["The 80/20 split is taken from inside each SensoDat collection, so the tools never face a fully unseen test suite; whether ITS4SDC's advantage survives whole-campaign holdout is an open question.","The pass/fail labels come from the original SensoDat creators; if those labels are oracle-dependent, a different oracle could reorder the tools, even though all tools were treated equally.","A practical takeaway implied but not stated by the authors is that for cost-sensitive regression testing, a lightweight predictor with negligible initialization and sub-second selection is more attractive than a slower model with similar precision.","The competition protocol could be extended to test selection across generators (training on Ambiegen campaigns, selecting from Frenetic campaigns), which would probe whether the learned predictors generalize across test generation distributions."],"forward_implications":["A prediction-based test selector can be substantially more cost-effective than random selection in simulation-based SDC testing, since the winner's precision was more than double the baseline's.","Tools that select fewer, mostly failing test cases can cut simulation time per fault by more than half compared with random selection.","Model complexity has observable cost consequences: the two tools with the slowest selection (DETOUR and DRVN Tool) are also the ones with costly or complex models, while ITS4SDC selects in under a second on average.","The defined metrics can be reused in future editions to compare new SDC test selection tools against these first-edition results.","Because no clear diversity difference was found, selection cost and fault precision, not road-curvature diversity, drove the ranking."],"supporting_citations":[{"why":"supplies the pre-executed SensoDat collections, the 36 test suites and their failure labels on which all tools are evaluated.","marker":"[4]"},{"why":"describes the AmbieGen generator that produced one class of the SensoDat collections.","marker":"[5]"},{"why":"describes the Frenetic generator that produced another class of the SensoDat collections.","marker":"[7]"},{"why":"describes the FreneticV generator, a further source of SensoDat collections.","marker":"[9]"},{"why":"archives the competition results data behind the reported statistics.","marker":"[3]"},{"why":"defines regression test selection, the problem the competition instantiates and the terminology the metrics build on.","marker":"[2]"}],"fun_headline_variants":["ITS4SDC wins first self-driving car test selection competition","SDC testing tool ITS4SDC beats random baseline in ICST competition","First SDC test competition: ITS4SDC top, 0.80 fault ratio","ITS4SDC doubles fault rate vs random in SDC test contest","ICST 2025: ITS4SDC best SDC test selector, 0.80 vs 0.38"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ranking rests on the assumption that the pre-computed failure labels in the SensoDat collections are a correct and complete oracle for SDC faults; if those labels are noisy or generator-specific, the measured ratios and the tool order could change.","fun_headline_variants_meta":{"raw":{"variants":["ITS4SDC wins first self-driving car test selection competition","SDC testing tool ITS4SDC beats random baseline in ICST competition","First SDC test competition: ITS4SDC top, 0.80 fault ratio","ITS4SDC doubles fault rate vs random in SDC test contest","ICST 2025: ITS4SDC best SDC test selector, 0.80 vs 0.38"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3192,"prompt_tokens":859,"completion_tokens":2333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":2223}},"tokens_in":475,"tokens_out":2333,"duration_ms":15422,"temperature":1.0,"reasoning_tokens":2223,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:49:07.828790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the identical protocol but hold out entire SensoDat collections instead of splitting each collection 80/20, and compare ITS4SDC with the random baseline on the untouched suite; if the winner's 0.805 fault-to-selection ratio collapses toward the baseline's 0.379, the in-collection split is what carried the result.","supporting_citations":[{"cited_title":"Birchler, S","cited_arxiv_id":null,"evidence_quote":"supplies the pre-executed SensoDat collections, the 36 test suites and their failure labels on which all tools are evaluated."},{"cited_title":"Birchler, C","cited_arxiv_id":null,"evidence_quote":"describes the AmbieGen generator that produced one class of the SensoDat collections."},{"cited_title":"Panichella, A","cited_arxiv_id":null,"evidence_quote":"describes the FreneticV generator, a further source of SensoDat collections."},{"cited_title":"Khatiri, S","cited_arxiv_id":null,"evidence_quote":"defines regression test selection, the problem the competition instantiates and the terminology the metrics build on."}],"review_version":1}