{"id":"a09df76f-3a5f-44fa-91fc-51d397efa835","arxiv_id":"2501.09866","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Autoware's human-written unit tests cover only 17.7% of lines in tested packages, LLM-generated tests have low build success rates, and the AwTest-LLM pipeline improves both.","lead":"This paper measures unit test coverage in Autoware, an open-source autonomous driving stack, and finds that developer-written tests cover only a fraction of functions. It then shows that LLM-generated C++ tests often fail to build, and proposes a dependency-aware prompting pipeline that raises build success rates and coverage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed improvement rests on a non-representative benchmark: only branch-containing functions from the 40 already-tested packages are used, and Table 6's uncovered dataset count (736) contradicts Section 3's 812, so the headline result may not generalize to most Autoware code.","rationale":"The reader's conditional verdict already focuses on the most load-bearing weakness: the evaluation's representativeness and the unresolved dataset-size inconsistency. My stress-test confirms this is the central threat to the paper's strongest claim. The claim is not that AwTest-LLM works on all Autoware code; it is that it improves LLM-generated unit tests for Autoware packages. Given the benchmark explicitly selects functions with branches and input arguments from only the 40 packages that already have developer tests, and given that the uncovered dataset itself shows low absolute performance (5–7% line coverage overall in Table 6), the generalization from this benchmark to the broader Autoware codebase is insecure. The 812-to-736 discrepancy is particularly damning because it suggests either an unrecorded filtering step or a reporting error; either way, the reader cannot trust the exact numbers that support the headline. The mixed module-level results further reinforce that the average improvement is not robust, and the lack of uncertainty quantification means we cannot distinguish signal from noise. These points justify the CONDITIONAL verdict without moving it: the paper's early-stage, self-aware discussion and the plausible mechanism (providing dependencies and call-graph examples) still warrant publication pending revision, but only after addressing the dataset consistency and broadening the evaluation. Therefore the verdict should remain UNCHANGED relative to the reader's conditional recommendation.","tokens_in":148,"tokens_out":2357,"duration_ms":37455,"concrete_test":"Re-run the uncovered-dataset experiments on the full 812 functions after reconciling the count with Table 6, and separately repeat the covered-dataset comparison on a stratified random sample of functions from the 63 packages without developer tests, including branchless functions and simple getters. If AwTest-LLM's build success rate and line coverage are not significantly above the basic-prompt baseline on this broader sample (using a paired or per-module analysis), the generalizability of the headline claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that AwTest-LLM improves build success rate and coverage of LLM-generated tests for Autoware—is empirically supported only on a carefully selected subset. Section 3 states that the evaluation collects 'functions with branches and input arguments' from packages that already have developer-written tests, yielding 390 covered and 812 uncovered functions. This excludes 63 of 103 packages and all branchless functions, which constitute a large share of the codebase (Table 1's 2206 total functions vs. 1129 branched functions). Consequently, the measured improvements (e.g., Table 5's +7.2% BS and +4.5% coverage for GPT-4o-mini, +11.6% coverage for GPT-4o) are not shown to transfer to the untested majority. This concern is compounded by two internal inconsistencies: (1) Section 3 reports 812 uncovered functions, but Table 6 lists only 736, an unexplained 76-function gap that could indicate selective exclusion of hard-to-process or low-performing functions; (2) module-level results in Table 5 are mixed—e.g., GPT-4o's control module BS drops by 11.6% and evaluator's RS_case drops by 26.4%—yet no statistical tests, confidence intervals, or cross-validation are provided, so the overall average improvement may hide instability. The paper itself concedes that uncovered functions are 'still very challenging' (RQ3), but the chosen benchmark and unaccounted dataset size make it difficult to quantify how much of the headline gain is an artifact of selection rather than a robust property of AwTest-LLM.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of unit testing for the Autoware autonomous driving framework, covering both human-written tests and LLM-generated tests. It reports that human-written tests have low coverage (36.4% function coverage across 40 tested packages), and that naive LLM prompts yield low build success rates (overall BS_file 22.3% for GPT-4o-mini and 26.7% for GPT-4o on a covered dataset of 390 functions). Based on a manual analysis of build errors (Table 3), the authors propose AwTest-LLM, which enriches the prompt with namespace/header metadata, call-graph examples, LLM-generated documentation, and CMake dependency information. On the covered dataset, AwTest-LLM improves overall BS_file by +7.2% and line coverage by +4.5% for GPT-4o-mini, and +6.6% and +11.6% for GPT-4o (Table 5). On an uncovered dataset (Table 6), results are lower and the paper concludes that untested functions are still very challenging. The central claim is that AwTest-LLM improves build success rate and coverage of LLM-generated tests for Autoware.","tokens_in":10145,"tokens_out":3615,"duration_ms":33518,"significance":"If the central claim holds, this is a worthwhile contribution to the software-testing literature: it provides the first fine-grained unit-testing study of an industrial ADS codebase, a careful qualitative error analysis, and a practical prompt-engineering method that demonstrably reduces build failures. The evaluation is external and grounded in the actual Autoware codebase: build success, run success, and lcov line coverage are measured against real compilation and execution. The manual categorization of error types (Tables 3 and 4) is useful diagnostic information for the community. However, the strength of the empirical claim is limited by the issues detailed below, particularly the dataset-size inconsistency and lack of statistical validation.","major_comments":[{"comment":"Section 3 states that the uncovered dataset comprises 812 functions not covered by official test cases, but Table 6 lists function counts that sum to 736 across the eight modules (46+19+1+10+10+639+8+3). This 76-function discrepancy is not explained. Moreover, Table 1 reports 1129 branched functions in covered packages, of which 390 are covered, implying 739 uncovered branched functions, which matches neither 812 nor 736. The inconsistency must be resolved and the actual dataset composition and selection criteria clarified, because it directly affects the validity of the results in Table 6 and the paper's general claims.","section":"Section 3 vs. Table 6"},{"comment":"All reported improvements are based on a single run per configuration, with no standard deviations, confidence intervals, or statistical tests. Since LLM test generation is stochastic, the headline differences (e.g., overall BS_file +7.2% for GPT-4o-mini, line coverage +11.6% for GPT-4o) could be within run-to-run noise. The module-level results also show notable regressions, such as GPT-4o BS_file dropping by 11.6% on the control module and RS_case dropping by 26.4% on the evaluator module. Multiple runs (e.g., at least 5) and paired significance tests are needed to support the claim that AwTest-LLM improves these metrics.","section":"Section 3.2, Table 5"},{"comment":"The evaluation benchmark is restricted to functions with branches and input arguments drawn only from the 40 of 103 packages that already have developer-written test cases. This excludes all branchless functions and the majority of the codebase (63 packages without tests). Consequently, the measured improvements (e.g., Table 5) do not support the broad statement in the abstract and conclusion that AwTest-LLM improves unit testing 'across Autoware packages.' The general claim should be tempered, or the benchmark should be widened to include packages without existing tests and branchless functions to demonstrate generality.","section":"Section 3, benchmark selection"},{"comment":"The proposed approach includes an LLM-based document generation step for undocumented functions, but the paper does not validate the accuracy of these generated documents or isolate their effect on the final results. Without an ablation study (e.g., with and without document generation, with and without call-graph examples), it is unclear which components of AwTest-LLM are responsible for the measured improvements. This is important for understanding the mechanism and for reproducibility.","section":"Section 3.3.1, Document Generation"}],"minor_comments":[{"comment":"The email addresses use 'ualbera.ca' instead of 'ualberta.ca'.","section":"Author affiliation"},{"comment":"'CmakeLists.txt' should be 'CMakeLists.txt'.","section":"Section 3.1, step 2"},{"comment":"The table does not show counts for 'Assertion on throw', 'Assertion on death behaviors', 'Invalid argument', and 'Other errors'; please complete the table or note zero counts.","section":"Table 4"},{"comment":"The abstract states the approach improves 'test case pass rates,' but the overall RS_case actually decreases for GPT-4o-mini (Table 5, -2.2%) and only marginally increases for GPT-4o (+1.5%); rephrase to focus on build success and coverage.","section":"Abstract and Section 4"},{"comment":"The framework name is written as 'Awtest-LLM' in the conclusion, which is inconsistent with 'AwTest-LLM' used elsewhere.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The dataset-size inconsistency (812 in Section 3 vs. 736 in Table 6) is a red flag that should be carefully checked, as it could indicate a post-hoc selection of functions. Please ask the authors to provide complete data and scripts to reproduce the numbers. The lack of repeated runs is also a serious weakness for an LLM-based study; without variance estimates, the headline improvements are not convincing. The paper's scope claim should be aligned with the actual evaluated subset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this one: it is the first quantitative look at unit test coverage in Autoware, and it shows clearly that LLM-generated C++/ROS tests are bottlenecked by build failures, not by test logic. The paper earns its keep with the coverage audit (2206 functions across 40 tested packages, 36.4% function coverage) and the error taxonomy from manual analysis of build failures. That taxonomy—symbol errors, missing namespaces, private access, hallucinated members—is genuinely useful for anyone trying to get LLMs to write compiling C++ tests against a large real codebase. The proposed AwTest-LLM pipeline (AST metadata, call-graph examples, CMake dependency parsing) is a sensible recipe, and the measured gains on the covered dataset are plausible: build success up about 7 points for GPT-4o-mini and 6.6 for GPT-4o, line coverage up 4.5 and 11.6 points respectively. Those are modest gains, but the direction is consistent across most modules, and the evaluation is external: generated tests actually have to compile and execute against the real Autoware build.\n\nThe soft spots are real but mostly fixable. The dataset size inconsistency stands out: Section 3 says 812 uncovered functions, while Table 6 sums to 736. The abstract's total of 1126 matches 390+736, so 812 is likely a typo, but it needs to be fixed because it undermines trust in the numbers. More substantively, the benchmark only includes functions with branches and input arguments drawn from the 40 of 103 packages that already have developer-written tests. That excludes the majority of packages and all branchless functions, so the measured improvements do not necessarily transfer to the untested bulk of the codebase. The paper is honest about this—RQ3 explicitly says uncovered functions remain very challenging—but the headline claim in the abstract is broader than the evidence supports. There are also no error bars or repeated runs, and module-level results are mixed (e.g., GPT-4o's control module build success drops 11.6 points), so the overall averages may hide instability.\n\nOn balance this is a solid empirical study for an under-explored area, and the authors are not overselling: they call it a study and early exploration, they identify the main failure mode, and they show a plausible mitigation. It deserves a serious referee, but a revised version should fix the count inconsistency, report variance or at least per-run data, and sharply delimit the generalization claim to functions like those in the benchmark. If you work on LLM-based test generation or ADS testing, this is worth citing for the dataset and the error taxonomy; the AwTest-LLM recipe is not groundbreaking, but it is reproducible and useful.","headline":"A useful exploratory study with a real empirical payoff, but the headline numbers need a cleanup pass and the benchmark's scope is narrower than it looks.","tokens_in":774,"tokens_out":857,"would_cite":true,"duration_ms":28789,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AwTest-LLM, a dependency-aware prompt method, raises build success and line coverage for LLM-generated unit tests on Autoware packages.","keywords":["Autoware","unit testing","LLM test generation","autonomous driving software","test coverage","build success rate","C++ testing","few-shot prompting"],"falsifier":"Run AwTest-LLM on the 60+ Autoware packages that currently lack any developer tests and on branchless functions, and compare the build success rate and line coverage against the paper's reported numbers. If the gains disappear on that broader set, the central claim about improving Autoware testing does not generalize. Alternatively, an ablation that removes the call-graph examples or the CMake dependency information from the prompt and shows no drop in build success would indicate the proposed pipeline's specific components are not the cause of the improvement.","tokens_in":9629,"feed_emoji":"🚗","tokens_out":4559,"duration_ms":39185,"temperature":0.7,"pith_summary":"This paper studies unit testing for Autoware, an industrial autonomous-driving software stack written in C++ and built on ROS 2. It finds that human-written tests cover only a small fraction of functions, and that large language models asked to write tests from raw source code fail to compile most of the time. The paper identifies the dominant failure modes—wrong symbols, missing headers, wrong namespaces, private access—and proposes AwTest-LLM, which enriches the prompt with namespace and header metadata, CMake-level external dependencies, and call-graph examples of the focal function. In the paper's evaluation, this raises the build success rate and line coverage of generated tests across Autoware packages compared with a basic prompt. The point of the work is that fine-grained, function-level testing for ADS software is both needed and partially attainable with context-aware LLM prompting.","feed_headline":"Dependency-aware prompts lift AI-written Autoware tests","feed_subtitle":"Adding headers, CMake dependencies, and call-graph examples raises build success and line coverage.","key_machinery":"The central object is the AwTest-LLM pipeline, a five-step preprocessing and prompting procedure. It parses each package's C++ files into abstract syntax trees, extracts the focal function's namespace, headers, and class metadata, builds a call graph to select calling functions as few-shot in-context examples, generates documentation for undocumented functions, and parses CMake files for external dependencies. These elements are assembled into the final prompt alongside the focal file, giving the LLM the API surface and usage patterns it needs to produce compilable tests. The pipeline is the mechanism that carries the argument: the paper's comparisons of basic prompts versus AwTest-LLM prompts attribute the gains to this added context.","core_discovery":"The central claim is that a large part of LLM failure in unit-testing Autoware comes from missing contextual information, not from the model's inability to write assertions. AwTest-LLM supplies that context: an AST pass extracts namespaces, associated headers, and class metadata; a call graph provides functions that invoke the focal function as few-shot examples; CMake parsing exposes external package dependencies; and undocumented functions get an LLM-generated description. On the 390-function covered benchmark, overall build success rises from 22.3% to 29.5% (GPT-4o-mini) and from 26.7% to 33.3% (GPT-4o), with line coverage rising from 13.9% to 18.4% and from 11.4% to 23.0%. On the 736-function uncovered dataset, improvements are smaller and absolute rates stay low, which the paper reads as evidence that never-tested functions remain a hard frontier.","pith_inferences":["If the dependency-aware context is the cause, then ablation studies could isolate which component (headers vs. CMake deps vs. call-graph examples) contributes most; the paper does not run these, but the architecture implies testable claims.","The low gains on the uncovered dataset suggest that coverage gains may partly reflect the LLM imitating existing test styles; on functions with no test precedent, the method would need a different mechanism, such as generated documentation or symbolic execution.","The dataset-size discrepancy (812 uncovered functions in Section 3 vs. 736 in Table 6) suggests either filtering after generation or a reporting inconsistency; reconciling it would make the evaluation reproducible.","The approach might generalize beyond testing: the same context extracted for Autoware could feed other code-generation tasks such as documentation or repair."],"forward_implications":["If AwTest-LLM works as reported, LLM-driven unit testing becomes a practical complement to human-written tests for industrial C++/ROS codebases.","The error taxonomy (symbol, header, namespace, type, access) gives a concrete checklist for improving future test-generation prompts.","The uncovered-dataset results imply that packages without any human tests are the harder target; tooling should focus there.","The approach could be applied to other ROS-based autonomous driving stacks or large C++ projects with similar build systems.","Improvements in build success and coverage do not by themselves fix assertion quality; the paper notes run-success per test case improves little, so assertion generation remains open."],"supporting_citations":[{"why":"Identifies Autoware as the industrial ADS platform under study and source of all experimental data.","marker":"[4]"},{"why":"Tree-sitter is used to parse C++ into ASTs for extracting metadata and building the call graph.","marker":"[2]"},{"why":"Provides the few-shot in-context learning mechanism that justifies using calling functions as prompt examples.","marker":"[6]"},{"why":"Represents prior LLM-based unit test generation on Java/Python, the baseline the Autoware results are contrasted against.","marker":"[13]"},{"why":"Another prior LLM unit test generation work used as baseline for evaluating ChatGPT-style models.","marker":"[18]"},{"why":"CodaMosa shows LLM-accelerated test generation, a related approach that informs the proposed pipeline.","marker":"[10]"},{"why":"AFL is cited as a classic C++ fuzzer that cannot handle Autoware's dependencies, motivating the shift to LLMs.","marker":"[1]"},{"why":"KLEE is cited as a classic symbolic execution tool that likewise faces barriers on Autoware, motivating the LLM route.","marker":"[7]"}],"fun_headline_variants":["Context-aware prompts lift Autoware unit test coverage","Missing context, not skill, limits LLM Autoware tests","AwTest-LLM: adding code context boosts Autoware test pass rates","Autoware unit tests improve when LLMs get code context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation only draws functions that have branches and input arguments and come from the 40 of 103 Autoware packages that already have developer-written tests, so the measured improvements may not hold for the many packages without any tests or for branchless functions; the paper's own uncovered dataset reports inconsistent function counts (812 vs. 736), which further weakens the representativeness of the result.","fun_headline_variants_meta":{"raw":{"variants":["Context-aware prompts lift Autoware unit test coverage","Missing context, not skill, limits LLM Autoware tests","AwTest-LLM: adding code context boosts Autoware test pass rates","Autoware unit tests improve when LLMs get code context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001275,"raw_usage":{"total_tokens":5182,"prompt_tokens":881,"completion_tokens":4301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":4227}},"tokens_in":497,"tokens_out":4301,"duration_ms":28293,"temperature":1.0,"reasoning_tokens":4227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:35:34.506081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AwTest-LLM on the 60+ Autoware packages that currently lack any developer tests and on branchless functions, and compare the build success rate and line coverage against the paper's reported numbers. If the gains disappear on that broader set, the central claim about improving Autoware testing does not generalize. Alternatively, an ablation that removes the call-graph examples or the CMake dependency information from the prompt and shows no drop in build success would indicate the proposed pipeline's specific components are not the cause of the improvement.","supporting_citations":[{"cited_title":"Autoware - the world’s leading open-source software project for autonomous driving","cited_arxiv_id":null,"evidence_quote":"Identifies Autoware as the industrial ADS platform under study and source of all experimental data."},{"cited_title":"Tree-sitter","cited_arxiv_id":null,"evidence_quote":"Tree-sitter is used to parse C++ into ASTs for extracting metadata and building the call graph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents prior LLM-based unit test generation on Java/Python, the baseline the Autoware results are contrasted against."},{"cited_title":"american fuzzy lop","cited_arxiv_id":null,"evidence_quote":"AFL is cited as a classic C++ fuzzer that cannot handle Autoware's dependencies, motivating the shift to LLMs."}],"review_version":1}