Fine-grained Testing for Autonomous Driving Software: a Study on Autoware with LLM-driven Unit Testing

An Ran Chen; Lei Ma; Renzhi Wang; Wenhan Wang; Xuan Xie; Yuheng Huang

REVIEW 4 major objections 5 minor 20 references

Fine-grained Testing for Autonomous Driving Software: a Study on Autoware with LLM-driven Unit Testing

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AwTest-LLM, a dependency-aware prompt method, raises build success and line coverage for LLM-generated unit tests on Autoware packages.

desk verdict A useful exploratory study with a real empirical payoff, but the headline numbers need a cleanup pass and the benchmark's scope is narrower than it looks. read the letter →

arxiv 2501.09866 v1 pith:NCEPHQA4 submitted 2025-01-16 cs.SE

Wenhan Wang , Xuan Xie , Yuheng Huang , Renzhi Wang , An Ran Chen , Lei Ma This is my paper · ORCID

classification cs.SE

keywords AutowareunittestingLLMtestgenerationautonomousdrivingsoftwarecoveragebuildsuccessrateC++few-shotprompting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies unit testing for Autoware, an industrial autonomous-driving software stack written in C++ and built on ROS 2. It finds that human-written tests cover only a small fraction of functions, and that large language models asked to write tests from raw source code fail to compile most of the time. The paper identifies the dominant failure modes—wrong symbols, missing headers, wrong namespaces, private access—and proposes AwTest-LLM, which enriches the prompt with namespace and header metadata, CMake-level external dependencies, and call-graph examples of the focal function. In the paper's evaluation, this raises the build success rate and line coverage of generated tests across Autoware packages compared with a basic prompt. The point of the work is that fine-grained, function-level testing for ADS software is both needed and partially attainable with context-aware LLM prompting.

What carries the argument

The central object is the AwTest-LLM pipeline, a five-step preprocessing and prompting procedure. It parses each package's C++ files into abstract syntax trees, extracts the focal function's namespace, headers, and class metadata, builds a call graph to select calling functions as few-shot in-context examples, generates documentation for undocumented functions, and parses CMake files for external dependencies. These elements are assembled into the final prompt alongside the focal file, giving the LLM the API surface and usage patterns it needs to produce compilable tests. The pipeline is the mechanism that carries the argument: the paper's comparisons of basic prompts versus AwTest-LLM prompts attribute the gains to this added context.

What would settle it

Run AwTest-LLM on the 60+ Autoware packages that currently lack any developer tests and on branchless functions, and compare the build success rate and line coverage against the paper's reported numbers. If the gains disappear on that broader set, the central claim about improving Autoware testing does not generalize. Alternatively, an ablation that removes the call-graph examples or the CMake dependency information from the prompt and shows no drop in build success would indicate the proposed pipeline's specific components are not the cause of the improvement.

Watch

Extended reading notes

Core claim

The central claim is that a large part of LLM failure in unit-testing Autoware comes from missing contextual information, not from the model's inability to write assertions. AwTest-LLM supplies that context: an AST pass extracts namespaces, associated headers, and class metadata; a call graph provides functions that invoke the focal function as few-shot examples; CMake parsing exposes external package dependencies; and undocumented functions get an LLM-generated description. On the 390-function covered benchmark, overall build success rises from 22.3% to 29.5% (GPT-4o-mini) and from 26.7% to 33.3% (GPT-4o), with line coverage rising from 13.9% to 18.4% and from 11.4% to 23.0%. On the 736-function uncovered dataset, improvements are smaller and absolute rates stay low, which the paper reads as evidence that never-tested functions remain a hard frontier.

Load-bearing premise

The evaluation only draws functions that have branches and input arguments and come from the 40 of 103 Autoware packages that already have developer-written tests, so the measured improvements may not hold for the many packages without any tests or for branchless functions; the paper's own uncovered dataset reports inconsistent function counts (812 vs. 736), which further weakens the representativeness of the result.

Editorial extensions

If this is right

If AwTest-LLM works as reported, LLM-driven unit testing becomes a practical complement to human-written tests for industrial C++/ROS codebases.
The error taxonomy (symbol, header, namespace, type, access) gives a concrete checklist for improving future test-generation prompts.
The uncovered-dataset results imply that packages without any human tests are the harder target; tooling should focus there.
The approach could be applied to other ROS-based autonomous driving stacks or large C++ projects with similar build systems.
Improvements in build success and coverage do not by themselves fix assertion quality; the paper notes run-success per test case improves little, so assertion generation remains open.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the dependency-aware context is the cause, then ablation studies could isolate which component (headers vs. CMake deps vs. call-graph examples) contributes most; the paper does not run these, but the architecture implies testable claims.
The low gains on the uncovered dataset suggest that coverage gains may partly reflect the LLM imitating existing test styles; on functions with no test precedent, the method would need a different mechanism, such as generated documentation or symbolic execution.
The dataset-size discrepancy (812 uncovered functions in Section 3 vs. 736 in Table 6) suggests either filtering after generation or a reporting inconsistency; reconciling it would make the evaluation reproducible.
The approach might generalize beyond testing: the same context extracted for Autoware could feed other code-generation tasks such as documentation or repair.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know about this one: it is the first quantitative look at unit test coverage in Autoware, and it shows clearly that LLM-generated C++/ROS tests are bottlenecked by build failures, not by test logic. The paper earns its keep with the coverage audit (2206 functions across 40 tested packages, 36.4% function coverage) and the error taxonomy from manual analysis of build failures. That taxonomy—symbol errors, missing namespaces, private access, hallucinated members—is genuinely useful for anyone trying to get LLMs to write compiling C++ tests against a large real codebase. The proposed AwTest-LLM pipeline (AST metadata, call-graph examples, CMake dependency parsing) is a sensible recipe, and the measured gains on the covered dataset are plausible: build success up about 7 points for GPT-4o-mini and 6.6 for GPT-4o, line coverage up 4.5 and 11.6 points respectively. Those are modest gains, but the direction is consistent across most modules, and the evaluation is external: generated tests actually have to compile and execute against the real Autoware build.

The soft spots are real but mostly fixable. The dataset size inconsistency stands out: Section 3 says 812 uncovered functions, while Table 6 sums to 736. The abstract's total of 1126 matches 390+736, so 812 is likely a typo, but it needs to be fixed because it undermines trust in the numbers. More substantively, the benchmark only includes functions with branches and input arguments drawn from the 40 of 103 packages that already have developer-written tests. That excludes the majority of packages and all branchless functions, so the measured improvements do not necessarily transfer to the untested bulk of the codebase. The paper is honest about this—RQ3 explicitly says uncovered functions remain very challenging—but the headline claim in the abstract is broader than the evidence supports. There are also no error bars or repeated runs, and module-level results are mixed (e.g., GPT-4o's control module build success drops 11.6 points), so the overall averages may hide instability.

On balance this is a solid empirical study for an under-explored area, and the authors are not overselling: they call it a study and early exploration, they identify the main failure mode, and they show a plausible mitigation. It deserves a serious referee, but a revised version should fix the count inconsistency, report variance or at least per-run data, and sharply delimit the generalization claim to functions like those in the benchmark. If you work on LLM-based test generation or ADS testing, this is worth citing for the dataset and the error taxonomy; the AwTest-LLM recipe is not groundbreaking, but it is reproducible and useful.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical study of unit testing for the Autoware autonomous driving framework, covering both human-written tests and LLM-generated tests. It reports that human-written tests have low coverage (36.4% function coverage across 40 tested packages), and that naive LLM prompts yield low build success rates (overall BS_file 22.3% for GPT-4o-mini and 26.7% for GPT-4o on a covered dataset of 390 functions). Based on a manual analysis of build errors (Table 3), the authors propose AwTest-LLM, which enriches the prompt with namespace/header metadata, call-graph examples, LLM-generated documentation, and CMake dependency information. On the covered dataset, AwTest-LLM improves overall BS_file by +7.2% and line coverage by +4.5% for GPT-4o-mini, and +6.6% and +11.6% for GPT-4o (Table 5). On an uncovered dataset (Table 6), results are lower and the paper concludes that untested functions are still very challenging. The central claim is that AwTest-LLM improves build success rate and coverage of LLM-generated tests for Autoware.

Significance. If the central claim holds, this is a worthwhile contribution to the software-testing literature: it provides the first fine-grained unit-testing study of an industrial ADS codebase, a careful qualitative error analysis, and a practical prompt-engineering method that demonstrably reduces build failures. The evaluation is external and grounded in the actual Autoware codebase: build success, run success, and lcov line coverage are measured against real compilation and execution. The manual categorization of error types (Tables 3 and 4) is useful diagnostic information for the community. However, the strength of the empirical claim is limited by the issues detailed below, particularly the dataset-size inconsistency and lack of statistical validation.

major comments (4)

[Section 3 vs. Table 6] Section 3 states that the uncovered dataset comprises 812 functions not covered by official test cases, but Table 6 lists function counts that sum to 736 across the eight modules (46+19+1+10+10+639+8+3). This 76-function discrepancy is not explained. Moreover, Table 1 reports 1129 branched functions in covered packages, of which 390 are covered, implying 739 uncovered branched functions, which matches neither 812 nor 736. The inconsistency must be resolved and the actual dataset composition and selection criteria clarified, because it directly affects the validity of the results in Table 6 and the paper's general claims.
[Section 3.2, Table 5] All reported improvements are based on a single run per configuration, with no standard deviations, confidence intervals, or statistical tests. Since LLM test generation is stochastic, the headline differences (e.g., overall BS_file +7.2% for GPT-4o-mini, line coverage +11.6% for GPT-4o) could be within run-to-run noise. The module-level results also show notable regressions, such as GPT-4o BS_file dropping by 11.6% on the control module and RS_case dropping by 26.4% on the evaluator module. Multiple runs (e.g., at least 5) and paired significance tests are needed to support the claim that AwTest-LLM improves these metrics.
[Section 3, benchmark selection] The evaluation benchmark is restricted to functions with branches and input arguments drawn only from the 40 of 103 packages that already have developer-written test cases. This excludes all branchless functions and the majority of the codebase (63 packages without tests). Consequently, the measured improvements (e.g., Table 5) do not support the broad statement in the abstract and conclusion that AwTest-LLM improves unit testing 'across Autoware packages.' The general claim should be tempered, or the benchmark should be widened to include packages without existing tests and branchless functions to demonstrate generality.
[Section 3.3.1, Document Generation] The proposed approach includes an LLM-based document generation step for undocumented functions, but the paper does not validate the accuracy of these generated documents or isolate their effect on the final results. Without an ablation study (e.g., with and without document generation, with and without call-graph examples), it is unclear which components of AwTest-LLM are responsible for the measured improvements. This is important for understanding the mechanism and for reproducibility.

minor comments (5)

[Author affiliation] The email addresses use 'ualbera.ca' instead of 'ualberta.ca'.
[Section 3.1, step 2] 'CmakeLists.txt' should be 'CMakeLists.txt'.
[Table 4] The table does not show counts for 'Assertion on throw', 'Assertion on death behaviors', 'Invalid argument', and 'Other errors'; please complete the table or note zero counts.
[Abstract and Section 4] The abstract states the approach improves 'test case pass rates,' but the overall RS_case actually decreases for GPT-4o-mini (Table 5, -2.2%) and only marginally increases for GPT-4o (+1.5%); rephrase to focus on build success and coverage.
[Section 4] The framework name is written as 'Awtest-LLM' in the conclusion, which is inconsistent with 'AwTest-LLM' used elsewhere.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: AwTest-LLM's build-success and coverage gains are measured externally against the Autoware codebase and are not forced by construction.

full rationale

The paper's derivation chain is empirical rather than definitional. The proposed method adds package metadata, dependency information, and call-graph examples to the LLM prompt, and the claimed improvements are measured by whether generated test files compile, run, and increase lcov line coverage on Autoware packages. None of these outcomes is determined by the prompt content: the LLM can still produce uncompilable or low-coverage tests, and indeed Tables 2 and 5 show substantial failures and mixed module-level results. No parameter is fitted to the evaluation data, and no prediction is statistically forced by an equation. The only self-citations, such as TESTEVAL, merely list prior LLM test-generation benchmarks and are not load-bearing; the headline result does not rest on a self-citation chain or on an imported uniqueness theorem. The mismatch between 812 uncovered functions in Section 3 and 736 in Table 6 is an internal-consistency and selection threat to generalizability, not a circularity, because it does not make the measured build-success or coverage numbers equal to the method's inputs by construction. The evaluation is external to the method's own fitted values, so the central claim retains independent content.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on measurement assumptions (lcov validity, branch representativeness, LLM-doc reliability) rather than fitted constants. No numerical free parameters are introduced; the pipeline's design choices are qualitative. The main exposure is that all conclusions are single-run empirical measurements on a narrow function subset.

assumptions (4)

domain assumption lcov line coverage is a faithful proxy for test effectiveness in this comparison.
The paper uses lcov through colcon-lcov-result as the sole coverage metric and explicitly discards branch coverage as inconsistent, without validating line coverage against other measures (Section 3.1).
domain assumption The awsim-stable branch of Autoware is representative of industrial ADS software for the purpose of this study.
Section 2 selects this branch because it is stable and integrates with AWSIM, but does not justify generalizability to other ADS codebases.
domain assumption Functions with branches and input arguments, in packages that already have tests, adequately represent the difficulty of unit testing Autoware.
Section 3 builds the benchmark from this subset, excluding all packages without tests and all branchless functions, which biases the difficulty assessment.
ad hoc to paper LLM-generated documentation for undocumented functions is accurate enough not to degrade test generation.
Section 3.3.1 step 3 lets the LLM write the function description used later in the test-generation prompt; no validation of these generated documents is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Fine-grained Testing for Autonomous Driving Software: a Study on Autoware with LLM-driven Unit Testing." pith.science (2026). https://pith.science/paper/NCEPHQA4

@misc{pith2026250109866,
  author       = {Pith},
  title        = {Pith review of: Fine-grained Testing for Autonomous Driving Software: a Study on Autoware with LLM-driven Unit Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCEPHQA4}},
  note         = {Machine review of arXiv:2501.09866}
}

read the original abstract

Testing autonomous driving systems (ADS) is critical to ensuring their reliability and safety. Existing ADS testing works focuses on designing scenarios to evaluate system-level behaviors, while fine-grained testing of ADS source code has received comparatively little attention. To address this gap, we present the first study on testing, specifically unit testing, for ADS source code. Our study focuses on an industrial ADS framework, Autoware. We analyze both human-written test cases and those generated by large language models (LLMs). Our findings reveal that human-written test cases in Autoware exhibit limited test coverage, and significant challenges remain in applying LLM-generated tests for Autoware unit testing. To overcome these challenges, we propose AwTest-LLM, a novel approach to enhance test coverage and improve test case pass rates across Autoware packages.

Figures

Figures reproduced from arXiv: 2501.09866 by the authors.

**Figure 1.** The overview of AwTest-LLM. From the generated ASTs, we extract dependency information for each function, including its namespace, associated header files, and class metadata(if the function is a class method). (2) Call Dependency Extraction: We also build a call graph from ASTs for each package to extract call dependencies between functions. From the call graph, we can extract functions that call the focal functi… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 12 canonical work pages

[1]

american fuzzy lop

2013. american fuzzy lop. https://github.com/google/AFL

work page 2013
[2]

Tree-sitter

2019. Tree-sitter. https://tree-sitter.github.io/tree-sitter/

work page 2019
[3]

ROS - Robot Operating System

2022. ROS - Robot Operating System . https://www.ros.org/

work page 2022
[4]

Autoware - the world’s leading open-source software project for autonomous driving

2024. Autoware - the world’s leading open-source software project for autonomous driving. https://github.com/autowarefoundation/autoware

work page 2024
[5]

2024. A WSIM. https://github.com/tier4/AWSIM

work page 2024
[6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

2020
[7]

Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.. In OSDI, Vol. 8. 209–224

2008
[8]

Mingfei Cheng, Yuan Zhou, and Xiaofei Xie. 2023. Behavexplor: Behavior di- versity guided testing for autonomous driving systems. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis . 488–500

work page 2023

Show all 20 references

[9]

Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. 2024. Testgeneval: A real world unit test generation and test completion benchmark. arXiv preprint arXiv:2410.00752 (2024)

2024 arXiv
[10]

Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen
[11]

Guannan Lou, Yao Deng, Xi Zheng, Mengshi Zhang, and Tianyi Zhang. 2022. Testing of autonomous driving systems: where are we and where should we go?. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engin...

2022
[12]

Niels Mündler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. 2024. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[13]

Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Mu- rali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-aware prompting: A study of coverage-guided test generation in regression setting using llm. Pro- ceedings of the ACM on Software Engineering 1, FSE...

2024
[14]

Shuncheng Tang, Zhenya Zhang, Yi Zhang, Jixiang Zhou, Yan Guo, Shuang Liu, Shengjian Guo, Yan-Fu Li, Lei Ma, Yinxing Xue, et al . 2023. A survey on automated driving system testing: Landscapes and trends. ACM Transactions on Software Engineering and Methodology 32, 5 (2023), 1–62

2023
[15]

Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th international conference on software engineering . 303–314

2018
[16]

Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2024. TESTEVAL: Bench- marking Large Language Models for Test Case Generation. arXiv preprint arXiv:2406.04531 (2024)

2024 arXiv
[17]

Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering . 1258–1268

2024
[18]

Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726

2024
[19]

Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khur- shid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In Proceedings of the 33rd ACM/IEEE international conference on automated software enginee...

2018
[2023]

In 2023 IEEE/ACM 45th International Conference on Soft- ware Engineering (ICSE)

Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on Soft- ware Engineering (ICSE). IEEE, 919–931

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

american fuzzy lop

2013. american fuzzy lop. https://github.com/google/AFL

work page 2013

[2] [2]

Tree-sitter

2019. Tree-sitter. https://tree-sitter.github.io/tree-sitter/

work page 2019

[3] [3]

ROS - Robot Operating System

2022. ROS - Robot Operating System . https://www.ros.org/

work page 2022

[4] [4]

Autoware - the world’s leading open-source software project for autonomous driving

2024. Autoware - the world’s leading open-source software project for autonomous driving. https://github.com/autowarefoundation/autoware

work page 2024

[5] [5]

2024. A WSIM. https://github.com/tier4/AWSIM

work page 2024

[6] [6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

2020

[7] [7]

Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.. In OSDI, Vol. 8. 209–224

2008

[8] [8]

Mingfei Cheng, Yuan Zhou, and Xiaofei Xie. 2023. Behavexplor: Behavior di- versity guided testing for autonomous driving systems. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis . 488–500

work page 2023