Verification and Validation of Autonomous Systems

Shreyas Renukuntla; Sneha Sudhir Shetiya; Vikas Vyas

REVIEW 3 major objections 6 minor 16 references

Verification and Validation of Autonomous Systems

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A V-model lifecycle with parallel testing phases is the route to reliable autonomous-vehicle software.

desk verdict A thin, poorly-sourced survey of well-known AV testing practice; the V-model premise is asserted without evidence and the paper contradicts itself on simulation, so it is not a serious research contribution. read the letter →

arxiv 2411.13614 v1 pith:UJ3WOYZJ submitted 2024-11-20 cs.SE cs.AI

Sneha Sudhir Shetiya , Vikas Vyas , Shreyas Renukuntla This is my paper · ORCID

classification cs.SEcs.AI

keywords verificationandvalidationautonomousvehiclesV-modelsoftware-in-the-loophardware-in-the-loopvehicle-in-the-loopfinalproductacceptancetestsimulation-basedtesting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper brings together the verification and validation practices that apply to autonomous-vehicle software and presents them as one lifecycle built on the V-model. The V-model pairs every development phase on the left side—requirements, analysis, design, coding—with a testing phase on the right, from unit and subsystem tests through integration and acceptance tests. The paper argues that this structure prevents defects, catches those that slip through, and builds assurance in the product development phase. It also makes the case that simulation, software-in-the-loop, hardware-in-the-loop, vehicle-in-the-loop, calibration checks, and final product acceptance testing are the layers that make the lifecycle work in practice. The payoff, if the paper is right, is a concrete route to reliable self-driving software that can be deployed on public roads.

What carries the argument

The central object is the V-model (also called the V-cycle), a variant of the waterfall model in which testing phases run parallel to development phases. The key identity is the one-to-one mapping between each left-side phase, from requirements through design and coding, and a right-side test activity, from unit and subsystem tests through integration and acceptance tests, with each identified defect fed back to the corresponding left-side phase for faster correction. The paper then populates that skeleton with named mechanisms: MIL, SIL, and HIL for integration testing, VIL for combining a real vehicle with a simulated environment, calibration checks on camera intrinsic and extrinsic parameters, and final product acceptance testing covering road tests, functional safety, cybersecurity, and regulatory compliance.

What would settle it

A controlled comparison of two similar autonomous-driving development programs, one using the V-model and one using an alternative lifecycle, would settle whether the paper's organizing claim holds: if the alternative produces equal or fewer field-reported safety defects, the paper's central recommendation loses its force.

Watch

Extended reading notes

Core claim

The central claim is that the V-model is the right organizing framework for autonomous-vehicle software development, with a testing phase running in parallel to every development phase. The paper equates verification with the objective tests that confirm a product meets the metrics of its requirements, and validation with demonstrating that the product meets the original intent; the mapping between the left and right sides of the V makes both explicit. From there it treats simulation, SIL, HIL, and VIL as complementary layers that combine virtual and physical testing, and calibration plus final product acceptance testing as the gates ensuring safety, reliability, and regulatory compliance before a vehicle is deployed. On this view, the path to highly reliable autonomous vehicles is not any single test but the disciplined application of the whole chain.

Load-bearing premise

In Section I.A, the paper assumes the V-model is the standard and appropriate lifecycle for autonomous-vehicle software development, and it offers no comparative evidence that this framework outperforms agile, spiral, or other process models.

Editorial extensions

If this is right

Following the V-model gives every development phase a matching test phase, so defects found on the right side can be traced back to the left side and fixed with shorter turnaround.
Simulation, SIL, HIL, and VIL together let teams test many scenarios cheaply and safely, reducing reliance on expensive physical prototypes and road testing.
Calibration accuracy must be checked and re-checked, because sensor mounting, data quality, and cross-calibration affect the reliability of perception.
Final product acceptance testing, including road tests, functional safety checks, cybersecurity tests, and regulatory compliance, is a required gate before an autonomous vehicle can be deployed.
Model-based design is a promising way to address the operational design domain problem by simulating many scenarios and edge cases that cannot be exhaustively tested on real roads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension of the paper's position is that teams using the V-model and layered X-in-the-loop testing will exhibit lower defect-escape rates than teams relying only on road testing; that comparison is not reported in the paper but follows from its argument.
The paper leaves open how to verify learned perception and prediction components, whose behavior is data-dependent; one could extend the V-model with explicit test-time monitoring and continuous data feedback loops.
The paper's emphasis on calibration suggests that a standardized set of calibration accuracy metrics across manufacturers would be a natural next step, since it notes no standard exists today.
A hybrid physical-virtual validation strategy implies that investment should shift toward reusable scenario libraries and simulation infrastructure rather than exclusive physical-proving-ground expansion.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is not a research paper. It is a summary of standard V-cycle testing content—SIL, HIL, VIL, calibration, and acceptance testing—that you can find in any automotive software engineering textbook. There is no new methodology, no comparative analysis, no empirical data, and no formal treatment. The reader's REJECT verdict is right.

To give credit where it is due: the high-level descriptions of the test levels are broadly accurate. The distinction between verification and validation is stated correctly, and the basic idea of VIL as a hybrid of simulation and real vehicle testing is conveyed without any serious errors. As a very short primer for someone completely new to the topic, the taxonomy could be useful. That is the ceiling of its value.

The soft spots are real and they matter. Section I.A asserts that the V-model is "Generally... considered" and Section I.C goes further, claiming "all the product companies have adopted V-cycle model"—with no citation and no consideration of agile or spiral lifecycles. That is an empirical, universal claim sitting at the foundation of the whole paper. It might be true, but the paper gives the reader no reason to believe it, and if it is not true the recommendations lose their footing. The citation errors are sloppy: reference [2] is a protein-structure paper, not the Cameo modeling tool, and reference [12] is about seismic data acquisition, not software-in-the-loop testing. Those suggest the reference list was not checked against the text.

The internal contradiction flagged in the stress test is real. Section IX.B says "Simulation only allows testing of a few vehicles and test scenarios," while Section III touts data augmentation and Section IX.C claims simulation "can simulate millions of scenarios." Both cannot be right, and since the paper's main practical recommendation is to lean on simulation, this inconsistency undercuts the argument. The writing is also poor in places, with incomplete sentences and awkward phrasing.

Who is this for? A reader who has literally never heard of HIL, SIL, or VIL might get a five-minute orientation. That is not enough to justify peer review time at a research venue. It reads like a draft of a blog post or an undergraduate project report, not a contribution to the literature. The authors would need to ground the V-model claim in industry data, fix the citations, resolve the simulation contradiction, and add some comparative or empirical analysis to make this worthy of a survey venue.

Recommendation: desk reject. Do not send it to reviewers.

Referee Report

3 major / 6 minor

Summary. This is a short expository overview of verification and validation (V&V) practices for autonomous-vehicle software. The paper describes the V-model lifecycle, simulation, software-in-the-loop (SIL), hardware-in-the-loop (HIL), vehicle-in-the-loop (VIL), calibration testing, and a final product acceptance test (FPAT), and it argues that model-based design and simulation can mitigate the cost and scenario-coverage challenges of physical testing.

Significance. If the overview were technically accurate and properly sourced, it could serve as a broad introductory orientation for practitioners entering the field. The high-level descriptions of SIL, HIL, and VIL in Sections VI, VII, and V are largely consistent with common engineering knowledge, which is a modest strength. However, the paper is entirely descriptive and offers no new evidence, methods, data, or falsifiable predictions. Its usefulness as a survey is seriously undermined by several incorrect citations, an unsupported universal claim about industry adoption of the V-model, and an internal contradiction about the capacity of simulation. These problems affect the paper's central recommendation to rely on simulation-based testing, so the manuscript does not currently meet the standard of a reliable scholarly survey.

major comments (3)

[Section I.C (and I.A)] The paper asserts in Section I.C that 'all the product companies have adopted V-cycle model for development' and in Section I.A that 'Generally, the V-model is considered for software development and testing methodology,' but no citation, industry survey, or comparative lifecycle analysis supports these empirical claims. This is load-bearing because the entire paper is organized around the V-model. The authors should either provide concrete evidence for this universal claim or explicitly limit the paper's scope to V-model-based development processes.
[Sections I.B, VI.A, I.B.6] Several references do not support the statements to which they are attached. Reference [2], cited for the Cameo modeling tool in Section I.B.2, is actually a protein-structure evaluation paper (CAMEO). Reference [12], cited in Section VI.A for SIL data analysis, is a seismic data acquisition system paper. Reference [5], cited in Section I.B.6 for ETAS HIL, is a connected-automated-vehicle HIL paper rather than a description of ETAS HIL products. Because this paper is a review, citation accuracy is central to its value; these errors require correction throughout.
[Section IX.B vs Section III and Section IX.C] Section IX.B states that 'Simulation only allows testing of a few vehicles and test scenarios,' which directly contradicts the 'Data Augmentation' benefit described in Section III and the claim in Section IX.C that model-based design can 'simulate millions of scenarios.' This internal inconsistency undermines a key recommendation of the paper—that simulation-based testing is the primary way to overcome the ODD coverage problem. The authors must reconcile these statements or the survey's guidance on simulation is not coherent.

minor comments (6)

[Author line] "3th" should be "3rd" in the author affiliation line.
[Section I.B.5] The phrase "one bug block" appears to be a typo for "one big block," and the referenced Figure 1 is not visible in the manuscript, making it difficult to follow the V-cycle description.
[Section I and throughout] There are numerous grammatical errors, e.g., "an increase reliability" should be "an increased reliance," and "which overflow one after the other" should be "which flow one after the other." A careful proofreading pass is needed.
[Section IV] The calibration discussion reads as a generic camera-calibration tutorial and is not tied to autonomous-vehicle V&V metrics; the authors should connect this material to the validation workflow or remove it.
[Section VIII.A] The formatting of "Regulatory Compliance Tests" is broken: it appears as a bullet with no following content. The list should be restructured for readability.
[References] Reference [1] contains a malformed author string "Craig L. [1] Silver"; the author name should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning found: the paper is a descriptive survey of V&V methods, with no derivation chain or fitted-prediction structure.

full rationale

This paper is a narrative overview of verification and validation (V&V) techniques for autonomous vehicles. It presents no equations, no formal derivation, no fitted parameters, and no predictive claim that could be equivalent to its inputs by construction. The central content—descriptions of the V-model, simulation, SIL, HIL, VIL, calibration, and FPAT—is assembled from external standards and cited prior work, and the conclusions simply summarize those descriptions. The V-model assumption in Section I.A is asserted without citation or comparative evidence, but an unsupported empirical premise is a correctness and rigor concern, not circular reasoning; the paper does not define the V-model in terms of its own conclusions. Similarly, the internal contradiction between Section IX.B's statement that 'Simulation only allows testing of a few vehicles and test scenarios' and Section IX.C's claim that model-based design 'can simulate millions of scenarios' is an inconsistency, not a circular step. No self-citation chain is load-bearing: the authors do not rely on their own prior results to justify the framework. Because there is no derivation to trace back to its own inputs, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no free parameters, no new entities, and no formal axioms. It relies on established engineering assumptions about the V-model and simulation, which are stated without empirical support.

assumptions (2)

domain assumption The V-model is the standard and appropriate development lifecycle for autonomous vehicle software.
Section I.A presents the V-model without justification, and the rest of the paper relies on it as the organizing framework for verification and validation activities.
domain assumption Simulation-based testing can sufficiently emulate real-world conditions to validate autonomous vehicle safety.
Section VI and IX assert that simulation is a viable alternative to physical testing, but the paper does not provide evidence for the fidelity or adequacy of simulated environments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Verification and Validation of Autonomous Systems." pith.science (2026). https://pith.science/paper/UJ3WOYZJ

@misc{pith2026241113614,
  author       = {Pith},
  title        = {Pith review of: Verification and Validation of Autonomous Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJ3WOYZJ}},
  note         = {Machine review of arXiv:2411.13614}
}

read the original abstract

This paper describes how to proficiently prevent software defects in autonomous vehicles, discover and correct defects if they are encountered, and create a higher level of assurance in the software product development phase. It also describes how to ensure high assurance on software reliability.

Figures

Figures reproduced from arXiv: 2411.13614 by the authors.

**Figure 1.** V-cycle Software model A. Phases of Software Product Development Generally, the V-model is considered for software development and testing methodology. The V-model is considered a variant of the traditional Waterfall approach. Besides the standard Waterfall methodology’s life cycle phases, which overflow one after the other in a linear development process. The V-model is categorized by a testing phase running in pa… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

[2]

best single template

Haas J, Gumienny R, Barbato A, Ackermann F, Tauriello G, Bertoni M, Studer G, Smolinski A, Schwede T. Introducing “best single template” models as reference baseline for the Continuous Automated Model Evaluation (CAMEO). Proteins: Structure, Function, and Bioinformatics. 2019 Dec;87(12):1378-87

work page 2019
[12]

The SIL data acquisition system—at present and beyond year 2000

Boovarsson R, R ¨ognvaldsson ST, Slunga R, Kjartansson E. The SIL data acquisition system—at present and beyond year 2000. Physics of the Earth and Planetary Interiors. 1999 Jun 1;113(1-4):89-101

work page 2000
[5]

Hardware-In-the-Loop for Connected Automated Vehicles Testing in Real Traffic

Kim Y , Tay S, Guanetti J, Borrelli F, Miller R. Hardware-in-the-loop for connected automated vehicles testing in real traffic. arXiv preprint arXiv:1907.09052. 2019 Jul 21

work page Pith review arXiv 1907
[1]

Developing and managing embedded systems and products: methods, techniques, tools, processes, and teamwork

Fowler K, editor. Developing and managing embedded systems and products: methods, techniques, tools, processes, and teamwork. Elsevier; 2014 Aug 30

work page 2014
[3]

Design and Development of Automotive Do- main Needs Using Rhapsody Model Manager

Varsha T, Hegde V . Design and Development of Automotive Do- main Needs Using Rhapsody Model Manager. In2023 7th International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS) 2023 Nov 2 (pp. 1-6). IEEE

work page 2023
[4]

ISO 26262 Compliant Automatic Requirements-Based Testing for TargetLink

Brockmeyer U, Valea A, Gross M. ISO 26262 Compliant Automatic Requirements-Based Testing for TargetLink. InEmbedded Real Time Software and Systems (ERTS2012) 2012 Feb 1

work page 2012
[6]

Compute solution for tesla’s full self-driving computer

Talpes E, Sarma DD, Venkataramanan G, Bannon P, McGee B, Floering B, Jalote A, Hsiong C, Arora S, Gorti A, Sachdev GS. Compute solution for tesla’s full self-driving computer. IEEE Micro. 2020 Feb 24;40(2):25- 35

work page 2020
[7]

Operational design domain for automated driving systems

Czarnecki K. Operational design domain for automated driving systems. Taxonomy of Basic Terms “, Waterloo Intelligent Systems Engineering (WISE) Lab, University of Waterloo, Canada. 2018 Jul 21;1

work page 2018

Show all 16 references

[8]

Heuristic approach for real-time multi- agent trajectory planning under uncertainty

Tahir H, Syed MN, Baroudi U. Heuristic approach for real-time multi- agent trajectory planning under uncertainty. IEEE Access. 2019 Dec 27;8:3812-26

2019
[9]

Camera calibration

Zhang YJ. Camera calibration. In3-D Computer Vision: Principles, Al- gorithms and Applications 2023 Feb 1 (pp. 37-65). Singapore: Springer Nature Singapore

2023
[10]

Vehicle-in-the-loop (VIL) and scenario-in-the-loop (SCIL) automotive simulation concepts from the perspectives of traffic simulation and traffic control

Horv ´ath MT, Lu Q, Tettamanti T, T¨or¨ok ´A, Szalay Z. Vehicle-in-the-loop (VIL) and scenario-in-the-loop (SCIL) automotive simulation concepts from the perspectives of traffic simulation and traffic control. Transport and Telecommunication Journal. 2019 Apr 1;20(2):153-61

2019
[11]

An Indoor Rapid Testing Platform for Autonomous Vehicles Using Vehicle-in-the- Loop Simulation

Cheng JJ, Xu ZG, Gao Y , Wang Z, Wang WW, Zhao XM. An Indoor Rapid Testing Platform for Autonomous Vehicles Using Vehicle-in-the- Loop Simulation. InCICTP 2020 2020 (pp. 411-422)

2020
[13]

Hardware-in-the- loop (hil) simulation technique for an automotive electronics course

Nissimagoudar PC, Mane V , Gireesha HM, Iyer NC. Hardware-in-the- loop (hil) simulation technique for an automotive electronics course. Procedia Computer Science. 2020 Jan 1;172:1047-52

2020
[14]

Functional safety and proof of compliance

Myklebust T, St ˚alhane T. Functional safety and proof of compliance. Springer; 2021 Dec

2021
[15]

Development of an energy efficient and cost effective autonomous vehicle research platform

Brown NE, Rojas JF, Goberville NA, Alzubi H, AlRousan Q, Wang C, Huff S, Rios-Torres J, Ekti AR, LaClair TJ, Meyer R. Development of an energy efficient and cost effective autonomous vehicle research platform. Sensors. 2022 Aug 11;22(16):5999

2022
[16]

(https://www.sciencedirect.com/science/article/pii/S0965856423002847)

Le Yi Koh, Kum Fai Yuen, Public acceptance of autonomous vehicles: Examining the joint influence of perceived vehicle performance and intelligent in-vehicle interaction quality,Transportation Research Part A: Policy and Practice,V olume 178,2023,103864,ISSN 0965-8564, https://...

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [2]

best single template

Haas J, Gumienny R, Barbato A, Ackermann F, Tauriello G, Bertoni M, Studer G, Smolinski A, Schwede T. Introducing “best single template” models as reference baseline for the Continuous Automated Model Evaluation (CAMEO). Proteins: Structure, Function, and Bioinformatics. 2019 Dec;87(12):1378-87

work page 2019

[2] [12]

The SIL data acquisition system—at present and beyond year 2000

Boovarsson R, R ¨ognvaldsson ST, Slunga R, Kjartansson E. The SIL data acquisition system—at present and beyond year 2000. Physics of the Earth and Planetary Interiors. 1999 Jun 1;113(1-4):89-101

work page 2000

[3] [5]

Hardware-In-the-Loop for Connected Automated Vehicles Testing in Real Traffic

Kim Y , Tay S, Guanetti J, Borrelli F, Miller R. Hardware-in-the-loop for connected automated vehicles testing in real traffic. arXiv preprint arXiv:1907.09052. 2019 Jul 21

work page Pith review arXiv 1907

[4] [1]

Developing and managing embedded systems and products: methods, techniques, tools, processes, and teamwork

Fowler K, editor. Developing and managing embedded systems and products: methods, techniques, tools, processes, and teamwork. Elsevier; 2014 Aug 30

work page 2014

[5] [3]

Design and Development of Automotive Do- main Needs Using Rhapsody Model Manager

Varsha T, Hegde V . Design and Development of Automotive Do- main Needs Using Rhapsody Model Manager. In2023 7th International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS) 2023 Nov 2 (pp. 1-6). IEEE

work page 2023

[6] [4]

ISO 26262 Compliant Automatic Requirements-Based Testing for TargetLink

Brockmeyer U, Valea A, Gross M. ISO 26262 Compliant Automatic Requirements-Based Testing for TargetLink. InEmbedded Real Time Software and Systems (ERTS2012) 2012 Feb 1

work page 2012

[7] [6]

Compute solution for tesla’s full self-driving computer

Talpes E, Sarma DD, Venkataramanan G, Bannon P, McGee B, Floering B, Jalote A, Hsiong C, Arora S, Gorti A, Sachdev GS. Compute solution for tesla’s full self-driving computer. IEEE Micro. 2020 Feb 24;40(2):25- 35

work page 2020

[8] [7]

Operational design domain for automated driving systems

Czarnecki K. Operational design domain for automated driving systems. Taxonomy of Basic Terms “, Waterloo Intelligent Systems Engineering (WISE) Lab, University of Waterloo, Canada. 2018 Jul 21;1

work page 2018