Pith. sign in

REVIEW 3 major objections 5 minor 45 references

On the Practices of Autonomous Systems Development: Survey-based Empirical Findings

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 2019 survey of 110 experts finds that autonomous systems in practice ran mostly on rule-based code, not machine learning.

desk verdict A transparent, genuinely useful 2019 snapshot of autonomy development practice, but the 'state-of-the-practice' framing overreaches its convenience sample. read the letter →

arxiv 2506.04438 v1 pith:UCY2YGUM submitted 2025-06-04 cs.SE

Katerina Goseva-Popstojanova , Denny Hood , Johann Schumann , Noble Nkwocha This is my paper
classification cs.SE
keywords autonomoussystemssurveystudystateofthepracticeverificationandvalidationmodel-basedsoftwareengineeringreusemachinelearningcomponents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish the state of the practice in developing autonomous systems as of 2019, using an anonymous online survey answered by 110 experts from industry and academia. It claims that high levels of autonomy were already common: for each of four autonomy tasks, 50% to 67% of autonomous components operated with full computer autonomy. It also claims that development was still dominated by traditional approaches, with rule-based algorithms used for 35% of components and machine learning for only 12%. The authors present these results as the first part of a longitudinal study meant to track how autonomous systems development evolves.

What carries the argument

The central object is the Autonomous Component (AUC), defined as a component that provides autonomous capabilities and supports autonomous operations; respondents provided details for 58 AUCs. The argument is carried by a 48-question branching online survey, administered from April 25 to June 20, 2019, whose sections cover domain, safety criticality, programming languages, autonomy levels, reuse, processes and standards, verification and validation, and bugs. The survey's four-level autonomy scale and its four information-processing tasks, adapted from prior frameworks, produce the headline percentages.

What would settle it

Repeating the survey with a sample that is randomly drawn or balanced across industry domains and finding markedly different rates of full autonomy or machine learning usage would falsify the state-of-the-practice claim. A simpler check is whether the 40% space share of respondents matches the actual distribution of autonomous-systems development effort across sectors.

Watch

Extended reading notes

Core claim

The paper's central claim is that in 2019, industrial practice in autonomous systems was more conventional than media attention to machine learning suggested. Across the four information-processing tasks used in the survey, between 50% and 67% of autonomous components had full computer autonomy, and the most common development methods were rule-based algorithms, planning systems, and statistical filtering. Only 9% of components used offline machine learning and 3% used online machine learning. The authors conclude that safety-critical systems with substantial autonomy were being built successfully with existing software technology, while naming system complexity, environmental uncertainty, and achieving the desired level of autonomy as the principal challenges.

Load-bearing premise

The 110 respondents, recruited through convenience sampling and snowballing, are representative of the broader population of autonomous systems developers, despite the sample's heavy skew toward space, aviation, and military domains.

Editorial extensions

If this is right

  • If these numbers describe 2019 practice, then full autonomy was not a distant prospect; it was already deployed across a majority of surveyed autonomous components.
  • If machine learning was used in only 12% of components, then assurance methods aimed at learning-enabled systems were addressing a small slice of the autonomous systems actually being built at that time.
  • If only 24% of systems went through certification and 26% of autonomous components were part of a certified system, then certification was a clear gap in 2019 practice.
  • If 25% of respondents reported reuse-specific bugs while only 19% verified reused artifacts, then reuse without dedicated verification and validation is a concrete risk in autonomous systems development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because 40% of respondents came from the space domain and another 29% from aviation or military, the 'state of the practice' likely describes safety-critical aerospace practice more accurately than it describes consumer automotive or service robotics; this is an inference from the reported domain distribution.
  • The sharpest test of the paper's claims will come from the planned second survey: if machine learning usage and certification rates have not moved by the mid-2020s, the 2019 baseline will look less like a lag and more like a persistent feature of the field.
  • The reuse findings hint that conventional assumptions about reuse lowering defect rates may not transfer to autonomous systems, because reuse-specific bugs were nearly as common as autonomy-specific bugs; the paper does not state this directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports the first wave of a longitudinal survey-based study of autonomous-systems development, based on an anonymous online survey administered in 2019. Of 129 respondents who started the survey, 110 reported having worked on autonomous systems and/or MBSwE and answered questions in six blocks covering industry domains, safety criticality, programming languages, code origins, autonomy details for up to 58 autonomous components (AUCs), reuse, processes and standards, verification and validation, and bugs. The main reported findings are a space/aviation/military-dominated respondent pool, high safety criticality, C/C++ as dominant languages, rule-based rather than ML algorithms (ML used for 12% of AUCs), full computer autonomy for 50–67% of AUCs across four autonomy tasks, low incidence of certification, and sparse V&V of security and reused artifacts. The paper claims four contributions: establishing the state-of-the-practice (C1), quantifying autonomy and reuse benefits and challenges (C2), identifying processes and standards (C3), and exploring V&V (C4), followed by practitioner recommendations.

Significance. If the descriptive results are accepted as an accurate snapshot of the 2019 respondent population, the paper is a valuable empirical complement to the many literature-review studies in this area. Its strengths are the structured survey design following Kitchenham and Pfleeger, the explicit reporting of the number of respondents behind each figure and table, the pilot study, the anonymous administration, and the stated intent to repeat the survey longitudinally. The usefulness of the findings as evidence about industry practice in general, however, is limited by the convenience sample, the domain skew toward space, aviation, and military, and the small denominators behind several headline percentages; these factors must be addressed before the 'state-of-the-practice' claim can be considered established.

major comments (3)
  1. [Section 1 (C1), Section 3, Section 5, Section 6] The central claim that the survey 'established the state-of-the-practice of developing autonomous systems' (C1) and the Section 6 conclusion that 'safety-critical systems with a substantial degree of autonomy have been successfully developed in the industry' generalize beyond the respondent pool, but the paper explicitly disclaims such generalization in Section 5: 'we cannot claim that the results based on one survey would be valid for all autonomous systems.' The sample was recruited via non-probabilistic convenience sampling and snowballing (Section 3), including contacts of the NASA-affiliated author team and over 300 authors of autonomy-related papers, and the respondent mix is heavily skewed toward space (40%), aviation (16%), and military (13%), with only 9% automotive (Figure 2). The authors' own prior MBSwE survey [24] had 33% automotive, which the paper itself cites as evidence that recruitment channels change the domain mix. Please reframe the contributions and conclusions as describing the surveyed sample, or provide substantive evidence of representativeness (e.g., comparison with known population demographics, weighting, or sensitivity analysis).
  2. [Section 4.2, Figures 7–8, Section 5] Key percentages are computed over very small and non-independent units: data about 58 AUCs were provided by 38 respondents, with 28 respondents reporting one AUC and the rest reporting multiple AUCs. Figure 7 is based on 50 AUCs and Figure 8 on 58 AUCs, so the headline full-autonomy figures (50–67%) and the ML-usage figure of 12% (7 of 58 AUCs) have wide sampling uncertainty and no adjustment for clustering of AUCs within respondents. Section 5's statement that 'these are fairly large sample sizes that allow drawing valid conclusions' is difficult to support for denominators of 29–58. Please report raw counts and confidence intervals for all key proportions, and either account for the clustering or qualify the precision of the estimates.
  3. [Section 4.5.1 vs. Section 6 recommendations] The paper reports two different numbers for V&V of security: Section 4.5.1 states 'only 11% of respondents verified and validated security' (about 4 of the 36 respondents who answered that question), while the final recommendations state 'Verification and validation of security was rare, done by only 4% of the respondents.' These cannot both be correct. Please reconcile the discrepancy and ensure Table 9 matches the detailed results.
minor comments (5)
  1. [Section 1] The text contains a typo: 'concpets' should be 'concepts'.
  2. [Section 2, reference [32]] The description of Taylor et al. uses 'reproducability' and should be 'reproducibility'.
  3. [Figure 23] The label 'Austonomous response and adaptation of mission' should read 'Autonomous response and adaptation of mission'.
  4. [Section 5] The sentence 'Humans are know to have evaluation apprehension' should read 'Humans are known to have evaluation apprehension'.
  5. [Figures 7–16] Several bar charts report only percentages without raw counts; adding the underlying counts would improve interpretability, especially given the small denominators discussed in the major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's findings are direct aggregations of survey responses, with no fitted parameters, equations, or self-citation chain that makes a prediction equivalent to its inputs.

full rationale

This is an empirical survey paper. The reported findings are descriptive statistics computed directly from the 110 respondents' answers (e.g., 50%–67% full autonomy per task, 12% ML usage, 24% certification). There are no mathematical derivations, fitted parameters, or predictive models whose output is constrained by construction to equal an input. The authors cite their own prior work [24] and [34], but those citations are used only for comparison of domain mix (33% vs. 9% automotive) and as an example of bug studies; neither citation is load-bearing for the central claims of this paper. The 'state-of-the-practice' framing is a generalization claim, not a circular derivation, and the paper itself explicitly limits it in Section 5: 'we cannot claim that the results based on one survey would be valid for all autonomous systems.' Any concern about convenience sampling or representativeness is an external validity threat, not circularity. Therefore the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central findings are aggregates of self-reported survey responses. They rest on assumptions about sampling representativeness and response accuracy, and on arbitrary analysis thresholds (e.g., 30% reuse cutoff). No new entities or fitted mathematical parameters are introduced.

free parameters (2)
  • Reuse extent threshold = 30%
    In Section 4.3.1, reuse extent was categorized as '30% or more' versus 'less than 30%'. This arbitrary cutoff determines reported reuse rates, e.g., 63% of AUC projects reused 30% or more of code.
  • Number of autonomy levels = 4
    Section 4.2.2 uses four levels of autonomy ('human primary', 'computer with human interaction', 'computer independent with limited human', 'full autonomy') instead of the SAE 6-level scale. This choice shapes the reported autonomy distribution.
assumptions (3)
  • domain assumption Self-selection and recall biases cancel out
    Section 5 states 'we believe that such biases canceled themselves out'; required for the aggregated percentages to reflect actual practice.
  • domain assumption Sample representativeness
    The claim of establishing 'state-of-the-practice' assumes the 110 respondents, recruited via convenience sampling and snowballing, represent the broader population. Section 5 admits this cannot be assured.
  • domain assumption Consistent interpretation of terms
    The survey supplied definitions of autonomy, MBSwE, reuse, and criticality, but the analysis assumes all respondents interpreted these terms consistently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Practices of Autonomous Systems Development: Survey-based Empirical Findings." pith.science (2026). https://pith.science/paper/UCY2YGUM

@misc{pith2026250604438,
  author       = {Pith},
  title        = {Pith review of: On the Practices of Autonomous Systems Development: Survey-based Empirical Findings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UCY2YGUM}},
  note         = {Machine review of arXiv:2506.04438}
}
read the original abstract

Autonomous systems have gained an important role in many industry domains and are beginning to change everyday life. However, due to dynamically emerging applications and often proprietary constraints, there is a lack of information about the practice of developing autonomous systems. This paper presents the first part of the longitudinal study focused on establishing state-of-the-practice, identifying and quantifying the challenges and benefits, identifying the processes and standards used, and exploring verification and validation (V&V) practices used for the development of autonomous systems. The results presented in this paper are based on data about software systems that have autonomous functionality and may employ model-based software engineering (MBSwE) and reuse. These data were collected using an anonymous online survey that was administered in 2019 and were provided by experts with experience in development of autonomous systems and /or the use of MBSwE. Our current work is focused on repeating the survey to collect more recent data and discover how the development of autonomous systems has evolved over time.

Figures

Figures reproduced from arXiv: 2506.04438 by the authors.

Figure 1
Figure 1. Flowchart of our survey indicated a high level of safety criticality. These observations are consistent with the results from our previous survey [24], which used different values for the levels of criticality but had similar percentage of high criticality of 46%. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Areas of industry to which the respondents to our survey belong. (99 respondents, multiple answers possible) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Safety criticality of applications (97 respondents) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: The use of different programming languages (97 respondents, multiple selections possible) [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: How was the code for autonomous functionality developed (69 respondents) [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Respondents’ roles in the project (34 respondents, multiple selections possible) [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Levels of autonomy for specific tasks (50 AUCs) [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Algorithms and modeling paradigms used for development of AUCs (58 AUCs, multiple selections possible) [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Specification of requirements for AUCs (51 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 11
Figure 11. Figure 11: Degree of difficulty for challenges encountered during development, deployment and use of AUCs (50 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: The extent of reuse of different software artifacts for AUCs (30 respondents) [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: The extent of reuse of different software artifacts for non-AUC (30 respondents) [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Negative aspects of reuse (30 respondents) [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Difficulties due to reuse (29 respondents) [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Benefits of reuse with respect to productivity, quality, and cost (30 respondents) [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: Life-cycle models used by projects (39 respondents) [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Modeling standards used by projects (40 respondents) None 26% Other 24% NASA coding standard 19% JPL coding standard 18% Motor Industry Software Reliability Association (MISRA) 13% [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 20
Figure 20. Figure 20: Methods used for verification and validation of the models (37 respondents, multiple selections possible) [PITH_FULL_IMAGE:figures/full_fig_p018_20.png]
Figure 21
Figure 21. Figure 21: Methods used for verification and validation of AUCs during development (34 respondents, multiple selec [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]
Figure 22
Figure 22. Figure 22: Methods used for monitoring/assurance of AUCs runtime behavior (34 respondents, multiple selections [PITH_FULL_IMAGE:figures/full_fig_p019_22.png]
Figure 23
Figure 23. Figure 23: Methods used for handling the violations and errors of AUC (34 respondents, multiple selections possible) [PITH_FULL_IMAGE:figures/full_fig_p019_23.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 43 canonical work pages

  1. [24]

    Sur- vey on Model-Based Software Engineering and Auto-Generated Code,

    K. Goseva-Popstojanova, T. Kahsai, M. Knudson, T. Kyanko, N. Nkwocha, and J. Schumann, “Sur- vey on Model-Based Software Engineering and Auto-Generated Code,” https://tkyanko.me/cv/Survey on Model-Based Software Engineering and Auto-Generated Code.pdf, NASA Report, Tech. Rep., 2016, accessed: 2025-04-16

    work page 2016

  2. [1]

    Computing systems for autonomous driving: State of the art and challenges,

    L. Liu, S. Lu, R. Zhong, B. Wu, Y . Yao, Q. Zhang, and W. Shi, “Computing systems for autonomous driving: State of the art and challenges,” IEEE Internet of Things Journal, vol. 8, no. 8, pp. 6469–6486, 2020

    work page 2020

  3. [2]

    Creating a foundation for next-generation autonomous systems,

    D. Harel, A. Marron, and J. Sifakis, “Creating a foundation for next-generation autonomous systems,” IEEE Design & Test, vol. 39, no. 1, pp. 49–56, 2021. 23 On the Practices of Autonomous Systems Development: Survey-based Empirical Findings A PREPRINT

    work page 2021

  4. [3]

    Validation of autonomous systems,

    C. Ebert and M. Weyrich, “Validation of autonomous systems,” IEEE Software, vol. 36, no. 5, pp. 15–23, 2019

    work page 2019

  5. [4]

    2015 NASA technology roadmaps,

    L. Hall, “2015 NASA technology roadmaps,” 2015

    work page 2015

  6. [5]

    A review of ver- ification and validation for space autonomous systems,

    R. C. Cardoso, G. Kourtis, L. A. Dennis, C. Dixon, M. Farrell, M. Fisher, and M. Webster, “A review of ver- ification and validation for space autonomous systems,” Current Robotics Reports, vol. 2, no. 3, pp. 273–283, 2021

    work page 2021

  7. [6]

    Tesla recalls 362,758 vehicles, says full self-driving beta software may cause crashes,

    “Tesla recalls 362,758 vehicles, says full self-driving beta software may cause crashes,” https://www.cnbc.com/ 2023/02/16/tesla-recalls-362758-vehicles-says-full-self-driving-beta-software-may-cause-crashes.html, Febru- ary 2023, accessed: 2025-04-16

    work page 2023

  8. [7]

    A model for types and levels of human interaction with automation,

    R. Parasuraman, T. B. Sheridan, and C. D. Wickens, “A model for types and levels of human interaction with automation,” IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans, vol. 30, no. 3, pp. 286–297, 2000

    work page 2000

Show all 45 references
  1. [8]

    Challenges of testing highly automated vehicles: A liter- ature review,

    D. Karunakaran, J. S. Berrio, S. Worrall, and E. Nebot, “Challenges of testing highly automated vehicles: A liter- ature review,” in 2022 IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE). IEEE, 2022, pp. 1–8

    2022

  2. [9]

    DO-178C/ED-12C: Software considerations in airborne systems and equipment certification,

    RTCA, “DO-178C/ED-12C: Software considerations in airborne systems and equipment certification,” 2012. [Online]. Available: http://www.rtca.org

    2012

  3. [10]

    DO-331, Model-Based Development and Verification Supplement to DO-178C and DO-278A, RTCA Std., 2013

    2013

  4. [11]

    ISO 26 262-6, 2018

    Road vehicles — Functional safety — Part 6: Product development at the software level, International Organiza- tion for Standardization Std. ISO 26 262-6, 2018

    2018

  5. [12]

    EASA concept paper: First usable guidance for level 1 machine learning applications,

    “EASA concept paper: First usable guidance for level 1 machine learning applications,” European Aviation Safety Agency, Tech. Rep., 2021

    2021

  6. [13]

    EASA concept paper: First usable guidance for level 1&2 machine learning applications,

    “EASA concept paper: First usable guidance for level 1&2 machine learning applications,” European Aviation Safety Agency, Tech. Rep., 2023

    2023

  7. [14]

    IEEE Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems, IEEE Standard 7009, 2024, published by the IEEE

    2024

  8. [15]

    A survey of autonomous control for UA V,

    H. Chen, X. Wang, and Y . Li, “A survey of autonomous control for UA V,” in2009 International Conference on Artificial Intelligence and Computational Intelligence, vol. 2, Nov 2009, pp. 267–271

    2009

  9. [16]

    Coverage based testing for V&V and safety assurance of self-driving autonomous ve- hicles: A systematic literature review,

    Z. Tahir and R. Alexander, “Coverage based testing for V&V and safety assurance of self-driving autonomous ve- hicles: A systematic literature review,” in2020 IEEE International Conference On Artificial Intelligence Testing (AITest). IEEE, 2020, pp. 23–30

    2020

  10. [17]

    Autonomous driving security: State of the art and challenges,

    C. Gao, G. Wang, W. Shi, Z. Wang, and Y . Chen, “Autonomous driving security: State of the art and challenges,” IEEE Internet of Things Journal, vol. 9, no. 10, pp. 7572–7595, 2021

    2021

  11. [18]

    Application of neural networks in high assurance systems: A survey,

    J. Schumann, P. Gupta, and Y . Liu, “Application of neural networks in high assurance systems: A survey,” in Applications of Neural Networks in High Assurance Systems. Springer, 2010

    2010

  12. [19]

    Verification and validation of system health management models using parametric testing,

    J. Schumann, E. Reed, and O. Mengshoel, “Verification and validation of system health management models using parametric testing,” in AIAA Infotech@Aerospace, 03 2011

    2011

  13. [20]

    Software health management with Bayesian networks,

    J. Schumann, T. Mbaya, O. Mengshoel, K. Pipatsrisawat, A. Srivastava, A. Choi, and A. Darwiche, “Software health management with Bayesian networks,” Innovations in Systems and Software Engineering , vol. 9, no. 4, pp. 271–292, 2013

    2013

  14. [21]

    Assurance of model-based fault diagnosis,

    A. Nikora, P. Srivastava, L. Fesq, S. Chung, and K. Kolcio, “Assurance of model-based fault diagnosis,” in2018 IEEE Aerospace Conference, March 2018, pp. 1–14

    2018

  15. [22]

    Concepts in testing of autonomous systems: Academic literature and industry practice,

    Q. Song, E. Engstr ¨om, and P. Runeson, “Concepts in testing of autonomous systems: Academic literature and industry practice,” in 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN) . IEEE, 2021, pp. 74–81

    2021

  16. [23]

    Security modeling of autonomous systems: A survey,

    F. Jahan, W. Sun, Q. Niyaz, and M. Alam, “Security modeling of autonomous systems: A survey,” ACM Com- puting Surveys (CSUR), vol. 52, no. 5, pp. 1–34, 2019

    2019

  17. [25]

    Intelligent control for spacecraft autonomy - an industry survey,

    D. LaVallee, J. Jacobsohn, C. Olsen, and J. Reilly, “Intelligent control for spacecraft autonomy - an industry survey,” inSpace 2006, 2006, p. 7384. 24 On the Practices of Autonomous Systems Development: Survey-based Empirical Findings A PREPRINT

    2006

  18. [26]

    T. B. Sheridan, Telerobotics, Automation, and Human Supervisory Control. Cambridge, MA, USA: MIT Press, 1992

    1992

  19. [27]

    Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems,

    SAE On-Road Automated Vehicle Standards Committee and others, “Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems,” SAE Standard J, vol. 3016, no. 1, pp. 1–16, 2014

    2014

  20. [28]

    Methods for determining the level of autonomy to design into a human spaceflight vehicle: A function specific approach,

    R. W. Proud, J. J. Hart, and R. B. Mrozinski, “Methods for determining the level of autonomy to design into a human spaceflight vehicle: A function specific approach,” in Proc. Performance Metrics for Intelligent Systems (PerMIS ’03), NIST Special Publication 1014, 2003

    2003

  21. [29]

    Mercedes says level 4 autonomous driving will be a reality this decade,

    “Mercedes says level 4 autonomous driving will be a reality this decade,” https://cars.usnews.com/cars-trucks/ features/mercedes-level-4-autonomy-is-a-reality, March 2023, accessed: 2025-04-16

    2023

  22. [30]

    A comprehensive study of autonomous vehicle bugs,

    J. Garcia, Y . Feng, J. Shen, S. Almanee, Y . Xia, and A. Qi, “A comprehensive study of autonomous vehicle bugs,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering, 2020, pp. 385–396

    2020

  23. [31]

    Issue categorization and analysis of an open-source driving assis- tant system,

    S. Tang, Z. Zhang, J. Tang, L. Ma, and Y . Xue, “Issue categorization and analysis of an open-source driving assis- tant system,” in 2021 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 2021, pp. 148–153

    2021

  24. [32]

    A study on software bugs in unmanned aircraft systems,

    M. Taylor, J. Boubin, H. Chen, C. Stewart, and F. Qin, “A study on software bugs in unmanned aircraft systems,” in 2021 International Conference on Unmanned Aircraft Systems (ICUAS). IEEE, 2021, pp. 1439–1448

    2021

  25. [33]

    An exploratory study of autopilot software bugs in unmanned aerial vehicles,

    D. Wang, S. Li, G. Xiao, Y . Liu, and Y . Sui, “An exploratory study of autopilot software bugs in unmanned aerial vehicles,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021...

    2021

  26. [34]

    The anatomy of software changes and bugs in autonomous operating system,

    K. Goseva-Popstojanova, D. Hood, J. Schumann, and N. Nkwocha, “The anatomy of software changes and bugs in autonomous operating system,” in47th IEEE Annual Computers, Software, and Applications Conference (COMPSAC), 2023, pp. 28–38

    2023

  27. [35]

    Mind the gaps: Assuring the safety of autonomous systems from an engineering, ethical, and legal perspective,

    S. Burton, I. Habli, T. Lawton, J. McDermid, P. Morgan, and Z. Porter, “Mind the gaps: Assuring the safety of autonomous systems from an engineering, ethical, and legal perspective,” Artificial Intelligence, vol. 279, p. 103201, 2020

    2020

  28. [36]

    Ethics and the safety of autonomous systems,

    C. Menon and R. Alexander, “Ethics and the safety of autonomous systems,” 02 2018

    2018

  29. [37]

    Waymo’s safety methodologies and safety readiness determinations,

    N. Webb, D. Smith, C. Ludwick, T. Victor, Q. Hommes, F. Favaro, G. Ivanov, and T. Daniel, “Waymo’s safety methodologies and safety readiness determinations,” 2020. [Online]. Available: https://arxiv.org/abs/2011.00054

    2020 arXiv

  30. [38]

    Personal opinion surveys,

    B. A. Kitchenham and S. L. Pfleeger, “Personal opinion surveys,” in Guide to Advanced Empirical Software Engineering, F. Shull, J. Singer, and D. Sjøberg, Eds. Springer London, 2008, pp. 63–92

    2008

  31. [39]

    SurveyMonkey,

    “SurveyMonkey,” https://www.surveymonkey.com/, accessed: 2025-04-16

    2025

  32. [40]

    Papyrus modeling environment,

    Eclipse.org, “Papyrus modeling environment,” https://eclipse.org/papyrus/, accessed: 2025-04-16

    2025

  33. [41]

    The MathWorks Automotive Advisory Board

    MathWorks, “The MathWorks Automotive Advisory Board.” [Online]. Available: http://www.mathworks.com/ solutions/automotive/standards/maab.html

  34. [42]

    SysML open source specification project,

    SysML.org, “SysML open source specification project,” http://sysml.org, accessed: 2025-04-16

    2025

  35. [43]

    “UPPAAL,” http://www.uppaal.org, accessed: 2025-04-16

    2025

  36. [44]

    Jenkins,

    jenkins.io, “Jenkins,” https://jenkins.io/, accessed: 2025-04-16

    2025

  37. [45]

    PRISM 4.0: Verification of Probabilistic Real-time Systems,

    M. Kwiatkowska, G. Norman, and D. Parker, “PRISM 4.0: Verification of Probabilistic Real-time Systems,” in Proceedings of the 23rd International Conference on Computer Aided Verification (CAV’11) , ser. LNCS, G. Gopalakrishnan and S. Qadeer, Eds., vol. 6806. Springer, 2011, pp...

    2011

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.