Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components

Cumhur Erkan Tuncali; Danil Prokhorov; Georgios Fainekos; Hisahiro Ito; James Kapinski

REVIEW 4 major objections 3 minor 72 references

Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components

T0 review · 4 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Writing autonomous-vehicle requirements in signal temporal logic and searching scenario parameters automatically uncovers failure cases, including sensor faults that lead to collisions, in a closed-loop simulator with a…

desk verdict Solid, honest tool paper for falsifying STL requirements in closed-loop AV simulation; the 'critical corner cases' language outruns the simulation-only evidence. read the letter →

arxiv 1908.01094 v1 pith:KHOSHQZJ submitted 2019-08-02 cs.RO cs.LGcs.SE

Cumhur Erkan Tuncali , Georgios Fainekos , Danil Prokhorov , Hisahiro Ito , James Kapinski This is my paper · ORCID

classification cs.ROcs.LGcs.SE

keywords signaltemporallogicautonomousvehiclestestgenerationfalsificationmachinelearningdeepneuralnetworkscoveringarrayssimulation-basedtesting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Building on an earlier version of the work, the paper extends a closed-loop simulation framework that tests autonomous-vehicle controllers by expressing safety and performance requirements in signal temporal logic (STL) and using robustness values as a cost function to search for failing scenarios. The search first explores combinations of discrete parameters with covering arrays and then refines the most promising cases with stochastic optimization, all while a deep-neural-network perception system processes synthetic camera images alongside lidar and radar models. The authors present five STL requirements and demonstrate the approach on three driving scenarios, automatically finding cases of collision, of visible agents going undetected, of sensor faults followed by collisions, and of unnecessary repeated hard braking. If the method works as described, designers receive replayable, parameter-traceable failure cases that can guide controller debugging and corner-case testing.

What carries the argument

The mechanism that carries the argument is the robustness semantics of signal temporal logic, which assigns each simulation trace a real number--negative when the trace violates a requirement and positive when it satisfies it--so that falsification becomes a numerical optimization problem over the scenario parameter space. Two search strategies are combined: covering arrays, which guarantee that every $t$-way combination of discrete parameters appears in the test suite, and simulated annealing, which refines the most promising covering-array cases by varying continuous parameters such as speeds and positions. The requirements themselves do the targeting: each of the five STL formulas focuses the search on one kind of critical behavior, so the framework can be asked for a collision, a missed detection, a sensor-to-system fault chain, or a ride-comfort violation rather than merely 'something wrong'.

What would settle it

Re-run the three scenarios in a physically validated high-fidelity simulator or on a test track using the same agent trajectories, and check whether the reported failures--the missed pedestrian detections, the sensor-fault-then-collision sequence, and the triple brake-release event--still occur; if they disappear, the discovered cases are simulator artifacts rather than genuine system corner cases.

Watch

Extended reading notes

Core claim

The central claim is that the same formal requirements used to evaluate a test case can also be used to discover failing test cases. Each STL formula is interpreted with robust semantics, giving every simulation trace a real number whose sign indicates satisfaction or falsification and whose magnitude measures the trace's distance to the requirement boundary; minimizing that robustness value over scenario parameters reformulates testing as adversarial search. Because the perception system contains a deep neural network, the search varies physical and environmental parameters such as agent trajectories, colors, and speeds instead of pixel-level images, so the findings target closed-loop behavior rather than isolated classifier errors. The paper proposes five requirements--no collision, timely detection of visible agents, bounded localization error, no sensor fault followed shortly by a collision, and no excessive or frequently repeated braking--and shows for three scenarios that the covering-array and optimization search returns falsifying traces that can be replayed and traced back to parameter values. These traces are presented as automatically generated evidence of specific critical behaviors to support, not replace, human debugging of the controller design.

Load-bearing premise

The load-bearing premise is that the simplified vehicle, sensor, and neural-perception models inside the simulator are faithful enough that failures found there correspond to real critical behaviors; the paper explicitly notes the simplified algorithms were chosen to evaluate methodology, not a production control design.

Editorial extensions

If this is right

Falsifying traces can be replayed and traced back to the parameter combination that produced them, giving designers concrete, reproducible failure cases rather than a pass/fail summary.
The sensor-to-system requirement makes it possible to automatically isolate collisions that are temporally correlated with a perception fault, which the authors demonstrate on the intersection scenario.
Covering-array cases with small positive robustness mark near-violations, so they identify boundary behaviors worth extra scrutiny even when the requirement is technically satisfied.
Because the requirements are expressed in STL, the same five formulas can be checked against any scenario built in the simulator without rewriting the testing logic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors list retraining the perception network with discovered counterexamples as future work, but if that loop were closed, the robustness value would give a quantitative before-and-after comparison of the perception system across the same scenarios.
The requirement patterns are mostly sensor-agnostic, so the same formulas should transfer to other sensor suites by changing what counts as visible and detected; the paper does not test this transfer.
The biggest obstacle to treating the discovered scenarios as real defects is simulator fidelity: until the same scenarios are re-run with higher-fidelity physics, sensors, and rendering, the corner cases are best read as candidate hypotheses for physical testing.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Solid tool paper. The core demonstration—automatically falsifying STL requirements in closed-loop Webots simulation with a DNN perception component—works and is reported honestly. The genuinely new parts are the lidar/radar sensor models with fusion logic, the visibility-aware STL requirements, and the search over agent trajectories; the five requirements R1–R5 are clear and sensible, and the M3 trace analysis is a good example of explaining why a requirement fails.

The main soft spot is the gap between 'critical corner cases' and what is actually shown. The paper itself says in Sec. V that the vehicle control subsystems are simplified and the purpose is to evaluate testing methodology, not a real AV control design. So a falsifying trace is a corner case for the simulated system; calling it a critical behavior of an autonomous vehicle goes beyond the evidence unless the simulator is validated against reality or a higher-fidelity model. That is not necessarily a flaw in a methodology paper, but the language should be scoped.

The experiments also have no baselines—no random search, no alternative falsification tool—and single runs for M2/M3, so the falsification counts are anecdotal rather than statistically grounded. There is an internal mismatch: the M2 section refers to Fig. 14 as a falsifying example for R2, the figure caption says M1, and Table I lists M2 under R4. That's likely a copy-paste error but needs fixing before publication. Reproducibility is partial: the toolbox is public, but the exact scenario files and model weights aren't, so the reported numbers cannot be reproduced from the artifact alone.

The methodology is sound and the contribution is real. I would send this to peer review. A referee should ask for the label fix, baseline comparisons, repeated trials with variance, and a more careful scoping of what a 'critical corner case' means outside the simulator. But the central capability is demonstrated, and the paper is a useful addition to the AV testing literature.

Referee Report

4 major / 3 minor

Summary. The paper presents Sim-ATAV, a simulation-based adversarial testing framework for autonomous vehicles with ML perception components. It formalizes five STL requirements (R1-R5) at component and system level, generates test scenarios via t-way covering arrays over discrete/discretized parameters, then uses simulated-annealing-based robustness falsification over continuous parameters. The evaluation uses three Webots scenarios (M1, M2, M3) with a SqueezeDet DNN and simplified lidar/radar/camera models; the paper reports falsification counts for R1, R2, R4, and R5, including a localization-error-then-collision trace and an unnecessary-braking trace, and claims these are 'critical corner cases' useful for development.

Significance. If the claims are supported, the paper would be a useful methodology contribution: it demonstrates closed-loop falsification of temporal-logic requirements with a DNN perception component in the loop, on a publicly released tool (Sim-ATAV add-on to S-TaLiRo), and it provides explicit STL formulas for system-level, sensor-level, and subsystem-to-system requirements. The falsification counts in Table I are direct simulation outcomes rather than quantities derived from fitted constants, so there is no equation-level circularity in the reported experiments. The main scientific value is the demonstration that combinatorial coverage of discrete scenario parameters can seed requirement falsification in a closed-loop AV simulator; however, the broader claim that the discovered traces are 'critical corner cases' of an AV depends on simulation fidelity, which is not established.

major comments (4)

[Abstract; Sec. V; Sec. VI/Table I] The abstract and Sec. VI claim that the framework identifies 'critical corner cases' and 'critical system behaviors,' but Sec. V explicitly states: 'we use simplified algorithms to implement the subsystems of the vehicle control, which is sufficient in this case, as the purpose of this investigation is to evaluate new testing methodologies and not to evaluate a real AV control design.' No fidelity validation, high-fidelity comparison, or sensitivity analysis is given. A falsifying trace such as the M2 localization-error-then-collision case can be an artifact of the simplified sensor models, the synthetic-to-real domain gap of the DNN, or the simplified dynamics. This makes the 'critical corner cases' claim load-bearing and unsupported. Please either restate the claim as 'falsifying behaviors inside the simulation model' or add evidence that the falsifying scenarios persist under higher-fidelity models or module replacement.
[Table I and Sec. VI] The experimental evaluation is baseline-free and appears to consist of a single run per scenario. Table I reports 195 covering-array cases plus 300 simulated-annealing iterations for M1, 58 iterations for M2, and 232 iterations for M3, but there is no comparison with uniform random test generation at the same budget and no repeated runs with different random seeds. Since both the ACTS covering-array construction and simulated annealing are stochastic, the reported falsification counts (e.g., 12 additional R4 falsifications from 300 M1 iterations) cannot be separated from randomness. The claim that the combined approach is an 'effective way' of finding corner cases requires at least a random-sampling baseline and seed-level statistics.
[Sec. IV, R2-R5; Sec. VI] The instantiated experimental requirements are under-specified. The formulas R2, R3, and R4 contain timing constants t1 and t2 and error thresholds ϵerr and ϵdist, but the scenario descriptions in Sec. VI do not give their values; only the R5 analysis gives t1 = 0.6 and t2 = 0.5. Without these constants (and the discretization step sizes for the covering array), the quantitative results in Table I cannot be reproduced or compared across scenarios. Please provide a table of all requirement thresholds and discretization parameters per scenario.
[Introduction, contribution bullet; Sec. IV, R4] One of the listed contributions says the framework automatically finds 'cases where a sensor failure leads to a collision case,' but R4 only formalizes a temporal correlation: a period of poor detection/visibility followed within a bounded interval by a collision. As Sec. IV itself notes, a violation of R4 'does not necessarily indicate that the sensor fault caused the system fault.' The causal phrasing in the introduction overstates what the experiments establish. Please either soften the language to 'cases where a sensor fault is temporally associated with a collision' or add a counterfactual analysis that actually supports a causal reading.

minor comments (3)

[Sec. VI, Fig. 14] The text states that Fig. 14 shows a falsifying example for model M2 with respect to requirement R2, but Table I indicates M2 was evaluated against R4 and Fig. 14's caption says 'model M1.' Please correct this inconsistency.
[Sec. IV, R5] The definition B↓ = B ∧ ©¬B makes the predicate true at the sample before the release of the brake, not at the release instant itself. The prose 'event of releasing the brake' should be clarified to avoid ambiguity about when B↓ is evaluated.
[Sec. V] The descriptions of the sensor-fusion logic, planner, and low-level controller are high-level ('simple logic', 'simple path and speed tracking'). Since the release URL is given, please also point to a specific version or commit and list the main parameters used in the experiments to aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the falsification results are direct simulation outcomes with respect to the authors' own STL requirements, and the cited prior work provides tooling and standard robust semantics rather than a forced conclusion.

full rationale

The paper's central claim is that the Sim-ATAV framework evaluates closed-loop simulation traces against STL requirements and automatically finds traces that falsify them. That is exactly the implemented procedure: Section III-C defines falsification as minimizing the robustness value (Eq. 1), Section V describes the S-TaLiRo/Sim-ATAV simulation and optimization loop, and Section VI reports concrete falsifying traces with negative robustness for requirements R1, R2, R4, and R5. The STL requirements in Section IV are inputs of the analysis, not outputs: no parameter is fitted to the falsification results, and no reported counterexample is reconstructed from the same data used to define success. Citations to S-TaLiRo [24], STL robust semantics [49], and the earlier Sim-ATAV paper [23] are provenance for the tool and the mathematical semantics; they do not supply a result that the experiments merely rename. The explicit limitation in Section V that 'we use simplified algorithms to implement the subsystems of the vehicle control' is a fidelity concern, not a circularity concern: it weakens the external validity of the term 'critical corner cases' but does not make the in-simulation falsification outcomes equivalent to their inputs. No equation-level reduction, fitted-input-as-prediction, or load-bearing self-citation chain can be exhibited from the paper's own text, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper contributes an empirical tool demonstration, not a formal derivation. Its central claim rests on standard STL semantics and on the unvalidated fidelity of the Webots sensor models and the synthetic-image DNN. The reported results also depend on hand-chosen requirement thresholds, scenario parameter ranges, and search budgets. No new physical or formal entities are introduced.

free parameters (3)

Requirement thresholds in R1-R5 = Hand-chosen; e.g., in Scenario 3, t1=0.6s, t2=0.5s, FC threshold 0.5m, braking threshold 0.5; epsilon_err and…
These constants define the specific requirements tested in Sec. VI. The reported falsification counts and criticality assessments apply only to these instantiations, and changing the thresholds would change the results.
Covering array strength selection = Mixed strength: 3-way covering for four intuitively critical parameters, 2-way for the rest
This design choice in Sec. VI sets the number of discrete test cases (195) and therefore affects how many falsifications are found in the covering-array stage.
Search budget for falsification = Maximum 300 additional simulations per requirement
The reported number of additional falsifications is capped by this budget. A different budget would change the counts and the apparent effectiveness of the method.

assumptions (5)

standard math STL robust semantics (Fainekos-Pappas) provide a sound quantitative measure of satisfaction that can be used as a cost function for falsification.
Invoked throughout Sec. III-B and used in all experiments. This is accepted background theory and is not a contribution of the paper.
domain assumption The Webots physics, CCD camera, lidar, radar, and vehicle dynamics models faithfully represent the closed-loop behavior of a real autonomous vehicle and its environment.
Sec. V states simplified algorithms are used because the purpose is to evaluate testing methodology, and no calibration or real-world validation is presented. The transfer of simulated failures to real critical corner cases depends on this assumption.
domain assumption SqueezeDet, trained on KITTI and further trained on virtual images, is a representative DNN perception component whose errors in simulation resemble real perception errors.
Sec. V describes the DNN and training but reports no accuracy statistics on synthetic images. The camera-related falsification results in Scenario 1 depend on the network's behavior in the virtual domain.
ad hoc to paper t-way covering arrays plus simulated annealing is an effective search strategy for finding DNN-sensitive parameter combinations in closed-loop cyber-physical systems.
Sec. III-D argues this by example and cites the earlier paper [23], but no formal guarantee or comparative study supports the claimed effectiveness beyond the chosen scenarios.
domain assumption The simulator is deterministic (Remark 1), so single-run falsification counts are meaningful as point estimates.
Sec. III Remark 1 assumes a deterministic sim. The optimizer is stochastic, and the paper does not report repeated runs or variance, so the point estimates rest on this assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components." pith.science (2026). https://pith.science/paper/KHOSHQZJ

@misc{pith2026190801094,
  author       = {Pith},
  title        = {Pith review of: Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KHOSHQZJ}},
  note         = {Machine review of arXiv:1908.01094}
}

read the original abstract

Autonomous vehicles are complex systems that are challenging to test and debug. A requirements-driven approach to the development process can decrease the resources required to design and test these systems, while simultaneously increasing the reliability. We present a testing framework that uses signal temporal logic (STL), which is a precise and unambiguous requirements language. Our framework evaluates test cases against the STL formulae and additionally uses the requirements to automatically identify test cases that fail to satisfy the requirements. One of the key features of our tool is the support for machine learning (ML) components in the system design, such as deep neural networks. The framework allows evaluation of the control algorithms, including the ML components, and it also includes models of CCD camera, lidar, and radar sensors, as well as the vehicle environment. We use multiple methods to generate test cases, including covering arrays, which is an efficient method to search discrete variable spaces. The resulting test cases can be used to debug the controller design by identifying controller behaviors that do not satisfy requirements. The test cases can also enhance the testing phase of development by identifying critical corner cases that correspond to the limits of the system's allowed behaviors. We present STL requirements for an autonomous vehicle system, which capture both component-level and system-level behaviors. Additionally, we present three driving scenarios and demonstrate how our requirements-driven testing framework can be used to identify critical system behaviors, which can be used to support the development process.

Figures

Figures reproduced from arXiv: 1908.01094 by the authors.

**Figure 1.** The simple test scenario of Example 1. stamp ti ∈ R≥0. Here, mout is the number of observable output variables. We denote the set of possible output traces by Y = (Y × R≥0) Nout . The output trace timestamps should satisfy (i) tNout = T, and (ii) the monotonicity property. For notational convenience, we will make an additional assumption that the simulator also returns an updated input trace u where N = Nin = Nout a… view at source ↗

**Figure 3.** Output trace y for the test scenario of Example 1. real-time properties of signals (simulation traces) (for an overview see [48]). STL formulae are built over predicates on the variables of a signal using Boolean and temporal operators. The temporal operators include eventually (♦I), always (I) and until (UI), where I is a time interval that encodes timing constraints. The boolean operators include conjunction ∧, di… view at source ↗

**Figure 4.** The resulting robustness landscape (heatmap) for [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

Figures from the paper (11 more)

**Figure 5.** Figure 5: Specific configurations impacting DNN performance. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Robotic pedestrian surrogate target with a Toyota [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Overview of the simulation environment. The simulation environment used in Sim-ATAV is based on the open source simulator Webots [64] and includes a vehicle perception system, a vehicle controller, and a model of the physical environment. The perception system processe…

**Figure 8.** Figure 8: Outputs from the SqueezeDet DNN, based on a [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

**Figure 10.** Figure 10: Flowcharts illustrating the combinatorial testing (a) [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

**Figure 9.** Figure 9: Sensor fusion outputs. them. Expected future positions of agent vehicles with respect to the ego vehicle are represented by red circles. Our simple planner receives as inputs the high level target path and target speed, and the outputs of the sensor fusion and trajecto…

**Figure 11.** Figure 11: Overview of the scenario 1. • R, G, B values for the colors of Agent car: [0, 1]; • R, G, B values for the pedestrian’s shirt and pants: [0, 1]. We chose the parameterized aspects of the scenarios such that their specific combinations would be challenging to a DNN-bas…

**Figure 12.** Figure 12: Overview of the scenario 2. • Agent 2 initial distance to the intersection: [50, 100]m. We evaluate Model M2 against requirement R4. The idea in using the sensor-to-system-level requirement is that it is relatively easy, in general, to find behaviors that result in a …

**Figure 13.** Figure 13: Robustness guided falsification utilizes global opti [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]

**Figure 15.** Figure 15: Analysis of falsification for Model M3. more realistic by using other scene rendering tools, such as those based on state-of-the-art game engines, e.g., [71]. Also, we note that the formal requirements that we considered were provided as an example of the type used w…

**Figure 14.** Figure 14: Time-ordered images from the falsifying example on [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 64 canonical work pages

[1]

NHTSA Federal Automated Vehicles Pol- icy,

“NHTSA Federal Automated Vehicles Pol- icy,” https://www.transportation.gov/A V/ federal-automated-vehicles-policy-september-2016, accessed: 2018-09- 11

work page 2016
[2]

Guidelines for safe on-road testing of sae level 3, 4, and 5 prototype automated driving systems (ADS), document j3018 201503,

SAE, “Guidelines for safe on-road testing of sae level 3, 4, and 5 prototype automated driving systems (ADS), document j3018 201503,” https://www.sae.org/standards/content/j3018 201503/, accessed: 2018- 09-12

work page 2018
[3]

Key considerations in the development of driving automation systems,

A. Christensen, A. Cunningham, J. Engelman, C. Green, C. Kawashima, S. Kiger, D. Prokhorov, L. Tellis, B. Wendling, and F. Barickman, “Key considerations in the development of driving automation systems,” in 24th Enhanced Safety of Vehicles Conferences , 2015

work page 2015
[4]

Euro NCAP,

“Euro NCAP,” https://www.euroncap.com/en, accessed: 2018-09-11

work page 2018
[5]

Pegasus,

“Pegasus,” https://www.pegasusprojekt.de/en/home, accessed: 2018-09- 11

work page 2018
[6]

Testing of advanced driver assistance towards automated driving: A survey and taxonomy on existing approaches and open questions,

J. E. Stellet, M. R. Zofka, J. Schumacher, T. Schamm, F. Niewels, and J. M. Z ¨ollner, “Testing of advanced driver assistance towards automated driving: A survey and taxonomy on existing approaches and open questions,” 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pp. 1455–1462, 2015

work page 2015
[7]

The sleepwalker framework: Veriﬁcation and validation of autonomous vehicles by mixed reality lidar stimulation,

M. R. Zofka, M. Essinger, T. Fleck, R. Kohlhaas, and J. M. Z ¨ollner, “The sleepwalker framework: Veriﬁcation and validation of autonomous vehicles by mixed reality lidar stimulation,” in SIMPAR. IEEE, 2018, pp. 151–157

work page 2018
[8]

Challenges in autonomous vehicle testing and validation,

P. Koopman and M. Wagner, “Challenges in autonomous vehicle testing and validation,” SAE Int. J. Trans. Safety , vol. 4, pp. 15–24, 04 2016. [Online]. Available: https://doi.org/10.4271/2016-01-0128

doi:10.4271/2016-01-0128 2016

Show all 72 references

[9]

The Mobile Revolution - Machine Intelligence for Autonomous Vehicles (Dagstuhl Seminar 15462),

W. Burgard, U. Franke, M. Enzweiler, and M. Trivedi, “The Mobile Revolution - Machine Intelligence for Autonomous Vehicles (Dagstuhl Seminar 15462),” Dagstuhl Reports , vol. 5, no. 11, pp. 62–70, 2016. [Online]. Available: http://drops.dagstuhl.de/opus/volltexte/2016/5764

2016
[10]

The worst-time- to-collision metric for situation identiﬁcation,

W. Wachenfeld, P. Junietz, R. Wenzel, and H. Winner, “The worst-time- to-collision metric for situation identiﬁcation,” in 2016 IEEE Intelligent Vehicles Symposium, IV 2016, Gotenburg, Sweden, June 19-22, 2016 , 2016, pp. 729–734

2016
[11]

Are we ready for autonomous driving? the kitti vision benchmark suite,

A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on . IEEE, 2012

2012
[12]

SqueezeDet: Uniﬁed, small, low power fully convolutional neural networks for real-time object detection for autonomous driving,

B. Wu, F. Iandola, P. H. Jin, and K. Keutzer, “SqueezeDet: Uniﬁed, small, low power fully convolutional neural networks for real-time object detection for autonomous driving,” in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , 2017, pp. 446– 454

2017
[13]

Alvinn: An autonomous land vehicle in a neural network,

D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in Advances in neural information processing systems , 1989, pp. 305–313

1989
[14]

DeepDriving: Learning affordance for direct perception in autonomous driving,

C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “DeepDriving: Learning affordance for direct perception in autonomous driving,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 2722–2730

2015
[15]

Deep steering: Learning end-to-end driving model from spatial and temporal visual cues,

L. Chi and Y . Mu, “Deep steering: Learning end-to-end driving model from spatial and temporal visual cues,” arXiv preprint arXiv: 1708.03798, 2017

2017 arXiv
[16]

Deep predictive models for collision risk assessment in autonomous driving,

M. Strickland, G. Fainekos, and H. B. Amor, “Deep predictive models for collision risk assessment in autonomous driving,” in IEEE Interna- tional Conference on Robotics and Automation (ICRA) , 2018

2018
[17]

Imagenet classiﬁcation with deep convolutional neural networks,

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classiﬁcation with deep convolutional neural networks,” in Advances in neural infor- mation processing systems , 2012, pp. 1097–1105

2012
[18]

Deeptest: Automated testing of deep-neural-network-driven autonomous cars,

Y . Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: Automated testing of deep-neural-network-driven autonomous cars,” in 40th International Conference on Software Engineering (ICSE) , 2018

2018
[19]

Practical black-box attacks against machine learning,

N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in ACM Asia Conference on Computer and Communications Security . ACM, 2017

2017
[20]

Feature-guided black- box safety testing of deep neural networks,

M. Wicker, X. Huang, and M. Kwiatkowska, “Feature-guided black- box safety testing of deep neural networks,” in International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), ser. LNCS, vol. 10805, 2018, pp. 408–426

2018
[21]

Software and hardware testing using combinatorial cov- ering suites,

A. Hartman, “Software and hardware testing using combinatorial cov- ering suites,” Graph theory, combinatorics and algorithms , vol. 34, pp. 237–266, 2005

2005
[22]

Probabilistic temporal logic falsiﬁcation of cyber-physical systems,

H. Abbas, G. E. Fainekos, S. Sankaranarayanan, F. Ivancic, and A. Gupta, “Probabilistic temporal logic falsiﬁcation of cyber-physical systems,” ACM Transactions on Embedded Computing Systems, vol. 12, no. s2, May 2013

2013
[23]

Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components,

C. E. Tuncali, G. Fainekos, H. Ito, and J. Kapinski, “Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components,” in IEEE Intelligent Vehicles Symposium (IV) , 2018

2018
[24]

S-TaLiRo: A tool for temporal logic falsiﬁcation for hybrid systems,

Y . S. R. Annapureddy, C. Liu, G. E. Fainekos, and S. Sankaranarayanan, “S-TaLiRo: A tool for temporal logic falsiﬁcation for hybrid systems,” in Tools and algorithms for the construction and analysis of systems , ser. LNCS, vol. 6605. Springer, 2011, pp. 254–257

2011
[25]

Description of light-vehicle pre-crash scenarios for safety applications based on vehicle-to-vehicle communications,

W. G. Najm, R. Ranganathan, G. Srinivasan, J. S. Toma, E. Swanson, and A. B. D. Smith, “Description of light-vehicle pre-crash scenarios for safety applications based on vehicle-to-vehicle communications,” DOT HS 811 731, Tech. Rep., 2013

2013
[26]

TrafﬁcNet: An open naturalistic driving scenario library,

D. Zhao, Y . Guo, and Y . J. Jia, “TrafﬁcNet: An open naturalistic driving scenario library,” in 20th IEEE International Conference on Intelligent Transportation Systems, ITSC , 2017

2017
[27]

Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques,

D. Zhao, H. Lam, H. Peng, S. Bao, D. J. LeBlanc, K. Nobukawa, and C. S. Pan, “Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques,” IEEE Transactions on Intelligent Transportation Systems , vol. 18, no. 3, pp. 59...

2017
[28]

Adaptive cruise control: Hybrid, distributed, and now formally veriﬁed,

S. M. Loos, A. Platzer, and L. Nistor, “Adaptive cruise control: Hybrid, distributed, and now formally veriﬁed,” in Formal Methods, ser. LNCS, vol. 6664. Springer, 2011, pp. 42–56

2011
[29]

Safety veriﬁcation of autonomous vehicles for coordinated evasive maneuvers,

M. Althoff, D. Althoff, D. Wollherr, and M. Buss, “Safety veriﬁcation of autonomous vehicles for coordinated evasive maneuvers,” in IEEE Intelligent Vehicles Symposium, 2010

2010
[30]

Utilizing S-TaLiRo as an automatic test generation framework for autonomous vehicles,

C. E. Tuncali, T. P. Pavlic, and G. Fainekos, “Utilizing S-TaLiRo as an automatic test generation framework for autonomous vehicles,” in IEEE Intelligent Transportation Systems Conference , 2016

2016
[31]

Rapidly-exploring random trees for testing automated vehicles,

C. E. Tuncali and G. Fainekos, “Rapidly-exploring random trees for testing automated vehicles,” in IEEE Intelligent Transportation Systems Conference (ITSC), 2019

2019
[32]

Automatic generation of safety-critical test scenarios for collision avoidance of road vehicles,

M. Althoff and S. Lutz, “Automatic generation of safety-critical test scenarios for collision avoidance of road vehicles,” in IEEE Intelligent Vehicles Symposium (IV), 2018

2018
[33]

Functional gradient descent optimization for automatic test case generation for vehicle controllers,

C. E. Tuncali, S. Yaghoubi, T. P. Pavlic, and G. Fainekos, “Functional gradient descent optimization for automatic test case generation for vehicle controllers,” in IEEE International Conference on Automation Science and Engineering , 2017

2017
[34]

Computer-aided design for safe autonomous vehicles,

M. O’Kelly, H. Abbas, and R. Mangharam, “Computer-aided design for safe autonomous vehicles,” in 2017 Resilience Week (RWS) , 2017

2017
[35]

Testing autonomous vehicle software in the virtual prototyping environment,

B. Kim, Y . Kashiba, S. Dai, and S. Shiraishi, “Testing autonomous vehicle software in the virtual prototyping environment,” Embedded Systems Letters, vol. 9, no. 1, pp. 5–8, 2017

2017
[36]

The SMT-based automatic road network generation in vehicle simulation environment,

B. Kim, A. Jarandikar, J. Shum, S. Shiraishi, and M. Yamaura, “The SMT-based automatic road network generation in vehicle simulation environment,” in International Conference on Embedded Software (EM- SOFT). ACM, 2016, pp. 18:1–18:10

2016
[37]

Data-driven formal reasoning and their applications in safety analysis of vehicle autonomy features,

C. Fan, B. Qi, and S. Mitra, “Data-driven formal reasoning and their applications in safety analysis of vehicle autonomy features,” IEEE Design Test, vol. 35, no. 3, pp. 31–38, June 2018

2018
[38]

APEX: Autonomous vehicle plan veriﬁcation and execution,

M. O’Kelly, H. Abbas, S. Gao, S. Shiraishi, S. Kato, and R. Mangharam, “APEX: Autonomous vehicle plan veriﬁcation and execution,” in SAE World Congress, 2016

2016
[39]

A committee of neural networks for trafﬁc sign classiﬁcation,

D. Cires ¸an, U. Meier, J. Masci, and J. Schmidhuber, “A committee of neural networks for trafﬁc sign classiﬁcation,” in Proceedings of the 2011 International Joint Conference on Neural Networks (IJCNN), 2011, pp. 1918–1921

2011
[40]

Trafﬁc light recognition in varying illumination using deep learning and saliency map,

V . John, K. Yoneda, B. Qi, Z. Liu, and S. Mita, “Trafﬁc light recognition in varying illumination using deep learning and saliency map,” in Pro- ceedings of the 2014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC) , 2014

2014
[41]

Pedestrian detection with a large-ﬁeld-of-view deep network,

A. Angelova, A. Krizhevsky, and V . Vanhoucke, “Pedestrian detection with a large-ﬁeld-of-view deep network,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA ’15), 2015

2015
[42]

Learning and veriﬁcation of feedback control systems using feedforward neural net- works,

S. Dutta, S. Jha, S. Sankaranarayanan, and A. Tiwari, “Learning and veriﬁcation of feedback control systems using feedforward neural net- works,” in Analysis and Design of Hybrid Systems , 2018

2018
[43]

Compositional falsication of cyber-physical systems with machine learning components,

T. Dreossi, A. Donze, and S. A. Seshia, “Compositional falsication of cyber-physical systems with machine learning components,” in NASA Formal Methods (NFM) , ser. LNCS, vol. 10227. Springer, 2017, pp. 357–372

2017
[44]

Semantic adversarial deep learning,

T. Dreossi, S. Jha, and S. A. Seshia, “Semantic adversarial deep learning,” vol. 10981, pp. 3–26, 2018

2018
[45]

Safe at any speed: A simulation-based test harness for autonomous vehicles,

H. Abbas, M. O’Kelly, A. Rodionova, and R. Mangharam, “Safe at any speed: A simulation-based test harness for autonomous vehicles,” in 7th International Workshop on Cyber-Physical Systems (CyPhy) , 2017

2017
[46]

Alur, Principles of Cyber-Physical Systems

R. Alur, Principles of Cyber-Physical Systems . MIT Press, 2015

2015
[47]

Robustness-guided temporal logic testing and veriﬁcation for stochastic cyber-physical systems,

H. Abbas, B. Hoxha, G. Fainekos, and K. Ueda, “Robustness-guided temporal logic testing and veriﬁcation for stochastic cyber-physical systems,” in IEEE International Conference on CYBER Technology in Automation, Control, and Intelligent Systems , 2014

2014
[48]

Speciﬁcation-based monitoring of cyber-physical systems: A survey on theory, tools and applications,

E. Bartocci, J. Deshmukh, A. Donz ´e, G. Fainekos, O. Maler, D. Nick- ovic, and S. Sankaranarayanan, “Speciﬁcation-based monitoring of cyber-physical systems: A survey on theory, tools and applications,” in Lectures on Runtime Veriﬁcation - Introductory and Advanced Topics , s...

2018
[49]

Robustness of temporal logic spec- iﬁcations for continuous-time signals,

G. E. Fainekos and G. J. Pappas, “Robustness of temporal logic spec- iﬁcations for continuous-time signals,” Theoretical Computer Science , vol. 410, no. 42, pp. 4262–4291, 2009

2009
[50]

Boyd and L

S. Boyd and L. Vandenberghe, Convex Optimization . Cambridge University Press, 2004

2004
[51]

What’s decidable about hybrid automata?

T. A. Henzinger, P. W. Kopke, A. Puri, and P. Varaiya, “What’s decidable about hybrid automata?” J. Comput. Syst. Sci., vol. 57, no. 1, pp. 94–124, 1998

1998
[52]

Towards formal speciﬁcation visualization for testing and monitoring of cyber-physical systems,

B. Hoxha, H. Bach, H. Abbas, A. Dokhanchi, Y . Kobayashi, and G. Fainekos, “Towards formal speciﬁcation visualization for testing and monitoring of cyber-physical systems,” in International Workshop on Design and Implementation of Formal Tools and Systems , 2014

2014
[53]

Simulation- based approaches for veriﬁcation of embedded control systems: An overview of traditional and advanced modeling, testing, and veriﬁcation techniques,

J. Kapinski, J. V . Deshmukh, X. Jin, H. Ito, and K. Butts, “Simulation- based approaches for veriﬁcation of embedded control systems: An overview of traditional and advanced modeling, testing, and veriﬁcation techniques,” IEEE Control Systems Magazine, vol. 36, no. 6, pp. 45–64, 2016

2016
[54]

D. R. Kuhn, R. N. Kacker, and Y . Lei, Introduction to combinatorial testing. CRC press, 2013

2013
[55]

On a formal model of safe and scalable self-driving cars,

S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a formal model of safe and scalable self-driving cars,” arXiv:1708.06374v2, Tech. Rep., 2017

2017 arXiv
[56]

Comparison and evaluation of advanced motion models for vehicle tracking,

R. Schubert, E. Richter, and G. Wanielik, “Comparison and evaluation of advanced motion models for vehicle tracking,” in 2008 11th Interna- tional Conference on Information Fusion , June 2008, pp. 1–6

2008
[57]

Advanced surrogate target development for evaluating pre-collision systems,

D. J. LeBlanc, M. Gilbert, S. Stachowski, D. Blower, C. A. C. Flannagan, S. Karamihas, and W. T. B. andRini Sherony, “Advanced surrogate target development for evaluating pre-collision systems,” in 23rd Enhanced Safety of Vehicles Conferences , 2013

2013
[58]

Model based vehicle detection and tracking for autonomous urban driving,

A. Petrovskaya and S. Thrun, “Model based vehicle detection and tracking for autonomous urban driving,” Autonomous Robots , vol. 26, no. 2, pp. 123–139, Apr 2009

2009
[59]

Evidence based evaluation method for grid-based environmental representation,

B. Grabe, T. Ike, and M. H ¨otter, “Evidence based evaluation method for grid-based environmental representation,” in FUSION. IEEE, 2009, pp. 1234–1240

2009
[60]

ISO Functional Safety,

“ISO Functional Safety,” https://en.wikipedia.org/wiki/ISO 26262, ac- cessed: 2018-09-11

2018
[61]

In the passenger seat: Inves- tigating ride comfort measures in autonomous cars,

M. Elbanhawi, M. Simic, and R. Jazar, “In the passenger seat: Inves- tigating ride comfort measures in autonomous cars,” IEEE Intelligent Transportation Systems Magazine, vol. 7, no. 3, pp. 4–17, Fall 2015

2015
[62]

Motion sickness and concerns for self-driving vehi- cles: A literature review,

P. Green, “Motion sickness and concerns for self-driving vehi- cles: A literature review,” http://umich.edu/ simdriving/publications/ Motion-Sickness--Report-061616pg-sent.pdf, accessed: 2018-09-11

2018
[63]

An efﬁcient algorithm for monitoring practical tptl speciﬁcations,

A. Dokhanchi, B. Hoxha, C. E. Tuncali, and G. Fainekos, “An efﬁcient algorithm for monitoring practical tptl speciﬁcations,” in Formal Meth- ods and Models for System Design (MEMOCODE), 2016 ACM/IEEE International Conference on . IEEE, 2016, pp. 184–193

2016
[64]

Cyberbotics ltd. Webots: professional mobile robot simula- tion,

O. Michel, “Cyberbotics ltd. Webots: professional mobile robot simula- tion,” International Journal of Advanced Robotic Systems , vol. 1, no. 1, p. 5, 2004

2004
[65]

Tensorﬂow: Large- scale machine learning on heterogeneous distributed systems,

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. , “Tensorﬂow: Large- scale machine learning on heterogeneous distributed systems,” arXiv preprint:1603.04467, 2016

2016 arXiv
[66]

A density-based algorithm for discovering clusters in large spatial databases with noise

M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. , “A density-based algorithm for discovering clusters in large spatial databases with noise.” in Kdd, vol. 96, no. 34, 1996, pp. 226–231

1996
[67]

Dbscan revisited, revisited: why and how you should (still) use dbscan,

E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu, “Dbscan revisited, revisited: why and how you should (still) use dbscan,” ACM Transactions on Database Systems (TODS) , vol. 42, no. 3, p. 19, 2017

2017
[68]

The unscented kalman ﬁlter for nonlinear estimation,

E. A. Wan and R. Van Der Merwe, “The unscented kalman ﬁlter for nonlinear estimation,” in Proceedings of the IEEE 2000 Adaptive Sys- tems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373) . Ieee, 2000, pp. 153–158

2000
[69]

Breach, a toolbox for veriﬁcation and parameter synthesis of hybrid systems,

A. Donze, “Breach, a toolbox for veriﬁcation and parameter synthesis of hybrid systems,” in Computer Aided Veriﬁcation (CAV) , ser. LNCS, vol. 6174. Springer, 2010, pp. 167–170

2010
[70]

Counterexample-guided data augmentation,

T. Dreossi, S. Ghosh, X. Yue, K. Keutzer, A. L. Sangiovanni-Vincentelli, and S. A. Seshia, “Counterexample-guided data augmentation,” in Proceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence, IJCAI , 2018, pp. 2071–2078

2018
[71]

CARLA: An open urban driving simulator,

A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning , 2017, pp. 1–16

2017
[72]

VISPEC: A graphical tool for elicitation of MTL requirements,

B. Hoxha, N. Mavridis, and G. Fainekos, “VISPEC: A graphical tool for elicitation of MTL requirements,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems , 2015

2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

NHTSA Federal Automated Vehicles Pol- icy,

“NHTSA Federal Automated Vehicles Pol- icy,” https://www.transportation.gov/A V/ federal-automated-vehicles-policy-september-2016, accessed: 2018-09- 11

work page 2016

[2] [2]

Guidelines for safe on-road testing of sae level 3, 4, and 5 prototype automated driving systems (ADS), document j3018 201503,

SAE, “Guidelines for safe on-road testing of sae level 3, 4, and 5 prototype automated driving systems (ADS), document j3018 201503,” https://www.sae.org/standards/content/j3018 201503/, accessed: 2018- 09-12

work page 2018

[3] [3]

Key considerations in the development of driving automation systems,

A. Christensen, A. Cunningham, J. Engelman, C. Green, C. Kawashima, S. Kiger, D. Prokhorov, L. Tellis, B. Wendling, and F. Barickman, “Key considerations in the development of driving automation systems,” in 24th Enhanced Safety of Vehicles Conferences , 2015

work page 2015

[4] [4]

Euro NCAP,

“Euro NCAP,” https://www.euroncap.com/en, accessed: 2018-09-11

work page 2018

[5] [5]

Pegasus,

“Pegasus,” https://www.pegasusprojekt.de/en/home, accessed: 2018-09- 11

work page 2018

[6] [6]

Testing of advanced driver assistance towards automated driving: A survey and taxonomy on existing approaches and open questions,

J. E. Stellet, M. R. Zofka, J. Schumacher, T. Schamm, F. Niewels, and J. M. Z ¨ollner, “Testing of advanced driver assistance towards automated driving: A survey and taxonomy on existing approaches and open questions,” 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pp. 1455–1462, 2015

work page 2015

[7] [7]

The sleepwalker framework: Veriﬁcation and validation of autonomous vehicles by mixed reality lidar stimulation,

M. R. Zofka, M. Essinger, T. Fleck, R. Kohlhaas, and J. M. Z ¨ollner, “The sleepwalker framework: Veriﬁcation and validation of autonomous vehicles by mixed reality lidar stimulation,” in SIMPAR. IEEE, 2018, pp. 151–157

work page 2018

[8] [8]

Challenges in autonomous vehicle testing and validation,

P. Koopman and M. Wagner, “Challenges in autonomous vehicle testing and validation,” SAE Int. J. Trans. Safety , vol. 4, pp. 15–24, 04 2016. [Online]. Available: https://doi.org/10.4271/2016-01-0128

doi:10.4271/2016-01-0128 2016