Pith. sign in

REVIEW 2 major objections 4 minor 183 references

Safety Monitoring of Machine Learning Perception Functions: a Survey

T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey argues that safety monitoring for machine-learning perception is best designed around five factors—threats, requirements, detection, reaction, evaluation—and that current research overweights out-of-distribution detection at…

desk verdict A genuinely useful top-down map of ML safety monitoring; the OOD-vs-OMS empirical claim leans on the authors' own benchmark, but the survey's value doesn't rest on it. read the letter →

arxiv 2412.06869 v1 pith:6BU7R2DQ submitted 2024-12-09 cs.LG cs.AIcs.CVcs.SE

Raul Sena Ferreira , Joris Guérin , Kevin Delmas , Jérémie Guiochet , Hélène Waeselynck This is my paper · ORCID
classification cs.LGcs.AIcs.CVcs.SE
keywords safetymonitoringmachinelearningperceptionruntimefaulttolerancesafety-criticalautonomoussystemsout-of-distributiondetectionmechanismsrecovery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to organize the scattered literature on runtime safety monitors for machine-learning perception systems used in critical applications such as self-driving cars and surgical robots. It argues that building a monitor is not just an error-detection problem, and structures the field around five questions: what threats to guard against, how to derive monitor requirements from safety objectives, how to detect failures, how to react, and how to evaluate. Its pointed central claim is that the dominant focus on out-of-distribution (OOD) detection is misplaced, because OOD-ness is ambiguous and monitors should instead be judged by whether they catch incorrect predictions. If true, this gives safety practitioners a checklist and a research agenda, and points to under-studied areas such as requirement elicitation, recovery actions, and standardized evaluation.

What carries the argument

The paper's central organizing device is a two-level taxonomy of safety-monitor design. At the top level are five questions every monitor must answer: which threats, which requirements, which detection mechanism, which reaction, and how to evaluate. At the detection level, it distinguishes internal mechanisms (built into the ML model, such as uncertainty estimation and rejection learning) from external mechanisms (independent components that watch the model's input, internal layer activations, output, or other sensors). The taxonomy carries the argument by turning 'safety monitoring' from a single technique into a design space, and it is what lets the authors argue that most surveys cover only the detection cell of this space.

What would settle it

Run the detection mechanisms surveyed here on a labeled corpus of genuine perception errors, not just out-of-distribution inputs, and observe them catching most errors at low false-positive rates; that outcome would refute the paper's central warning that out-of-distribution-centric monitors are misaligned with error detection.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the safety-monitoring literature for ML perception is best understood top-down through five design factors—threat identification, requirements elicitation, error detection, reaction, and evaluation—rather than bottom-up through detection techniques. It classifies detection mechanisms as internal (uncertainty estimation, domain-knowledge integration, learning with rejection) or external (monitoring inputs, internal activations, outputs, or external sensors), and advocates replacing the OOD-detection framing with an out-of-model-scope framing in which monitors are evaluated by their ability to detect actual prediction errors. It also gathers the comparatively sparse literature on recovery mechanisms and evaluation protocols, and concludes that the biggest open challenges are aligning monitor objectives with system-level safety analysis, combining monitors with plausibility checks, meeting embedded implementation constraints, and standardizing benchmarks.

Load-bearing premise

The survey's coverage is representative because the authors chose the corpus by expert judgment without documenting a search protocol or inclusion criteria, so undetected gaps in the literature would make the claimed open challenges incomplete.

Editorial extensions

If this is right

  • Evaluation of monitors should shift its target from out-of-distribution detection to the detection of actual model errors, with test sets labeled by whether the model's prediction is wrong.
  • Requirement elicitation becomes a first-class research problem: deriving monitor specifications from system-level hazard analysis (for instance, identifying state-space regions where an ML failure is safety-critical) is a promising direction the survey identifies.
  • Practical monitors will likely combine several mechanisms—data-driven detectors alongside plausibility checks, model assertions, and failure-mode analysis—because no single category covers all threats.
  • Recovery mechanisms need more attention than they currently receive: control switching, input enhancement, and alternative perception components are underdeveloped relative to detection.
  • Progress toward certifying ML-based perception will depend on unified benchmarks and metrics that reflect the full monitor lifecycle, not just detection accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the out-of-model-scope framing is right, then widely used OOD benchmark suites should be complemented by 'error benchmarks' that record whether the deployed model is actually wrong, which would make many published detection results non-comparable to the monitor's real purpose.
  • The five-factor structure could be instantiated as a reusable design checklist for safety practitioners, effectively turning the survey into a template for deriving monitor requirements from a hazard analysis.
  • As detection mechanisms mature, the practical bottleneck should shift to the reaction step: testable engineering work could pair this detection taxonomy with control-theoretic safety envelopes to measure end-to-end safety contribution, not just detection quality.
  • The internal/external distinction suggests a certification hypothesis worth testing: external monitors, being separable from the ML model, may be easier to verify and certify than internal mechanisms, which would make external monitoring attractive even when it is less accurate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper is a survey of runtime safety monitoring for ML-based perception functions in safety-critical autonomous systems. It organizes the literature around five design questions: threat identification (Section 3), requirements elicitation (Section 4), failure detection (Section 5), reaction/recovery (Section 6), and evaluation (Section 7). The detection taxonomy distinguishes internal mechanisms (uncertainty estimation, domain knowledge, learning with rejection) from external mechanisms (input, internal-representation, output, and multi-sensor monitoring). The survey also advances a research agenda centered on out-of-model-scope (OMS) evaluation rather than pure out-of-distribution (OOD) detection, and concludes with open challenges including standardized benchmarking and certification. The principal contribution is a top-down map of the field that covers aspects (requirements, reaction, system-level evaluation) that are rare in prior bottom-up surveys.

Significance. If the survey's organizational structure is adopted, it provides a useful reference map for both safety practitioners and ML researchers, and it identifies several genuine gaps: the scarcity of systematic requirements-to-monitor derivation, the underdevelopment of recovery mechanisms beyond basic alerts, and the absence of standardized evaluation benchmarks. The paper is careful in many places, explicitly acknowledging that evidence on detector efficacy is conflicting (Section 5) and that evaluation protocols vary widely (Section 7). Its strongest assets are the breadth of the detection taxonomy, the clear separation of runtime threat types, and the concrete discussion of evaluation metrics and system-level safety gain. The main caveats are that the survey does not document a systematic search protocol and that its central OOD-to-OMS critique rests heavily on a single self-cited benchmark whose protocol is not described in the paper.

major comments (2)
  1. [Section 8 and Section 7.1.2] The claim that "most detection mechanisms based on out-of-distribution detection suffer from a high number of false positives and false negatives" when detecting actual ML model failures is load-bearing: it motivates the OMS evaluation paradigm in Section 7.1.2 and appears as the first open challenge in Section 8. Yet the only evidence cited is Ferreira et al. [37], and the survey does not report that benchmark's datasets, detector selection, threat distributions, labeling scheme, or metrics. Since [37] is the authors' own prior work and the survey itself notes that the broader literature gives conflicting results (Section 5), the claim should either be accompanied by a concise description of the benchmark protocol and its limitations, or be explicitly presented as a preliminary finding from a single study rather than a general negative result.
  2. [Section 2] The survey describes itself as an "extensive literature review" and claims that the top-down approach allows it to "uncover specific areas where research is lacking," but it provides no search protocol, inclusion/exclusion criteria, or coverage statistics. The corpus appears to have been assembled through expert judgment, which is a legitimate method for a survey, but without any documentation of the selection process the representativeness of the coverage cannot be independently verified. Please add a short methodology paragraph describing how references were identified and screened, or temper the extensiveness claim accordingly.
minor comments (4)
  1. [Figure 3] The reference lists inside the taxonomy boxes (e.g., "67,68,69,11") are not consistently sorted and some references appear in multiple boxes without explanation; consider reordering and adding a note on whether multiple appearances indicate multi-purpose mechanisms.
  2. [Section 3.3.2] The distinction between semantic shift and covariate shift is clear conceptually, but the statement that semantic shifts "cannot be handled with denoising or backup sensors" (Section 3.3.2) could be read as contradicting the later discussion of input reconstruction and alternative sensors in Section 6.2; a cross-reference clarifying the scope of that statement would help.
  3. [Section 7.1.2] The metrics list is useful, but the definition of P@80R (precision at recall 0.8) is placed after AUPR without an explicit formula; a one-line formal definition would improve precision.
  4. [References] A few references have inconsistent metadata, e.g., [174] lists a Master's thesis in the venue field and [13] is missing page numbers; a final reference cleanup pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey is organizational and its self-citations are external, reproducible evidence.

full rationale

This paper is a literature survey rather than a derivation: it does not derive a quantitative result from assumptions, so there is no derivation chain in which an output could collapse into an input. The five-factor taxonomy (threat identification, requirements elicitation, detection of failure, reaction, and evaluation) is an organizational structure, not a prediction, and it is presented as a top-down framing rather than as a theorem. The places where the authors cite their own prior work are used as supporting evidence for open challenges: Section 8 cites Ferreira et al. 37 for the empirical finding that OOD-based detection mechanisms suffer from high false-positive and false-negative rates when judged against actual model failures, and Section 7.1.2 cites Guerin et al. 29 for the out-of-model-scope evaluation perspective. However, these citations are to peer-reviewed, externally reproducible studies with their own datasets and protocols; the survey does not fit parameters to data, rename a fitted value as a prediction, or use an author-imported uniqueness theorem. Even the claim that not all threats lead to errors and some in-distribution images lead to wrong predictions is a conceptual point stated directly in the paper, independent of the cited benchmark. No self-definitional equation, fitted-input-as-prediction step, or load-bearing self-citation chain appears, so the survey is self-contained in the sense relevant to circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey introduces no fitted parameters and no new entities. Its claims rest on domain assumptions about safety engineering and on the authors' organizational and evaluative choices, several of which are self-cited.

assumptions (3)
  • domain assumption Conventional offline safety measures are insufficient for ML-based perception, so online safety monitors are necessary.
    Adopted in Section 1 to frame the survey's relevance; not proved in the paper.
  • ad hoc to paper The correct decomposition of safety-monitor design is threat identification, requirements elicitation, detection, reaction, and evaluation.
    This five-part structure is the paper's organizing choice, asserted in Section 1 and used to select and arrange the literature.
  • domain assumption Out-of-model-scope error detection is the right evaluation target rather than out-of-distribution detection.
    Presented in Section 7.1.2, citing the authors' own prior work (ref 29); it is an argued position, not a demonstrated fact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Safety Monitoring of Machine Learning Perception Functions: a Survey." pith.science (2026). https://pith.science/paper/6BU7R2DQ

@misc{pith2026241206869,
  author       = {Pith},
  title        = {Pith review of: Safety Monitoring of Machine Learning Perception Functions: a Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6BU7R2DQ}},
  note         = {Machine review of arXiv:2412.06869}
}
read the original abstract

Machine Learning (ML) models, such as deep neural networks, are widely applied in autonomous systems to perform complex perception tasks. New dependability challenges arise when ML predictions are used in safety-critical applications, like autonomous cars and surgical robots. Thus, the use of fault tolerance mechanisms, such as safety monitors, is essential to ensure the safe behavior of the system despite the occurrence of faults. This paper presents an extensive literature review on safety monitoring of perception functions using ML in a safety-critical context. In this review, we structure the existing literature to highlight key factors to consider when designing such monitors: threat identification, requirements elicitation, detection of failure, reaction, and evaluation. We also highlight the ongoing challenges associated with safety monitoring and suggest directions for future research.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

183 extracted references · 66 canonical work pages

  1. [37]

    Benchmarking Safety Monitors for Image Classifiers with Machine Learning

    Ferreira Raul Sena, Arlat Jean, Guiochet Jérémie, Waeselynck Hélène. Benchmarking Safety Monitors for Image Classifiers with Machine Learning. In: 2021 IEEE 26th Pacific Rim International Symposium on Dependable Computing (PRDC):7–16IEEE; 2021

    2021

  2. [1]

    Runtime Monitoring of Cyber-Physical Systems Using Data-driven Models

    Calvi Michele Giovanni. Runtime Monitoring of Cyber-Physical Systems Using Data-driven Models. PhD thesisUniversity of Illinois at Chicago2019

  3. [2]

    Autonomy for surgical robots: Concepts and paradigms.IEEE Transactions on Medical Robotics and Bionics.2019;1(2):65–76

    Haidegger Tamás. Autonomy for surgical robots: Concepts and paradigms.IEEE Transactions on Medical Robotics and Bionics.2019;1(2):65–76

    2019

  4. [3]

    Certifying Emergency Landing for Safe Urban UA V

    Guérin Joris, Delmas Kevin, Guiochet Jérémie. Certifying Emergency Landing for Safe Urban UA V . In:7th International Workshop on Safety and Security of Intelligent Vehicles (SSIV 2021) at IEEE/IFIP Intern. Conf. on Dependable Systems and Networks (DSN):55–62; 2021

    2021

  5. [4]

    Intelligent Robotic Perception Systems

    Premebida Cristiano, Ambrus Rares, Marton Zoltan-Csaba. Intelligent Robotic Perception Systems. In: Hurtado Efren Gorrostieta, ed.Applications of Mobile RobotsRijeka: IntechOpen 2018

    2018

  6. [5]

    Benchmarking deep reinforcement learning for continuous control

    Duan Yan, Chen Xi, Houthooft Rein, Schulman John, Abbeel Pieter. Benchmarking deep reinforcement learning for continuous control. In: International conference on machine learning:1329–1338PMLR; 2016

    2016

  7. [6]

    Computer vision and deep learning techniques for pedestrian detection and tracking: A survey

    Brunetti Antonio, Buongiorno Domenico, Trotta Gianpaolo Francesco, Bevilacqua Vitoantonio. Computer vision and deep learning techniques for pedestrian detection and tracking: A survey. Neurocomputing. 2018;300:17–33

    2018

  8. [7]

    Evaluation of runtime monitoring for UA V emergency landing

    Guerin Joris, Delmas Kevin, Guiochet Jérémie. Evaluation of runtime monitoring for UA V emergency landing. In:2022 International Conference on Robotics and Automation (ICRA):9703–9709IEEE; 2022

    2022

Show all 183 references
  1. [8]

    On the safety of machine learning: Cyber-physical systems, decision sciences, and data products

    Varshney Kush R, Alemzadeh Homa. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big data. 2017;5(3):246–255

    2017

  2. [9]

    Machine learning safety: An overview

    Faria José M. Machine learning safety: An overview. In: Proceedings of the 26th Safety-Critical Systems Symposium, York, UK:6–8; 2018

    2018

  3. [10]

    Practical solutions for machine learning safety in autonomous vehicles

    Mohseni Sina, Pitale Mandar, Singh Vasu, Wang Zhangyang. Practical solutions for machine learning safety in autonomous vehicles. In: Proceedings of the Workshop on Artificial Intelligence Safety:162–169; 2020

    2020

  4. [11]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Gal Yarin, Ghahramani Zoubin. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: International conference on machine learning (ICML), New York, United States:1050–1059; 2016

    2016

  5. [12]

    Basic concepts and taxonomy of dependable and secure computing

    Avizienis Algirdas, Laprie J-C, Randell Brian, Landwehr Carl. Basic concepts and taxonomy of dependable and secure computing. IEEE transactions on dependable and secure computing. 2004;1(1):11–33

    2004

  6. [13]

    SMOF: A safety monitoring framework for autonomous systems

    Machin Mathilde, Guiochet Jérémie, Waeselynck Hélène, Blanquart Jean-Paul, Roy Matthieu, Masson Lola. SMOF: A safety monitoring framework for autonomous systems. IEEE Transactions on Systems, Man, and Cybernetics: Systems. 2018;48(5):702–715

    2018

  7. [14]

    Kernels for safety

    Rushby John. Kernels for safety. Safe and Secure Computing Systems. 1989;:210–220

    1989

  8. [15]

    A safety integrated architecture for an autonomous safety excavator

    Pace Conrad, Seward Derek. A safety integrated architecture for an autonomous safety excavator. In:International Symposium on Automation and Robotics in Construction; 2000

    2000

  9. [16]

    The ranger robotic satellite servicer and its autonomous software-based safety system

    Roderick Stephen, Roberts Brian, Atkins Ella, Akin Dave. The ranger robotic satellite servicer and its autonomous software-based safety system. IEEE Intelligent Systems. 2004;19(5):12–19

    2004

  10. [17]

    Dependable execution control for autonomous robots

    Py Frédéric, Ingrand Félix. Dependable execution control for autonomous robots. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566):1136–1141IEEE; 2004

    2004

  11. [18]

    Safe and sound

    Fox John, Das Subrata. Safe and sound. Artificial intelligence in hazardous applications. 2000;307

    2000

  12. [19]

    The safety-bag expert system in the electronic railway interlocking system Elektra

    Klein Peter. The safety-bag expert system in the electronic railway interlocking system Elektra. In: Operational Expert System Applications in EuropeElsevier 1991 (pp. 1–15)

    1991

  13. [20]

    Towards the robotic co-worker

    Haddadin Sami, Suppa Michael, Fuchs Stefan, Bodenmüller Tim, Albu-Schäffer Alin, Hirzinger Gerd. Towards the robotic co-worker. In: Robotics ResearchSpringer 2011 (pp. 261–282)

    2011

  14. [21]

    Safety monitoring for autonomous systems: interactive elicitation of safety rules

    Masson Lola. Safety monitoring for autonomous systems: interactive elicitation of safety rules. PhD thesisUniversité Paul Sabatier-Toulouse III2019

  15. [22]

    Machine learning for reliability engineering and safety applications: Review of current status and future opportunities

    Xu Zhaoyi, Saleh Joseph Homer. Machine learning for reliability engineering and safety applications: Review of current status and future opportunities. Reliability Engineering & System Safety. 2021;211:107530

    2021

  16. [23]

    Deep learning for safe autonomous driving: Current challenges and future directions

    Muhammad Khan, Ullah Amin, Lloret Jaime, Del Ser Javier, Albuquerque Victor Hugo C. Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Transactions on Intelligent Transportation Systems.2020

    2020

  17. [24]

    Deep learning-based applications for safety management in the AEC industry: A review

    Hou Lei, Chen Haosen, Zhang Guomin, Wang Xiangyu. Deep learning-based applications for safety management in the AEC industry: A review. Applied Sciences. 2021;11(2):821

    2021

  18. [25]

    Sensing and machine learning for automotive perception: A review

    Pandharipande Ashish, Cheng Chih-Hong, Dauwels Justin, et al. Sensing and machine learning for automotive perception: A review. IEEE Sensors Journal. 2023;. 20 Ferreira ET AL

    2023

  19. [26]

    Taxonomy of machine learning safety: A survey and primer

    Mohseni Sina, Wang Haotao, Xiao Chaowei, Yu Zhiding, Wang Zhangyang, Yadawa Jay. Taxonomy of machine learning safety: A survey and primer. ACM Computing Surveys. 2022;55(8):1–38

    2022

  20. [27]

    Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends

    Rahman Quazi Marufur, Corke Peter, Dayoub Feras. Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends. IEEE Access. 2021;9:20067–20075

    2021

  21. [28]

    A survey on learning to reject.Proceedings of the IEEE

    Zhang Xu-Yao, Xie Guo-Sen, Li Xiuli, Mei Tao, Liu Cheng-Lin. A survey on learning to reject.Proceedings of the IEEE. 2023;111(2):185–215

    2023

  22. [29]

    Out-Of-Distribution Detection Is Not All You Need

    Guerin Joris, Delmas Kevin, Ferreira Raul Sena, Guiochet Jérémie. Out-Of-Distribution Detection Is Not All You Need. In: The 37th AAAI conference on artificial intelligence (2023); 2023

    2023

  23. [30]

    Review of data preprocessing techniques in data mining

    Alasadi Suad A, Bhaya Wesam S. Review of data preprocessing techniques in data mining. Journal of Engineering and Applied Sciences. 2017;12(16):4102–4107

    2017

  24. [31]

    Training deep neural-networks based on unreliable labels

    Bekker Alan Joseph, Goldberger Jacob. Training deep neural-networks based on unreliable labels. In:IEEE International Conference on Acoustics, Speech and Signal Processing, China:2682–2686IEEE; 2016

    2016

  25. [32]

    The ml test score: A rubric for ml production readiness and technical debt reduction

    Breck Eric, Cai Shanqing, Nielsen Eric, Salib Michael, Sculley D. The ml test score: A rubric for ml production readiness and technical debt reduction. In: 2017 IEEE International Conference on Big Data (Big Data):1123–1132IEEE; 2017

    2017

  26. [33]

    Hidden technical debt in machine learning systems

    Sculley David, Holt Gary, Golovin Daniel, et al. Hidden technical debt in machine learning systems. In:Advances in neural information processing systems (NeurIPS), Montreal, Canada:2503–2511; 2015

    2015

  27. [34]

    A review of novelty detection

    Pimentel Marco A.F., Clifton David A., Clifton Lei, Tarassenko Lionel. A review of novelty detection. Signal Processing. 2014;99:215 - 249

    2014

  28. [35]

    Adversarial attacks and defences: A survey

    Chakraborty Anirban, Alam Manaar, Dey Vishal, Chattopadhyay Anupam, Mukhopadhyay Debdeep. Adversarial attacks and defences: A survey. arXiv preprint arXiv:1810.00069. 2018

    2018 arXiv

  29. [36]

    DOCTOR: A Simple Method for Detecting Misclassification Errors

    Granese Federica, Romanelli Marco, Gorla Daniele, Palamidessi Catuscia, Piantanida Pablo. DOCTOR: A Simple Method for Detecting Misclassification Errors. Advances in Neural Information Processing Systems. 2021;34

    2021

  30. [38]

    Towards out-of-distribution generalization: A survey.arXiv preprint arXiv:2108.13624

    Shen Zheyan, Liu Jiashuo, He Yue, et al. Towards out-of-distribution generalization: A survey.arXiv preprint arXiv:2108.13624. 2021

    2021 arXiv

  31. [39]

    SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images

    Ferreira Raul Sena, Guérin Joris, Guiochet Jérémie, Waeselynck Helene. SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images. In: 2022 IEEE 27th Pacific Rim International Symposium on Dependable Computing (PRDC):68–77IEEE; 2022

    2022

  32. [40]

    The Cityscapes Dataset for Semantic Urban Scene Understanding

    Cordts Marius, Omran Mohamed, Ramos Sebastian, et al. The Cityscapes Dataset for Semantic Urban Scene Understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016

    2016

  33. [41]

    Segmentation transformer: Object-contextual representations for semantic segmentation

    Yuan Yuhui, Chen Xiaokang, Chen Xilin, Wang Jingdong. Segmentation transformer: Object-contextual representations for semantic segmentation. In: European Conference on Computer Vision (ECCV); 2021

    2021

  34. [42]

    Imagenet: A large-scale hierarchical image database

    Deng Jia, Dong Wei, Socher Richard, Li Li-Jia, Li Kai, Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In:2009 IEEE conference on computer vision and pattern recognition:248–255Ieee; 2009

    2009

  35. [43]

    Microsoft coco: Common objects in context

    Lin Tsung-Yi, Maire Michael, Belongie Serge, et al. Microsoft coco: Common objects in context. In: European conference on computer vision:740–755Springer; 2014

    2014

  36. [44]

    Swin Transformer V2: Scaling Up Capacity and Resolution

    Liu Ze, Hu Han, Lin Yutong, et al. Swin Transformer V2: Scaling Up Capacity and Resolution. arXiv preprint arXiv:2111.09883. 2021

    2021 arXiv

  37. [45]

    Uncertainty in Machine Learning: A Safety Perspective on Autonomous Driving

    Shafaei Sina, Kugele Stefan, Osman Mohd Hafeez, Knoll Alois. Uncertainty in Machine Learning: A Safety Perspective on Autonomous Driving. In: International Conference on Computer Safety, Reliability, and Security (SAFECOMP):458–464; 2018

    2018

  38. [46]

    Generalized out-of-distribution detection: A survey

    Yang Jingkang, Zhou Kaiyang, Li Yixuan, Liu Ziwei. Generalized out-of-distribution detection: A survey. preprint arXiv:2110.11334. 2021

    2021 arXiv

  39. [47]

    Fishyscapes: A benchmark for safe semantic segmentation in autonomous driving

    Blum Hermann, Sarlin Paul-Edouard, Nieto Juan, Siegwart Roland, Cadena Cesar. Fishyscapes: A benchmark for safe semantic segmentation in autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops:0–0; 2019

    2019

  40. [48]

    Performance measures for classification systems with rejection

    Condessa Filipe, Bioucas-Dias José, Kovaˇcevi´c Jelena. Performance measures for classification systems with rejection. Pattern Recognition. 2017;63:437–450

    2017

  41. [49]

    AMANDA: Semi-supervised Density-based Adaptive Model for Non-stationary Data with Extreme Verification Latency.Information Sciences

    Ferreira Raul S, Zimbrão Geraldo, Alvim Leandro GM. AMANDA: Semi-supervised Density-based Adaptive Model for Non-stationary Data with Extreme Verification Latency.Information Sciences. 2019;488:219-237

    2019

  42. [50]

    Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

    Hendrycks Dan, Dietterich Thomas. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In: International Conference on Learning Representations; 2019

    2019

  43. [51]

    Sensor fault detection and diagnosis for autonomous systems

    Khalastchi Eliahu, Kalech Meir, Rokach Lior. Sensor fault detection and diagnosis for autonomous systems. In: Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems:15–22Citeseer; 2013. 21

    2013

  44. [52]

    Active tuning of intrinsic camera parameters

    Micheloni Christian, Foresti Gian Luca. Active tuning of intrinsic camera parameters. IEEE transactions on automation science and engineering. 2009;6(4):577–587

    2009

  45. [53]

    Deployment of backup sensors in wireless sensor networks for structural health monitoring

    Surya S, Ravi R. Deployment of backup sensors in wireless sensor networks for structural health monitoring. In: 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI):1526–1533IEEE; 2018

    2018

  46. [54]

    Crash and disengagement data of autonomous vehicles on public roads in California

    Sinha Amolika, Chand Sai, Vu Vincent, Chen Huang, Dixit Vinayak. Crash and disengagement data of autonomous vehicles on public roads in California. Scientific data. 2021;8(1):1–10

    2021

  47. [55]

    Survey of image denoising techniques

    Motwani Mukesh C, Gadiya Mukesh C, Motwani Rakhi C, Harris Frederick C. Survey of image denoising techniques. In: Proceedings of GSPX:27–30Proceedings of GSPX; 2004

    2004

  48. [56]

    Deep learning on image denoising: An overview.Neural Networks

    Tian Chunwei, Fei Lunke, Zheng Wenxian, Xu Yong, Zuo Wangmeng, Lin Chia-Wen. Deep learning on image denoising: An overview.Neural Networks. 2020

    2020

  49. [57]

    It’s Not All About Size: On the Role of Data Properties in Pedestrian Detection

    Rasouli Amir, Kotseruba Iuliia, Tsotsos John K. It’s Not All About Size: On the Role of Data Properties in Pedestrian Detection. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops:0–0; 2018

    2018

  50. [58]

    Autopilot Blamed for Tesla’s Crash Into Overturned Truck

    Stumpf Rob. Autopilot Blamed for Tesla’s Crash Into Overturned Truck. Online; accessed 07 June 2022; 2020

    2022

  51. [59]

    Attribute-aware pedestrian detection in a crowd

    Zhang Jialiang, Lin Lixiang, Zhu Jianke, et al. Attribute-aware pedestrian detection in a crowd. IEEE Transactions on Multimedia. 2020

    2020

  52. [60]

    Threat of adversarial attacks on deep learning in computer vision: A survey

    Akhtar Naveed, Mian Ajmal. Threat of adversarial attacks on deep learning in computer vision: A survey. Ieee Access. 2018;6:14410–14430

    2018

  53. [61]

    Adversarial examples in the physical world

    Kurakin Alexey, Goodfellow Ian J, Bengio Samy. Adversarial examples in the physical world. In:Artificial intelligence safety and securityChapman and Hall/CRC 2018 (pp. 99–112)

    2018

  54. [62]

    Attacking vision-based perception in end-to-end autonomous driving models

    Boloor Adith, Garimella Karthik, He Xin, Gill Christopher, V orobeychik Yevgeniy, Zhang Xuan. Attacking vision-based perception in end-to-end autonomous driving models. Journal of Systems Architecture.2020;110:101766

    2020

  55. [63]

    Experience with model-based user-centered risk assessment for service robots

    Guiochet Jeremie, Martin-Guillerez Damien, Powell David. Experience with model-based user-centered risk assessment for service robots. In: 2010 IEEE 12th International Symposium on High Assurance Systems Engineering:104–113IEEE; 2010

    2010

  56. [64]

    Compositional falsification of cyber-physical systems with machine learning components

    Dreossi Tommaso, Donzé Alexandre, Seshia Sanjit A. Compositional falsification of cyber-physical systems with machine learning components. Journal of Automated Reasoning. 2019;63(4):1031–1053

    2019

  57. [65]

    Verifai: A toolkit for the formal design and analysis of artificial intelligence-based systems

    Dreossi Tommaso, Fremont Daniel J, Ghosh Shromona, et al. Verifai: A toolkit for the formal design and analysis of artificial intelligence-based systems. In: International Conference on Computer Aided Verification:432–442Springer; 2019

    2019

  58. [66]

    A safety analysis method for perceptual components in automated driving

    Salay Rick, Angus Matt, Czarnecki Krzysztof. A safety analysis method for perceptual components in automated driving. In: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE):24–34IEEE; 2019

    2019

  59. [67]

    Efficient uncertainty estimation for semantic segmentation in videos

    Huang Po-Yu, Hsu Wan-Ting, Chiu Chun-Yueh, Wu Ting-Fan, Sun Min. Efficient uncertainty estimation for semantic segmentation in videos. In: Proceedings of the European Conference on Computer Vision (ECCV):520–535; 2018

    2018

  60. [68]

    Calibrating uncertainty models for steering angle estimation

    Hubschneider Christian, Hutmacher Robin, Zöllner J Marius. Calibrating uncertainty models for steering angle estimation. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC):1511–1518IEEE; 2019

    2019

  61. [69]

    Uncertainty Estimation for Data-Driven Visual Odometry

    Costante Gabriele, Mancini Michele. Uncertainty Estimation for Data-Driven Visual Odometry. IEEE Transactions on Robotics. 2020;PP:1-20

    2020

  62. [70]

    Superpixel-based Domain-Knowledge Infusion in Computer Vision

    Chhablani Gunjan, Sharma Abheesht, Pandey Harshit, Dash Tirtharaj. Superpixel-based Domain-Knowledge Infusion in Computer Vision. arXiv preprint arXiv:2105.09448. 2021

    2021 arXiv

  63. [71]

    Learning semantic relationships for better action retrieval in images

    Ramanathan Vignesh, Li Congcong, Deng Jia, et al. Learning semantic relationships for better action retrieval in images. In: Proceedings of the IEEE conference on computer vision and pattern recognition:1100–1109; 2015

    2015

  64. [72]

    A semantic loss function for deep learning with symbolic knowledge

    Xu Jingyi, Zhang Zilu, Friedman Tal, Liang Yitao, Broeck Guy. A semantic loss function for deep learning with symbolic knowledge. In: International conference on machine learning:5502–5511PMLR; 2018

    2018

  65. [73]

    Logic tensor networks for semantic image interpretation

    Donadello I, Serafini L, Garcez AS. Logic tensor networks for semantic image interpretation. In: IJCAI International Joint Conference on Artificial Intelligence:1596–1602IJCAI; 2017

    2017

  66. [74]

    Boosting with abstention

    Cortes Corinna, DeSalvo Giulia, Mohri Mehryar. Boosting with abstention. Advances in Neural Information Processing Systems. 2016;29:1660– 1668

    2016

  67. [75]

    Selectivenet: A deep neural network with an integrated reject option

    Geifman Yonatan, El-Yaniv Ran. Selectivenet: A deep neural network with an integrated reject option. In:International Conference on Machine Learning:2151–2159PMLR; 2019

    2019

  68. [76]

    Automated evaluation of semantic segmentation robustness for autonomous driving

    Zhou Wei, Berrio Julie Stephany, Worrall Stewart, Nebot Eduardo. Automated evaluation of semantic segmentation robustness for autonomous driving. IEEE Transactions on Intelligent Transportation Systems.2019;21(5):1951–1963

    2019

  69. [77]

    Failing to learn: Autonomously identifying perception failures for self-driving cars

    Ramanagopal Manikandasriram Srinivasan, Anderson Cyrus, Vasudevan Ram, Johnson-Roberson Matthew. Failing to learn: Autonomously identifying perception failures for self-driving cars. IEEE Robotics and Automation Letters. 2018;3(4):3860–3867. 22 Ferreira ET AL

    2018

  70. [78]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks

    Hendrycks Dan, Gimpel Kevin. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136. 2016

    2016 arXiv

  71. [79]

    Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks

    Liang Shiyu, Li Yixuan, Srikant R.. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In:International Conference on Learning Representations; 2018

    2018

  72. [80]

    Generalized odin: Detecting out-of-distribution image without learning from out-of- distribution data

    Hsu Yen-Chang, Shen Yilin, Jin Hongxia, Kira Zsolt. Generalized odin: Detecting out-of-distribution image without learning from out-of- distribution data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition:10951–10960; 2020

    2020

  73. [81]

    Model assertions for debugging machine learning

    Kang Daniel, Raghavan Deepti, Bailis Peter, Zaharia Matei. Model assertions for debugging machine learning. In: NeurIPS MLSys Workshop:10; 2018

    2018

  74. [82]

    Safety Validation of Autonomous Vehicles using Assertion-based Oracles

    Harper Christopher, Chance Greg, Ghobrial Abanoub, Alam Saquib, Pipe Tony, Eder Kerstin. Safety Validation of Autonomous Vehicles using Assertion-based Oracles. arXiv preprint arXiv:2111.04611. 2021

    2021 arXiv

  75. [83]

    Monitoring Object Detection Abnormalities via Data-Label and Post-Algorithm Abstractions

    Chen Yuhang, Cheng Chih-Hong, Yan Jun, Yan Rongjie. Monitoring Object Detection Abnormalities via Data-Label and Post-Algorithm Abstractions. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS):6688–6693IEEE; 2021

    2021

  76. [84]

    Robust Detection of Objects under Periodic Motion with Gaussian Process Filtering

    Guérin Joris, Paula Canuto Anne Magaly, Goncalves Luiz Marcos Garcia. Robust Detection of Objects under Periodic Motion with Gaussian Process Filtering. In: 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA):685–692IEEE; 2020

    2020

  77. [85]

    A consensus novelty detection ensemble approach for anomaly detection in activities of daily living

    Yahaya Salisu Wada, Lotfi Ahmad, Mahmud Mufti. A consensus novelty detection ensemble approach for anomaly detection in activities of daily living. Applied Soft Computing. 2019;83:105613

    2019

  78. [86]

    Informed democracy: voting-based novelty detection for action recognition

    Roitberg Alina, Al-Halah Ziad, Stiefelhagen Rainer. Informed democracy: voting-based novelty detection for action recognition. British Machine Vision Conference.2018

    2018

  79. [87]

    VisionGuard: Runtime detection of adversarial inputs to perception systems.arXiv preprint arXiv:2002.09792

    Kantaros Yiannis, Carpenter Taylor, Park Sangdon, et al. VisionGuard: Runtime detection of adversarial inputs to perception systems.arXiv preprint arXiv:2002.09792. 2020

    2002 arXiv

  80. [88]

    Adversarial sample detection for deep neural network through model mutation testing

    Wang Jingyi, Dong Guoliang, Sun Jun, Wang Xinyu, Zhang Peixin. Adversarial sample detection for deep neural network through model mutation testing. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE):1245–1256IEEE; 2019

    2019

  81. [89]

    Input Validation for Neural Networks via Runtime Local Robustness Verification.arXiv preprint arXiv:2002.03339

    Liu Jiangchao, Chen Liqian, Mine Antoine, Wang Ji. Input Validation for Neural Networks via Runtime Local Robustness Verification.arXiv preprint arXiv:2002.03339. 2020

    2002 arXiv

  82. [90]

    Did you miss the sign? A false negative alarm system for traffic sign detectors

    Rahman Quazi Marufur, Sünderhauf Niko, Dayoub Feras. Did you miss the sign? A false negative alarm system for traffic sign detectors. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS):3748–3753IEEE; 2019

    2019

  83. [91]

    ReAct: Out-of-distribution Detection With Rectified Activations

    Sun Yiyou, Guo Chuan, Li Yixuan. ReAct: Out-of-distribution Detection With Rectified Activations. In: Ranzato M., Beygelzimer A., Dauphin Y ., Liang P.S., Vaughan J. Wortman, eds.Advances in Neural Information Processing Systems:144–157Curran Associates, Inc.; 2021

    2021

  84. [92]

    Into the unknown: Active monitoring of neural networks

    Lukina Anna, Schilling Christian, Henzinger Thomas A. Into the unknown: Active monitoring of neural networks. In: International Conference on Runtime Verification:42–61Springer; 2021

    2021

  85. [93]

    Runtime monitoring neuron activation patterns

    Cheng Chih-Hong, Nührenberg Georg, Yasuoka Hirotoshi. Runtime monitoring neuron activation patterns. In:2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), Florence, Italy:300–303IEEE; 2019

    2019

  86. [94]

    Outside the Box: Abstraction-Based Monitoring of Neural Networks

    Henzinger Thomas A, Lukina Anna, Schilling Christian. Outside the Box: Abstraction-Based Monitoring of Neural Networks. In: 24th European Conference on Artificial Intelligence-ECAI 2020:2433–2440; 2020

    2020

  87. [95]

    SENA: Similarity-based Error-checking of Neural Activations

    Ferreira Raul S, Guerin Joris, Guiochet Jeremie, Waeselynck Helene. SENA: Similarity-based Error-checking of Neural Activations. In:26th European Conference on Artificial Intelligence-ECAI 2023; 2023

    2023

  88. [96]

    Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes

    Wu Changshun, Falcone Yliès, Bensalem Saddek. Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes. arXiv preprint arXiv:2104.14435. 2021

    2021 arXiv

  89. [97]

    Dissector: Input validation for deep learning applications by crossing-layer dissection

    Wang Huiyan, Xu Jingwei, Xu Chang, Ma Xiaoxing, Lu Jian. Dissector: Input validation for deep learning applications by crossing-layer dissection. In: 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE):727–738IEEE; 2020

    2020

  90. [98]

    FACER: A universal framework for detecting anomalous operation of deep neural networks

    Schorn Christoph, Gauerhof Lydia. FACER: A universal framework for detecting anomalous operation of deep neural networks. In:2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC):1–6IEEE; 2020

    2020

  91. [99]

    A simple unified framework for detecting out-of-distribution samples and adversarial attacks

    Lee Kimin, Lee Kibok, Lee Honglak, Shin Jinwoo. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems. 2018;31

    2018

  92. [100]

    Task-Aware Novelty Detection for Visual-based Deep Learning in Autonomous Systems

    Chen Valerie, Yoon Man-Ki, Shao Zhong. Task-Aware Novelty Detection for Visual-based Deep Learning in Autonomous Systems. In:2020 IEEE International Conference on Robotics and Automation (ICRA):11060–11066IEEE; 2020

    2020

  93. [101]

    Signal processing-based anomaly detection techniques: a comparative analysis

    Ndong Joseph, Salamatian Kavé. Signal processing-based anomaly detection techniques: a comparative analysis. In: Proc. 2011 3rd International Conference on Evolving Internet:32–39; 2011. 23

    2011

  94. [102]

    Image-based anomaly detection technique: algorithm, implementation and effectiveness

    Kim Seong Soo, Reddy AL Narasimha. Image-based anomaly detection technique: algorithm, implementation and effectiveness. IEEE Journal on Selected Areas in Communications. 2006;24(10):1942–1954

    2006

  95. [103]

    Video-based water drop detection and removal method for a moving vehicle.Information Technology Journal

    Liao H, Wang D, Yang C, Shine J. Video-based water drop detection and removal method for a moving vehicle.Information Technology Journal. 2013;12(4):569–583

    2013

  96. [104]

    Adversarially learned one-class classifier for novelty detection

    Sabokrou Mohammad, Khalooei Mohammad, Fathy Mahmood, Adeli Ehsan. Adversarially learned one-class classifier for novelty detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition:3379–3388; 2018

    2018

  97. [105]

    Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance

    Denouden Taylor, Salay Rick, Czarnecki Krzysztof, Abdelzad Vahdat, Phan Buu, Vernekar Sachin. Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance. arXiv preprint arXiv:1812.02765. 2018

    2018 arXiv

  98. [106]

    Misbehaviour prediction for autonomous driving systems

    Stocco Andrea, Weiss Michael, Calzana Marco, Tonella Paolo. Misbehaviour prediction for autonomous driving systems. In:Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering:359–371; 2020

    2020

  99. [107]

    Real-time out-of-distribution detection in learning-enabled cyber-physical systems

    Cai Feiyang, Koutsoukos Xenofon. Real-time out-of-distribution detection in learning-enabled cyber-physical systems. In: 2020 ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS):174–183IEEE; 2020

    2020

  100. [108]

    A general framework for uncertainty estimation in deep learning

    Loquercio Antonio, Segu Mattia, Scaramuzza Davide. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters. 2020;5(2):3153–3160

    2020

  101. [109]

    Run-Time Assurance for Learning-Based Aircraft Taxiing

    Cofer Darren, Amundson Isaac, Sattigeri Ramachandra, et al. Run-Time Assurance for Learning-Based Aircraft Taxiing. In:2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC):1–9IEEE; 2020

    2020

  102. [110]

    Fault-Tolerant Perception for Automated Driving A Lightweight Monitoring Approach

    Buerkle Cornelius, Geissler Florian, Paulitsch Michael, Scholl Kay-Ulrich. Fault-Tolerant Perception for Automated Driving A Lightweight Monitoring Approach. arXiv preprint arXiv:2111.12360. 2021

    2021 arXiv

  103. [111]

    A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective

    Mena José, Pujol Oriol, Vitrià Jordi. A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective. ACM Computing Surveys (CSUR). 2021;54(9):1–35

    2021

  104. [112]

    A survey of uncertainty in deep neural networks

    Gawlikowski Jakob, Tassi Cedrique Rovile Njieutcheu, Ali Mohsin, et al. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342. 2021

    2021 arXiv

  105. [113]

    A Survey on Evidential Deep Learning For Single-Pass Uncertainty Estimation

    Ulmer Dennis. A Survey on Evidential Deep Learning For Single-Pass Uncertainty Estimation. arXiv preprint arXiv:2110.03051. 2021

    2021 arXiv

  106. [114]

    Evidential deep learning to quantify classification uncertainty

    Sensoy Murat, Kaplan Lance, Kandemir Melih. Evidential deep learning to quantify classification uncertainty. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems:3183–3193; 2018

    2018

  107. [115]

    Bayesian modelling in machine learning: A tutorial review.: Saarland University, Saarbruecken, Germany; 2006

    Seeger Matthias. Bayesian modelling in machine learning: A tutorial review.: Saarland University, Saarbruecken, Germany; 2006

    2006

  108. [116]

    Bayesian inference in statistical analysis

    Box George EP, Tiao George C. Bayesian inference in statistical analysis. John Wiley & Sons; 2011

    2011

  109. [117]

    A variational dirichlet framework for out-of-distribution detection

    Chen Wenhu, Shen Yilin, Jin Hongxia, Wang William. A variational dirichlet framework for out-of-distribution detection. arXiv preprint arXiv:1811.07308. 2018

    2018 arXiv

  110. [118]

    Dirichlet-based gaussian processes for large-scale calibrated classification

    Milios Dimitrios, Camoriano Raffaello, Michiardi Pietro, Rosasco Lorenzo, Filippone Maurizio. Dirichlet-based gaussian processes for large-scale calibrated classification. Advances in Neural Information Processing Systems. 2018;31

    2018

  111. [119]

    Predictive uncertainty estimation via prior networks

    Malinin Andrey, Gales Mark. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems. 2018;31

    2018

  112. [120]

    Good initializations of variational bayes for deep models

    Rossi Simone, Michiardi Pietro, Filippone Maurizio. Good initializations of variational bayes for deep models. In: International Conference on Machine Learning:5487–5497PMLR; 2019

    2019

  113. [121]

    Bayesian learning via stochastic gradient Langevin dynamics

    Welling Max, Teh Yee W. Bayesian learning via stochastic gradient Langevin dynamics. In:Proceedings of the 28th international conference on machine learning (ICML-11):681–688Citeseer; 2011

    2011

  114. [122]

    Bayesian neural networks: An introduction and survey

    Goan Ethan, Fookes Clinton. Bayesian neural networks: An introduction and survey. In: Case Studies in Applied Bayesian Data ScienceSpringer 2020 (pp. 45–87)

    2020

  115. [123]

    A review and comparative study on probabilistic object detection in autonomous driving

    Feng Di, Harakeh Ali, Waslander Steven L, Dietmayer Klaus. A review and comparative study on probabilistic object detection in autonomous driving. IEEE Transactions on Intelligent Transportation Systems.2021

    2021

  116. [124]

    Evaluating bayesian deep learning methods for semantic segmentation

    Mukhoti Jishnu, Gal Yarin. Evaluating bayesian deep learning methods for semantic segmentation. arXiv preprint arXiv:1811.12709. 2018

    2018 arXiv

  117. [125]

    Concrete dropout

    Gal Yarin, Hron Jiri, Kendall Alex. Concrete dropout. Advances in neural information processing systems. 2017;30

    2017

  118. [126]

    Incorporating Domain Knowledge into Deep Neural Networks

    Dash Tirtharaj, Chitlangia Sharad, Ahuja Aditya, Srinivasan Ashwin. Incorporating Domain Knowledge into Deep Neural Networks. arXiv preprint arXiv:2103.00180. 2021

    2021 arXiv

  119. [127]

    Semantic hierarchies for visual object recognition

    Marszalek Marcin, Schmid Cordelia. Semantic hierarchies for visual object recognition. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition:1–7IEEE; 2007. 24 Ferreira ET AL

    2007

  120. [128]

    Support vector machines with embedded reject option

    Fumera Giorgio, Roli Fabio. Support vector machines with embedded reject option. In: International Workshop on Support Vector Machines:68– 82Springer; 2002

    2002

  121. [129]

    The nearest neighbor classification rule with a reject option

    Hellman Martin E. The nearest neighbor classification rule with a reject option. IEEE Transactions on Systems Science and Cybernetics. 1970;6(3):179–185

    1970

  122. [130]

    Machine learning with a reject option: A survey

    Hendrickx Kilian, Perini Lorenzo, Plas Dries, Meert Wannes, Davis Jesse. Machine learning with a reject option: A survey. arXiv preprint arXiv:2107.11277. 2021

    2021 arXiv

  123. [131]

    Detecting adversarial examples in learning-enabled cyber-physical systems using variational autoencoder for regression

    Cai Feiyang, Li Jiani, Koutsoukos Xenofon. Detecting adversarial examples in learning-enabled cyber-physical systems using variational autoencoder for regression. In: 2020 IEEE Security and Privacy Workshops (SPW):208–214IEEE; 2020

    2020

  124. [132]

    Towards anomaly detectors that learn continuously

    Stocco Andrea, Tonella Paolo. Towards anomaly detectors that learn continuously. In:2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW):201–208IEEE; 2020

    2020

  125. [133]

    Robust out-of-distribution motion detection and localization in autonomous CPS: wip abstract

    Feng Yeli, Easwaran Arvind. Robust out-of-distribution motion detection and localization in autonomous CPS: wip abstract. In:Proceedings of the ACM/IEEE 12th International Conference on Cyber-Physical Systems:225–226; 2021

    2021

  126. [134]

    Combining pretrained CNN feature extractors to enhance clustering of complex natural images

    Guérin Joris, Thiery Stephane, Nyiri Eric, Gibaru Olivier, Boots Byron. Combining pretrained CNN feature extractors to enhance clustering of complex natural images. Neurocomputing. 2021;423:551–571

    2021

  127. [135]

    Learn from experience: probabilistic prediction of perception performance to avoid failure

    Gur˘au Corina, Rao Dushyant, Tong Chi Hay, Posner Ingmar. Learn from experience: probabilistic prediction of perception performance to avoid failure. The International Journal of Robotics Research. 2018;37(9):981–995

    2018

  128. [136]

    Introspective black box failure prediction for autonomous driving

    Kuhn Christopher B, Hofbauer Markus, Petrovic Goran, Steinbach Eckehard. Introspective black box failure prediction for autonomous driving. In: 2020 IEEE Intelligent Vehicles Symposium (IV):1907–1913IEEE; 2020

    2020

  129. [137]

    No One Representation to Rule Them All: Overlapping Features of Training Methods

    Gontijo-Lopes Raphael, Dauphin Yann, Cubuk Ekin D. No One Representation to Rule Them All: Overlapping Features of Training Methods. arXiv preprint arXiv:2110.12899. 2021

    2021 arXiv

  130. [138]

    Real-esrgan: Training real-world blind super-resolution with pure synthetic data

    Wang Xintao, Xie Liangbin, Dong Chao, Shan Ying. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision:1905–1914; 2021

    1905

  131. [139]

    ViM: Out-Of-Distribution with Virtual-logit Matching

    Wang Haoqi, Li Zhizhong, Feng Litong, Zhang Wayne. ViM: Out-Of-Distribution with Virtual-logit Matching. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition:4921–4930; 2022

    2022

  132. [140]

    Energy-based out-of-distribution detection.Advances in neural information processing systems

    Liu Weitang, Wang Xiaoyun, Owens John, Li Yixuan. Energy-based out-of-distribution detection.Advances in neural information processing systems. 2020;33:21464–21475

    2020

  133. [141]

    IVOA: Introspective vision for obstacle avoidance

    Rabiee Sadegh, Biswas Joydeep. IVOA: Introspective vision for obstacle avoidance. In:2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS):1230–1235IEEE; 2019

    2019

  134. [142]

    Machine learning: a Bayesian and optimization perspective

    Theodoridis Sergios. Machine learning: a Bayesian and optimization perspective. Academic press; 2015

    2015

  135. [143]

    Runtime Monitoring of Deep Neural Networks Using Top-Down Context Models Inspired by Predictive Processing and Dual Process Theory

    Roy Anirban, Cobb Adam, Bastian Nathaniel D, Jalaian Brian, Jha Susmit. Runtime Monitoring of Deep Neural Networks Using Top-Down Context Models Inspired by Predictive Processing and Dual Process Theory. AAAI 2022 Workshop on Designing Artificial Intelligence for Open Worlds. 2022

    2022

  136. [144]

    DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems

    Zhang Mengshi, Zhang Yuqun, Zhang Lingming, Liu Cong, Khurshid Sarfraz. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In: 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE):132– 142IEEE; 2018

    2018

  137. [145]

    Camera-IMU Extrinsic Calibration Quality Monitoring for Autonomous Ground Vehicles

    Li Binbin, Xiao Xuesu, Zhang Yulin, Li Haifeng, Wang Hongpeng. Camera-IMU Extrinsic Calibration Quality Monitoring for Autonomous Ground Vehicles. IEEE Robotics and Automation Letters. 2022

    2022

  138. [146]

    Neural Simplex Architecture

    Phan Dung, Paoletti Nicola, Grosu Radu, Jansen Nils, Smolka Scott A., Stoller Scott D.. Neural Simplex Architecture. 2019

    2019

  139. [147]

    Medical image denoising using convolutional denoising autoencoders

    Gondara Lovedeep. Medical image denoising using convolutional denoising autoencoders. In: 2016 IEEE 16th international conference on data mining workshops (ICDMW):241–246IEEE; 2016

    2016

  140. [148]

    Abdulkareem Karrar Hameed, Arbaiy Nureize, Zaidan AA, et al. A new standardisation and selection framework for real-time image dehazing algorithms from multi-foggy scenes based on fuzzy Delphi and hybrid multi-criteria decision analysis methods. Neural Computing and Applicatio...

    2021

  141. [149]

    High dynamic range imaging via gradient-aware context aggregation network

    Yan Qingsen, Gong Dong, Shi Javen Qinfeng, et al. High dynamic range imaging via gradient-aware context aggregation network. Pattern Recognition. 2022;122:108342

    2022

  142. [150]

    Dual-attention-guided network for ghost-free high dynamic range imaging.International Journal of Computer Vision

    Yan Qingsen, Gong Dong, Shi Javen Qinfeng, et al. Dual-attention-guided network for ghost-free high dynamic range imaging.International Journal of Computer Vision. 2021;:1–19

    2021

  143. [151]

    ADNet: Attention-guided deformable convolutional network for high dynamic range imaging

    Liu Zhen, Lin Wenjie, Li Xinpeng, et al. ADNet: Attention-guided deformable convolutional network for high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition:463–470; 2021. 25

    2021

  144. [152]

    Attentive generative adversarial network for raindrop removal from a single image

    Qian Rui, Tan Robby T, Yang Wenhan, Su Jiajun, Liu Jiaying. Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition:2482–2491; 2018

    2018

  145. [153]

    When awgn-based denoiser meets real noises

    Zhou Yuqian, Jiao Jianbo, Huang Haibin, et al. When awgn-based denoiser meets real noises. In:Proceedings of the AAAI Conference on Artificial Intelligence:13074–13081; 2020

    2020

  146. [154]

    Countering Adversarial Examples: Combining Input Transformation and Noisy Training

    Zhang Cheng, Gao Pan. Countering Adversarial Examples: Combining Input Transformation and Noisy Training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision:102–111; 2021

    2021

  147. [155]

    Deep learning for image super-resolution: A survey.IEEE transactions on pattern analysis and machine intelligence

    Wang Zhihao, Chen Jian, Hoi Steven CH. Deep learning for image super-resolution: A survey.IEEE transactions on pattern analysis and machine intelligence. 2020

    2020

  148. [156]

    Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

    Shi Wenzhe, Caballero Jose, Huszár Ferenc, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition:1874–1883; 2016

    2016

  149. [157]

    Photo-realistic single image super-resolution using a generative adversarial network

    Ledig Christian, Theis Lucas, Huszár Ferenc, et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition:4681–4690; 2017

    2017

  150. [158]

    Bayesian Image Reconstruction using Deep Generative Models

    Marinescu Razvan, Moyer Daniel, Golland Polina. Bayesian Image Reconstruction using Deep Generative Models. In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; 2021

    2021

  151. [159]

    Palette: Image-to-Image Diffusion Models

    Saharia Chitwan, Chan William, Chang Huiwen, et al. Palette: Image-to-Image Diffusion Models. In:NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; 2021

    2021

  152. [160]

    Color Image Demosaicing Using Progressive Collaborative Representation

    Ni Zhangkai, Ma Kai-Kuang, Zeng Huanqiang, Zhong Baojiang. Color Image Demosaicing Using Progressive Collaborative Representation. IEEE Transactions on Image Processing.2020;29:4952–4964

    2020

  153. [161]

    Gradient-Based Feature Extraction From Raw Bayer Pattern Images

    Zhou Wei, Zhang Ling, Gao Shengyu, Lou Xin. Gradient-Based Feature Extraction From Raw Bayer Pattern Images. IEEE Transactions on Image Processing. 2021;30:5122–5137

    2021

  154. [162]

    Class Retrieval of Adversarial Attacks

    Al-Afandi Jalal, Horváth András. Class Retrieval of Adversarial Attacks. Workshop on Adversarial Machine Learning in Real-World Computer Vision Systems and Online Challenges (AML-CV). 2021

    2021

  155. [163]

    On the Reversibility of Adversarial Attacks

    Li Chau Yi, Sánchez-Matilla Ricardo, Shamsabadi Ali Shahin, Mazzon Riccardo, Cavallaro Andrea. On the Reversibility of Adversarial Attacks. In: 2021 IEEE International Conference on Image Processing (ICIP):3073–3077IEEE; 2021

    2021

  156. [164]

    A survey on fault tolerance techniques in wireless sensor networks

    Kakamanshadi Gholamreza, Gupta Savita, Singh Sukhwinder. A survey on fault tolerance techniques in wireless sensor networks. In: 2015 international conference on green computing and internet of things (ICGCIoT):168–173IEEE; 2015

    2015

  157. [165]

    Tesla autonomous driving talk at CVPR 2021

    Andrej Karpathy . Tesla autonomous driving talk at CVPR 2021. Online; accessed 07 June 2022; 2021-06

    2021

  158. [166]

    Three scenarios for continual learning

    Ven Gido M, Tolias Andreas S. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734. 2019

    1904 arXiv

  159. [167]

    Adaptive aggregation networks for class-incremental learning

    Liu Yaoyao, Schiele Bernt, Sun Qianru. Adaptive aggregation networks for class-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition:2544–2553; 2021

    2021

  160. [168]

    The MNIST database of handwritten digit images for machine learning research [best of the web]

    Deng Li. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine. 2012;29(6):141–142

    2012

  161. [169]

    Learning multiple layers of features from tiny images

    Krizhevsky Alex, Hinton Geoffrey, others . Learning multiple layers of features from tiny images. : University of Toronto; 2009

    2009

  162. [170]

    A less biased evaluation of out-of-distribution sample detectors

    Shafaei Alireza, Schmidt Mark, Little James J. A less biased evaluation of out-of-distribution sample detectors. 30th British Machine Vision Conference, Cardiff, Wales.2019

    2019

  163. [171]

    CARLA: An Open Urban Driving Simulator

    Dosovitskiy Alexey, Ros German, Codevilla Felipe, Lopez Antonio, Koltun Vladlen. CARLA: An Open Urban Driving Simulator. In:Proceedings of the 1st Annual Conference on Robot Learning:1–16; 2017

    2017

  164. [172]

    Comparing offline and online testing of deep neural networks: An autonomous car case study

    Haq Fitash Ul, Shin Donghwan, Nejati Shiva, Briand Lionel C. Comparing offline and online testing of deep neural networks: An autonomous car case study. In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST):85–95IEEE; 2020

    2020

  165. [173]

    Unifying evaluation of machine learning safety monitors

    Guerin Joris, Ferreira Raul Sena, Delmas Kevin, Guiochet Jérémie. Unifying evaluation of machine learning safety monitors. In: 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE):414–422IEEE; 2022

    2022

  166. [174]

    Evaluating the simulation gap for training off-road self-driving

    Zandbergen Lars. Evaluating the simulation gap for training off-road self-driving. PhD thesisMaster’s thesis, University of Amsterdam, Amsterdam, The Netherlands2021

  167. [175]

    Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry

    Borg Markus, Englund Cristofer, Wnuk Krzysztof, et al. Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry. Journal of Automotive Software Engineering. 2019;1(1):1–19

    2019

  168. [176]

    Oracle problem in software testing

    Jahangirova Gunel. Oracle problem in software testing. In: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis:444–447; 2017. 26 Ferreira ET AL

    2017

  169. [177]

    Failing loudly: An empirical study of methods for detecting dataset shift

    Rabanser Stephan, Günnemann Stephan, Lipton Zachary. Failing loudly: An empirical study of methods for detecting dataset shift. Advances in Neural Information Processing Systems. 2019;32

    2019

  170. [178]

    Pass-fail criteria for scenario-based testing of automated driving systems

    Myers Robert, Saigol Zeyn. Pass-fail criteria for scenario-based testing of automated driving systems. arXiv preprint arXiv:2005.09417. 2020

    2005 arXiv

  171. [179]

    Procedure for the Safety Assessment of an Autonomous Vehicle Using Real-World Scenarios

    De Gelder Erwin, Den Camp Olaf Op. Procedure for the Safety Assessment of an Autonomous Vehicle Using Real-World Scenarios. arXiv preprint arXiv:2012.00643. 2020

    2012 arXiv

  172. [180]

    Risk Quantification for Automated Driving Systems in Real-World Driving Scenarios.IEEE Access

    De Gelder Erwin, Elrofai Hala, Saberi Arash Khabbaz, Paardekooper Jan-Pieter, Den Camp Olaf Op, De Schutter Bart. Risk Quantification for Automated Driving Systems in Real-World Driving Scenarios.IEEE Access. 2021

    2021

  173. [181]

    JARUS guidelines on Specific Operations Risk Assessment (SORA) v2.0

    Joint Authorities for Rulemaking of Unmanned Systems (JARUS) . JARUS guidelines on Specific Operations Risk Assessment (SORA) v2.0. Guidelines: EASA; 2019

    2019

  174. [182]

    Phase Plane-based Approaches for Event Detection and Plausibility Check of Vehicle Dynamics

    Kontos János, Vathy-Fogarassy Ágnes, Kránicz Balázs. Phase Plane-based Approaches for Event Detection and Plausibility Check of Vehicle Dynamics. In: 2021 IEEE 25th International Conference on Intelligent Engineering Systems (INES):000031–000036IEEE; 2021

    2021

  175. [183]

    Model assertions for monitoring and improving ML models

    Kang Daniel, Raghavan Deepti, Bailis Peter, Zaharia Matei. Model assertions for monitoring and improving ML models. Proceedings of Machine Learning and Systems. 2020;2:481–496

    2020

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.