{"id":"9ecca3f8-ca79-42c0-8251-d5a07e34f482","arxiv_id":"2412.06869","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that organizes research on runtime safety monitors for ML perception into threat identification, requirements, detection, reaction, and evaluation, and lists open challenges.","lead":"This paper surveys how to build runtime safety monitors for machine-learning perception in autonomous systems, organizing the literature around five design questions: threats, requirements, detection, reaction, and evaluation. It is a structured review for safety engineers and ML researchers, not a new method, and it highlights open challenges like standardized benchmarks and certification.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The survey's claim that OOD-based monitors fail to detect actual model errors rests mainly on one self-cited benchmark; if that benchmark is protocol-dependent, the paper's central 'out-of-model-scope' research agenda is weakened.","rationale":"The reader correctly identified the absence of a documented systematic search as a transparency weakness. I agree that this is a limitation, but I do not see it as the most load-bearing concern: the survey's central value is its top-down organization, and a broader corpus would refine but not overturn that structure. The stronger concern is that the survey's distinctive empirical conclusion, that OOD-focused monitoring is misaligned with actual safety-relevant errors, rests on a single self-cited benchmark whose protocol is not described in the survey. This claim is used to justify the OMS evaluation paradigm and one of the six open challenges. If the benchmark's labeling or detector selection was favorable to the authors' position, the contribution would lose some force. However, this is a narrow empirical anchor within an otherwise comprehensive review, and the taxonomy, threat definitions, and detection-mechanism classification stand independently. The paper itself acknowledges that the evidence on detector efficacy is inconclusive (Section 5), which limits the strength of any recommendation based on [37]. A verdict of ACCEPT therefore remains appropriate, with the caveat that the authors should ideally disclose or cite the full protocol behind their key negative result.","tokens_in":25595,"tokens_out":2851,"duration_ms":34375,"concrete_test":"Independently reproduce the Ferreira et al. benchmark with both labeling paradigms explicitly separated: (1) OOD labels based on input-distribution shift (e.g., CIFAR-10 vs. a disjoint-label dataset, corrupted/shifted variants), and (2) OMS labels based on actual model misclassification on the same test inputs. Evaluate a representative sample of detectors from the survey's Section 5 categories (softmax confidence, ODIN, Mahalanobis distance, MC Dropout, autoencoder reconstruction, an ensemble method) using identical AUROC and TPR@95TNR metrics under both labelings. If OOD detectors perform comparably under OMS labeling, the survey's 'unsatisfactory results' claim is protocol-dependent; if a large gap persists, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey is a useful top-down map, and most of its taxonomy is organizational rather than empirical. The load-bearing empirical claim appears in Section 5 and Section 8: that 'most detection mechanisms based on out-of-distribution detection suffer from a high number of false positives and false negatives when considering their ability to detect a failure of the ML model' (Section 8, citing Ferreira et al. 37). This result is used to motivate the paper's central contribution over earlier bottom-up surveys: the shift from OOD detection to out-of-model-scope (OMS) evaluation (Section 7.1.2), and the open challenge that monitor requirements are misaligned with system-level safety objectives. The survey does not report the evaluation protocol, datasets, metrics, or detector selection of [37], and it cites the authors' own related work [29] as the basis for this position. If [37] evaluated detectors under an OMS labeling scheme that differs from what detectors were designed for, or if the selected detectors and threat distributions were not representative, the negative result may not generalize. Because this empirical claim is what distinguishes the survey's framing from prior work, it is load-bearing. That said, this is one empirical assertion within an otherwise valid organizational survey; the five-factor structure and the mapping of the literature remain useful even if this particular result is later qualified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a survey of runtime safety monitoring for ML-based perception functions in safety-critical autonomous systems. It organizes the literature around five design questions: threat identification (Section 3), requirements elicitation (Section 4), failure detection (Section 5), reaction/recovery (Section 6), and evaluation (Section 7). The detection taxonomy distinguishes internal mechanisms (uncertainty estimation, domain knowledge, learning with rejection) from external mechanisms (input, internal-representation, output, and multi-sensor monitoring). The survey also advances a research agenda centered on out-of-model-scope (OMS) evaluation rather than pure out-of-distribution (OOD) detection, and concludes with open challenges including standardized benchmarking and certification. The principal contribution is a top-down map of the field that covers aspects (requirements, reaction, system-level evaluation) that are rare in prior bottom-up surveys.","tokens_in":25787,"tokens_out":3170,"duration_ms":34628,"significance":"If the survey's organizational structure is adopted, it provides a useful reference map for both safety practitioners and ML researchers, and it identifies several genuine gaps: the scarcity of systematic requirements-to-monitor derivation, the underdevelopment of recovery mechanisms beyond basic alerts, and the absence of standardized evaluation benchmarks. The paper is careful in many places, explicitly acknowledging that evidence on detector efficacy is conflicting (Section 5) and that evaluation protocols vary widely (Section 7). Its strongest assets are the breadth of the detection taxonomy, the clear separation of runtime threat types, and the concrete discussion of evaluation metrics and system-level safety gain. The main caveats are that the survey does not document a systematic search protocol and that its central OOD-to-OMS critique rests heavily on a single self-cited benchmark whose protocol is not described in the paper.","major_comments":[{"comment":"The claim that \"most detection mechanisms based on out-of-distribution detection suffer from a high number of false positives and false negatives\" when detecting actual ML model failures is load-bearing: it motivates the OMS evaluation paradigm in Section 7.1.2 and appears as the first open challenge in Section 8. Yet the only evidence cited is Ferreira et al. [37], and the survey does not report that benchmark's datasets, detector selection, threat distributions, labeling scheme, or metrics. Since [37] is the authors' own prior work and the survey itself notes that the broader literature gives conflicting results (Section 5), the claim should either be accompanied by a concise description of the benchmark protocol and its limitations, or be explicitly presented as a preliminary finding from a single study rather than a general negative result.","section":"Section 8 and Section 7.1.2"},{"comment":"The survey describes itself as an \"extensive literature review\" and claims that the top-down approach allows it to \"uncover specific areas where research is lacking,\" but it provides no search protocol, inclusion/exclusion criteria, or coverage statistics. The corpus appears to have been assembled through expert judgment, which is a legitimate method for a survey, but without any documentation of the selection process the representativeness of the coverage cannot be independently verified. Please add a short methodology paragraph describing how references were identified and screened, or temper the extensiveness claim accordingly.","section":"Section 2"}],"minor_comments":[{"comment":"The reference lists inside the taxonomy boxes (e.g., \"67,68,69,11\") are not consistently sorted and some references appear in multiple boxes without explanation; consider reordering and adding a note on whether multiple appearances indicate multi-purpose mechanisms.","section":"Figure 3"},{"comment":"The distinction between semantic shift and covariate shift is clear conceptually, but the statement that semantic shifts \"cannot be handled with denoising or backup sensors\" (Section 3.3.2) could be read as contradicting the later discussion of input reconstruction and alternative sensors in Section 6.2; a cross-reference clarifying the scope of that statement would help.","section":"Section 3.3.2"},{"comment":"The metrics list is useful, but the definition of P@80R (precision at recall 0.8) is placed after AUPR without an explicit formula; a one-line formal definition would improve precision.","section":"Section 7.1.2"},{"comment":"A few references have inconsistent metadata, e.g., [174] lists a Master's thesis in the venue field and [13] is missing page numbers; a final reference cleanup pass is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The OOD/OMS critique is an important differentiator for the survey, but its empirical support comes almost entirely from the authors' own benchmark and position paper. This is not inappropriate per se, but it does mean the survey's research agenda is more advocacy-oriented than a neutral synthesis would be. The requested additions (benchmark protocol summary, search methodology) should be feasible without changing the paper's structure. The survey's organizational contribution is solid enough that I would not reject it, but the load-bearing empirical claim needs to be either substantiated or appropriately qualified before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It's a survey, so no new mechanism, but it's a good one. The five-question structure—threats, requirements, detection, reaction, evaluation—is a real improvement over the bottom-up detector taxonomies you usually see. The internal/external detector split is also more useful than the threat-based split. Each section delivers what it promises, and the open challenges list in Section 8 is concrete enough to seed a research agenda.\n\nCredit where due: the coverage is broad and current, and they're honest about the evidence being inconclusive. Section 5 explicitly says the literature doesn't support definitive conclusions about detector efficacy. That honesty extends to their own position: the OMS-vs-OOD framing is presented as an argument, not a settled result.\n\nThe biggest soft spot is exactly what the stress-test note flags: the claim that OOD-based monitors fail at detecting actual model errors, used to motivate the OMS agenda, traces back to one self-cited benchmark, Ferreira et al. 37. The survey doesn't give the protocol or dataset details for that benchmark, so the negative result isn't independently checkable. But I don't think this is load-bearing. The survey's contribution is the map and the taxonomy; the OMS recommendation is a position held by the authors, and they themselves note the evidence is inconclusive. If that benchmark later fails to generalize, the survey's organizational value survives. What's more problematic is the absence of a documented search protocol. For a survey claiming 'extensive literature review', there's no inclusion criteria or coverage statistics, so you can't verify representativeness. That's a transparency issue and should be fixed before publication.\n\nWho it's for: practitioners entering this area, safety engineers wanting a structured view, and anyone building benchmarks. I'd send it to a serious referee—surveys this well-organized deserve peer review even if the empirical claim gets challenged. My own verdict is accept with minor revisions, mostly around the search protocol and a bit more context on [37]. I'd bring it to a reading group as a shared frame for discussion.","headline":"A genuinely useful top-down map of ML safety monitoring; the OOD-vs-OMS empirical claim leans on the authors' own benchmark, but the survey's value doesn't rest on it.","tokens_in":26371,"tokens_out":2231,"would_cite":true,"duration_ms":23220,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey argues that safety monitoring for machine-learning perception is best designed around five factors—threats, requirements, detection, reaction, evaluation—and that current research overweights out-of-distribution detection at…","keywords":["safety monitoring","machine learning perception","runtime monitoring","fault tolerance","safety-critical autonomous systems","out-of-distribution detection","detection mechanisms","recovery mechanisms"],"falsifier":"Run the detection mechanisms surveyed here on a labeled corpus of genuine perception errors, not just out-of-distribution inputs, and observe them catching most errors at low false-positive rates; that outcome would refute the paper's central warning that out-of-distribution-centric monitors are misaligned with error detection.","tokens_in":25343,"feed_emoji":"🛡️","tokens_out":6605,"duration_ms":58495,"temperature":0.7,"pith_summary":"The paper tries to organize the scattered literature on runtime safety monitors for machine-learning perception systems used in critical applications such as self-driving cars and surgical robots. It argues that building a monitor is not just an error-detection problem, and structures the field around five questions: what threats to guard against, how to derive monitor requirements from safety objectives, how to detect failures, how to react, and how to evaluate. Its pointed central claim is that the dominant focus on out-of-distribution (OOD) detection is misplaced, because OOD-ness is ambiguous and monitors should instead be judged by whether they catch incorrect predictions. If true, this gives safety practitioners a checklist and a research agenda, and points to under-studied areas such as requirement elicitation, recovery actions, and standardized evaluation.","feed_headline":"ML perception monitors: five design choices, not just error detection","feed_subtitle":"A map of the field that argues monitors should catch wrong predictions, not just out-of-distribution inputs.","key_machinery":"The paper's central organizing device is a two-level taxonomy of safety-monitor design. At the top level are five questions every monitor must answer: which threats, which requirements, which detection mechanism, which reaction, and how to evaluate. At the detection level, it distinguishes internal mechanisms (built into the ML model, such as uncertainty estimation and rejection learning) from external mechanisms (independent components that watch the model's input, internal layer activations, output, or other sensors). The taxonomy carries the argument by turning 'safety monitoring' from a single technique into a design space, and it is what lets the authors argue that most surveys cover only the detection cell of this space.","core_discovery":"The paper's central discovery is that the safety-monitoring literature for ML perception is best understood top-down through five design factors—threat identification, requirements elicitation, error detection, reaction, and evaluation—rather than bottom-up through detection techniques. It classifies detection mechanisms as internal (uncertainty estimation, domain-knowledge integration, learning with rejection) or external (monitoring inputs, internal activations, outputs, or external sensors), and advocates replacing the OOD-detection framing with an out-of-model-scope framing in which monitors are evaluated by their ability to detect actual prediction errors. It also gathers the comparatively sparse literature on recovery mechanisms and evaluation protocols, and concludes that the biggest open challenges are aligning monitor objectives with system-level safety analysis, combining monitors with plausibility checks, meeting embedded implementation constraints, and standardizing benchmarks.","pith_inferences":["If the out-of-model-scope framing is right, then widely used OOD benchmark suites should be complemented by 'error benchmarks' that record whether the deployed model is actually wrong, which would make many published detection results non-comparable to the monitor's real purpose.","The five-factor structure could be instantiated as a reusable design checklist for safety practitioners, effectively turning the survey into a template for deriving monitor requirements from a hazard analysis.","As detection mechanisms mature, the practical bottleneck should shift to the reaction step: testable engineering work could pair this detection taxonomy with control-theoretic safety envelopes to measure end-to-end safety contribution, not just detection quality.","The internal/external distinction suggests a certification hypothesis worth testing: external monitors, being separable from the ML model, may be easier to verify and certify than internal mechanisms, which would make external monitoring attractive even when it is less accurate."],"forward_implications":["Evaluation of monitors should shift its target from out-of-distribution detection to the detection of actual model errors, with test sets labeled by whether the model's prediction is wrong.","Requirement elicitation becomes a first-class research problem: deriving monitor specifications from system-level hazard analysis (for instance, identifying state-space regions where an ML failure is safety-critical) is a promising direction the survey identifies.","Practical monitors will likely combine several mechanisms—data-driven detectors alongside plausibility checks, model assertions, and failure-mode analysis—because no single category covers all threats.","Recovery mechanisms need more attention than they currently receive: control switching, input enhancement, and alternative perception components are underdeveloped relative to detection.","Progress toward certifying ML-based perception will depend on unified benchmarks and metrics that reflect the full monitor lifecycle, not just detection accuracy."],"supporting_citations":[{"why":"Defines the safety-monitoring framework (SMOF) that the survey uses to frame monitors as components checking safety properties and triggering corrective actions.","marker":"[13]"},{"why":"The most closely related prior survey of runtime failure detection for ML robotic perception; the survey positions its own top-down structure against this bottom-up, detection-only taxonomy.","marker":"[27]"},{"why":"Prior survey of ML safety in autonomous vehicles that classifies fail-safe mechanisms; represents the detection-centric approach the paper argues is incomplete.","marker":"[10]"},{"why":"Survey on learning to reject; a reference point for the detection-only coverage of monitoring in existing literature.","marker":"[28]"},{"why":"The authors' own argument that out-of-distribution detection is not sufficient, which supplies the survey's central critical claim that monitors should target incorrect predictions.","marker":"[29]"},{"why":"Benchmark study showing that OOD detection mechanisms have high false-positive and false-negative rates when used to detect actual model failures; the key evidence for the misalignment claim.","marker":"[37]"},{"why":"Baseline softmax-confidence detection method that anchors the survey's category of output-monitoring mechanisms.","marker":"[78]"},{"why":"Monte-Carlo Dropout, the uncertainty-estimation technique that anchors the survey's category of internal detection mechanisms.","marker":"[11]"}],"fun_headline_variants":["ML perception monitors: five design factors, not just OOD","Safety monitors for ML perception: beyond out-of-distribution","Perception monitors: catch prediction errors, not just OOD inputs","Survey maps ML perception monitors via five design axes","Five factors for designing ML perception safety monitors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's coverage is representative because the authors chose the corpus by expert judgment without documenting a search protocol or inclusion criteria, so undetected gaps in the literature would make the claimed open challenges incomplete.","fun_headline_variants_meta":{"raw":{"variants":["ML perception monitors: five design factors, not just OOD","Safety monitors for ML perception: beyond out-of-distribution","Perception monitors: catch prediction errors, not just OOD inputs","Survey maps ML perception monitors via five design axes","Five factors for designing ML perception safety monitors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1119,"prompt_tokens":825,"completion_tokens":294,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":215}},"tokens_in":441,"tokens_out":294,"duration_ms":3650,"temperature":1.0,"reasoning_tokens":215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:42:33.450124+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the detection mechanisms surveyed here on a labeled corpus of genuine perception errors, not just out-of-distribution inputs, and observe them catching most errors at low false-positive rates; that outcome would refute the paper's central warning that out-of-distribution-centric monitors are misaligned with error detection.","supporting_citations":[],"review_version":1}