Robustness Requirement Coverage using a Situation Coverage Approach for Vision-based AI Systems

Boda Li; Nawshin Mannan Proma; Richard Hawkins; Sepeedeh Shahbeigi; Valentina Donzella; Victoria Hodge

REVIEW 3 major objections 5 minor 12 references

Robustness Requirement Coverage using a Situation Coverage Approach for Vision-based AI Systems

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that robustness safety requirements for camera-based AI perception should be written as performance requirements qualified by bounded operational contexts (PODs) generated from a situational grid over camera noise factors.

desk verdict Clear position paper, new integration, but completeness claim overreaches and the illustrative grid violates the paper's own dependencies. read the letter →

arxiv 2507.12986 v1 pith:EWA5PZZU submitted 2025-07-17 cs.RO

Sepeedeh Shahbeigi , Nawshin Mannan Proma , Victoria Hodge , Richard Hawkins , Boda Li , Valentina Donzella This is my paper

classification cs.RO

keywords requirementselicitationrobustnessnoisefactorsidentificationsituationcoverageanalysiscameradegradationAIperceptionautomotivesafetyoperationaldesigndomain

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Faced with an effectively infinite space of camera-degraded images, the paper proposes to compress that space into a finite grid of representative degraded conditions and to write robustness requirements against the grid. It adopts an existing P-Diagram taxonomy of thirty automotive camera noise factors, filters those that affect pixel intensity or position, assigns discrete states and dependency rules through expert input, and then builds situational coverage grids for the five noise-factor types. Each row of these grids is treated as a bounded operational context (a POD), and the paper claims that the situational grid 'defines part of the operational contexts' in which robustness safety requirements will be derived, with the union of all PODs intended to span the complete intended operational context. If this works, safety cases can express robustness as a set of concrete, testable slices rather than an unmanageable 'all conditions' promise. Because the paper is a position paper, the expert inputs and grid counts are illustrative rather than validated.

What carries the argument

The central object is the situational coverage grid over the noise factor hierarchy Type, Noise Factor, State, where each cell is a distinct subset of operating context. The argument runs through three stages: a noise factor table with discretized states and dependency rules, the enumeration of all feasible state combinations per factor type into grids of size $n_1 = 72$, $n_2 = 36$, $n_3 = 108$, $n_4 = 12$, and $n_5 = 24$, and the conversion of each grid row into a bounded operational context POD. The POD is the load-bearing object: it turns 'robustness' into a named, enumerable set of degraded conditions that can qualify a performance requirement, making the safety requirement modular and traceable.

What would settle it

Check the Usage-type grid in Table II against the dependency rules in Table I: if a row such as the one with Vehicle Dynamic Settings = True and Misplacement = False appears despite Table I declaring a co-occurrence dependency, the pruning step claimed in Section III-B has not actually been applied, and the 108-row count cannot serve as a faithful representation of feasible operational contexts.

Watch

Extended reading notes

Core claim

The central claim of the paper is that robustness safety requirements for camera-based AI perception are best formulated as performance requirements qualified by explicitly bounded operational contexts, generated by applying situation coverage analysis to a camera noise factor table. The framework starts from thirty expert-curated noise factors grouped under piece-to-piece variation, change over time, usage, environment, and system interactions, keeps only those that change pixel intensity or pixel position, and discretizes each factor into a small number of operational states with co-occurrence dependencies. Stage 2 then enumerates every consistent combination of states within each factor type, producing pruned grids whose rows are called PODs; Stage 3 attaches each POD as the operational-context qualifier to a performance requirement, yielding safety requirements of the form performance $\mathrm{SR} + [\mathrm{PODs}]$. The paper's completeness statement is that the union of all PODs should span the complete intended operational context. This recasts robustness from an implicit promise over all possible degraded inputs into an explicit, inspectable envelope defined by the grid.

Load-bearing premise

The completeness of the POD envelope rests on the expert-supplied noise factor list, the chosen discrete states, and the dependency rules faithfully capturing every degradation mode that actually harms the AI perception; if any of these is wrong or incomplete, the resulting grid silently omits safety-relevant contexts.

Editorial extensions

If this is right

Each POD can be assessed and tested in isolation, so a requirement failure is traceable to a specific degradation context rather than to a vague 'all conditions' claim.
Completeness becomes an explicit, inspectable property of the requirement set: if the union of PODs does not span the intended operational context, the gap is visible as a missing row or type.
The combinatorial explosion of degraded scenarios is contained: the usage type alone is reduced to 108 representative situations, a finite set that can be enumerated in a coverage table.
Safety cases built on the machine-learning assurance guidance in [5] can cite POD-qualified requirements directly, giving the safety argument a concrete operational envelope.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves the state boundaries (e.g., what counts as 'low' versus 'high' scratch severity) to experts; a natural extension is sensitivity analysis showing how the POD set changes when those boundaries shift.
The same POD mechanism should transfer to other perception sensors such as lidar or radar, since the five P-Diagram categories are sensor-general and only the noise-factor table would need re-population.
If adopted, a safety case could make a stronger completeness argument by claiming coverage of every POD in the grid, then defending the taxonomy itself; this is a more auditable claim than the traditional 'all conditions' phrasing.
The POD structure closely resembles an operational design domain decomposition, so the framework could slot directly into ODD-based safety arguments in automated vehicle standards.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The useful idea here is to pair Li et al.'s P-Diagram noise-factor taxonomy with situation coverage, and use the combination to derive robustness requirements for camera-based AI perception. That integration is genuinely new in the cited literature, and the three-stage structure—noise-factor table, situational grids, then requirements expressed as performance SRs plus bounded operational contexts (PODs)—is clearly described. The paper is honest about being a position paper and about using expert input illustratively.

The main soft spot is the completeness claim. Section III-C states that the union of all PODs should span the complete intended operational context. But Stage 1 deliberately restricts factors to those affecting pixel intensity or pixel position, excluding frame-rate and dropped-frame effects, and Section II admits the underlying taxonomy omits corner cases like ghosting. So the union of PODs derived from this reduced factor set cannot logically span the full intended operational context unless that context is defined as just the camera-degradation subspace. The paper does not define it that way, and offers no argument that the excluded factors are covered elsewhere. That is a load-bearing gap, not a missing validation.

The second issue is internal: the illustrative Usage grid in Table II contradicts the dependency column of Table I. Row 2 shows Dynamics=True with Misplacement=False, despite the declared co-occurrence dependency. Row 3 shows Lens Scratch=Low with Obstructions=False, again violating the stated co-occurrence. If these are representative rows of the generated grid, the claimed pruning of invalid combinations is not demonstrated, and the stated grid sizes are not credible as presented. This is fixable, but it should not have shipped in this form.

On the plus side, the writing is direct, the related work is appropriate, and the idea of qualifying performance requirements with bounded operational contexts is practically useful for safety cases. The paper would benefit from a case study, but as a position paper its job is to set up the framework.

My take: this deserves a serious referee, but with major revision. The authors need to either weaken the completeness claim or show how the full operational context is covered by other parts of the safety case, and they need to fix the example and state the pruning rule explicitly. I'd bring it to a reading group as a discussion piece, but I wouldn't cite it in its current form.

Referee Report

3 major / 5 minor

Summary. The paper proposes a three-stage framework for deriving robustness safety requirements for camera-based AI perception systems. Stage 1 builds an updated noise-factor table from the P-Diagram taxonomy of Li et al., adding expert-informed discrete states and dependencies. Stage 2 generates situational coverage grids by enumerating feasible combinations of states per factor type. Stage 3 turns each grid row into a bounded operational context, expressing safety requirements as performance requirements qualified by PODs, with the claim that the union of all PODs should span the complete intended operational context. The work is positioned as an initial, illustrative step rather than a validated method, and it explicitly builds on the authors' prior situation coverage work and Li et al.'s noise-factor analysis.

Significance. If the framework's completeness claim held, it would offer a tractable and traceable bridge between sensor degradation analysis and ML safety requirements, a genuinely useful contribution to safety cases for autonomous systems. The strengths are the clear three-stage decomposition, the reuse of established taxonomies, the explicit formulation of requirements as performance SRs plus PODs, and the reproducible combinatorial arithmetic for the grid sizes. The paper also honestly discloses that the expert inputs are illustrative and that corner cases such as surface-reflection ghosting are excluded. However, the significance is currently limited by an internal mismatch between the scoped construction of the grids and the unscoped completeness claim, and by the inconsistency between the stated dependencies and the example rows in Table II.

major comments (3)

[Section III-B, Tables I and II] The claimed pruned grid size for the Usage type, n3 = 108, is inconsistent with the dependencies listed in Table I. Without pruning, the Usage factors give 2 x 2 x 2 x 3 x 3 x 2 = 144 combinations. Enforcing only the dependency 'Vehicle Impact Co-Occurrence with Misplacement' removes 36 combinations and yields 108, which matches n3. However, Table I also states 'Lens Scratch Co-occurrence with Obstructions' and 'Vehicle Dynamic Settings Co-occurrence with Misplacement'; enforcing either of those dependencies would reduce the count below 108. Moreover, Table II rows 2 and 3 explicitly show Dynamics = True with Misplacement = False, and Lens Scratch = Low with Obstructions = False, both of which violate the stated co-occurrence dependencies. The pruning claim is therefore not supported by the paper's own example, and the combinatorial arithmetic needs to be recomputed or the dependency semantics (e.g., one-way vs. two-way co-occurrence) must be clarified.
[Section III-C, with Sections II and III-A] The central completeness claim is not derivable from the framework's construction. Stage 3 states that 'the union of all PODs should span the complete intended operational context,' but Stage 2 defines the situational grid as covering only 'part of the operational contexts, associate with sensor degradation,' and Stage 1 restricts the factor set to those affecting pixel intensity or pixel position, explicitly excluding factors that affect only frame rate or dropped frames. Section II additionally admits that the adopted taxonomy omits corner cases such as surface-reflection ghosting. Even if every retained factor and every discretized state were correct, the union of PODs generated from this reduced factor set can only span the camera-degradation subspace, not the complete intended operational context as stated. The authors should either restrict the completeness claim to that subspace and redefine the intended operational context accordingly, or provide an argument or evidence that the excluded factors cannot affect the safety-relevant robustness requirements.
[Section III-A] The discretization thresholds and dependency rules are the mechanism that makes the infinite degradation space tractable, but they are presented as assumed illustrative inputs rather than as validated knowledge. The text says that factor relevance, states, and dependencies are 'informed by expert input, empirical data, and the operational context specifications,' yet the example uses assumed inputs and no validation protocol is described. Because the completeness of the resulting PODs depends entirely on these inputs, the paper should at least sketch how the thresholds and dependencies would be validated in practice, for example through sensitivity analysis, data collection, or a structured expert elicitation protocol. Absent that, the completeness assessment remains heuristic rather than principled.

minor comments (5)

[Section III-B] The sentence 'The resulting situational grid defines part of the operational contexts, associate with sensor degradation' contains a grammatical error and an apparent missing word; it should read something like 'associated with sensor degradation.'
[Table II] The table shows '...' for rows 4 through 107, but if the dependency constraints in Table I are enforced, the row count and row IDs would change; the table should be updated to be consistent with the pruned enumeration.
[Section III-C] The notation is inconsistent: the RQ example uses '[PODs#1]' while the following paragraph uses '[POD#1]'. The authors should settle on a single notation and define precisely how a POD is formed when the overall operational context spans multiple type grids.
[References] Reference [4] contains a formatting artifact in 'Accepted for , 44th International Conference...' and reference [5] has an incomplete journal-style citation; these should be cleaned up.
[Throughout] There are several typographical glitches, including 'I NTRODUCTION' and 'T HE G RID COVERAGE-BASED FRAMEWORK' section headings with extra spaces, and inconsistent author initials such as 'V . Donzella' in the references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework explicitly integrates prior taxonomies and a coverage method into a new requirements-elicitation procedure, with no fitted parameter renamed as a prediction and no derivation that assumes its own conclusion.

full rationale

The paper is a position paper that combines two prior strands of work – the camera noise-factor taxonomy of Li et al. [3] and the situation-coverage framework of Proma et al. [4] – into a proposed procedure for eliciting robustness requirements. The core derivation is the generation of situational grids from Table I's explicit factor states and dependency rules, followed by the illustrative formulation of a performance requirement qualified by a POD. No quantity is fitted to data and then reported as a prediction; the expert inputs in Table I are explicitly presented as illustrative assumptions ('expert input is used illustratively to highlight how this process would be conducted in practice'). The paper does make a completeness-style assertion – 'The union of all PODs should span the complete intended operational context' – but this is an aspiration stated in Stage 3, not a result derived from the framework's inputs. In fact, the paper itself acknowledges that Stage 2's grid 'defines part of the operational contexts' and that the adopted taxonomy omits corner cases such as surface-reflection ghosting, which weakens the completeness claim but does not make the derivation circular. The self-citations to [3] and [4] are load-bearing in the straightforward sense that the paper builds on them, which is normal and explicit; there is no hidden reliance on an unverified self-cited 'uniqueness theorem' or a renaming of a known result as a new one. Therefore, the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework's output is a direct consequence of its inputs: the set of 30 noise factors, their discretized states, and the dependency rules are all assumed or adopted rather than derived, so the completeness and utility of the resulting requirements inherit the validity of those inputs.

free parameters (3)

state discretization thresholds (e.g., Low/High for Obstructions, Lens Scratch, Ageing of Electronics)
Each noise factor is discretized manually into binary or categorical states informed by expert input and operational context specifications (Section III-A); the resulting grid sizes and POD coverage depend on these thresholds, which are not derived from data.
set of 30 noise factors = 30
The taxonomy from Li et al. [3] is adopted wholesale; completeness of this set is an input, not a result.
dependency rules (co-occurrence)
Co-occurrence relations in Table I are chosen by domain experts (e.g., 'obstructions and lens scratches often co-occur') and are not derived or validated; they directly determine which grid cells are pruned.

assumptions (4)

domain assumption The 30-factor P-Diagram taxonomy from Li et al. [3] covers all relevant camera noise factors for AI perception.
Adopted in Section II; the paper notes corner cases such as ghosting are excluded.
domain assumption Noise factors that only affect frame rate or dropped frames can be excluded because they do not directly affect spatial or visual content.
Section III-A; this exclusion shapes the noise factor table.
ad hoc to paper Expert-determined co-occurrence and mutual exclusivity dependencies capture real-world correlations.
Section III-A states dependencies 'usually depend on domain assumptions ... determined in close consultation with domain experts'; Table I implements them ad hoc.
ad hoc to paper The union of all PODs spans the complete intended operational context.
Section III-C asserts this without a verification method.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Robustness Requirement Coverage using a Situation Coverage Approach for Vision-based AI Systems." pith.science (2026). https://pith.science/paper/EWA5PZZU

@misc{pith2026250712986,
  author       = {Pith},
  title        = {Pith review of: Robustness Requirement Coverage using a Situation Coverage Approach for Vision-based AI Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EWA5PZZU}},
  note         = {Machine review of arXiv:2507.12986}
}

read the original abstract

AI-based robots and vehicles are expected to operate safely in complex and dynamic environments, even in the presence of component degradation. In such systems, perception relies on sensors such as cameras to capture environmental data, which is then processed by AI models to support decision-making. However, degradation in sensor performance directly impacts input data quality and can impair AI inference. Specifying safety requirements for all possible sensor degradation scenarios leads to unmanageable complexity and inevitable gaps. In this position paper, we present a novel framework that integrates camera noise factor identification with situation coverage analysis to systematically elicit robustness-related safety requirements for AI-based perception systems. We focus specifically on camera degradation in the automotive domain. Building on an existing framework for identifying degradation modes, we propose involving domain, sensor, and safety experts, and incorporating Operational Design Domain specifications to extend the degradation model by incorporating noise factors relevant to AI performance. Situation coverage analysis is then applied to identify representative operational contexts. This work marks an initial step toward integrating noise factor analysis and situational coverage to support principled formulation and completeness assessment of robustness requirements for camera-based AI perception.

Figures

Figures reproduced from arXiv: 2507.12986 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

P. H. Chan, G. Dhadyalla, and V. Donzella, ``A framework to analyze noise factors of automotive perception sensors,'' IEEE Sensors Letters , vol. 4, no. 6, pp. 1--4, 2020

work page 2020
[2]

Molloy, S

J. Molloy, S. Shahbeigi, and J. A. McDermid, ``Hazard and safety analysis of machine-learning-based perception capabilities in autonomous vehicles,'' Computer , vol. 57, no. 11, pp. 60--70, 2024

work page 2024
[3]

B. Li, P. H. Chan, G. Baris, M. D. Higgins, and V. Donzella, ``Analysis of automotive camera sensor noise factors and impact on object detection,'' IEEE Sensors Journal , 2022

work page 2022
[4]

N. M. Proma, V. J. Hodge, and R. Alexander, `` SCALOFT: An Initial Approach for Situation Coverage-Based Safety Analysis of an Autonomous Aerial Drone in a Mine Environment ,'' in Accepted for, 44th International Conference on Computer Safety, Reliability and Security (safecomp 2025) , 2025. https://arxiv.org/abs/2505.20969

arXiv 2025
[5]

Hawkins, C

R. Hawkins, C. Paterson, C. Picardi, Y. Jia, R. Calinescu, and I. Habli, ``Guidance on the assurance of machine learning in autonomous systems (amlas),'' arXiv preprint arXiv:2102.01564 , 2021

arXiv 2021
[6]

ISO 21448:2022

International Organization for Standardization , `` ISO 21448:2022 Road vehicles -- Safety of the intended functionality .'' https://www.iso.org/standard/70939.html, 2022. ISO 21448:2022

work page 2022
[7]

Burton, L

S. Burton, L. Gauerhof, and C. Heinzemann, ``Making the case for safety of machine learning in highly automated driving,'' in Computer Safety, Reliability, and Security, Trento, Italy, September 12, 2017, Proceedings 36 , pp. 5--16, Springer, 2017

work page 2017
[8]

Proma, V

N. Proma, V. Hodge, and R. Alexander, ``Situation coverage based safety analysis of an autonomous aerial drone in a mine environment,'' in The Yorkshire Innovation in Science and Engineering Conference (YISEC), York UK , June 2024

work page 2024

Show all 12 references

[9]

Alexander, H

R. Alexander, H. R. Hawkins, and A. J. Rae, ``Situation coverage--a coverage criterion for testing autonomous robots,'' tech. rep., Department of Computer Science, University of York, 2015

2015
[10]

N. M. Proma and R. Alexander, ``Systematic situation coverage versus random situation coverage for safety testing in an autonomous car simulation,'' in Procs of the 12th Latin-American Symposium on Dependable and Secure Computing , LADC '23, p. 208–213, 2023

2023
[11]

Gauerhof, R

L. Gauerhof, R. Hawkins, C. Picardi, C. Paterson, Y. Hagiwara, and I. Habli, ``Assuring the safety of machine learning for pedestrian detection at crossings,'' in Computer Safety, Reliability, and Security: 39th International Conference, SAFECOMP 2020, Lisbon, Portugal, Septem...

2020
[12]

I love you

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEco...

2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

P. H. Chan, G. Dhadyalla, and V. Donzella, ``A framework to analyze noise factors of automotive perception sensors,'' IEEE Sensors Letters , vol. 4, no. 6, pp. 1--4, 2020

work page 2020

[2] [2]

Molloy, S

J. Molloy, S. Shahbeigi, and J. A. McDermid, ``Hazard and safety analysis of machine-learning-based perception capabilities in autonomous vehicles,'' Computer , vol. 57, no. 11, pp. 60--70, 2024

work page 2024

[3] [3]

B. Li, P. H. Chan, G. Baris, M. D. Higgins, and V. Donzella, ``Analysis of automotive camera sensor noise factors and impact on object detection,'' IEEE Sensors Journal , 2022

work page 2022

[4] [4]

N. M. Proma, V. J. Hodge, and R. Alexander, `` SCALOFT: An Initial Approach for Situation Coverage-Based Safety Analysis of an Autonomous Aerial Drone in a Mine Environment ,'' in Accepted for, 44th International Conference on Computer Safety, Reliability and Security (safecomp 2025) , 2025. https://arxiv.org/abs/2505.20969

arXiv 2025

[5] [5]

Hawkins, C

R. Hawkins, C. Paterson, C. Picardi, Y. Jia, R. Calinescu, and I. Habli, ``Guidance on the assurance of machine learning in autonomous systems (amlas),'' arXiv preprint arXiv:2102.01564 , 2021

arXiv 2021

[6] [6]

ISO 21448:2022

International Organization for Standardization , `` ISO 21448:2022 Road vehicles -- Safety of the intended functionality .'' https://www.iso.org/standard/70939.html, 2022. ISO 21448:2022

work page 2022

[7] [7]

Burton, L

S. Burton, L. Gauerhof, and C. Heinzemann, ``Making the case for safety of machine learning in highly automated driving,'' in Computer Safety, Reliability, and Security, Trento, Italy, September 12, 2017, Proceedings 36 , pp. 5--16, Springer, 2017

work page 2017

[8] [8]

Proma, V

N. Proma, V. Hodge, and R. Alexander, ``Situation coverage based safety analysis of an autonomous aerial drone in a mine environment,'' in The Yorkshire Innovation in Science and Engineering Conference (YISEC), York UK , June 2024

work page 2024