Pith. sign in

REVIEW 4 major objections 6 minor 107 references

SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networks

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read SeFaR evaluates a vision model's semantic robustness by perturbing one feature at a time and pairing each failure with the feature that caused it.

desk verdict SeFaR is a genuine extension of RBT4DNN with a clean hierarchical feature model; its main weakness is unmeasured edit fidelity, which blurs the causal link between named features and failures. read the letter →

arxiv 2608.10289 v1 pith:IO6ROR3C submitted 2026-08-10 cs.CV cs.LG

classification cs.CVcs.LG
keywords semanticrobustnesstestingvisionmodelsdiffusion-basedimageeditinghierarchicalconceptmodelfailure-inducingfeaturesrequirementpreconditionsvision-languageanalysisopen-worldfeaturediscovery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SeFaR sets out to turn semantic-feature robustness testing from a single-feature exercise into a systematic, hierarchical exploration. Given a requirement of the form "if precondition then postcondition" and images that satisfy it, SeFaR generates photorealistic edits that vary one semantic feature at a time, checks whether the model still satisfies the requirement, and then uses a vision-language model to identify which features correlate with failures, including features absent from the user's initial model. Across three perception settings, the paper reports that 93.4% of generated tests preserve the precondition and that an average of 16.3% of tests expose feature non-robustness. If this holds, developers gain not just failing inputs but a named, testable semantic cause for each failure.

What carries the argument

The central mechanism is the forest of feature trees (FFT), a user-populated hierarchy in which nodes are semantic features and edges are refinement relations expressed with `or` and `xor` constructors plus an open-world placeholder for unstated features. Around it, the pipeline is: a perturbation operator $\Delta$ that invokes a diffusion editing model with a structured prompt to change only the target feature; Algorithm 1, perturb-and-check, which filters for postcondition violations; and Algorithm 2, detect-violating-features, which uses a vision-language model to describe each original/perturbed pair, embeds and clusters the descriptions, and pairs each cluster of failures with its semantic feature. Adaptive exploration then refines the feature model by descending into features that cause failures, removing robust ones, and adding newly discovered features.

What would settle it

Take a set of SeFaR-generated color-only or type-only edits, mask the task-relevant objects such as the front vehicle, lane, and traffic light, and ask human annotators or a segmentation model whether any non-target object or spatial relation changed; if a substantial fraction of edits alter scene layout, object identity, or other requirement-relevant semantics while the test still flags the target feature, the attribution claim is falsified. Alternatively, run the same editing prompt multiple times for one seed image and check whether the same feature is consistently reported, since genuine single-feature perturbations should yield stable diagnoses.

Watch

Extended reading notes

Core claim

The paper's central claim is that a vision model's violations of a semantic requirement can be systematically traced to individual or composite semantic features by combining a user-defined feature hierarchy, diffusion-based editing, and vision-language-model-based failure analysis. SeFaR formalizes robustness under requirement-irrelevant features as invariance: for every input satisfying the precondition and every feature outside the precondition's semantics, violating the postcondition on a perturbed version of that input constitutes feature non-robustness. The method reports that this reveals feature-level vulnerabilities, such as vehicle color changes to green (39.7% failures) and blue (11.3%), vehicle type truck (10.5%), dust storm (2.4%), and lighting and traffic-light-pole edits (26.7% and 29.1% individually, 51.3% combined), and that such failures are witnessed by generated test inputs paired with the responsible feature.

Load-bearing premise

The entire pipeline assumes that the diffusion editor changes exactly the targeted semantic feature, such as vehicle color, and leaves everything else, including the precondition-relevant scene, intact; if it also alters scene layout, lighting, or task-relevant objects, the reported failure-inducing features are artifacts of the edit rather than causes of model non-robustness.

Editorial extensions

If this is right

  • Testers can go from "model failed on these images" to "model fails when the front vehicle is green or a truck," because each failing input is paired with the feature whose variation caused the failure.
  • Robustness should be assessed per feature and per feature combination, since interactions amplify failures (for example, individual lighting and traffic-light-pole edits fail around 27% to 29% of tests, while together they fail 51.3%).
  • Open-world modeling pays off: features absent from the initial hierarchy, such as a teal vehicle color, lighting, and dust storm, can be discovered automatically and folded back into the feature model for the next round.
  • Requirement preconditions are largely preserved under perturbation (93.4% average), but not perfectly, so generated tests still need a precondition check before being trusted.
  • The iterative refinement terminates with increasingly precise failure characterizations without requiring a complete feature model up front, easing adoption in domains where experts cannot enumerate all relevant attributes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond testing, the discovered failure features read as evidence of spurious correlations; a provider could use them to prioritize retraining data (for example, collecting more dust-storm terrain images) rather than only reporting bugs.
  • One testable extension is to use the same method to audit object detectors and segmentation models, not just control or classifier outputs, provided the postcondition can be expressed as a predicate over the model's output.
  • Another extension is to measure attribution stability across repeated edits: if the same feature is flagged consistently across several independent generations for the same seed image, the diagnosis is more trustworthy than if it depends on a single generated image.
  • The reported failure rates are tied to a particular generative editor and vision-language model; as editors improve, the bottleneck shifts from photorealism to ensuring the editor truly changes only the targeted attribute, which could be checked with human or perceptual evaluation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SeFaR, a framework for semantic-feature-aware robustness testing of vision models. Given a natural-language requirement (precondition implies postcondition), a set of satisfying inputs, and a user-provided forest of feature trees (FFT), SeFaR perturbs inputs along semantic features using a diffusion-based editing model (Qwen-Image-Edit), checks whether the perturbed inputs still satisfy the precondition and violate the postcondition, and then uses a vision-language model (Qwen2.5-VL) with embedding-based clustering to discover and name features that correlate with model failures. The FFT supports xor/or refinement relations and an open-world placeholder, and the framework iteratively refines feature selections based on detected failure-inducing features. The evaluation covers three datasets (SGSM, RRAV, AI4MARS) and reports an average precondition satisfaction of 93.4% (RQ1) and an average of 16.3% of tests revealing feature non-robustness (RQ2), with specific findings such as vehicle color Green causing 39.7% failures and Lighting causing 19.2% and 29.1% failures under requirements S3 and S5. The paper's central claim is that SeFaR can both uncover faults and relate them to semantic features.

Significance. If the central claim holds, SeFaR addresses an important gap: most prior testing work is pixel-level or simulator-constrained, and the paper provides a structured, interpretable way to test vision models against semantic feature variation that preserves requirement satisfaction. The formalization of semantic feature functional requirements and the FFT grammar are useful contributions, and the open-world placeholder is a sensible mechanism for discovering features not anticipated by the user. The artifact is publicly available, which supports reproducibility. However, the significance is currently conditional: the causal attribution of failures to named features rests on unvalidated assumptions about the editing operator and the VLM-based feature detector, and the empirical support is at the level of illustrative case studies without baselines, confidence intervals, or ground-truth checks. These issues are fixable within the scope of the manuscript, so the significance is real but not yet fully established.

major comments (4)
  1. [Section 4.2; Algorithm 1; Section 5.2 (Figure 7)] The central causal claim that model failures are caused by variation in a particular named feature rests on requirement (a) in Section 4.2: Delta must produce inputs that differ only in the target feature. RQ1 only measures precondition match rates (Figure 4), which is compatible with large unintended changes to layout, lighting, objects, or other semantics. If the editing model changes such factors, the failure rates in Figure 7 and the feature attributions in Section 4.3 are artifacts of confounded edits rather than sensitivity to the named feature. The manuscript should validate Delta fidelity directly, for example by human annotation or automated semantic/detection-based comparison of paired images for each feature, and report per-feature unintended-change rates.
  2. [Algorithm 2; Section 4.3; Section 6] Algorithm 2 attributes failure-inducing features based on Qwen2.5-VL descriptions clustered by a sentence transformer, but there is no human or ground-truth oracle to verify that the discovered attributes (e.g., Dust Storm, TrafficLightPole) are actually present in the perturbed images and absent from the seeds. Section 4.3 reports only a within-method coverage statistic (69% for Blue), and Section 6 acknowledges dependence on foundation-model quality without quantifying it. The paper should add a validation study, for example using known-edit images with ground-truth attribute labels to measure precision and recall of the detected features, and a human agreement study on a sample of generated pairs.
  3. [Section 5.1.2; Section 5.2.2] The evaluation is a convenience sample without quantitative baselines or statistical grounding. Section 5.1.2 explicitly states that the FFT subtrees are 'a limited convenience sample selected to facilitate this evaluation,' and Figure 7 reports failure percentages without confidence intervals or significance tests. To support the general claim that SeFaR effectively reveals model sensitivity to individual and composite features, the paper needs comparisons to relevant baselines (e.g., RBT4DNN [55], random instruction-guided edits, or counterfactual methods [52-54]) and uncertainty quantification for the reported rates; otherwise the 16.3% average and per-feature rates are descriptive case-study numbers.
  4. [Section 5.2.1; Section 5.2 (RQ2 interpretation)] The precondition-satisfaction measurement is not validated for the cases that matter most. Section 5.2.1 reports average classifier accuracy but not per-requirement accuracy, and for S3_B it relies on manual visual inspection to assert that classifiers 'underestimate precondition satisfaction'; for AI4MARS it uses MiniCPM-o without reporting its accuracy. Since RQ2 interprets low pass rates (e.g., S3_B 36.3%, S5_B 9.7%) as model non-robustness after excluding precondition failures, inaccurate precondition detectors can inflate apparent non-robustness. Please report per-requirement validation and error analysis for the precondition detectors, and either exclude or reclassify ambiguous images.
minor comments (6)
  1. [Section 3.2, equations] The formula '∀𝑥∈𝑋,∀𝑓∈𝐹 𝜙 :(𝜙 𝑥(𝑥)⇒𝜙 𝑦(𝑁(𝑥)))⇒...' has ambiguous parsing due to the placement of the colon and the mixed use of phi and Greek subscripts; use explicit parentheses to clarify the quantifier scope.
  2. [Section 5.2, text before RQ1] The sentence 'Across FFT-requirement pairs, no root-level postcondition violations are observed for Background (S1, S2, S4, S7) or for SurfaceElements and Weather under A1. We exclude these cases...' should explain how these exclusions affect the reported averages and the denominators in Figure 7.
  3. [Figures 4 and 7 captions] The figure captions say the X-axis shows requirement-feature tree pairs but do not define the abbreviations V, B, and W; please define them in the captions.
  4. [Algorithm 2, line 14] The expression '𝑐ℎ𝑎𝑛𝑔𝑒[𝑥′]⊆𝑐' is unclear; it should presumably mean that the recorded changes belong to cluster c, and the subset notation should be replaced with a defined membership predicate.
  5. [Section 4.1] The statement that 'The full graph defined by these rules is not itself a tree, because subtrees may be shared among roots' sits awkwardly with the term 'forest of feature trees'; clarify whether shared subtrees are allowed and how tree-paths resolve the resulting ambiguity.
  6. [References] Reference [68] duplicates reference [60] (Qwen-Image), and reference [41] appears unused in the text; please reconcile the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the evaluation is empirical and the central claims do not reduce to their inputs.

full rationale

SeFaR's claimed derivation chain is empirical rather than definitional. Failure rates (RQ2) are computed on generated images against the model under test; the feature labels produced by Algorithm 2 do not determine pass/fail, and the reported figures (e.g., 16.3% average non-robustness) are measured on generated tests, not on the violating pairs used for discovery. No equation in Sections 3-4 defines a predicted quantity in terms of a fitted parameter: F_phi and Delta are inputs/assumptions, and the postcondition check phi_y(N(x')) is an independent oracle. The paper reuses artifacts from the authors' prior work [55] (requirements, data generators, precondition classifiers, and the SGSM model), which is a notable self-citation burden, but these artifacts are supporting infrastructure rather than the framework's conclusion: the new perturbations are edited from those seeds and evaluated with external foundation models and task-specific oracles. The main caveat is that Delta's fidelity (varying only the target feature) and the VLM's attributions are unvalidated; the threats-to-validity section admits dependence on foundation-model quality. That is a correctness/validity threat, not a circular reduction: an edit that confounds layout or lighting would make the feature attributions wrong, but it would not make the measured failure rates equal to the chosen features by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The framework introduces no fitted physical parameters and no new physical entities. It does depend on three domain assumptions: feature orthogonality to requirements, edit fidelity, and VLM label reliability. These assumptions, stated or implied in Sections 4.1 to 4.3, carry most of the burden for the central claim.

assumptions (3)
  • domain assumption Feature-tree roots and nodes do not appear in the requirement precondition, so perturbing them preserves precondition satisfaction.
    Section 4.1 states "we assume that the roots of the defined feature trees ... do not appear in the requirement under consideration." If this fails, generated images may no longer satisfy the precondition and failure attribution breaks.
  • domain assumption The diffusion edit model with structured prompts preserves all non-target semantics and only changes the target feature.
    Section 4.2 gives requirements (a) to (c) for Delta and relies on prompt templates and generative fidelity instead of verification. If edits introduce unintended changes, the detected failure-inducing features may be artifacts.
  • domain assumption The vision-language model reliably and consistently names semantic changes, including features outside the user-defined feature forest.
    Section 4.3 uses VLM outputs with embedding clustering to construct feature explanations, but no ground-truth evaluation of these labels is reported and the prompts are generated by ChatGPT.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networks." pith.science (2026). https://pith.science/paper/IO6ROR3C

@misc{pith2026260810289,
  author       = {Pith},
  title        = {Pith review of: SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IO6ROR3C}},
  note         = {Machine review of arXiv:2608.10289}
}
read the original abstract

Deep neural networks are increasingly deployed in safety-critical domains as perception modules, where failures are often caused due to rare and under-represented scenarios. This necessitates the need to evaluate the semantic robustness of perception models; conformance of behavior to high-level requirements over real-world perceptual variability. To address this, we propose SeFaR, a framework for systematic semantic-feature-centric testing of vision models. Given a natural-language requirement and a set of satisfying inputs, SeFaR evaluates robustness with respect to diverse realistic semantic variations that preserve requirement satisfaction. The approach employs a novel hierarchical concept model enabling structured exploration of the feature space and incorporation of domain knowledge via user-defined concepts. State-of-the-art diffusion and vision-language models are leveraged to generate photorealistic semantics-preserving perturbations and identification of previously unknown features impacting behavior. A feedback-driven adaptive process is adopted to generate interpretable failure-inducing semantic concepts along with corresponding test inputs. Evaluation on case studies demonstrates that the proposed framework effectively satisfies requirement preconditions while identifying requirement-independent features that influence model decisions, enabling it to both uncover faults and relate them to such features.

Figures

Figures reproduced from arXiv: 2608.10289 by the authors.

Figure 1
Figure 1. Examples of SeFaR generated misclassified images and corresponding failure-inducing semantic features from SGSM and AI4Mars case-studies. The leftmost image in each panel satisfies the respective requirement. The rest of the images show perturbations with respect to a specific concept while preserving other scene semantics, leading to incorrect model behavior. Our Approach. We propose SeFaR, a framework for systemat… view at source ↗
Figure 2
Figure 2. SeFaR Overview 4 Approach [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A forest of feature trees. A feature can be defined by [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Percentage of generated tests that match the precondition. X-axis shows requirement–feature tree pairs, where each [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Image panel of precondition Satisfaction. S3_B: [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Image panel for S3_B: Background (top), Natural (middle), TreeDensity (bottom). In each row, the leftmost image is the seed, followed by perturbed images that fail the precondition classifier. As the VLM captures fine-grained changes, unconstrained Level 0 exploration …
Figure 7
Figure 7. Figure 7: Percentage of generated tests for which the model passed. X-axis shows requirement–feature tree pairs, where each [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

107 extracted references · 70 canonical work pages

  1. [55]

    RBT4DNN: Requirements-based Testing of Neural Networks

    N. J. Mozumder, F. Toledo, S. Dola, and M. B. Dwyer, “Rbt4dnn: Requirements- based testing of neural networks, ”arXiv preprint arXiv:2504.02737, 2025

  2. [1]

    To- wards unsupervised object detection from lidar point clouds,

    L. Zhang, A. J. Yang, Y. Xiong, S. Casas, B. Yang, M. Ren, and R. Urtasun, “To- wards unsupervised object detection from lidar point clouds, ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 9317–9328

  3. [2]

    Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline,

    P. Wu, X. Jia, L. Chen, J. Yan, H. Li, and Y. Qiao, “Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline, ” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 6119–6132

  4. [3]

    Safety-enhanced autonomous driving using interpretable sensor fusion transformer,

    H. Shao, L. Wang, R. Chen, H. Li, and Y. Liu, “Safety-enhanced autonomous driving using interpretable sensor fusion transformer, ” inProceedings of the Conference on Robot Learning (CoRL). PMLR, 2023, pp. 726–737

  5. [4]

    Deeper notions of correct- ness for machine learning-based systems,

    F. Toledo, D. Shriver, S. Elbaum, and M. B. Dwyer, “Deeper notions of correct- ness for machine learning-based systems, ” inProceedings of the International Conference on Software Engineering (ICSE), 2021

  6. [5]

    End to end learning for self-driving cars,

    M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. Jackel, M. Monfort, U. Muller, J. Zhang et al., “End to end learning for self-driving cars, ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2016

  7. [6]

    Learning by cheating,

    D. Chen, V. Koltun, and P. Krähenbühl, “Learning by cheating, ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 5127–5136

  8. [7]

    Machine learning for au- tonomous navigation in unstructured environments,

    A. Giusti, J. Guzzi, D. C. Ciresan, F.-L. He, J. P. Rodriguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. A. Di Caro et al., “Machine learning for au- tonomous navigation in unstructured environments, ” in IEEE International Conference on Robotics and Automation (ICRA), 2016

Show all 107 references
  1. [8]

    Dronet: Learning to fly by driving,

    A. Loquercio, A. I. Maqueda, C. R. del Blanco, and D. Scaramuzza, “Dronet: Learning to fly by driving, ” inIEEE Robotics and Automation Letters (RA-L), 2018

  2. [9]

    Deep learning for real-time uav navigation and obstacle avoidance,

    H. Pham and H. La, “Deep learning for real-time uav navigation and obstacle avoidance, ”IEEE Transactions on Intelligent Transportation Systems, 2018

  3. [10]

    Deep learning for spacecraft pose estimation from monocular images,

    P. Proenca and Y. Gao, “Deep learning for spacecraft pose estimation from monocular images, ”IEEE Transactions on Aerospace and Electronic Systems, vol. 56, no. 2, pp. 1497–1510, 2019

  4. [11]

    You only look twice: Rapid multi-scale object detection in satel- lite imagery,

    A. Van Etten, “You only look twice: Rapid multi-scale object detection in satel- lite imagery, ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

  5. [12]

    Deepglobe 2018: A challenge to parse the earth through satellite images,

    I. Demir, K. Koperski, D. Lindenbaum, G. Pang, J. Huang, S. Basu, F. Hughes, D. Tuia, and R. Raska, “Deepglobe 2018: A challenge to parse the earth through satellite images, ” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018

  6. [13]

    Deep learning in remote sensing: A comprehensive review and list of resources,

    X. X. Zhu, D. Tuia, L. Mou, G.-S. Xia, L. Zhang, F. Xu, and F. Fraundorfer, “Deep learning in remote sensing: A comprehensive review and list of resources, ”IEEE Geoscience and Remote Sensing Magazine, vol. 5, no. 4, pp. 8–36, 2017

  7. [14]

    De- tecting spacecraft anomalies using lstms and nonparametric dynamic thresh- olding,

    K. Hundman, V. Constantinou, C. Laporte, I. Colwell, and T. Soderstrom, “De- tecting spacecraft anomalies using lstms and nonparametric dynamic thresh- olding, ” inProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2018

  8. [15]

    Driver error or designer error: Using the perceptual cycle model to explore the circumstances surrounding the fatal tesla crash,

    V. Banks, K. Plant, and N. Stanton, “Driver error or designer error: Using the perceptual cycle model to explore the circumstances surrounding the fatal tesla crash, ”Transportation Research Part F, 2017

  9. [16]

    Effects of autonomous vehicle crashes on public perception of self-driving technology,

    P. Penmetsa et al., “Effects of autonomous vehicle crashes on public perception of self-driving technology, ”Transportation Research Part F, 2021

  10. [17]

    A matched case-control analysis of autonomous vs human- driven vehicle accidents,

    M. Abdel-Aty et al., “A matched case-control analysis of autonomous vs human- driven vehicle accidents, ”Nature Communications Engineering, 2024

  11. [18]

    Crash: Cognitive reasoning agent for safety hazards in autonomous driving,

    E. Silva, R. Yasmin, and A. Shoker, “Crash: Cognitive reasoning agent for safety hazards in autonomous driving, ”arXiv preprint arXiv:2603.15364, 2026

  12. [19]

    ISO/PAS 21 448:2019, 2019, sOTIF standard for safety of autonomous driving systems

    Road Vehicles – Safety of the Intended Functionality, International Organiza- tion for Standardization Std. ISO/PAS 21 448:2019, 2019, sOTIF standard for safety of autonomous driving systems

  13. [20]

    On the generation of synthetic event-based vision datasets for navigation and landing,

    L. J. Azzalini, E. Blazquez, A. Hadjiivanov, G. Meoni, and D. Izzo, “On the generation of synthetic event-based vision datasets for navigation and landing, ” in Proceedings of the ESA Guidance, Navigation and Control Conference (ESA GNC), 2023, pp. 1–14

  14. [21]

    Generation of artificial infrared camera images for visual naviga- tion simulation,

    K. Zysk, M. Hałoń, K. Kaczmarek, M. Kasprzyk, P. Rodo, O. Skromak, and M. Sochacki, “Generation of artificial infrared camera images for visual naviga- tion simulation, ” inProceedings of the ESA Guidance, Navigation and Control Conference (ESA GNC), Sopot, Poland, 2023

  15. [22]

    Cit4dnn: Generating diverse and rare inputs for neural networks using latent space combinatorial testing,

    S. Dola, R. McDaniel, M. B. Dwyer, and M. L. Soffa, “Cit4dnn: Generating diverse and rare inputs for neural networks using latent space combinatorial testing, ” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), 2024, pp. 1–13

  16. [23]

    Deeptest: Automated testing of deep-neural- network-driven autonomous cars,

    Y. Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: Automated testing of deep-neural- network-driven autonomous cars, ” in Proceedings of the 40th International Conference on Software Engineering (ICSE), 2018, pp. 303–314

  17. [24]

    Deepxplore: Automated whitebox testing of deep learning systems,

    K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems, ” in Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP). ACM, 2017, pp. 1–18

  18. [25]

    Dlfuzz: Differential fuzzing testing of deep learning systems,

    J. Guo, Y. Jiang, Y. Zhao, Q. Chen, and J. Sun, “Dlfuzz: Differential fuzzing testing of deep learning systems, ” in Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2018,...

  19. [26]

    Effective white-box testing of deep neural networks with adaptive neuron-selection strategy,

    S. Lee, S. Cha, D. Lee, and H. Oh, “Effective white-box testing of deep neural networks with adaptive neuron-selection strategy, ” inProceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2020, pp. 165–176

  20. [27]

    Bet: Black-box efficient testing for convolutional neural networks,

    J. Wang, H. Qiu, Y. Rong, H. Ye, Q. Li, Z. Li, and C. Zhang, “Bet: Black-box efficient testing for convolutional neural networks, ” inProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2022, pp. 164–175

  21. [28]

    Efficient and effective feature space exploration for testing deep learning systems,

    T. Zohdinasab, V. Riccio, A. Gambi, and P. Tonella, “Efficient and effective feature space exploration for testing deep learning systems, ”ACM Transactions on Software Engineering and Methodology (TOSEM), 2022

  22. [29]

    Scenic: A language for scenario specification and data gener- ation,

    D. J. Fremont, E. Kim, T. Dreossi, S. Ghosh, X. Yue, A. L. Sangiovanni-Vincentelli, and S. A. Seshia, “Scenic: A language for scenario specification and data gener- ation, ”Machine Learning, vol. 112, no. 10, pp. 3805–3849, 2023

  23. [30]

    VerifAI: A toolkit for the formal design and analysis of artificial intelligence-based systems,

    T. Dreossi, D. J. Fremont, S. Ghosh, E. Kim, H. Ravanbakhsh, M. Vazquez- Chanlatte, and S. A. Seshia, “VerifAI: A toolkit for the formal design and analysis of artificial intelligence-based systems, ” inProceedings of the 31st International Conference on Computer Aided Verific...

  24. [31]

    A framework for autonomous vehi- cle testing using semantic models,

    Y. He, M. Razi, J. Z. Gao, and C. Tao, “A framework for autonomous vehi- cle testing using semantic models, ” in Proceedings of the IEEE International Conference on Artificial Intelligence Testing (AITest), 2023, pp. 66–73

  25. [32]

    Autonomous vehicles scenario testing framework and model of computation: On generation and coverage,

    A. J. Alnaser, A. Sargolzaei, M. I. Akbas, R. Razdan, R. Sell, M. Bellone, M. Menase, and M. Malayjerdi, “Autonomous vehicles scenario testing framework and model of computation: On generation and coverage, ”IEEE Access, vol. 9, pp. 60 617–60 628, 2021

  26. [33]

    Formal scenario-based testing of autonomous vehicles: From simulation to the real world,

    D. J. Fremont, E. Kim, Y. V. Pant, S. A. Seshia, A. Acharya, X. Bruso, P. Wells, S. Lemke, Q. Lu, and S. Mehta, “Formal scenario-based testing of autonomous vehicles: From simulation to the real world, ” in2020 IEEE 23rd International Conference on Intelligent Transportation S...

  27. [34]

    S3c: Spatial semantic scene coverage for autonomous vehicles,

    T. Woodlief, F. Toledo, S. Elbaum, and M. B. Dwyer, “S3c: Spatial semantic scene coverage for autonomous vehicles, ” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

  28. [35]

    Search-based DNN testing and retraining with gan-enhanced simulations,

    M. O. Attaoui, F. Pastore, and L. C. Briand, “Search-based DNN testing and retraining with gan-enhanced simulations, ”IEEE Trans. Software Eng., vol. 51, no. 4, pp. 1086–1103, 2025

  29. [36]

    Designator: a toolset for automated gan-enhanced search-based testing and retraining of dnns in martian environments,

    M. O. Attaoui and F. Pastore, “Designator: a toolset for automated gan-enhanced search-based testing and retraining of dnns in martian environments, ” in Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE 2025) Tool Demonstration T...

  30. [37]

    Efficient domain augmentation for autonomous driving testing using diffusion models,

    L. Baresi, D. Y. X. Hu, A. Stocco, and P. Tonella, “Efficient domain augmentation for autonomous driving testing using diffusion models, ” inProceedings of the 47th International Conference on Software Engineering (ICSE 2025), 2025

  31. [38]

    Flux: Text-to-image generation model,

    B. F. Labs, “Flux: Text-to-image generation model, ” 2024, https://huggingface. co/black-forest-labs

  32. [39]

    Flux-kontext: Context-aware image editing with diffusion models,

    ——, “Flux-kontext: Context-aware image editing with diffusion models, ” 2024, https://huggingface.co/black-forest-labs

  33. [40]

    Concept sliders: Interpretable control of diffusion models,

    R. Gandikota et al., “Concept sliders: Interpretable control of diffusion models, ” arXiv preprint arXiv:2304.xxxxx, 2024

  34. [41]

    Qwen-vl: A versatile vision-language model for understanding, localization, and generation,

    J. Bai et al., “Qwen-vl: A versatile vision-language model for understanding, localization, and generation, ”arXiv preprint arXiv:2308.12966, 2023

  35. [42]

    Benchmarking semantic segmentation models via appearance and geometry attribute editing,

    e. a. Yin, “Benchmarking semantic segmentation models via appearance and geometry attribute editing, ” inCVPR, 2024

  36. [43]

    Concept algebra for fine-grained control in diffusion models,

    Y. Du, S. Li, A. Torralba, and J. B. Tenenbaum, “Concept algebra for fine-grained control in diffusion models, ” in Advances in Neural Information Processing Systems (NeurIPS), 2023

  37. [44]

    Conceptor: Concept-oriented controllable generation for diffusion models,

    D. Rueckert, M. Niemeyer, A. Vedaldi, and C. Rupprecht, “Conceptor: Concept-oriented controllable generation for diffusion models, ” inInternational Conference on Learning Representations (ICLR), 2024

  38. [45]

    Closed-form factorization of la- tent semantics in gans,

    Y. Shen, C. Yang, X. Tang, and B. Zhou, “Closed-form factorization of la- tent semantics in gans, ” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 1532–1540

  39. [46]

    Noiseclr: Noise- guided disentangled representation learning for diffusion models,

    M. Subedar, K. Ridgeway, A. G. Wilson, H. Liu, and J.-Y. Zhu, “Noiseclr: Noise- guided disentangled representation learning for diffusion models, ” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 11 mozumder et al

  40. [47]

    Riemannian geodesics in diffusion generative models,

    F. Lastname, F. Lastname, and F. Lastname, “Riemannian geodesics in diffusion generative models, ” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

  41. [48]

    Cns- bench: Benchmarking image classifier robustness under continuous nuisance shifts,

    O. Dünkel, A. Jesslen, J. Xie, C. Theobalt, C. Rupprecht, and A. Kortylewski, “Cns- bench: Benchmarking image classifier robustness under continuous nuisance shifts, ” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025

  42. [49]

    Benchmarking robustness to text-guided corruptions,

    M. Mofayezi and Y. Medghalchi, “Benchmarking robustness to text-guided corruptions, ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023

  43. [50]

    Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object,

    C. Zhang, F. Pan, J. Kim, I. S. Kweon, and C. Mao, “Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object, ” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

  44. [51]

    Dillema: Diffusion and large language models for multi-modal augmentation,

    D. P. authors, “Dillema: Diffusion and large language models for multi-modal augmentation, ” 2025

  45. [52]

    Lance: Stress- testing visual models by generating language-guided counterfactual images,

    V. Prabhu, S. Yenamandra, P. Chattopadhyay, and J. Hoffman, “Lance: Stress- testing visual models by generating language-guided counterfactual images, ” in Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023

  46. [53]

    Decodex: Confounder detector guidance for diffusion-based counterfactual explanations,

    N. Fathi et al., “Decodex: Confounder detector guidance for diffusion-based counterfactual explanations, ” in Proceedings of Machine Learning Research (PMLR), 2024

  47. [54]

    Visual counterfactual explanations via conceptual edits,

    “Visual counterfactual explanations via conceptual edits, ” in NeurIPS, 2025

  48. [56]

    Collision between vehicle controlled by developmental automated driving system and pedestrian,

    National Transportation Safety Board, “Collision between vehicle controlled by developmental automated driving system and pedestrian, ” 2019, uber self- driving crash, Tempe, Arizona

  49. [57]

    Collision between a sport utility vehicle operating with partial driving automation and a crash attenuator,

    ——, “Collision between a sport utility vehicle operating with partial driving automation and a crash attenuator, ” 2020, tesla Autopilot crash, Mountain View, California

  50. [58]

    Collision between a car operating with automated vehicle control systems and a tractor-semitrailer truck,

    ——, “Collision between a car operating with automated vehicle control systems and a tractor-semitrailer truck, ” 2021, failure to detect truck crossing highway

  51. [59]

    Battery fire after high-speed collision involving electric vehicle,

    ——, “Battery fire after high-speed collision involving electric vehicle, ” 2018, post-crash battery fire risk highlighting perception and response limitations

  52. [61]

    Qwen models on hugging face,

    Qwen Team, “Qwen models on hugging face, ” https://huggingface.co/Qwen, 2025, accessed: 2026-03-13

  53. [62]

    Specifying and moni- toring safe driving properties with scene graphs,

    F. Toledo, T. Woodlief, S. Elbaum, and M. B. Dwyer, “Specifying and moni- toring safe driving properties with scene graphs, ” inProceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 15 577–15 584

  54. [63]

    Ai4mars: A dataset for terrain-aware autonomous driving on mars,

    R. M. Swan, D. Atha, H. A. Leopold, M. Gildner, S. Oij, C. Chiu, X. Hong, and M. Ono, “Ai4mars: A dataset for terrain-aware autonomous driving on mars, ” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021

  55. [64]

    Assessing visually continuous corruption ro- bustness of neural networks relative to human performance,

    B. C. Hu and M. Chechik, “Assessing visually continuous corruption ro- bustness of neural networks relative to human performance, ”arXiv preprint arXiv:2402.19401, 2024

  56. [65]

    Deeproad: Gan-based metamorphic autonomous driving sys- tem testing,

    M. Zhang et al., “Deeproad: Gan-based metamorphic autonomous driving sys- tem testing, ” inASE, 2018

  57. [66]

    Deepgauge: Multi-granularity testing criteria for deep learning systems,

    L. Ma et al., “Deepgauge: Multi-granularity testing criteria for deep learning systems, ” inASE, 2018

  58. [67]

    Deepxplore: Automated whitebox testing of deep learning systems,

    K. Peiet al., “Deepxplore: Automated whitebox testing of deep learning systems, ” in SOSP, 2017

  59. [69]

    Explaining and harnessing adver- sarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adver- sarial examples, ” inProceedings of the International Conference on Learning Representations (ICLR), 2015, arXiv:1412.6572

  60. [70]

    Adversarial examples in the physical world,

    A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial examples in the physical world, ” inProceedings of the International Conference on Learning Representations (ICLR), Workshop Track, 2017, arXiv:1607.02533

  61. [71]

    Prac- tical black-box attacks against machine learning,

    N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Prac- tical black-box attacks against machine learning, ” inProceedings of the 2017 ACM Asia Conference on Computer and Communications Security (AsiaCCS). ACM, 2017, pp. 506–519

  62. [72]

    Benchmarking neural network robustness to common corruptions and perturbations,

    D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations, ” inProceedings of the International Conference on Learning Representations (ICLR), 2019, iMAGENET-C and IMAGENET-P benchmarks

  63. [73]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., “On the opportunities and risks of foundation models, ”arXiv preprint arXiv:2108.07258, 2021

  64. [74]

    Language models are few-shot learn- ers,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakan- tan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learn- ers, ”Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877–1901, 2020

  65. [75]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding, ” in Proceedings of NAACL-HLT, 2019, pp. 4171–4186

  66. [76]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision, ” inICML, 2021

  67. [77]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y. Yang, Y. Xia, Y.-C. Chen, Z. Parekh, H. Pham, Q. V. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision, ” inProceedings of ICML, 2021

  68. [78]

    Flamingo: a visual language model for few- shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al., “Flamingo: a visual language model for few- shot learning, ”Advances in Neural Information Processing Systems (NeurIPS), 2022

  69. [79]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. C. H. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models, ” Proceedings of ICML, 2023

  70. [80]

    Vqa: Visual question answering,

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering, ” inProceedings of ICCV, 2015, pp. 2425–2433

  71. [81]

    Bottom-up and top-down attention for image captioning and visual question answering,

    P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering, ” inProceedings of CVPR, 2018

  72. [82]

    Deep unsu- pervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsu- pervised learning using nonequilibrium thermodynamics, ” inICML, 2015

  73. [83]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models, ” Advances in Neural Information Processing Systems (NeurIPS), 2020

  74. [84]

    Score- based generative modeling through stochastic differential equations,

    Y. Song, J. Sohl-Dickstein, D. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score- based generative modeling through stochastic differential equations, ” inICLR, 2021

  75. [85]

    Hierarchi- cal text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchi- cal text-conditional image generation with clip latents, ” in arXiv preprint arXiv:2204.06125, 2022

  76. [86]

    Photorealistic text-to-image diffusion models with deep language understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, K. Ghasemipour, B. K. Ayan, S. Mahdavi, R. Lopes et al., “Photorealistic text-to-image diffusion models with deep language understanding, ” inNeurIPS, 2022

  77. [87]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models,

    A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models, ” arXiv preprint arXiv:2112.10741, 2021

  78. [88]

    Reliable evaluation of adversarial robustness with an ensemble of diverse attacks,

    F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse attacks, ” inICML, 2020

  79. [89]

    Improving robustness using generated data,

    S. Gowal, C. Qin, J. Uesato, T. Mann, and P. Kohli, “Improving robustness using generated data, ” inNeurIPS, 2021

  80. [90]

    Certified robustness for deep neural networks,

    L. Li, T. Xie, and B. Li, “Certified robustness for deep neural networks, ” inIEEE Symposium on Security and Privacy, 2020

  81. [91]

    A survey of adversarial examples in computer vision,

    K. Xu et al., “A survey of adversarial examples in computer vision, ” Wuhan University Journal of Natural Sciences, 2025

  82. [92]

    On evaluating adversarial robustness,

    N. Carlini et al., “On evaluating adversarial robustness, ” arXiv preprint arXiv:1902.06705, 2019

  83. [93]

    Deepsafe: A data- driven approach for assessing robustness of neural networks,

    D. Gopinath, G. Katz, C. S. Pasareanu, and C. Barrett, “Deepsafe: A data- driven approach for assessing robustness of neural networks, ” inInternational Symposium on Automated Technology for Verification and Analysis (ATVA). Springer, 2018, pp. 3–19

  84. [94]

    Baader, D

    F. Baader, D. Calvanese, D. McGuinness, D. Nardi, and P. Patel-Schneider, The Description Logic Handbook: Theory, Implementation, and Applications. Cambridge University Press, 2003

  85. [95]

    Instructpix2pix: Learning to follow image editing instructions,

    T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions, ” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 18 392–18 402

  86. [96]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models, ” inProceedings of CVPR, 2022

  87. [97]

    SEFAR Artifact Repository,

    Anonymous Authors, “SEFAR Artifact Repository, ” https://doi.org/10.5281/ zenodo.19341589, 2026, accessed: 2026-03-30

  88. [98]

    Artificial analysis ai model leaderboards,

    Artificial Analysis, “Artificial analysis ai model leaderboards, ” https:// huggingface.co/spaces/ArtificialAnalysis/Text-to-Image-Leaderboard, 2025, ac- cessed: 2026-03-13

  89. [99]

    Qwen2.5-vl,

    Q. Team, “Qwen2.5-vl, ” January 2025. [Online]. Available: https://qwenlm. github.io/blog/qwen2.5-vl/

  90. [100]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks, ” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational 12 SeFaR: Semantic Feature-aware Robustness Testing of Deep Ne...

  91. [101]

    AgglomerativeClustering — scikit-learn docu- mentation,

    Scikit-learn Developers, “AgglomerativeClustering — scikit-learn docu- mentation, ” https://scikit-learn.org/stable/modules/generated/sklearn.cluster. AgglomerativeClustering.html, 2026, accessed: 2026-03-25

  92. [102]

    Chatgpt,

    OpenAI, “Chatgpt, ” https://chat.openai.com, 2025, accessed: 2026-03-13

  93. [103]

    CARLA Leaderboard 1.0 – SENSORS Track (0.9.10.1),

    “CARLA Leaderboard 1.0 – SENSORS Track (0.9.10.1), ” https://leaderboard.carla. org/leaderboard/, [Online; accessed 23-Oct-2024]

  94. [104]

    NASA experimental rover,

    “NASA experimental rover, ” https://ntrs.nasa.gov/api/citations/20250004071/ downloads/RRAV_2025AmesParternshipDays.mp4, accessed: 2026-01-20

  95. [105]

    Coco 2017 dataset,

    COCO Consortium, “Coco 2017 dataset, ” https://cocodataset.org, 2017

  96. [106]

    Tracon: A novel dataset for real-time traffic cones detection using deep learning,

    I. Katsamenis, E. E. Karolou, A. Davradou, E. Protopapadakis, A. Doulamis, N. Doulamis, and D. Kalogeras, “Tracon: A novel dataset for real-time traffic cones detection using deep learning, ”arXiv preprint arXiv:2205.11830, 2022

  97. [107]

    Marssimnav: Mars simulation navigation dataset and tools,

    Pushkar Hue, “Marssimnav: Mars simulation navigation dataset and tools, ” https://github.com/pushkar-hue/MarsSimNav, 2025, accessed: 2026-03-13

  98. [108]

    MiniCPM-o-2_6,

    OpenBMB, “MiniCPM-o-2_6, ” 2025, accessed: March 3, 2026. [Online]. Available: https://huggingface.co/openbmb/MiniCPM-o-2_6 13

  99. [2025]

    Available: https://arxiv.org/abs/2508.02324

    [Online]. Available: https://arxiv.org/abs/2508.02324

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.