Pith. sign in

REVIEW 4 major objections 5 minor 64 references

TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Traffic-light detectors lose 39.8% mAP on augmented scenes.

desk verdict Well-scoped augmentation tool with genuinely new traffic light transforms, but the headline testing claim rests on aggregate mAP rather than the per-image metamorphic checks the paper defines. read the letter →

arxiv 2507.05932 v1 pith:ERBAKLQT submitted 2025-07-08 cs.SE cs.CV

You Lu , Dingji Wang , Kaifeng Huang , Bihuan Chen , Xin Peng This is my paper · ORCID
classification cs.SEcs.CV
keywords trafficlightdetectionautonomousdrivingdataaugmentationmetamorphictestingdeeplearningretrainingobjectmeanaverageprecision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TigAug is a data-augmentation pipeline for testing traffic light detection in autonomous driving. It applies twelve transformations over three families of effects, namely weather, camera, and traffic-light properties, to labeled traffic-light images, and uses metamorphic relations to check whether a detector's output changes as it should. On two real-world datasets with four detectors, the pipeline reports that original models lose on average 39.8% mean average precision on augmented test images, which it interprets as evidence of erroneous behaviors. Retraining on the same augmented images raises average precision by an average of 67.5% on augmented tests while keeping performance on original tests roughly unchanged. TigAug therefore offers an automated alternative to manual collection and labeling of diverse traffic-light data.

What carries the argument

The load-bearing mechanism is the pairing of two families of metamorphic relations with twelve concrete image transformations. The weather family (rain, snow, fog, lens flare) and camera family (overexposure, underexposure, motion blur) share the relation that a detector's output should be identical on the original and transformed image; the traffic-light family (color change, position move, addition, rotation, scaling) uses the relation that the output should shift exactly as the light was shifted. The equality criterion is mAP@[.50,.95], averaged over ten intersection-over-union thresholds, which the paper uses to tolerate small bounding-box drift. These relations turn labeled images into test oracles without manual labeling of the augmented images.

What would settle it

A single decisive check is to run the original models on the cleaned augmented test sets, the paper's '-' variants, and see whether the average 39.8% mAP drop persists when only human-visible transformed images are included; if it mostly disappears, the testing claim rests on impossible inputs rather than detector errors.

Watch

Extended reading notes

Core claim

The paper's central claim is that metamorphic-relation-guided augmentation can both reveal and repair weaknesses in traffic-light detection models. For weather and camera transformations, the relation states that the detection output should be unchanged when rain, snow, fog, lens flare, exposure changes, or motion blur are applied to an image. For traffic-light transformations, the relation states that the output should change in the same way the light was changed, whether in color, position, count, orientation, or scale. Violations are detected by comparing mean average precision, mAP@[.50,.95], before and after transformation. The experimental claim is that YOLOv5, YOLOX, Faster R-CNN, and SSD all drop on average 39.8% mAP on augmented test data, and that retrained models gain 67.5% mAP on that same data, with the largest per-transformation improvements coming from rotation and fog.

Load-bearing premise

The testing result assumes that a drop in detection accuracy on artificially transformed images means the model is faulty, even though some transformed images hide traffic lights so completely that no detector could reasonably be expected to find them.

Editorial extensions

If this is right

  • Traffic-light detection models can be tested automatically against weather, camera, and traffic-light variations that are expensive or dangerous to collect on the road, with an average 39.8% mAP drop exposing their lack of robustness.
  • Adding 20% of each augmented training dataset to the original training data produces retrained models whose average mAP on augmented test data is 67.5% higher than the original models, with little or no loss on original test data.
  • Retraining with all twelve transformations together outperforms retraining with any single transformation, so the full transformation set is better than its parts.
  • The roughly 27.9% of synthesized images judged unnatural do not hurt detection on natural data when used for retraining, so the pipeline can be used without manual cleaning.
  • The cost is modest, about 0.88 seconds per synthesized image and about 36 hours to retrain a model on all augmented data, making the approach practical for repeated testing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit implication is that the same metamorphic-relation scheme could be transferred to other small, safety-critical perception targets, such as traffic signs or railroad crossings, where labeled data are scarce and weather effects dominate.
  • The paper's testing evidence is an aggregate mAP comparison; a stricter implementation would check each augmented image against its metamorphic relation individually, and would separate images where the transformation makes the light invisible even to humans before counting a detection failure as a bug.
  • Because per-transformation mAP drops range from about 17% for scaling to 64% for rotation, a practitioner should interpret the 39.8% average as a portfolio effect and prioritize transformations that match the operating environment.
  • A testable extension is to measure whether the 67.5% retraining gain persists under distribution shift, for example by applying TigAug transformations to a third dataset collected in a different country or season.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes TigAug, a data augmentation tool for testing traffic-light detection models in autonomous driving systems. TigAug defines two families of metamorphic relations (Eqs. 1-2) and twelve transformations grouped into weather, camera, and traffic-light families, then uses the augmented images in two ways: to detect erroneous behaviors of existing detectors (RQ1) and to retrain improved detectors (RQ2). The evaluation uses four detection models (YOLOv5, YOLOX, Faster R-CNN, SSD) on two datasets (LISA and Bosch). The headline results are that original models lose on average 39.8% mAP on augmented test data, and retrained models gain on average 67.5% mAP on that data; additional studies report synthesis/retraining time (RQ3) and manual naturalness of the generated images (RQ4). The paper claims TigAug is effective and efficient for testing traffic-light detection models, and it makes code and data available.

Significance. If the central claims are validated, TigAug would be a useful contribution to a relatively under-studied area of ADS testing: automated metamorphic testing of traffic-light detectors. The work has several concrete strengths: the transformation families are domain-specific and reasonably motivated; the evaluation spans four architectures and two datasets; the effects are large and consistent across those conditions; and the artifact is publicly available. The paper also honestly documents the naturalness limitations of some transformations. However, the main testing-effectiveness claim rests on a fragile proxy: dataset-level mAP drops on augmented images are treated as evidence of erroneous behaviors, without per-image metamorphic-relation checking and despite the authors' own admission that some augmented images make traffic lights invisible to humans. The retraining result is also measured on the same augmented distribution used for training. These issues affect the two headline quantitative claims, so the paper needs substantive revision rather than minor polishing.

major comments (4)
  1. [Sec. 2.1 and Sec. 3.3 (RQ1)] The metamorphic relations in Eqs. (1)-(2) are per-image statements quantified over i in I, but the RQ1 evaluation never checks them per image. The only quantitative evidence is the aggregate mAP drop on augmented testing datasets (e.g., the 39.8% average drop in Sec. 3.3). No per-image violation counts, consistency rates, or matched-detection comparisons are reported, and the "Detected Erroneous Behaviors" paragraph only shows selected examples in Fig. 4. A mAP drop is compatible with many phenomena other than metamorphic-relation violations, so as written the data do not support the claim that TigAug detects erroneous behaviors. Please report per-image metamorphic checks (e.g., detection-set comparison before/after each transformation) and quantify the fraction of images violating each relation.
  2. [Sec. 3.7 and Sec. 3.3 (RQ1)] The authors concede in Sec. 3.7 that "the large part of unnatural images synthesized by SW and FG are those that make traffic lights invisible" and that "it seems meaningless for models to detect" them. For such inputs, the expected detection output is undefined or impossible, so a detection failure is not evidence of a model bug. Because these images are included in the augmented test sets used for the headline mAP-drop numbers, the 39.8% figure conflates genuine detector faults with failures on impossible inputs. The RQ1 analysis should either exclude such images or report results separately for human-visible and human-invisible augmented images, with a precise statement of which transformations are affected.
  3. [Sec. 3.4 (RQ2)] The retraining evaluation is partly circular: models are retrained on augmented training data generated by a transformation tau and evaluated on augmented test data generated by the same transformation family. An mAP gain on that test distribution is expected from matching the training and test covariate shifts and does not by itself demonstrate that TigAug improves robustness more broadly. The authors do report that retrained models retain similar mAP on the original test sets, which is useful, but the 67.5% gain claim needs additional support. Please add evaluation on held-out transformation intensities, on combinations of transformations, or on naturally degraded data, and compare against a standard augmentation baseline.
  4. [Sec. 3.3-3.4 (evaluation design)] No existing augmentation or metamorphic-testing approach is compared as a baseline. The related work discusses DeepTest [19], LiRTest [22], and Bai et al. [28], but the experiments do not include any of them. Consequently, the claim that TigAug is effective for testing and retraining cannot be distinguished from the generic effect of adding transformed training data. A baseline comparison (e.g., DeepTest-style affine/weather transformations, or standard flip/color-jitter augmentation) is needed to support the paper's specificity claim.
minor comments (5)
  1. [Sec. 2.1] The statement "we use mean average precision to derive the equality criterion E" is imprecise: Eqs. (1)-(2) define E as a per-image predicate, while mAP is a dataset-level aggregate. Please specify how per-image outputs are compared (e.g., matched bounding boxes with IoU and label agreement) and then how those per-image results are aggregated.
  2. [Sec. 3.3-3.6] Reported mAP values are group means without variance estimates or confidence intervals; since each model/dataset combination appears to be trained once, adding repeated runs or at least error bars would strengthen the quantitative claims.
  3. [Sec. 2.2 and Sec. 3.7] The transformation parameters (e.g., severity=2 for SW/FG, kernel size 15 for MB, offset equal to box width for MP) are fixed without sensitivity analysis; the threat-to-validity paragraph acknowledges this, but a small parameter sweep for one or two transformations would make the robustness claim more convincing.
  4. [Sec. 3.6] The naturalness study is conducted only on LISA, not Bosch; this limitation is mentioned, but it would be helpful to state explicitly in the RQ4 summary that the 27.9% unnatural-image figure applies only to LISA.
  5. [Sec. 4] Bai et al. [28] is discussed as the closest prior work but is not included in any experimental comparison; even a small case study of the color-change transformation against that method would clarify the incremental contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the testing and retraining claims are empirical measurements on external datasets, not derivations that reduce to their own inputs.

full rationale

TigAug's central assertions are supported by measured mAP differences on held-out LISA and Bosch splits (Sec. 3.3-3.4), not by a chain of definitions or by fitting parameters to the evaluated quantity. The metamorphic relations in Sec. 2.1 are oracle specifications; RQ1's aggregate mAP drop and RQ2's retraining gains are empirical outcomes, and no equation in the paper equates a prediction with an input by construction. There are no load-bearing self-citations or imported uniqueness theorems. Two apparent concerns are validity threats rather than circularity: (i) RQ2 retrains on TigAug-augmented data and tests on TigAug-augmented data from the same transformation families, making the 67.5% gain unsurprising but still an empirical generalization result; (ii) RQ1 treats mAP drop as evidence of erroneous behaviors although the MRs are per-image, and Sec. 3.7 concedes that 'the large part of unnatural images synthesized by SW and FG are those that make traffic lights invisible' and 'it seems meaningless for models to detect' them, so part of the drop reflects impossible inputs rather than detector faults. These are construct-validity limitations, not reductions-by-definition, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical claims depend on hand-set transformation intensities, on the invariance assumptions in Eqs. 1 and 2, and on treating dataset-level mAP as a proxy for MR violations. No new physical entities are introduced.

free parameters (3)
  • Weather and camera transformation intensities = RN drop_size=(0.1,0.2), speed=(0.2,0.3); SW/FG severity=2; OE severity=4; UE severity=1; MB kernel k=15
    Chosen by hand to make augmented images look natural; not swept or benchmarked. They directly control how much mAP drops, so RQ1 and RQ2 results depend on them.
  • Traffic light transformation geometry = MP offset delta = bbox width; SC canvas expansion +320x+180; RT and CC rotation rules
    Hand-set placement and scaling choices with no sensitivity analysis reported.
  • Retraining mix fraction = 20% of each augmented training dataset merged into the original training dataset
    Arbitrary and likely favorable; more augmented data could change the magnitude of retraining gains.
assumptions (3)
  • domain assumption Weather and camera transformations preserve traffic light position and state, so a correct detector should output the same boxes and labels (Eq. 1).
    Load-bearing for RQ1. The paper later concedes some SW and FG images make traffic lights invisible even to humans, so the invariance assumption is violated for the most aggressive transformations.
  • domain assumption Traffic light transformations change outputs in a prescribed way (Eq. 2) while respecting regulation, such as opposite color and red on top or left.
    Used for CC, RT, MP, AD, and SC; if the semantic edits are unrealistic, detected erroneous behaviors may be artifacts of the edit rather than detector faults.
  • ad hoc to paper Dataset-level mAP on augmented test images is a valid proxy for metamorphic-relation violations.
    No per-image MR check is reported; mAP drop under distribution shift is treated as evidence of erroneous behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systems." pith.science (2026). https://pith.science/paper/ERBAKLQT

@misc{pith2026250705932,
  author       = {Pith},
  title        = {Pith review of: TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ERBAKLQT}},
  note         = {Machine review of arXiv:2507.05932}
}
read the original abstract

Autonomous vehicle technology has been developed in the last decades with recent advances in sensing and computing technology. There is an urgent need to ensure the reliability and robustness of autonomous driving systems (ADSs). Despite the recent achievements in testing various ADS modules, little attention has been paid on the automated testing of traffic light detection models in ADSs. A common practice is to manually collect and label traffic light data. However, it is labor-intensive, and even impossible to collect diverse data under different driving environments. To address these problems, we propose and implement TigAug to automatically augment labeled traffic light images for testing traffic light detection models in ADSs. We construct two families of metamorphic relations and three families of transformations based on a systematic understanding of weather environments, camera properties, and traffic light properties. We use augmented images to detect erroneous behaviors of traffic light detection models by transformation-specific metamorphic relations, and to improve the performance of traffic light detection models by retraining. Large-scale experiments with four state-of-the-art traffic light detection models and two traffic light datasets have demonstrated that i) TigAug is effective in testing traffic light detection models, ii) TigAug is efficient in synthesizing traffic light images, and iii) TigAug generates traffic light images with acceptable naturalness.

Figures

Figures reproduced from arXiv: 2507.05932 by the authors.

Figure 1
Figure 1. Approach Overview of TigAug 2. Methodology We design TigAug as a systematic data augmentation approach to automatically aug￾ment labeled traffic light images for testing traffic light detection models in ADSs [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Sample Traffic Light Images Synthesized by Our Transformations [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. mAP Comparison of the Original Models between Original and Augmented Testing Datasets [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Samples of Traffic Light Images Revealing Erroneous Behaviors Found by [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: mAP Comparison between Original and Retrained Models on the Original Testing Datasets [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: mAP Comparison between Original and Retrained Models on the Augmented Testing Datasets [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: mAP Comparison between Original Model, Models Retrained by Augmented Datasets Separately, [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 9
Figure 9. Figure 9: mAP of the Original Model and Retrained YOLOv5 Models (by One Transformation) on the Original LISA Testing Dataset When model is retrained using all transformations together, the average retraining time increases to 36 hours. Notice that we provide detailed time cost o…
Figure 10
Figure 10. Figure 10: mAP of the Original Model and Retrained YOLOv5 Models (by One Transformation) on the Augmented and Cleaned Augmented LISA Testing Dataset symbol, e.g., RN+) as well as on the cleaned augmented LISA testing dataset (denoted with a ‘-’ symbol, e.g., RN-). First, the ori…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 63 canonical work pages

  1. [19]

    Y. Tian, K. Pei, S. Jana, B. Ray, Deeptest: Automated testing of deep-neural-network-driven au- tonomous cars, in: Proceedings of the 40th international conference on software engineering, 2018, pp. 303–314

    work page 2018

  2. [22]

    A. Guo, Y. Feng, Z. Chen, Lirtest: augmenting lidar point clouds for automated testing of autonomous driving systems, in: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 480–492

    work page 2022

  3. [28]

    T. Bai, Y. Fan, Y. Pan, M. Qing, Metamorphic testing for traffic light recognition in autonomous driving systems, in: Proceedings of the IEEE 21st International Conference on Software Quality, Reliability and Security Companion, 2021, pp. 38–44

    work page 2021

  4. [1]

    Paden, M

    B. Paden, M. ˇC´ ap, S. Z. Yong, D. Yershov, E. Frazzoli, A survey of motion planning and control techniques for self-driving urban vehicles, IEEE Transactions on intelligent vehicles 1 (2016) 33–55

    work page 2016

  5. [2]

    Badue, R

    C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. M. Paixao, F. Mutz, et al., Self-driving cars: A survey, Expert Systems with Applications 165 (2021) 113816

    work page 2021

  6. [3]

    Atherton, Incident 434: Sudden braking by tesla allegedly on self-driving mode caused multi-car pileup in tunnel, in: K

    D. Atherton, Incident 434: Sudden braking by tesla allegedly on self-driving mode caused multi-car pileup in tunnel, in: K. Lam (Ed.), AI Incident Database, Responsible AI Collaborative, 2022. URL: https://incidentdatabase.ai/cite/434/

    work page 2022

  7. [4]

    Khemka, Incident 347: Waymo self-driving taxi behaved unexpectedly, driving away from support crew, in: K

    S. Khemka, Incident 347: Waymo self-driving taxi behaved unexpectedly, driving away from support crew, in: K. Lam (Ed.), AI Incident Database, Responsible AI Collaborative, 2021. URL:https: //incidentdatabase.ai/cite/347/

    work page 2021

  8. [5]

    McGregor, Incident 4: Uber av killed pedestrian in arizona, in: S

    S. McGregor, Incident 4: Uber av killed pedestrian in arizona, in: S. McGregor (Ed.), AI Incident Database, Responsible AI Collaborative, 2018. URL:https://incidentdatabase.ai/cite/4

    work page 2018

Show all 64 references
  1. [6]

    P. Kaur, S. Taghavi, Z. Tian, W. Shi, A survey on simulators for testing self-driving cars, in: Pro- ceedings of the Fourth International Conference on Connected and Autonomous Driving, 2021, pp. 62–70

    2021

  2. [7]

    P. Ji, R. Li, Y. Xue, Q. Dong, L. Xiao, R. Xue, Perspective, survey and trends: Public driving datasets and toolsets for autonomous driving virtual test, in: Proceedings of the IEEE International Intelligent Transportation Systems Conference, 2021, pp. 264–269

    2021

  3. [8]

    Ben Abdessalem, S

    R. Ben Abdessalem, S. Nejati, L. C. Briand, T. Stifter, Testing advanced driver assistance systems using multi-objective search and neural networks, in: Proceedings of the 31st IEEE/ACM international conference on automated software engineering, 2016, pp. 63–74

    2016

  4. [9]

    R. B. Abdessalem, A. Panichella, S. Nejati, L. C. Briand, T. Stifter, Testing autonomous cars for feature interaction failures using many-objective search, in: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, 2018, pp. 143–154

    2018

  5. [10]

    R. B. Abdessalem, S. Nejati, L. C. Briand, T. Stifter, Testing vision-based control systems using learnable evolutionary algorithms, in: Proceedings of the 40th International Conference on Software Engineering, 2018, pp. 1016–1026

    2018

  6. [11]

    Gladisch, T

    C. Gladisch, T. Heinz, C. Heinzemann, J. Oehlerking, A. von Vietinghoff, T. Pfitzer, Experience paper: Search-based testing in automated driving control applications, in: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering, 2019, pp. 26–37

    2019

  7. [12]

    J. C. Han, Z. Q. Zhou, Metamorphic fuzz testing of autonomous vehicles, in: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, 2020, pp. 380–385

    2020

  8. [13]

    G. Li, Y. Li, S. Jha, T. Tsai, M. Sullivan, S. K. S. Hari, Z. Kalbarczyk, R. Iyer, Av-fuzzer: Finding safety violations in autonomous driving systems, in: Proceedings of the IEEE 31st international symposium on software reliability engineering, 2020, pp. 25–36. 20

    2020

  9. [14]

    H. Tian, Y. Jiang, G. Wu, J. Yan, J. Wei, W. Chen, S. Li, D. Ye, Mosat: finding safety violations of autonomous driving systems using multi-objective genetic algorithm, in: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundati...

    2022

  10. [15]

    Y. Sun, C. M. Poskitt, J. Sun, Y. Chen, Z. Yang, Lawbreaker: An approach for specifying traffic laws and fuzzing autonomous vehicles, in: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12

    2022

  11. [16]

    Gambi, M

    A. Gambi, M. Mueller, G. Fraser, Automatically testing self-driving cars with search-based procedural content generation, in: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2019, pp. 318–328

    2019

  12. [17]

    Zhong, Z

    Z. Zhong, Z. Hu, S. Guo, X. Zhang, Z. Zhong, B. Ray, Detecting multi-sensor fusion errors in advanced driver-assistance systems, in: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 493–505

    2022

  13. [18]

    K. Pei, Y. Cao, J. Yang, S. Jana, Deepxplore: Automated whitebox testing of deep learning systems, in: Proceedings of the 26th Symposium on Operating Systems Principles, 2017, pp. 1–18

    2017

  14. [20]

    J. Shao, Testing object detection for autonomous driving systems via 3d reconstruction, in: Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering Companion, 2021, pp. 117– 119

    2021

  15. [21]

    S. Wang, Z. Su, Metamorphic object insertion for testing object detection systems, in: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 2020, pp. 1053– 1065

    2020

  16. [23]

    Redmon, A

    J. Redmon, A. Farhadi, Yolo9000: better, faster, stronger, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7263–7271

    2017

  17. [24]

    S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, in: Proceedings of the Advances in neural information processing systems, 2015, pp. 1–9

    2015

  18. [25]

    W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector, in: Proceedings of the 14th European Conference on Computer Vision, 2016, pp. 21–37

    2016

  19. [26]

    Lam, Incident 145: Tesla’s autopilot misidentified the moon as yellow stop light, in: S

    K. Lam, Incident 145: Tesla’s autopilot misidentified the moon as yellow stop light, in: S. McGregor (Ed.), AI Incident Database, Responsible AI Collaborative, 2021. URL:https://incidentdatabase. ai/cite/145

    2021

  20. [27]

    McGregor (Ed.), AI Incident Database, Responsible AI Collaborative, 2014

    Anonymous, Incident 8: Uber autonomous cars running red lights, in: S. McGregor (Ed.), AI Incident Database, Responsible AI Collaborative, 2014. URL:https://incidentdatabase.ai/cite/8

    2014

  21. [29]

    T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. Tse, Z. Q. Zhou, Metamorphic testing: A review of challenges and opportunities, ACM Computing Surveys 51 (2018) 1–27

    2018

  22. [30]

    Segura, G

    S. Segura, G. Fraser, A. B. Sanchez, A. Ruiz-Cort´ es, A survey on metamorphic testing, IEEE Trans- actions on software engineering 42 (2016) 805–824

    2016

  23. [31]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. Williams, J. Winn, A. Zisserman, The pascal visual object classes (voc) challenge, International journal of computer vision 88 (2009) 303–308

    2009

  24. [32]

    T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, C. L. Zitnick, Microsoft coco: Common objects in context, in: Proceedings of the 13th European Conference on Computer Vision, 2014, pp. 740–755. 21

    2014

  25. [33]

    A. B. Jung, K. Wada, J. Crall, S. Tanaka, J. Graving, C. Reinders, S. Yadav, J. Banerjee, G. Vecsei, A. Kraft, Z. Rui, J. Borovec, C. Vallentin, S. Zhydenko, K. Pfeiffer, B. Cook, I. Fern´ andez, F.-M. De Rainville, C.-H. Weng, A. Ayala-Acevedo, R. Meudec, M. Laporte, et al., ...

    2020

  26. [34]

    URL:https://www.adobe.com/ creativecloud/photography/discover/lens-flare.html

    Adobe, How to add a lens flare in adobe photoshop, 2023. URL:https://www.adobe.com/ creativecloud/photography/discover/lens-flare.html

    2023

  27. [35]

    M. B. Jensen, M. P. Philipsen, A. Møgelmose, T. B. Moeslund, M. M. Trivedi, Vision for looking at traffic lights: Issues, survey, and perspectives, IEEE Transactions on Intelligent Transportation Systems 17 (2016) 1800–1815

    2016

  28. [36]

    M. P. Philipsen, M. B. Jensen, A. Møgelmose, T. B. Moeslund, M. M. Trivedi, Traffic light detec- tion: A learning algorithm and evaluations on challenging dataset, in: Proceedings of the IEEE 18th international conference on intelligent transportation systems, 2015, pp. 2341–2345

    2015

  29. [37]

    Behrendt, L

    K. Behrendt, L. Novak, R. Botros, A deep learning approach to traffic lights: Detection, tracking, and classification, in: Proceedings of the 2017 IEEE International Conference on Robotics and Automation, 2017, pp. 1370–1377

    2017

  30. [38]

    W. Pan, Y. Chen, B. Liu, Traffic light detection for self-driving vehicles based on deep learning, in: Proceedings of the 15th International Conference on Computational Intelligence and Security, 2019, pp. 63–67

    2019

  31. [39]

    J. Kim, H. Cho, M. Hwangbo, J. Choi, J. Canny, Y. P. Kwon, Deep traffic light detection for self- driving cars from a large-scale dataset, in: Prceedings of the 21st International Conference on Intelligent Transportation Systems, 2018, pp. 280–285

    2018

  32. [40]

    M¨ uller, K

    J. M¨ uller, K. Dietmayer, Detecting traffic lights by single shot detection, in: Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems, 2018, pp. 266–273

    2018

  33. [41]

    Z. Li, Q. Zeng, Y. Liu, J. Liu, L. Li, An improved traffic lights recognition algorithm for autonomous driving in complex scenarios, International Journal of Distributed Sensor Networks 17 (2021)

    2021

  34. [42]

    W. Ma, M. Papadakis, A. Tsakmalis, M. Cordy, Y. L. Traon, Test selection for deep learning systems, ACM Transactions on Software Engineering and Methodology 30 (2021) 1–22

    2021

  35. [43]

    Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13 (2004) 600–612

    2004

  36. [44]

    Zhang, P

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang, The unreasonable effectiveness of deep features as a perceptual metric, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018) 586–595

    2018

  37. [45]

    Dalal, B

    N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) 1 (2005) 886–893 vol. 1

    2005

  38. [46]

    Knauss, J

    A. Knauss, J. Schroder, C. Berger, H. Eriksson, Software-related challenges of testing automated vehicles, in: Proceedings of the IEEE/ACM 39th International Conference on Software Engineering Companion, 2017, pp. 328–330

    2017

  39. [47]

    K. Czarnecki, Software engineering for automated vehicles: Addressing the needs of cars that run on software and data, in: Proceedings of the IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings, 2019, pp. 6–8

    2019

  40. [48]

    Tahir, R

    Z. Tahir, R. Alexander, Coverage based testing for v&v and safety assurance of self-driving autonomous vehicles: A systematic literature review, in: Proceedings of the IEEE International Conference On Artificial Intelligence Testing, 2020, pp. 23–30

    2020

  41. [49]

    H. Tian, G. Wu, J. Yan, Y. Jiang, J. Wei, W. Chen, S. Li, D. Ye, Generating critical test scenarios for autonomous driving systems via influential behavior patterns, in: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12

    2022

  42. [50]

    Zhang, Y

    M. Zhang, Y. Zhang, L. Zhang, C. Liu, S. Khurshid, Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems, in: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, 2018, pp. 132–142

    2018

  43. [51]

    Goodfellow, J

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, Communications of the ACM 63 (2020) 139–144. 22

    2020

  44. [52]

    H. Zhou, W. Li, Z. Kong, J. Guo, Y. Zhang, B. Yu, L. Zhang, C. Liu, Deepbillboard: Systematic physical-world testing of autonomous driving systems, in: Proceedings of the ACM/IEEE 42nd Inter- national Conference on Software Engineering, 2020, pp. 347–358

    2020

  45. [53]

    Gambi, T

    A. Gambi, T. Huynh, G. Fraser, Generating effective test cases for self-driving cars from police reports, in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019, pp. 257–267

    2019

  46. [54]

    Secci, A

    F. Secci, A. Ceccarelli, On failures of rgb cameras and their effects in autonomous driving applications, in: Proceedings of the IEEE 31st International Symposium on Software Reliability Engineering, 2020, pp. 13–24

    2020

  47. [55]

    Y. Deng, X. Zheng, M. Zhang, G. Lou, T. Zhang, Scenario-based test reduction and prioritization for multi-module autonomous driving systems, in: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering,...

    2022

  48. [56]

    S. Tang, Z. Zhang, Y. Zhang, J. Zhou, Y. Guo, S. Liu, S. Guo, Y.-F. Li, L. Ma, Y. Xue, et al., A survey on automated driving system testing: Landscapes and trends, arXiv preprint arXiv:2206.05961 (2022,)

    2022 arXiv

  49. [57]

    Koopman, M

    P. Koopman, M. Wagner, Challenges in autonomous vehicle testing and validation, SAE International Journal of Transportation Safety 4 (2016) 15–24

    2016

  50. [58]

    G. Lou, Y. Deng, X. Zheng, M. Zhang, T. Zhang, Testing of autonomous driving systems: where are we and where should we go?, in: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022,, pp. 31–43

    2022

  51. [59]

    Dwarakanath, M

    A. Dwarakanath, M. Ahuja, S. Sikand, R. M. Rao, R. J. C. Bose, N. Dubash, S. Podder, Identify- ing implementation bugs in machine learning based image classifiers using metamorphic testing, in: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and...

    2018

  52. [60]

    Z. Sun, J. M. Zhang, M. Harman, M. Papadakis, L. Zhang, Automatic testing and improvement of machine translation, in: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 974–985

    2020

  53. [61]

    S. Chen, S. Jin, X. Xie, Validation on machine reading comprehension software without annotated labels: A property-based method, in: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ...

    2021

  54. [62]

    Z. Liu, Y. Feng, Z. Chen, Dialtest: Automated testing for recurrent-neural-network-driven dialogue systems, in: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, pp. 115–126

    2021

  55. [63]

    P. Ji, Y. Feng, J. Liu, Z. Zhao, Z. Chen, Asrtest: automated testing for deep-neural-network-driven speech recognition systems, in: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 189–201

    2022

  56. [64]

    B. Yu, Z. Zhong, X. Qin, J. Yao, Y. Wang, P. He, Automated testing of image captioning systems, in: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 467–479. 23

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.