Pith. sign in

REVIEW 2 major objections 5 minor 63 references

Black-box ASR testing works better by interpolating TTS phoneme embeddings than by perturbing waveforms, matching white-box quality without gradients.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 15:12 UTC pith:E3O2BMCF

load-bearing objection Solid black-box ASR testing via phoneme-latent multi-objective search; the WavefoRm ablation is clean, the white-box claim is overstated. the 2 major comments →

arxiv 2607.09833 v1 pith:E3O2BMCF submitted 2026-07-10 cs.CR cs.LG

Generative Testing of Automated Speech Recognition Systems

Yanis Xabier Wilbrand Pe\~na , Oliver Wei{\ss}l , Andrea Stocco This is my paper
classification cs.CR cs.LG
keywords automatic speech recognitionblack-box testingadversarial audiotext-to-speech latent spacephoneme embeddingsmulti-objective optimizationperceptual qualityuntargeted attacks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Automatic speech recognition systems can be fooled by carefully crafted audio that still sounds natural to people. This paper shows that the right place to search for such test cases is not the raw waveform but the phoneme-level latent space of a modern text-to-speech model. By linearly mixing each phoneme embedding toward a noise direction and decoding with the original prosody and style, the method keeps the audio on the natural-speech manifold while changing the ASR transcript. Multi-objective search balances how much the meaning drifts against how little the audio is distorted. On short English sentences and a transformer ASR, the approach reaches 98 percent success with lower distortion and higher human-rated quality than waveform and prosody baselines, and stays competitive with a white-box gradient attack. The practical claim is that choosing a semantically aligned representation matters more than having access to model internals when the goal is realistic untargeted testing.

Core claim

Untargeted multi-objective search over per-phoneme interpolation weights in a TTS acoustic embedding space produces natural-sounding audio that induces semantic transcription failures in a black-box ASR at rates and quality comparable to white-box methods, outperforming waveform-space and prosody-feature black-box baselines.

What carries the argument

GATAS: per-phoneme scalar interpolation κ between ground-truth and noise-matched phoneme embeddings of StyleTTS2, decoded with fixed original prosody/style, optimized by NSGA-II on set-overlap semantic divergence and inverted PESQ.

Load-bearing premise

Linear mixes of the TTS phoneme embeddings, decoded with the original prosody and style, stay natural enough and still expose the same kinds of ASR failures outside the single small model and short balanced sentences used in the study.

What would settle it

Run the same interpolation-and-decode pipeline against a larger multi-speaker ASR or longer spontaneous speech and measure whether success rate and human MOS stay near the reported 98 percent / near-ground-truth levels, or collapse once the manifold or decoder assumptions break.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Black-box ASR testing can reach white-box levels of naturalness and success without gradients if the search space is a phoneme-level TTS latent.
  • Waveform or pure prosody search spaces force a quality-versus-divergence trade-off that latent phoneme interpolation largely avoids.
  • Untargeted objectives are sufficient and often preferable for open-vocabulary speech, because they avoid hard temporal alignment to a fixed target phrase.
  • Human studies remain necessary: automatic scores alone can miss the gap between signal destruction and controlled phonetic drift.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same phoneme-level manipulator could be reused as a controlled data-augmentation tool for hardening ASR models rather than only attacking them.
  • If failures concentrate on specific phoneme substitutions, the method supplies a diagnostic map of which acoustic dimensions the ASR is brittle to.
  • Downstream systems that consume ASR output (dialogue managers, voice commands) could be stress-tested by feeding these latent-generated transcripts without needing physical replay.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes GATAS, a black-box untargeted testing method for ASR systems that generates failure-inducing audio by linearly interpolating StyleTTS2 phoneme-level acoustic embeddings toward a noise reference matched to ground-truth statistics, with per-phoneme weights optimized by NSGA-II under SetOverlap (semantic divergence) and inverted PESQ (perceptual quality) objectives. Evaluation on 100 Harvard Sentences against Whisper-tiny compares GATAS to a matched WavefoRm ablation, adapted SMACK, and white-box PGD, reporting ~98% success with superior UTMOS and lower spectrogram/embedding distortion (Tables 1–2), human MOS near ground truth (Table 3), and competitive runtime under higher query budgets (Table 4, Fig. 5). The central claim is that phoneme-latent representation and multi-objective perceptual alignment matter more than gradient access for realistic ASR test generation.

Significance. If the representation-over-internals claim holds under fairer white-box controls, the work supplies a practical black-box generator of natural, failure-inducing ASR test cases and a clean ablation isolating search-space structure (WavefoRm). Strengths include the matched multi-objective design for black-box methods, statistical tests with effect sizes, human MOS (10 raters × 50 clips), qualitative spectrogram analysis, and a public replication package. The untargeted formulation is well-motivated for open-vocabulary ASR. Even with the white-box caveat, the black-box results and human validation are useful for ASR testing and adversarial audio research.

major comments (2)
  1. [Abstract, §5, Table 1, §4.3.3, §4.4 Eq. (8)] Abstract and §5 claim that GATAS is competitive with white-box methods and that “representation and perceptual alignment are more critical than access to model internals.” This rests almost entirely on PGD (Table 1, §4.3.3). PGD uses original parameters, a fixed 200-query budget, and no multi-objective early-stopping of the form SetOverlap ≤ 0.5 ∧ PESQnorm ≤ 0.2 that defines success for GATAS/WavefoRm (§4.4, Eq. 8). The reported UTMOS/dF-Spec edge of GATAS over PGD may therefore reflect mismatched objectives rather than structural superiority of the phoneme latent space. A white-box run that jointly optimizes the same two objectives (or is early-stopped on the same thresholds) is needed; without it the representation-over-internals claim is not supported by the evidence. The WavefoRm ablation cleanly isolates the search-space effect among black-box methods; the white-box comparison does
  2. [§4.6.2, §3.2, §4.3] External validity (§4.6.2) is limited to Whisper-tiny and short, phonetically balanced Harvard Sentences with a single-speaker StyleTTS2 pipeline. The claim that latent-space optimization yields realistic, generalizable test cases therefore rests on the untested assumption that StyleTTS2 phoneme embeddings, when linearly interpolated toward noise and decoded with fixed prosody/style, remain on the natural-speech manifold and induce failures beyond this narrow setting. At least one additional ASR (e.g., a larger Whisper variant or a non-transformer system) or a more diverse speech corpus is needed before the generality of the representation advantage can be asserted.
minor comments (5)
  1. [Throughout] Placeholder venue headers (“Conference’17, July 2017, Washington, DC, USA Trovato et al.”) appear repeatedly and should be removed.
  2. [§4.4] Typo “indipedent” → “independent” in the human-study description.
  3. [Table 2] Table 2 header has a garbled column label (“𝑑𝐶-Embmb”); fix for readability.
  4. [§4.6.3, §8] Replication package is cited as [41] with an anonymous link; ensure the final version points to a stable, non-anonymous archive.
  5. [§3.4, §4.2] Clarify whether PESQ is computed on TTS-resynthesized ground truth vs. original recordings, since all methods start from StyleTTS2 audio.

Circularity Check

0 steps flagged

No circularity: empirical black-box method whose success and quality claims are measured against external ASR outputs and independent perceptual metrics, not re-derived from fitted inputs.

full rationale

GATAS is an empirical testing method, not a first-principles derivation. The multi-objective search (SetOverlap + inverted PESQ) guides generation of candidate audio; success, UTMOS, spectrogram/embedding distances, and human MOS are then evaluated on the independent Whisper-tiny transcriptions and on separate quality models/human raters. The WavefoRm ablation uses identical objectives and early-stopping thresholds yet yields markedly worse quality, so the headline comparison is not forced by construction. No parameters are fitted to a data subset and then re-presented as predictions; no uniqueness theorem or ansatz is imported via self-citation to forbid alternatives; the few self-citations appear only in related-work discussion of generative testing and do not underwrite the central empirical claim. The paper is therefore self-contained against its external benchmarks.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central empirical claim rests on standard optimization and speech-quality machinery plus domain choices about which TTS latent to edit and what counts as semantic success. Free parameters are search/threshold knobs, not physical constants. No new physical entities are postulated; GATAS is a method. Load-bearing domain assumptions are that the chosen latent stays natural under interpolation and that the chosen metrics capture meaningful ASR failures.

free parameters (4)
  • early_stop SetOverlap threshold
    Success requires SetOverlap ≤ 0.5; chosen to require at least half of content words changed (§4.4). Directly defines when optimization stops and contributes to reported success/efficiency.
  • early_stop PESQnorm threshold
    PESQnorm ≤ 0.2 (~raw PESQ ≥ 3.5) is an author-selected quality gate aligned with prior audio testing work but still a free design choice affecting which solutions count as successful.
  • NSGA-II population size and max generations
    Population 100 (selected from {50,100,200}) and 100 generations set the query budget (up to 10k) and exploration dynamics; preliminary tuning is reported but values remain free hyperparameters.
  • per-phoneme interpolation vector κ and noise sampling
    κ ∈ [0,1]^p and h_noise ~ N(μ_GT, σ_GT) define the entire search parameterization; not fitted to a closed-form law but chosen as the attack representation.
axioms (4)
  • domain assumption StyleTTS2 phoneme-level acoustic embeddings h_text encode phonetic content such that linear interpolation toward noise, with other TTS features fixed, yields natural speech on the data manifold.
    Core design premise in §3.2 and Discussion; without it, latent search would not dominate waveform search on quality.
  • domain assumption PESQ (and UTMOS/human MOS) are valid proxies for perceptual similarity/naturalness of adversarial speech test cases.
    F2 objective and RQ1/RQ2 metrics depend on this standard speech-quality assumption (§3.4, §4.2).
  • ad hoc to paper Content-word SetOverlap after stop-word removal and WordNet lemmatization measures semantically meaningful transcription divergence for untargeted ASR testing.
    Authors define F1 this way (§3.4); alternative semantic metrics could change optimization trajectories and success labels.
  • domain assumption Black-box query access to final transcriptions only, with deterministic greedy decoding (temperature 0), is a valid evaluation setting for deployed ASR testing.
    Stated SUT assumptions in §3.1 and §4.3.2; non-determinism is noted as a minor internal-validity threat.
invented entities (1)
  • GATAS (phoneme-latent multi-objective ASR test generator) no independent evidence
    purpose: Name and package the proposed black-box test-generation pipeline combining StyleTTS2 h_text interpolation with NSGA-II on SetOverlap and PESQ.
    Methodological construct rather than a physical entity; independent evidence is the empirical evaluation itself, not an external predicted observable outside the paper’s experiments.

pith-pipeline@v1.1.0-grok45 · 22849 in / 3449 out tokens · 41311 ms · 2026-07-14T15:12:01.601603+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Generative Testing of Automated Speech Recognition Systems." pith.science (2026). https://pith.science/paper/E3O2BMCF

@misc{pith2026260709833,
  author       = {Pith},
  title        = {Pith review of: Generative Testing of Automated Speech Recognition Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3O2BMCF}},
  note         = {Machine review of arXiv:2607.09833}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Automatic speech recognition (ASR) systems have achieved high accuracy with transformer-based models, enabling deployment in critical applications. However, they remain vulnerable to adversarial manipulation, particularly in black-box settings where attacks must preserve perceptual naturalness. This work introduces GATAS, a black-box testing approach that generates failure inducing inputs by operating in the phoneme-level latent space of a text- to-speech model. Instead of perturbing waveforms directly, the approach interpolates latent representations to induce transcription errors while remaining within the manifold of natural speech. The attack is formulated as a multi-objective optimization problem balancing semantic divergence and perceptual quality. Our empirical evaluation against both white-box and black-box baselines shows that GATAS achieves a 98% success rate while producing lower distortion and higher perceptual quality, as confirmed by human studies. Despite operating without gradient access, GATAS remains competitive against white-box methods, highlighting that representation and perceptual alignment are more critical than access to model internals. Overall, our results demonstrate that untargeted latent-space optimization enables the efficient generation of realistic and effective test cases for ASR systems.

Figures

Figures reproduced from arXiv: 2607.09833 by Andrea Stocco, Oliver Wei{\ss}l, Yanis Xabier Wilbrand Pe\~na.

Figure 1
Figure 1. Figure 1: Overview of GATAS. The initial input text [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Simplified StyleTTS2 pipeline [31]. is available, including architecture, parameters, gradients, or inter￾mediate outputs. In short, the only observable output is the final transcription produced by the system. We focus on untargeted test generation, where any semantically meaningful deviation from the original transcription constitutes success, because this formulation reflects realistic scenarios, where … view at source ↗
Figure 3
Figure 3. Figure 3: Spectrogram comparison of the original audio versus the generated test case for each approach. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example change in frequency band amplitudes. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cumulative success rate (RQ3 ). RQ3 (efficiency): GATAS achieves competitive runtime while operating under substantially higher query budgets than fixed￾budget baselines, indicating more efficient optimization dynam￾ics compared to WavefoRm. While gradient-based methods (PGD) and constrained approaches (SMACK) are faster due to lower query requirements, GATAS benefits from early stopping and converges more… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 11 linked inside Pith

  1. [1]

    Hadi Abdullah, Muhammad Sajidur Rahman, Washington Garcia, Kevin Warren, Anurag Swarnim Yadav, Tom Shrimpton, and Patrick Traynor. 2021. Hear” no evil”, see” kenansville”: Efficient and transferable black-box attacks on speech recognition and voice identification systems. In2021 IEEE Symposium on Security and Privacy (SP) . IEEE, 712–729

    2021

  2. [2]

    Harsh Ahlawat, Naveen Aggarwal, and Deepti Gupta. 2025. Automatic Speech Recognition: A survey of deep learning techniques and approaches.International Journal of Cognitive Computing in Engineering 6 (2025), 201–237. doi:10.1016/j. ijcce.2024.12.007

    doi:10.1016/j 2025

  3. [3]

    Harsh Ahlawat, Naveen Aggarwal, and Deepti Gupta. 2025. Automatic Speech Recognition: A survey of deep learning techniques and approaches.International Journal of Cognitive Computing in Engineering 6 (2025), 201–237

    2025

  4. [4]

    Moustafa Alzantot, Bharathan Balaji, and Mani Srivastava. 2018. Did you hear that? adversarial examples against automatic speech recognition.arXiv preprint arXiv:1801.00554 (2018)

    Pith/arXiv arXiv 2018

  5. [5]

    Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guo- liang Chen, et al. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning . PMLR, 173–182

    2016

  6. [6]

    Muhammad Hilmi Asyrofi, Ferdian Thung, David Lo, and Lingxiao Jiang. 2020. Crossasr: Efficient differential testing of automatic speech recognition via text- to-speech. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 640–650

    2020

  7. [7]

    Muhammad Hilmi Asyrofi, Zhou Yang, and David Lo. 2021. Crossasr++: A mod- ular differential testing framework for automatic speech recognition. InProceed- ings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1575–1579

    2021

  8. [8]

    Shenglin Bao, Nan Jiang, Weijie Zhu, and Pei Zhang. 2024. Generative Model- Based Test Case Generation and Operational Testing for Deep Learning. In2024 5th International Conference on Big Data & Artificial Intelligence & Software En- gineering (ICBASE). IEEE, 565–570

    2024

  9. [9]

    Marie Biolková and Bac Nguyen. 2022. Neural predictor for black-box adversar- ial attacks on speech recognition.arXiv preprint arXiv:2203.09849 (2022)

    Pith/arXiv arXiv 2022

  10. [10]

    Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit . ” O’Reilly Media, Inc. ”

    2009

  11. [11]

    Igor Buzhinsky, Arseny Nerinovsky, and Stavros Tripakis. 2023. Metrics and methods for robustness evaluation of neural networks with generative models. Machine Learning 112, 10 (2023), 3977–4012

    2023

  12. [12]

    Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) . Ieee, 39–57

    2017

  13. [13]

    Nicholas Carlini and David Wagner. 2018. Audio adversarial examples: Targeted attacks on speech-to-text. In 2018 IEEE security and privacy workshops (SPW) . IEEE, 1–7

    2018

  14. [14]

    William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. 2015. Listen, attend and spell. arXiv preprint arXiv:1508.01211 (2015)

    Pith/arXiv arXiv 2015

  15. [15]

    Xingcheng Chen, Oliver Weissl, and Andrea Stocco. 2026. Feature-Aware Test Generation for Deep Learning Models. arXiv:2601.14081 [cs.SE] https://arxiv. org/abs/2601.14081

    arXiv 2026

  16. [16]

    Yuxuan Chen, Xuejing Yuan, Jiangshan Zhang, Yue Zhao, Shengzhi Zhang, Kai Chen, and XiaoFeng Wang. 2020. {Devil’s} whisper: A general approach for physical adversarial attacks against commercial black-box speech recognition devices. In 29th USENIX Security Symposium (USENIX Security 20) . 2667–2684

    2020

  17. [17]

    Peng Cheng, Yuwei Wang, Peng Huang, Zhongjie Ba, Xiaodong Lin, Feng Lin, Li Lu, and Kui Ren. 2024. Alif: Low-cost adversarial audio attacks on black-box speech platforms using linguistic features. In 2024 IEEE Symposium on Security and Privacy (SP) . IEEE, 1628–1645

    2024

  18. [18]

    Kalyanmoy Deb, Aravind Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation 6, 2 (2002), 182–197

    2002

  19. [19]

    Tianyu Du, Shouling Ji, Jinfeng Li, Qinchen Gu, Ting Wang, and Raheem Beyah

  20. [20]

    In Proceedings of the 15th ACM Asia conference on computer and communications security

    Sirenattack: Generating adversarial audio for end-to-end acoustic systems. In Proceedings of the 15th ACM Asia conference on computer and communications security. 357–369

  21. [21]

    Zheng Fang, Tao Wang, Lingchen Zhao, Shenyi Zhang, Bowen Li, Yunjie Ge, Qi Li, Chao Shen, and Qian Wang. 2024. Zero-query adversarial attack on black- box automatic speech recognition systems. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security . 630–644

    2024

  22. [22]

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572 (2014)

    Pith/arXiv arXiv 2014

  23. [23]

    Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber

  24. [24]

    InProceedings of the 23rd international con- ference on Machine learning

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. InProceedings of the 23rd international con- ference on Machine learning . 369–376

  25. [25]

    Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, An- drea Stocco, and Paolo Tonella. 2020. Taxonomy of Real Faults in Deep Learning Systems. In Proceedings of 42nd International Conference on Software Engineer- ing (Seoul, Republic of Korea) (ICSE ’20) . ACM, New York, NY, USA, 12 pages. doi:10.1145/3377811.3380395

    doi:10.1145/3377811.3380395 2020

  26. [26]

    ITU-T. 2003. Mapping function for transforming P.862 raw result scores to MOS-LQO. Recommendation P.862.1. International Telecommunication Union, Telecommunication Standardization Sector, Geneva, Switzerland

    2003

  27. [27]

    ITU-T. 2017. Recommendation ITU-T P.10: Vocabulary for performance and qual- ity of service. ITU-T Recommendation. International Telecommunication Union. https://www.itu.int/rec/T-REC-P.10/en

    2017

  28. [28]

    ITU-T Recommendation P.800. 1996. Methods for subjective determination of transmission quality. Technical Report. International Telecommunication Union, Geneva, Switzerland

    1996

  29. [29]

    Pin Ji, Yang Feng, Jia Liu, Zhihong Zhao, and Zhenyu Chen. 2022. ASRTest: automated testing for deep-neural-network-driven speech recognition systems. In Proceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis . 189–201

    2022

  30. [30]

    Sungmin Kang, Robert Feldt, and Shin Yoo. 2020. Sinvad: Search-based image space navigation for dnn image classifier test input generation. InProceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops . 521–528

    2020

  31. [31]

    Davenport, and Dessislava Pachamanova

    Julia Kokina, Shay Blanchette, Thomas H. Davenport, and Dessislava Pachamanova. 2025. Challenges and opportunities for artificial intelligence in auditing: Evidence from the field. International Journal of Accounting Informa- tion Systems 56 (2025), 100734. doi:10.1016/j.accinf.2025.100734

    doi:10.1016/j.accinf.2025.100734 2025

  32. [32]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis.Advances in neural information processing systems 33 (2020), 17022–17033

    2020

  33. [33]

    Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mes- garani. 2023. Styletts 2: Towards human-level text-to-speech through style dif- fusion and adversarial training with large speech language models.Advances in Neural Information Processing Systems 36 (2023), 19594–19621

    2023

  34. [34]

    Maryam Maryam, Matteo Biagiola, Andrea Stocco, and Vincenzo Riccio. 2025. Benchmarking Generative AI Models for Deep Learning Test Input Generation. In 2025 IEEE Conference on Software Testing, Verification and Validation (ICST) . IEEE, 174–185

    2025

  35. [35]

    Giorgi Merabishvili, Oliver Weißl, and Andrea Stocco. 2026. Latent Regulariza- tion in Generative Test Input Generation.arXiv preprint arXiv:2602.15552 (2026)

    arXiv 2026

  36. [36]

    George A Miller. 1995. WordNet: a lexical database for English.Commun. ACM 38, 11 (1995), 39–41

    1995

  37. [37]

    Sondess Missaoui, Simos Gerasimou, and Nicholas Matragkas. 2023. Semantic data augmentation for deep learning testing using generative AI. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 1694–1698

    2023

  38. [38]

    Raphael Olivier and Bhiksha Raj. 2022. There is more than one kind of robust- ness: Fooling whisper with adversarial examples.arXiv preprint arXiv:2210.17316 (2022)

    Pith/arXiv arXiv 2022

  39. [39]

    1993.Fundamentals of speech recog- nition

    Lawrence Rabiner and Biing-Hwang Juang. 1993.Fundamentals of speech recog- nition. Prentice-Hall, Inc

    1993

  40. [40]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak super- vision. In International conference on machine learning . PMLR, 28492–28518

    2023

  41. [41]

    Vyas Raina, Rao Ma, Charles McGhee, Kate Knill, and Mark Gales. 2024. Muting whisper: A universal acoustic adversarial attack on speech foundation models. arXiv preprint arXiv:2405.06134 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    Sai Sathiesh Rajan, Sakshi Udeshi, and Sudipta Chattopadhyay. 2022. Aequevox: Automated fairness testing of speech recognition systems. InInternational Con- ference on Fundamental Approaches to Software Engineering . Springer, 245–267

    2022

  43. [43]

    Replication Package

    rep-package 2026. Replication Package. https://anonymous.4open.science/r/ GATTS-6FA8

    2026

  44. [44]

    Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing Machine Learning based Sys- tems: A Systematic Mapping. Empirical Software Engineering (2020)

    2020

  45. [45]

    Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. 2001. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality Conference’17, July 2017, Washington, DC, USA Trovato et al. assessment of telephone networks and codecs. In2001 IEEE international confer- ence on acoustics, speech, and signal processing. Proceedin...

    2001

  46. [46]

    Ernst H Rothauser. 1969. IEEE recommended practice for speech quality mea- surements. IEEE Transactions on Audio and Electroacoustics17, 3 (1969), 225–246

    1969

  47. [47]

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022. InProceedings of Interspeech 2022

    2022

  48. [48]

    Lea Schönherr, Katharina Kohls, Steffen Zeiler, Thorsten Holz, and Dorothea Kolossa. 2018. Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding.arXiv preprint arXiv:1808.05665 (2018)

    Pith/arXiv arXiv 2018

  49. [49]

    Mona Sloane, Hilke Schellmann, Katelyn Xiaoying Mei, Anna Seo Gyeong Choi, and Allison Koenecke. 2026. The case for stakeholder-driven AI auditing in au- tomatic speech recognition. Nature Machine Intelligence (2026). doi:10.1038/ s42256-026-01207-x

    2026

  50. [50]

    Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. 2018. Constructing unrestricted adversarial examples with generative models. Advances in neural information processing systems 31 (2018)

    2018

  51. [51]

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Er- han, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural net- works. arXiv preprint arXiv:1312.6199 (2013)

    Pith/arXiv arXiv 2013

  52. [52]

    Rohan Taori, Amog Kamsetty, Brenton Chu, and Nikita Vemuri. 2019. Targeted adversarial examples for black box audio systems. In2019 IEEE security and pri- vacy workshops (SPW) . IEEE, 15–20

    2019

  53. [53]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  54. [54]

    Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135 (2017)

    Pith/arXiv arXiv 2017

  55. [55]

    Oliver Weißl, Amr Abdellatif, Xingcheng Chen, Giorgi Merabishvili, Vincenzo Riccio, Severin Kacianka, and Andrea Stocco. 2025. Targeted deep learning sys- tem boundary testing. ACM Transactions on Software Engineering and Methodol- ogy (2025)

    2025

  56. [56]

    Oliver Weißl, Vincenzo Riccio, Severin Kacianka, and Andrea Stocco. 2026. HyperNet-Adaptation for Diffusion-Based Test Case Generation.arXiv preprint arXiv:2601.15041 (2026)

    Pith/arXiv arXiv 2026

  57. [57]

    Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods.Biometrics Bulletin 1, 6 (dec 1945), 80. doi:10.2307/3001968

    doi:10.2307/3001968 1945

  58. [58]

    Yi Xie, Zhuohang Li, Cong Shi, Jian Liu, Yingying Chen, and Bo Yuan. 2021. Enabling fast and universal audio adversarial attack using generative model. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 14129–14137

    2021

  59. [59]

    2023.{SMACK}: Semantically meaningful adversarial audio attack

    Zhiyuan Yu, Yuanhaur Chang, Ning Zhang, and Chaowei Xiao. 2023.{SMACK}: Semantically meaningful adversarial audio attack. In32nd USENIX security sym- posium (USENIX security 23) . 3799–3816

    2023

  60. [60]

    Guoming Zhang, Xiaohui Ma, Huiting Zhang, Riccardo Spolaor, Yanni Yang, Xi- aoyu Ji, Xiuzhen Cheng, and Pengfei Hu. 2025. UltraAdv: An Ultrasonic Adver- sarial Attack on Black-Box Speech Recognition Systems. IEEE Transactions on Mobile Computing (2025)

    2025

  61. [61]

    Guoming Zhang, Chen Yan, Xiaoyu Ji, Tianchen Zhang, Taimin Zhang, and Wenyuan Xu. 2017. Dolphinattack: Inaudible voice commands. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security . 103–117

    2017

  62. [62]

    Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2017. Generating natural adver- sarial examples. arXiv preprint arXiv:1710.11342 (2017)

    Pith/arXiv arXiv 2017

  63. [63]

    Baolin Zheng, Peipei Jiang, Qian Wang, Qi Li, Chao Shen, Cong Wang, Yunjie Ge, Qingyang Teng, and Shenyi Zhang. 2021. Black-box adversarial attacks on commercial speech platforms with minimal information. In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security. 86–107

    2021