pith. sign in

arxiv: 2604.12389 · v1 · submitted 2026-04-14 · 📡 eess.AS

VoxEffects: A Speech-Oriented Audio Effects Dataset and Benchmark

Zhe Zhang , Yigitcan \"Ozer , Junichi Yamagishi This is my paper

Pith reviewed 2026-05-10 14:36 UTC · model grok-4.3

classification 📡 eess.AS
keywords speech audio effectsaudio effect identificationeffect chain supervisiondatasetbenchmarkAudioMAErobustness evaluation
0
0 comments X

The pith

VoxEffects pairs produced speech waveforms with exact effect-chain annotations to enable precise identification of applied audio effects and parameters.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper creates VoxEffects to fill the gap in speech datasets that lack exact details on which audio effects and settings were used during production. It generates the data by starting with clean speech and applying known effects through a controlled rendering process, producing versions at different levels of detail about the effects. This setup supports a benchmark with three main tasks: spotting which effects are active, recognizing specific presets, and estimating how strongly each effect is applied. The work also tests how well models hold up when the audio is degraded by recording or platform issues. A baseline model based on AudioMAE is provided to show initial performance across these tasks and under varying conditions like input length and speaker gender.

Core claim

VoxEffects is a speech audio effects dataset that pairs produced speech with exact effect-chain supervision at multiple granularities, enabling speech-oriented audio effect identification where a model infers which effects are present and how they are applied from a given produced waveform. Built from minimally edited clean speech, the dataset supplies an extensible rendering pipeline for offline synthesis and on-the-fly rendering. The included benchmark covers effect presence detection, preset classification, and intensity prediction, together with a robustness protocol for capture-side and platform-side degradations.

What carries the argument

The extensible rendering pipeline that starts from clean speech and applies known effect chains to produce supervised waveforms at multiple annotation granularities.

If this is right

  • Models can be trained and evaluated to detect effect presence, classify presets, and predict intensity levels with exact ground-truth supervision.
  • The robustness protocol quantifies performance drops under capture-side and platform-side audio degradations.
  • The rendering pipeline supports both pre-generated datasets and on-the-fly data creation during model training.
  • Analyses quantify effects of domain shift, input duration, and speaker gender on identification accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The supervised effect chains could support development of tools that automatically reverse or adjust specific production effects in existing recordings.
  • Extending the same pipeline to music or other audio types would test whether the identification approach generalizes beyond speech.
  • The multi-granularity annotations make it possible to study how coarse versus fine effect labels affect model training efficiency.

Load-bearing premise

Effects applied via the rendering pipeline to minimally edited clean speech sufficiently represent real-world produced speech for the identification benchmark to be meaningful.

What would settle it

Run the trained identification models on a collection of real-world produced speech recordings where the applied effects and parameters are known independently and measure whether accuracy falls sharply relative to the synthetic test set.

Figures

Figures reproduced from arXiv: 2604.12389 by Junichi Yamagishi, Yigitcan \"Ozer, Zhe Zhang.

Figure 1
Figure 1. Figure 1: Overview of VoxEffects framework: dataset creation via a speech post-processing chain with curated presets, and benchmark evaluation via multi-granularity prediction. unknown effects [13] or predicting effect parameters from lan￾guage descriptions [14]. However, these efforts typically target music production regimes, specific implementations, or do not evaluate robustness to distribution artifacts such as… view at source ↗
Figure 2
Figure 2. Figure 2: Effect-wise radar analysis. Each panel reports evalu￾ation metrics on In-Domain vs. Out-of-Domain test. teristics (even in clean speech) affect how DN presets mod￾ify low-energy regions, thereby altering the DN-induced arti￾facts used for detection. For fine-grained preset classification, RVB transfers relatively well, while DRC and LIM show larger OOD drops, indicating higher sensitivity of preset recogni… view at source ↗
Figure 4
Figure 4. Figure 4: shows closely matched performance under both None and Both degradation settings. The primary performance drop is driven by degradations rather than gender. A larger audit with content-controlled matching is left for future work [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Presence detection accuracy with various input dura￾tion from 0.2 s to 5 s on OOD test [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
read the original abstract

Speech audio in the wild is often processed by post-production effects, but existing speech datasets rarely provide precise annotations of effects and parameters, limiting systematic study. We introduce VoxEffects, a speech audio effects dataset that pairs produced speech with exact effect-chain supervision at multiple granularities. VoxEffects supports speech-oriented audio effect identification: given a produced waveform, infer which effects are present and how they are applied. Built from minimally edited clean speech, it provides an extensible rendering pipeline for both offline synthesis and on-the-fly rendering for efficient training and evaluation. The audio effect identification benchmark includes effect presence detection, preset classification, and intensity prediction, with a robustness protocol covering capture-side and platform-side degradations. We provide an AudioMAE-based multi-task baseline and analyses of domain shift, robustness, input duration, and gender fairness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces VoxEffects, a dataset pairing produced speech waveforms with exact effect-chain supervision at multiple granularities. It is built from minimally edited clean speech via an extensible rendering pipeline supporting offline and on-the-fly synthesis, and establishes an audio effect identification benchmark covering effect presence detection, preset classification, and intensity prediction. The work includes an AudioMAE-based multi-task baseline together with analyses of domain shift, robustness to capture/platform degradations, input duration, and gender fairness.

Significance. If the synthetic rendering pipeline produces outputs whose effect distributions and interactions match those of real post-production speech, the dataset would address a clear gap in existing speech corpora by enabling precise supervised study of audio effects. The multi-granularity annotations and robustness protocol could support development of more reliable effect-identification and restoration models for wild audio, while the provided baseline and analyses would serve as a reproducible starting point for the community.

major comments (2)
  1. [Abstract] Abstract: The central claim that VoxEffects supplies 'exact effect-chain supervision at multiple granularities' and thereby enables a 'meaningful identification benchmark' rests on the assumption that effects applied via the rendering pipeline to minimally edited clean speech are representative of real-world produced speech. No validation is reported that compares the chosen effect chains, parameter ranges, or correlations against distributions observed in actual studio, broadcast, or podcast material.
  2. [Abstract] Abstract and robustness protocol description: While domain-shift and robustness analyses are mentioned, the manuscript provides no quantitative evidence (e.g., statistical tests or distribution comparisons) that the synthetic effect parameters reproduce nonlinear interactions, room acoustics, or artistic choices typical of real post-production, which directly affects whether benchmark performance generalizes beyond the pipeline.
minor comments (1)
  1. [Abstract] The abstract would be strengthened by including one or two key quantitative results (e.g., baseline F1 or MSE on the identification tasks) so readers can immediately gauge benchmark difficulty.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the need to better contextualize the synthetic nature of VoxEffects relative to real post-production speech. We address each major comment below and outline planned revisions to improve clarity without overstating the dataset's fidelity to real-world distributions.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that VoxEffects supplies 'exact effect-chain supervision at multiple granularities' and thereby enables a 'meaningful identification benchmark' rests on the assumption that effects applied via the rendering pipeline to minimally edited clean speech are representative of real-world produced speech. No validation is reported that compares the chosen effect chains, parameter ranges, or correlations against distributions observed in actual studio, broadcast, or podcast material.

    Authors: We agree that the manuscript does not report direct quantitative validation (e.g., distribution comparisons or statistical tests) of the synthetic effect chains against real studio, broadcast, or podcast material. This is a fair observation. The dataset's core value lies in supplying exact multi-granularity supervision, which real post-production recordings almost never provide. Effect types, presets, and parameter ranges were selected to span commonly used speech-oriented processing based on standard audio engineering references rather than to statistically match any particular real-world corpus. In the revised version we will (i) expand the abstract to explicitly state that the benchmark is a controlled testbed, (ii) add a new subsection under Dataset Construction that documents the parameter-selection rationale with citations to production practices, and (iii) clarify the intended scope of generalization. These changes address the concern without requiring unavailable real annotated data. revision: partial

  2. Referee: [Abstract] Abstract and robustness protocol description: While domain-shift and robustness analyses are mentioned, the manuscript provides no quantitative evidence (e.g., statistical tests or distribution comparisons) that the synthetic effect parameters reproduce nonlinear interactions, room acoustics, or artistic choices typical of real post-production, which directly affects whether benchmark performance generalizes beyond the pipeline.

    Authors: The existing domain-shift and robustness experiments evaluate model behavior under controlled degradations that approximate capture- and platform-side distortions, but we acknowledge they do not include formal statistical comparisons of nonlinear interactions, room acoustics, or artistic intent against real post-production material. The rendering pipeline employs standard DSP models for these phenomena; however, artistic choices remain inherently variable. In revision we will augment the robustness-protocol description with additional detail on the modeled interactions and add an explicit limitations paragraph stating that performance on highly stylized or complex real-world productions may require domain adaptation. No new quantitative real-data comparisons are possible without ground-truth annotations that do not exist, but the textual expansions will better bound the claims about generalization. revision: partial

Circularity Check

0 steps flagged

No circularity: dataset and benchmark introduction with no derivations

full rationale

The paper introduces VoxEffects as a new dataset pairing produced speech with effect-chain annotations and supplies an identification benchmark plus baseline. No mathematical derivations, first-principles results, fitted parameters, or predictions are claimed anywhere in the abstract or described structure. The dataset construction is presented as an explicit pipeline starting from clean speech; this is definitional resource creation rather than any reduction of an output to its inputs by construction. No self-citation load-bearing steps, uniqueness theorems, or ansatzes appear. The work is self-contained as a data resource and evaluation protocol against external benchmarks, warranting a score of 0.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper's contribution is the creation and annotation of a new dataset rather than a derivation from axioms; the baseline relies on standard pre-trained audio models.

axioms (1)
  • domain assumption Audio effects applied to clean speech can be precisely controlled and annotated to simulate produced speech.
    The dataset is built from minimally edited clean speech with added effects via a rendering pipeline.

pith-pipeline@v0.9.0 · 5437 in / 1235 out tokens · 47437 ms · 2026-05-10T14:36:12.324477+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages · 1 internal anchor

  1. [1]

    real vs. fake

    Introduction Real-world speech audio is rarely “raw.” In practice, recordings are typically processed by a small set of quality-oriented post- production effects. These operations improve intelligibility and perceived broadcast quality, but also introduce audio artifacts and shift signal statistics that downstream systems rely on. In this paper, we studya...

    work page

  2. [2]

    VoxEffects: A Speech-Oriented Audio Effects Dataset and Benchmark

    VoxEffects Dataset V oxEffects is a speech-oriented audio effects dataset and bench- mark that pairs produced speech with multi-granularity super- 1https://github.com/nii-yamagishilab/ VoxEffects arXiv:2604.12389v1 [eess.AS] 14 Apr 2026 vision. It includes a reproducible pipeline supporting both of- fline synthesis and on-the-fly rendering for scalable tr...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  3. [3]

    Audio Effect Identification Benchmark 3.1. Audio effect identification tasks Main task: effect presence detection.We define a binary pres- ence vectory∈ {0,1} |V| from the effect-chain configuration: ye=1if effecteis active (i.e., its preset is notbypass), and ye=0otherwise. Given an input waveformx, the model pre- dicts per-effect probabilities ˆy∈[0,1] ...

    work page

  4. [4]

    AudioMAE-Fx: fine-tuning AudioMAE for speech AEI We proposeAudioMAE-Fx, a strong baseline that fine-tunes AudioMAE [15] for speech AEI on V oxEffects

    Baseline Model 4.1. AudioMAE-Fx: fine-tuning AudioMAE for speech AEI We proposeAudioMAE-Fx, a strong baseline that fine-tunes AudioMAE [15] for speech AEI on V oxEffects. Given an input waveform, we extract log-mel filterbank features and feed them to the AudioMAE backbone. AudioMAE-Fx is trained in a multi-task manner with lightweight prediction heads on...

    work page

  5. [5]

    Experimental setup We evaluate AudioMAE-Fx on (i) ID test splits of DAPS/EARS/TSP and (ii) an OOD VCTK test set, re- porting results under the five degradation settings in § 2.3

    Evaluation and Discussion 5.1. Experimental setup We evaluate AudioMAE-Fx on (i) ID test splits of DAPS/EARS/TSP and (ii) an OOD VCTK test set, re- porting results under the five degradation settings in § 2.3. Because evaluating all utterances under all 2520 preset tuples is expensive, we report results on fixed subsets: 60 ID utterances (20 per corpus) a...

    work page

  6. [6]

    Limitations V oxEffects has several limitations. First, it assumes a fixed post-production chain and a finite preset bank, which yields tractable supervision but does not cover alternative orderings, repeated stages, or continuously tuned parameters found in real workflows. Second, rendering relies on a single effect imple- mentation stack (Pedalboard[20]...

    work page

  7. [7]

    Conclusion We introduced V oxEffects, a speech-oriented dataset and benchmark for AEI, where the goal is to infer which post- production effects are present in a processed waveform and how strongly they are applied. V oxEffects is built from clean speech recordings and a quality-oriented speech post-production chain with curated preset banks, providing ex...

    work page

  8. [8]

    This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo

    Acknowledgments This study is supported by a project (JPNP22007) commis- sioned by the New Energy and Industrial Technology Devel- opment Organization (NEDO). This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo

    work page

  9. [9]

    A new approach to voice authenticity,

    N. M. M ¨uller, P. Kawa, S. Hu, M. Neu, J. Williams, P. Sperl, and K. B¨ottinger, “A new approach to voice authenticity,” inProceed- ings of the Annual Conference of the International Speech Com- munication Association (Interspeech), Dublin, Ireland, 2024, pp. 2245–2249

    work page 2024

  10. [10]

    Audio forgery detec- tion techniques: Present and past review,

    P. R. Bevinamarad and M. S. Shirldonkar, “Audio forgery detec- tion techniques: Present and past review,” inProceedings of the International Conference on Trends in Electronics and Informat- ics (ICOEI), Tirunelveli, India, 2020, pp. 613–618

    work page 2020

  11. [11]

    Digital multimedia au- dio forensics: Past, present and future,

    M. Zakariah, M. K. Khan, and H. Malik, “Digital multimedia au- dio forensics: Past, present and future,”Multimedia Tools and Ap- plications, pp. 1009–1040, 2018

    work page 2018

  12. [12]

    Edit detection in speech recordings via instantaneous electric network frequency variations,

    P. A. A. Esquef, J. A. Apolin ´ario, and L. W. P. Biscainho, “Edit detection in speech recordings via instantaneous electric network frequency variations,”IEEE Transactions on Information F oren- sics and Security, pp. 2314–2326, 2014

    work page 2014

  13. [13]

    Differentiable black-box and gray-box modeling of nonlinear audio effects,

    M. Comunit `a, C. J. Steinmetz, and J. D. Reiss, “Differentiable black-box and gray-box modeling of nonlinear audio effects,” Frontiers in Signal Processing, vol. 5, 2025

    work page 2025

  14. [14]

    Blind estimation of audio effects us- ing an auto-encoder approach and differentiable digital signal pro- cessing,

    C. Peladeau and G. Peeters, “Blind estimation of audio effects us- ing an auto-encoder approach and differentiable digital signal pro- cessing,” inProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, South Korea, 2024, pp. 1–5

    work page 2024

  15. [15]

    Blind acoustic parameter estimation through task-agnostic embeddings using latent approximations,

    P. G ¨otz, C. Tuna, A. Brendel, A. Walther, and E. A. P. Ha- bets, “Blind acoustic parameter estimation through task-agnostic embeddings using latent approximations,” inProceedings of the International Workshop on Acoustic Signal Enhancement (IWAENC), Bayreuth, Germany, 2024, pp. 289–293

    work page 2024

  16. [16]

    Automatic recognition of cascaded guitar effects,

    J. Guo and B. McFee, “Automatic recognition of cascaded guitar effects,” inProceedings of the International Conference on Digi- tal Audio Effects (DAFx), Copenhagen, Denmark, 2023

    work page 2023

  17. [17]

    Guitar effects recog- nition and parameter estimation with convolutional neural net- works,

    M. Comunit `a, D. Stowell, and J. D. Reiss, “Guitar effects recog- nition and parameter estimation with convolutional neural net- works,”Journal of the Audio Engineering Society, vol. 69, no. 7/8, pp. 594–604, 2021

    work page 2021

  18. [18]

    Recognizing guitar effects and their parameter settings,

    H. J ¨urgens, R. Hinrichs, and J. Ostermann, “Recognizing guitar effects and their parameter settings,” inProceedings of the In- ternational Conference on Digital Audio Effects (DAFx), Vienna, Austria, 2020

    work page 2020

  19. [19]

    Automatic de- tection of audio effects in guitar and bass recordings,

    J. Abeßer, M. Stein, C. Dittmar, and G. Schuller, “Automatic de- tection of audio effects in guitar and bass recordings,” inProceed- ings of the Audio Engineering Society Convention (AES), London, UK, 2010

    work page 2010

  20. [20]

    DiffV ox: A differentiable model for capturing and analysing vocal effects distributions,

    C. Yu, M. A. Mart ´ınez-Ram´ırez, J. Koo, B. Hayes, W. Liao, G. Fazekas, and Y . Mitsufuji, “DiffV ox: A differentiable model for capturing and analysing vocal effects distributions,” inPro- ceedings of the International Conference on Digital Audio Effects (DAFx), Ancona, Italy, 2025

    work page 2025

  21. [21]

    General purpose audio effect removal,

    M. Rice, C. J. Steinmetz, G. Fazekas, and J. D. Reiss, “General purpose audio effect removal,” inProceedings of the IEEE Work- shop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY , USA, 2023

    work page 2023

  22. [22]

    Can large language models predict audio effects parameters from natural language?

    S. Doh, J. Koo, M. A. Mart ´ınez-Ram´ırez, W. Liao, J. Nam, and Y . Mitsufuji, “Can large language models predict audio effects parameters from natural language?” inProceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), Tahoe City, CA, USA, 2025, pp. 1–5

    work page 2025

  23. [23]

    Masked autoencoders that lis- ten,

    P.-Y . Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, “Masked autoencoders that lis- ten,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 35, New Orleans, LA, USA, 2022, pp. 28 708– 28 720

    work page 2022

  24. [24]

    G. J. Mysore, “Can we automatically transform speech recorded on common consumer devices in real-world environments into professional production quality speech? A dataset, insights, and challenges,”IEEE Signal Processing Letters, vol. 22, no. 8, pp. 1006–1010, 2015

    work page 2015

  25. [25]

    Izhaki,Mixing Audio: Concepts, Practices, and Tools, 4th ed

    R. Izhaki,Mixing Audio: Concepts, Practices, and Tools, 4th ed. Focal Press, 2023

    work page 2023

  26. [26]

    Senior,Mixing Secrets for the Small Studio

    M. Senior,Mixing Secrets for the Small Studio. Focal Press, 2011

    work page 2011

  27. [27]

    R. A. Katz,Mastering Audio: The Art and the Science, 3rd ed. Focal Press, 2015

    work page 2015

  28. [28]

    Pedalboard,

    P. Sobot, “Pedalboard,” 2023. [Online]. Available: https: //zenodo.org/records/7817839

    work page arXiv 2023

  29. [29]

    EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and derever- beration,

    J. Richter, Y .-C. Wu, S. Krenn, S. Welker, B. Lay, S. Watan- abe, A. Richard, and T. Gerkmann, “EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and derever- beration,” inProceedings of the Annual Conference of the Inter- national Speech Communication Association (Interspeech), Kos, Greece, 2024, pp. 4873–4877

    work page 2024

  30. [30]

    TSP Speech Database,

    P. Kabal, “TSP Speech Database,” Telecommunications & Signal Processing Laboratory, Department of Electrical & Computer En- gineering, McGill University, Nov. 2018

    work page 2018

  31. [31]

    CSTR VCTK Cor- pus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92),

    C. Veaux, J. Yamagishi, and K. MacDonald, “CSTR VCTK Cor- pus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92),” 2019

    work page 2019

  32. [32]

    Audio Set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” inProceedings of the IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 2017

    work page 2017

  33. [33]

    Decoupled weight decay regulariza- tion,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” inInternational Conference on Learning Representations, 2019

    work page 2019

  34. [34]

    BEiT: BERT pre-training of image transformers,

    H. Bao, L. Dong, S. Piao, and F. Wei, “BEiT: BERT pre-training of image transformers,” inProceedings of the International Con- ference on Learning Representations (ICLR), Virtual Event, 2022

    work page 2022

  35. [35]

    Curricu- lum learning,

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curricu- lum learning,” inProceedings of the International Conference on Machine Learning (ICML), New York, NY , USA, 2009, pp. 41– 48

    work page 2009

  36. [36]

    We consider five augmenta- tion settings applied at training and/or evaluation time:None, Pre(pre-effect),Post(post-effect),Either(randomly pre or post), andBoth(pre and post)

    Appendix Table 2 reports a comprehensive robustness evaluation under a full train/test degradation grid. We consider five augmenta- tion settings applied at training and/or evaluation time:None, Pre(pre-effect),Post(post-effect),Either(randomly pre or post), andBoth(pre and post). Each cell is written asIn- Domain / Out-of-Domain, where In-Domain aggregat...

    work page