REVIEW 3 major objections 5 minor 29 references
Designed Vocalizations Dataset: Sound-Designed Human and Animal Voices for Non-human Voice Conversion
T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read This paper introduces a public dataset of 231,800 sound-designed vocalizations for converting human voices into non-human ones, with a standardized benchmark to measure how well systems generalize to unseen styles and sources.
desk verdict A genuinely useful, honest dataset paper — the first public benchmark for designed-voice conversion — with a thin baseline evaluation and one split-overlap detail that needs clarifying. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is the preset-specific DSP operator G_p, a composition of effect modules (delay pitch shifting, flanger/chorus, granular, noise generator, pitch shifting, ring modulator, spectral shifting) arranged in serial, parallel, or hybrid chains. These operators define the mapping from raw vocalization to designed vocalization, and they make the benchmark's structure possible: because every reference is G_p applied to the same source, timbre conversion is evaluated independently of content mismatch, while the seen/unseen split over presets and source types probes generalization.
What would settle it
Take the released test set and compare two conditions: (1) the standard same-source pairs, and (2) pairs where the reference is the designed version of a different source with different content, matched by preset. If the cosine similarity or MOS drops substantially in condition (2), then the benchmark's aligned pairs are not a fair proxy for production use. A second, simpler check is to verify whether the 60 source-seen VCTK test clips are disjoint from the 3,270 VCTK clips used in training; overlapping clips would inflate seen-source scores.
Extended reading notes
Core claim
The central claim is that the Designed Vocalizations Dataset is a viable public resource for training and evaluating human-to-non-human voice conversion. Each designed sample is generated as x^(p) = G_p(x_raw), where G_p is a preset-specific DSP operator built from effect modules, and the test set uses reference samples that are the preset-processed versions of the same source: x_r = G_p(x_s). This construction makes content identical between source and reference, isolating the timbre-transfer task. The test set covers 120 sources and 47 presets—40 seen and 7 unseen—with source timbre groups split into seen and unseen categories, enabling systematic evaluation of how models handle unseen spe
Load-bearing premise
The benchmark assumes that a reference designed from the same source as the input is representative of how voice conversion models will actually be used, where the reference usually has different content—if same-content pairs artificially make the task easier, the reported conversion quality may overstate real-world performance.
Editorial extensions
If this is right
- Research groups can train and compare non-human voice conversion systems on a common public dataset instead of internally curated, unreleased data.
- The explicit seen/unseen splits let future work quantify how much performance drops when a model encounters an unseen style or an unseen source type, guiding progress toward production-ready generalization.
- The released preset-level effect summaries and full effect-chain parameters could enable research on effect-aware or interpretable voice conversion, where models estimate or control the underlying sound-design chain rather than only matching a target timbre.
- The baseline results establish a concrete reference point, so subsequent papers can state whether their method improves on the standard setting.
- Because the dataset includes non-linguistic sources like animal sounds and vocal mimicry, it expands voice conversion research beyond speech into broader sound-design use cases for games, film, and interactive media.
Reading between the lines
- A natural extension is to test whether the same-content pairing in the test set masks a key difficulty of real use: production references usually have different content from the input. A variant benchmark where reference and source are different utterances would reveal how much of the reported performance depends on content alignment.
- The dataset's structure could support an effect-decoding task: given the raw source and the designed output, can a model predict which preset was used? One could treat the preset labels as a supervision signal for learning interpretable timbre representations.
- The benchmark could be extended to a text-conditioned setting where the source is also synthesized or edited, connecting non-human voice conversion to text-to-speech pipelines for character voices.
- Because only one baseline model is evaluated, the dataset's difficulty is not yet strongly characterized; running existing zero-shot voice conversion systems on the same splits would show whether the seen/unseen gaps are model-specific or reflect intrinsic dataset difficulty.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Designed Vocalizations Dataset, a publicly released resource for non-human voice conversion (H2NH-VC). Raw vocalizations (3,270 VCTK speech clips and 2,384 non-linguistic Freesound clips, plus 120 test sources) are processed with 47 DSP presets—30 built-in Dehumaniser 2 presets and 17 in-house-designed presets—to create 231,800 designed clips. The dataset provides non-parallel training data, a test split of 120 sources × 47 presets with aligned (source, reference) pairs, and explicit seen/unseen splits over source timbre groups and preset styles. The authors report baseline results using their own H2NH-VC model [3] across four conditions (seen/unseen source × seen/unseen preset), with objective metrics (cosine similarity, PCC-E, RMSE-E, CER, WER) and a MOS listening test. The central claim is that this is the first public benchmark enabling controlled evaluation of designed/non-human voice conversion.
Significance. If the dataset is released as described and the splits are clean, this is a valuable contribution. Prior H2NH-VC work relies on internally curated, non-public resources, making fair comparison impossible; a public benchmark with metadata for effect-chain structures and explicit generalization splits directly addresses that gap. The arithmetic is consistent (5,654×40=226,160; 120×47=5,640; total clips including raw sources are 237,574), the construction pipeline is described step-by-step, and the authors appropriately flag the CER/WER interpretation as speculative. The decision to use a self-authored baseline is disclosed and is acceptable for a dataset paper. The significance depends on the integrity of the seen/unseen split and on the ecological validity of the test-pair construction, both of which require scrutiny.
major comments (3)
- [Sec. 2.4.1 and Sec. 2.3.1] The source-seen test condition is load-bearing for the benchmark claim of ‘explicit seen/unseen splits over source timbre groups.’ The training set uses 30 VCTK samples from each of 109 speakers; the test source-seen linguistic samples are ‘drawn from 10 VCTK speakers.’ The paper does not state whether these 10 speakers are a subset of the 109, nor whether the specific test utterances are disjoint from the 30 training utterances per speaker. If any test speaker or exact test clip also appears in training, the seen-to-seen scores in Table 2 (Cos. Sim. 0.667, MOS 3.81) are inflated and the seen/unseen comparison is not a valid generalization test. Please state the disjointness explicitly, provide a verification script in the released file lists, and if there is any overlap, reselect the test clips and rerun the baseline.
- [Sec. 2.1] Each test reference is defined as x_r = G_p(x_s), so source and reference always have identical content. This isolates timbre matching, but in typical H2NH-VC production use the reference designed sample comes from a different vocalization with different content. The current test protocol may therefore overstate performance for content-mismatched reference conditions. The paper should explicitly discuss this limitation and, ideally, add a secondary content-mismatched evaluation condition. This does not invalidate the controlled timbre-transfer protocol, but it is an important boundary on what the benchmark claims to measure.
- [Table 2 and Sec. 3.3] Objective metrics are reported as point estimates without confidence intervals or significance tests, and the MOS is based on 8 participants / 40 samples. The paper interprets small differences (e.g., 0.15–0.17 MOS) as a monotone trend across conditions. Since benchmark results are intended to support reproducible comparison, the release should include per-sample statistics, bootstrap confidence intervals, or at least error bars so that readers can assess whether the seen/unseen ordering is reliable. This is fixable without changing the dataset.
minor comments (5)
- [Sec. 2.4.1] The term ‘source timbre group’ is used for both VCTK speaker identities and non-linguistic categories. Please define what constitutes a timbre group and how the 10 non-linguistic types were selected, beyond the tag-based + manual listening description.
- [Sec. 2.4.3] The claim that the training set avoids over-representation of any effect module is not quantitatively supported. A table or histogram of effect-module counts would make this checkable.
- [Sec. 3.1] Please specify the exact training/validation split for the baseline model (number of training clips, validation procedure, hyperparameters) to ensure reproducibility of the baseline numbers in Table 2.
- [Sec. 2.2] The dataset is described as publicly available, but the license terms for the released audio—especially regarding Dehumaniser 2 processing and Freesound sources—are not stated. Clarify the licensing conditions in the dataset release.
- [Sec. 3.3] The lower CER/WER for unseen-source conditions is interpreted as possibly reflecting weaker conversion leaving the output closer to the source. This speculation is flagged, but it would be useful to report an oracle condition (source only, no conversion) to anchor the ASR metrics.
Circularity Check
No significant circularity: the dataset is a constructed public resource; the only self-citation (baseline model) is disclosed and non-load-bearing.
full rationale
The paper's contribution is a dataset and benchmark, not a derivation whose conclusion is assumed in its premises. The paired-test definition x_r = G_p(x_s) (Sec. 2.1) is an explicit protocol for controlling timbre transfer; the model is trained on non-parallel data and scored with external metrics (BEATs/SALMONN cosine similarity, Whisper CER/WER, energy correlation), so no fitted parameter is renamed as a prediction. The baseline is the authors' prior H2NH-VC model (Ref [3]), used only as a disclosed benchmark and not to justify the dataset's validity; other self-citations (Refs [14]-[16]) are ordinary related-work context. The one flagged data-hygiene gap is that Sec. 2.3.1 describes 30 VCTK samples from each of 109 training speakers, while Sec. 2.4.1 selects source-seen linguistic test samples from 10 VCTK speakers without stating disjointness from the training speakers/utterances; if overlap exists it would threaten the seen-source condition's validity, but this is a benchmark-correctness risk, not circular reasoning. No circular step can be exhibited as a quote-and-reduction, so the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Dehumaniser 2 built-in and in-house presets are representative of professional sound design output
- domain assumption Same-content test pairs, x_r = G_p(x_s), validly measure conversion quality
- domain assumption Tag-based classification plus manual listening correctly assigns source timbre seen/unseen labels
- domain assumption BEATs/SALMONN cosine similarity, energy correlation, and Whisper CER/WER are valid proxies for the stated qualities
Cite this review
Pith. "Pith review of Designed Vocalizations Dataset: Sound-Designed Human and Animal Voices for Non-human Voice Conversion." pith.science (2026). https://pith.science/paper/HDHW7BTS
@misc{pith2026260720951,
author = {Pith},
title = {Pith review of: Designed Vocalizations Dataset: Sound-Designed Human and Animal Voices for Non-human Voice Conversion},
year = {2026},
howpublished = {\url{https://pith.science/paper/HDHW7BTS}},
note = {Machine review of arXiv:2607.20951}
}
read the original abstract
Advances in AI-based voice conversion have enabled a wide range of media applications, including films, audiobooks, and games. However, most research and public benchmarks still focus on natural human speech, leaving designed vocalizations, such as monster growls and robotic voices, underexplored, partly due to the lack of publicly available resources. To address this gap, we introduce the Designed Vocalizations Dataset, constructed by curating diverse raw vocal sources, including speech and animal vocalizations, and applying professional vocal effects processing to produce corresponding effect modified variants. We further provide a standardized test set with explicit seen/unseen splits over source timbre groups and preset styles to assess generalization under controlled conditions. Finally, we report baseline benchmark results to support reproducible evaluation and future research. The dataset and demo samples are available at https://ncai-official.github.io/speech/publications/designed-vocalizations-dataset/.
Figures
Reference graph
Works this paper leans on
-
[3]
The experiments assessed model performance on the four seen/unseen combinations of sourcex s and reference xr =x (p) using the test set, following training on the train- ing set
Experiment This section discusses the evaluation of the quality and usabil- ity of the proposed dataset by assessing the performance of the voice conversion model and identifying directions for fu- ture research. The experiments assessed model performance on the four seen/unseen combinations of sourcex s and reference xr =x (p) using the test set, followi...
-
[1]
Introduction The growth of interactive and creative media industries (games, film, animation, VR/AR, etc.) has driven increasing demand for diverse vocal sounds that enhance character expression and immersion. In particular, non-natural/non-human vocalizations such as monster roars, robotic voices, and stylized character ut- terances are essential for sha...
arXiv 2026
-
[2]
Designed V ocalizations Dataset 2.1. Description of Human-to-Nonhuman V oice Conversion Given a source vocalizationx s and a reference vocalizationx r, a conversion modelF θ generatesˆy=F θ(xs, xr), whereˆyis expected to preserve the content and temporal structure ofx s while reflecting the target timbre or style represented byx r. In conventional voice c...
-
[4]
The dataset in- cludes raw audio and preset-specific designed variants gener- ated through sound-design operators, covering diverse linguistic and non-linguistic sources
Conclusion This study introduces the Designed V ocalizations Dataset, a public dataset and benchmark for training and evaluating designed vocalization conversion models. The dataset in- cludes raw audio and preset-specific designed variants gener- ated through sound-design operators, covering diverse linguistic and non-linguistic sources. We report baseli...
-
[5]
RS-2025- 25441313, Professional AI Talent Development Program for Multimodal AI Agents, Contribution: 50%)
Acknowledgments This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2025- 25441313, Professional AI Talent Development Program for Multimodal AI Agents, Contribution: 50%). This research was supported by Culture, Sports and Tourism R&D P...
2025
-
[6]
The scientific content, analysis, and conclusions were developed by the authors
Generative AI Use Disclosure Generative AI tools were used to assist with English translation and language editing of the manuscript. The scientific content, analysis, and conclusions were developed by the authors. All authors take full responsibility for the content of the paper and approve its submission
-
[7]
Speak like a dog: Human to non-human creature voice conversion,
K. Suzuki, S. Sakamoto, T. Taniguchi, and H. Kameoka, “Speak like a dog: Human to non-human creature voice conversion,” in2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, Nov. 2022, p. 1388–1393. [Online]. Available: http://dx.doi.org/10.23919/APSIPAASC55919.2022.9980306
arXiv 2022
-
[8]
Ai-assisted human-pet artistic musical co-creation for wellness therapy,
Z. Wang, L. Ma, Y . Jin, Y . Feng, X. Pan, S. Ji, and K. Zhang, “Ai-assisted human-pet artistic musical co-creation for wellness therapy,” inProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, J. Kwok, Ed. International Joint Conferences on Artificial Intelligence Organization, 8 2025, pp. 10 216–10 224, a...
Show all 29 references
-
[9]
When humans growl and birds speak: High-fidelity voice conversion from human to animal and designed sounds,
M. Kang, S. Lee, C. Lee, and N. Cho, “When humans growl and birds speak: High-fidelity voice conversion from human to animal and designed sounds,” inProc. Interspeech, 05 2025
2025
-
[10]
Cartoonsing: Unifying human and nonhuman timbres in singing generation,
J. Han, J. Shi, Z. Tao, Y . Tang, Y . Zhao, G. Xia, and S. Watanabe, “Cartoonsing: Unifying human and nonhuman timbres in singing generation,” 2025. [Online]. Available: https://arxiv.org/abs/2511.21045
2025
-
[11]
Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,
C. Veaux, J. Yamagishi, K. MacDonaldet al., “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,”Uni- versity of Edinburgh. The Centre for Speech Technology Research (CSTR), vol. 6, p. 15, 2017
2017
-
[12]
Libritts: A corpus derived from librispeech for text- to-speech,
H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text- to-speech,”arXiv preprint arXiv:1904.02882, 2019
1904 arXiv
-
[13]
Libri-light: A benchmark for asr with limited or no su- pervision,
J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazar´e, J. Karadayi, V . Liptchinsky, R. Collobert, C. Fuegen et al., “Libri-light: A benchmark for asr with limited or no su- pervision,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Sign...
2020
-
[14]
Hi-fi multi-speaker english tts dataset,
E. Bakhturina, V . Lavrukhin, B. Ginsburg, and Y . Zhang, “Hi-fi multi-speaker english tts dataset,”arXiv preprint arXiv:2104.01497, 2021
2021 arXiv
-
[15]
Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions,
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerrv-Ryanet al., “Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions,” in2018 IEEE international conference on acoustics, speech and signal processing...
2018
-
[16]
Fastspeech 2: Fast and high-quality end-to-end text to speech,
Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” arXiv preprint arXiv:2006.04558, 2020
2006 arXiv
-
[17]
Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,
J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” inInter- national Conference on Machine Learning. PMLR, 2021, pp. 5530–5540
2021
-
[18]
Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,
E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. G ¨olge, and M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” inInternational Con- ference on Machine Learning. PMLR, 2022, pp. 2709–2720
2022
-
[19]
Freevc: Towards high-quality text-free one-shot voice conversion,
J. Li, W. Tu, and L. Xiao, “Freevc: Towards high-quality text-free one-shot voice conversion,” inICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5
2023
-
[20]
Hiervst: Hierarchical adap- tive zero-shot voice style transfer,
S. Lee, H. Choi, H. Oh, and S. Lee, “Hiervst: Hierarchical adap- tive zero-shot voice style transfer,” inProc. Interspeech, 2023
2023
-
[21]
Diff-hiervc: Diffusion-based hierar- chical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation,
H.Choi, S.Lee, and S.Lee, “Diff-hiervc: Diffusion-based hierar- chical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation,” inProc. Interspeech, 2023, pp. 2283–2287
2023
-
[22]
Dddm-vc: Decoupled denoising dif- fusion models with disentangled representation and prior mixup for verified robust voice conversion,
H. Choi, S. Lee, and S. Lee, “Dddm-vc: Decoupled denoising dif- fusion models with disentangled representation and prior mixup for verified robust voice conversion,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 17 862–17 870
2024
-
[23]
Dehumaniser 2,
“Dehumaniser 2,” Krotos Ltd., software. [Online]. Available: https://www.krotosaudio.com/dehumaniser-2
-
[24]
Cubase 12 pro,
“Cubase 12 pro,” Steinberg Media Technologies, software. [Online]. Available: https://www.steinberg.net/cubase/
-
[25]
Freesound datasets: a platform for the creation of open audio datasets,
E. Fonseca, J. Pons Puig, X. Favory, F. Font Corbera, D. Bog- danov, A. Ferraro, S. Oramas, A. Porter, and X. Serra, “Freesound datasets: a platform for the creation of open audio datasets,” in Hu X, Cunningham SJ, Turnbull D, Duan Z, editors. Proceed- ings of the 18th ISMIR C...
2017
-
[26]
High-fidelity audio compression with improved rvqgan,
R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” inAdvances in Neural Information Processing Sys- tems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran As- sociates, I...
2023
-
[27]
BEATs: Audio pre-training with acoustic tokenizers,
S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “BEATs: Audio pre-training with acoustic tokenizers,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Re- search, vol. 202, 23–29 Jul...
2023
-
[28]
SALMONN: Towards generic hearing abilities for large language models,
C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=14rn7HpKVk
2024
-
[29]
Robust speech recognition via large-scale weak su- pervision,
A. Radford, J. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Re- search, vol. 202. PMLR, 23–29...
2023
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.