REVIEW 2 major objections 4 minor 7 references
WildSpoof Challenge Evaluation Plan
T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read WildSpoof pairs in-the-wild TTS with spoof-aware speaker verification
desk verdict A concrete, mostly well-specified challenge plan for in-the-wild TTS and spoofing detection; the science is absent by design, and the scoring tooling is underspecified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the challenge protocol itself: the pairing of two independently treated tracks, each with its own data, submission format, and metric. The SASV side is anchored by the a-DCF metric, which assigns a single cost to a system by weighting one miss rate and two false-alarm rates; the specified priors and costs make the formula numerically concrete. The TTS side is anchored by the five fixed metrics computed by the designated scoring tool. Together these define the only basis on which systems will be compared, and the plan states explicitly that TTS systems will not be ranked.
What would settle it
If, by the announced data-release date, the TITW-KSKT and TITW-KSUT trial lists or the SASV evaluation waveform package are not distributed, or the reference a-DCF implementation is absent from the evaluation package, the plan cannot be executed as written. A simpler check: run the provided baseline systems and confirm the metric code produces finite scores on the released trial lists.
Extended reading notes
Core claim
This document is an evaluation plan rather than a research result. Its central claim is that a coordinated pair of tasks—generating spoofed speech and detecting it—can be run on in-the-wild datasets under a single protocol. The TTS track fixes the training data (TITW), the evaluation lists (TITW-KSKT and TITW-KSUT), and five metrics computed by a single scoring tool, with no participant ranking. The SASV track fixes the SpoofCeleb training data, a three-class trial list (target, non-target, spoof), and the a-DCF metric with priors and costs specified exactly: π_tar = 0.9405, π_non = 0.0095, π_spf = 0.05, and C_miss = 1, C_fa = 10, C_fa,spoof = 10. The plan's feasibility depends on evaluation
Load-bearing premise
The challenge can only run if the evaluation trial lists, waveform packages, and scoring scripts described as 'to be released' are actually distributed before the submission deadline, and if the designated scoring tool computes the TTS metrics as described.
Editorial extensions
If this is right
- If the plan executes, TTS participants produce 9,113 and 8,000 WAV files for the two evaluation conditions, forming a new in-the-wild TTS evaluation corpus.
- The SASV track will produce a public score set over target, non-target, and spoof trials, creating a common benchmark for spoofing-robust speaker verification on wild audio.
- Because participants must choose one track, the challenge generates independent solutions from each community rather than a single joint system.
- The absence of a TTS ranking means the public outcome will be a summary rather than a leaderboard, reducing competitive pressure to game the metrics.
- The fixed priors and costs of a-DCF make the SASV results directly comparable to future challenges that adopt the same metric.
Reading between the lines
- The paired-track design implicitly assumes that better TTS systems produce harder spoofs; if that coupling holds, improvements in generation quality should translate into more challenging detection trials in future iterations.
- Because the same participants cannot enter both tracks, the challenge does not directly measure how well a single team's TTS fools a single team's detector; the organizers' summary may need to cross-evaluate submissions to test that coupling.
- The plan's TTS metrics cover intelligibility, naturalness, and speaker similarity but not human listening; adding a small human evaluation could ground the neural metrics in perception.
- The dependency on materials 'to be released' means the protocol could execute even without a TTS ranking, but a missing a-DCF reference implementation in the evaluation package would block the SASV track entirely.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript is an evaluation plan for the WildSpoof Challenge, a community challenge with two parallel tracks: a TTS synthesis track and a spoofing-robust SASV track. The TTS track uses the TITW-Easy/Hard training sets and the TITW-KSKT/KSUT evaluation protocols; participants submit 9,113 and 8,000 waveform files, and the organizers compute MCD, UTMOS, DNSMOS, WER, and SPK-sim, with no system ranking. The SASV track uses SpoofCeleb training data and a forthcoming evaluation package; participants submit trial scores, and systems are ranked by a-DCF with priors pi_tar=0.9405, pi_non=0.0095, pi_spf=0.05 and costs C_miss=1, C_fa=10, C_fa_spoof=10. The plan also specifies baselines, submission formats, rules, ethics, registration, and a schedule. The central content is the protocol itself; there are no experimental results or derivations.
Significance. If the challenge executes as specified, it would provide a useful community resource: it explicitly targets in-the-wild speech data, connects TTS generation with spoofing detection, and provides open baselines with public repositories. The protocol is largely concrete: submission file counts, formats, sampling rates, score file schemas, and the a-DCF priors/costs are all explicitly stated, and the priors sum to one. The two baseline repositories are named with URLs. The plan is less complete on evaluation tooling: the TTS metrics are delegated to an undefined tool called Versa, and the a-DCF reference implementation is said to live in an evaluation package that is never identified. Since these tools are the sole mechanism for computing both tracks' metrics, the protocol as written is not fully reproducible. These gaps are fixable and do not undermine the challenge's motivation, but they need to be resolved before the plan can serve as the definitive specification.
major comments (2)
- [Section 2.4] The statement 'All these metrics will be calculated by Versa' is load-bearing for the TTS track, yet Versa is not cited, described, or linked. The reader cannot tell which UTMOS checkpoint is used, how SPK-sim is computed (e.g., which speaker embedding model), how MCD is aligned, or how WER is decoded. Without this information, the metric definitions are not reproducible and participants cannot verify the organizers' scoring. Please name the Versa repository or API, its version, and the exact configuration of each metric.
- [Section 3.4] The a-DCF reference implementation is described as 'included in the evaluation package in GitHub repository mentioned in Section 4,' but Section 4 mentions only the TTS_baselines and SASV_baselines repositories, neither of which is identified as containing an evaluation package. The path to the a-DCF implementation is therefore missing. Since a-DCF is the sole ranking metric for the SASV track, the plan must give a direct URL or repository name and, ideally, a version/commit for the reference implementation.
minor comments (4)
- [Section 2.2] 'Both evaluation protocols cover 40.' appears to be missing a noun (presumably '40 speakers' or '40 utterances'). Please clarify.
- [Table 1] The Evaluation row for the TTS track says 'to be released,' while Section 2.2 refers to the TITW-KSKT/KSUT evaluation protocols. Clarify whether the evaluation protocols already exist (as part of TITW) and only the file lists are being released, or whether the protocols themselves are new.
- [Section 3.5] In the last bullet, 'If a participants is uncertain' contains a subject-verb agreement typo.
- [Section 1 and 5] The challenge objectives and ethics are described twice in similar terms (Introduction and Ethics); consolidating would reduce redundancy, though this is not essential.
Circularity Check
No significant circularity: the evaluation plan makes no derivational predictions, and its self-references are normal challenge infrastructure rather than load-bearing circular arguments.
full rationale
WildSpoof is a challenge evaluation plan, not a derivation or prediction paper. It specifies data, submission formats, metrics, and rules; it does not claim to derive an empirical result from first principles. The metrics (MCD, UTMOS, DNSMOS, WER, SPK-sim, a-DCF) are defined externally or by reference, and the a-DCF priors and costs are stated as protocol constants, not fitted to participant outputs. The paper's reliance on the organizers' own prior work (TITW, SpoofCeleb, a-DCF, the SASV baseline) is self-reference in infrastructure but is not circular: those works are cited as datasets, baselines, and metric definitions, and the challenge does not use them to justify a novelty claim that reduces to itself. The identified missing references (Versa in Section 2.4 and the un-named evaluation package in Section 3.4) are specification gaps that affect reproducibility and execution, but they are not circularity: no equation or prediction is equivalent to its input by construction. No fitted parameter is renamed as a prediction, and no uniqueness claim is imported from self-citation. Therefore the appropriate score is 0.
Assumptions & free parameters
free parameters (2)
- a-DCF priors (pi_tar, pi_non, pi_spf) =
pi_tar=0.9405, pi_non=0.0095, pi_spf=0.05
- a-DCF costs (C_miss, C_fa, C_fa,spoof) =
C_miss=1, C_fa=10, C_fa,spoof=10
assumptions (4)
- domain assumption TITW dataset (Easy/Hard training sets and KSKT/KSUT evaluation protocols) is available and usable as described
- domain assumption SpoofCeleb training data exists and is accessible to participants
- domain assumption a-DCF as defined in [3] is a valid architecture-agnostic metric for spoofing-robust SASV
- domain assumption MCD, UTMOS, DNSMOS, WER, and SPK-sim, as computed by the Versa toolkit, are valid proxies for TTS quality
Cite this review
Pith. "Pith review of WildSpoof Challenge Evaluation Plan." pith.science (2026). https://pith.science/paper/D6K6DDJL
@misc{pith2026250816858,
author = {Pith},
title = {Pith review of: WildSpoof Challenge Evaluation Plan},
year = {2026},
howpublished = {\url{https://pith.science/paper/D6K6DDJL}},
note = {Machine review of arXiv:2508.16858}
}
read the original abstract
The WildSpoof Challenge aims to advance the use of in-the-wild data in two intertwined speech processing tasks. It consists of two parallel tracks: (1) Text-to-Speech (TTS) synthesis for generating spoofed speech, and (2) Spoofing-robust Automatic Speaker Verification (SASV) for detecting spoofed speech. While the organizers coordinate both tracks and define the data protocols, participants treat them as separate and independent tasks. The primary objectives of the challenge are: (i) to promote the use of in-the-wild data for both TTS and SASV, moving beyond conventional clean and controlled datasets and considering real-world scenarios; and (ii) to encourage interdisciplinary collaboration between the spoofing generation (TTS) and spoofing detection (SASV) communities, thereby fostering the development of more integrated, robust, and realistic systems.
Reference graph
Works this paper leans on
-
[1]
Text-to-speech synthesis in the wild,
J.-w. Jung, W. Zhang, S. Maiti, et al., “Text-to-speech synthesis in the wild,” in Proc. Inter- speech, 2025
work page 2025
-
[2]
Spoofceleb: Speech deepfake detection and sasv in the wild,
J.-w. Jung, Y. Wu, X. Wang, et al., “Spoofceleb: Speech deepfake detection and sasv in the wild,” IEEE Open Journal of Signal Processing, 2025
work page 2025
-
[3]
a-DCF: An architecture agnostic metric with application to spoofing-robust speaker verification,
H.-j. Shim, J.-w. Jung, T. Kinnunen, et al., “a-DCF: An architecture agnostic metric with application to spoofing-robust speaker verification,” in Proc. Speaker Odyssey, To appear, 2024. arXiv: 2403.01355 [eess.AS]
arXiv 2024
-
[4]
H. Delgado, N. Evans, J.-w. Jung, et al., “Asvspoof 5 evaluation plan,” Online, 2024
work page 2024
-
[5]
Grad-tts: A diffusion probabilistic model for text-to-speech,
V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. A. Kudinov, “Grad-tts: A diffusion probabilistic model for text-to-speech,” in ICML 2021, 18-24 July 2021, Virtual Event, vol. 139, PMLR, 2021, pp. 8599–8608
work page 2021
-
[6]
Diffwave: A versatile diffusion model for audio synthesis,
Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021
work page 2021
-
[7]
Towards single integrated spoofing-aware speaker veri- fication embeddings,
S. H. Mun, H.-j. Shim, H. Tak, et al., “Towards single integrated spoofing-aware speaker veri- fication embeddings,” in Proc. Interspeech, 2023. 6
work page 2023
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.