REVIEW 3 major objections 5 minor 28 references
Voice Adaptation for Swiss German
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper establishes that a voice-adaptation model for Swiss German can be built from roughly 5,000 hours of automatically transcribed podcast audio, and that human listeners rate its output close to reference-quality speech.
desk verdict A genuinely useful Swiss German TTS corpus and recipe, but the dialect-control claim rests on a circular metric and Zurich DID drops to 0.071. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a fully automated data pipeline that turns raw Swiss podcast audio into weakly labeled training material: speaker diarization and voice activity detection split the audio into single-speaker segments, a large automatic speech recognition model (Whisper-Large-V3) produces Standard German transcripts, and a phoneme n-gram Naive Bayes classifier assigns each speaker one of seven Swiss German dialect regions. This corpus, SRG, is then used to fine-tune XTTSv2, a multilingual zero-shot text-to-speech model that takes a text prompt and a short reference voice as inputs. The same phoneme-based classifier is reused as the dialect-recognition evaluation metric (DID), which is why it both enables and constrains the dialect-rendering claim.
What would settle it
Ask native Swiss German speakers from each region to listen to a blinded set of generated utterances and name the dialect; if Zurich and Central Switzerland utterances are confused at rates near the classifier's own confusion (Zurich DID as low as 0.071–0.457 in the paper's long scenario), the claim that the model correctly renders the desired dialect would be falsified.
Extended reading notes
Core claim
The paper's central claim is that a Standard German text-to-speech model can be adapted to render Swiss German in a user-selected dialect while preserving a reference voice, using only automatically produced training data. The supporting result is that fine-tuning XTTSv2, first on the noisy pseudo-labeled SRG corpus and then on the clean STT4SG-350 corpus, produces a model whose output is rated close to reference quality by human listeners, with the strongest combination showing CMOS -0.29 and SMOS 3.81 in the long scenario and intelligibility 4.51 in the short scenario. The authors also claim that dialect correctness is high across regions, with the known exception of Zurich being under-recognized because the dialect classifier itself confuses Zurich with Central Switzerland.
Load-bearing premise
The whole result rests on the automatic dialect classifier's labels being right, but the same classifier is known to confuse Zurich with Central Switzerland, so a non-trivial share of the training and evaluation labels could be wrong without being detected.
Editorial extensions
If this is right
- A voice-adaptation system for Swiss German can be trained from automatically processed podcasts, without manual parallel data, and can render the requested dialect for most regions.
- The two-stage recipe—pretrain on the large noisy SRG corpus, then fine-tune on clean STT4SG-350—gives the best intelligibility and back-translation scores on short utterances, at a small cost in speaker similarity.
- Dialect coverage is uneven: Valais, with under 1% of the data, is the weakest dialect, while Central Switzerland benefits from the abundance of Zurich data.
- Zurich dialect rendering is systematically underestimated in automated dialect recognition because the evaluation classifier itself confuses Zurich with Central Switzerland, so the true Zurich accuracy remains uncertain.
- Human raters judged the SRG-trained models as close to or better than the baseline on naturalness (CMOS) and speaker similarity (SMOS), especially for longer 10–15 second utterances.
Reading between the lines
- Because the same classifier supplies both the labels that condition the model and the DID score that evaluates it, an independent human dialect-judgment test would be needed to confirm the dialect-rendering claim.
- The same automated pipeline should transfer to other dialect continua or low-resource languages wherever a strong speech recognizer and a small seed corpus for dialect classification exist.
- Since the transcripts are translations rather than faithful transcriptions, the back-translation WER/BLEU scores mix rendering errors with transcription-level differences; a separate human intelligibility check already partly covers this.
- The strong long-scenario result for SRG+STT4SG hints that matching training segment length to deployment length may matter more than additional clean data.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper addresses voice adaptation (text-to-speech with a reference voice) for Swiss German, a low-resource dialect continuum. The authors build the SRG corpus from Swiss public-broadcast podcasts: voice activity detection and diarization produce roughly 1.8 million segments, WhisperX provides Standard German transcripts, and a phoneme n-gram Naive Bayes classifier (Section 2.4) assigns one of seven Swiss German dialect regions or Standard German to each speaker. They then fine-tune XTTS-v2 on this data, with and without a final fine-tuning pass on the STT4SG-350 corpus, yielding three models. Evaluations on the STT4SG-350 test set cover automatic back-translation WER/BLEU, ECAPA2 speaker similarity, dialect recognition (DID), and human SMOS/CMOS/intelligibility ratings. The best model, SRG+STT4SG++, reaches WER 0.328/BLEU 0.607 in the Short scenario and 0.156/0.786 in the Long scenario, with human intelligibility up to 4.51; the paper concludes that translating Standard German text to Swiss German speech is feasible and that the desired dialect is correctly rendered.
Significance. If the results hold, this is a useful step for a genuinely under-resourced language group: it demonstrates that weakly supervised podcast data can improve an open TTS backbone, and it reports human ratings from native raters, which is stronger evidence than purely automatic metrics. The dataset-construction pipeline and the consistent relative ordering of the three models (SRG+STT4SG++ best on intelligibility and back-translation, SRG+STT4SG best on speaker similarity in the Long scenario) are concrete contributions. The main weakness is that the dialect-rendering claim relies on a DID metric computed with the same classifier that produced the training labels, and no independent dialect check is offered; for Zurich the internal metric collapses. The central feasibility claim (intelligible, speaker-similar synthetic Swiss German) is nevertheless supported by non-circular human and back-translation evidence.
major comments (3)
- [Section 4.1 / Tables 3-4] The DID column is measured with "our phoneme-based classification pipeline (c.f. 2.4)"—the same classifier that assigned every dialect label in the SRG training corpus. Because the model is conditioned on those labels, near-ceiling DID scores for most regions may simply show that the model reproduces the classifier's dialect prototypes, including its systematic errors. The paper's own numbers show the metric is not trustworthy for Zurich: SRG+STT4SG++ achieves DID 0.457 (Short) and 0.071 (Long), and even the Baseline reaches only 0.313/0.200. Since Section 4 explicitly states that human raters did not judge dialect accuracy, there is no independent evidence for the abstract's claim that the model "can correctly render the desired dialect." I ask for an external dialect classifier or an expert listening test, or a substantial narrowing of the claim.
- [Section 2.4] The same classifier is applied out-of-distribution to label the SRG corpus. It is trained on STT4SG-350 plus 30 hours of Common Voice and evaluated on 30-second concatenated samples (macro F1 0.88), but it is then applied to merged same-speaker segments from spontaneous podcasts, for which no error analysis is reported. The paper concedes that the classifier "often confuses" Zurich and Central Switzerland. Because these labels are the only dialect supervision for the 4,979-hour corpus, the Zurich split (23.67% of the data, Table 1) is potentially contaminated with Central Swiss or mixed-dialect speech. The manuscript should quantify this label noise—for instance, with a manual audit of a sample of speaker-level labels or a confusion analysis on held-out podcast audio—and discuss how such noise propagates to the Zurich DID collapse.
- [Section 4.3 / Abstract] The statement "dialect correctness is generally high, except for Zurich misclassification due to classifier limitations" treats the DID scores as ground truth. Given the circularity described above, the correct summary is that dialect correctness is not independently measured for any region, and for Zurich even the internal metric fails. The conclusion and abstract should be revised to reflect this evidential gap, or new independent evidence should be supplied.
minor comments (5)
- [Section 1 / References] References [7] and [9] are both the AutoPrep paper, while the sentence in Section 1 that cites "Emilia [7]" points to the wrong reference; the Emilia entry appears to be missing.
- [Section 4] The evaluation description says 1400 generated utterances (200 per dialect region), but the tables contain eight regions (seven Swiss German regions plus German), which would imply 1600 utterances; please clarify the dialect-region count and the sampling procedure.
- [Table 5] The significance markers * and † are not tied to a stated statistical test; please specify the test used and the significance threshold.
- [Figure 1 / Table 1] Figure 1 reports 1.8M segments and 4.7k hours, while Table 1 reports 1,764K samples and 4,979.1 hours; please reconcile these numbers or explain the difference.
- [Sections 1 and 4.1] Calling a CMOS of -0.28 "almost human quality" is an interpretive leap, since CMOS is a relative comparison against a reference recording; please justify or soften this phrasing.
Circularity Check
Dialect correctness is measured by the same classifier that created the training labels, so the central claim 'can correctly render the desired dialect' is largely a self-consistency result rather than an independent verification.
-
self definitional
[Section 2.4 and Section 4.1, Tables 3-4]
"To control the dialect during the generation, we automatically annotate the dialect of each sample. For this, we use the approach described in [19], which is based on a pipeline that converts audio to phonemes using [20] and then trains a simple Naive Bayes model on top of phoneme n-grams. ... Dialect Recognition Accuracy (DID): Measured using our phoneme-based classification pipeline (c.f. 2.4)."
The same phoneme n-gram Naive Bayes classifier assigns the dialect labels that condition the XTTSv2 model during training (Section 2.4) and also produces the DID scores reported as dialect correctness (Section 4.1). Thus the DID evaluation measures how well generated audio matches the classifier's own label distribution, not an independent ground truth. The paper concedes the classifier 'often confuses the two geographically close regions, Zurich and Central Switzerland', and the numbers show it: Zurich DID collapses to 0.412/0.457 (Short, SRG models) and 0.071 (Long, SRG+STT4SG++).
full rationale
The central dialect claim rests on the DID column of Tables 3 and 4, which is generated by the same phoneme Naive Bayes classifier (Section 2.4) that assigned every dialect label in the SRG training corpus. The model is conditioned on those labels, and the evaluation asks whether the generated audio receives the same label from the same classifier. High DID scores are therefore partly a measure of self-consistency with the labeling instrument rather than an independent confirmation of dialect correctness. The paper's own caveat that the classifier 'often confuses' Zurich and Central Switzerland, together with the Zurich DID collapse to 0.071-0.457, demonstrates that the metric is sensitive to the instrument's biases without correcting them. Because human raters were not asked to judge dialect accuracy (the paper states such evaluation is 'often unreliable'), no independent dialect check is present. The WER, BLEU, SMOS, CMOS and intelligibility results are independent external evaluations of intelligibility and voice quality, so the paper is not wholly circular; however, the abstract's 'correctly render the desired dialect' claim is load-bearing and reduces, to a significant degree, to the classifier's own labels.
Assumptions & free parameters
free parameters (5)
- Learning rate 6e-5 =
6e-5
- Segment length cap =
15 seconds
- Diarization speaker range =
2 to 6 speakers
- Minimum segment duration =
2 seconds
- Dialect classifier training recipe =
30h per dialect plus 30h German Common Voice
assumptions (5)
- domain assumption Whisper-Large-V3 produces adequate Standard German translations of Swiss German podcast speech
- domain assumption pyannote diarization yields single-speaker segments that are good enough for TTS training
- domain assumption The seven-region dialect taxonomy is the right conditioning granularity
- domain assumption XTTS-v2 can absorb a new dialect condition from weakly labeled data
- ad hoc to paper CMOS near zero can be interpreted as 'almost human quality'
Cite this review
Pith. "Pith review of Voice Adaptation for Swiss German." pith.science (2026). https://pith.science/paper/YERTPMHE
@misc{pith2026250522054,
author = {Pith},
title = {Pith review of: Voice Adaptation for Swiss German},
year = {2026},
howpublished = {\url{https://pith.science/paper/YERTPMHE}},
note = {Machine review of arXiv:2505.22054}
}
read the original abstract
This work investigates the performance of Voice Adaptation models for Swiss German dialects, i.e., translating Standard German text to Swiss German dialect speech. For this, we preprocess a large dataset of Swiss podcasts, which we automatically transcribe and annotate with dialect classes, yielding approximately 5000 hours of weakly labeled training material. We fine-tune the XTTSv2 model on this dataset and show that it achieves good scores in human and automated evaluations and can correctly render the desired dialect. Our work shows a step towards adapting Voice Cloning technology to underrepresented languages. The resulting model achieves CMOS scores of up to -0.28 and SMOS scores of 3.8.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction V oice Adaptation has seen an unprecedented increase in perfor- mance and utility in recent years [1, 2]. It is now possible to clone a voice across languages with less than a minute of audio required [3, 4]. The two factors that led to these advancements are using large amounts of data and computing power, which are well-leveraged using the ...
-
[2]
Voice Adaptation for Swiss German
Data Collection Pipeline In this section, we describe the creation of the pseudo-labeled dataset, which we ref to as SRG-corpus. We leverage data from the Swiss Broadcasting Cooperation (SRG) using their API 2, where we access available podcasts 3. These podcasts cover various topics, from politics to entertainment and science. Fig- ure 1 shows the proces...
work page Pith review arXiv 2025
-
[3]
For our first model, we fine- tuned XTTS-v2 using the SRG and STT4SG-350 data mix
Model Training As a baseline, we fine-tuned the pre-trained XTTS-v2 [4] on the STT4SG-350 data, which consists of 343 hours of training data, we refer to this model as Baseline. For our first model, we fine- tuned XTTS-v2 using the SRG and STT4SG-350 data mix. We Model Description Steps Baseline XTTSv2 fine-tuned on STT4SG. 170k SRG+STT4SG XTTSv2 fine-tun...
-
[4]
for the training setup, applying an AdamW optimizer with betas 0.9 and 0.96 and weight decay 0.01. The learning rate was changed to 6e-5 from the original 5e-5 due to internal tests and listening to the generated audio files. Weight decay was applied only to the weights, and the learning rate was decayed using MultiStepLR with a gamma of 0.5 using milesto...
-
[5]
Evaluation and Results We conducted two types of evaluations on the STT4SG-350 test set: automated and human. The training and test sets were pre- partitioned by the dataset authors to ensure speaker indepen- dence, such that no speaker or sample appears in both splits. The evaluation covers two scenarios. • Short: Texts from the STT4SG-350 test set corre...
-
[6]
Conclusion This work presented the transfer of voice adaptation technolo- gies to the challenging scenario of Swiss German dialects. We showed that translating Standard German text to Swiss German dialect speech is feasible and yields satisfactory results. For this, we created the SRG corpus of approximately 5000 hours of Swiss German dialect speech and f...
-
[7]
Neural codec language mod- els are zero-shot text to speech synthesizers,
C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Neural codec language mod- els are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023
arXiv 2023
-
[8]
Vall-e 2: Neural codec language models are hu- man parity zero-shot text to speech synthesizers,
S. Chen, S. Liu, L. Zhou, Y . Liu, X. Tan, J. Li, S. Zhao, Y . Qian, and F. Wei, “Vall-e 2: Neural codec language models are hu- man parity zero-shot text to speech synthesizers,” arXiv preprint arXiv:2406.05370, 2024
arXiv 2024
Show all 28 references
-
[9]
Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,
Z. Zhang, L. Zhou, C. Wang, S. Chen, Y . Wu, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,”arXiv preprint arXiv:2303.03926, 2023
2023 arXiv
-
[10]
XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model,
E. Casanova, K. Davis, E. G ¨olge, G. G ¨oknar, I. Gulea, L. Hart, A. Aljafari, J. Meyer, R. Morais, S. Olayemi, and J. We- ber, “XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model,” in Interspeech 2024, 2024, pp. 4978–4982
2024
-
[11]
Atten- tion is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Atten- tion is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Ga...
2017
-
[12]
Libriheavy: A 50,000 hours asr corpus with punc- tuation casing and context,
W. Kang, X. Yang, Z. Yao, F. Kuang, Y . Yang, L. Guo, L. Lin, and D. Povey, “Libriheavy: A 50,000 hours asr corpus with punc- tuation casing and context,” in ICASSP 2024 - 2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 10 991–10 995
2024
-
[14]
Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model bench- mark,
L. Ma, D. Guo, K. Song, Y . Jiang, S. Wang, L. Xue, W. Xu, H. Zhao, B. Zhang, and L. Xie, “Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model bench- mark,” in Interspeech 2024, 2024, pp. 1840–1844
2024
-
[15]
Autoprep: An automatic preprocessing framework for in-the-wild speech data,
J. Yu, H. Chen, Y . Bian, X. Li, Y . Luo, J. Tian, M. Liu, J. Jiang, and S. Wang, “Autoprep: An automatic preprocessing framework for in-the-wild speech data,” in ICASSP 2024 - 2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. ...
2024
-
[16]
SDS-200: A Swiss German speech to Standard German text corpus,
M. Pl ¨uss, M. H ¨urlimann, M. Cuny, A. St ¨ockli, N. Kapotis, J. Hartmann, M. A. Ulasik, C. Scheller, Y . Schraner, A. Jain, J. Deriu, M. Cieliebak, and M. V ogel, “SDS-200: A Swiss German speech to Standard German text corpus,” in Proceedings of the Thirteenth Language Resou...
2022
-
[17]
STT4SG-350: A speech corpus for all Swiss German dialect regions,
M. Pl ¨uss, J. Deriu, Y . Schraner, C. Paonessa, J. Hartmann, L. Schmidt, C. Scheller, M. H ¨urlimann, T. Samardˇzi´c, M. V ogel, and M. Cieliebak, “STT4SG-350: A speech corpus for all Swiss German dialect regions,” in Proceedings of the 61st Annual Meeting of the Association ...
2023
-
[18]
Fine-tuning whisper on low-resource languages for real- world applications,
V . Timmel, C. Paonessa, R. Kakooee, M. V ogel, and D. Perru- choud, “Fine-tuning whisper on low-resource languages for real- world applications,” arXiv preprint arXiv:2412.15726, 2024
2024 arXiv
-
[19]
Swissdial: Parallel multidialectal corpus of spoken swiss german,
P. Dogan-Sch ¨onberger, J. M ¨ader, and T. Hofmann, “Swissdial: Parallel multidialectal corpus of spoken swiss german,” arXiv preprint arXiv:2103.11401, 2021
2021 arXiv
-
[20]
Natural tts synthesis by condi- tioning wavenet on mel spectrogram predictions,
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerrv-Ryan, R. A. Saurous, Y . Agiomvrgiannakis, and Y . Wu, “Natural tts synthesis by condi- tioning wavenet on mel spectrogram predictions,” in 2018 IEEE International Conferenc...
2018
-
[21]
pyannote.audio 2.1 speaker diarization pipeline: prin- ciple, benchmark, and recipe,
H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: prin- ciple, benchmark, and recipe,” in Proc. INTERSPEECH 2023 , 2023
2023
-
[22]
Powerset multi-class cross entropy loss for neural speaker diarization,
A. Plaquet and H. Bredin, “Powerset multi-class cross entropy loss for neural speaker diarization,” in Proc. INTERSPEECH 2023, 2023
2023
-
[23]
ELAN (Version 6.8) [Computer software],
“ELAN (Version 6.8) [Computer software],” Nijmegen, The Netherlands, 2024. [Online]. Available: https://archive.mpi.nl/tla/ elan
2024
-
[24]
Robust speech recognition via large-scale weak su- pervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in Proceedings of the 40th International Conference on Machine Learning, ser. ICML’23. JMLR.org, 2023
2023
-
[25]
Automatische erkennung schweizerdeutscher dialekte anhand von audiodaten via phonem- transkriptionen,
L. Bolliger and S. Waldburger, “Automatische erkennung schweizerdeutscher dialekte anhand von audiodaten via phonem- transkriptionen,” ZHAW Z ¨urcher Hochschule f ¨ur Angewandte Wissenschaften, Tech. Rep., 2024
2024
-
[26]
Simple and effective zero-shot cross-lingual phoneme recognition,
Q. Xu, A. Baevski, and M. Auli, “Simple and effective zero-shot cross-lingual phoneme recognition,” arXiv preprint arXiv:2109.11680, 2021
2021 arXiv
-
[27]
Common voice: A massively-multilingual speech corpus,
R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari, F. B ´echet, P. ...
2020
-
[28]
Ecapa2: A hybrid neural net- work architecture and training strategy for robust speaker embed- dings,
J. Thienpondt and K. Demuynck, “Ecapa2: A hybrid neural net- work architecture and training strategy for robust speaker embed- dings,” in 2023 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU). IEEE, 2023, pp. 1–8
2023
-
[29]
Dialect transfer for Swiss German speech translation,
C. Paonessa, Y . Schraner, J. Deriu, M. H ¨urlimann, M. V ogel, and M. Cieliebak, “Dialect transfer for Swiss German speech translation,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association fo...
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.