Pith. sign in

REVIEW 2 major objections 6 minor 53 references

Prosody Labeling with Phoneme-BERT and Speech Foundation Models

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining frozen speech and linguistic foundation models improves automatic Japanese prosody label prediction beyond either modality alone, reaching 89.8% accuracy on accent labels, 93.2% on high-low accents, and 94.3% on break indices.

desk verdict A clean, incremental fusion study for Japanese prosody labeling; the central fusion benefit over speech-only features is small and unquantified, but the paper is honest and deserves refereeing. read the letter →

arxiv 2507.03912 v1 pith:GDKRF7S2 submitted 2025-07-05 eess.AS cs.SD

Tomoki Koriyama This is my paper · ORCID
classification eess.AScs.SD
keywords prosodylabelingautomaticannotationpitchaccentbreakindexPnGBERTPL-BERTHuWhisper
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes an automatic prosody annotation system that predicts phoneme-level pitch accents, high-low accents, break indices, and pause presence from Japanese speech. Its central claim is that concatenating features from a frozen speech foundation model (HuBERT, wav2vec2, WavLM, or Whisper) with features from a phoneme-input linguistic foundation model (PnG BERT or PL-BERT) gives higher label prediction accuracy than either modality alone. On the Corpus of Spontaneous Japanese, the best combination reaches 89.8% accuracy for accent symbols, 93.2% for high-low accents, and 94.3% for break indices. The intended use is to generate reliable prosodic labels automatically for training prosody-controllable text-to-speech systems.

What carries the argument

The mechanism is a frozen-feature fusion pipeline: a speech waveform passes through a frozen SSL or Whisper encoder, frame-level hidden states are averaged over phoneme intervals using VAE-based alignment durations, and the resulting phoneme-level acoustic vector is concatenated with the corresponding phoneme-level representation from a frozen phoneme-input BERT model. A weighted sum over encoder layers, learned during training, selects informative layers, and a small CNN annotation model predicts four prosodic label types under multi-task learning.

What would settle it

Run the same fused system on the same CSJ evaluation set but replace the VAE-based alignment with a standard forced aligner (or with oracle phoneme boundaries from CSJ); if the accuracy gap between the fused and speech-only systems narrows substantially, the fusion benefit depends on the alignment method rather than on the foundation models themselves.

Watch

Extended reading notes

Core claim

The paper establishes that speech and linguistic foundation models carry complementary information for Japanese prosody annotation. Acoustic features, obtained by averaging SSL or Whisper encoder hidden layers over phoneme intervals, capture boundary pitch movements, break-index confusions, and pause presence that text alone cannot provide. Linguistic features from PnG BERT or PL-BERT improve word-boundary distinctions, filled pauses, and disfluencies. Combining the two, with a weighted sum of hidden layers feeding a six-layer CNN, outperforms either modality alone and also beats traditional mel-spectrogram and F0 features by a clear margin.

Load-bearing premise

The accuracy of the speech-side features rests on the VAE-based phoneme alignment being correct, because frame-level acoustic features are averaged over the aligned phoneme intervals; if those durations are systematically wrong, the acoustic contribution is distorted and the reported fusion gain may not survive.

Editorial extensions

If this is right

  • Automatic prosodic labels accurate enough for training prosody-controllable TTS can be derived from waveform plus text, reducing reliance on expert manual annotation.
  • Speech foundation models trained on Japanese yield higher accuracy than English or multilingual SSL models, so language-matched pretraining matters for prosody labeling.
  • Phoneme-input BERT alone predicts accent symbols at 82.5% accuracy, so text-only prosody prediction for TTS also benefits from linguistic foundation models.
  • Boundary pitch movements and pause presence are largely acoustic in nature; speech features are necessary for these labels, while linguistic models mainly help word-boundary and filled-pause decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same fusion pattern generalizes to other languages, concatenating speech and linguistic foundation models could become a default recipe for prosody labeling; the paper only tests Japanese, so this remains an open extrapolation.
  • The learned layer weights indicate later HuBERT layers (8-10) carry most prosodic information; a testable extension is whether this layer-selection pattern transfers across corpora and languages.
  • Because all foundation models are frozen, the approach is parameter-efficient; fine-tuning the acoustic or linguistic models might boost accuracy further but would complicate training and risk overfitting the limited labeled data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes an automatic prosody annotation model for Japanese that concatenates frozen features from a speech foundation model (SSL such as HuBERT or Whisper encoder) with frozen features from a phoneme-input linguistic foundation model (PnG BERT or PL-BERT), and trains a small CNN annotation head with multi-task losses for accent symbols, high-low accents, break indices, and pause presence. The speech features are phoneme-averaged using VAE-based alignment. Experiments on CSJ core data report accuracy and macro F1 on a held-out split, with headline numbers of 89.8% for accent labels, 93.2% for high-low accents, and 94.3% for break indices using HuBERT-base plus PnG BERT. Additional experiments compare several SSL and Whisper models, and the paper includes confusion matrices, an analysis of layer weights, and a qualitative example of predicted labels.

Significance. The paper is carefully designed in several respects: target labels are external X-JToBI manual annotations, the evaluation is on a held-out split, the foundation models are frozen so that only the annotation head and layer-combination weights are fitted, and the ablations over acoustic and linguistic input types are systematic. If the fusion result is real, the approach is practically useful for building prosody-controllable TTS without large amounts of prosodically labeled data and provides a clean recipe for combining speech and linguistic foundation models. However, the quantitative evidence for the central claim is fragile: all reported numbers come from a single training run on a small evaluation set, and the fusion gains over speech-only input are small. The paper's value therefore depends on whether the main effect survives repeated-run or resampled evaluation.

major comments (2)
  1. [Section 5.4, Table 1] The central claim that the combination beats either modality alone rests on differences of only 0.4–1.5 accuracy points over the speech-only baseline. For example, with HuBERT-base, adding PnG BERT changes ACC from 0.890 to 0.898, HL from 0.928 to 0.932, and BI from 0.928 to 0.943, while the PAU score is unchanged. Every number is from a single run on an 896-utterance evaluation set, with no standard deviation, confidence interval, seed variation, or significance test. Since the annotation model is randomly initialized and trained for 100k steps, run-to-run variation of this magnitude is plausible. The paper should report results over multiple training seeds or provide bootstrap/talk-level confidence intervals and a paired significance test (e.g., McNemar or a talk-wise paired test) for the fusion-versus-speech contrast. Without this, the abstract's statement that the combination 'enhanced the prediction accuracy compared to using either a speech or linguistic input alone' is not established at the claimed precision. The same limitation affects the model rankings in Section 5.6 and Tables 2 and 3, where differences as small as 0.001–0.01 are interpreted as meaningful.
  2. [Section 5.3] All acoustic features are computed by phoneme-averaging frame-level SSL or Whisper features using phoneme durations from a single VAE-based alignment method. The paper provides no analysis of how alignment errors affect the fusion result, even though the entire acoustic pathway depends on these boundaries. If the aligner is systematically biased, the speech-model contribution could be distorted, and the reported fusion gains might not transfer to another alignment method or to cases where phoneme transcriptions are not available. Please add a sensitivity analysis with an alternative aligner or an oracle alignment, or at least report alignment error rates on the evaluation set, to support the generality of the proposed phoneme-averaging approach.
minor comments (6)
  1. [Section 5.3] The text contains the typo 'V AE-based alignment'; the extra space before 'AE' should be removed.
  2. [Figure 3] The figure caption labels PnG BERT as 'phoneme+text', but the text and Figure 1 define PnG BERT as using phonemes and graphemes; please use consistent terminology.
  3. [Section 2] The phrase 'SSL-based linguistic models such as BERT' is confusing because the paper elsewhere uses 'SSL' to refer to speech self-supervised models; BERT can be described as a pretrained transformer model rather than an 'SSL-based linguistic model'.
  4. [Section 5.2] The paper states that all labels are mora-level, but BI and PAU are boundary-level events; please clarify how boundary labels are attached to mora-core phonemes when computing losses.
  5. [Section 5.7, Figure 10] The layer-weight optimization is not described; please state whether the weights are softmax-normalized or otherwise constrained, and add this detail to the experimental conditions.
  6. [Section 5.4] The paper does not compare with prior automatic prosody annotation systems (e.g., [7], [8], [12]) on the same data; adding such a comparison would help position the practical contribution, even though the internal fusion comparison is the main claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the prosody-label predictions are evaluated against external manual X-JToBI annotations on a held-out split.

full rationale

The paper's central claim is that concatenating frozen speech and linguistic foundation-model features improves automatic prosody-label prediction. The target labels (ACC, HL, BI, PAU) are manually annotated X-JToBI-style labels from CSJ, independent of the model inputs. Reported accuracies are measured on a held-out evaluation set of 896 utterances; the only trained quantities are the layer-combination weights and the annotation-model parameters, fitted on the training split. No equation or table entry reduces a reported prediction to a fitted parameter, and no 'prediction' is defined in terms of the target labels. The self-citation to the author's prior VAE-based alignment [35] is load-bearing as a preprocessing choice (phoneme durations for averaging speech features), but that prior work is a separate alignment method whose outputs are not derived from prosodic labels and do not include the claimed result; this is independent support rather than circularity. The paper's main empirical weakness is statistical: fusion-vs-speech gains are 0.4-1.5 percentage points with no error bars or repeated runs, which is a robustness/correctness concern, not a circularity concern.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No invented entities are introduced and no theory-level free parameters are fitted; the trained network parameters and hyperparameters are standard supervised-learning components. The main unstated premises are the reliability of the CSJ labels and of the VAE alignment used to convert frame features to phoneme features.

free parameters (1)
  • Annotation model hyperparameters = 6 CNN layers, hidden size 256, kernel size 5, learning rate 1e-5, batch size 4, 100k steps
    Hand-chosen without a reported hyperparameter search; all compared configurations use the same values, so relative comparisons are fair, but the absolute scores could shift with better tuning.
assumptions (3)
  • domain assumption CSJ manual prosodic labels are reliable ground truth
    All accuracy and F1 scores are measured against these labels (Section 5.2); annotation errors or inconsistencies would bias every reported number.
  • domain assumption VAE-based phoneme alignment is accurate
    Phoneme durations from [35] are used to average frame features into phoneme features (Section 5.3); systematic alignment errors would distort the acoustic features.
  • domain assumption The retrained Japanese PnG BERT and PL-BERT are faithful to the original models
    Both models are pretrained from scratch on Japanese Wikipedia and CC-100 (Section 5.1); the paper treats them as the same foundation models as in [23,24], without releasing or evaluating them independently.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prosody Labeling with Phoneme-BERT and Speech Foundation Models." pith.science (2026). https://pith.science/paper/GDKRF7S2

@misc{pith2026250703912,
  author       = {Pith},
  title        = {Pith review of: Prosody Labeling with Phoneme-BERT and Speech Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDKRF7S2}},
  note         = {Machine review of arXiv:2507.03912}
}
read the original abstract

This paper proposes a model for automatic prosodic label annotation, where the predicted labels can be used for training a prosody-controllable text-to-speech model. The proposed model utilizes not only rich acoustic features extracted by a self-supervised-learning (SSL)-based model or a Whisper encoder, but also linguistic features obtained from phoneme-input pretrained linguistic foundation models such as PnG BERT and PL-BERT. The concatenation of acoustic and linguistic features is used to predict phoneme-level prosodic labels. In the experimental evaluation on Japanese prosodic labels, including pitch accents and phrase break indices, it was observed that the combination of both speech and linguistic foundation models enhanced the prediction accuracy compared to using either a speech or linguistic input alone. Specifically, we achieved 89.8% prediction accuracy in accent labels, 93.2% in high-low pitch accents, and 94.3% in break indices.

Figures

Figures reproduced from arXiv: 2507.03912 by the authors.

Figure 1
Figure 1. Training scheme of PnG BERT. Masked phonemes [M] and masked token [MASK] are inferred. Transformer layers [CLS] IY T IY Z S AH N IY [SEP] [CLS] IY T [M] [M] S AH N IY [SEP] [CLS] it it is is sunny sunny sunny sunny [SEP] Phonemes Graphemes [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Proposed model architecture. a sj i t a w a h a r e d e s u k a ACC [ * ] # [ ] * * ? HL L H H L L H L L L BI 0 0 1 2 0 1 0 1 3 PAU N N N Y N N N N NNN [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Accent symbols (ACC): This notation was based on end-to￾end TTS symbols introduced by Kurihara et al. [41]. To ex￾press high-low pitch accents of Japanese, the transition points between high and low moras were denoted by accent sym￾bols. Moreover, the accent phrase boundaries were repre￾sented by accent symbols, and boundary pitch movements such as those in an interrogative sentence were also distin￾guished by the s… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Confusion matrices on accent symbol (ACC) prediction. L H L H 16646 978 1367 15894 L H L H 15292 2132 2377 14884 L H L H 16381 1043 1449 15812 (a) HuBERT & PnG BERT (b) HuBERT & no linguistic input (c) no acoustic input & PnG BERT [PITH_FULL_IMAGE:figures/full_fig_p00…
Figure 7
Figure 7. Figure 7: Confusion matrices on break index (BI) prediction. N Y N Y 33855 173 261 396 N Y N Y 33840 188 256 401 N Y N Y 33989 39 545 112 (a) HuBERT & PnG BERT (b) HuBERT & no linguistic input (c) no acoustic input & PnG BERT [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]
Figure 9
Figure 9. Figure 9: An example of automatically annotated labels. The sentence is “ee shumi ga ongaku nandesu keredomo” in Japanese that means “Well, my hobby is music.” notation. When we used PnG BERT with no acoustic model, we saw 82.5% accuracy in the ACC label prediction. Since this c…
Figure 10
Figure 10. Figure 10: Layer weights for the weighted sum for HUBERT and PnG BERT. and multiple language databases. Among Whisper models, Whisper-large-v3 gave the highest prediction accuracy and F1 scores. 5.7. Analysis of the proposed model Finally, we illustrate the predicted prosody lab…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 51 canonical work pages

  1. [8]

    Mora-level prosody prediction for text-to-speech us- ing japanese BERT without accentual labels,

    T. Ogura, T. Okamoto, Y . Ohtani, E. Cooper, T. Toda, and H. Kawai, “Mora-level prosody prediction for text-to-speech us- ing japanese BERT without accentual labels,” in Proc. ICASSP, 2025

    work page 2025

  2. [1]

    Prosody Labeling with Phoneme-BERT and Speech Foundation Models

    Introduction Recent text-to-speech (TTS) technology has enabled us to gen- erate human-like utterances. To achieve more human-like speech synthesis, it is necessary to accurately produce not only phonetic information but also prosody. For example, syllable- level prosody such as tones and stress/pitch accents is used to distinguish the different meanings ...

    work page Pith review arXiv 2025

  3. [2]

    [8] have also conducted automatic prosody an- notation for constructing a speech synthesis database, similar to the objective of our study

    Related Work Shirahata et al. [8] have also conducted automatic prosody an- notation for constructing a speech synthesis database, similar to the objective of our study. Their annotation model con- sists of a Whisper-based encoder-decoder architecture, with a pretrained Whisper encoder for feature extraction and a de- coder that predicts a mixed sequence ...

    work page

  4. [3]

    ashita wa hare desuka

    Linguistic and Speech Foundation Models 3.1. Linguistic foundation models with phoneme inputs SSL-based linguistic models such as BERT have been incorpo- rated into TTS models and have made it possible to improve the synthetic speech quality due to the rich linguistic informa- tion extraction [17–20]. Following the success of BERT, ex- tended BERT models ...

    work page

  5. [4]

    Figure 3 shows the outline of the proposed model

    Proposed prosody annotation model We propose an automatic prosody annotation model using speech and linguistic foundation models. Figure 3 shows the outline of the proposed model. A speech waveform is input into a speech foundation model such as an SSL encoder or Whisper encoder, and the acoustic features at a frame level are obtained from the encoder’s h...

    work page

  6. [5]

    *”: Other symbol that was neither a high-low transition nor a phrase boundary. • “[

    Experiments 5.1. Pretraining of phoneme-input BERT models We pre-trained language models of PnG BERT and PL-BERT. We used Japanese Wikipedia corpus [37] containing 4.9 GB and the Japanese part of the CC-100 dataset [38] consisting of 70 GB sentences. The total number of sentences were approxi- mately 630 millions. In the original PnG BERT and PL-BERT, wor...

    work page

  7. [6]

    Conclusions In this paper, we proposed an automatic prosody annotation model that uses the encoder hidden layers of both speech and linguistic foundation models. The experimental results showed that the combination of speech and linguistic models enhanced the prosody label prediction accuracies compared to using ei- ther acoustic or linguistic inputs alon...

    work page

  8. [7]

    Automatic prosody annotation with pre-trained text- speech model,

    Z. Dai, J. Yu, Y . Wang, N. Chen, Y . Bian, G. Li, D. Cai, and D. Yu, “Automatic prosody annotation with pre-trained text- speech model,” in Proc. Interspeech, 2022, pp. 5513–5517

    work page 2022

Show all 53 references
  1. [9]

    Semi-supervised prosody model- ing using deep Gaussian process latent variable model,

    T. Koriyama and T. Kobayashi, “Semi-supervised prosody model- ing using deep Gaussian process latent variable model,” in Proc. Interspeech, 2019, pp. 4450–4454

    2019

  2. [10]

    Ac- cent modeling of low-resourced dialect in pitch accent language using variational autoencoder,

    K. Yufune, T. Koriyama, S. Takamichi, and H. Saruwatari, “Ac- cent modeling of low-resourced dialect in pitch accent language using variational autoencoder,” in Proc. 11th ISCA Speech Syn- thesis Workshop (SSW 11), 2021, pp. 189–194

    2021

  3. [11]

    Cross-dialect text- to-speech in pitch-accent language incorporating multi-dialect phoneme-level BERT,

    K. Yamauchi, Y . Saito, and H. Saruwatari, “Cross-dialect text- to-speech in pitch-accent language incorporating multi-dialect phoneme-level BERT,” inProc. IEEE Spoken Language Technol- ogy Workshop (SLT), 2024, pp. 750–757

    2024

  4. [12]

    Multi-modal automatic prosody anno- tation with contrastive pretraining of speech-silence and word- punctuation,

    J. Zhong, Y . Li, H. Huang, K. Richmond, J. Liu, Z. Su, J. Guo, B. Tang, and F. Zhu, “Multi-modal automatic prosody anno- tation with contrastive pretraining of speech-silence and word- punctuation,” in Proc. Interspeech, 2024, pp. 2305–2309

    2024

  5. [13]

    Language-independent prosody-enhanced speech representations for multilingual speech synthesis,

    C. Liu, Z.-H. Ling, and Y .-J. Hu, “Language-independent prosody-enhanced speech representations for multilingual speech synthesis,” inProc. IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 482–488

    2024

  6. [14]

    BERT: Pre- training of deep bidirectional transformers for language under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language under- standing,” in Proc. NAACL, 2019, pp. 4171–4186

    2019

  7. [15]

    Audio- conditioned phonemic and prosodic annotation for building text- to-speech models from unlabeled speech data,

    Y . Shirahata, B. Park, R. Yamamoto, and K. Tachibana, “Audio- conditioned phonemic and prosodic annotation for building text- to-speech models from unlabeled speech data,” in Proc. Inter- speech, 2024, pp. 2795–2799

    2024

  8. [16]

    Low-resourced phonetic and prosodic feature estimation with self-supervised-learning-based acoustic modeling,

    K. Kurihara and M. Sano, “Low-resourced phonetic and prosodic feature estimation with self-supervised-learning-based acoustic modeling,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) , 2024, pp. 640–644

    2024

  9. [17]

    Cross-lingual speech- based ToBI label generation using bidirectional LSTM,

    M. Vetter, S. Sakti, and S. Nakamura, “Cross-lingual speech- based ToBI label generation using bidirectional LSTM,” in Proc. ICASSP, 2019, pp. 6620–6624

    2019

  10. [18]

    A unified accent esti- mation method based on multi-task learning for Japanese text-to- speech,

    B. Park, R. Yamamoto, and K. Tachibana, “A unified accent esti- mation method based on multi-task learning for Japanese text-to- speech,” in Proc. Interspeech, 2022, pp. 1931–1935

    2022

  11. [19]

    Wav2ToBI: a new approach to automatic ToBI transcription,

    W. Zhai and M. Hasegawa-Johnson, “Wav2ToBI: a new approach to automatic ToBI transcription,” in Proc. Interspeech, 2023, pp. 2748–2752

    2023

  12. [20]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Proc. Interspeech, 2020, pp. 5036–5040

    2020

  13. [21]

    Investigation of enhanced Tacotron text-to-speech synthesis systems with self- attention for pitch accent language,

    Y . Yasuda, X. Wang, S. Takaki, and J. Yamagishi, “Investigation of enhanced Tacotron text-to-speech synthesis systems with self- attention for pitch accent language,” in Proc. ICASSP, 2019, pp. 6905–6909

    2019

  14. [22]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in Proc. ICML, 2023, pp. 28 492–28 518

    2023

  15. [23]

    wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” Proc. NeurIPS, vol. 33, pp. 12 449–12 460, 2020

    2020

  16. [24]

    Pre-trained text embeddings for enhanced text-to- speech synthesis,

    T. Hayashi, S. Watanabe, T. Toda, K. Takeda, S. Toshniwal, and K. Livescu, “Pre-trained text embeddings for enhanced text-to- speech synthesis,” in Proc. Interspeech, 2019, pp. 4430–4434

    2019

  17. [25]

    Improving prosody with linguistic and BERT derived features in multi-speaker based Mandarin Chinese neural TTS,

    Y . Xiao, L. He, H. Ming, and F. K. Soong, “Improving prosody with linguistic and BERT derived features in multi-speaker based Mandarin Chinese neural TTS,” inProc. ICASSP, 2020, pp. 6704– 6708

    2020

  18. [26]

    Improving the prosody of RNN-based English text-to-speech synthesis by incorporating a BERT model,

    T. Kenter, M. Sharma, and R. Clark, “Improving the prosody of RNN-based English text-to-speech synthesis by incorporating a BERT model,” in Proc. Interspeech, 2020, pp. 4412–4416

    2020

  19. [27]

    Im- proving prosody modelling with cross-utterance BERT embed- dings for end-to-end speech synthesis,

    G. Xu, W. Song, Z. Zhang, C. Zhang, X. He, and B. Zhou, “Im- proving prosody modelling with cross-utterance BERT embed- dings for end-to-end speech synthesis,” in Proc. ICASSP, 2021, pp. 6079–6083

    2021

  20. [28]

    WavLM: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selecte...

    2022

  21. [29]

    PE- Wav2vec: A prosody-enhanced speech model for self-supervised prosody learning in TTS,

    Z.-C. Liu, L. Chen, Y .-J. Hu, Z.-H. Ling, and J. Pan, “PE- Wav2vec: A prosody-enhanced speech model for self-supervised prosody learning in TTS,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 4199–4210, 2024

    2024

  22. [30]

    PnG BERT: Aug- mented BERT on phonemes and graphemes for neural TTS,

    Y . Jia, H. Zen, J. Shen, Y . Zhang, and Y . Wu, “PnG BERT: Aug- mented BERT on phonemes and graphemes for neural TTS,” in Proc. Interspeech, 2021, pp. 151–155

    2021

  23. [31]

    Phoneme-level BERT for enhanced prosody of text-to-speech with grapheme pre- dictions,

    Y . A. Li, C. Han, X. Jiang, and N. Mesgarani, “Phoneme-level BERT for enhanced prosody of text-to-speech with grapheme pre- dictions,” arXiv preprint arXiv:2301.08810, 2023

    2023 arXiv

  24. [32]

    Detection of prosodic bound- aries in speech using wav2vec 2.0,

    M. Kune ˇsov´a and M. ˇRez´aˇckov´a, “Detection of prosodic bound- aries in speech using wav2vec 2.0,” in Proc. International Con- ference on Text, Speech, and Dialogue, 2022, p. 377–388

    2022

  25. [33]

    Corpus of Spontaneous Japanese: Its design and evaluation,

    K. Maekawa, “Corpus of Spontaneous Japanese: Its design and evaluation,” in ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, 2003

    2003

  26. [34]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

    2021

  27. [35]

    V AE-based phoneme alignment using gradient an- nealing and SSL acoustic features,

    T. Koriyama, “V AE-based phoneme alignment using gradient an- nealing and SSL acoustic features,” in Proc. Interspeech, 2024, pp. 3814–3818

    2024

  28. [36]

    Self-supervised speech representation learning: A review,

    A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe et al. , “Self-supervised speech representation learning: A review,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179–1210, 2022

    2022

  29. [37]

    StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,

    Y . A. Li, C. Han, V . Raghavan, G. Mischler, and N. Mesgarani, “StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,” in Proc. NeurIPS, vol. 36, 2023, pp. 19 594–19 621

    2023

  30. [38]

    Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark,

    Z. Ma, M. Chen, H. Zhang, Z. Zheng, W. Chen, X. Li, J. Ye, X. Chen, and T. Hain, “Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark,” in Proc. In- terspeech, 2024, pp. 1580–1584

    2024

  31. [39]

    Non-intrusive speech intelligibility pre- diction for hearing-impaired users using intermediate ASR fea- tures and human memory models,

    R. Mogridge, G. Close, R. Sutherland, T. Hain, J. Barker, S. Goetze, and A. Ragni, “Non-intrusive speech intelligibility pre- diction for hearing-impaired users using intermediate ASR fea- tures and human memory models,” in Proc. ICASSP, 2024, pp. 306–310

    2024

  32. [40]

    Montreal forced aligner: Trainable text-speech align- ment using kaldi

    M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Son- deregger, “Montreal forced aligner: Trainable text-speech align- ment using kaldi.” inProc. Interspeech, vol. 2017, 2017, pp. 498– 502

    2017

  33. [41]

    One TTS alignment to rule them all,

    R. Badlani, A. Ła ´ncucki, K. J. Shih, R. Valle, W. Ping, and B. Catanzaro, “One TTS alignment to rule them all,” in Proc. ICASSP, 2022, pp. 6092–6096

    2022

  34. [42]

    X-JToBI: an extended J-ToBI for spontaneous speech,

    K. Maekawa, H. Kikuchi, Y . Igarashi, and J. Venditti, “X-JToBI: an extended J-ToBI for spontaneous speech,” in Proc. 7th ICSLP, 2002, pp. 1545–1548

    2002

  35. [43]

    Stress in Thai,

    P. Peyasantiwong, “Stress in Thai,” in Papers from a Conference on Thai Studies in Honor of William J. Gedney. Michigan Pa- pers on South and Southeast Asia, Center for South and South- east Asian Studies, University of Michigan, Ann Arbor , 1986, pp. 19–39

    1986

  36. [44]

    ee shumi ga ongaku nandesu keredomo

    database [45]. For the Whisper encoder except in Sect. 5.6, Whisper-small was used. Both the HuBERT-base and Whisper- small encoder had a hidden size of 768 and 12 hidden layers. The metrics for experimental evaluations were accuracy and macro F1. The macro F1 was calculated b...

  37. [45]

    Japanese Wikipedia Corpus, https://dumps.wikimedia.org/jawiki/

  38. [46]

    CC-100: Monolingual Datasets from Web Crawl Data, https://data.statmt.org/cc-100/

  39. [47]

    UniDic, https://clrd.ninjal.ac.jp/unidic/

  40. [48]

    Adam: A method for stochastic opti- mization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” in Proc. ICLR, 2015

    2015

  41. [49]

    Prosodic features con- trol by symbols as input of sequence-to-sequence acoustic mod- eling for neural TTS,

    K. Kurihara, N. Seimiya, and T. Kumano, “Prosodic features con- trol by symbols as input of sequence-to-sequence acoustic mod- eling for neural TTS,” IEICE Transactions on Information and Systems, vol. E104.D, no. 2, pp. 302–311, 2021

    2021

  42. [50]

    ESPnet: End-to-end speech processing toolkit,

    S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y . Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chenet al., “ESPnet: End-to-end speech processing toolkit,” in Proc. Inter- speech, 2018

    2018

  43. [51]

    ReazonSpeech https://research.reazon.jp/projects/ReazonSpeech/

  44. [52]

    Rinna/japanese-hubert-base https://huggingface.co/rinna/japanese-hubert-base/

  45. [53]

    WORLD: A vocoder- based high-quality speech synthesis system for real-time appli- cations,

    M. Morise, F. Yokomori, and K. Ozawa, “WORLD: A vocoder- based high-quality speech synthesis system for real-time appli- cations,” IEICE Transactions on Information and Systems , vol. E99.D, no. 7, pp. 1877–1884, 2016

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.