Score-Based Training for Energy-Based TTS Models

Anton Ragni; Wanli Sun

REVIEW 5 major objections 6 minor 41 references

Score-Based Training for Energy-Based TTS Models

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A simple delta loss trains energy-based TTS score functions to point directly from a noisy hypothesis to the reference, yielding one-step-friendly inference and better subjective ratings than sliced score matching.

desk verdict Delta loss is a t=0 flow-matching loss in disguise; the paper is honest about the link but overclaims novelty, and the empirical comparison is confounded, though the study is a useful and readable contribution to EBM TTS training. read the letter →

arxiv 2505.13771 v1 pith:HZIQ3MV3 submitted 2025-05-19 cs.SD cs.LGeess.AS

Wanli Sun , Anton Ragni This is my paper · ORCID

classification cs.SDcs.LGeess.AS

keywords energy-basedmodelstext-to-speechdeltalossslicedscorematchingnoisecontrastiveestimationflowfunctiontrainingfirst-orderinference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a new training objective, the delta loss, for energy-based models of text-to-speech, and argues that it learns score functions suited to the first-order inference these models use. It points out that noise contrastive estimation and sliced score matching spend their effort on matching energies or gradients without considering the shape of the log-likelihood, so a perfectly learned score may still fail to move a noisy sample toward the reference in a few steps. The delta loss instead trains the score at a noisy hypothesis to equal the displacement vector from that hypothesis to the reference speech, making the ideal update direction explicit. On a single-speaker benchmark, the resulting model is competitive with sliced-score-matching training on objective measures and receives better subjective naturalness ratings, with fewer low scores and more high scores.

What carries the argument

The central object is the delta loss, a regression objective that trains the model score at a noisy sample $Y^-$ to return the displacement vector $Y^+ - Y^-$ that points from that sample to the reference $Y^+$. Its purpose is to make the score realize the one-step identity $Y^+ = Y^- + S_\theta(x,Y^-)$ for the training pairs, so the first-order update used in inference is aligned with the actual path to the target. The paper shows this target is the same velocity field used by flow matching along the linear path $Y_t = tY^+ + (1-t)Y^-$ evaluated at $t = 0$, which is why the loss is both simple and geometrically meaningful. The paper contrasts this with NCE and SSM, which learn energies or scores without regard for whether the resulting log-likelihood supports few-step gradient ascent.

What would settle it

Use a trained delta-loss EBM and run one-step inference from initializations drawn from a different distribution than the acoustic-model hypotheses used in training, for example references corrupted by Gaussian noise; if the one-step output no longer moves toward the reference (measured by mel-cepstral distortion or a perceptual metric), the claimed one-step suitability does not generalize beyond the training initialization distribution.

Watch

Extended reading notes

Core claim

The central claim is that an energy-based TTS score function should satisfy the one-step identity $Y^+ = Y^- + S_\theta(x,Y^-)$ at the noisy starting points $Y^-$ supplied by the acoustic model, and that the delta loss $L^\Delta_\theta = \frac{1}{2}\lVert S_\theta(x,Y^-) - (Y^+ - Y^-)\rVert_2^2$ is the training objective that enforces it. Because a score that satisfies this identity points directly from any training hypothesis toward its reference, the paper argues that the resulting log-likelihood landscape is far friendlier to the first-order gradient updates used in EBM and diffusion inference than landscapes produced by NCE or SSM. The paper also establishes a mathematical link to flow matching, noting that the delta-loss target is the straight-line velocity of the interpolated path between the noisy sample and the reference, evaluated at the starting point. Experimental results show the delta-trained model performs comparably to sliced score matching on objective metrics while receiving better listening-test ratings.

Load-bearing premise

The load-bearing premise is that the hypothesis-reference pairs seen during training are representative of the starting points used at inference, so a score calibrated to point from the acoustic model's outputs to references also points correctly from other initializations.

Editorial extensions

If this is right

If the delta-loss claim holds, energy-based TTS training no longer needs carefully engineered negative samples or expensive Hessian-style computations; a simple squared-distance regression on displacement vectors suffices.
The objective results suggest the trained score models reach their best scores after one inference step from the acoustic model's output, so the hundreds of MCMC iterations traditionally used for EBM inference may be unnecessary.
Because delta loss is closely linked to flow matching, advances in flow-matching theory or training practice can be carried over to score-based EBM training, and vice versa.
The subjective results indicate that the delta-trained model reduces the frequency of severely unnatural outputs (scores 1 through 3) and increases the frequency of high ratings (4 and 5), so the practical benefit is concentrated in avoiding bad synthesis rather than in improving average quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, one could train delta loss at several points along the interpolation between $Y^-$ and $Y^+$, not just at $Y^-$; that would recover full flow matching and might make the score robust to initializations that differ from the acoustic model's training hypotheses.
Beyond the paper, the paper's comparison uses a fixed inference learning rate and a fixed number of inference steps, so sweeping the step size would test whether delta-trained scores really are one-step-optimal or merely improve over SSM at the chosen settings.
Beyond the paper, the one-step identity suggests delta loss could be used to initialize or fine-tune diffusion or flow-based TTS models, since it enforces the straight-line velocity property those models aim to approximate.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Know this before reading: the 'delta loss' in Eq. (9) is exactly the t=0 endpoint of the conditional flow-matching loss in Eq. (10), with Y0 = Y− and no time interpolation. The authors acknowledge the link in Section 3.3, but they still frame delta loss as a new criterion. That framing is the paper's main weakness, not the math.

What the paper does well: it is the first to apply sliced score matching (SSM) to EBM TTS, and it gives a clear, readable comparison of NCE, SSM, delta loss, and their combination. The training setup is described in enough detail to reproduce (architecture from Matcha-TTS, LJSpeech split, 50K steps, optimizer settings). The simple one-step result is genuinely interesting: both SSM and delta loss reach competitive objective metrics after a single gradient step, which contrasts with the hundreds of steps often needed for NCE-trained EBMs. The paper is also honest about the connection to flow matching, even if it does not fully embrace the consequence.

The soft spots are real but not fatal. First, the novelty claim does not survive contact with Eq. (10); delta loss is a special case, not a new objective. Second, Eq. (8) claims one-step optimality 'using any noisy speech sample as initialisation,' but training only sees force-aligned Tacotron2 hypotheses. Inference starts from hypotheses with predicted durations, so the initialization distribution shifts. The authors never test with other initializations, so this central generalization claim is unsupported. Third, Table 3 compares NCE on the full validation set with score-based models on a 600-clip subset, and the objective metrics have no error bars or significance tests. Fourth, the MOS test uses only three listeners, and the 1-step delta vs. SSM difference (3.95 vs. 3.90) is within overlapping error bars; the 'better' claim is weak.

For whom is this paper useful? People working on EBM or score-based generative models for speech, especially those tired of NCE's negative-sampling pain. It is a good engineering recipe and a decent case study of why evaluation details matter. It is not a foundational contribution.

My recommendation: send it to peer review, not because the contribution is new, but because the question is relevant and the paper is clearly written. Expect the reviewers to push for a reframed contribution, a clean comparison on the same validation set, more listeners, and at least one experiment that varies the initialization distribution.

Referee Report

5 major / 6 minor

Summary. The paper proposes 'delta loss' (Eq. 9) for training energy-based text-to-speech (TTS) models. The loss minimizes the squared error between the model score evaluated at a noisy acoustic-model hypothesis Y- and the displacement (Y+ - Y-) to the reference speech. The authors argue that this makes learned scores suitable for one-step first-order inference (Eq. 8), note a close connection to flow matching (Sec. 3.3), and report experiments on LJSpeech comparing delta loss with sliced score matching (SSM), noise contrastive estimation (NCE), and Tacotron2. Objective metrics (MCD, log f0, SpeechBERTScore, UTMOSv2) and a small MOS study are reported; the conclusion claims comparable objective performance and better subjective performance for delta loss.

Significance. If the claims were fully supported, the paper would provide a simple and efficient alternative to NCE and SSM for EBM training and would clarify the relationship between score-based EBM training and flow matching. The derivation is transparent, the SSM baseline for EBM TTS is a legitimate first application, and the paper is honest about the flow-matching connection. However, the empirical evidence is currently insufficient: the headline comparison against NCE is confounded by different evaluation subsets, no uncertainty quantification or significance tests are provided, the subjective evaluation has only three listeners, and the central 'any initialization' claim is only tested on one source distribution. These issues are load-bearing for the paper's main claims.

major comments (5)

[Section 3.2, Eq. (8)] The claim that Eq. (8) holds 'using any noisy speech sample as initialisation' is not supported by the training objective. Eq. (9) is optimized only over Y- produced by force-aligned Tacotron2 outputs (Sec. 4.1.2), while inference initializations use predicted durations and therefore fall outside this training manifold. One-step optimality is exact only for the training pairs (if the loss reaches zero); for arbitrary Y- it is a generalization assumption. The experiments never vary the initialization distribution (e.g., Gaussian noise, unaligned hypotheses, random starts), so they cannot distinguish a globally useful score from a map memorized for the acoustic-model distribution. Please either add such experiments or restrict the claim to initializations from the same source distribution.
[Table 3, Sec. 4.1.1] The comparison of NCE with score-based EBMs is confounded: NCE is evaluated on the full validation set (1,800 clips) while SSM and Delta are evaluated on a 600-clip subset. The text states that score-based training is 'significantly better' than NCE, but no significance test is reported and the evaluation sets differ. All systems must be scored on the same validation set, with paired significance tests or bootstrap confidence intervals, before a superiority claim over NCE can be made.
[Tables 1-2] Objective metrics are reported as point estimates without error bars, confidence intervals, or significance tests. Many differences are very small (e.g., MCD 5.292 vs 5.294 for Delta and SSM at one step in Table 2) and could easily be within noise. Moreover, Table 2 shows that SSM also attains its best objective results after one step, so the experiments do not demonstrate that delta loss improves one-step inference relative to SSM. Please provide utterance-level statistics or bootstrap CIs and a paired significance test.
[Table 4, Sec. 4.1.3] The subjective evaluation rests on only three listeners, and the MOS difference between SSM-1 (3.90 ± 0.06) and Delta-1 (3.95 ± 0.05) is within the reported confidence intervals. The conclusion's claim of 'better subjective evaluations' is therefore not established by the MOS scores alone; it relies on the raw score-count histogram in Figure 3 without any statistical test. A larger listening panel and a significance test (e.g., Wilcoxon signed-rank or paired t-test on utterance-level scores) are needed.
[Section 3.3, Eq. (10)] The paper correctly acknowledges the close link between delta loss and flow matching, but Eq. (9) is essentially a t=0 conditional flow-matching loss whose source distribution is the acoustic model. Without a comparison to a flow-matching or rectified-flow baseline trained with the same network architecture and data, the experiments cannot attribute the observed behavior to the proposed 'scores suitable for first-order schemes' mechanism rather than to the already-known flow-matching objective. Please add such a baseline or clearly reposition delta loss as an application/reexamination of flow matching for EBM training.

minor comments (6)

[Section 4.1.3] The description 'Three listeners, who are all native English speakers, were asked to rate ... on the Amazon Mechanical Turk platform' is unclear, since MTurk is typically associated with many anonymous workers. Please clarify the recruitment procedure and listener compensation.
[Table 3] The asterisk on NCE* is never explained in the text or in the table caption.
[Abstract] The sentence 'Experiments contrasts these approaches for training EBMs' contains a subject-verb agreement error; it should be 'Experiments contrast...'.
[Table 2] The column header 'log fo' should read 'log f0' for consistency with the rest of the paper.
[Figure 2] The caption says 'hypotheses list' but the figure illustrates different score/log-likelihood shapes; please rephrase to match the actual content.
[Eq. (5)] The score notation S_theta(x, Y(N), N) introduces an iteration index that is not used elsewhere in the same way; this may confuse readers comparing it with S_theta(x, Y) in Eq. (8).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: delta loss is an explicit training objective, and the one-step result is empirically evaluated rather than derived from its own inputs.

full rationale

Delta loss in Eq. (9) is not a prediction obtained from hidden inputs; it is an explicitly defined regression objective whose target is the residual Y+ - Y-. Eq. (8) merely restates what the score would need to equal for one-step inference to recover Y+, and the paper does not present this identity as an empirical finding. The experimental success of one-step inference with delta loss is a genuine empirical result, because the trained U-Net must generalize to held-out Tacotron2 hypotheses and, as Sec. 4.1.2 notes, inference is performed without the force-aligned duration constraint used in training ('No such constraint is enforced in inference'). The admitted relationship to flow matching in Sec. 3.3 is disclosed rather than hidden, and it does not make the central evaluation circular. The only self-citation is [3], earlier work by the same authors used as the NCE baseline and as motivation for EBM TTS; this citation is not load-bearing for the new loss or its evaluation. The unsupported generalization in the phrase 'using any noisy speech sample as initialisation' in Sec. 3.2 exceeds the training distribution of Tacotron2 hypotheses, but that is a correctness risk, not a circularity of derivation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim relies on the empirical assumption that acoustic-model hypotheses are a valid proxy for MCMC initialization noise, and that the learned score generalizes beyond training pairs. No free parameters are fitted to data in the loss itself; the learning rates and step counts are standard hyperparameters.

assumptions (3)

domain assumption Tacotron2 hypotheses provide a representative distribution of noise for MCMC initialization
The delta loss is trained exclusively on pairs of references and Tacotron2 hypotheses (Section 4.1.2); no argument or experiment shows that the learned score generalizes to other initialization distributions.
domain assumption A score that equals Y+ - Y- for all Y- is realizable by the network class
The one-step optimality argument in Eq. (8) requires the learned score to exactly match the delta for every possible starting point, which is not guaranteed by the U-net architecture or the finite training set.
domain assumption First-order inference with the learned score will be stable at the chosen learning rate
The experiments use gradient descent with learning rate 3e-5 and report degradation with more steps for delta loss, indicating the inference dynamics are not governed by the one-step argument alone.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Score-Based Training for Energy-Based TTS Models." pith.science (2026). https://pith.science/paper/HZIQ3MV3

@misc{pith2026250513771,
  author       = {Pith},
  title        = {Pith review of: Score-Based Training for Energy-Based TTS Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HZIQ3MV3}},
  note         = {Machine review of arXiv:2505.13771}
}

read the original abstract

Noise contrastive estimation (NCE) is a popular method for training energy-based models (EBM) with intractable normalisation terms. The key idea of NCE is to learn by comparing unnormalised log-likelihoods of the reference and noisy samples, thus avoiding explicitly computing normalisation terms. However, NCE critically relies on the quality of noisy samples. Recently, sliced score matching (SSM) has been popularised by closely related diffusion models (DM). Unlike NCE, SSM learns a gradient of log-likelihood, or score, by learning distribution of its projections on randomly chosen directions. However, both NCE and SSM disregard the form of log-likelihood function, which is problematic given that EBMs and DMs make use of first-order optimisation during inference. This paper proposes a new criterion that learns scores more suitable for first-order schemes. Experiments contrasts these approaches for training EBMs.

Figures

Figures reproduced from arXiv: 2505.13771 by the authors.

**Figure 1.** Two ways of computing scores for EBMs: analytic (top) and predictive (bottom) could be performed using the following version of Langevin Markov Chain Monte-Carlo (MCMC) [3] Y (N+1) ← Y (N) + ρ∇Y log pθ(Y (N) |x), (3) where ρ is an learning rate. Typically, a large number of iterations is required to yield accurate speech samples [3], which is consistent with other related models [15, 18, 19]. 2.1. Noise contrastive… view at source ↗

**Figure 2.** Different score functions: from left to right, the hypotheses list from the worst to the best. side of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Detailed breakdown of MOS score counts suggests that the delta loss trained EBM has much fewer 1, 2 and 3 scores and more 4 and 5 scores than SSM trained EBM. 5. Conclusion This paper presented two new score-based approaches to training energy-based models (EBMs) for text-to-speech (TTS) as an alternative to more challenging Noise Contrastive Estimation (NCE). The first approach is sliced score matching (SSM) that… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 27 canonical work pages

[1]

Introduction Recently, energy-based models (EBMs) [1, 2] have drawn some attention in the text-to-speech (TTS) area [3]. These proba- bilistic models define the log-likelihood of speech given text as the difference between negative (unnormalised) energy function and the logarithm of the corresponding normalisation term. Due to the difficulty of computing ...

work page
[2]

first application of SSM to training EBMs in TTS

work page
[3]

Section 2 discusses EBMs and related work, including training methods and con- nections between EBMs and diffusion models

a new score-based loss function for EBMs; and the rest of it is organised as follows. Section 2 discusses EBMs and related work, including training methods and con- nections between EBMs and diffusion models. Section 3 de- scribes existing score-based training approached and introduces delta loss as a possible alternative. Experimental results and dis- cu...

work page
[4]

Energy-Based Models Given a sequence of text tokens x, an energy-based model (EBM) of spectral feature sequencesY could be defined as [3] pθ(Y|x) = 1 Zθ(x) exp(−Eθ(x,Y )), (1) whereEθ(x,Y ) is an energy function with parameters θ that connects text and speech such that smaller values correspond to better matches between text and speech and vice versa, and...

work page Pith review arXiv 2025
[5]

Score-based Training This section will first discuss score matching approaches and then will introduce a new criterion, which is shown to be linked with currently popular flow matching [16, 25, 26], that ad- dresses some of score matching limitations. 3.1. Sliced score matching (SSM) Rather than using NCE it is also possible to optimise EBMs by matching t...

work page
[6]

(9) Compared to both NCE and (S)SM, delta loss is a very efficient training approach. 3.3. Connection to flow matching Interestingly, the delta loss in equation (9) can be linked to the Flow Matching (FM) loss [16, 25, 26] Lfm θ (x,Y +,Y 0,t ) = 1 2∥V (x,Yt,t )− (Y +−Y 0)||2, (10) where t ∈ [0, 1], Y 0 is a source or anchor sample possibly influenced by x...

work page
[7]

Experimental setup 4.1.1

Experiments 4.1. Experimental setup 4.1.1. Dataset The dataset utilized in this work is LJSpeech [29], which con- sists of 13,100 audio clips totalling approximately 24 hours of speech from a single female speaker reading passages from pub- lic domain texts. The dataset is randomly divided into training (10,000 clips), validation (1,800 clips), and test (...

work page
[8]

The first approach is sliced score matching (SSM) that has been previously used with EBMs in other areas but not in TTS

Conclusion This paper presented two new score-based approaches to train- ing energy-based models (EBMs) for text-to-speech (TTS) as an alternative to more challenging Noise Contrastive Estima- tion (NCE). The first approach is sliced score matching (SSM) that has been previously used with EBMs in other areas but not in TTS. The second approach is a new ob...

work page

Show all 41 references

[9]

A tutorial on energy-based learning,

Y . LeCun, S. Chopra, R. Hadsell, M. Ranzato, F. Huang et al., “A tutorial on energy-based learning,”Predicting structured data, vol. 1, no. 0, 2006

2006
[10]

Learning deep kernels for exponential family densities,

L. Wenliang, D. J. Sutherland, H. Strathmann, and A. Gretton, “Learning deep kernels for exponential family densities,” in In- ternational Conference on Machine Learning. PMLR, 2019, pp. 6737–6746

2019
[11]

Energy-based models for speech synthesis,

W. Sun, Z. Tu, and A. Ragni, “Energy-based models for speech synthesis,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 12 667–12 671

2024
[12]

Residual energy-based models for end-to-end speech recognition,

Q. Li, Y . Zhang, B. Li, L. Cao, and P. C. Woodland, “Residual energy-based models for end-to-end speech recognition,” arXiv preprint arXiv:2103.14152, 2021

2021 arXiv
[13]

Energy-based reranking: Improving neural machine translation using energy-based models,

S. Bhattacharyya, A. Rooshenas, S. Naskar, S. Sun, M. Iyyer, and A. McCallum, “Energy-based reranking: Improving neural machine translation using energy-based models,” arXiv preprint arXiv:2009.13267, 2020

2009 arXiv
[14]

Residual energy-based models for text,

A. Bakhtin, Y . Deng, S. Gross, M. Ott, M. Ranzato, and A. Szlam, “Residual energy-based models for text,” Journal of Machine Learning Research, vol. 22, no. 40, pp. 1–41, 2021

2021
[15]

Recurrent neural network language model training with noise contrastive es- timation for speech recognition,

X. Chen, X. Liu, M. J. Gales, and P. C. Woodland, “Recurrent neural network language model training with noise contrastive es- timation for speech recognition,” in2015 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2015, pp. 5411–5415

2015
[16]

Estimation of non-normalized sta- tistical models by score matching

A. Hyv ¨arinen and P. Dayan, “Estimation of non-normalized sta- tistical models by score matching.” Journal of Machine Learning Research, vol. 6, no. 4, 2005

2005
[17]

A connection between score matching and denois- ing autoencoders,

P. Vincent, “A connection between score matching and denois- ing autoencoders,” Neural computation, vol. 23, no. 7, pp. 1661– 1674, 2011

2011
[18]

Sliced score matching: A scalable approach to density and score estimation,

Y . Song, S. Garg, J. Shi, and S. Ermon, “Sliced score matching: A scalable approach to density and score estimation,” inUncertainty in Artificial Intelligence. PMLR, 2020, pp. 574–584

2020
[19]

How to train your energy-based mod- els,

Y . Song and D. P. Kingma, “How to train your energy-based mod- els,” arXiv preprint arXiv:2101.03288, 2021

2021 arXiv
[20]

Imita- tion with neural density models,

K. Kim, A. Jindal, Y . Song, J. Song, Y . Sui, and S. Ermon, “Imita- tion with neural density models,”Advances in Neural Information Processing Systems, vol. 34, pp. 5360–5372, 2021

2021
[21]

On out-of-distribution detection with energy-based models,

S. Elflein, B. Charpentier, D. Z ¨ugner, and S. G ¨unnemann, “On out-of-distribution detection with energy-based models,” arXiv preprint arXiv:2107.08785, 2021

2021 arXiv
[22]

Effortless training of joint energy-based models with sliced score matching,

X. Liu, D. Staudt, C.-T. Lin, and C. Zach, “Effortless training of joint energy-based models with sliced score matching,” in 2022 26th International Conference on Pattern Recognition (ICPR) . IEEE, 2022, pp. 2643–2649

2022
[23]

Generative modeling by estimating gra- dients of the data distribution,

Y . Song and S. Ermon, “Generative modeling by estimating gra- dients of the data distribution,” Advances in neural information processing systems, vol. 32, 2019

2019
[24]

Flow straight and fast: Learning to generate and transfer data with rectified flow,

X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” arXiv preprint arXiv:2209.03003, 2022

2022 arXiv
[25]

Glow-tts: A genera- tive flow for text-to-speech via monotonic alignment search,

J. Kim, S. Kim, J. Kong, and S. Yoon, “Glow-tts: A genera- tive flow for text-to-speech via monotonic alignment search,”Ad- vances in Neural Information Processing Systems , vol. 33, pp. 8067–8077, 2020

2020
[26]

Score-based generative modeling through stochas- tic differential equations,

Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochas- tic differential equations,” arXiv preprint arXiv:2011.13456 , 2020

2011 arXiv
[27]

Denoising diffusion probabilis- tic models,

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilis- tic models,” Advances in neural information processing systems , vol. 33, pp. 6840–6851, 2020

2020
[28]

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,

M. Gutmann and A. Hyv ¨arinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in Proceedings of the thirteenth international conference on artifi- cial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010,...

2010
[29]

Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions,

J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerrv-Ryan et al. , “Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions,” in 2018 IEEE international conference on acoustics, speech and signal process...

2018
[30]

Neural speech synthe- sis with transformer network,

N. Li, S. Liu, Y . Liu, S. Zhao, and M. Liu, “Neural speech synthe- sis with transformer network,” in Proceedings of the AAAI con- ference on artificial intelligence, vol. 33, no. 01, 2019, pp. 6706– 6713

2019
[31]

Fastspeech 2: Fast and high-quality end-to-end text to speech,

Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” arXiv preprint arXiv:2006.04558, 2020

2006 arXiv
[32]

Matcha-tts: A fast tts architecture with conditional flow match- ing,

S. Mehta, R. Tu, J. Beskow, ´E. Sz ´ekely, and G. E. Henter, “Matcha-tts: A fast tts architecture with conditional flow match- ing,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 341–11 345

2024
[33]

Flow matching for generative modeling,

Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” arXiv preprint arXiv:2210.02747, 2022

2022 arXiv
[34]

Iterative α-(de) blending: A minimalist deterministic diffusion model,

E. Heitz, L. Belcour, and T. Chambon, “Iterative α-(de) blending: A minimalist deterministic diffusion model,” inACM SIGGRAPH 2023 Conference Proceedings, 2023, pp. 1–8

2023
[35]

Estimation of non-normalized statistical models,

A. Hyv ¨arinen, J. Hurri, P. O. Hoyer, A. Hyv ¨arinen, J. Hurri, and P. O. Hoyer, “Estimation of non-normalized statistical models,” Natural Image Statistics: A Probabilistic Approach to Early Com- putational Vision, pp. 419–426, 2009

2009
[36]

Estimating the hessian by back-propagating curvature,

J. Martens, I. Sutskever, and K. Swersky, “Estimating the hessian by back-propagating curvature,” arXiv preprint arXiv:1206.6464, 2012

2012 arXiv
[37]

The lj speech dataset,

K. Ito and L. Johnson, “The lj speech dataset,” https://keithito. com/LJ-Speech-Dataset/, 2017

2017
[38]

U-net: Convolutional networks for biomedical image segmentation,

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. S...

2015
[39]

Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,

J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,” Advances in neural information processing systems , vol. 33, pp. 17 022– 17 033, 2020

2020
[40]

SpeechBERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,

T. Saeki, S. Maiti, S. Takamichi, S. Watanabe, and H. Saruwatari, “SpeechBERTScore: Reference-aware automatic evaluation of speech generation leveraging nlp evaluation metrics,” in Inter- speech 2024, 2024, pp. 4943–4947

2024
[41]

The t05 sys- tem for the voicemos challenge 2024: Transfer learning from deep image classifier to naturalness mos prediction of high-quality syn- thetic speech,

K. Baba, W. Nakata, Y . Saito, and H. Saruwatari, “The t05 sys- tem for the voicemos challenge 2024: Transfer learning from deep image classifier to naturalness mos prediction of high-quality syn- thetic speech,” in2024 IEEE Spoken Language Technology Work- shop (SLT). IEEE, 2...

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Introduction Recently, energy-based models (EBMs) [1, 2] have drawn some attention in the text-to-speech (TTS) area [3]. These proba- bilistic models define the log-likelihood of speech given text as the difference between negative (unnormalised) energy function and the logarithm of the corresponding normalisation term. Due to the difficulty of computing ...

work page

[2] [2]

first application of SSM to training EBMs in TTS

work page

[3] [3]

Section 2 discusses EBMs and related work, including training methods and con- nections between EBMs and diffusion models

a new score-based loss function for EBMs; and the rest of it is organised as follows. Section 2 discusses EBMs and related work, including training methods and con- nections between EBMs and diffusion models. Section 3 de- scribes existing score-based training approached and introduces delta loss as a possible alternative. Experimental results and dis- cu...

work page

[4] [4]

Energy-Based Models Given a sequence of text tokens x, an energy-based model (EBM) of spectral feature sequencesY could be defined as [3] pθ(Y|x) = 1 Zθ(x) exp(−Eθ(x,Y )), (1) whereEθ(x,Y ) is an energy function with parameters θ that connects text and speech such that smaller values correspond to better matches between text and speech and vice versa, and...

work page Pith review arXiv 2025

[5] [5]

Score-based Training This section will first discuss score matching approaches and then will introduce a new criterion, which is shown to be linked with currently popular flow matching [16, 25, 26], that ad- dresses some of score matching limitations. 3.1. Sliced score matching (SSM) Rather than using NCE it is also possible to optimise EBMs by matching t...

work page

[6] [6]

(9) Compared to both NCE and (S)SM, delta loss is a very efficient training approach. 3.3. Connection to flow matching Interestingly, the delta loss in equation (9) can be linked to the Flow Matching (FM) loss [16, 25, 26] Lfm θ (x,Y +,Y 0,t ) = 1 2∥V (x,Yt,t )− (Y +−Y 0)||2, (10) where t ∈ [0, 1], Y 0 is a source or anchor sample possibly influenced by x...

work page

[7] [7]

Experimental setup 4.1.1

Experiments 4.1. Experimental setup 4.1.1. Dataset The dataset utilized in this work is LJSpeech [29], which con- sists of 13,100 audio clips totalling approximately 24 hours of speech from a single female speaker reading passages from pub- lic domain texts. The dataset is randomly divided into training (10,000 clips), validation (1,800 clips), and test (...

work page

[8] [8]

The first approach is sliced score matching (SSM) that has been previously used with EBMs in other areas but not in TTS

Conclusion This paper presented two new score-based approaches to train- ing energy-based models (EBMs) for text-to-speech (TTS) as an alternative to more challenging Noise Contrastive Estima- tion (NCE). The first approach is sliced score matching (SSM) that has been previously used with EBMs in other areas but not in TTS. The second approach is a new ob...

work page