Pith. sign in

REVIEW 5 major objections 5 minor 92 references

FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FlowDubber claims that replacing duration prediction with LLM-based semantic-aware learning, dual contrastive lip-phoneme alignment, and flow-matching voice enhancement yields near-ground-truth lip-sync and competitive word error rates on…

desk verdict Plausible dubbing architecture with a real lip-sync contribution, but the headline numbers rest on automatic predictors that may reward artifacts; worth refereeing with a demand for human evaluation. read the letter →

arxiv 2505.01263 v2 pith:P6QJYHEL submitted 2025-05-02 cs.MM cs.CVcs.SDeess.AS

Gaoxiang Cong , Liang Li , Jiadong Pan , Zhedong Zhang , Amin Beheshti , Anton van den Hengel , Yuankai Qi , Qingming Huang This is my paper · ORCID
classification cs.MMcs.CVcs.SDeess.AS
keywords moviedubbingvisualvoicecloninglipsynchronizationflowmatchingspeechlanguagemodelcontrastivealignmentclassifier-freeguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FlowDubber sets out to fix two weaknesses it attributes to earlier movie-dubbing systems: coarse lip-sync, because duration prediction only approximates timing, and degraded acoustic quality, because standard vocoder pipelines blur spectral detail. The paper's solution is an architecture that replaces the duration predictor with a speech language model that learns phoneme-level semantics from the script and reference voice, a dual contrastive alignment that ties those phonemes to lip motion in both directions, and a flow-matching voice-enhancement stage that sharpens clarity while preserving speaker identity. The headline result is that on the Chem and GRID dubbing benchmarks this architecture reaches lip-sync scores near the ground-truth audio: on Chem setting 1.0, lip-sync confidence (LSE-C) is 8.21 versus ground truth 8.12 and the best baseline 3.87, while word error rate stays comparable to the previous state of the art. If true, the payoff is that dubbing could achieve near-ground-truth automatic lip-sync and voice cloning from a short reference clip without a separate duration-prediction stage.

What carries the argument

Three modules carry the argument. LLM-based Semantic-aware Learning (LLM-SL) first converts the reference audio into semantic and global tokens through speech tokenization, then feeds script text, semantic tokens, and global tokens to a pretrained text-based language model that autoregressively predicts speech tokens, and finally adapts those token semantics to the phoneme level with cross-modal transformers whose keys and values come from the LLM-mapped speech features. Dual Contrastive Aligning (DCA) takes phoneme embeddings and lip-motion embeddings extracted from the silent video, applies contrastive losses in both directions so each lip frame retrieves its temporally aligned phoneme and vice versa, builds a similarity matrix from the learned alignment, and uses monotonic alignment search to convert that matrix into a per-phoneme frame mapping. Flow-based Voice Enhancing (FVE) then trains a conditional flow-matching network with an optimal-transport interpolation from noise to mel-spectrogram, injects speaker style through an affine prior, and refines the generated spectrum with a decoupled classifier-free guidance term that amplifies the LLM-conditioned vector field over a zero-semantics field. Together these convert script, reference voice, and silent video into a mel-spectrogram prior that already carries the lip alignment before any vocoder is applied.

What would settle it

A forced-choice human study on held-out Chem and GRID clips would settle the perceptual claim: if raters do not consistently prefer FlowDubber's dubs to the strongest prior system for lip-sync and clarity, or if the guidance-scale setting that maximizes the neural clarity scores is not the one humans prefer, the reported metric gap is not evidence of perceptual gain. A cheaper diagnostic is to check whether the lip-sync metric still rises when synthetic sharpening is added at the wrong moments; if it does, the metric is gameable.

Watch

Extended reading notes

Core claim

The paper claims that an LLM-based semantic-aware learning module plus dual contrastive alignment makes lip-sync a first-class objective rather than a byproduct of duration prediction, and that a flow-matching decoder with LLM-based guidance produces clearer audio than prior dubbing vocoders. Concretely, FlowDubber reports state-of-the-art results on both benchmark datasets. On Chem setting 1.0 it reports LSE-C 8.21 (ground truth 8.12, best prior dubbing baseline 3.87), LSE-D 6.89 (ground truth 6.59, best prior 10.56), UTMOS 3.91, SIM-O 0.754, and WER 9.96; on GRID it reaches LSE-C 7.27 (ground truth 7.13) at setting 1.0. In a zero-shot test with an unseen reference speaker it beats all compared methods on every reported metric. The ablations attribute pronunciation to the LLM semantic learning, lip-sync to the dual contrastive alignment, and identity to the style prior in flow matching.

Load-bearing premise

The load-bearing premise is that the automated scores used as headline evidence—a learned lip-sync confidence/distance measure and neural estimates of clarity and naturalness—faithfully capture what human viewers and listeners perceive, since the paper reports no human evaluation and on one setting its lip-sync score exceeds the ground-truth audio's score.

Editorial extensions

If this is right

  • On the paper's evidence, a movie-dubbing system can reach near-ground-truth lip-sync on automatic metrics without a duration predictor, by learning mutual alignment between phoneme and lip-motion embeddings.
  • The LLM's phoneme-level semantic features are what keep pronunciation accurate: removing LLM-SL in the ablation raises WER from 9.96 to 48.33 on Chem setting 1.0.
  • The division of labor is clean: the flow-matching style prior carries speaker identity, and the LLM-based guidance scale monotonically improves clarity metrics (DNSMOS, SNR, UTMOS) when increased from 0.0 to 0.8.
  • Because lip alignment is baked into the mel-spectrogram prior before the vocoder, the same front-end can work with different neural vocoders and still preserve synchronization; the paper shows this by swapping in alternative generators and retaining high LSE-C/LSE-D.
  • The system generalizes to unseen reference voices in a zero-shot setting, so it does not require per-speaker adaptation to keep its lip-sync advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported LSE-C exceeding ground-truth audio's LSE-C on Chem (8.21 vs 8.12) suggests the lip-sync metric may reward artifacts; a natural extension is to test whether human raters still prefer FlowDubber over real audio.
  • The guidance scale alpha is presented only through clarity metrics; testing whether higher alpha changes LSE-C/LSE-D or speaker similarity would map the trade-off between the LLM-enhancement path and the visual alignment path.
  • The decoupling of LLM semantics from lip-aligning features suggests the method could be ported to cross-lingual dubbing by keeping the alignment path and swapping the text-phoneme side, but the paper does not test this.
  • Since all acoustic-quality conclusions rest on learned predictors, an independent human-subject study is the cheapest way to decide whether the voice-enhancing contribution is perceptual or only numeric.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FlowDubber, an end-to-end movie dubbing system combining a Qwen2.5-based speech language model with semantic-aware phoneme learning, dual contrastive alignment (DCA) between lip motion and phonemes, and a flow-matching decoder enhanced by classifier-free guidance. The authors report large improvements over three prior dubbing methods on the Chem and GRID benchmarks, particularly in SyncNet-based lip-sync metrics (LSE-C/LSE-D), with comparable or slightly better WER and UTMOS. The ablations show that removing DCA collapses lip-sync performance, removing LLM-SL degrades WER significantly, and the proposed FVE guidance raises automatic clarity scores. The paper claims state-of-the-art results on both benchmarks.

Significance. If the results are valid, the paper makes a strong contribution: it demonstrates that a lip-motion-conditioned alignment module can be integrated into an LLM-based speech synthesis pipeline, and that the resulting system can achieve near-ground-truth automatic lip-sync scores while cloning a reference voice. The architectural combination is innovative and internally consistent, and the ablations are informative: Table 5 clearly attributes the lip-sync gain to DCA and the pronunciation gain to LLM-SL. The paper also ships a demo page and uses reproducible public benchmarks. The main qualification is epistemic: all headline metrics are automatic predictors, no human evaluation is reported, and some recent dubbing baselines cited in the related work are absent from the comparisons. These issues affect the strength of the central SOTA claim but do not invalidate the architecture itself.

major comments (5)
  1. [Tables 1-3] The central claim that FlowDubber 'sets SOTA results on both Chem and GRID benchmarks' is based only on comparisons with StyleDubber, Speaker2Dubber, and ProDubber. Several more recent dubbing methods cited in Section 2.1 and the reference list, including EmoDubber [12], MCDubber [83], VoiceCraft-Dub [56], DeepDubber-V1 [84], and FCConDubber [36], are not evaluated in any table. To support the SOTA claim, at least the strongest of these recent methods should be included in the benchmark comparison, or the claim should be qualified to the three compared baselines.
  2. [Section 4.3 and Table 4 caption] The paper's headline claims of 'high-quality audio-visual sync' and 'better acoustic quality' rest entirely on automatic predictors: LSE-C/LSE-D from SyncNet, UTMOS, DNSMOS, SNR, SIM-O, and WER. The Table 4 caption itself states that DNSMOS, SNR, and UTMOS are not human subjective metrics, but no human listening or subjective MOS evaluation is reported anywhere in the paper. This is a load-bearing gap because the lip-sync claim is the main differentiator of the method. A human perceptual evaluation, or at least an analysis of how well the automatic metrics correlate with human judgments on these datasets, is needed before the SOTA claim can be accepted.
  3. [Table 1, Setting 1.0] FlowDubber's LSE-C on Chem (8.21) exceeds the ground-truth audio's LSE-C (8.12), and its LSE-D (6.89) is close to ground truth (6.59). If LSE-C/D measure genuine synchronization with the video, a generated utterance should not systematically outperform the real recorded audio. This anomaly suggests that SyncNet may reward artifacts of the generated mel-spectrogram or that the metric is not a faithful proxy for perceptual sync quality. The paper should explain this behavior or provide evidence, such as per-sample analysis or a human sync evaluation, that the generated audio is truly as well synchronized as the ground truth.
  4. [Section 4.5 and Table 4] The guidance scale alpha is tuned empirically to maximize DNSMOS, SNR, and UTMOS on the same test set, and the same three metrics are then used to claim that FVE improves clarity. This is circular: the monotonic trend in Table 4 is expected when alpha is selected to optimize those exact predictors. The paper should report alpha selection on a validation set and evaluate the chosen model with held-out metrics or human listening, otherwise the clarity improvement claim is not independently established.
  5. [Tables 1-6] All reported numbers are single-run point estimates without error bars, confidence intervals, or statistical significance tests. Given that the performance gaps in Tables 1 and 3 are extremely large for LSE-C/D, a small number of runs across multiple seeds would materially strengthen the claim and would help rule out metric instability or training stochasticity as an explanation.
minor comments (5)
  1. [Table 5] The header 'UTMOS↓' is incorrect: higher UTMOS is better, as stated in Table 1. The arrow should be 'UTMOS↑'.
  2. [Eq. (10)] The formula 'mu = F([C_lip, Up(LLM_p, z_p, t_ab))' has mismatched parentheses; it should be 'F([C_lip, Up(LLM_p, z_p, t_ab)])'.
  3. [Eq. (4)] The second line of Eq. (4) appears to have duplicated arguments: 'f_theta(LN(Z_hat) + LN(Z_hat))' should likely be 'LN(Z_hat) + LN(Z_prev)' or similar. Please clarify.
  4. [Section 4.4, Chem results] The text says 'absolute improvements of 5.63% and 5.65%' for LSE-C and LSE-D, but these appear to be absolute differences in metric values, not percentages. Consider rewording to 'improvements of 5.63 and 5.65 in LSE-C and LSE-D'.
  5. [Section 4.7] The ablation of audio generators is useful, but the statement that for DAC and CV the original waveform is generated and then reconstructed is unclear. Clarify whether the flow-matching mel-spectrogram is decoded, re-encoded, and then vocoded, or whether the codec is used in some other reconstruction pipeline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SOTA claims rest on external benchmarks and metrics, not on self-derived identities or fitted predictions.

full rationale

FlowDubber's central claims are empirical comparisons against external baselines (Tables 1–3, 7) on the Chem and GRID benchmarks, measured with independent pretrained metrics (SyncNet LSE-C/LSE-D, Whisper WER, UTMOS, SIM-O). None of the method equations reduces to its own evaluation target by construction: DCA uses ground-truth MFA timing as supervision during training, which is a legitimate training signal rather than a metric fitted to the test set, and the final lip-sync scores come from an external SyncNet not used in the loss. The FVE clarity analysis in Table 4 sweeps the guidance scale alpha and reports DNSMOS/SNR/UTMOS on the same sweep; this is a hyperparameter behavior report, not a renamed fitted parameter presented as an independent prediction, and the caption explicitly cautions that these are not human subjective metrics. That caveat is a correctness/validity limitation, not circularity. Self-citations appear for baseline methods, the lip-motion extractor, and attention components, but these are normal component reuse and do not carry the load-bearing SOTA argument. No equation, fitted parameter, or uniqueness claim was found that is equivalent to its input by definition.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The experimental claims depend on pretrained model transfer and on automatic metrics; the only explicitly tuned scalar is the CFG guidance scale alpha, selected on the quality metrics used for evaluation.

free parameters (2)
  • Guidance scale alpha = 0.0 to 0.8, chosen empirically; best quality claims at 0.8
    Equation 12. Section 4.5 tunes alpha by measuring DNSMOS, SNR, and UTMOS, the same metrics used to claim FVE improves clarity. Main tables use alpha=0.0.
  • InfoNCE temperature tau = 0.1
    Set in Implementation Details for both L_mp and L_pm; the contrastive learning dynamics depend on it.
assumptions (4)
  • domain assumption Montreal Forced Aligner (MFA) provides accurate phoneme-to-audio timing for constructing positive pairs in the dual contrastive loss.
    Section 3.3, Eq. 7. If MFA alignments are wrong, the contrastive objective trains on incorrect lip-phoneme correspondences and the claimed lip-sync accuracy is not grounded.
  • domain assumption SyncNet LSE-C/LSE-D and UTMOS/DNSMOS/SNR are valid proxies for human-perceived lip-sync and acoustic clarity.
    Section 4.3 and Table 4 caption. All headline comparisons rest on these automatic metrics; the paper itself notes they are not human subjective metrics, and no listening test is reported.
  • domain assumption The pretrained modules (wav2vec 2.0, Qwen2.5-0.5B, ECAPA-TDNN, lip encoder, HiFi-GAN) transfer reliably to the movie dubbing domain.
    Sections 3.2-3.4 and 4.7. If these modules degrade on noisy, accented movie audio or on the Chem/GRID distributions, the stacked pipeline inherits the errors.
  • domain assumption The lip-motion features extracted by the frozen lip encoder are informative enough for phonetic alignment at 25 FPS.
    Section 3.3, Eq. 6. The DCA module can only align phonemes to lip movements if the lip features carry articulatory timing information; the paper does not analyze failure cases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing." pith.science (2026). https://pith.science/paper/P6QJYHEL

@misc{pith2026250501263,
  author       = {Pith},
  title        = {Pith review of: FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P6QJYHEL}},
  note         = {Machine review of arXiv:2505.01263}
}
read the original abstract

Movie Dubbing aims to convert scripts into speeches that align with the given movie clip in both temporal and emotional aspects while preserving the vocal timbre of a given brief reference audio. Existing methods focus primarily on reducing the word error rate while ignoring the importance of lip-sync and acoustic quality. To address these issues, we propose a large language model (LLM) based flow matching architecture for dubbing, named FlowDubber, which achieves high-quality audio-visual sync and pronunciation by incorporating a large speech language model and dual contrastive aligning while achieving better acoustic quality via the proposed voice-enhanced flow matching than previous works. First, we introduce Qwen2.5 as the backbone of LLM to learn the in-context sequence from movie scripts and reference audio. Then, the proposed semantic-aware learning focuses on capturing LLM semantic knowledge at the phoneme level. Next, dual contrastive aligning (DCA) boosts mutual alignment with lip movement, reducing ambiguities where similar phonemes might be confused. Finally, the proposed Flow-based Voice Enhancing (FVE) improves acoustic quality in two aspects, which introduces an LLM-based acoustics flow matching guidance to strengthen clarity and uses affine style prior to enhance identity when recovering noise into mel-spectrograms via gradient vector field prediction. Extensive experiments demonstrate that our method outperforms several state-of-the-art methods on two primary benchmarks.

Figures

Figures reproduced from arXiv: 2505.01263 by the authors.

Figure 1
Figure 1. (a) V2C task. (b) Unlike dubbing with duration pre [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of FlowDubber. It consists of LLM-based Semantic-aware Learning (LLM-SL), Dual Contrastive [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The visualization of the mel-spectrograms of ground truth (GT) and synthesized audios obtained by different models. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 49 canonical work pages

  1. [12]

    Gaoxiang Cong, Jiadong Pan, Liang Li, Yuankai Qi, Yuxin Peng, Anton van den Hengel, Jian Yang, and Qingming Huang. 2024. EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing.arXiv preprint arXiv:2412.08988 (2024)

    arXiv 2024

  2. [83]

    Yuan Zhao, Zhenqi Jia, Rui Liu, De Hu, Feilong Bao, and Guanglai Gao. 2024. MCDubber: Multimodal Context-Aware Expressive Video Dubbing.arXiv preprint arXiv:2408.11593 (2024)

    arXiv 2024

  3. [56]

    Kim Sung-Bin, Jeongsoo Choi, Puyuan Peng, Joon Son Chung, Tae-Hyun Oh, and David Harwath. 2025. VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models. arXiv preprint arXiv:2504.02386 (2025)

    arXiv 2025

  4. [84]

    Junjie Zheng, Zihao Chen, Chaofan Ding, and Xinhan Di. 2025. DeepDubber- V1: Towards High Quality and Dialogue, Narration, Monologue Adaptive Movie Dubbing Via Multi-Modal Chain-of-Thoughts Reasoning Guidance.arXiv preprint arXiv:2503.23660 (2025)

    arXiv 2025

  5. [36]

    Qiulin Li, Zhichao Wu, Hanwei Li, Xin Dong, and Qun Yang. 2025. FCConDubber: Fine And Coarse Grained Prosody Alignment For Expressive Video Dubbing via Contrastive Audio-Motion Pretraining. In ICASSP. 1–5

    work page 2025

  6. [1]

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al . 2024. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430 (2024)

    arXiv 2024

  7. [2]

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representa- tions. In NIPS

    2020

  8. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS

    2020

Show all 92 references
  1. [4]

    Qi Chen, Mingkui Tan, Yuankai Qi, Jiaqiu Zhou, Yuanqing Li, and Qi Wu. 2022. V2C: Visual Voice Cloning. In CVPR. 21210–21219

    2022

  2. [5]

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural ordinary differential equations. Advances in neural information processing systems 31 (2018)

    2018

  3. [6]

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. 2024. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370 (2024)

    2024 arXiv

  4. [7]

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2024. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885 (2024)

    2024 arXiv

  5. [8]

    Jeongsoo Choi, Ji-Hoon Kim, Jinyu Li, Joon Son Chung, and Shujie Liu. 2025. V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow. In ICASSP. IEEE, 1–5

    2025

  6. [9]

    Jeongsoo Choi, Se Jin Park, Minsu Kim, and Yong Man Ro. 2024. Av2av: Direct audio-visual speech to audio-visual speech translation with unified audio-visual speech representation. In CVPR. 27325–27337

    2024

  7. [10]

    Joon Son Chung and Andrew Zisserman. 2016. Out of Time: Automated Lip Sync in the Wild. In ACCV Workshop. 251–263

    2016

  8. [11]

    Gaoxiang Cong, Liang Li, Yuankai Qi, Zheng-Jun Zha, Qi Wu, Wenyu Wang, Bin Jiang, Ming-Hsuan Yang, and Qingming Huang. 2023. Learning to Dub Movies via Hierarchical Prosody Models. In CVPR. 14687–14697

    2023

  9. [13]

    Gaoxiang Cong, Yuankai Qi, Liang Li, Amin Beheshti, Zhedong Zhang, Anton van den Hengel, Ming-Hsuan Yang, Chenggang Yan, and Qingming Huang. 2024. StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing. InFindings of ACL. 6767–6779

    2024

  10. [14]

    Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. 2006. An audio- visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America 120, 5 (2006), 2421–2424

    2006

  11. [15]

    Yiming Cui, Liang Li, Jiehua Zhang, Chenggang Yan, Hongkui Wang, Shuai Wang, Heng Jin, and Li Wu. 2024. Stochastic context consistency reasoning for domain adaptive object detection. In ACM MM. 1331–1340

    2024

  12. [16]

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407 (2024)

    2024 arXiv

  13. [17]

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al . 2024. Cosyvoice 2: Scal- able streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117 (2024)

    2024 arXiv

  14. [18]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  15. [19]

    Hao-Han Guo, Kun Liu, Fei-Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. 2024. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03283 (2024)

    2024 arXiv

  16. [20]

    Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. 2024. VoiceFlow: Efficient Text-To-Speech with Rectified Flow Matching. InICASSP. 11121–11125

    2024

  17. [21]

    Chenxu Hu, Qiao Tian, Tingle Li, Yuping Wang, Yuxuan Wang, and Hang Zhao

  18. [22]

    Youngjoon Jang, Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Hongsun Yang, Yooncheol Ju, Ilhwan Kim, Byeong-Yeol Kim, and Joon Son Chung. 2024. Faces that Speak: Jointly Synthesising Talking Face and Speech from Text. In CVPR. 8818–8828

    2024

  19. [24]

    Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al . 2024. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532 (2024)

    2024 arXiv

  20. [25]

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiangyang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao. 2024. Nat- uralSpeech 3: Zero-Shot Speech Synthesis...

    2024

  21. [26]

    Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. 2020. Glow- TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search. In NeurIPS

    2020

  22. [27]

    Ji-Hoon Kim, Jeongsoo Choi, Jaehun Kim, Chaeyoung Jung, and Joon Son Chung

  23. [28]

    Shih, Rohan Badlani, João Felipe Santos, Evelina Bakh- turina, Mikyas Desta, Rafael Valle, Sungroh Yoon, and Bryan Catanzaro

    Sungwon Kim, Kevin J. Shih, Rohan Badlani, João Felipe Santos, Evelina Bakh- turina, Mikyas Desta, Rafael Valle, Sungroh Yoon, and Bryan Catanzaro. 2023. P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting. In NeurIPS

    2023

  24. [29]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. In NIPS. 17022–17033

    2020

  25. [30]

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. 2023. High-fidelity audio compression with improved rvqgan. InNeurIPS

    2023

  26. [31]

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. 2023. Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale. In NeurIPS

    2023

  27. [32]

    Jiyoung Lee, Joon Son Chung, and Soo-Whan Chung. 2023. Imaginary Voice: Face-Styled Diffusion Model for Text-to-Speech. In ICASSP. 1–5

    2023

  28. [33]

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon

  29. [34]

    Hao Li, DeLiang Wang, Xueliang Zhang, and Guanglai Gao. 2020. Frame-Level Signal-to-Noise Ratio Estimation Using Deep Learning.. In Interspeech. 4626– 4630

    2020

  30. [35]

    Sheng, Qingming Huang, and Ming-Hsuan Yang

    Liang Li, Gaoxiang Cong, Yuankai Qi, Zheng-Jun Zha, Qi Wu, Quan Z. Sheng, Qingming Huang, and Ming-Hsuan Yang. 2025. Dubbing Movies via Hierarchical Phoneme Modeling and Acoustic Diffusion Denoising. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025), 1–17

    2025

  31. [37]

    Raghavan, Gavin Mischler, and Nima Mesgarani

    Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, and Nima Mesgarani. 2023. StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models. In NeurIPS

    2023

  32. [38]

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le

  33. [39]

    Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Zechao Li, Qi Tian, and Qingming Huang. 2023. Entity-Enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding. IEEE PAMI 45, 3 (2023), 3003–3018

    2023

  34. [40]

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022. A ConvNet for the 2020s. In CVPR. 11966–11976

    2022

  35. [41]

    Junchen Lu, Berrak Sisman, Rui Liu, Mingyang Zhang, and Haizhou Li. 2022. Visualtts: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over. In ICASSP. 8032–8036

    2022

  36. [42]

    Pingchuan Ma, Brais Martinez, Stavros Petridis, and Maja Pantic. 2021. Towards Practical Lipreading with Distilled and Efficient Models. In ICASSP. 7608–7612

    2021

  37. [43]

    Brais Martinez, Pingchuan Ma, Stavros Petridis, and Maja Pantic. 2020. Lipreading Using Temporal Convolutional Networks. In ICASSP. 6319–6323

    2020

  38. [44]

    Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi.. In Interspeech. 498–502. MM ’25, October 27–31, 2025, Dublin, Ireland Gaoxiang Cong et al

    2017

  39. [45]

    Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter

  40. [46]

    Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. 2024. Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.08551 (2024)

    2024 arXiv

  41. [47]

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. 2024. Finite Scalar Quantization: VQ-VAE Made Simple. InICLR

    2024

  42. [48]

    Andrew Cameron Morris, Viktoria Maier, and Phil D. Green. 2004. From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. In Interspeech. 2765–2768

    2004

  43. [49]

    K. R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C. V. Jawahar

  44. [50]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In ICML. 28492–28518

    2023

  45. [51]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

    2019

  46. [52]

    Chandan KA Reddy, Vishak Gopal, and Ross Cutler. 2021. DNSMOS: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6493–6497

    2021

  47. [53]

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu

  48. [54]

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152 (2022)

    2022 arXiv

  49. [55]

    Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. 2021. Maximum like- lihood training of score-based diffusion models. Advances in neural information processing systems 34 (2021), 1415–1428

    2021

  50. [57]

    Wei Tang, Liang Li, Xuejing Liu, Lu Jin, Jinhui Tang, and Zechao Li. 2024. Context disentangling and prototype inheriting for robust visual grounding. IEEE PAMI 46, 5 (2024), 3213–3229

    2024

  51. [58]

    Wei Tang, Yanpeng Sun, Qinying Gu, , and Zechao Li. 2025. Visual Position Prompt for MLLM based Visual Grounding. IEEE Trans. Multimedia (2025)

    2025

  52. [59]

    Yuandong Tian. 2022. Understanding Deep Contrastive Learning via Coordinate- wise Optimization. In NeurIPS

    2022

  53. [60]

    FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. In ICLR

  54. [61]

    Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural Discrete Representation Learning. In NeurIPS. 6306–6315

    2017

  55. [62]

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al . 2023. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111 (2023)

    2023 arXiv

  56. [63]

    Tan, and Haizhou Li

    Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan, and Haizhou Li. 2023. Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert. In CVPR. 14653–14662

    2023

  57. [64]

    Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. 2025. Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710 (2025)

    2025 arXiv

  58. [65]

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. 2024. Maskgct: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750 (2024)

    2024 arXiv

  59. [66]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  60. [67]

    Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, and Qingming Huang. 2024. SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning. IEEE PAMI 46, 7 (2024), 4926–4943

    2024

  61. [68]

    Jiaxin Ye, Boyuan Cao, and Hongming Shan. 2025. Emotional Face-to-Speech. arXiv preprint arXiv:2502.01046 (2025)

    2025 arXiv

  62. [69]

    Jiaxin Ye and Hongming Shan. 2025. Shushing! Let’s Imagine an Authentic Speech from the Silent Video. arXiv preprint arXiv:2503.14928 (2025)

    2025 arXiv

  63. [70]

    Jiaxin Ye, Xin-Cheng Wen, Yujie Wei, Yong Xu, Kunhong Liu, and Hongming Shan. 2023. Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition. In ICASSP. 1–5

    2023

  64. [71]

    Zhaoda Ye, Xiangteng He, and Yuxin Peng. 2022. Unsupervised Cross-Media Hashing Learning via Knowledge Graph. Chinese Journal of Electronics 31, 6 (2022), 1081–1091

    2022

  65. [72]

    Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. 2024. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. arXiv preprint arXiv:2408.17175 (2024)

    2024 arXiv

  66. [73]

    Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi DAI, et al . 2025. Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis. arXiv preprint arXiv:2502.04128 (2025)

    2025 arXiv

  67. [74]

    Jixun Yao, Yuguang Yang, Yu Pan, Ziqian Ning, Jiaohao Ye, Hongbin Zhou, and Lei Xie. 2024. Stablevc: Style controllable zero-shot voice conversion with conditional flow matching. arXiv preprint arXiv:2412.04724 (2024)

    2024 arXiv

  68. [75]

    Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. LibriTTS: A Corpus Derived from LibriSpeech for Text- to-Speech. In Interspeech. 1526–1530

    2019

  69. [76]

    Beichen Zhang, Liang Li, Shuhui Wang, Shaofei Cai, Zheng-Jun Zha, Qi Tian, and Qingming Huang. 2024. Inductive State-Relabeling Adversarial Active Learning With Heuristic Clique Rescaling. IEEE PAMI 46, 12 (2024), 9780–9796

    2024

  70. [77]

    Haomin Zhang, Chang Liu, Junjie Zheng, Zihao Chen, Chaofan Ding, and Xinhan Di. 2025. DeepAudio-V1: Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generation. arXiv preprint arXiv:2503.22265 (2025)

    2025 arXiv

  71. [78]

    Tao Zhang, Ying Fu, and Jun Zhang. 2024. Deep Guided Attention Network for Joint Denoising and Demosaicing in Real Image. Chinese Journal of Electronics 33, 1 (2024), 303–312

    2024

  72. [79]

    Xueyao Zhang, Yuancheng Wang, Chaoren Wang, Ziniu Li, Zhuo Chen, and Zhizheng Wu. 2025. Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment. In ACL. 12251–12270

    2025

  73. [80]

    Zhedong Zhang, Liang Li, Gaoxiang Cong, YIN Haibing, Yuhan Gao, Chenggang Yan, Anton van den Hengel, and Yuankai Qi. 2024. From Speaker to Dubber: Movie Dubbing with Prosody and Duration Consistency Learning. In ACM MM

    2024

  74. [81]

    Yochai Yemini, Aviv Shamsian, Lior Bracha, Sharon Gannot, and Ethan Fetaya

  75. [82]

    LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading. In ICLR

  76. [85]

    Xinfa Zhu, Wenjie Tian, and Lei Xie. 2024. Autoregressive Speech Synthesis with Next-Distribution Prediction. arXiv preprint arXiv:2412.16846 (2024)

    2024

  77. [89]

    Zhedong Zhang, Liang Li, Chenggang Yan, Chunshan Liu, Anton van den Hengel, and Yuankai Qi. 2025. Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing. arXiv preprint arXiv:2503.12042 (2025)

    2025 arXiv

  78. [90]

    Zhedong Zhang, Liang Li, Jiehua Zhang, Zhenghui Hu, Hongkui Wang, Cheng- gang Yan, Jian Yang, and Yuankai Qi. 2024. Generating High-Quality Symbolic Music Using Fine-Grained Discriminators. In ICPR. 332–344

    2024

  79. [2020]

    Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis. In CVPR. 13793–13802

  80. [2021]

    In NeurIPS

    Neural Dubber: Dubbing for Videos According to Scripts. In NeurIPS. 16582–16595

  81. [2022]

    arXiv preprint arXiv:2210.02747 (2022)

    Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022)

    2022 arXiv

  82. [2023]

    BigVGAN: A Universal Neural Vocoder with Large-Scale Training. InICLR

  83. [2024]

    In ICASSP

    Matcha-TTS: A fast TTS architecture with conditional flow matching. In ICASSP. 11341–11345

  84. [2025]

    arXiv preprint arXiv:2503.16956 (2025)

    From Faces to Voices: Learning Hierarchical Representations for High- quality Video-to-Speech. arXiv preprint arXiv:2503.16956 (2025)

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.