Pith. sign in

REVIEW 3 major objections 4 minor 37 references

Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Expressive video dubbing improves when the surrounding multimodal context is modeled at both sentence and phoneme scales and is allowed to interact with the current sentence, rather than being treated as a static conditioning signal.

desk verdict Competent, reimplementable dubbing paper with thorough ablations, but a missing control for the added pretrained emotion features leaves the main gain over MCDubber unproven. read the letter →

arxiv 2412.18748 v2 pith:5QRGLHCU submitted 2024-12-25 cs.MM cs.CLcs.SDeess.AS

Yuan Zhao , Rui Liu , Gaoxiang Cong This is my paper · ORCID
classification cs.MMcs.CLcs.SDeess.AS
keywords automaticvideodubbingprosodyexpressivenessmultiscalecontextmodelingmultimodalinteractiongraphattentionfusionspeechsynthesiscontext-awareprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that expressive video dubbing improves when the system models the surrounding context at two scales—sentence-level global prosody and phoneme-level local prosody—across video, text, and audio, and when those context cues interact with the current sentence rather than being treated in isolation. On the Context Chem dataset, the proposed M2CI-Dubber achieves the lowest gross pitch error and F0 frame error among baselines, and the highest subjective ratings for prosody alignment with context and similarity to ground truth, with the gains reported as statistically significant. If the claim holds, the practical recipe for better dubbing is not just more context but a hierarchy of context scales plus explicit interaction pathways between context and the sentence being spoken.

What carries the argument

The central mechanism is the M2CI encoder, a shared module applied to the previous and following sentences. It operates in three stages: Multiscale Feature Extraction produces a global sentence-level feature per modality from pretrained emotion/expression models (MAE-DFER for video, a RoBERTa-based emotion classifier for text, Wav2Vec 2.0 fine-tuned on IEMOCAP for audio) and a local quasi-phoneme-level feature per modality from frame-level extractors followed by temporal downsampling encoders; Interaction-based Multiscale Aggregation concatenates the current text with the global feature, applies self-attention, then cross-attends from that output to the local feature, with a residual concatenation of the global feature; Interaction-based Multimodal Fusion builds an undirected graph whose nodes are the aggregated global-local features and the current text, with intra-modal edges along time, inter-modal edges across modalities at the same time step, and interaction-based edges linking the current text to every modality node at the same time step, then fuses nodes with a multi-head graph attention encoder. The fused features are fed into a Context-Aware Adaptor that uses gated fusion and cross-attention to inject them into a HPMDubbing-style synthesizer.

What would settle it

Run the full model on Context Chem with the three global feature extractors replaced by fixed random vectors of the same dimension while keeping all interaction modules intact; if GPE, FFE, and MOS-C do not degrade substantially relative to the reported 38.01, 30.91, and 3.943, then the claimed gains are not attributable to multiscale context modeling. Alternatively, retrain the global extractors on the Context Chem training set itself and compare: if the gap to baselines shrinks, the original frozen features were the source of the improvement.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that prosody expressiveness in automatic video dubbing reduces to a two-scale, three-modality context modeling problem: the current sentence's pitch and energy should be conditioned on global sentence-level prosody features and local phoneme-level features extracted from the previous and following sentences' face frames, text, and audio, aggregated by attention mechanisms that first let the current text interact with the global features and then with the local features, and finally fused through a graph attention network that adds explicit edges from the current text to every modality at every time step. The authors report that M2CI-Dubber significantly outperforms FastSpeech2, DSU-AVO, HPMDubbing, and MCDubber on the Context Chem dataset on all prosody-related metrics (GPE 38.01, FFE 30.91, MOS-C 3.943, MOS-S 3.866), and that ablations removing any component—global features, local features, aggregation, fusion, interaction, a modality, or a neighboring sentence—degrade these metrics.

Load-bearing premise

The global prosody features come from frozen pretrained models trained on other datasets, and the paper assumes these features transfer prosody information to the Context Chem speaker; if they do not carry such information, the multiscale context gains would largely disappear.

Editorial extensions

If this is right

  • If the claim is correct, dubbing systems should condition on both the previous and the following sentence's multimodal context, not just the current sentence's face and text.
  • The multiscale design implies that sentence-level global prosody and phoneme-level local prosody carry complementary information; dropping either scale costs roughly 4-7 points in GPE in the ablations.
  • The interaction edges in aggregation and fusion are load-bearing: removing them from both stages produces the largest subjective drop, so context that is merely concatenated, not interacted with, is insufficient.
  • The gains in MOS-C and MOS-S (about 0.17-0.20 over MCDubber) suggest the model aligns prosody with the surrounding conversational context, which matters for dubbing feature films and serialized content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same two-scale interaction recipe could transfer to conversational speech synthesis and audiobook narration, where neighboring sentences also carry global speaking style and local emphasis cues; the paper does not test this.
  • A natural testable extension is to replace the frozen global extractors (MAE-DFER, RoBERTa emotion, Wav2Vec 2.0) with random projections or average pooling: if prosody gains persist, the interaction modules alone are responsible; if they vanish, the pretrained features dominate. The paper does not isolate this.
  • The graph interaction pattern—edges from the current token to each modality at the same time step—is a generic multimodal fusion idea that could be applied to other conditional generation tasks (e.g., expressive TTS conditioned on scene context), though that claim is not in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes M2CI-Dubber, an automatic video dubbing system that models multiscale (global sentence-level and local phoneme/frame-level) multimodal context from previous and following sentences and interacts it with the current text through two proposed modules: Interaction-based Multiscale Aggregation (IMA) and Interaction-based Multimodal Fusion (IMF) using a graph attention network. Experiments on the Context Chem dataset report that M2CI-Dubber outperforms FastSpeech2, DUS-AVO, HPMDubbing, and MCDubber on GPE, FFE, MOS-C, and MOS-S, with ablations showing contributions of global/local features, IMA, IMF, the context-aware adaptor, modalities, and context sides.

Significance. If the reported gains are attributable to the proposed multiscale multimodal context interaction architecture, this would be a useful advance for expressive video dubbing, where prosody expressiveness depends on surrounding context. The paper is clearly written, the architecture is specified in detail, the code and demos are promised, and the ablation study covers many components and interaction variants. However, the central attribution claim is not fully supported because the paper does not control for the additional frozen pretrained emotion/prosody features introduced by M2CI-Dubber relative to the strongest baseline, and the statistical evidence for the headline improvements is incompletely reported.

major comments (3)
  1. [Section II-A and Table II] The main claimed gain over MCDubber may come from the three frozen pretrained global feature extractors (MAE-DFER, RoBERTa emotion, Wav2Vec 2.0 fine-tuned on IEMOCAP) rather than from IMA/IMF. MCDubber does not use these features, and the ablation study removes whole components (w/o Global, w/o Local, w/o IMA, w/o IMF) but never adds the same pretrained features into a simple fusion baseline. To support the claim that interaction-based aggregation and fusion cause the improvement, the authors should add a control that feeds the same global/local features into MCDubber via a simple concatenation or a single cross-attention layer, and show that IMA/IMF still provide a meaningful gain over that control.
  2. [Section IV-A, Table I] The claim that M2CI-Dubber significantly outperforms baselines with p-value < 0.001 is not backed by a described statistical test. The objective metrics are reported as point estimates without variance or number of runs, and the subjective MOS confidence intervals are computed from raters but no test procedure, rater sample size per system, or multiple-comparison correction is described. The authors should specify the test used, the unit of analysis, and report variance or confidence intervals for objective metrics, or soften the significance claim accordingly.
  3. [Section III-C and Section IV-A] The subjective evaluation uses only 20 raters and 12 generated samples. Given that the central claim is about prosody expressiveness and context alignment, this is a small evaluation. Please report how the 12 samples were selected, whether each rater rated all systems, the inter-rater agreement, and ideally increase the number of samples or raters. This limitation does not by itself invalidate the results, but it should be acknowledged and the claimed generality should be tempered.
minor comments (4)
  1. [Table I header] The sentence 'THE M2CI-DUBBER SIGNIFICANTLY OUTPERFORM THE BASELINES' contains a subject-verb agreement error; it should be 'OUTPERFORMS'. Also, the table caption says 'WITH 95% CONFIDENCE INTERVAL' but only subjective metrics have intervals.
  2. [Section IV-B] There is a duplicated word in 'removed both both in IMA and IMF'; one 'both' should be deleted.
  3. [Section III-A] The dataset is referred to as 'Chem dataset' but the description points to prior work [20] which is a Hugging Face emotion recognition model, not the dataset. Please provide a proper citation for the Context Chem dataset or clarify its provenance.
  4. [General] Some acronyms and terms are written inconsistently, e.g., 'A VD' with a space in the abstract and introduction. Please ensure consistent formatting throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are evaluated against external re-run baselines with fixed pretrained feature extractors; self-citations are background components, not load-bearing evidence.

full rationale

The paper's derivation chain is an empirical architecture comparison. The proposed M2CI encoders use fixed, externally pretrained feature extractors (MAE-DFER, RoBERTa emotion recognition, Wav2Vec 2.0 fine-tuned on IEMOCAP, and the emotion-aware SSL model [24]) and then train attention and graph-fusion modules on the Context Chem training split to predict ground-truth prosody. No equation defines the reported GPE, FFE, MOS-C, or MOS-S in terms of the same quantities used to fit a parameter, and no fitted value is renamed as a prediction. The ablations in Table II remove components and report changes, but they do not exhibit a reduction of the headline result to the model's own inputs by construction. The self-citations (HPMDubbing [10] as backbone, MCDubber [11] as baseline, and [24] as a local audio feature extractor) are used as background components or comparison systems, not as the justification for the central claim that M2CI-Dubber improves prosody expressiveness. A missing control that feeds the pretrained features through a simpler fusion would be an experimental confound or ablation gap, but it is not circularity under the defined patterns. The p-value claim is a statistical assertion without circular consequences. Therefore no significant circularity is found.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claim is empirical; it does not rest on a derivation. The listed free parameters are hand-chosen hyperparameters and feature dimensions, not fitted constants in a formula. The axioms are the transferability of pretrained emotion and prosody extractors and the design hypothesis that global-local context interaction helps prosody.

free parameters (4)
  • Representation dimension = 256
    All global, local, current, and fused features are set to 256 dimensions; no sensitivity analysis is reported.
  • GAE attention heads = 2
    Graph Attention Encoder uses two heads; no ablation on head count is given.
  • Temporal downsample factors = video 4x, audio 16x
    Chosen to approximate quasi-phoneme-level resolution; not justified with experiments.
  • Training schedule = 40k steps, lr 0.00625, batch 8
    Reported as implementation detail; no convergence or sensitivity analysis is provided.
assumptions (5)
  • domain assumption Pretrained emotion models transfer to dubbing prosody
    MAE-DFER, RoBERTa emotion, and Wav2Vec 2.0-IEMOCAP features are used as global sentence-level prosody signals without adaptation; their transferability is assumed (Section II-A).
  • ad hoc to paper Global and local context features are complementary and useful
    The core design premise, tested only through the paper's ablations, is that sentence-level and phoneme-level features from each modality improve current-sentence prosody.
  • ad hoc to paper Interaction with current text in aggregation and fusion is beneficial
    The paper's main novel mechanism; supported by ablation results but not derived from theory.
  • domain assumption Context Chem is a valid proxy for expressive dubbing
    Evaluation is limited to one single-speaker dataset collected by prior work; generalization to other speakers and languages is assumed.
  • standard math Standard attention and graph attention operations are reliable building blocks
    Self-attention, cross-attention, and graph attention are used as off-the-shelf modules without re-derivation.
invented entities (1)
  • Interaction-based edge in the graph constructor
    purpose: Connects current text feature to nodes from each modality at the same time step to enable deep interaction during multimodal fusion
    This is a modeling construct introduced by the paper; its value is supported only by the paper's own ablations, not by external evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction." pith.science (2026). https://pith.science/paper/5QRGLHCU

@misc{pith2026241218748,
  author       = {Pith},
  title        = {Pith review of: Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5QRGLHCU}},
  note         = {Machine review of arXiv:2412.18748}
}
read the original abstract

Automatic Video Dubbing (AVD) generates speech aligned with lip motion and facial emotion from scripts. Recent research focuses on modeling multimodal context to enhance prosody expressiveness but overlooks two key issues: 1) Multiscale prosody expression attributes in the context influence the current sentence's prosody. 2) Prosody cues in context interact with the current sentence, impacting the final prosody expressiveness. To tackle these challenges, we propose M2CI-Dubber, a Multiscale Multimodal Context Interaction scheme for AVD. This scheme includes two shared M2CI encoders to model the multiscale multimodal context and facilitate its deep interaction with the current sentence. By extracting global and local features for each modality in the context, utilizing attention-based mechanisms for aggregation and interaction, and employing an interaction-based graph attention network for fusion, the proposed approach enhances the prosody expressiveness of synthesized speech for the current sentence. Experiments on the Chem dataset show our model outperforms baselines in dubbing expressiveness. The code and demos are available at \textcolor[rgb]{0.93,0.0,0.47}{https://github.com/AI-S2-Lab/M2CI-Dubber}.

Figures

Figures reproduced from arXiv: 2412.18748 by the authors.

Figure 1
Figure 1. The proposed M2CI-Dubber contains two shared Multiscale Multimodal Context Interaction Encoders, a Text Encoder, and a Video Dubbing Synthesizer. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 19 canonical work pages

  1. [1]

    Neural dubber: Dubbing for videos according to scripts,

    C. Hu, Q. Tian, T. Li, W. Yuping, Y . Wang, and H. Zhao, “Neural dubber: Dubbing for videos according to scripts,” Advances in neural information processing systems , vol. 34, pp. 16 582–16 595, 2021

    work page 2021

  2. [2]

    Prosody Modeling with 3D Visual Information for Expressive Video Dubbing,

    Z. Yang, S. Liu, X. Li, H. Wu, Z. Wu, Y . Shan, and J. Jia, “Prosody Modeling with 3D Visual Information for Expressive Video Dubbing,” in Proc. INTERSPEECH 2023 , 2023, pp. 4863–4867

    work page 2023

  3. [3]

    Styledubber: Towards multi-scale style learning for movie dubbing,

    G. Cong, Y . Qi, L. Li, A. Beheshti, Z. Zhang, A. v. d. Hengel, M.-H. Yang, C. Yan, and Q. Huang, “Styledubber: Towards multi-scale style learning for movie dubbing,” arXiv preprint arXiv:2402.12636 , 2024

    arXiv 2024

  4. [4]

    V2c: Visual voice cloning,

    Q. Chen, M. Tan, Y . Qi, J. Zhou, Y . Li, and Q. Wu, “V2c: Visual voice cloning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 21 242–21 251

    work page 2022

  5. [5]

    From speaker to dubber: Movie dubbing with prosody and duration consistency learning,

    Z. Zhang, L. Li, G. Cong, Y . Haibing, Y . Gao, C. Yan, A. van den Hengel, and Y . Qi, “From speaker to dubber: Movie dubbing with prosody and duration consistency learning,” in ACM Multimedia 2024

    work page 2024

  6. [6]

    Emodubber: Towards high quality and emotion controllable movie dubbing,

    G. Cong, J. Pan, L. Li, Y . Qi, Y . Peng, A. v. d. Hengel, J. Yang, and Q. Huang, “Emodubber: Towards high quality and emotion controllable movie dubbing,” arXiv preprint arXiv:2412.08988 , 2024

    arXiv 2024

  7. [7]

    High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Units

    J. Lu, B. Sisman, M. Zhang, and H. Li, “High-quality automatic voice over with accurate alignment: Supervision through self-supervised discrete speech units,” arXiv preprint arXiv:2306.17005 , 2023

    work page Pith review arXiv 2023

  8. [8]

    Dubwise: Video-guided speech duration control in multimodal llm- based text-to-speech for dubbing,

    N. Sahipjohn, A. Gudmalwar, N. Shah, P. Wasnik, and R. R. Shah, “Dubwise: Video-guided speech duration control in multimodal llm- based text-to-speech for dubbing,” arXiv preprint arXiv:2406.08802 , 2024

    arXiv 2024

Show all 37 references
  1. [9]

    More than words: In-the-wild visually-driven prosody for text-to-speech,

    M. Hassid, M. T. Ramanovich, B. Shillingford, M. Wang, Y . Jia, and T. Remez, “More than words: In-the-wild visually-driven prosody for text-to-speech,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 587–10 597

    2022

  2. [10]

    Learning to dub movies via hierarchical prosody models,

    G. Cong, L. Li, Y . Qi, Z.-J. Zha, Q. Wu, W. Wang, B. Jiang, M.-H. Yang, and Q. Huang, “Learning to dub movies via hierarchical prosody models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14 687–14 697

    2023

  3. [11]

    Mcdubber: Multimodal context-aware expressive video dubbing,

    Y . Zhao, Z. Jia, R. Liu, D. Hu, F. Bao, and G. Gao, “Mcdubber: Multimodal context-aware expressive video dubbing,” arXiv preprint arXiv:2408.11593, 2024

    2024 arXiv

  4. [12]

    To- wards Multi-Scale Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis,

    S. Lei, Y . Zhou, L. Chen, J. Hu, Z. Wu, S. Kang, and H. Meng, “To- wards Multi-Scale Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis,” in Proc. Interspeech 2022, 2022, pp. 5523–5527

    2022

  5. [13]

    Msstyletts: Multi-scale style modeling with hierarchical context infor- mation for expressive speech synthesis,

    S. Lei, Y . Zhou, L. Chen, Z. Wu, X. Wu, S. Kang, and H. Meng, “Msstyletts: Multi-scale style modeling with hierarchical context infor- mation for expressive speech synthesis,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2023

    2023

  6. [14]

    Unsupervised multi-scale expressive speaking style modeling with hierarchical context information for audiobook speech synthesis,

    X. Chen, S. Lei, Z. Wu, D. Xu, W. Zhao, and H. Meng, “Unsupervised multi-scale expressive speaking style modeling with hierarchical context information for audiobook speech synthesis,” in Proceedings of the 29th International Conference on Computational Linguistics , 2022, pp....

    2022

  7. [15]

    Mae-dfer: Efficient masked au- toencoder for self-supervised dynamic facial expression recognition,

    L. Sun, Z. Lian, B. Liu, and J. Tao, “Mae-dfer: Efficient masked au- toencoder for self-supervised dynamic facial expression recognition,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 6110–6121

    2023

  8. [16]

    Dfew: A large-scale database for recognizing dynamic facial expressions in the wild,

    X. Jiang, Y . Zong, W. Zheng, C. Tang, W. Xia, C. Lu, and J. Liu, “Dfew: A large-scale database for recognizing dynamic facial expressions in the wild,” in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 2881–2889

    2020

  9. [17]

    Estimation of continuous valence and arousal levels from faces in naturalistic conditions,

    A. Toisoul, J. Kossaifi, A. Bulat, G. Tzimiropoulos, and M. Pantic, “Estimation of continuous valence and arousal levels from faces in naturalistic conditions,” Nature Machine Intelligence , vol. 3, no. 1, pp. 42–50, 2021

    2021

  10. [18]

    Towards multi- scale style control for expressive speech synthesis,

    X. Li, C. Song, J. Li, Z. Wu, J. Jia, and H. Meng, “Towards multi- scale style control for expressive speech synthesis,” arXiv preprint arXiv:2104.03521, 2021

    2021 arXiv

  11. [19]

    Fastspeech: Fast, robust and controllable text to speech,

    Y . Ren, Y . Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech: Fast, robust and controllable text to speech,” Advances in neural information processing systems , vol. 32, 2019

    2019

  12. [20]

    Emotion-english-roberta-large,

    J. Hartmann, “Emotion-english-roberta-large,” https://huggingface.co/ j-hartmann/emotion-english-roberta-large, 2021

    2021

  13. [21]

    Fastspeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” arXiv preprint arXiv:2006.04558, 2020

    2006 arXiv

  14. [22]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems , vol. 33, pp. 12 449– 12 460, 2020

    2020

  15. [23]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation , vol. 42, pp. 335–359, 2008

    2008

  16. [24]

    Emotion-aware speech self-supervised representation learning with intensity knowledge,

    R. Liu and Z. Ma, “Emotion-aware speech self-supervised representation learning with intensity knowledge,” arXiv preprint arXiv:2406.06646 , 2024

    2024 arXiv

  17. [25]

    Graph attention networks,

    P. Veli ˇckovi´c, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y . Ben- gio, “Graph attention networks,” arXiv preprint arXiv:1710.10903, 2017

    2017 arXiv

  18. [26]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” Advances in neural information processing systems , vol. 33, pp. 17 022–17 033, 2020

    2020

  19. [27]

    A method for fundamental frequency estimation and voicing decision: Application to infant utterances recorded in real acoustical environments,

    T. Nakatani, S. Amano, T. Irino, K. Ishizuka, and T. Kondo, “A method for fundamental frequency estimation and voicing decision: Application to infant utterances recorded in real acoustical environments,” Speech Communication, vol. 50, no. 3, pp. 203–214, 2008

    2008

  20. [28]

    Reducing f0 frame error of f0 tracking algorithms under noisy conditions with an unvoiced/voiced classification frontend,

    W. Chu and A. Alwan, “Reducing f0 frame error of f0 tracking algorithms under noisy conditions with an unvoiced/voiced classification frontend,” in 2009 IEEE International Conference on Acoustics, Speech and Signal Processing . IEEE, 2009, pp. 3969–3972

    2009

  21. [29]

    Out of time: automated lip sync in the wild,

    J. S. Chung and A. Zisserman, “Out of time: automated lip sync in the wild,” in Computer Vision–ACCV 2016 Workshops: ACCV 2016 In- ternational Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13 . Springer, 2017, pp. 251–263

    2016

  22. [30]

    A lip sync expert is all you need for speech to lip generation in the wild,

    K. Prajwal, R. Mukhopadhyay, V . P. Namboodiri, and C. Jawahar, “A lip sync expert is all you need for speech to lip generation in the wild,” in Proceedings of the 28th ACM international conference on multimedia , 2020, pp. 484–492

    2020

  23. [31]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  24. [32]

    Text-to-speech for low-resource agglutinative language with morphology-aware language model pre-training,

    R. Liu, Y . Hu, H. Zuo, Z. Luo, L. Wang, and G. Gao, “Text-to-speech for low-resource agglutinative language with morphology-aware language model pre-training,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  25. [33]

    Multi-source spatial knowledge understanding for immersive visual text-to-speech,

    S. He, R. Liu, and H. Li, “Multi-source spatial knowledge understanding for immersive visual text-to-speech,” arXiv preprint arXiv:2410.14101 , 2024

    2024 arXiv

  26. [34]

    Multi-modal and multi-scale spatial environment understanding for immersive visual text-to-speech,

    R. Liu, S. He, Y . Hu, and H. Li, “Multi-modal and multi-scale spatial environment understanding for immersive visual text-to-speech,” arXiv preprint arXiv:2412.11409, 2024

    2024 arXiv

  27. [35]

    Emphasis rendering for conversational text-to-speech with multi-modal multi-scale context modeling,

    R. Liu, Z. Jia, J. Yang, Y . Hu, and H. Li, “Emphasis rendering for conversational text-to-speech with multi-modal multi-scale context modeling,” arXiv preprint arXiv:2410.09524 , 2024

    2024 arXiv

  28. [36]

    Generative expressive conversational speech synthesis,

    R. Liu, Y . Hu, Y . Ren, X. Yin, and H. Li, “Generative expressive conversational speech synthesis,” in Proceedings of the 32nd ACM International Conference on Multimedia , 2024, pp. 4187–4196

    2024

  29. [37]

    Fctalker: Fine and coarse grained context modeling for expressive conversational speech synthesis,

    Y . Hu, R. Liu, G. Gao, and H. Li, “Fctalker: Fine and coarse grained context modeling for expressive conversational speech synthesis,” in 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP). IEEE, 2024, pp. 299–303

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.