Pith. sign in

REVIEW 4 major objections 5 minor 33 references

MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Movie dubbing that reads scene type and speaker attributes from video outperforms prior state-of-the-art dubbing models on sync, timbre, and emotion.

desk verdict Useful VLM-conditioned dubbing integration with a real annotated dataset, but the SOTA claim rests on an unmatched F5-TTS baseline. read the letter →

arxiv 2505.16279 v1 pith:HA5MWQ6P submitted 2025-05-22 cs.MM cs.CV

Junjie Zheng , Zihao Chen , Chaofan Ding , Yunming Liang , Yihan Fan , Huan Yang , Lei Xie , Xinhan Di This is my paper · ORCID
classification cs.MMcs.CV
keywords moviedubbingmulti-modallearningvision-languagemodelspeechgenerationdiffusiontransformerflowmatchingaudio-visualsynchronizationdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that movie dubbing improves when the dubbing system explicitly understands the scene and the speaker rather than relying only on a reference voice and low-level video features. The proposed framework, MM-MovieDubber, first uses a multi-modal vision-language model to turn a silent video clip into a structured conclusion: scene type (dialogue, narration, or monologue) and fine-grained speaker attributes (gender, age, emotion). A diffusion-transformer speech generator, built on the F5-TTS architecture, is then conditioned on this conclusion, the video frames, and the script, with a duration loss for synchronization and classifier-free guidance over all three conditions. The authors also contribute a 7.2-hour movie dubbing dataset annotated with these dubbing types and attributes. Across the V2C-Animation and GRID benchmarks, plus zero-shot and no-reference-reference settings, they report improvements over prior state-of-the-art dubbing models in lip-sync distance, speaker similarity, emotion similarity, and mel-cepstral distortion, up to 1.09%, 8.80%, 19.08%, and 18.74% respectively.

What carries the argument

The key mechanism is a two-stage generative pipeline. Stage 1 is a multi-modal large language model (mLLM) trained with instruction tuning to convert a silent video clip into a structured 'understanding conclusion' $C_v = \{C_s, C_g, C_a, C_e\}$ covering scene type, speaker gender, speaker age, and emotion. Stage 2 is a diffusion-transformer speech generator (the F5-TTS architecture with conditional flow matching) whose denoising is conditioned on CLIP-encoded video frames, the conclusion, and the script, using cross-attention to fuse the conclusion and visual features. A duration loss constrains speech-video length consistency, and inference uses classifier-free guidance over video, conclusion, and script conditions.

What would settle it

Fine-tune F5-TTS on the same V2C-Animation and GRID training splits with the same reference-audio settings and duration loss, and compare it with MM-MovieDubber when the VLM conclusion is ablated; if the fine-tuned F5-TTS matches or exceeds the reported SPK-SIM, EMO-SIM, and MCD improvements, the central claim that the video-understanding conditions drive the gains is contradicted.

Watch

Extended reading notes

Core claim

The central claim is that conditioning speech generation on a vision-language-model-derived 'understanding conclusion' — composed of scene type, speaker gender, speaker age, and voice emotion — yields dubbing that is better synchronized, more similar to the target voice, and more emotionally expressive than state-of-the-art dubbing models, including HPMDubbing, StyleDubber, and Speaker2Dubber. The paper further claims this works without a reference voice: given a silent video and script alone, the model produces speech that is intelligible and emotionally appropriate. Supporting evidence includes objective metrics on V2C-Animation and GRID and subjective MOS, with gains such as SPK-SIM rising from 79.81% to 83.30% and EMO-SIM from 59.71% to 64.93% in the Dub 2.0 setting on V2C-Animation, and MCD falling from 9.11 to 8.80.

Load-bearing premise

The paper's comparison assumes the F5-TTS baseline is the off-the-shelf Emilia-trained model, while the proposed model is additionally trained on the dubbing datasets; if most of the reported gains come from that fine-tuning alone and not from the VLM-derived dubbing-type and attribute conditions, the central claim collapses.

Editorial extensions

If this is right

  • Dubbing without a reference voice: silent video plus script is enough to produce intelligible, emotionally appropriate speech, as shown in the proposed fine-grained test.
  • Explicit scene-type and speaker-attribute labels act as effective conditioning, reflected in improved EMO-SIM and SPK-SIM and reduced MCD on both V2C-Animation and GRID.
  • The new annotated dataset provides a benchmark for dubbing that distinguishes dialogue, narration, and monologue and fine-grained attributes, which existing benchmarks do not cover.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled comparison that fine-tunes the same F5-TTS backbone on the identical dubbing data, with and without the VLM conclusion, would isolate how much of the reported gain comes from the understanding conditions versus from fine-tuning on in-domain audio-video pairs.
  • Because the 'understanding conclusion' is an interpretable intermediate representation, its accuracy against human labels for scene type, gender, age, and emotion could be measured directly; if speech improvements do not track correct labels, the conditioning mechanism would need a different explanation.
  • The same scene-type and attribute conditions could serve as a shared semantic control layer for dubbing into other languages or for generating character-specific voices, since the conditions are language-agnostic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MM-MovieDubber, a two-stage multi-modal framework for movie dubbing. In Stage 1, a vision-language model is trained (via instruction tuning) to analyze silent video and produce an 'understanding conclusion' that includes dubbing type (dialogue/narration/monologue) and fine-grained speaker attributes (gender, age, emotion). In Stage 2, a diffusion-transformer speech generator (architecture inherited from F5-TTS) is pre-trained on Emilia-EN and then tuned with ControlNet-style multi-modal conditioning using video features, the VLM conclusion, and the script. The authors also construct a 7.2-hour annotated movie dubbing dataset. Experiments on V2C-Animation, GRID, a zero-shot setting, and the proposed dataset report improvements over several dubbing baselines on metrics including LSE-D, SPK-SIM, EMO-SIM, and MCD, with the largest gains on the proposed dataset.

Significance. If the claimed improvements are robust and correctly attributed, the framework would be a useful contribution to movie dubbing, particularly the ability to dub without a reference speech and to condition on scene type and speaker attributes. The paper's strengths include evaluation on external benchmarks, a newly constructed annotated dataset, and a zero-shot experimental setting that is practically relevant. However, the central claim that the multi-modal VLM/ControlNet conditions are the source of the gains is not adequately supported: the F5-TTS baseline appears unmatched in training data, the ablation differences are small, and the 'superior performance' claim is contradicted by the GRID results. The experimental protocol is also ambiguous regarding which datasets are used for the fine-tuning stage.

major comments (4)
  1. [Section 3.2 and Tables 1-2] The training and evaluation protocol is ambiguous, and the F5-TTS baseline is not matched in training data. Section 3.2 states that the speech-generation tuning stage (Stage 2.2) is implemented with only the proposed movie-dubbing dataset, yet Tables 1 and 2 report results on V2C-Animation and GRID. It is not stated whether the proposed model was fine-tuned on those benchmarks' training splits or evaluated in a zero-shot/cross-dataset manner, nor whether the F5-TTS row in Table 1 is the off-the-shelf Emilia-pretrained model or a model fine-tuned on dubbing data. If the proposed model received additional training on movie dubbing data (even from a different dataset) while F5-TTS did not, the gains over F5-TTS (e.g., WER 22.52 vs 24.41 in Table 1) could be due to domain adaptation rather than to the VLM-derived conditions and ControlNet. The paper must clarify the protocol and provide a matched comparison, such as fine-tuning F5-TTS on the same dubbing data and budget, or evaluating an ablated version of the proposed model without Stage 2.2, to support the attribution.
  2. [Abstract and Section 4.1.1] The claim of 'superior performance compared to state-of-the-art (SOTA) methods' is overstated and not supported by the GRID results. In Table 2, the proposed method is worse than F5-TTS on WER (14.71 vs 11.94 in Dub 1.0) and SPK-SIM (95.73 vs 96.51), and in Table 3 it is worse than F5-TTS on both NMOS (4.33 vs 4.43) and SMOS (3.14 vs 3.32). The abstract should be qualified to specify the metrics and datasets for which improvements hold, and the main text should acknowledge that the method does not dominate F5-TTS on GRID. The current wording implies global superiority, which is inaccurate.
  3. [Table 6] The ablation study does not establish that the multi-modal conditions are the driver of the reported improvements. Removing the video conclusion changes SPK-SIM by only 0.40 (82.99 vs 82.59) and EMO-SIM by 1.56 (64.74 vs 63.18), and removing the video clip changes SPK-SIM by 0.36. No error bars or significance tests are provided, yet the text claims that removing video clip control 'causes all metrics to drop significantly' and that the conclusion control 'improves SPK-SIM and EMO-SIM.' Given the small effect sizes relative to the gaps over baselines in Tables 1 and 2 (e.g., SPK-SIM gains of 0.44 over F5-TTS on V2C-Animation), the attribution of the main benchmark improvements to the VLM conclusion and visual conditioning is not supported.
  4. [Section 2.3.2 and Table 6] The paper's novelty rests on fine-grained attribute understanding, but no experiment isolates the contributions of scene type (Cs), gender (Cg), age (Ca), and emotion (Ce). In Eq. (6), these are concatenated into a single condition Cv, and the ablation in Table 6 removes the entire 'conclusion' rather than individual attributes. Consequently, the specific value of each fine-grained attribute for dubbing quality is not demonstrated. The authors should provide an ablation that removes each attribute separately or, at minimum, report the model's accuracy in predicting these attributes and its effect on the final output.
minor comments (5)
  1. [Figure 1 and Section 4.1.1] There are several typos: 'Curent' in Figure 1, 'decreaded' in the V2C-Animation results paragraph, and 'SIP-SIM' for 'SPK-SIM' in the zero-shot results paragraph. The heading of Table 5 reads 'towards fined-grained movie dubbing test' and should be corrected to 'toward fine-grained movie dubbing test.'
  2. [Equation (9)] Equation (9) mixes the symbols g_theta and v0 for the model output; the classifier-free guidance formula should use consistent notation, e.g., v_theta throughout, to avoid confusion about the velocity estimate being combined.
  3. [References] References [4] and [11] are the same paper (Cong et al., 'Learning to dub movies via hierarchical prosody models'), and references [1] and [22] are also the same work (Chen et al., V2C). These duplicates should be consolidated.
  4. [Section 4.1.1, GRID paragraph] The sentence 'slightly lower than the best fine-tuned F5-TTS model, Speaker2Dub and StyleDubber' is unclear; it appears to mean the proposed method's WER is slightly higher than the best values among F5-TTS, Speaker2Dub, and StyleDubber. This should be rephrased for clarity.
  5. [Section 2.3.2] The description of how the mLLM is trained in Stage 1 (Section 2.2) and how its conclusions are generated is underspecified: no details are given about the base mLLM, the instruction template, the amount of training data, or the annotation process for the proposed dataset. Adding these details would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical system evaluation against external benchmarks, with no equation-level equivalence or self-citation chain.

full rationale

MM-MovieDubber is an empirical system paper rather than a derivation. The proposed model trains an mLLM to output dubbing-type and attribute conclusions, fine-tunes a flow-matching speech generator on dubbing datasets, and is evaluated on the external V2C-Animation and GRID benchmarks as well as on its own proposed dataset. The reported metrics (LSE-D, LSE-C, SPK-SIM, EMO-SIM, MCD, WER, MOS) are measured against ground-truth audio or pretrained external estimators, so the improvements are not equivalent by construction to the conditioning inputs. No equation in the paper reduces a predicted quantity to a fitted input: the conditions Cv, Vl, and Tv enter the loss Lg, but the evaluation quantities are independent external measurements. The proposed dataset is used both for training and for the fine-grained test, which is an in-distribution evaluation concern, but it is only one of three test beds and does not make the central claim definitionally circular. The paper contains no load-bearing self-citations: the cited baselines and components are external works, and no claimed uniqueness or prior-theorem result by the present authors is invoked. The skeptic's point about the F5-TTS baseline possibly being off-the-shelf while the proposed model is fine-tuned is a legitimate experimental-fairness concern, but it is a correctness/comparability issue, not a circularity issue, because the comparison is not forced by the paper's definitions or equations. Accordingly, no circular step is identified and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central system is an empirical pipeline built from existing components; no new physical or mathematical entities are postulated. The main uncharged inputs are the reliability of the VLM's attribute predictions, the adequacy of the small annotated dataset, and the hand-chosen guidance scales. The fitted neural-network weights are standard and not listed individually.

free parameters (2)
  • classifier-free guidance scales lambda_V, lambda_C, lambda_T = not reported
    Equation 9 defines three guidance weights for video, conclusion, and script; their values are chosen by hand but never reported, so the sampling behavior cannot be reproduced.
  • condition dropout probability = 5%
    Section 2.3.2 sets each condition to empty with 5% probability; this is a hyperparameter affecting classifier-free guidance, presumably tuned but no sensitivity analysis is given.
assumptions (4)
  • domain assumption The vision-language model can predict scene type, gender, age, and emotion from silent video frames plus subtitles.
    Stage 1 trains an mLLM on the proposed dataset; if these predictions are unreliable, the conditioning in Stage 2 is based on noise. The paper provides no accuracy numbers for the VLM itself.
  • domain assumption CLIP visual features and the F5-TTS DiT backbone are sufficient base representations for dubbing.
    Section 2.3.2 reuses CLIP [19] and F5-TTS [16]; the paper does not test alternative visual encoders or speech backbones.
  • domain assumption The proposed 7.2-hour annotated dataset is large enough and its annotations accurate enough to train both the mLLM and speech generator.
    The dataset is constructed by the authors (Section 3.1), and no annotation statistics or validation of label quality are reported.
  • standard math The flow-matching objective L_CFM is a valid training objective for the speech generator.
    Equation 5 follows the standard flow-matching formulation of Lipman et al. [17].

how reviews work

0 comments
Cite this review

Pith. "Pith review of MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing." pith.science (2026). https://pith.science/paper/HA5MWQ6P

@misc{pith2026250516279,
  author       = {Pith},
  title        = {Pith review of: MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HA5MWQ6P}},
  note         = {Machine review of arXiv:2505.16279}
}
read the original abstract

Current movie dubbing technology can produce the desired speech using a reference voice and input video, maintaining perfect synchronization with the visuals while effectively conveying the intended emotions. However, crucial aspects of movie dubbing, including adaptation to various dubbing styles, effective handling of dialogue, narration, and monologues, as well as consideration of subtle details such as speaker age and gender, remain insufficiently explored. To tackle these challenges, we introduce a multi-modal generative framework. First, it utilizes a multi-modal large vision-language model (VLM) to analyze visual inputs, enabling the recognition of dubbing types and fine-grained attributes. Second, it produces high-quality dubbing using large speech generation models, guided by multi-modal inputs. Additionally, a movie dubbing dataset with annotations for dubbing types and subtle details is constructed to enhance movie understanding and improve dubbing quality for the proposed multi-modal framework. Experimental results across multiple benchmark datasets show superior performance compared to state-of-the-art (SOTA) methods. In details, the LSE-D, SPK-SIM, EMO-SIM, and MCD exhibit improvements of up to 1.09%, 8.80%, 19.08%, and 18.74%, respectively.

Figures

Figures reproduced from arXiv: 2505.16279 by the authors.

Figure 1
Figure 1. Curent Dubbing models [11, 12, 13] (Left). Proposed Dubbing Models (Right) For dubbing types and fine-grained at￾tributes. large vision-language model (VLM) is trained to improve the understanding of dubbing types and fine-grained attributes from video inputs. Second, a large speech generation model is trained with designed control mechanisms guided by multi-modal con￾ditions. Finally, we construct a movie dubbing d… view at source ↗
Figure 2
Figure 2. MM-MovieDubber pipeline with multi-stage, multi-modal training. which can be formalized as: Sˆ = F 2 generator(Vl, Cv, Tv) (3) where Vl represents the input video clip, Cv represents the un￾derstanding conclusion, and Tv represents the expected script. The speech Sˆ is generated under conditioning. 2.2. Stage 1: Multi-modal Large Language Model (mLLM) Learning Building upon the success of multi-modal large language … view at source ↗
Figure 3
Figure 3. Proposed dataset with multi-type annotations, including annotation for lips, faces, scene-type, speaker gender, speaker age, voice emotion. 3. Experiments 3.1. Datasets Emilia is a comprehensive multilingual speech generation dataset containing a total of 101,654 hours of speech data across six languages [21]. The English portion of this dataset, compris￾ing approximately 46,800 hours, is utilized to train our found… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 23 canonical work pages

  1. [1]

    Exist- ing dubbing methods can be categorized into two groups, each focusing on learning different styles of key prior information to generate high-quality voices

    Introduction Dubbing involves adding the correct human voice to a video’s dialogue, ensuring synchronization with the characters’ lip movements, and conveying the emotions of the scene. Exist- ing dubbing methods can be categorized into two groups, each focusing on learning different styles of key prior information to generate high-quality voices. The fir...

    work page

  2. [2]

    Method 2.1. Overview Given a silent video clip Vl, a corresponding subtitle Tv, and the goal of generating a fully dubbed video, the proposed model (MM-MovieDubber) aims to produce speech ˆS that matches the video, ensures contextual and prosodic relevance, and maintains speech-video synchronization. The model can be formalized as follows: ˆS = F n dubber...

    work page Pith review arXiv 2025

  3. [3]

    Datasets Emilia is a comprehensive multilingual speech generation dataset containing a total of 101,654 hours of speech data across six languages [21]

    Experiments 3.1. Datasets Emilia is a comprehensive multilingual speech generation dataset containing a total of 101,654 hours of speech data across six languages [21]. The English portion of this dataset, compris- ing approximately 46,800 hours, is utilized to train our founda- tional text-to-speech (TTS) model. V2C-Animation is a specialized dataset des...

    work page

  4. [4]

    To as- sess pronunciation accuracy, we use Word Error Rate (WER) with Whisper-V3[24] as the ASR model

    Evaluation We evaluate using both objective and subjective metrics. To as- sess pronunciation accuracy, we use Word Error Rate (WER) with Whisper-V3[24] as the ASR model. Timbre consistency is evaluated with speaker encoder cosine similarity (SPK-SIM) [12]. We also calculate mel cepstral distortion dynamic time warping (MCD) and speech length variance (MC...

    work page

  5. [5]

    Additionally, we have de- veloped a movie dubbing dataset with multi-type annotations to enhance movie understanding and improve dubbing quality

    Conclusion In this paper, we propose a multi-stage, multi-modal genera- tive framework for movie dubbing. Additionally, we have de- veloped a movie dubbing dataset with multi-type annotations to enhance movie understanding and improve dubbing quality. In the future, we plan to incorporate reasoning ability into our movie dubbing model to improve its under...

    work page

  6. [6]

    V2c: Vi- sual voice cloning,

    Q. Chen, M. Tan, Y . Qi, J. Zhou, Y . Li, and Q. Wu, “V2c: Vi- sual voice cloning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 21 242– 21 251

    work page 2022

  7. [7]

    More than words: In-the-wild visually-driven prosody for text-to-speech,

    M. Hassid, M. T. Ramanovich, B. Shillingford, M. Wang, Y . Jia, and T. Remez, “More than words: In-the-wild visually-driven prosody for text-to-speech,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 587–10 597

    work page 2022

  8. [8]

    Generalized end-to-end loss for speaker verification,

    L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end loss for speaker verification,” in 2018 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4879–4883

    2018

Show all 33 references
  1. [9]

    Learning to dub movies via hierarchical prosody models,

    G. Cong, L. Li, Y . Qi, Z.-J. Zha, Q. Wu, W. Wang, B. Jiang, M.- H. Yang, and Q. Huang, “Learning to dub movies via hierarchical prosody models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14 687– 14 697

    2023

  2. [10]

    Neu- ral dubber: Dubbing for videos according to scripts,

    C. Hu, Q. Tian, T. Li, W. Yuping, Y . Wang, and H. Zhao, “Neu- ral dubber: Dubbing for videos according to scripts,” Advances in neural information processing systems , vol. 34, pp. 16 582– 16 595, 2021

    2021

  3. [11]

    Imaginary voice: Face- styled diffusion model for text-to-speech,

    J. Lee, J. S. Chung, and S.-W. Chung, “Imaginary voice: Face- styled diffusion model for text-to-speech,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  4. [12]

    Mcdubber: Multimodal context-aware expressive video dubbing,

    Y . Zhao, Z. Jia, R. Liu, D. Hu, F. Bao, and G. Gao, “Mcdubber: Multimodal context-aware expressive video dubbing,” inNational Conference on Man-Machine Speech Communication. Springer, 2024, pp. 168–182

    2024

  5. [13]

    Audiopedia: Audio qa with knowledge,

    A. S. Penamakuri, K. Chhatre, and A. Jain, “Audiopedia: Audio qa with knowledge,” 2024

    2024

  6. [14]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models,

    Y . Dong, Z. Liu, H.-L. Sun, J. Yang, W. Hu, Y . Rao, and Z. Liu, “Insight-v: Exploring long-chain visual reasoning with multimodal large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2411.14432

    2024 arXiv

  7. [15]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhao, and L. Bing, “Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,” 2024. [Online]. Available: https://arxiv.org/abs/2406.07476

    2024 arXiv

  8. [16]

    Learning to dub movies via hierarchical prosody models,

    G. Cong, L. Li, Y . Qi, Z. Zha, Q. Wu, W. Wang, B. Jiang, M.-H. Yang, and Q. Huang, “Learning to dub movies via hierarchical prosody models,” 2023

    2023

  9. [17]

    Styledubber: Towards multi-scale style learning for movie dubbing,

    G. Cong, Y . Qi, L. Li, A. Beheshti, Z. Zhang, A. van den Hengel, M.-H. Yang, C. Yan, and Q. Huang, “Styledubber: Towards multi-scale style learning for movie dubbing,” 2024. [Online]. Available: https://arxiv.org/abs/2402.12636

    2024 arXiv

  10. [18]

    From speaker to dubber: Movie dubbing with prosody and duration consistency learning,

    Z. Zhang, L. Li, G. Cong, H. YIN, Y . Gao, C. Yan, A. van den Hengel, and Y . Qi, “From speaker to dubber: Movie dubbing with prosody and duration consistency learning,” in ACM Multimedia 2024 , 2024. [Online]. Available: https: //openreview.net/forum?id=QHRNR64J1m

    2024

  11. [19]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/ forum?id=w0H2xGHlkw

    2023

  12. [20]

    Llava-cot: Let vision language models reason step-by-step,

    G. Xu, P. Jin, H. Li, Y . Song, L. Sun, and L. Yuan, “Llava-cot: Let vision language models reason step-by-step,” 2025. [Online]. Available: https://arxiv.org/abs/2411.10440

    2025 arXiv

  13. [21]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” 2024. [Online]. Available: https://arxiv.org/abs/2410.06885

    2024 arXiv

  14. [22]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” in The Eleventh International Conference on Learning Representations ,

  15. [23]

    An audio-visual corpus for speech perception and automatic speech recognition,

    M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio-visual corpus for speech perception and automatic speech recognition,” The Journal of the Acoustical Society of America , vol. 120, no. 5, pp. 2421–2424, 11 2006. [Online]. Available: https://doi.org/10.1121/1.2229005

    2006 doi

  16. [24]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res., vol. 21, pp. 140:1–140:67, 2020. [Online]. Available: https://jmlr.org/pa...

    2020

  17. [25]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine L...

    2021

  18. [26]

    DiVE: Dit-based video generation with enhanced control,

    J. Jiang, G. Hong, L. Zhou, E. Ma, H. Hu, xia zhou, J. Xiang, F. Liu, K. Yu, H. Sun, K. Zhan, P. Jia, and M. Zhang, “DiVE: Dit-based video generation with enhanced control,” in ECCV 2024 Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving,...

    2024

  19. [27]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,

    H. He, Z. Shang, C. Wang, X. Li, Y . Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, Y . Wang, K. Chen, P. Zhang, and Z. Wu, “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” 2024. [Online]. Available: https://arxiv.org/abs/2407.05361

    2024 arXiv

  20. [28]

    V2c: Visual voice cloning,

    Q. Chen, Y . Li, Y . Qi, J. Zhou, M. Tan, and Q. Wu, “V2c: Visual voice cloning,” 2021. [Online]. Available: https://arxiv.org/abs/2111.12890

    2021 arXiv

  21. [30]

    Robust speech recognition via large- scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large- scale weak supervision,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.04356

    2022 arXiv

  22. [31]

    Location-relative attention mechanisms for robust long-form speech synthesis,

    E. Battenberg, R. Skerry-Ryan, S. Mariooryad, D. Stanton, D. Kao, M. Shannon, and T. Bagby, “Location-relative attention mechanisms for robust long-form speech synthesis,” 2020. [Online]. Available: https://arxiv.org/abs/1910.10288

    2020 arXiv

  23. [32]

    Tem- poral modeling matters: A novel temporal emotional modeling approach for speech emotion recognition,

    J. Ye, X.-C. Wen, Y . Wei, Y . Xu, K. Liu, and H. Shan, “Tem- poral modeling matters: A novel temporal emotional modeling approach for speech emotion recognition,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023...

    2023

  24. [33]

    Out of time: automated lip sync in the wild,

    J. S. Chung and A. Zisserman, “Out of time: automated lip sync in the wild,” in Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13 . Springer, 2017, pp. 251–263

    2016

  25. [2023]

    Available: https://openreview.net/forum?id= PqvMRDCJT9t

    [Online]. Available: https://openreview.net/forum?id= PqvMRDCJT9t

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.