Pith. sign in

REVIEW 4 major objections 5 minor 89 references

OmniCustom claims a single zero-shot model can generate a talking video that keeps a reference face while cloning a reference voice, with the spoken words supplied as text.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:05 UTC pith:RMVW25X5

load-bearing objection A new task and a plausible method, but the SOTA claim rests on a 100-example benchmark with no error bars. the 4 major comments →

arxiv 2602.12304 v5 pith:RMVW25X5 submitted 2026-02-12 cs.SD cs.AIcs.MMeess.AS

OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model

Maomao Li , Zhen Li , Kaipeng Zhang , Guosheng Yin , Zhifeng Li , Dong Xu This is my paper
classification cs.SD cs.AIcs.MMeess.AS
keywords sync audio-video customizationidentity preservationtimbre cloningzero-shot generationLoRAflow matchingcontrastive learningjoint audio-video generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces a new task, sync audio-video customization: from one reference image and one reference audio clip, generate a video whose face matches the image, whose voice matches the timbre of the audio, and whose spoken content is whatever the user types. It then presents OmniCustom, a diffusion-transformer audio-video model fine-tuned to do this zero-shot. The core mechanism is to feed the reference image and audio as static tokens through separate low-rank adapters in the model's self-attention layers, and to add a contrastive objective that pushes reference-conditioned predictions away from no-reference predictions. Trained on a constructed one-million-clip talking-head dataset, the model is reported to beat existing video-customization systems on identity and quality metrics while matching dedicated TTS systems on timbre cloning. If true, this collapses the usual split between video customization and voice cloning into one flexible pipeline.

Core claim

The central claim is that identity and timbre can be synchronized in a single generative pass rather than stitched from separate video-customization and TTS models. The paper argues that a joint audio-video diffusion transformer can absorb both conditions at once: the reference image is injected into the video branch's self-attention and the reference audio into the audio branch's self-attention, each through low-rank adapters, with face and timbre embeddings added as global tokens. A contrastive flow-matching loss then regularizes the velocity field by repelling the reference-conditioned flow from the text-only, no-reference flow. The authors report that this combination yields the best ide

What carries the argument

Two independent LoRA adapters (one for reference-image tokens, one for reference-audio tokens) inserted into the QKV projections of self-attention in the video and audio branches of a joint audio-video diffusion transformer. The reference tokens are given a static time step of 0, so they act as time-invariant conditioning. Around this, the paper adds a contrastive regularization term: the velocity field conditioned on references (positive) is pushed away from the velocity field with references masked out and embeddings zeroed (negative), with a stop gradient on the negative. The face/timbre embeddings enter as one-global-token conditions. These pieces together are what the paper credits for

Load-bearing premise

The load-bearing premise is that a reference clip and a training clip drawn from the same video teach timbre rather than incidental recording conditions, and that the no-reference flow is a meaningful negative rather than a degenerate baseline.

What would settle it

Run a controlled swap: use reference audio from speaker A and a prompt with content spoken by speaker B. If the generated audio's speaker embedding is no closer to A than to a random held-out speaker—while word error rate stays low—then the timbre-cloning claim fails. Similarly, if the contrastive loss's gains vanish when the no-reference baseline is replaced by a well-trained text-only flow, the regularization is not doing what the paper says.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Video customization stops being a silent-film technology: the same model supplies synchronized speech with freely editable wording.
  • Voice cloning no longer needs a dedicated TTS stack; a joint audio-video model can carry timbre alongside identity, and even produce text-relevant background sound.
  • The contrastive flow-matching trick may transfer to other reference-conditioned generation tasks that need strong separation from no-reference baselines.
  • A single benchmark with 100 held-out identities and a 1M-clip dataset is proposed as a foundation for comparing sync audio-video customization systems.
  • The reported results suggest that modeling the joint distribution of video and audio is better for lip-sync than modeling audio-driven conditional video generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the same recipe extends beyond 5-second English clips; testing the LoRA-plus-contrastive approach on longer or multilingual generations is a direct next step.
  • A sharper test of the timbre claim would be cross-gender and cross-accent references: the paper shows one cross-gender example and notes a slight timbre drop, suggesting the model carries implicit gender priors.
  • If the contrastive term is doing real work, it should also improve generalization to unseen identities; a held-out identity study with error bars on the 30 unseen persons would make that visible.
  • The same-video pairing of reference and target could be probed by swapping reference audio from a different speaker while keeping content; if speaker similarity stays high, the model may be copying prosody rather than timbre.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces 'sync audio-video customization,' a new task in which a model receives a reference image and a reference audio clip and must produce a video that preserves the identity of the image while imitating the timbre of the audio, with the spoken content specified by a free-form text prompt. The proposed system, OmniCustom, extends the OVI joint audio-video diffusion transformer by adding reference-image and reference-audio branches with LoRA adapters in self-attention layers, plus global face and timbre embeddings. A 'contrastive' regularizer (Eq. 8) pushes reference-conditioned flows away from no-reference flows. The model is trained on a new 1M-clip dataset derived from SpeakerVid-5M, and evaluated on a self-built 100-example benchmark with video, audio, and TTS baselines. The paper reports state-of-the-art identity preservation, timbre cloning, and video quality, with a user study showing preference over prior methods.

Significance. If the reported results hold, this is a genuinely new capability: simultaneous zero-shot identity and timbre customization with free text control, enabled by a clean architectural extension of an open audio-video generation model. The dataset construction and the use of an existing TTS benchmark for timbre comparison are useful contributions. The paper also provides ablations isolating the contribution of the embeddings and the contrastive loss, and includes a user study. However, the central SOTA claim currently rests on a small, unreleased benchmark with no statistical inference, and the novel objective in Eq. (8) is not characterized enough to establish that it provides the claimed benefit rather than a training artifact.

major comments (4)
  1. [§6.3, Table 2] The paper's headline claim of state-of-the-art performance rests on a self-built benchmark of only 100 examples (300 videos with 3 seeds) and reports no confidence intervals, error bars, or significance tests. The differences in FaceSim-Arc (e.g., 0.60 vs. 0.49 for ConsisID) and FID/FVD may be within noise, especially given that the benchmark construction is not publicly released. I request error bars, statistical significance tests (e.g., paired bootstrap or Wilcoxon), and release of the benchmark to make the SOTA claim verifiable.
  2. [§4.2.3, Eq. (8)] The 'contrastive' identity/timbre objective is a stop-gradient L2 repulsion from the no-reference flow. The paper does not characterize this no-reference flow: it is obtained by the same network with masked attention and zeroed embeddings, and the baseline may be high-variance or poorly calibrated. If so, the repulsion term can inflate distances to a noisy predictor and improve FaceSim/Speaker-Sim on the small benchmark without transferring to new identities. The paper should analyze the no-reference flow (e.g., variance, stability), test the objective without the stop-gradient, and verify that the gains persist on a larger or cross-session set. Also note that the objective is not 'contrastive' in the standard sense—there is no positive pair; the positive examples are only 'implied' in the flow-matching loss, so the terminology may mislead.
  3. [§5.3 and §6.1] Training reference and target clips are sampled from the same source video (first 4s vs. last 5s). This pairing may teach the model to exploit recording/session conditions rather than speaker identity and timbre alone. More importantly, the Speaker-Sim evaluation in Table 2 is ambiguous: it compares the synthesized speech with 'ground-truth speech segments,' but for a freely specified text prompt there is no natural ground truth. If the comparison is against audio from the same source video as the reference, the metric may be inflated by channel/room acoustics. Please clarify the exact protocol and, ideally, construct a benchmark where reference and target are from different sessions/recordings.
  4. [§5 and §6.1] The paper presents OmniCustom-1M as a core contribution, but the dataset is not released, and the 100-example benchmark is not released. This makes the reported results impossible to reproduce or compare against by other groups. Given that the central claim depends on this data, I consider a release (even a subset or detailed metadata) and the benchmark essential for the claim to be substantiated.
minor comments (5)
  1. [§4.2.3, Eq. (8)] The equation has a mismatched parenthesis: `StopGrad(v_θ(Z_ti, φ, C, ti)||^2` is missing a closing `)`. Please fix.
  2. [§4.2.3] 'Positive examples are implied in the flow matching objectives' is vague. Clarify how the flow-matching loss acts as a positive term and why this design is preferable to explicit positives.
  3. [§1, Table 1] Typo: 'the first to propose sync' is missing a space. Also, the sentence 'proposes a more compelling new task' is subjective; consider rewording.
  4. [§2, References] Reference [6] has the author name 'JianZhao JianZhao' duplicated. Please correct.
  5. [§6.4] The statement 'timbre embeddings and contrastive timbre loss boost timbre similarity by 31.0% and 23.7%' should specify whether this is a relative or absolute improvement and what the baseline value is, for reproducibility.

Circularity Check

0 steps flagged

No significant circularity: OmniCustom is an empirical training recipe on an external base model, with no fitted output masquerading as a derived prediction and no load-bearing self-citation chain.

full rationale

The paper's chain is a standard training/evaluation claim, not a mathematical derivation. The base model OVI [46], flow matching [42,45], LoRA [23], face/timbre extractors [9,30], and evaluation metrics are all external and independently specified; the authors' own prior work [84] is a related image-customization method cited in passing and is not load-bearing. Eq. (8) is an auxiliary regularizer that maximizes distance between reference-conditioned and no-reference velocity fields; it does not define the identity/timbre metrics, nor is any Table-2 number computed from Eq. (8) by construction. The ablation gains are empirical and could fail if the no-reference baseline is unstable, but that is a correctness/robustness risk, not circularity. The nearest issue is Sec. 5.3, where reference and training clips are sampled from the same video ('Each training clip and its corresponding reference clip are sampled from the same video'), so timbre fidelity may partly inherit recording-consistency cues; however, the benchmark reserves unseen identities and compares against external TTS and customization systems, and no prediction reduces to its input by definition. I therefore find no significant circularity and score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

No new physical or conceptual entities (particles, forces, conserved quantities) are introduced. The paper contributes a dataset, a task definition, and model components, which are not 'invented entities' in the ledger sense.

free parameters (5)
  • Contrastive loss weights λ_Ir, λ_Ar = 0.1 each
    Chosen by hand in Sec. 6.1; the ablation only compares 0 vs 0.1, so the reported gains may depend on these values.
  • Flow-matching loss weights λ_V, λ_A = 1.0 each
    Chosen by hand in Sec. 6.1; no sensitivity analysis is provided.
  • LoRA rank = 128
    Architecture hyperparameter set in Sec. 6.1; no study of rank effects on identity/timbre fidelity.
  • Guidance scales for audio and video = audio 3.0, video 4.0
    Inference-time hyperparameters set in Sec. 6.1; no guidance sweep is reported.
  • Dataset filtering thresholds = |offset|≤3, SyncNet confidence>1.5, aesthetics≥0.3
    Empirical curation choices in Sec. 5.2; changing these thresholds changes the training distribution and downstream quality.
axioms (5)
  • domain assumption OVI provides a working joint audio-video generator whose paired cross-attention preserves synchronization after LoRA fine-tuning.
    The entire framework is built on OVI [46] and assumes its audio-video alignment survives the fine-tuning of self-attention layers (Sec. 4.2).
  • standard math Flow matching loss (Eq. 2) is a valid generative training objective.
    Taken from the flow-matching literature and used without modification (Sec. 3).
  • domain assumption Static reference tokens with time step 0, concatenated in self-attention (Eq. 6), provide sufficient conditioning for identity and timbre.
    The method assumes that injection through self-attention plus global face/timbre embeddings is enough to control both modalities (Sec. 4.2.1).
  • ad hoc to paper The stop-gradient repulsion loss (Eq. 8) improves identity/timbre fidelity without degrading text following.
    This is a new regularizer introduced by the paper; its behavior is validated only by the three-row ablation in Table 2, not by analysis.
  • domain assumption Filtered SpeakerVid-5M clips, with 4-second reference / 5-second target from the same video, teach timbre rather than content and generalize to new identities.
    The dataset construction in Sec. 5.3 assumes temporal separation prevents speech-content leakage and that the model learns a reusable timbre representation.

pith-pipeline@v1.3.0-alltime-deepseek · 20086 in / 11124 out tokens · 121124 ms · 2026-08-03T00:05:52.844661+00:00 · methodology

0 comments
read the original abstract

Existing mainstream video customization methods focus on generating identity-consistent videos based on given reference images and textual prompts. Benefiting from the rapid advancement of joint audio-video generation, this paper proposes a more compelling new task: sync audio-video customization, which aims to synchronously customize both video identity and audio timbre. Specifically, given a reference image $I^{r}$ and a reference audio $A^{r}$, this novel task requires generating videos that maintain the identity of the reference image while imitating the timbre of the reference audio, with spoken content freely specifiable through user-provided textual prompts. To this end, we propose OmniCustom, a powerful DiT-based audio-video customization framework that can synthesize a video following reference image identity, audio timbre, and text prompts all at once in a zero-shot manner. Our framework is built on three key contributions. First, identity and audio timbre control are achieved through separate reference identity and audio LoRA modules that operate through self-attention layers within the base audio-video generation model. Second, we introduce a contrastive learning objective alongside the standard flow matching objective. It uses predicted flows conditioned on reference inputs as positive examples and those without reference conditions as negative examples, thereby enhancing the model ability to preserve identity and timbre. Third, we train OmniCustom on our constructed large-scale, high-quality audio-visual human dataset. Extensive experiments demonstrate that OmniCustom outperforms existing methods in generating audio-video content with consistent identity and timbre fidelity. Project page: https://omnicustom-project.github.io/page/.

Figures

Figures reproduced from arXiv: 2602.12304 by Dong Xu, Guosheng Yin, Kaipeng Zhang, Maomao Li, Zhen Li, ZhiFeng Li.

Figure 1
Figure 1. Figure 1: We propose OmniCustom, a novel framework to deal with sync audio-video customization. Given a reference image [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) Overview of our OmniCustom architecture. We extend the joint audio-video generation model OVI [ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison with state-of-the-art video customization methods. The speech content of HunyuanCustom [ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study. Face embeddings and the contrastive identity objective boost identity consistency, respectively. See the video [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

89 extracted references · 31 linked inside Pith

  1. [1]

    Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video genera- tion.arXiv preprint arXiv:2304.08477, 2023

    Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video genera- tion.arXiv preprint arXiv:2304.08477, 2023. 1

    Pith/arXiv arXiv 2023

  2. [2]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. InCVPR, pages 22563–22575, 2023. 1

    2023

  3. [3]

    Humo: Human-centric video generation via collaborative multi-modal conditioning.arXiv preprint arXiv:2509.08519, 2025

    Liyang Chen, Tianxiang Ma, Jiawei Liu, Bingchuan Li, Zhuowei Chen, Lijie Liu, Xu He, Gen Li, Qian He, and Zhiyong Wu. Humo: Human-centric video generation via collaborative multi-modal conditioning.arXiv preprint arXiv:2509.08519, 2025. 2, 3, 6, 8, 9, 1, 5

    Pith/arXiv arXiv 2025

  4. [4]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self- supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16 (6):1505–1518, 2022. 7

    2022

  5. [5]

    A simple framework for contrastive learn- ing of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learn- ing of visual representations. InICML, pages 1597–1607. PmLR, 2020. 5

    2020

  6. [6]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chun- hui Wang, JianZhao JianZhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6255–6271, 2025. 3, 8

    2025

  7. [7]

    Mmaudio: Taming multimodal joint training for high-quality video-to- audio synthesis

    Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to- audio synthesis. InCVPR, pages 28901–28911, 2025. 3, 4, 5, 7

    2025

  8. [8]

    Google DeepMind. Veo 3. https://https://deepmind.google/models/veo, 2025. 2, 3

    2025

  9. [9]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. InCVPR, pages 4690–4699, 2019. 4, 5, 7

    2019

  10. [10]

    Vall-t: Decoder-only generative transducer for robust and decoding-controllable text-to-speech

    Chenpeng Du, Yiwei Guo, Hankun Wang, Yifan Yang, Zhikang Niu, Shuai Wang, Hui Zhang, Xie Chen, and Kai Yu. Vall-t: Decoder-only generative transducer for robust and decoding-controllable text-to-speech. InICASSP, pages 1–5. IEEE, 2025. 3

    2025

  11. [11]

    Cosyvoice: A scalable multilingual zero-shot text- to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text- to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024. 3, 8 9

    Pith/arXiv arXiv 2024

  12. [12]

    Skyreels-a2: Compose anything in video dif- fusion transformers.arXiv preprint arXiv:2504.02436, 2025

    Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, et al. Skyreels-a2: Compose anything in video dif- fusion transformers.arXiv preprint arXiv:2504.02436, 2025. 2, 3

    Pith/arXiv arXiv 2025

  13. [13]

    Preserve your own correlation: A noise prior for video diffusion models

    Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. InICCV, pages 22930–22941, 2023. 1

    2023

  14. [14]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. InCVPR, pages 15180–15190, 2023. 3

    2023

  15. [15]

    Generative adversarial networks.Commu- nications of the ACM, 63(11):139–144, 2020

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Commu- nications of the ACM, 63(11):139–144, 2020. 1

    2020

  16. [16]

    Reuse and diffuse: Iterative denoising for text-to-video generation.arXiv preprint arXiv:2309.03549, 2023

    Jiaxi Gu, Shicong Wang, Haoyu Zhao, Tianyi Lu, Xing Zhang, Zuxuan Wu, Songcen Xu, Wei Zhang, Yu-Gang Jiang, and Hang Xu. Reuse and diffuse: Iterative denoising for text-to-video generation.arXiv preprint arXiv:2309.03549, 2023. 1

    Pith/arXiv arXiv 2023

  17. [17]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning.ICLR, 2024

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to- image diffusion models without specific tuning.ICLR, 2024. 1

    2024

  18. [18]

    Pulid: Pure and lightning id customiza- tion via contrastive alignment.NeurIPS, 37:36777–36804,

    Zinan Guo, Yanze Wu, Zhuowei Chen, Lang Chen, Peng Zhang, and Qian He. Pulid: Pure and lightning id customiza- tion via contrastive alignment.NeurIPS, 37:36777–36804,

  19. [19]

    Vall-e r: Robust and efficient zero-shot text- to-speech synthesis via monotonic alignment.arXiv preprint arXiv:2406.07855, 2024

    Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, and Furu Wei. Vall-e r: Robust and efficient zero-shot text- to-speech synthesis via monotonic alignment.arXiv preprint arXiv:2406.07855, 2024. 3

    Pith/arXiv arXiv 2024

  20. [20]

    Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024

    Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, and Jie Zhang. Id-animator: Zero-shot identity-preserving human video generation.arXiv preprint arXiv:2404.15275, 2024. 2, 3, 7, 8, 9, 5

    Pith/arXiv arXiv 2024

  21. [21]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.NeurIPS, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.NeurIPS, 30, 2017. 7

    2017

  22. [22]

    Denoising dif- fusion probabilistic models.NeurIPS, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.NeurIPS, 33:6840–6851, 2020. 1

    2020

  23. [23]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022. 2, 4, 5

    2022

  24. [24]

    Hunyuancustom: A multimodal-driven architecture for customized video gener- ation.arXiv preprint arXiv:2505.04512, 2025

    Teng Hu, Zhentao Yu, Zhengguang Zhou, Sen Liang, Yuan Zhou, Qin Lin, and Qinglin Lu. Hunyuancustom: A multimodal-driven architecture for customized video gener- ation.arXiv preprint arXiv:2505.04512, 2025. 2, 3, 6, 7, 8, 9, 1, 5

    Pith/arXiv arXiv 2025

  25. [25]

    Make-an-audio 2: Temporal-enhanced text- to-audio generation.arXiv preprint arXiv:2305.18474, 2023

    Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text- to-audio generation.arXiv preprint arXiv:2305.18474, 2023. 3

    Pith/arXiv arXiv 2023

  26. [26]

    Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang

    Y . Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricular- face: Adaptive curriculum learning loss for deep face recog- nition.CVPR, pages 5900–5909, 2020. 7

    2020

  27. [27]

    Conceptmaster: Multi-concept video customiza- tion on diffusion transformer models without test-time tun- ing.arXiv preprint arXiv:2501.04698, 2025

    Yuzhou Huang, Ziyang Yuan, Quande Liu, Qiulin Wang, Xintao Wang, Ruimao Zhang, Pengfei Wan, Di Zhang, and Kun Gai. Conceptmaster: Multi-concept video customiza- tion on diffusion transformer models without test-time tun- ing.arXiv preprint arXiv:2501.04698, 2025. 2, 3

    Pith/arXiv arXiv 2025

  28. [28]

    Videobooth: Diffusion-based video generation with image prompts

    Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. InCVPR, pages 6689–6700, 2024. 2

    2024

  29. [29]

    Vace: All-in-one video creation and editing.ICCV, 2025

    Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing.ICCV, 2025. 2, 3, 7, 8, 9, 5

    2025

  30. [30]

    Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models.arXiv preprint arXiv:2403.03100, 2024

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models.arXiv preprint arXiv:2403.03100, 2024. 3, 4, 5

    Pith/arXiv arXiv 2024

  31. [31]

    Fr\’echet audio distance: A metric for evaluating music enhancement algorithms.arXiv preprint arXiv:1812.08466, 2018

    Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr\’echet audio distance: A metric for evaluating music enhancement algorithms.arXiv preprint arXiv:1812.08466, 2018. 7

    Pith/arXiv arXiv 2018

  32. [32]

    Glow-tts: A generative flow for text-to-speech via monotonic alignment search.NeurIPS, 33:8067–8077, 2020

    Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search.NeurIPS, 33:8067–8077, 2020. 3

    2020

  33. [33]

    Glow: Generative flow with invertible 1x1 convolutions.NeurIPS, 31, 2018

    Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.NeurIPS, 31, 2018. 1

    2018

  34. [34]

    Auto-encoding varia- tional bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes.arXiv preprint arXiv:1312.6114, 2013. 4

    Pith/arXiv arXiv 2013

  35. [35]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024. 1

    Pith/arXiv arXiv 2024

  36. [36]

    V oicebox: Text-guided multilingual universal speech generation at scale.NeurIPS, 36:14005–14034, 2023

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. V oicebox: Text-guided multilingual universal speech generation at scale.NeurIPS, 36:14005–14034, 2023. 3

    2023

  37. [37]

    Bigvgan: A universal neural vocoder with large-scale training.arXiv preprint arXiv:2206.04658,

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training.arXiv preprint arXiv:2206.04658,

    Pith/arXiv arXiv

  38. [38]

    Sound-guided se- mantic video generation

    Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chany- oung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. Sound-guided se- mantic video generation. InECCV, pages 34–50. Springer,

  39. [39]

    Neural speech synthesis with transformer network

    Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. InProceedings of the AAAI conference on artificial intelli- gence, pages 6706–6713, 2019. 3

    2019

  40. [40]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. InICCV, pages 13401–13412, 2021. 3

    2021

  41. [41]

    Fish-speech: Lever- aging large language models for advanced multilingual text- to-speech synthesis.arXiv preprint arXiv:2411.01156, 2024

    Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. Fish-speech: Lever- aging large language models for advanced multilingual text- to-speech synthesis.arXiv preprint arXiv:2411.01156, 2024. 3, 8

    Pith/arXiv arXiv 2024

  42. [42]

    Flow matching for generative mod- eling.ICLR, 2023

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.ICLR, 2023. 1, 3

    2023

  43. [43]

    Audioldm 2: Learning holistic audio gen- eration with self-supervised pretraining.IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 32: 2871–2883, 2024

    Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio gen- eration with self-supervised pretraining.IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 32: 2871–2883, 2024. 3

    2024

  44. [44]

    Phantom: Subject-consistent video generation via cross- modal alignment.ICCV, 2025

    Lijie Liu, Tianxiang Ma, Bingchuan Li, Zhuowei Chen, Ji- awei Liu, Gen Li, Siyu Zhou, Qian He, and Xinglong Wu. Phantom: Subject-consistent video generation via cross- modal alignment.ICCV, 2025. 2, 3, 7, 8, 9, 5

    2025

  45. [45]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.ICLR, 2023

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.ICLR, 2023. 1, 3

    2023

  46. [46]

    Ovi: Twin backbone cross-modal fusion for audio-video genera- tion.arXiv preprint arXiv:2510.01284, 2025

    Chetwin Low, Weimin Wang, and Calder Katyal. Ovi: Twin backbone cross-modal fusion for audio-video genera- tion.arXiv preprint arXiv:2510.01284, 2025. 2, 3, 4, 6, 7, 8

    Pith/arXiv arXiv 2025

  47. [47]

    Magic-me: Identity-specific video customized diffu- sion

    Ze Ma, Daquan Zhou, Xue-She Wang, Chun-Hsiao Yeh, Xi- uyu Li, Huanrui Yang, Zhen Dong, Kurt Keutzer, and Jiashi Feng. Magic-me: Identity-specific video customized diffu- sion. InECCV, pages 19–37. Springer, 2024. 2, 3

    2024

  48. [48]

    Sora2.https://openai.com/index/ sora-2/, 2024

    OpenAI. Sora2.https://openai.com/index/ sora-2/, 2024. Accessed: 2025-09-30. 2, 3

    2024

  49. [49]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, pages 4195–4205, 2023. 2, 3

    2023

  50. [50]

    V oicecraft: Zero-shot speech editing and text-to-speech in the wild.arXiv preprint arXiv:2403.16973, 2024

    Puyuan Peng, Po-Yao Huang, Shang-Wen Li, Abdelrah- man Mohamed, and David Harwath. V oicecraft: Zero-shot speech editing and text-to-speech in the wild.arXiv preprint arXiv:2403.16973, 2024. 3

    Pith/arXiv arXiv 2024

  51. [51]

    Grad-tts: A diffusion prob- abilistic model for text-to-speech

    Vadim Popov, Ivan V ovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion prob- abilistic model for text-to-speech. InICML, pages 8599–

  52. [52]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. InProceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020. 1

    2020

  53. [53]

    Composeme: Attribute-specific image prompts for control- lable human image generation

    Guocheng Qian, Daniil Ostashev, Egor Nemchinov, Sergey Tulyakov, Kuan-Chieh Jackson Wang, and Kfir Aberman. Composeme: Attribute-specific image prompts for control- lable human image generation. InProceedings of the SIG- GRAPH Asia 2025 Conference Papers, pages 1–12, 2025. 3

    2025

  54. [54]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, pages 8748–8763. PmLR, 2021. 7

    2021

  55. [55]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInterna- tional conference on machine learning, pages 28492–28518. PMLR, 2023. 6, 7

    2023

  56. [56]

    Syncnet: Using causal con- volutions and correlating objective for time delay estimation in audio signals.arXiv preprint arXiv:2203.14639, 2022

    Akshay Raina and Vipul Arora. Syncnet: Using causal con- volutions and correlating objective for time delay estimation in audio signals.arXiv preprint arXiv:2203.14639, 2022. 6

    Pith/arXiv arXiv 2022

  57. [57]

    Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, 2020. 7

    2020

  58. [58]

    Fastspeech 2: Fast and high-quality end-to-end text to speech.arXiv preprint arXiv:2006.04558,

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech.arXiv preprint arXiv:2006.04558,

    Pith/arXiv arXiv 2006

  59. [59]

    Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. InCVPR, pages 10219– 10228, 2023. 2

    2023

  60. [60]

    Facenet: A unified embedding for face recognition and clus- tering

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clus- tering. InCVPR, pages 815–823, 2015. 5

    2015

  61. [61]

    Natural tts synthesis by conditioning wavenet on mel spectrogram predictions

    Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In ICASSP, pages 4779–4783. IEEE, 2018. 3

    2018

  62. [62]

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers.arXiv preprint arXiv:2304.09116,

    Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers.arXiv preprint arXiv:2304.09116,

    Pith/arXiv arXiv

  63. [63]

    S. S. Stevens, John E. V olkmann, and Edwin B. Newman. A scale for the measurement of the psychological magnitude pitch.Journal of the Acoustical Society of America, 8:185– 190, 1937. 4

    1937

  64. [64]

    Contrastive flow match- ing.ICCV, 2025

    George Stoica, Vivek Ramanujan, Xiang Fan, Ali Farhadi, Ranjay Krishna, and Judy Hoffman. Contrastive flow match- ing.ICCV, 2025. 5

    2025

  65. [65]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

  66. [66]

    Rethinking the inception ar- chitecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception ar- chitecture for computer vision. InCVPR, pages 2818–2826,

  67. [67]

    To- wards accurate generative models of video: A new metric & challenges.arXiv preprint arXiv:1812.01717, 2018

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric & challenges.arXiv preprint arXiv:1812.01717, 2018. 7

    Pith/arXiv arXiv 2018

  68. [68]

    Wan: Open and advanced large-scale video gen- erative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video gen- erative models.arXiv preprint arXiv:2503.20314, 2025. 1, 3

    Pith/arXiv arXiv 2025

  69. [69]

    Neural codec language models are zero-shot text to speech synthesizers.arXiv preprint arXiv:2301.02111, 2023

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huam- ing Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers.arXiv preprint arXiv:2301.02111, 2023. 3

    Pith/arXiv arXiv 2023

  70. [70]

    Universe-1: Unified audio-video generation via stitching of experts.arXiv preprint arXiv:2509.06155, 2025

    Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. Universe-1: Unified audio-video generation via stitching of experts.arXiv preprint arXiv:2509.06155, 2025. 2

    Pith/arXiv arXiv 2025

  71. [71]

    Customvideo: Customizing text-to- video generation with multiple subjects.arXiv preprint arXiv:2401.09962, 2024

    Zhao Wang, Aoxue Li, Lingting Zhu, Yong Guo, Qi Dou, and Zhenguo Li. Customvideo: Customizing text-to- video generation with multiple subjects.arXiv preprint arXiv:2401.09962, 2024. 2

    arXiv 2024

  72. [72]

    In- teracthuman: Multi-concept human animation with layout- aligned audio conditions.arXiv preprint arXiv:2506.09984,

    Zhenzhi Wang, Jiaqi Yang, Jianwen Jiang, Chao Liang, Gao- jie Lin, Zerong Zheng, Ceyuan Yang, and Dahua Lin. In- teracthuman: Multi-concept human animation with layout- aligned audio conditions.arXiv preprint arXiv:2506.09984,

    arXiv

  73. [73]

    Dreamvideo: Composing your dream videos with customized subject and motion

    Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhi- heng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hong- ming Shan. Dreamvideo: Composing your dream videos with customized subject and motion. InCVPR, pages 6537– 6549, 2024. 2

    2024

  74. [74]

    Exploring video quality assessment on user generated contents from aesthetic and technical perspectives

    Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jing- wen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. InICCV, pages 20144–20154, 2023. 6

    2023

  75. [75]

    Mo- tionbooth: Motion-aware customized text-to-video genera- tion.NeurIPS, 37:34322–34348, 2024

    Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Mo- tionbooth: Motion-aware customized text-to-video genera- tion.NeurIPS, 37:34322–34348, 2024. 2

    2024

  76. [76]

    Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners

    Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual- audio generation with diffusion latent aligners. InCVPR, pages 7151–7161, 2024. 3

    2024

  77. [77]

    Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo ...

    Pith/arXiv arXiv 2025

  78. [78]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 1

    Pith/arXiv arXiv 2025

  79. [79]

    Diffsound: Discrete diffusion model for text-to-sound generation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1720–1733, 2023

    Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1720–1733, 2023. 3

    2023

  80. [80]

    Cogvideox: Text-to-video diffusion models with an expert transformer.ICLR, 2025

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.ICLR, 2025. 1

    2025

Showing first 80 references.