Pith. sign in

REVIEW 4 major objections 5 minor 86 references

OmniMate generates real-time interactive avatars that switch between speaking and listening while preserving identity across long conversations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:49 UTC pith:PP6XYQIK

load-bearing objection Worth a referee, but the load-bearing interactive claim sits on an unvalidated LLM duration estimate and an evaluation that is thinner than the results suggest. the 4 major comments →

arxiv 2607.23023 v2 pith:PP6XYQIK submitted 2026-07-25 cs.CV

OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars

Quanyue Song , Yishan He , Yanbo Ding , Zhixiang He , Yongxiang Li , Caigui Jiang , Zhi Zhi Guo This is my paper
classification cs.CV
keywords real-time generationaudio-visual synthesisinteractive avatarsstreaming diffusionidentity preservationgeneration progress controllermulti-reference conditioningstate transitions
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that open-ended, real-time streaming generation of audio-visual avatars is achievable by explicitly modeling generation progress and providing persistent multi-reference identity cues. The key problem is that the generation horizon is unknown and identity drifts over long interactions. OmniMate's Generation Progress Controller (GPC) lets the model know how far along it is in a response, enabling seamless transitions between execution and listening states based on an LLM-estimated duration. The Multi-Reference Conditioning Module (MRCM) uses multiple reference images and a reference speech segment to keep appearance and voice consistent across many turns. If true, this would make interactive avatars feel natural, responsive, and stable in real-time use.

Core claim

The central discovery is that a diffusion-based streaming model can be trained to condition each generated chunk on an explicit progress signal (GPC) and on multiple reference images plus reference audio (MRCM), yielding real-time joint audio-visual generation with state transitions and long-term identity preservation. The model achieves high generation speed (27.64 FPS) and low time-to-first-frame (3.49s) while maintaining strong identity scores and the best speech accuracy among compared methods.

What carries the argument

Two modules carry the argument. GPC maps latent-aligned progress values (1 to 1000 during execution, 0 during listening) into a sinusoidal time embedding added to the latents, giving the model explicit awareness of how much of a response remains and enabling duration-aware chunk generation. MRCM concatenates multiple reference image latents with noisy video latents along the temporal dimension (with negative RoPE to avoid disturbing temporal positions) and replaces warm-up audio latents with reference speech latents during denoising, providing persistent visual and speaker identity cues. Training relies on a DMD-distilled bidirectional teacher to construct state-transition ODE trajectories,

Load-bearing premise

The system relies on the LLM's estimated execution duration being accurate; if the estimate is wrong, responses get truncated or repeated, breaking the natural interaction.

What would settle it

Feed the system instructions of systematically varied lengths and compare the actual response durations to the LLM-predicted durations; if a significant fraction of responses show perceptible word omissions or repetitions due to duration mismatch, the progress-conditioning mechanism does not robustly guarantee coherent responses as claimed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Real-time interactive avatars can sustain multi-turn conversations with low latency and seamless execution-to-listening transitions.
  • Long-form generation (tested up to 240 seconds) shows no significant degradation in visual quality, audio quality, or cross-modal consistency.
  • The framework builds on an existing joint audio-video foundation model with only lightweight adaptation, avoiding the need for massive new training data or architectures.
  • Explicit progress conditioning prevents premature termination and repetitive speech during open-ended interactions.
  • Using multiple reference images and a reference speech segment preserves identity under unseen poses and viewpoints over extended dialogues.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach suggests that accurate duration estimation is a critical bottleneck; learning to predict response length from instruction semantics (rather than relying solely on an LLM) could further reduce word omissions and repetitions.
  • Treating identity preservation as a multi-reference conditioning problem, rather than extending temporal context, may generalize to other streaming generative tasks where long-term consistency matters.
  • The state-transition framework (execution vs listening) could be extended to other interactive content generation, such as live narration or real-time game avatars, where the generation horizon is unknown.
  • The reliance on a distilled teacher for constructing supervision trajectories hints that the method's effectiveness may scale with the teacher's temporal consistency; a stronger teacher could yield sharper transitions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. OmniMate proposes a streaming framework built on LTX2.3 that jointly generates video, speech, and sound effects for interactive avatars. It introduces a Generation Progress Controller (GPC), which injects progress embeddings computed from an LLM-predicted execution duration to control execution/listening transitions, and a Multi-Reference Conditioning Module (MRCM), which concatenates multiple reference image latents and optionally replaces audio latents with reference speech to preserve visual and speaker identity. Training uses a DMD-distilled teacher to construct state-transition trajectories, followed by ODE initialization and self-forcing DMD distillation. The paper reports quantitative comparisons on an interaction-oriented adaptation of VerseBench, showing the highest FPS, the lowest TTFF, and strong audio-visual consistency over 30s–240s segments.

Significance. If the system operates as claimed, the two architectural ideas—explicit progress conditioning and multi-reference identity injection—are useful steps toward open-ended interactive audio-visual avatars. The paper provides a detailed training pipeline and attempts to compare with recent offline and streaming models. However, the evidence in the current manuscript is not commensurate with the strength of the central claims. The real-time claim covers only the generation model, the GPC's dependence on LLM duration estimates is not validated, and the benchmark is self-constructed without release or error bars. The core ideas are plausible and the writing is clear, but the manuscript needs substantial empirical additions before the claims are supported.

major comments (4)
  1. [Method Overview / Table 1] The real-time claim is not demonstrated end-to-end. FPS and TTFF in Table 1 measure only the streaming generation model on an H100 GPU; the method overview states that user speech is transcribed by an ASR system and processed by an LLM before generation. No end-to-end interaction latency or system latency breakdown is reported. The abstract's claim of 'real-time interactive avatar generation' and 'low-latency state transitions' therefore applies only to the generator, not the full interactive loop. Please report the complete ASR→LLM→generation pipeline latency, or explicitly state that the real-time claim is generator-only.
  2. [Equations (1)–(2) / Generation Progress Controller] The load-bearing GPC mechanism depends on an unvalidated LLM-estimated execution duration. At inference, T_v and T_a in Eqs. (1)–(2) are derived from this predicted duration, while training progress values are computed from ground-truth teacher-generated token counts. The paper never measures the accuracy of the LLM duration estimate, nor its effect on response completeness and execution/listening transitions. The 30s/60s/240s rows in Table 2 vary total generation length, not the mismatch between predicted and actual duration. The Limitations section itself concedes that 'inaccurate duration estimation may occasionally cause word omissions or repetitions.' Because seamless transitions and complete responses are a central claimed contribution, please add a quantitative evaluation of duration-prediction error and an ablation that deliberately perturbs the predicted duration to show gracefu
  3. [Tables 1 and 2 / Evaluation Data Construction] The evaluation evidence is insufficient to support the strong quantitative conclusions. Tables 1 and 2 report no error bars, confidence intervals, significance tests, or number of trials, and many metric differences are small (e.g., AQ 6.22 vs 5.92 in Table 1; V-ID 0.957 vs 0.952). The benchmark is an 'interaction-oriented adaptation of VerseBench' constructed by the authors, but no benchmark samples, annotation details, or evaluation code are provided. This makes it difficult to assess whether the reported advantages are stable or an artifact of a self-constructed test set. Please provide variance estimates, exact metric definitions, and make the benchmark and evaluation code publicly available.
  4. [Table 1 / Experimental Setup] The FPS/TTFF comparison is confounded by resolution and inference settings. The experimental setup states that OmniMate is evaluated at 480p (with DMD distillation at 384p), while baselines include MOVA-360P and other methods that are likely at their native resolutions with different chunk sizes and sampling steps. The paper does not report the resolution, number of steps, or chunk configuration for each compared method in Table 1. Thus the claimed 'highest streaming efficiency' may reflect favorable settings rather than architectural advantage. Please report matched-resolution comparisons, or provide a latency/quality Pareto plot across resolutions and steps for all methods.
minor comments (5)
  1. [Figure 2] The caption labels (b) as GPC and (c) as MRCM, but the figure body shows '(b) MRCM' and '(c) GPC' with swapped tags. This should be corrected.
  2. [Method Overview] The acronym DMD is used without definition at first occurrence in the Method Overview. Define 'distribution matching distillation' or add it to the Related Work introduction.
  3. [Table 2] The rows 'OmniMate(30s)', 'OmniMate(60s)', and 'OmniMate(240s)' are not explicitly described in the text. Clarify that these are the full model evaluated at different generation durations, and state what changes (e.g., the LLM-estimated duration) between them.
  4. [Eq. (4)] Equation (4) defines m_i = 1 for i < τ and then states 'm_i = 1 denotes the execution state'. The notation should be made consistent: currently the condition is m_i = 1 when i < τ, which matches, but the sentence is slightly confusing and could be rewritten for clarity.
  5. [Dataset / Evaluation] The paper references an appendix for further details on metrics and data construction, but the appendix content is not included in the submitted text. The references to 'VerseBench' also vary in capitalization ('Versebench' vs 'VerseBench'). Please ensure the appendix is available and standardize the capitalization.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are empirical and its conditioning mechanisms are not definitionally identical to their outputs.

full rationale

The central claims of OmniMate—real-time joint audio-visual streaming, long-term identity consistency, and smooth execution/listening transitions—are empirical results supported by quantitative comparisons, ablations, and qualitative demonstrations, rather than quantities derived from fitted constants. The GPC progress schedule in Eqs. (1)–(2) is a control signal computed from the LLM-estimated execution duration; training progress values are computed from ground-truth token counts of teacher-generated sequences, and inference uses the LLM estimate. This is standard conditioning, not a fitted parameter renamed as a prediction. The paper explicitly acknowledges the dependency and its failure mode in the Limitations: 'inaccurate duration estimation may occasionally cause word omissions or repetitions.' MRCM's reference-latent replacement in Eq. (8) is direct conditioning on the reference, not a claim to have predicted speaker identity from first principles; its effectiveness is tested by ablation (w/o SpeakerIdentity). The evaluation uses an author-adapted subset of the external VerseBench benchmark, but adapting an external benchmark for interaction scenarios is an evaluation design choice, not a derivational step that reduces to the paper's own outputs. Self-citations to InteractiveAvatar and Unison appear only in related work and are not load-bearing for the main contribution. No equation in the paper equals another by construction in a way that undercuts the claimed results, and no load-bearing argument rests on an unverified self-citation. The identified limitation about duration-estimation accuracy is a correctness risk, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The framework rests on standard deep-learning infrastructure (LTX-2.3 backbone, DMD distillation, self-forcing, RollingSink) rather than on new physical postulates. The free parameters are engineering hyperparameters. The key domain assumption — that an LLM can predict execution duration accurately — is load-bearing and acknowledged by the authors as a failure mode.

free parameters (4)
  • GPC progress scale (1..1000) = 1–1000
    Video and audio progress are linearly normalized into the range 1–1000 (Eq. 1–2); the range is a design choice with no sensitivity analysis.
  • Streaming chunk sizes = 3 video latents / 25 audio latents per chunk
    Chunk sizes are hyperparameters balancing latency and quality; no ablation is shown for these values.
  • Reference image count N = 4 (up to 4)
    Four identity frames are sampled as references during training and inference (duplicated if fewer); the choice is hand-set.
  • Training steps per stage = 50K/5K/8K/5K
    The four-stage training schedule (50K, 5K, 8K, 5K steps) is chosen without reported sensitivity.
axioms (4)
  • domain assumption LTX-2.3 backbone and its V-VAE/A-VAE latents are an adequate base model
    The framework is built on LTX-2.3; the temporal compression (first video latent = 1 frame, others = 8 frames) is assumed correct.
  • domain assumption DMD-distilled bidirectional teacher provides reliable state-transition supervision
    The teacher generates ODE trajectories used as supervision; the paper claims the multi-step teacher leaks prompts, but the distilled teacher's reliability is assumed.
  • domain assumption LLM accurately estimates response duration and content
    GPC progress is derived from the LLM-estimated duration; errors cause truncation/repetition (noted in Limitations).
  • domain assumption Reference images/audio persistently encode identity across poses and viewpoints
    MRCM conditioning assumes face/voice cues from a few references suffice across large appearance/viewpoint changes; this is a core design premise whose validity is under test.

pith-pipeline@v1.3.0-alltime-deepseek · 11484 in / 12889 out tokens · 119466 ms · 2026-08-01T03:49:28.815752+00:00 · methodology

0 comments
read the original abstract

Recent advances in diffusion-based generative models have enabled real-time audio-driven avatar generation and unified audio-visual synthesis, providing a promising foundation for interactive avatar systems. However, extending unified audio-visual synthesis to real-time interactive streaming remains challenging, as the generation horizon is unknown in advance and the generated identity may drift over long-term generation. To address these challenges, we propose OmniMate, a unified framework for open-ended real-time interactive audio-visual avatar generation. OmniMate jointly synthesizes visual content, speech, and sound effects in real time, enabling natural and immersive multi-turn interactions. To achieve adaptive response progression, we introduce a Generation Progress Controller (GPC) that explicitly models the generation progress of each streaming chunk, allowing the model to complete responses according to the desired progress and achieve seamless transitions between execution and listening states. To preserve long-term cross-modal identity consistency, we propose a Multi-Reference Conditioning Module (MRCM), which leverages multiple reference images and a reference speech segment to provide persistent visual and speaker identity cues throughout long-duration streaming interactions. Extensive experiments on an interaction-oriented adaptation of VerseBench demonstrate that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency. The results further show that OmniMate supports realistic, coherent, and responsive interactive avatar experiences over extended multi-turn conversations.

Figures

Figures reproduced from arXiv: 2607.23023 by Caigui Jiang, Quanyue Song, Yanbo Ding, Yishan He, Yongxiang Li, Zhixiang He, Zhi Zhi Guo.

Figure 1
Figure 1. Figure 1: We propose OmniMate, an open-ended real-time streaming audio-visual interactive avatar generation framework, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of OmniMate. (a) Streaming inference pipeline of our framework for real-time interactive audio-visual [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training procedure of GPC. (a) A DMD-distilled bidirectional teacher model is used to (b) generate ODE trajectories [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparisons with state-of-the-art methods. Our method demonstrates smooth transitions between interac [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative ablation of OmniMate. Ablation studies demonstrate that our full model achieves more natural and accurate [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

86 extracted references · 27 linked inside Pith

  1. [1]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

  2. [2]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

  3. [3]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  4. [4]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

  5. [5]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    doi:10.1016/s0020-7373(84)80003-6 1984

  6. [6]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

  7. [7]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

  8. [8]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

  9. [9]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

  10. [10]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    2023

  11. [11]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

  12. [12]

    arXiv preprint arXiv:2506.12479 , year=

    Ai flow: Perspectives, scenarios, and approaches (2025) , author=. arXiv preprint arXiv:2506.12479 , year=

    Pith/arXiv arXiv 2025

  13. [13]

    IEEE Transactions on Neural Networks and Learning Systems , volume=

    Positive-incentive noise , author=. IEEE Transactions on Neural Networks and Learning Systems , volume=. 2022 , publisher=

    2022

  14. [14]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

    Variational positive-incentive noise: How noise benefits models , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

  15. [15]

    Vicinagearth , volume=

    A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges , author=. Vicinagearth , volume=. 2024 , publisher=

    2024

  16. [16]

    Vicinagearth , volume=

    Data augmentation in human-centric vision , author=. Vicinagearth , volume=. 2024 , publisher=

    2024

  17. [17]

    Proceedings of the 33rd ACM International Conference on Multimedia , pages=

    Fantasytalking: Realistic talking portrait generation via coherent motion synthesis , author=. Proceedings of the 33rd ACM International Conference on Multimedia , pages=

  18. [18]

    arXiv preprint arXiv:2505.22647 , year=

    Let them talk: Audio-driven multi-person conversational video generation , author=. arXiv preprint arXiv:2505.22647 , year=

    Pith/arXiv arXiv

  19. [19]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    From slow bidirectional to fast autoregressive video diffusion models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  20. [20]

    arXiv preprint arXiv:2512.04677 , year=

    Live avatar: Streaming real-time audio-driven avatar generation with infinite length , author=. arXiv preprint arXiv:2512.04677 , year=

    Pith/arXiv arXiv

  21. [21]

    arXiv preprint arXiv:2509.21574 , year=

    X-streamer: Unified human world modeling with audiovisual interaction , author=. arXiv preprint arXiv:2509.21574 , year=

    arXiv

  22. [22]

    arXiv preprint arXiv:2512.22065 , year=

    StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars , author=. arXiv preprint arXiv:2512.22065 , year=

    arXiv

  23. [23]

    arXiv preprint arXiv:2601.10103 , year=

    FlowAct-R1: Towards Interactive Humanoid Video Generation , author=. arXiv preprint arXiv:2601.10103 , year=

    arXiv

  24. [24]

    arXiv preprint arXiv:2506.03099 , year=

    Talkingmachines: Real-time audio-driven facetime-style video via autoregressive diffusion models , author=. arXiv preprint arXiv:2506.03099 , year=

    Pith/arXiv arXiv

  25. [25]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  26. [26]

    Proceedings of the 28th ACM international conference on multimedia , pages=

    A lip sync expert is all you need for speech to lip generation in the wild , author=. Proceedings of the 28th ACM international conference on multimedia , pages=

  27. [27]

    Advances in Neural Information Processing Systems , volume=

    Vasa-1: Lifelike audio-driven talking faces generated in real time , author=. Advances in Neural Information Processing Systems , volume=

  28. [28]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  29. [29]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  30. [30]

    European Conference on Computer Vision , pages=

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  31. [31]

    arXiv preprint arXiv:2505.20156 , year=

    Hunyuanvideo-avatar: High-fidelity audio-driven human animation for multiple characters , author=. arXiv preprint arXiv:2505.20156 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2506.18866 , year=

    Omniavatar: Efficient audio-driven avatar video generation with adaptive body animation , author=. arXiv preprint arXiv:2506.18866 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:2507.03905 , year=

    Echomimicv3: 1.3 b parameters are all you need for unified multi-modal and multi-task human animation , author=. arXiv preprint arXiv:2507.03905 , year=

    arXiv

  34. [34]

    Advances in Neural Information Processing Systems , volume=

    Diffusion forcing: Next-token prediction meets full-sequence diffusion , author=. Advances in Neural Information Processing Systems , volume=

  35. [35]

    arXiv preprint arXiv:2506.08009 , year=

    Self forcing: Bridging the train-test gap in autoregressive video diffusion , author=. arXiv preprint arXiv:2506.08009 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2510.02283 , year=

    Self-forcing++: Towards minute-scale high-quality video generation , author=. arXiv preprint arXiv:2510.02283 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2509.22622 , year=

    Longlive: Real-time interactive long video generation , author=. arXiv preprint arXiv:2509.22622 , year=

    Pith/arXiv arXiv

  38. [38]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Learning to listen: Modeling non-deterministic dyadic facial motion , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  39. [39]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Diffusion-based realistic listening head generation via hybrid motion modeling , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  40. [40]

    arXiv preprint arXiv:2508.19320 , year=

    Midas: Multimodal interactive digital-human synthesis via real-time autoregressive video generation , author=. arXiv preprint arXiv:2508.19320 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv e-prints , pages=

    Omnitalker: Real-time text-driven talking head generation with in-context audio-visual style replication , author=. arXiv e-prints , pages=

  42. [42]

    ACM Transactions On Graphics (TOG) , volume=

    Makelttalk: speaker-aware talking-head animation , author=. ACM Transactions On Graphics (TOG) , volume=. 2020 , publisher=

    2020

  43. [43]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Learning individual styles of conversational gesture , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  44. [44]

    arXiv preprint arXiv:2503.20314 , year=

    Wan: Open and advanced large-scale video generative models , author=. arXiv preprint arXiv:2503.20314 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2512.23851 , year=

    Pretraining Frame Preservation in Autoregressive Video Memory Compression , author=. arXiv preprint arXiv:2512.23851 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2502.14786 , year=

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features , author=. arXiv preprint arXiv:2502.14786 , year=

    Pith/arXiv arXiv

  47. [47]

    Chung, J. S. and Nagrani, A. and Zisserman, A. VoxCeleb2: Deep Speaker Recognition. INTERSPEECH. 2018

    2018

  48. [48]

    Yu, Jianhui and Zhu, Hao and Jiang, Liming and Loy, Chen Change and Cai, Weidong and Wu, Wayne , booktitle=

  49. [49]

    arXiv preprint arXiv:2412.00115 , year=

    OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation , author=. arXiv preprint arXiv:2412.00115 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2312.17090 , year=

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels , author=. arXiv preprint arXiv:2312.17090 , year=

    Pith/arXiv arXiv

  51. [51]

    Advances in neural information processing systems , volume=

    Gans trained by a two time-scale update rule converge to a local nash equilibrium , author=. Advances in neural information processing systems , volume=

  52. [52]

    arXiv preprint arXiv:1812.01717 , year=

    Towards accurate generative models of video: A new metric & challenges , author=. arXiv preprint arXiv:1812.01717 , year=

    Pith/arXiv arXiv

  53. [53]

    Asian conference on computer vision , pages=

    Out of time: automated lip sync in the wild , author=. Asian conference on computer vision , pages=. 2016 , organization=

    2016

  54. [54]

    2024 , eprint=

    VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models , author=. 2024 , eprint=

    2024

  55. [55]

    2025 , eprint=

    StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation , author=. 2025 , eprint=

    2025

  56. [56]

    2026 , eprint=

    UniTalking: A Unified Audio-Video Framework for Talking Portrait Generation , author=. 2026 , eprint=

    2026

  57. [57]

    arXiv preprint arXiv:2505.10238 , year=

    Mtvcrafter: 4d motion tokenization for open-world human image animation , author=. arXiv preprint arXiv:2505.10238 , year=

    arXiv

  58. [58]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  59. [59]

    arXiv preprint arXiv:2509.17765 , year=

    Qwen3-omni technical report , author=. arXiv preprint arXiv:2509.17765 , year=

    Pith/arXiv arXiv

  60. [60]

    arXiv preprint arXiv:2304.11277 , year=

    Pytorch fsdp: experiences on scaling fully sharded data parallel , author=. arXiv preprint arXiv:2304.11277 , year=

    Pith/arXiv arXiv

  61. [61]

    Advances in neural information processing systems , volume=

    Gpipe: Efficient training of giant neural networks using pipeline parallelism , author=. Advances in neural information processing systems , volume=

  62. [62]

    2022 , eprint=

    LAION-5B: An open large-scale dataset for training next generation image-text models , author=. 2022 , eprint=

    2022

  63. [63]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  64. [64]

    2026 , eprint=

    Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation , author=. 2026 , eprint=

    2026

  65. [65]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Uniavgen: Unified audio and video generation with asymmetric cross-modal interactions , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  66. [66]

    arXiv preprint arXiv:2510.01284 , year=

    Ovi: Twin backbone cross-modal fusion for audio-video generation , author=. arXiv preprint arXiv:2510.01284 , year=

    Pith/arXiv arXiv

  67. [67]

    arXiv preprint arXiv:2601.03233 , year=

    LTX-2: Efficient Joint Audio-Visual Foundation Model , author=. arXiv preprint arXiv:2601.03233 , year=

    Pith/arXiv arXiv

  68. [68]

    arXiv preprint arXiv:2602.08794 , year=

    Mova: Towards scalable and synchronized video-audio generation , author=. arXiv preprint arXiv:2602.08794 , year=

    arXiv

  69. [69]

    arXiv preprint arXiv:2512.13677 , year=

    JoVA: Unified Multimodal Learning for Joint Video-Audio Generation , author=. arXiv preprint arXiv:2512.13677 , year=

    arXiv

  70. [70]

    arXiv preprint arXiv:2603.21986 , year=

    Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model , author=. arXiv preprint arXiv:2603.21986 , year=

    arXiv

  71. [71]

    and Park, Taesung , title =

    Yin, Tianwei and Gharbi, Micha\"el and Zhang, Richard and Shechtman, Eli and Durand, Fr\'edo and Freeman, William T. and Park, Taesung , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

    2024

  72. [72]

    Advances in neural information processing systems , volume=

    Improved distribution matching distillation for fast image synthesis , author=. Advances in neural information processing systems , volume=

  73. [73]

    arXiv preprint arXiv:2606.22905 , year=

    InteractiveAvatar: Real-Time Streaming Video Generation for Consistent and Intent-Aware Avatars , author=. arXiv preprint arXiv:2606.22905 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2605.25659 , year=

    StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration , author=. arXiv preprint arXiv:2605.25659 , year=

    Pith/arXiv arXiv

  75. [75]

    1: End-to-end Real-time Interactive Foundation Models , author=

    Wan-Streamer v0. 1: End-to-end Real-time Interactive Foundation Models , author=. arXiv preprint arXiv:2606.25041 , year=

    Pith/arXiv arXiv

  76. [76]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  77. [77]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vfhq: A high-quality dataset and benchmark for video face super-resolution , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  78. [78]

    arXiv preprint arXiv:1806.05622 , year=

    Voxceleb2: Deep speaker recognition , author=. arXiv preprint arXiv:1806.05622 , year=

    Pith/arXiv arXiv

  79. [79]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Celebv-text: A large-scale facial text-video dataset , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  80. [80]

    arXiv preprint arXiv:1804.03619 , year=

    Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation , author=. arXiv preprint arXiv:1804.03619 , year=

    Pith/arXiv arXiv

Showing first 80 references.