Pith. sign in

REVIEW 4 major objections 4 minor 124 references

LeapTalk claims that real-time, open-ended talking-head generation is achievable in a single forward step per video chunk, using a Brownian-bridge student distilled from a diffusion teacher, with identity preserved over arbitrarily long str

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:21 UTC pith:2FHKMLSU

load-bearing objection Genuinely novel bridge-forcing chunking idea, but the heterogeneous DMD core is under-derived and the experimental support is too thin to accept the headline numbers at face value. the 4 major comments →

arxiv 2608.00079 v1 pith:2FHKMLSU submitted 2026-07-29 cs.CV cs.SD

LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

Rongxiang Zhang , Songhua Liu This is my paper
classification cs.CV cs.SD
keywords talking-head generationsingle-step distillationBrownian bridgestreaming video synthesisdistribution matching distillationSNR-aligned time transformationaudio-driven animationidentity consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LeapTalk claims that real-time, open-ended talking-head generation is possible at one diffusion step per chunk, without the usual speed-versus-quality trade-off. Its idea is to stop generating each new video chunk from random noise, which causes errors to accumulate; instead, every chunk is generated along a Brownian bridge that starts from the same reference image and lands on the target frames. A flow-matching diffusion teacher is distilled into this one-step bridge student using a time transformation that matches signal-to-noise ratios, and an audio-conditioned guidance term keeps lip motion sharp. If correct, a single GPU can synthesize 55-200 frames per second while preserving identity over 30 minutes or more, which would make live interactive digital humans practical.

Core claim

The central claim is that the latency-quality trade-off in talking-head generation is not fundamental: by reformulating chunk generation as a data-to-data transport problem anchored to a persistent reference image, a single forward step of a Brownian-bridge student can replace dozens of denoising steps of a flow-matching diffusion teacher. The student is trained with a heterogeneous distribution-matching distillation that maps teacher time to student time through the SNR equality t = 1/(1 + sqrt((1-tau)/tau)), and augments the teacher score with audio-driven classifier-free guidance. The result is a streaming model that generates each chunk in one step, matches or beats multi-step baselines

What carries the argument

The Brownian bridge X_tau = (1-tau)I + tau*X1 + sqrt(tau(1-tau))*epsilon, with I being a persistent reference image and X1 the target chunk, turns each chunk into data-to-data transport. Bridge Forcing builds chunk inputs by splicing the last K frames of the previous chunk onto repeated reference frames, preserving motion continuity while re-anchoring identity. The SNR-aligned time transform t = Phi(tau) = 1/(1 + sqrt((1-tau)/tau)) lets a flow-matching teacher supervise a bridge student at matched noise levels. Audio-driven classifier-free guidance replaces the teacher score with a conditional/unconditional combination to preserve lip detail and motion diversity under one step.

Load-bearing premise

The load-bearing premise is that matching signal-to-noise ratios via the time transformation makes the flow-matching teacher's score and the Brownian-bridge student's score comparable enough for distribution-matching distillation; if that comparison is not well-defined, the one-step student may only be learning from direct reconstruction loss, and the claimed DMD advantage would be unsupported.

What would settle it

Train the same bridge student with only the weighted reconstruction and perceptual losses, leaving out the score-matching DMD term under identical hyperparameters, and compare FID/FVD and Sync-C on HDTF and CelebV-HQ. If removing the DMD term does not degrade the metrics at one step, the SNR-aligned heterogeneous distillation is not the source of the gains and the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Real-time streaming: at chunk size 33 frames and one step per chunk, the next chunk is generated in about 20% of the current chunk's playback time, so playback can be continuous and gap-free.
  • Open-ended identity: because every chunk is anchored to the same reference image, identity consistency stays flat in long rollouts of 30 minutes or more, whereas noise-to-data autoregressive baselines visibly drift.
  • Lip-sync survives extreme step reduction: the audio-driven CFG term prevents static or mismatched mouth motion, giving Sync-C/Sync-D scores comparable to or better than 4-50-step diffusion baselines.
  • The heterogeneous distillation scheme offers a reusable recipe: a pre-trained flow-matching diffusion teacher can be distilled into a bridge-based student via the SNR-aligned time transformation, not only for talking heads.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the SNR-aligned heterogeneous distillation could extend to other image-to-video or paired-translation tasks where a pre-trained noise-to-data diffusion model is too slow and a data-to-data bridge is a better prior, such as general image animation or controllable video editing.
  • Beyond the paper: the persistent-reference bridge design suggests the same architecture could be steered by non-audio controls (e.g., pose or text) while keeping identity anchored, enabling interactive agents; the paper does not test this.
  • Beyond the paper: because the bridge treats the reference image as a deterministic bias, extremely large deviations from the reference identity or head pose may be under-produced; a targeted test would measure head-pose diversity on side-view or profile inputs, which the paper only evaluates qualitatively.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LeapTalk, a framework for real-time, streaming talking-head generation. It replaces the usual noise-to-data autoregressive diffusion formulation with a reference-anchored Brownian-bridge process: each chunk is generated from a fixed reference image to a target frame, with the previous chunk's last frames as a motion prefix. To distill a pretrained flow-matching teacher into this one-step bridge student, the authors introduce a 'heterogeneous DMD' scheme with an SNR-aligned time transformation Φ(τ), plus an audio-driven classifier-free guidance term. The reported experiments claim 1-step inference at up to 200 FPS, stable identity over 30-minute streams, and FID/FVD/Sync-C/Sync-D improvements over multi-step baselines on HDTF and CelebV-HQ.

Significance. If the central claims are correct, this is a significant systems contribution: stable open-ended talking-head generation at 1 NFE with 55–200 FPS would break a real latency-quality trade-off. The bridge-forcing idea is simple and appealing, and the long-video qualitative results are striking. The paper also includes a fairly complete ablation structure and a user study. However, the theoretical centerpiece—heterogeneous DMD—is not rigorously defined: the score-matching gradient as written compares scores at points from two different forward processes, and the connection to any proper distribution-matching objective is asserted rather than derived. The empirical evaluation also lacks error bars and tunes hyperparameters on the test set. These issues are load-bearing for the paper's main novelty claim.

major comments (4)
  1. [Methods, 'Heterogeneous Distribution Matching Distillation', Eqs. (15)–(17); Algorithm 1 lines 12–19] The central DMD gradient is not well-defined. Eq. (17) subtracts a 'real' teacher score evaluated at xΦ(τ) from a 'fake' student score evaluated at xτ. These are two different random vectors produced by two different forward processes (flow-matching vs. Brownian bridge); equalizing a scalar SNR does not make the score functions comparable, and the difference is not the derivative of any KL divergence between student and teacher. Algorithm 1 never constructs xΦ(τ): line 15 calls V_real(xt, t, ...) and line 19 uses xt, but xt is never defined anywhere in the paper. If xt is a teacher-process sample, the teacher score is evaluated at a point unrelated to the student sample; if xt is the bridge state xτ, the teacher is queried outside its training distribution. The paper needs to (i) specify xΦ(τ) explicitly, (ii) derive the gradient from a well-defined distribution-matching objective, or (i
  2. [Methods, 'Bridge Training' and Eq. (18); Algorithm 1 lines 22–25] The final loss also contains L_rec and L_perc, which directly regress the one-step output to multi-step teacher rollouts. The reported ablation removes the time transformation but never removes the DMD term itself, so it is impossible to attribute the one-step quality to the distribution-matching component. Add an ablation trained with only L_rec + L_perc (no DMD) and one with only DMD (no regression/perceptual losses). Without these, the claim that heterogeneous DMD 'enables' high-quality one-step generation is unsupported.
  3. [Experiments, Implementation Details; Tables 1 and 3; Figure 8] All main quantitative results appear to come from a single run, with no error bars or random seeds reported. In addition, α and λ_perc are selected by looking at HDTF metrics (Figure 8 and Table 3) and then the same HDTF numbers are used in the headline table. The reported differences on HDTF (e.g., Sync-C 8.38 vs. 8.11) are within plausible run-to-run variation. Report mean±std over at least 3 seeds and perform hyperparameter selection on a validation split.
  4. [Methods, 'Reference-Anchored Bridge Forcing'; Figure 5] The DINO-similarity identity metric measures similarity to the reference image I, which is explicitly concatenated/repeated into every chunk input (Eqs. 10–11). A degenerate model that always copies the reference would score highly on this metric, so high DINO-vs-reference similarity is partly by construction. Report also identity consistency between generated frames at increasing temporal offsets, or evaluate baselines under the same reference-anchored chunk input, to separate the method's contribution from the conditioning signal.
minor comments (4)
  1. [Eqs. (12)–(17) and Algorithm 1] The paper mixes score functions s and velocity fields v without stating the conversion (e.g., Tweedie/score-velocity relation). Algorithm 1 computes v_cfg and then uses it as if it were a score. Please define the notation precisely.
  2. [Appendix, 'Derivation of the SNR-aligned Time Transformation'] Equation (14) is called 'Theorem 1,' but it is an algebraic solution to an SNR equality, not a theorem about score comparability. Rename and clarify what the mapping actually guarantees.
  3. [Abstract and Appendix Table 9] The main text claims 'up to 200 FPS,' but Appendix Table 9 shows that this is on H200; on A100 at 512×512 the speed is 104 FPS. Include the hardware and resolution caveat in the abstract or main text.
  4. [User Study, Table 4] User-study percentages are reported without confidence intervals or significance tests. Add them if space permits.

Circularity Check

1 steps flagged

The SNR-aligned time transformation is a definitional construction, and the claimed well-defined DMD score matching rests on that definition rather than on a derived equivalence.

specific steps
  1. self definitional [Methods, Eq. (13)-(17); Appendix 'Derivation of the SNR-aligned Time Transformation', Eq. (32)-(37)]
    "Let SNRteacher(t) and SNRstudent(τ) denote the SNR of the flow-matching teacher and the bridge student, respectively. There exists a monotonic time transformation t=Φ(τ) that aligns the two processes such that SNRteacher(t)=SNRstudent(τ). ... Since the objective is to recover Xtgt, we take (1−τ)Xtgt as the signal component and treat τXsrc as a deterministic bias term. ... We align the two processes by enforcing SNRTeacher(t)=SNRStudent(τ) ... Solving for t yields ... t=Φ(τ)=1/(1+sqrt((1−τ)/τ))."

    The theorem is an algebraic restatement of the definition: Φ is solved from the very equality it is claimed to establish, and the 'signal vs bias' split is chosen to make SNRStudent take the desired form. The paper then calls this 'consistent score evaluation' and uses it to justify the DMD gradient in Eq. (17), where s_real(xΦ(τ)) and s_fake(xτ) are evaluated at states from different forward processes. Scalar SNR equality does not make the score difference a valid divergence gradient; the conclusion of 'well-defined score matching' is not derived from the theorem but assumed in the definition of alignment.

full rationale

Most empirical validation is external: FID/FVD, Sync-C/D, FPS, and user studies compare against unrelated baselines on HDTF and CelebV-HQ, and the one-step generator is also trained with direct regression and perceptual losses to teacher rollouts, so the framework as a whole is not a closed self-consistent loop. However, the central theoretical justification of heterogeneous DMD is circular in a load-bearing way: the SNR-aligned time transformation is constructed by solving for t=Φ(τ) from the equality SNR_teacher(t)=SNR_student(τ), so the theorem states only what was put into the definition. The paper then uses that definitional equality to claim 'consistent score evaluation' and a valid DMD objective, even though the teacher and student scores are evaluated at different states generated by different forward processes. This is a self-definitional step in the derivation chain, not an independent mathematical justification. Separately, Algorithm 1 calls the teacher on an undefined xt, which is a missing-definition issue rather than circularity. Overall, partial circularity in the theoretical claim, but the empirical comparisons are independent, so the score is moderate.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central contribution rests on (i) a representational assumption that a fixed reference image plus audio is sufficient to span the target chunk via a bridge, (ii) a pretrained flow-matching teacher whose multi-step rollouts are reliable targets, and (iii) an alignment assumption that matching a hand-defined SNR makes teacher and student scores comparable for DMD. Hyperparameters α and λperc are tuned on the same evaluation sets. No new physical or conceptual entities are introduced with independent falsifiable handles.

free parameters (4)
  • Audio CFG scale α = 1.6
    Chosen from HDTF sensitivity analysis (Fig 8) to maximize Sync-C; used for final HDTF/CelebV-HQ results without a separate validation split.
  • Perceptual loss weight λ_perc = 4.0
    Chosen via Table 3 sensitivity on PSNR/SSIM/LPIPS; λ_perc=8.0 severely degrades metrics, indicating instability.
  • Chunk length = 33 frames
    Selected from Table 8 to minimize T_gen/T_chunk at 512×512 on A100.
  • Bridge loss and spatial-mask weights (λ_br, λ_face, λ_lip) = not disclosed
    Appear in Eq. 18 and Algorithm 1 but no values are reported; they modulate the final objective and affect reproduction.
axioms (5)
  • domain assumption Each target chunk can be modeled as the endpoint of a Brownian bridge from a fixed reference image I; the reference image carries enough identity information.
    Methods 'Reference-Anchored Bridge Forcing', Eq. 8-9. If I cannot represent new appearance or identity variation, the model cannot generate it.
  • domain assumption A pretrained flow-matching video diffusion model (Wan2.1-T2V) is a valid teacher, and its multi-step rollouts are reliable training targets.
    Used throughout the distillation; Algorithm 1 line 9 calls multiStepRollout(V_real). Teacher quality on VividHead is not independently verified.
  • ad hoc to paper Equalizing the hand-defined SNR makes teacher and student score functions comparable for DMD.
    Appendix 'Derivation of the SNR-aligned Time Transformation', Eq. 33-37. The mapping only equates a chosen SNR; it does not prove the two conditional distributions are comparable, yet Eq. 16 subtracts their scores.
  • standard math Brownian bridge Gaussian conditioning and the DMD gradient formula hold as standard mathematics.
    Preliminaries Eq. 3-6; Eq. 12 from Yin et al. 2024. No errors apparent in these background statements.
  • domain assumption SyncNet, DINOv2, FID/FVD, and the small user study reliably measure lip-sync, identity, and quality.
    Used in Experiments; automated metrics are partial proxies and the user study has n=30 with no confidence intervals.

pith-pipeline@v1.3.0-alltime-deepseek · 15036 in / 19147 out tokens · 202818 ms · 2026-08-04T01:21:28.131822+00:00 · methodology

0 comments
read the original abstract

Long-form and real-time talking-head generation remains challenging due to a latency-quality trade-off: inefficient multi-step diffusion prohibits streaming generation, whereas real-time autoregressive approaches suffer from error accumulation and identity drift. To address this drawback, we propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step, scaling to arbitrarily long videos. At the heart of our approach lies a single-step bridge distillation scheme. On the one hand, departing from the conventional noise-to-data paradigm, we introduce a data-to-data transport formulation based on a Brownian bridge. Anchored by a persistent reference, this strategy effectively mitigates identity drift and enhances long-term temporal stability. On the other hand, to enable smooth knowledge transfer from a pre-trained diffusion teacher to the student bridge model, we explore a heterogeneous distillation framework with an SNR-aligned time transformation $\Phi(\tau)$, which bridges the functional discrepancy between the two models. Moreover, we propose an audio-driven classifier-free guidance mechanism to maintain fine-grained lip synchronization under extreme step reduction. Extensive experiments demonstrate that our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS, significantly outperforming existing approaches in both efficiency and stability. Project Page: https://zhangrongxiang.github.io/leaptalk-page/

Figures

Figures reproduced from arXiv: 2608.00079 by Rongxiang Zhang, Songhua Liu.

Figure 1
Figure 1. Figure 1: Overview of LeapTalk. Given audio and a reference image, our method enables open-ended streaming talking-head generation with consistent identity. It achieves 1-step inference per chunk at up to 200 FPS, delivering up to 15000× speedup while maintaining strong lip-sync accuracy. and static motion in one-step generation. In summary, our key contributions include: • We propose LeapTalk, a real-time, streamin… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison between conventional noise-to-data flow matching and our Bridge Forcing paradigm. The identity consistency is preserved along bridge process, while error accumulates in the forward diffusion process. Heterogeneous Distribution Matching Distillation Real-time talking-head generation requires autoregressive generation with few-step (ideally 1 step) synthesis. We propose a heterogeneous DMD (See [… view at source ↗
Figure 3
Figure 3. Figure 3: Distillation pipeline of our approach. The one-step student generates frames conditioned on a static reference. To enable stable distillation across heterogeneous processes, we apply a time transformation t = Φ(τ ) to align noise levels between the flow-matching teacher and the bridge-based student, allowing consistent score supervision and a well-defined DMD objective. the student is defined on a Brownian… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison on long-video streaming generation. Frames are sampled from streaming rollouts at increasing time indices. The red boxes mark identity incon￾sistency and drift, and the blue boxes mark slight or incorrect lip-sync. LeapTalk maintains stable identity and lip motion as generation proceeds. model always keeps a high and stable similarity curve, whereas baselines such as OmniAvatar degra… view at source ↗
Figure 6
Figure 6. Figure 6: Ablation results demonstrating the contribution of each component in our framework. …Under... …Administration... …Job... …More... LeapTalk w/o Audio- driven CFG [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of ablation effects of audio-driven [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Sensitivity analysis of audio-driven CFG. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Generated results under diverse and challenging [PITH_FULL_IMAGE:figures/full_fig_p007_10.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visual comparison across different VAEs. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: More qualitative results on challenging condi [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

124 extracted references · 33 linked inside Pith

  1. [2]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  2. [3]

    Zhu, Hao and Wu, Wayne and Zhu, Wentao and Jiang, Liming and Tang, Siwei and Zhang, Li and Liu, Ziwei and Loy, Chen Change , booktitle=

  3. [4]

    Diffusion Schr\"

    De Bortoli, Valentin and Thornton, James and Heng, Jeremy and Doucet, Arnaud , booktitle =. Diffusion Schr\"

  4. [6]

    International Conference on Machine Learning , year=

    FrameBridge: Improving Image-to-Video Generation with Bridge Models , author=. International Conference on Machine Learning , year=

  5. [7]

    2025 , eprint=

    Stochastic Interpolants: A Unifying Framework for Flows and Diffusions , author=. 2025 , eprint=

    2025

  6. [8]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Flow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  7. [9]

    Arxiv , year=

    Packing Input Frame Contexts in Next-Frame Prediction Models for Video Generation , author=. Arxiv , year=

  8. [10]

    2026 , eprint=

    Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation , author=. 2026 , eprint=

    2026

  9. [11]

    2024 , eprint=

    Rolling Diffusion Models , author=. 2024 , eprint=

    2024

  10. [12]

    2025 , eprint=

    End-to-End Training for Autoregressive Video Diffusion via Self-Resampling , author=. 2025 , eprint=

    2025

  11. [13]

    2025 , eprint=

    Stable Video Infinity: Infinite-Length Video Generation with Error Recycling , author=. 2025 , eprint=

    2025

  12. [14]

    2024 , eprint=

    Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion , author=. 2024 , eprint=

    2024

  13. [15]

    NeurIPS , year =

    FIFO-Diffusion: Generating Infinite Videos from Text without Training , author =. NeurIPS , year =

  14. [16]

    arXiv preprint arXiv:2603.04379 , year=

    Helios: Real Real-Time Long Video Generation Model , author=. arXiv preprint arXiv:2603.04379 , year=

    arXiv

  15. [17]

    2026 , eprint=

    SoulX-FlashHead: Oracle-guided Generation of Infinite Real-time Streaming Talking Heads , author=. 2026 , eprint=

    2026

  16. [18]

    Proceedings of the 33rd ACM International Conference on Multimedia , pages=

    Ditto: Motion-space diffusion for controllable realtime talking head synthesis , author=. Proceedings of the 33rd ACM International Conference on Multimedia , pages=

  17. [20]

    2026 , eprint=

    REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation , author=. 2026 , eprint=

    2026

  18. [21]

    2025 , eprint=

    Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation , author=. 2025 , eprint=

    2025

  19. [22]

    2025 , eprint=

    Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length , author=. 2025 , eprint=

    2025

  20. [23]

    arXiv:2304.07193 , year=

    DINOv2: Learning Robust Visual Features without Supervision , author=. arXiv:2304.07193 , year=

    Pith/arXiv arXiv

  21. [25]

    arXiv preprint arXiv:2403.17694 , year=

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation , author=. arXiv preprint arXiv:2403.17694 , year=

    Pith/arXiv arXiv

  22. [26]

    arXiv preprint arXiv:2412.04448 , year=

    MEMO: Memory-guided diffusion for expressive talking video generation , author=. arXiv preprint arXiv:2412.04448 , year=

    Pith/arXiv arXiv

  23. [27]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    SadTalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  24. [28]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    High-resolution image synthesis with latent diffusion models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  25. [29]

    2024 , booktitle=

    Video generation models as world simulators , author=. 2024 , booktitle=

    2024

  26. [31]

    arXiv preprint arXiv:1312.6114 , year=

    Auto-encoding variational bayes , author=. arXiv preprint arXiv:1312.6114 , year=

    Pith/arXiv arXiv

  27. [33]

    arXiv e-prints , pages=

    SoulX-LiveTalk: Real-Time Infinite Streaming of Audio-Driven Avatars via Self-Correcting Bidirectional Distillation , author=. arXiv e-prints , pages=

  28. [34]

    International Conference on Medical image computing and computer-assisted intervention , pages=

    U-net: Convolutional networks for biomedical image segmentation , author=. International Conference on Medical image computing and computer-assisted intervention , pages=. 2015 , organization=

    2015

  29. [35]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  30. [39]

    arXiv preprint arXiv:2501.00103 , year=

    LTX-Video: Realtime video latent diffusion , author=. arXiv preprint arXiv:2501.00103 , year=

    Pith/arXiv arXiv

  31. [43]

    arXiv preprint arXiv:2410.10122 , year=

    MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling , author=. arXiv preprint arXiv:2410.10122 , year=

    Pith/arXiv arXiv

  32. [45]

    Learning Transferable Visual Models From Natural Language Supervision , booktitle =

    Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever , editor =. Learning Transferable Visual Models From Natural Language Supervision , booktitle =. 2021 , url =

    2021

  33. [46]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    EchoMimic: Lifelike audio-driven portrait animations through editable landmark conditions , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  34. [47]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Sonic: Shifting focus to global audio perception in portrait animation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  35. [48]

    arXiv preprint arXiv:2411.19509 , year=

    Ditto: Motion-space diffusion for controllable realtime talking head synthesis , author=. arXiv preprint arXiv:2411.19509 , year=

    arXiv

  36. [49]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vfhq: A high-quality dataset and benchmark for video face super-resolution , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  37. [50]

    ECCV , month = Augest, year =

    Wang, Kaisiyuan and Wu, Qianyi and Song, Linsen and Yang, Zhuoqian and Wu, Wayne and Qian, Chen and He, Ran and Qiao, Yu and Loy, Chen Change , title =. ECCV , month = Augest, year =

  38. [51]

    Castellano, Brandon , title =

  39. [52]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Scalable diffusion models with transformers , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  40. [53]

    Advances in neural information processing systems , volume=

    Gans trained by a two time-scale update rule converge to a local nash equilibrium , author=. Advances in neural information processing systems , volume=

  41. [54]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    One-step diffusion with distribution matching distillation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  42. [55]

    Advances in neural information processing systems , volume=

    wav2vec 2.0: A framework for self-supervised learning of speech representations , author=. Advances in neural information processing systems , volume=

  43. [56]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Effective whole-body pose estimation with two-stages distillation , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  44. [57]

    arXiv preprint arXiv:1804.03619 , year=

    Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation , author=. arXiv preprint arXiv:1804.03619 , year=

    Pith/arXiv arXiv

  45. [58]

    arXiv preprint arXiv:2510.02283 , year=

    Self-Forcing++: Towards Minute-Scale High-Quality Video Generation , author=. arXiv preprint arXiv:2510.02283 , year=

    Pith/arXiv arXiv

  46. [59]

    Asian conference on computer vision , pages=

    Out of time: automated lip sync in the wild , author=. Asian conference on computer vision , pages=. 2016 , organization=

    2016

  47. [60]

    FVD: A new metric for video generation , author=

  48. [61]

    Huang, Ziqi and He, Yinan and Yu, Jiashuo and Zhang, Fan and Si, Chenyang and Jiang, Yuming and Zhang, Yuanhan and Wu, Tianxing and Jin, Qingyang and Chanpaisit, Nattapol and Wang, Yaohui and Chen, Xinyuan and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei , booktitle=

  49. [62]

    arXiv preprint arXiv:2312.17090 , year=

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels , author=. arXiv preprint arXiv:2312.17090 , year=

    Pith/arXiv arXiv

  50. [63]

    arXiv preprint arXiv:2507.09862 , year=

    SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation , author=. arXiv preprint arXiv:2507.09862 , year=

    Pith/arXiv arXiv

  51. [65]

    arXiv preprint arXiv:2309.14509 , year=

    Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models , author=. arXiv preprint arXiv:2309.14509 , year=

    Pith/arXiv arXiv

  52. [66]

    arXiv preprint arXiv:2310.01889 , year=

    Ring attention with blockwise transformers for near-infinite context , author=. arXiv preprint arXiv:2310.01889 , year=

    Pith/arXiv arXiv

  53. [67]

    GitHub repository , howpublished =

    LightX2V Contributors , title =. GitHub repository , howpublished =. 2025 , publisher =

    2025

  54. [68]

    and Ermon, Stefano and Rudra, Atri and R

    Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. Flash. Advances in Neural Information Processing Systems (NeurIPS) , year=

  55. [69]

    Dao, Tri , booktitle=. Flash

  56. [70]

    Advances in Neural Information Processing Systems , volume=

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision , author=. Advances in Neural Information Processing Systems , volume=

  57. [72]

    2025 , journal =

    LBM: Latent Bridge Matching for Fast Image-to-Image Translation , author=. 2025 , journal =

    2025

  58. [73]

    2025 , howpublished =

    Boer Bohan, Ollin , title =. 2025 , howpublished =

    2025

  59. [74]

    CVPR , year=

    Shen, Shuai and Zhao, Wenliang and Meng, Zibin and Li, Wanhua and Zhu, Zheng and Zhou, Jie and Lu, Jiwen , title=. CVPR , year=

  60. [75]

    2022 , eprint=

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow , author=. 2022 , eprint=

    2022

  61. [76]

    and Shechtman, Eli and Wang, Oliver , title =

    Zhang, Richard and Isola, Phillip and Efros, Alexei A. and Shechtman, Eli and Wang, Oliver , title =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , month =

  62. [77]

    2026 , eprint=

    TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation , author=. 2026 , eprint=

    2026

  63. [78]

    2026 , eprint=

    EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control , author=. 2026 , eprint=

    2026

  64. [79]

    2023 , eprint=

    Flow Matching for Generative Modeling , author=. 2023 , eprint=

    2023

  65. [80]

    2021 , eprint=

    Score-Based Generative Modeling through Stochastic Differential Equations , author=. 2021 , eprint=

    2021

  66. [81]

    2023 , howpublished =

    Google , title =. 2023 , howpublished =

    2023

  67. [82]

    2025 , eprint=

    TalkVid: A Large-Scale Diversified Dataset for Audio-Driven Talking Head Synthesis , author=. 2025 , eprint=

    2025

  68. [83]

    Proceedings of the IEEE international conference on computer vision , pages=

    Flownet: Learning optical flow with convolutional networks , author=. Proceedings of the IEEE international conference on computer vision , pages=

  69. [84]

    arXiv preprint arXiv:2304.11277 , year=

    Pytorch fsdp: experiences on scaling fully sharded data parallel , author=. arXiv preprint arXiv:2304.11277 , year=

    Pith/arXiv arXiv

  70. [85]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. J. Mach. Learn. Res. , volume =. 2020 , url =

    2020

  71. [86]

    arXiv preprint arXiv:2207.12598 , year=

    Classifier-free diffusion guidance , author=. arXiv preprint arXiv:2207.12598 , year=

    Pith/arXiv arXiv

  72. [87]

    2023 , eprint=

    FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling , author=. 2023 , eprint=

    2023

  73. [88]

    The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Bardia Doosti and Shujon Naha and Majid Mirbagheri and David Crandall , title =. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , month =

  74. [89]

    Baevski, A.; Zhou, Y.; Mohamed, A.; and Auli, M. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449--12460

    2020

  75. [90]

    Boer Bohan, O. 2025. TAEHV: Tiny AutoEncoder for Hunyuan Video. https://github.com/madebyollin/taehv

    2025

  76. [91]

    Brooks, T.; Peebles, B.; Holmes, C.; DePue, W.; Guo, Y.; Jing, L.; Schnurr, D.; Taylor, J.; Luhman, T.; Luhman, E.; Ng, C.; Wang, R.; and Ramesh, A. 2024. Video generation models as world simulators. In openai

    2024

  77. [92]

    Chadebec, C.; Tasar, O.; Sreetharan, S.; and Aubin, B. 2025. LBM: Latent Bridge Matching for Fast Image-to-Image Translation. arXiv preprint arXiv:2503.07535

    Pith/arXiv arXiv 2025

  78. [93]

    M.; Du, Y.; Simchowitz, M.; Tedrake, R.; and Sitzmann, V

    Chen, B.; Monso, D. M.; Du, Y.; Simchowitz, M.; Tedrake, R.; and Sitzmann, V. 2024. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. arXiv:2407.01392

    Pith/arXiv arXiv 2024

  79. [94]

    Chen, Z.; Cao, J.; Chen, Z.; Li, Y.; and Ma, C. 2025. EchoMimic: Lifelike audio-driven portrait animations through editable landmark conditions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2403--2410

    2025

  80. [95]

    Chern, E.; Hu, Z.; Tang, B.; Su, J.; Chern, S.; Deng, Z.; and Liu, P. 2025. LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation. arXiv preprint arXiv:2512.23576

    arXiv 2025

Showing first 80 references.