Pith. sign in

REVIEW 4 major objections 6 minor 66 references

A low-bitrate codebook bottleneck can force video representations to separate motion from content, and a diffusion decoder turns the separated latents back into frames.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A self-supervised diffusion framework with a low-bitrate vector-quantization bottleneck learns disentangled motion and content latents supporting motion transfer and auto-regressive generation.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A novel bitrate-bottleneck approach to self-supervised video disentanglement with strong reported numbers, but test-set bitrate tuning and an unvalidated content-invariance assumption mean the evidence is not yet conclusive. the 4 major comments →

arxiv 2509.08376 v1 pith:XQXOVBBH submitted 2025-09-10 cs.CV

Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video

Xiao Li , Qi Chen , Xiulian Peng , Kai Yu , Xie Chen , Yan Lu This is my paper
classification cs.CV
keywords video representation learningmotion-content disentanglementlow-bitrate vector quantizationinformation bottlenecklatent diffusion modelsmotion transferautoregressive video generationself-supervised learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a general, self-supervised information bottleneck—not task-specific priors like landmarks or 3D face models—is enough to disentangle video into motion and content. The proposed pipeline, BCD, encodes a video clip into per-frame motion features and a clip-wide content feature, then pushes the motion features through a low-bitrate vector-quantization codebook so that only the information needed to reconstruct the dynamics survives. Reconstruction is done by a latent diffusion model conditioned on the quantized motion and the content feature, with a cross-driven objective that forces the model to use one half of the clip's content to reconstruct the other half's motion. If the claim holds, motion transfer becomes a simple content-token swap, and the discrete motion space is structured enough to support autoregressive generation. The paper reports stronger cross-identity motion transfer on talking-head videos than methods using keypoints or 3D priors, and it shows the same pipeline works on 2D cartoon sprites.

Core claim

The central claim is that an information bottleneck in the form of a target-bitrate codebook is sufficient to learn genuinely disentangled motion and content representations from video, without face-specific inductive biases. In BCD, a T5-style transformer encodes a video's latent frames into per-frame motion features and clip-wise content features; motion features are split into groups and quantized through a distance-based Gumbel-softmax VQ, with the empirical codebook entropy matched to a target bitrate (4 kbps on talking heads). A DiT-based latent diffusion decoder reconstructs the latent frames from the quantized motion and content, and a cross-driven training strategy reconstructs the

What carries the argument

The key object is the bitrate-controlled group vector quantizer applied to motion features: motion latents are split into groups, each quantized with a small codebook using distance-based Gumbel-softmax sampling, and a loss matches the average codebook entropy to a target bitrate. This low-bitrate quantization is the information bottleneck that is supposed to block content from leaking into motion while preserving enough information to reconstruct dynamics. A second mechanism is the cross-driven training strategy, which splits each training clip into two halves and reconstructs the second half from the first half's content plus the second half's motion, preventing the model from trivially co

Load-bearing premise

The cross-driven training strategy assumes every training clip can be split into two temporal halves that show the same static content with only motion changing; if a clip contains viewpoint, lighting, or appearance change, the content branch can absorb dynamics and the claimed separation weakens.

What would settle it

Take a training clip made by splicing two different scenes together so that content changes mid-clip and check whether the content token changes across frames while the motion bitrate stays at target; if the model reconstructs it well, the content branch has absorbed dynamics, violating separation. More concretely, run cross-identity motion transfer on videos with viewpoint or lighting changes inside a clip: a sharp degradation in transfer quality would show the separation depends on the static-content assumption rather than on the bottleneck alone.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the separation holds, cross-identity motion transfer reduces to content-feature swapping: take motion tokens from one video and the content token from another, and the diffusion decoder animates the second identity with the first video's expression and pose.
  • The learned motion space is discrete and structured enough that a GPT-2 autoregressive model can generate new motion sequences conditioned on a single content frame, with reasonable temporal patterns.
  • Because the method avoids face-specific priors, the same pipeline transfers to non-face video domains such as 2D cartoon sprites, where it achieves perfect attribute accuracy on the test set.
  • Bitrate becomes a dial for disentanglement: too low a target loses motion detail, too high lets identity leak into the motion stream, and the lowest cross-driven error sits near 4 kbps, slightly below the average bitrate of talking-head codecs.
  • The paper's 3D-mesh evaluation metrics (shape error, motion error, cross-transfer error) provide a more pose-robust alternative to cosine-similarity identity metrics for talking-head generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same bitrate-bottleneck principle could be applied to other factor splits, such as separating viewpoint from object identity or expression from speaker identity, because the cap is domain-agnostic rather than tied to human faces.
  • Editorial inference: making the target bitrate adaptive per clip, or learned rather than fixed, is a natural extension the paper leaves open, especially since it notes the optimal bitrate is dataset-dependent.
  • Editorial inference: if the learned motion codebook is as cleanly separated as the talking-head results suggest, downstream applications like video editing or compression could reuse the codebook without retraining the encoder, though this is not demonstrated in the paper.
  • Editorial inference: the cross-driven assumption that content is static within a half-clip means the method's performance on in-the-wild clips with camera motion or scene changes is an open question that the paper does not directly measure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes BCD (Bitrate-Controlled Diffusion), a self-supervised framework that disentangles video into a clip-wise content feature and frame-wise motion features. A T5 encoder extracts both features from VAE-tokenized video latents; the motion branch is passed through a low-bitrate group-VQ bottleneck, and a latent diffusion decoder reconstructs the video conditioned on motion and content. Training uses a rate-distortion objective with a bitrate loss and a cross-driven strategy in which the content from the first half of a clip and motion from the second half reconstruct the second half. The method is evaluated on LRS3 talking-head motion transfer and autoregressive motion generation, and on LPC Sprites for attribute transfer. The authors report state-of-the-art or competitive results on LRS3 against keypoint-, 3D-, and linear-motion-basis baselines, and perfect attribute accuracy on Sprites.

Significance. If the central claim holds, BCD would be a meaningful advance: it learns a discrete, low-bitrate motion space without face-specific priors, supports cross-identity motion transfer and autoregressive generation, and the 3D-mesh-based evaluation metrics are a useful complement to CSIM/FID. The paper also provides several ablations (bitrate, cross-driven strategy, bitrate loss, content-frame count) and a user study. However, the evidence for true motion/content disentanglement rests on assumptions that are not empirically validated, and the headline quantitative comparison is weakened by test-set hyperparameter selection and the absence of error bars or significance tests. The contribution is potentially significant, but the current manuscript does not yet establish the claims at the level required for a top-tier venue.

major comments (4)
  1. [§4.4, Table 3] The target bitrate is selected by minimizing the cross-driven error on the test set ('the minimum cross-driven error occurs at approximately 4kbps'), and the same cross-driven error is used as the main disentanglement metric in Table 1. This creates a circular evaluation: a hyperparameter is fitted to the test-set metric that is then used to claim state-of-the-art disentanglement. The manuscript should either use a held-out validation split for bitrate selection and report test-set results only for the selected value, or report a sensitivity analysis with error bars. Without this, the comparison against baselines is not a fair assessment of the method's true performance.
  2. [Appendix A.1 / §3.4] The cross-driven strategy is load-bearing for disentanglement. It assumes 'minimal content changes within each training clip' and says this 'can be ensured through data preprocessing,' but no check is provided that LRS3 clips satisfy this assumption. Because only the motion branch passes through the VQ bottleneck and the content feature c is unconstrained, any appearance/pose/lighting drift within a clip can be encoded into the low-bitrate motion code, since the bottleneck limits bitrate, not semantics. The low cross-driven error would then reflect the model's ability to carry residual appearance through motion, not true motion/content separation. I ask for concrete evidence: e.g., measure content-feature invariance across clip halves, or probe whether identity/pose information can be decoded from motion codes, or show that content codes do not change when the same identity is shown with
  3. [Tables 1 and 3; §4.2] No error bars, confidence intervals, or significance tests are reported for any of the quantitative results, including the main motion-transfer comparison and the bitrate ablation. The reported differences between 2, 4, and 6 kbps are small (e.g., cross error 5.68 vs 3.67 vs 4.13 in Table 3), and without variance estimates it is impossible to know whether 4 kbps is genuinely optimal or whether the claim is noise. The same applies to the claim that BCD 'performs the best' in Table 1 for several metrics. Please report standard errors or confidence intervals over multiple runs or over test-set resampling, and state the number of test videos used.
  4. [§1 and §3.2] The paper repeatedly claims that BCD makes 'less assumptions and inductive biases' than prior work. In practice, the method assumes that training clips are content-stationary (cross-driven strategy) and that a low-bitrate bottleneck on motion alone yields the correct semantic split. These are real assumptions. I recommend softening the framing or providing empirical evidence that the assumptions are met. The bitrate loss in Eq. (7) controls the entropy of the codebook, not the semantic content of the motion codes; the connection between bitrate and disentanglement is argued qualitatively and should not be stated as an established fact.
minor comments (6)
  1. [Eq. (7)] The notation is inconsistent: H appears both as the model bitrate and as the target bitrate in Eq. (7), while Eq. (5) defines an estimate \hat{H}. Please define separate symbols (e.g., H_model and H_target) for clarity.
  2. [Appendix A.1, Eq. (13)] The AdaGN equations appear to contain a typo: '(t_s, t_b) and (z_s, z_b)' should likely be '(t_s, t_b) and (m_s, m_b)' to match the motion conditioning described in the text.
  3. [§4.3] There is a typo: 'first two colums' should be 'first two columns.' Also, the user study reports only 18 participants and 15 video groups; this is quite small and should be mentioned as a limitation in the main text.
  4. [Table 3 / §4.4] The row label '4.0 (single ref.)' is not clearly explained in the table caption. I assume it refers to using a single content frame; please make this explicit in the caption.
  5. [Appendix A.1] The claim that cross-driven strategy 'can be ensured through data preprocessing' is vague. What preprocessing exactly? Cropping to 4-second clips is mentioned, but no verification of content invariance is described. This is connected to a major comment, but it also hinders reproducibility.
  6. [§4.2 and Table A7] Some hyperparameters appear only in the appendix, but the main text says 'target bitrate H_target=160' while the appendix lists 'Target bitrate 4kbps.' Please reconcile these values and clarify the relation between bits/frame and kbps.

Circularity Check

0 steps flagged

No load-bearing circularity; minor same-group citations and bitrate tuning do not reduce the central claim to its inputs.

full rationale

BCD's derivation chain is self-contained with respect to its empirical claims. The disentanglement mechanism is a low-bitrate group-VQ bottleneck on motion features (Sec. 3.2) plus a cross-driven training objective (Sec. 3.4). Neither is defined in terms of the evaluation metrics: training uses latent MSE (Eq. 6) and bitrate MSE (Eq. 7), while evaluation uses external 3D mesh fitting (App. C) and FID/CSIM on held-out videos. The bitrate target is a hyperparameter chosen partly from an external codec estimate [19] and validated over a grid (Tab. 3); this is model selection, not a fitted parameter renamed as a prediction. The same-group citations [33, 19] are motivational or provide a reference bitrate; the central claim does not reduce to them. The cross-driven training objective is a self-supervised proxy that mirrors the structure of motion transfer, but test-time transfer combines content and motion from different videos, which training never does, so the evaluation remains an extrapolation rather than a restatement of the loss. The Appendix A.1 assumption of minimal content changes within clips is a real correctness risk—if violated, the unconstrained content branch could push appearance dynamics into the motion code—but this is an assumption/limitation, not circularity. External benchmarks and held-out comparisons support the central claim.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on a dataset-tuned bitrate, an information-bottleneck heuristic, a content-stationarity assumption for the cross-driven loss, and a pretrained image VAE. These are not derived from first principles.

free parameters (3)
  • Target bitrate H_target = 4 kbps (LRS3), 150 bps (Sprites)
    Sets the strength of the information bottleneck on motion; chosen empirically from dataset statistics and validated by minimizing the reported cross-driven error (Sec. 4.4).
  • Rate-distortion weight lambda = 0.04
    Balances diffusion denoising loss and bitrate loss; set by hand (Sec. 3.4).
  • Content query count K = Not stated
    Number of learnable content prefix tokens in the transformer; fixed architectural choice (Sec. 3.1).
axioms (4)
  • domain assumption An information bottleneck causes disentanglement
    The core mechanism relies on IB theory from [1,3,55]; applied without proof to video motion and content separation (Sec. 3.2).
  • domain assumption Content is stationary within each training clip
    Cross-driven training splits clips and assumes minimal content changes (Appendix A.1); load-bearing for the training signal.
  • domain assumption The pretrained Stable Diffusion 2.0 VAE is a valid latent space for video
    All reconstruction happens in VAE latents; the VAE was trained on images and may distort motion details (Sec. 3).
  • standard math Empirical batch entropy approximates codebook entropy
    Eq. (4)-(5) estimate entropy from batch histograms; a standard estimator, though biased for small batches.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video." pith.science (2026). https://pith.science/paper/XQXOVBBH

@misc{pith2026250908376,
  author       = {Pith},
  title        = {Pith review of: Bitrate-Controlled Diffusion for Disentangling Motion and Content in Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQXOVBBH}},
  note         = {Machine review of arXiv:2509.08376}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We propose a novel and general framework to disentangle video data into its dynamic motion and static content components. Our proposed method is a self-supervised pipeline with less assumptions and inductive biases than previous works: it utilizes a transformer-based architecture to jointly generate flexible implicit features for frame-wise motion and clip-wise content, and incorporates a low-bitrate vector quantization as an information bottleneck to promote disentanglement and form a meaningful discrete motion space. The bitrate-controlled latent motion and content are used as conditional inputs to a denoising diffusion model to facilitate self-supervised representation learning. We validate our disentangled representation learning framework on real-world talking head videos with motion transfer and auto-regressive motion generation tasks. Furthermore, we also show that our method can generalize to other types of video data, such as pixel sprites of 2D cartoon characters. Our work presents a new perspective on self-supervised learning of disentangled video representations, contributing to the broader field of video analysis and generation.

Figures

Figures reproduced from arXiv: 2509.08376 by Kai Yu, Qi Chen, Xiao Li, Xie Chen, Xiulian Peng, Yan Lu.

Figure 1
Figure 1. Figure 1: Overview of our method. (a) BCD learns compact and disentangled features to represent video data in a self-supervised manner: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: BCD is a diffusion model which consists of (a) a pre-trained image encoder to generate latent features for each video frame; [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Motion transfer results on different input videos. The reference appearance and motion frames are shown in the first two colums. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Motion generation results. We use the content and motion feature of a single frame as the prompt shown on the left; five of the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Motion transfer results on Sprites test set. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 42 canonical work pages · 3 internal anchors

  1. [1]

    Emergence of in- variance and disentanglement in deep representations.The Journal of Machine Learning Research, 19(1):1947–1980,

    Alessandro Achille and Stefano Soatto. Emergence of in- variance and disentanglement in deep representations.The Journal of Machine Learning Research, 19(1):1947–1980,

    work page 1947

  2. [2]

    Lrs3-ted: a large-scale dataset for visual speech recog- nition.arXiv preprint arXiv:1809.00496, 2018

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisser- man. Lrs3-ted: a large-scale dataset for visual speech recog- nition.arXiv preprint arXiv:1809.00496, 2018. 2, 6

    Pith/arXiv arXiv 2018

  3. [3]

    Deep variational information bottleneck.arXiv preprint arXiv:1612.00410, 2016

    Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck.arXiv preprint arXiv:1612.00410, 2016. 3, 4

    Pith/arXiv arXiv 2016

  4. [4]

    vq- wav2vec: Self-supervised learning of discrete speech repre- sentations.arXiv preprint arXiv:1910.05453, 2019

    Alexei Baevski, Steffen Schneider, and Michael Auli. vq- wav2vec: Self-supervised learning of discrete speech repre- sentations.arXiv preprint arXiv:1910.05453, 2019. 4

    Pith/arXiv arXiv 1910

  5. [5]

    Con- trastively disentangled sequential variational autoencoder

    Junwen Bai, Weiran Wang, and Carla P Gomes. Con- trastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems, 34: 10105–10118, 2021. 1, 3, 8, 13

    work page 2021

  6. [6]

    Multifactor Sequential Disentanglement via Structured Koopman Autoencoders

    Nimrod Berman, Ilan Naiman, and Omri Azencot. Multifac- tor sequential disentanglement via structured koopman au- toencoders.arXiv preprint arXiv:2303.17264, 2023. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    Sequential disentanglement by extracting static information from a single sequence element.arXiv preprint arXiv:2406.18131, 2024

    Nimrod Berman, Ilan Naiman, Idan Arbiv, Gal Fadlon, and Omri Azencot. Sequential disentanglement by extracting static information from a single sequence element.arXiv preprint arXiv:2406.18131, 2024. 1, 3, 4

    work page arXiv 2024

  8. [8]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 5

    work page 2023

  9. [9]

    Hyperreenact: One-shot reenactment via jointly learning to refine and re- target faces

    Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, Ioannis Patras, and Georgios Tzimiropoulos. Hyperreenact: One-shot reenactment via jointly learning to refine and re- target faces. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 3, 6

    work page 2023

  10. [10]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  11. [11]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PMLR, 2020. 2

    work page 2020

  12. [12]

    John Wiley & Sons, 1999

    Thomas M Cover.Elements of information theory. John Wiley & Sons, 1999. 4

    1999

  13. [13]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. InCVPR, 2019. 6

    work page 2019

  14. [14]

    Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set

    Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. InProceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition workshops, pages 0–0, 2019. 6, 14

    work page 2019

  15. [15]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 2

    Pith/arXiv arXiv 2010

  16. [16]

    Headgan: One-shot neural head synthesis and editing

    Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. InProceedings of the IEEE/CVF International con- ference on Computer Vision, pages 14398–14407, 2021. 15

    work page 2021

  17. [17]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 2

    2021

  18. [18]

    High-fidelity and freely controllable talking head video generation

    Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu. High-fidelity and freely controllable talking head video generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5609–5619, 2023. 1, 3, 15

    work page 2023

  19. [19]

    Implicit mo- tion function

    Yue Gao, Jiahao Li, Lei Chu, and Yan Lu. Implicit mo- tion function. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19278– 19289, 2024. 8

    work page 2024

  20. [20]

    Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.Advances in neural information processing systems, 27, 2014. 2

    work page 2014

  21. [21]

    Neu- ral head avatars from monocular rgb videos

    Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neu- ral head avatars from monocular rgb videos. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18653–18664, 2022. 3

    work page 2022

  22. [22]

    Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020. 2

    2020

  23. [23]

    Vec- tor quantized diffusion model for text-to-image synthesis

    Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vec- tor quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 10696–10706, 2022. 2

    work page 2022

  24. [24]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control.arXiv preprint arXiv:2407.03168, 2024

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control.arXiv preprint arXiv:2407.03168, 2024. 1, 3, 6

    Pith/arXiv arXiv 2024

  25. [25]

    Disentangled Recurrent Wasserstein Autoencoder

    Jun Han, Martin Renqiang Min, Ligong Han, Li Erran Li, and Xuan Zhang. Disentangled recurrent wasserstein autoen- coder.arXiv preprint arXiv:2101.07496, 2021. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  26. [26]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 9729–9738, 2020. 2

    2020

  27. [27]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017. 6

    2017

  28. [28]

    Look ma, no markers: holistic perfor- mance capture without the hassle.ACM Transactions on Graphics (TOG), 43(6), 2024

    Charlie Hewitt, Fatemeh Saleh, Sadegh Aliakbarian, Lohit Petikam, Shideh Rezaeifar, Louis Florentin, Zafiirah Hose- nie, Thomas J Cashman, Julien Valentin, Darren Cosker, and Tadas Baltruˇsaitis. Look ma, no markers: holistic perfor- mance capture without the hassle.ACM Transactions on Graphics (TOG), 43(6), 2024. 6, 13

    work page 2024

  29. [29]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 5, 12

    work page 2020

  30. [30]

    Implicit identity representation conditioned memory compensation network for talking head video generation

    Fa-Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation network for talking head video generation. InICCV, 2023. 3, 6

    work page 2023

  31. [31]

    Neu- ral compression-based feature learning for video restoration

    Cong Huang, Jiahao Li, Bin Li, Dong Liu, and Yan Lu. Neu- ral compression-based feature learning for video restoration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5872–5881, 2022. 3

    work page 2022

  32. [32]

    SODA: Bottleneck Diffusion Models for Representation Learning

    Drew A Hudson, Daniel Zoran, Mateusz Malinowski, An- drew K Lampinen, Andrew Jaegle, James L McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. arXiv preprint arXiv:2311.17901, 2023. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  33. [33]

    Dis- entangled feature learning for real-time neural speech cod- ing

    Xue Jiang, Xiulian Peng, Yuan Zhang, and Yan Lu. Dis- entangled feature learning for real-time neural speech cod- ing. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 3, 4

    work page 2023

  34. [34]

    Elucidating the design space of diffusion-based generative models.Advances in Neural Information Processing Sys- tems, 35:26565–26577, 2022

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models.Advances in Neural Information Processing Sys- tems, 35:26565–26577, 2022. 5

    work page 2022

  35. [35]

    Deep contextual video com- pression.Advances in Neural Information Processing Sys- tems, 34:18114–18125, 2021

    Jiahao Li, Bin Li, and Yan Lu. Deep contextual video com- pression.Advances in Neural Information Processing Sys- tems, 34:18114–18125, 2021. 1, 3, 5

    work page 2021

  36. [36]

    Hybrid spatial-temporal en- tropy modelling for neural video compression

    Jiahao Li, Bin Li, and Yan Lu. Hybrid spatial-temporal en- tropy modelling for neural video compression. InProceed- ings of the 30th ACM International Conference on Multime- dia, 2022

    work page 2022

  37. [37]

    Neural video compression with diverse contexts

    Jiahao Li, Bin Li, and Yan Lu. Neural video compression with diverse contexts. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22616–22626, 2023. 3

    work page 2023

  38. [38]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023. 3

    Pith/arXiv arXiv 2023

  39. [39]

    Motion-focused contrastive learning of video representations

    Rui Li, Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, and Tao Mei. Motion-focused contrastive learning of video representations. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 2105–2114,

    work page

  40. [40]

    Self- conditioned image generation via generating representations

    Tianhong Li, Dina Katabi, and Kaiming He. Self- conditioned image generation via generating representations. arXiv preprint arXiv:2312.03701, 2023. 2

    Pith/arXiv arXiv 2023

  41. [41]

    Disentangled sequential autoencoder.arXiv preprint arXiv:1803.02991, 2018

    Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder.arXiv preprint arXiv:1803.02991, 2018. 1, 3, 8

    Pith/arXiv arXiv 2018

  42. [42]

    Anitalker: animate vivid and di- verse talking faces through identity-decoupled facial motion encoding

    Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. Anitalker: animate vivid and di- verse talking faces through identity-decoupled facial motion encoding. InProceedings of the 32nd ACM International Conference on Multimedia, pages 6696–6705, 2024. 1

    work page 2024

  43. [43]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 6

    Pith/arXiv arXiv 2017

  44. [44]

    Implicit warping for animation with image sets.Advances in Neural Information Processing Systems, 35:22438–22450, 2022

    Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu. Implicit warping for animation with image sets.Advances in Neural Information Processing Systems, 35:22438–22450, 2022. 1, 3

    work page 2022

  45. [45]

    Sample and predict your latent: modality-free sequential disentan- glement via contrastive estimation

    Ilan Naiman, Nimrod Berman, and Omri Azencot. Sample and predict your latent: modality-free sequential disentan- glement via contrastive estimation. InInternational Con- ference on Machine Learning, pages 25694–25717. PMLR,

    work page

  46. [46]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  47. [47]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers.arXiv preprint arXiv:2208.06366, 2022

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers.arXiv preprint arXiv:2208.06366, 2022. 2

    Pith/arXiv arXiv 2022

  48. [48]

    Language models are unsu- pervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners.OpenAI blog, 1(8):9, 2019. 7

    work page 2019

  49. [49]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learn- ing Research, 21(140):1–67, 2020. 4

    2020

  50. [50]

    Deep visual analogy-making.Advances in neural informa- tion processing systems, 28, 2015

    Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making.Advances in neural informa- tion processing systems, 28, 2015. 2, 6

    work page 2015

  51. [51]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3, 5

    2022

  52. [52]

    Temporal context mining for learned video compression

    Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu. Temporal context mining for learned video compression. IEEE Transactions on Multimedia, 2022. 3

    work page 2022

  53. [53]

    First order motion model for image animation.Advances in neural information processing systems, 32, 2019

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation.Advances in neural information processing systems, 32, 2019. 1, 3, 6, 15

    work page 2019

  54. [54]

    Sequential representation learning via static- dynamic conditional disentanglement

    Mathieu Cyrille Simon, Pascal Frossard, and Christophe De Vleeschouwer. Sequential representation learning via static- dynamic conditional disentanglement. InEuropean Confer- ence on Computer Vision, pages 110–126. Springer, 2024. 1, 3, 4, 8

    work page 2024

  55. [55]

    The information bottleneck method.arXiv preprint physics/0004057, 2000

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method.arXiv preprint physics/0004057, 2000. 1, 3, 4

    Pith/arXiv arXiv 2000

  56. [56]

    Neural discrete representation learning.Advances in neural information pro- cessing systems, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advances in neural information pro- cessing systems, 30, 2017. 2

    2017

  57. [57]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 2, 3

    2017

  58. [58]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. InProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 10039–10049,

    work page

  59. [59]

    Latent image animator: Learning to animate im- ages via latent space navigation

    Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate im- ages via latent space navigation. InInternational Conference on Learning Representations, 2021. 3, 6

    work page 2021

  60. [60]

    Huggingface’s transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´emi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. 12

    Pith/arXiv arXiv 1910

  61. [61]

    Fake it till you make it: face analysis in the wild using synthetic data alone

    Erroll Wood, Tadas Baltru ˇsaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild using synthetic data alone. InProceedings of the IEEE/CVF international conference on computer vision, pages 3681–3691, 2021. 14

    work page 2021

  62. [62]

    3d face reconstruction with dense landmarks

    Erroll Wood, Tadas Baltru ˇsaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen, Nikola Milosavljevi´c, Daniel Wilde, Stephan Garbin, Toby Sharp, Ivan Stojiljkovi´c, et al. 3d face reconstruction with dense landmarks. InEuropean Confer- ence on Computer Vision, pages 160–177. Springer, 2022. 6, 13, 14

    work page 2022

  63. [63]

    N ¨uwa: Visual synthesis pre- training for neural visual world creation

    Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. N ¨uwa: Visual synthesis pre- training for neural visual world creation. InEuropean con- ference on computer vision, pages 720–736. Springer, 2022. 2

    work page 2022

  64. [64]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653–9663, 2022. 2

    work page 2022

  65. [65]

    Videogpt: Video generation using vq-vae and trans- formers.arXiv preprint arXiv:2104.10157, 2021

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers.arXiv preprint arXiv:2104.10157, 2021. 2

    Pith/arXiv arXiv 2021

  66. [66]

    S3vae: Self-supervised sequential vae for representation disentanglement and data generation

    Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6538–6547, 2020. 1, 3, 8 Appendix The appendix contains additional implementation details for our trainin...

    work page 2020

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.