Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Jointly learned motion and appearance codebooks, retrieved by a transformer at multiple scales, are claimed to yield state-of-the-art talking-head videos with only 4.3M parameters.

desk verdict A well-ablated codebook-based talking-head generator with a solid same-identity story, undermined by an unseeded 50-video eval and a cross-identity claim that its own supplementary table contradicts. read the letter →

arxiv 2412.00719 v2 pith:DT4UBEQS submitted 2024-12-01 cs.CV

Shuling Zhao , Fa-Ting Hong , Xiaoshui Huang , Dan Xu This is my paper · ORCID
classification cs.CV
keywords talkingheadvideogenerationmulti-scalemotioncodebookappearancecompensationtransformerretrievalflowrefinementone-shotfacereenactment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the two bottlenecks in talking-head video generation, inaccurate motion flow and incomplete appearance information from a single source image, can be addressed together by learning two dictionaries, or codebooks, from the training data. One codebook stores recurring local motion patterns; the other stores recurring facial texture and structure patterns. At each of several scales, a transformer retrieves entries from both codebooks to refine the estimated motion flow and to repair the source features warped by that flow. The paper reports that this joint compensation improves reconstruction quality, motion transfer, and identity preservation on VoxCeleb1, and stays competitive on the harder CelebV-HQ test set, using a 4.3M-parameter model running at about 15 FPS on an RTX 3090.

What carries the argument

The central object is a pair of multi-scale codebooks: a motion codebook $\mathcal{C}_M = \{m_k\}_{k=1}^K$ of 32-dimensional local motion-flow patterns and an appearance codebook $\mathcal{C}_A = \{a_k\}_{k=1}^T$ of 256-dimensional local appearance patterns. Codes are split into $N$ groups, and scale $i$ uses the first $i$ groups, so general patterns are shared while finer patterns are reserved for larger scales. A shared transformer $\mathcal{T}_M$ for motion and a window-partitioned transformer $\mathcal{T}_A$ for appearance retrieve codes by cross-attention: the motion transformer queries with the encoded current flow, warped source feature, and driving keypoint feature and outputs a residual flow; the appearance transformer queries with the warped source feature and outputs a compensated feature. The codebooks are updated with a VQ-VAE-style code-level loss while the rest of the network is trained with reconstruction, equivariance, keypoint-distance, and adversarial losses.

What would settle it

Re-run the same evaluation on multiple independently drawn 50-video samples from VoxCeleb1 and CelebV-HQ with the released checkpoints, reporting means and standard deviations; if the FID, LPIPS, or AKD gaps over MRFA and LIA fall within one standard deviation or flip sign across samples, the claimed state-of-the-art advantage is not established.

Watch

Extended reading notes

Core claim

The central claim is that motion and appearance should be compensated as a coupled process rather than separately. Starting from a coarse keypoint-based motion flow, the method refines the flow coarse-to-fine: at scale i a motion-codebook compensation module retrieves the nearest codes for the current flow, warped source feature, and driving keypoint feature, adds the decoded residual to the flow, and uses the refined flow to warp the source feature. An appearance-codebook compensation module then retrieves appearance codes to repair the warped feature before the image decoder fuses all scales. The two codebooks are learned jointly with the whole network, so gradients from appearance refinement shape the motion codes and vice versa. The paper argues that this coupling produces motion flows that are more flexible and appearance features with fewer distortions, and it reports quantitative gains over unsupervised and diffusion-based baselines on same-identity reconstruction and cross-identity reenactment.

Load-bearing premise

The claim of state-of-the-art quality rests on a comparison made on a single unseeded random sample of 50 test videos per dataset, with small metric gaps and no significance testing.

Editorial extensions

If this is right

  • On VoxCeleb1 same-identity reconstruction, the paper reports the best FID, PSNR, L1, LPIPS, AKD, and AED among the compared open-source methods, including diffusion-based methods trained on larger data.
  • Multi-scale motion codebook compensation alone improves motion transfer and image quality over the baseline, and adding multi-scale appearance compensation further improves FID, LPIPS, and AKD, with the largest gains when both codebooks are learned jointly.
  • The framework generalizes to CelebV-HQ without fine-tuning, staying competitive and achieving the best AKD among unsupervised methods there.
  • At 4.3M parameters and 352.91 GFLOPs, the model runs at 15.13 FPS on an RTX 3090, faster than the reported speeds of MRFA, AniPortrait, Follow-Your-Emoji, and LivePortrait.
  • Because motion and appearance compensation are coupled at every scale, the compensated multi-scale features can be fed into a standard VQGAN-style decoder, making the codebook mechanism a plug-in enhancement to that decoder design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated implication is that the same two-codebook compensation idea could be applied to other image-animation tasks, such as bodies, hands, or general objects, where motion and appearance also interact; the codebooks would need to capture the corresponding motion and texture patterns.
  • The ablation suggests the code allocation scheme, sharing early codes across scales and reserving later codes for fine detail, is doing real work; a testable prediction is that reordering or reindexing code groups would change the quality-detail tradeoff.
  • The reported FID advantage over diffusion models on VoxCeleb1 is notable because those models were trained on larger datasets; if this pattern persists on broader benchmarks, it would suggest that learned motion and appearance priors can substitute for large-scale pre-training in this task.
  • The paper's own stated limitation, appearance leakage from the keypoint-based motion estimator, implies that replacing the keypoint backbone with a shape-agnostic motion estimator would be a direct test of how much of the remaining leakage is due to the codebook design versus the motion estimator.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a talking-head video generation framework that jointly learns multi-scale motion and appearance codebooks and uses transformer-based compensation modules to refine motion flows and warped source features across scales. The motion codebook stores local motion patterns and the appearance codebook stores facial texture patterns; a code-allocation scheme shares coarse codes across scales and adds finer codes at larger scales. Experiments on VoxCeleb1 and CelebV-HQ compare the method with GAN-based and diffusion-based baselines, reporting strong same-identity reconstruction numbers and competitive cross-identity reenactment, with ablations on the codebooks, code allocation, codebook size, and model design.

Significance. If the performance claims are sustained, the paper makes a useful contribution: the joint learning of motion and appearance codebooks with multi-scale compensation is a well-motivated and clearly described design, and the ablation study supports the role of each component. The authors also provide a user study, an inference-speed comparison, and an honest limitation statement about appearance leakage. The main value would be a lightweight, efficient alternative to diffusion-based talking-head generators. However, the significance is currently bounded by evaluation weaknesses: the central state-of-the-art claim rests on a single unseeded 50-video test sample, small metric margins, and a cross-identity table whose numbers are partly in tension with the text.

major comments (3)
  1. [Sec. 4.1, Table 1] The evaluation protocol uses only 50 randomly sampled videos per dataset with no seed, no repeated sampling, and no standard deviations or significance tests. Several of the margins that support the main claim are very small, e.g., VoxCeleb1 PSNR 25.30 vs. 25.26 for MRFA, LPIPS 0.1846 vs. 0.1872, and FID 43.15 vs. 43.25 for FYE. Without error bars or a significance test, these differences are within plausible noise, so the statement in Sec. 4.2 that the method 'outperforms the other methods almost on all metrics' is not yet established at the level of a state-of-the-art claim. Please report mean and standard deviation over multiple samples/seeds and, where feasible, a paired significance test.
  2. [Supplementary Table 3 and Sec. 7.2.2] The conclusion that the method 'generally demonstrates the highest overall performance' in cross-identity reenactment is not directly supported by the table: LIA outperforms Ours on FID (72.55 vs. 76.47) and CSIM (0.6505 vs. 0.6142), and MRFA outperforms Ours on ARD (1.5903 vs. 1.6234). The text acknowledges these individual cases but does not provide an explicit aggregate criterion, such as mean rank or a combined score, that would justify the overall claim. Either define and apply such a criterion or temper the conclusion to match the reported numbers.
  3. [Sec. 4.3, Table 2] In the ablation table, the 'Baseline + SMC' row shows an FID regression from 47.83 to 49.00 and an AED regression from 0.1106 to 0.1167, yet the text states that single-scale motion codebook compensation 'already improves motion transfer and image quality' and lists only PSNR, L1, LPIPS, and AKD. The unaddressed FID and AED regressions make the component-wise claim appear selective. Please report and discuss all metrics, including tradeoffs, for each ablation variant.
minor comments (5)
  1. [Sec. 4.1] Please specify the random seed used to sample the 50 test videos and release the exact evaluation split, so that the numbers can be reproduced and compared by other groups.
  2. [Table 1 and Supplementary Table 3] The main-table comparison includes AniPortrait and FYE, which were trained on larger datasets, but this is only disclosed in the supplementary material. Adding a note in the main table caption would prevent the numbers from being read as a fair comparison with methods trained on the same data.
  3. [Figures 7 and 8] The captions contain the typo 'Visulization'; please change to 'Visualization'.
  4. [Throughout] The spacing in 'V oxCeleb1' is inconsistent and should be 'VoxCeleb1'.
  5. [Sec. 3.2.1] The notation N i_m = i × K/N appears confusing; using an upper index, e.g., N_m^i, would make the scale dependence clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the codebooks are learned with VQ reconstruction losses and evaluated on held-out test videos; self-cited baselines are comparisons, not load-bearing evidence.

full rationale

The paper's central derivation is self-contained rather than circular. The motion codebook is learned by quantizing encoded motion flows with a VQ loss (Eq. 3) and the appearance codebook by quantizing encoded driving features with a code-level loss (Eq. 5); neither loss is defined in terms of the evaluation metrics (PSNR, LPIPS, FID, AKD, AED, ARD, CSIM). The final training objective (Sec. 3.3) combines these codebook losses with equivariance, keypoint distance, reconstruction, and adversarial losses, and the model is tested on videos randomly sampled from held-out test splits (Sec. 4.1). The ablation study compares the full model against a baseline without codebooks and against partial variants, so the reported gains are empirically grounded rather than forced by construction. The authors' own prior works (DaGAN [16], MCNet [15]) appear only as comparison baselines and are not used to justify the method's premises. The acknowledged limitation in Sec. 8 (appearance leakage) is a real empirical weakness but not a circularity. The concern that the claimed superiority is undercut by the unseeded 50-video evaluation and by the supplementary cross-identity results in Table 3 is a matter of evidence strength and internal consistency, not of the derivation reducing to its own inputs.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The paper's contribution is an architectural recipe: multi-scale motion and appearance codebooks, a code-allocation scheme, transformer-based retrieval, and VQ losses. None of the design choices are derived from first principles; they are validated empirically. The free parameters listed are the main hand-chosen hyperparameters. The axioms are the standard deep-learning assumptions plus the domain assumptions specific to keypoint-based talking-head animation.

free parameters (7)
  • codebook sizes K and T = 1024
    Number of motion and appearance codes; chosen by hand; ablation shows 256 and 512 variants (Tab. 8).
  • number of scales N = 4
    Multi-scale compensation depth; set in supplementary without ablation.
  • loss weight lambda_recon_m = 32
    Weight for motion flow reconstruction in the VQ loss (Eq. 3); chosen manually.
  • loss weight beta = 0.25
    Commitment loss weight in code-level losses (Eqs. 3 and 5).
  • loss weights lambda_adv and lambda_1 = 0.8 and 0.5
    Adversarial and low-resolution reconstruction weights in the overall objective (Sec. 3.3).
  • latent dimensions dm and da = 32 and 256
    Dimensionality of motion and appearance code vectors; set in supplementary.
  • learning rate and training iterations = 8e-5, 250K
    Optimization schedule; standard choices, no sensitivity study.
assumptions (6)
  • domain assumption A keypoint-based motion flow estimator (FOMM) provides an adequate initial coarse motion flow for the compensation pipeline.
    Used as base in Sec. 3.3; the paper acknowledges this estimator entangles shape and motion (Sec. 8), which limits cross-identity appearance preservation.
  • standard math Nearest-neighbor vector quantization with straight-through gradient is a valid training signal for both codebooks.
    Adopted from VQ-VAE [26]; standard practice, not justified in this paper.
  • ad hoc to paper Stop-gradient on the input motion flow prevents codebook learning from interfering with the flow estimator (Sec. 3.2.2).
    Design choice with no theoretical or empirical analysis of interference.
  • ad hoc to paper A code-level loss alone (without reconstruction through the image decoder) is sufficient to learn a useful appearance codebook.
    Stated in Sec. 3.2.2; supported only by indirect ablation.
  • ad hoc to paper The code allocation scheme (first i groups used at scale i) is beneficial.
    Alternative schemes tested in Tab. 6, but no analysis of why sharing coarse codes works.
  • domain assumption Transformers with convolution layers can retrieve complementary information from codebooks via cross-attention.
    Core architectural assumption; no analysis of retrieval quality or failure cases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation." pith.science (2026). https://pith.science/paper/DT4UBEQS

@misc{pith2026241200719,
  author       = {Pith},
  title        = {Pith review of: Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DT4UBEQS}},
  note         = {Machine review of arXiv:2412.00719}
}
read the original abstract

Talking head video generation aims to generate a realistic talking head video that preserves the person's identity from a source image and the motion from a driving video. Despite the promising progress made in the field, it remains a challenging and critical problem to generate videos with accurate poses and fine-grained facial details simultaneously. Essentially, facial motion is often highly complex to model precisely, and the one-shot source face image cannot provide sufficient appearance guidance during generation due to dynamic pose changes. To tackle the problem, we propose to jointly learn motion and appearance codebooks and perform multi-scale codebook compensation to effectively refine both the facial motion conditions and appearance features for talking face image decoding. Specifically, the designed multi-scale motion and appearance codebooks are learned simultaneously in a unified framework to store representative global facial motion flow and appearance patterns. Then, we present a novel multi-scale motion and appearance compensation module, which utilizes a transformer-based codebook retrieval strategy to query complementary information from the two codebooks for joint motion and appearance compensation. The entire process produces motion flows of greater flexibility and appearance features with fewer distortions across different scales, resulting in a high-quality talking head video generation framework. Extensive experiments on various benchmarks validate the effectiveness of our approach and demonstrate superior generation results from both qualitative and quantitative perspectives when compared to state-of-the-art competitors.

Figures

Figures reproduced from arXiv: 2412.00719 by the authors.

Figure 1
Figure 1. Effect of motion and appearance compensation with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the framework. For each scale, multi-scale motion and appearance codebook compensation consists of two sub [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Illustration of appearance codebook learning and com [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Illustration of the code allocation scheme. We take the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison with state-of-the-art methods for (a) same-identity reconstruction and (b) cross-identity reenactment on [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 10
Figure 10. Figure 10: Visualization of the motion flow compensation process. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 9
Figure 9. Figure 9: Qualitative ablation study on multi-scale motion and [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 12
Figure 12. Figure 12: User study results ranking the quality of videos gener [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparison with more state-of-the-art approaches for (a) same-identity reconstruction and (b) cross-identity reenact￾ment on VoxCeleb1 or examples from the corresponding papers or project pages for closed-source methods (i.e., OSFV [29], PECHead [9], and M…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 42 canonical work pages

  1. [1]

    A morphable model for the synthesis of 3d faces

    V olker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2 , pages 157–164. Association for Computing Machinery, 2023. 2

    work page 2023

  2. [2]

    Hyperreenact: One-shot reenactment via jointly learning to refine and retar- get faces

    Stella Bounareli, Christos Tzelepis, Vasileios Argyriou, Ioannis Patras, and Georgios Tzimiropoulos. Hyperreenact: One-shot reenactment via jointly learning to refine and retar- get faces. In ICCV, 2023. 3

    work page 2023

  3. [3]

    Learning weights for codebook in image classification and retrieval

    Hongping Cai, Fei Yan, and Krystian Mikolajczyk. Learning weights for codebook in image classification and retrieval. In CVPR, 2010. 3

    work page 2010

  4. [4]

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022. 3

    work page 2022

  5. [5]

    Visual categorization with bags of keypoints

    Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and C ´edric Bray. Visual categorization with bags of keypoints. In ECCV Workshop, 2004. 3

    work page 2004

  6. [6]

    Megaportraits: One-shot megapixel neural head avatars

    Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Alek- sei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In ACM MM, 2022. 2

    work page 2022

  7. [7]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR,

  8. [8]

    Learning an animatable detailed 3d face model from in-the- wild images

    Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the- wild images. ACM TOG, 40(4):1–13, 2021. 2

    work page 2021

Show all 50 references
  1. [9]

    High-fidelity and freely controllable talking head video generation

    Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu. High-fidelity and freely controllable talking head video generation. In CVPR, 2023. 2

    2023

  2. [10]

    Vqfr: Blind face restoration with vector-quantized dictionary and parallel de- coder

    Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. Vqfr: Blind face restoration with vector-quantized dictionary and parallel de- coder. In ECCV, 2022. 3

    2022

  3. [11]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control. arXiv preprint arXiv:2407.03168, 2024. 2

    2024 arXiv

  4. [12]

    Marionette: Few-shot face reenact- ment preserving identity of unseen targets

    Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. Marionette: Few-shot face reenact- ment preserving identity of unseen targets. In AAAI, 2020. 1, 2

    2020

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  6. [14]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In NeurIPS, 2017. 6, 1

    2017

  7. [15]

    Implicit identity representation conditioned memory compensation network for talking head video generation

    Fa-Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation network for talking head video generation. In ICCV, 2023. 3, 6, 7, 1

    2023

  8. [16]

    Depth-aware generative adversarial network for talking head video generation

    Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In CVPR, 2022. 1, 6, 7, 3

    2022

  9. [17]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 3

    2019

  10. [18]

    Analyzing and improving the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020. 3

    2020

  11. [19]

    Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation

    Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. In SIGGRAPH Asia, 2024. 6, 7, 1, 2

    2024

  12. [20]

    V oxceleb: a large-scale speaker identification dataset

    Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. V oxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612, 2017. 6, 1

    2017 arXiv

  13. [21]

    Robust one-shot face video re-enactment using hybrid latent spaces of stylegan2

    Trevine Oorloff and Yaser Yacoob. Robust one-shot face video re-enactment using hybrid latent spaces of stylegan2. In ICCV, 2023. 3

    2023

  14. [22]

    Li, and Shan Liu

    Yurui Ren, Ge Li, Yuanqi Chen, Thomas H. Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In ICCV, 2021. 2

    2021

  15. [23]

    Animating arbitrary objects via deep motion transfer

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In CVPR, 2019. 2, 3, 6

    2019

  16. [24]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In NeurIPS, 2019. 1, 2, 6, 7, 3

    2019

  17. [25]

    Learn- ing motion refinement for unsupervised face animation

    Jiale Tao, Shuhang Gu, Wen Li, and Lixin Duan. Learn- ing motion refinement for unsupervised face animation. In NeurIPS, 2024. 1, 2, 6, 7, 3

    2024

  18. [26]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017. 2, 3, 5

    2017

  19. [27]

    Continuous piecewise-affine based mo- tion model for image animation

    Hexiang Wang, Fengqi Liu, Qianyu Zhou, Ran Yi, Xin Tan, and Lizhuang Ma. Continuous piecewise-affine based mo- tion model for image animation. In AAAI, 2024. 1, 2

    2024

  20. [28]

    Lipformer: High- fidelity and generalizable talking face generation with a pre- learned facial codebook

    Jiayu Wang, Kang Zhao, Shiwei Zhang, Yingya Zhang, Yu- jun Shen, Deli Zhao, and Jingren Zhou. Lipformer: High- fidelity and generalizable talking face generation with a pre- learned facial codebook. In CVPR, 2023. 3

    2023

  21. [29]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. In CVPR, 2021. 2

    2021

  22. [30]

    Recovering realistic texture in image super-resolution by deep spatial feature transform

    Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR, 2018. 6

    2018

  23. [31]

    Latent image animator: Learning to animate im- ages via latent space navigation

    Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate im- ages via latent space navigation. In ICLR, 2022. 6, 7, 1, 3

    2022

  24. [32]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024. 6, 7, 1, 2 9

    2024 arXiv

  25. [33]

    Reenactgan: Learning to reenact faces via boundary transfer

    Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, and Chen Change Loy. Reenactgan: Learning to reenact faces via boundary transfer. In ECCV, 2018. 2

    2018

  26. [34]

    Vfhq: A high-quality dataset and benchmark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution. In CVPRW, 2022. 1

    2022

  27. [35]

    Mesh guided one-shot face reenactment using graph convo- lutional networks

    Guangming Yao, Yi Yuan, Tianjia Shao, and Kun Zhou. Mesh guided one-shot face reenactment using graph convo- lutional networks. In ACM MM, 2020. 2

    2020

  28. [36]

    Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan

    Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In ECCV,

  29. [37]

    Few-shot adversarial learning of realis- tic neural talking head models

    Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realis- tic neural talking head models. In ICCV, 2019. 1

    2019

  30. [38]

    Fast bi-layer neural synthesis of one- shot realistic head avatars

    Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of one- shot realistic head avatars. In ECCV, 2020. 2

    2020

  31. [39]

    Face animation with an attribute-guided diffusion model

    Bohan Zeng, Xuhui Liu, Sicheng Gao, Boyu Liu, Hong Li, Jianzhuang Liu, and Baochang Zhang. Face animation with an attribute-guided diffusion model. In CVPRW, 2023. 2

    2023

  32. [40]

    Metaportrait: Identity-preserving talking head genera- tion with fast personalized adaptation

    Bowen Zhang, Chenyang Qi, Pan Zhang, Bo Zhang, Hsiang- Tao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. Metaportrait: Identity-preserving talking head genera- tion with fast personalized adaptation. In CVPR, 2023. 1

    2023

  33. [41]

    Learning non-redundant codebooks for classifying complex objects

    Wei Zhang, Akshat Surve, Xiaoli Fern, and Thomas Diet- terich. Learning non-redundant codebooks for classifying complex objects. In ICML, 2009. 3

    2009

  34. [42]

    Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset. In CVPR, 2021. 1

    2021

  35. [43]

    Thin-plate spline motion model for image animation

    Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In CVPR, 2022. 1, 2

    2022

  36. [44]

    Sparse to dense motion transfer for face image animation

    Ruiqi Zhao, Tianyi Wu, and Guodong Guo. Sparse to dense motion transfer for face image animation. In ICCV Work- shops, 2021. 2

    2021

  37. [45]

    Towards robust blind face restora- tion with codebook lookup transformer

    Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restora- tion with codebook lookup transformer. In NeurIPS, 2022. 3

    2022

  38. [46]

    Celebv- hq: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv- hq: A large-scale video facial attributes dataset. In ECCV,

  39. [47]

    Face alignment in full pose range: A 3d total solution

    Xiangyu Zhu, Xiaoming Liu, Zhen Lei, and Stan Z Li. Face alignment in full pose range: A 3d total solution. IEEE TPAMI, 41(1):78–92, 2017. 2 10 Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation Supplementary Material

    2017

  40. [48]

    We employ the keypoint-based motion flow estimator from FOMM [24]

    Additional Implementation Details We perform multi-scale compensation acrossN = 4scales. We employ the keypoint-based motion flow estimator from FOMM [24]. The multi-scale motion flows are estimated at a size of 64 × 64. We use convolution layers to en- code the motion flows i...

  41. [49]

    Base- line*

    More Details on Experiments 7.1. Additional Details on the Compared Methods We evaluate the performance of the compared methods us- ing their released pre-trained models, and we present the training datasets used for each method in Tab. 3. All the GAN-based methods [15, 16, 24...

    1970

  42. [50]

    Limitation A limitation of our method is the appearance leakage prob- lem in cross-identity reenactment, where the face in the gen- erated video tends to have a shape similar to that of the driv- ing face rather than the source face. This issue arises from the keypoint-based m...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.