{"id":"91b19062-b895-449b-a2fa-29d9192df9bb","arxiv_id":"2412.00719","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A talking-head video generation method that jointly learns multi-scale motion and appearance codebooks and compensates warped features with transformer-based code retrieval, showing improved reconstruction quality on VoxCeleb1 and competitive results on CelebV-HQ.","lead":"This paper trains two sets of learned codebooks, one for facial motion and one for appearance, and uses a transformer to retrieve complementary patterns that refine the warped features in a talking-head generator. If the results hold, it offers a practical quality boost for animating a single portrait photo with a driving video, with lower compute than many recent diffusion baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed superiority is undercut by the paper's own supplementary Table 3, where LIA beats Ours on FID and CSIM and MRFA beats Ours on ARD in cross-identity reenactment, and by the unseeded 50-video evaluation protocol.","rationale":"The reader's weakest_assumption correctly identifies the unseeded 50-video evaluation without significance tests as a key vulnerability. My independent reading reinforces this and finds an additional, more concrete internal inconsistency: the supplementary Table 3 shows the method is not the best on any cross-identity reenactment metric, which directly contradicts the strongest_claim's implied general superiority. This is not merely about external robustness or noise; it is a disagreement between the paper's stated conclusion and its own reported numbers. The method has genuine strengths: the multi-scale codebook design is coherent, the ablations are informative, and the parameter count and speed are attractive. No formal verification exists, but that is not a load-bearing concern for this type of systems paper. The correct disposition is CONDITIONAL: the central SOTA claim should be either substantiated with error bars and a fair cross-identity baseline, or explicitly narrowed to same-identity reconstruction. I therefore agree with the reader's verdict and recommend no change to the CONDITIONAL outcome, while noting that the cross-identity contradiction gives the condition more urgency.","tokens_in":16599,"tokens_out":2752,"duration_ms":22775,"concrete_test":"Run the cross-identity reenactment comparison of Table 3 on the same 10 source-driving pairs with a seeded protocol and report mean±std for FID, CSIM, and ARD. If LIA remains better on FID and CSIM while MRFA remains better on ARD, then the paper's claim of generally highest overall performance in cross-identity reenactment must be retracted or significantly softened. Additionally, recompute VoxCeleb1 same-identity results with 5 independent 50-video samples to check whether the small FID/PSNR/LPIPS gaps versus MRFA and FYE persist.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that the method outperforms state-of-the-art 'almost on all metrics on VoxCeleb1 and remains competitive on CelebV-HQ' is load-bearing but not fully supported by the paper's own numbers. First, the evaluation protocol in Sec. 4.1 uses 50 randomly sampled videos per dataset with no seed, no standard deviations, and no significance tests; many same-identity gaps in Table 1 are small (e.g., VoxCeleb1 PSNR 25.30 vs MRFA 25.26, FID 43.15 vs FYE 43.25, LPIPS 0.1846 vs 0.1872), so the claimed advantage could be noise. Second, and more concretely, the supplementary cross-identity reenactment results in Table 3 show that LIA outperforms Ours on FID (72.55 vs 76.47) and CSIM (0.6505 vs 0.6142), while MRFA outperforms Ours on ARD (1.5903 vs 1.6234). The paper acknowledges these individual weaknesses in the text but still concludes 'our method generally demonstrates the highest overall performance.' That conclusion is internally inconsistent with the presented table. The strongest_claim in Sec. 4.2 is therefore only conditionally true: it holds for same-identity reconstruction on VoxCeleb1, but the cross-identity superiority claim is contradicted by the supplementary results. The method itself is well-structured, with informative ablations and a plausible codebook design, so the issue is not soundness of the architecture but the mismatch between the stated claim and the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a talking-head video generation framework that jointly learns multi-scale motion and appearance codebooks and uses transformer-based compensation modules to refine motion flows and warped source features across scales. The motion codebook stores local motion patterns and the appearance codebook stores facial texture patterns; a code-allocation scheme shares coarse codes across scales and adds finer codes at larger scales. Experiments on VoxCeleb1 and CelebV-HQ compare the method with GAN-based and diffusion-based baselines, reporting strong same-identity reconstruction numbers and competitive cross-identity reenactment, with ablations on the codebooks, code allocation, codebook size, and model design.","tokens_in":17051,"tokens_out":7095,"duration_ms":63742,"significance":"If the performance claims are sustained, the paper makes a useful contribution: the joint learning of motion and appearance codebooks with multi-scale compensation is a well-motivated and clearly described design, and the ablation study supports the role of each component. The authors also provide a user study, an inference-speed comparison, and an honest limitation statement about appearance leakage. The main value would be a lightweight, efficient alternative to diffusion-based talking-head generators. However, the significance is currently bounded by evaluation weaknesses: the central state-of-the-art claim rests on a single unseeded 50-video test sample, small metric margins, and a cross-identity table whose numbers are partly in tension with the text.","major_comments":[{"comment":"The evaluation protocol uses only 50 randomly sampled videos per dataset with no seed, no repeated sampling, and no standard deviations or significance tests. Several of the margins that support the main claim are very small, e.g., VoxCeleb1 PSNR 25.30 vs. 25.26 for MRFA, LPIPS 0.1846 vs. 0.1872, and FID 43.15 vs. 43.25 for FYE. Without error bars or a significance test, these differences are within plausible noise, so the statement in Sec. 4.2 that the method 'outperforms the other methods almost on all metrics' is not yet established at the level of a state-of-the-art claim. Please report mean and standard deviation over multiple samples/seeds and, where feasible, a paired significance test.","section":"Sec. 4.1, Table 1"},{"comment":"The conclusion that the method 'generally demonstrates the highest overall performance' in cross-identity reenactment is not directly supported by the table: LIA outperforms Ours on FID (72.55 vs. 76.47) and CSIM (0.6505 vs. 0.6142), and MRFA outperforms Ours on ARD (1.5903 vs. 1.6234). The text acknowledges these individual cases but does not provide an explicit aggregate criterion, such as mean rank or a combined score, that would justify the overall claim. Either define and apply such a criterion or temper the conclusion to match the reported numbers.","section":"Supplementary Table 3 and Sec. 7.2.2"},{"comment":"In the ablation table, the 'Baseline + SMC' row shows an FID regression from 47.83 to 49.00 and an AED regression from 0.1106 to 0.1167, yet the text states that single-scale motion codebook compensation 'already improves motion transfer and image quality' and lists only PSNR, L1, LPIPS, and AKD. The unaddressed FID and AED regressions make the component-wise claim appear selective. Please report and discuss all metrics, including tradeoffs, for each ablation variant.","section":"Sec. 4.3, Table 2"}],"minor_comments":[{"comment":"Please specify the random seed used to sample the 50 test videos and release the exact evaluation split, so that the numbers can be reproduced and compared by other groups.","section":"Sec. 4.1"},{"comment":"The main-table comparison includes AniPortrait and FYE, which were trained on larger datasets, but this is only disclosed in the supplementary material. Adding a note in the main table caption would prevent the numbers from being read as a fair comparison with methods trained on the same data.","section":"Table 1 and Supplementary Table 3"},{"comment":"The captions contain the typo 'Visulization'; please change to 'Visualization'.","section":"Figures 7 and 8"},{"comment":"The spacing in 'V oxCeleb1' is inconsistent and should be 'VoxCeleb1'.","section":"Throughout"},{"comment":"The notation N i_m = i × K/N appears confusing; using an upper index, e.g., N_m^i, would make the scale dependence clearer.","section":"Sec. 3.2.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the architecture is described in enough detail to be reproducible, but the current evaluation does not fully support the strong state-of-the-art claim. The cross-identity table in the supplementary material is the most concrete mismatch between text and evidence. I recommend major revision with emphasis on repeated-seed evaluation, explicit aggregation of cross-identity metrics, and a more careful discussion of ablation tradeoffs. I do not see grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the architecture is genuinely worth a look, but the claims need to be scaled back. The joint multi-scale motion and appearance codebooks with code allocation and transformer-based compensation are a real combination, and the ablations do a good job showing each component earns its keep. Same-identity reconstruction on VoxCeleb1 is competitive, and the efficiency numbers (4.3M params, ~15 FPS on a 3090) make it practically interesting.\n\nThe soft spots are mostly around evaluation and framing. The protocol uses one unseeded sample of 50 videos per dataset, no standard deviations, no significance tests. Several claimed wins on VoxCeleb1 are within noise (PSNR 25.30 vs MRFA 25.26; FID 43.15 vs FYE 43.25). More concretely, the paper's own supplementary Table 3 for cross-identity reenactment shows LIA beating it on FID and CSIM, and MRFA beating it on ARD, which directly contradicts the main text's claim of being \"almost on all metrics\" superior. The authors acknowledge these individual weaknesses but still conclude \"highest overall performance\"—that's internally inconsistent.\n\nThere's also a numbers issue: Table 7 lists the method as 82.2M parameters, while Table 4 says 4.3M. That needs reconciling. The limitation section honestly admits appearance leakage in cross-identity settings, but the issue is not quantified, and the supplementary cross-identity numbers suggest it's not fully resolved.\n\nWhat's genuinely good: the code allocation scheme is interesting and is ablated against alternatives; the joint training of motion and appearance codebooks is clearly described; and the model is efficient. No code is released, but the project page has video results.\n\nWho is this for? Someone working on non-diffusion talking-head generation or codebook-based video synthesis will get value from the architecture and ablations. It's not a field reorg. The claims need to be scaled back to match the evidence, and the eval should be re-run with multiple seeds or at least error bars.\n\nRecommendation: send it to peer review, with a clear request to fix the evaluation protocol, reconcile Tables 3, 4, and 7, and rewrite the SOTA claims. Conditional accept, not reject.","headline":"A well-ablated codebook-based talking-head generator with a solid same-identity story, undermined by an unseeded 50-video eval and a cross-identity claim that its own supplementary table contradicts.","tokens_in":17491,"tokens_out":2911,"would_cite":false,"duration_ms":26853,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jointly learned motion and appearance codebooks, retrieved by a transformer at multiple scales, are claimed to yield state-of-the-art talking-head videos with only 4.3M parameters.","keywords":["talking head video generation","multi-scale motion codebook","multi-scale appearance codebook","codebook compensation","transformer retrieval","motion flow refinement","appearance compensation","one-shot face reenactment"],"falsifier":"Re-run the same evaluation on multiple independently drawn 50-video samples from VoxCeleb1 and CelebV-HQ with the released checkpoints, reporting means and standard deviations; if the FID, LPIPS, or AKD gaps over MRFA and LIA fall within one standard deviation or flip sign across samples, the claimed state-of-the-art advantage is not established.","tokens_in":16407,"feed_emoji":"🎥","tokens_out":6459,"duration_ms":53372,"temperature":0.7,"pith_summary":"The paper tries to show that the two bottlenecks in talking-head video generation, inaccurate motion flow and incomplete appearance information from a single source image, can be addressed together by learning two dictionaries, or codebooks, from the training data. One codebook stores recurring local motion patterns; the other stores recurring facial texture and structure patterns. At each of several scales, a transformer retrieves entries from both codebooks to refine the estimated motion flow and to repair the source features warped by that flow. The paper reports that this joint compensation improves reconstruction quality, motion transfer, and identity preservation on VoxCeleb1, and stays competitive on the harder CelebV-HQ test set, using a 4.3M-parameter model running at about 15 FPS on an RTX 3090.","feed_headline":"Paired codebooks beat previous methods for talking-head videos","feed_subtitle":"Jointly learned motion and appearance dictionaries refine warped frames across scales, at 4.3M parameters and 15 FPS.","key_machinery":"The central object is a pair of multi-scale codebooks: a motion codebook $\\mathcal{C}_M = \\{m_k\\}_{k=1}^K$ of 32-dimensional local motion-flow patterns and an appearance codebook $\\mathcal{C}_A = \\{a_k\\}_{k=1}^T$ of 256-dimensional local appearance patterns. Codes are split into $N$ groups, and scale $i$ uses the first $i$ groups, so general patterns are shared while finer patterns are reserved for larger scales. A shared transformer $\\mathcal{T}_M$ for motion and a window-partitioned transformer $\\mathcal{T}_A$ for appearance retrieve codes by cross-attention: the motion transformer queries with the encoded current flow, warped source feature, and driving keypoint feature and outputs a residual flow; the appearance transformer queries with the warped source feature and outputs a compensated feature. The codebooks are updated with a VQ-VAE-style code-level loss while the rest of the network is trained with reconstruction, equivariance, keypoint-distance, and adversarial losses.","core_discovery":"The central claim is that motion and appearance should be compensated as a coupled process rather than separately. Starting from a coarse keypoint-based motion flow, the method refines the flow coarse-to-fine: at scale i a motion-codebook compensation module retrieves the nearest codes for the current flow, warped source feature, and driving keypoint feature, adds the decoded residual to the flow, and uses the refined flow to warp the source feature. An appearance-codebook compensation module then retrieves appearance codes to repair the warped feature before the image decoder fuses all scales. The two codebooks are learned jointly with the whole network, so gradients from appearance refinement shape the motion codes and vice versa. The paper argues that this coupling produces motion flows that are more flexible and appearance features with fewer distortions, and it reports quantitative gains over unsupervised and diffusion-based baselines on same-identity reconstruction and cross-identity reenactment.","pith_inferences":["An unstated implication is that the same two-codebook compensation idea could be applied to other image-animation tasks, such as bodies, hands, or general objects, where motion and appearance also interact; the codebooks would need to capture the corresponding motion and texture patterns.","The ablation suggests the code allocation scheme, sharing early codes across scales and reserving later codes for fine detail, is doing real work; a testable prediction is that reordering or reindexing code groups would change the quality-detail tradeoff.","The reported FID advantage over diffusion models on VoxCeleb1 is notable because those models were trained on larger datasets; if this pattern persists on broader benchmarks, it would suggest that learned motion and appearance priors can substitute for large-scale pre-training in this task.","The paper's own stated limitation, appearance leakage from the keypoint-based motion estimator, implies that replacing the keypoint backbone with a shape-agnostic motion estimator would be a direct test of how much of the remaining leakage is due to the codebook design versus the motion estimator."],"forward_implications":["On VoxCeleb1 same-identity reconstruction, the paper reports the best FID, PSNR, L1, LPIPS, AKD, and AED among the compared open-source methods, including diffusion-based methods trained on larger data.","Multi-scale motion codebook compensation alone improves motion transfer and image quality over the baseline, and adding multi-scale appearance compensation further improves FID, LPIPS, and AKD, with the largest gains when both codebooks are learned jointly.","The framework generalizes to CelebV-HQ without fine-tuning, staying competitive and achieving the best AKD among unsupervised methods there.","At 4.3M parameters and 352.91 GFLOPs, the model runs at 15.13 FPS on an RTX 3090, faster than the reported speeds of MRFA, AniPortrait, Follow-Your-Emoji, and LivePortrait.","Because motion and appearance compensation are coupled at every scale, the compensated multi-scale features can be fed into a standard VQGAN-style decoder, making the codebook mechanism a plug-in enhancement to that decoder design."],"supporting_citations":[{"why":"Supplies the keypoint-based motion flow estimator and the unsupervised training pipeline that the paper adopts for initial flow estimation and self-reconstruction.","marker":"[24]"},{"why":"Provides the motion flow updater used as the residual decoder in motion compensation, and serves as the main non-diffusion baseline for comparison.","marker":"[25]"},{"why":"Introduces the vector-quantization code-level loss used to learn both codebooks.","marker":"[26]"},{"why":"Supplies the image encoder/decoder architecture into which the compensated multi-scale features are fed.","marker":"[7]"},{"why":"Source of the Taylor-expansion approximation used to derive the initial motion flow and warping procedure.","marker":"[23]"},{"why":"The memory-compensation baseline closest in spirit; the paper compares against it and distinguishes its own joint motion and appearance codebooks.","marker":"[15]"}],"fun_headline_variants":["Coupled motion and appearance codebooks improve talking head videos","Joint codebook compensation sharpens talking-head motion and details","Multi-scale codebooks couple motion and appearance for better talking heads","Talking heads get finer details with joint motion-appearance codebooks","Coupled codebooks refine talking heads more effectively than separate ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of state-of-the-art quality rests on a comparison made on a single unseeded random sample of 50 test videos per dataset, with small metric gaps and no significance testing.","fun_headline_variants_meta":{"raw":{"variants":["Coupled motion and appearance codebooks improve talking head videos","Joint codebook compensation sharpens talking-head motion and details","Multi-scale codebooks couple motion and appearance for better talking heads","Talking heads get finer details with joint motion-appearance codebooks","Coupled codebooks refine talking heads more effectively than separate ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00088,"raw_usage":{"total_tokens":3816,"prompt_tokens":968,"completion_tokens":2848,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2763}},"tokens_in":584,"tokens_out":2848,"duration_ms":18973,"temperature":1.0,"reasoning_tokens":2763,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:04:26.816617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same evaluation on multiple independently drawn 50-video samples from VoxCeleb1 and CelebV-HQ with the released checkpoints, reporting means and standard deviations; if the FID, LPIPS, or AKD gaps over MRFA and LIA fall within one standard deviation or flip sign across samples, the claimed state-of-the-art advantage is not established.","supporting_citations":[{"cited_title":"First order motion model for image animation","cited_arxiv_id":null,"evidence_quote":"Supplies the keypoint-based motion flow estimator and the unsupervised training pipeline that the paper adopts for initial flow estimation and self-reconstruction."},{"cited_title":"Learn- ing motion refinement for unsupervised face animation","cited_arxiv_id":null,"evidence_quote":"Provides the motion flow updater used as the residual decoder in motion compensation, and serves as the main non-diffusion baseline for comparison."},{"cited_title":"Animating arbitrary objects via deep motion transfer","cited_arxiv_id":null,"evidence_quote":"Source of the Taylor-expansion approximation used to derive the initial motion flow and warping procedure."},{"cited_title":"Implicit identity representation conditioned memory compensation network for talking head video generation","cited_arxiv_id":null,"evidence_quote":"The memory-compensation baseline closest in spirit; the paper compares against it and distinguishes its own joint motion and appearance codebooks."}],"review_version":1}