{"id":"8bc16a51-b7f3-49ac-9100-cc6fe8872859","arxiv_id":"2411.15436","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ConsistentAvatar aligns a Fourier high-frequency detail map through a diffusion model and uses it, with normals and emotion text, to condition talking-head avatar generation, reducing temporal and expression inconsistency.","lead":"This paper builds a talking-head avatar pipeline that first cleans up a high-frequency detail map between neighboring frames, then uses it to guide a diffusion model that renders the final face. It reports better temporal, 3D, and expression consistency than earlier avatar methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The temporal-consistency mechanism is asserted but not isolated: TSD is a per-frame high-frequency map, and the only direct temporal evidence is a qualitative optical-flow plot; a temporally-mismatched TSD ablation would settle whether the central claim holds.","rationale":"I read the paper in good faith. The framework is coherent: a coarse 3D proxy from INSTA drives a detail-alignment diffusion module, and the aligned high-frequency map is used as a condition in a second diffusion module along with normals and emotion embeddings. The ablation structure (w/o stage 2, w/o normal, w/o emotion) is reasonable, and the reported image-quality numbers are plausible. The point of agreement with the reader is that the role of TSD, especially the cutoff w=10, is under-validated. My sharper concern is that the paper does not actually demonstrate a temporal mechanism: TSD is a per-frame static map, TSDM aligns frames independently, and no experiment isolates temporal information from per-frame appearance improvement. Fig. 7 is qualitative, and the quantitative tables do not include a dedicated temporal-consistency metric, despite temporal consistency being the headline contribution. This is not an accusation of dishonesty; it is a request for an experiment that separates the temporal channel from the appearance channel. The proposed temporally-mismatched TSD test is feasible, uses the paper's own components, and would decisively show whether the aligned TSD encodes temporal patterns or only improves per-frame fidelity. Because the reader already set a conditional verdict, I keep that verdict: the paper should not be unconditionally accepted until the TSD mechanism is isolated, the w=10 choice is sensitivity-checked, and the temporal metric is reported quantitatively.","tokens_in":13535,"tokens_out":6195,"duration_ms":61014,"concrete_test":"Train or evaluate the same FCSD with three TSD conditions while holding normal and emotion conditions fixed: (a) aligned TSD from the correct frame i, (b) aligned TSD from a randomly chosen frame j of the same video (so the TSD is temporally mismatched but visually similar), and (c) no TSD condition. Measure both per-frame fidelity (PSNR/LPIPS to ground truth) and a temporal metric that does not trivially track per-frame fidelity, such as average warping error or LPIPS between adjacent generated frames. If condition (a) and condition (b) give the same temporal-metric values while per-frame fidelity also matches, then the TSD is not conveying temporal information and the central mechanism collapses; if (a) clearly beats (b), the temporal-guidance claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that the aligned TSD 'represents the temporal patterns' and constrains the diffusion process to generate temporally stable talking heads (Abstract, Sec. 1). The load-bearing premise is that TSD carries temporally specific information beyond per-frame appearance. However, the TSD defined in Eq. 2-3 is a per-frame high-frequency/contour map, and the TSDM in Sec. 4.1 aligns each frame independently using only the coarse TSD, pose, and expression; there is no recurrence, no adjacent-frame input, and no video-level loss. Any temporal consistency must therefore emerge from reducing per-frame conditioning error, not from an explicit temporal model. The paper's only direct temporal evidence is the qualitative optical-flow comparison in Fig. 7; Tab. 2 reports pose and expression errors, which are per-frame reconstruction metrics and can improve even without temporal modeling. Thus the phrase 'represents the temporal patterns' is asserted rather than demonstrated. Additionally, Eq. 2 is not a well-defined filter specification: it writes a one-dimensional Fourier integral over a time variable t, defines W only as 'the frequency set of the image,' and then fixes w=10 with no units or 2D masking rule, so the TSD construction itself is not reproducible from the text. If the observed temporal benefit is only the result of denoising high frequencies of the coarse proxy, the central novelty reduces to image-conditional diffusion with no temporal component.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces ConsistentAvatar, a diffusion-based framework for talking-head avatar generation from monocular RGB video. The method first obtains coarse RGB and normal outputs from INSTA, then defines a Temporally-Sensitive Detail (TSD) map by high-pass Fourier filtering. A temporal consistency diffusion module (TSDM) is trained to align the coarse TSD to the TSD of the ground-truth frame. A fully consistent diffusion module (FCSD) then generates the final avatar conditioned on the aligned TSD, INSTA normal, and an emotion text embedding obtained by matching the target expression to MEAD via DECA. Experiments on the INSTA, PointAvatar, and NeRFace datasets show improved L2, PSNR, SSIM, LPIPS, pose error, and expression error over several baselines, and a qualitative optical-flow comparison indicates improved temporal stability. The paper also uses LCM and an SDXL-style refiner to reduce inference to about 2.4 seconds. The central claim is that aligning TSD, which 'represents the temporal patterns,' constrains the diffusion process to produce temporally stable and fully consistent avatars.","tokens_in":13855,"tokens_out":6433,"duration_ms":53188,"significance":"If the temporal-alignment mechanism is real, the TSD representation is a useful idea for improving consistency in diffusion-based talking-head generation. The paper provides a staged design that is easy to follow and reports consistent gains over multiple metrics and datasets, along with ablations of the TSD, normal, and emotion conditions. The computational efficiency gain from LCM is a positive aspect. However, the central temporal-consistency claim is not supported by a quantitative temporal metric or an ablation that isolates temporal ordering, and the emotion-labeling protocol has a potential leakage that may inflate the reported expression-consistency improvement. These issues need to be resolved before the claims can be accepted.","major_comments":[{"comment":"The definition of the Temporally-Sensitive Detail (TSD) map is not reproducible from the text. Equation (2) writes a one-dimensional Fourier integral with respect to a continuous variable t for an image I_rgb_i, defines W only as 'the frequency set of the image,' and then fixes w=10 without specifying units or the two-dimensional masking procedure. Since TSD is the central contribution and the input to both diffusion modules, please provide the exact 2D filtering algorithm (e.g., FFT radius, band-pass mask, or high-pass threshold) and a sensitivity analysis for the cutoff w=10.","section":"Sec. 4.1, Eq. (2)-(3)"},{"comment":"The temporal-consistency mechanism is asserted but not isolated. The TSDM (Eqs. 4-6) aligns each frame independently using only per-frame TSD, pose, and expression; there is no recurrence, no adjacent-frame input, and no video-level loss. Therefore any temporal stability improvement could stem from better per-frame reconstruction rather than from modeling temporal patterns. The only temporal evidence is the qualitative optical-flow plot in Fig. 7, with no quantitative numbers. Please add a quantitative temporal-consistency metric and an ablation that feeds temporally mismatched TSD (e.g., TSD from a different frame) to show that the temporal alignment itself, not just the aligned TSD's per-frame accuracy, is responsible for the improvement.","section":"Sec. 4.1, Sec. 5.2 (Fig. 7)"},{"comment":"The emotion-labeling protocol introduces a circularity in the expression-consistency evaluation. Equation (7) assigns an emotion label by matching the DECA expression vector of the target frame I_i against MEAD, and the Expression Error (EE) metric in Sec. 5.2 also uses DECA to compare generated expression coefficients to the same ground-truth target. Thus, at test time, the conditioning contains information directly derived from the target expression, which can inflate EE results independently of the generative model. Please evaluate expression consistency with emotion labels that are not computed from the target frame (e.g., from audio, manual annotation, or a separate emotion-conditioning experiment), and report whether the EE gain persists.","section":"Sec. 4.2 (Eq. 7) and Sec. 5.2 (Expression Error)"},{"comment":"The evaluation protocol uses a per-identity split: the last 350 frames of each video are held out for testing, and the rest are used for training. Consequently, the method is only tested on identities seen during training, and the claim of 'fully consistent talking head avatar' is not evaluated for generalization to unseen identities. If the method is intended as a per-identity personalization system, please state that explicitly; if the title's generality is intended, add a cross-identity evaluation. Also, the optical-flow quantitative comparison promised in the Fig. 1 caption does not appear in the paper; the numbers should be reported.","section":"Sec. 5.1 (Evaluation Protocol) and Sec. 5.2 (Temporal consistency)"}],"minor_comments":[{"comment":"The abstract and Sec. 1 claim 'fully consistent' avatars, but Sec. 6 lists teeth and eyeball inaccuracies; consider using 'improved consistency' or 'consistent' with caveats.","section":"Sec. 6"},{"comment":"The text uses 'STD' once near the emotion-condition description; this should read 'TSD.'","section":"Sec. 4.2"},{"comment":"The dataset description says 'a resolution of 5122'; this should be '512×512.'","section":"Sec. 5.1"},{"comment":"The sentence about reducing inference time is vague; it should explicitly compare the 'w/o LCM' row (8.20s) with the 'Ours' row (2.40s) to attribute the gain to LCM.","section":"Tab. 1 and Sec. 5.2"},{"comment":"Given the small number of training videos (10 for INSTA), please report per-sequence results or error bars across repeated runs to support the quantitative claims.","section":"Tab. 1 and Tab. 2"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the emotion-label leakage in Eq. (7); it directly affects the reported expression-consistency gains and needs to be addressed before the paper can be considered for acceptance. The TSD definition also needs to be made fully reproducible. If the authors provide code or a detailed appendix for the TSD extraction and the temporal-mismatch ablation, the revisions would be straightforward to verify."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read ConsistentAvatar. The genuinely new piece is the TSD representation and the two-stage align-then-generate pipeline: extract a high-frequency detail map from the INSTA proxy, align it to the target frame's ground-truth detail map with a small diffusion module, then feed the aligned map plus normal and emotion embeddings into a ControlNet-style stable diffusion. That combination is absent from the prior work they cite, and the ablations support each component having a real effect. The comparative numbers across the INSTA, PointAvatar, and NeRFace datasets are consistently better than the stated baselines.\n\nThe soft spots are real but not disqualifying. The title and abstract say \"fully consistent,\" but the paper's own limitation section admits teeth and eyeballs are not modeled accurately. That overclaim should be fixed. More substantively, I agree with your concern that the temporal mechanism is asserted rather than isolated. TSD is a per-frame high-pass map; the TSDM aligns each frame independently with no adjacent-frame input or video-level loss. So any temporal stability must emerge indirectly from better per-frame conditioning, and the only direct temporal evidence is the qualitative optical-flow plot in Fig. 7. A clean test would be to feed a temporally mismatched TSD (e.g., from a neighboring frame) and show the output degrades; the current \"w/o stage2\" baseline is a start but does not isolate the temporal role. The Fourier definition in Eq. 2 is also informal: a 1D integral over a time variable with an unspecified frequency set, then w=10 with no units. Reproducibility suffers, and no code is released.\n\nOne protocol point: the emotion label for each frame is assigned by matching its expression vector to the nearest MEAD frame, and the same expression coefficients are used to compute Expression Error. That is a form of label leakage that favors your own pipeline, and the paper doesn't acknowledge it. It's not fatal to the main appearance results, but it weakens the expression-consistency comparison.\n\nOverall: this is an incremental but useful contribution for diffusion-based talking head generation. It deserves a serious referee — the core idea is credible and the experiments are honestly reported in most places — but the review should require a better temporal isolation experiment, a formal TSD definition, and toned-down claims. I would send it to review rather than desk reject, and I'd cite it as related work if the authors fix the reproducibility points.","headline":"A plausible incremental diffusion-based talking-head method with a genuinely new TSD detail-alignment idea, but the temporal-consistency claim is under-supported and the evaluation has leakage and overclaim issues.","tokens_in":14342,"tokens_out":2693,"would_cite":true,"duration_ms":24392,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConsistentAvatar claims that aligning a high-frequency Fourier detail map from a coarse 3D proxy to real video frames makes diffusion-based talking-head generation consistent in time, pose, and expression, and reports state-of-the-art…","keywords":["talking head generation","diffusion models","temporal consistency","3D consistency","expression consistency","temporally-sensitive detail map","Fourier high-frequency filtering","neural avatars"],"falsifier":"Run the full pipeline with several Fourier cutoffs (for instance, $w=5$, $10$, $20$, and a learned high-pass) on the same videos and measure adjacent-frame optical-flow error and DECA pose/expression error; if $w=10$ is not clearly the best or if the improvement over using unaligned TSD disappears for some cutoff, the claim that TSD defined by this cutoff carries the temporal pattern is not supported. A second decisive check is to evaluate on an unseen identity with large pose and expression changes: if temporal consistency degrades there, the alignment does not generalize beyond training conditions.","tokens_in":13359,"feed_emoji":"🗣️","tokens_out":6115,"duration_ms":52840,"temperature":0.7,"pith_summary":"The paper claims that diffusion-based talking-head avatar generation can be made temporally, 3D, and expression consistent by first learning to align a Fourier high-frequency detail representation between a coarse 3D proxy render and the real video frames. The aligned representation, called the Temporally-Sensitive Detail (TSD) map, then serves as a conditioning signal for a second diffusion model that produces the final portrait. The authors argue that existing diffusion approaches suffer from jitter and drift because per-frame conditions are inaccurate and the generator has no temporal model; the aligned TSD supplies the missing temporal pattern and suppresses error accumulation. If this holds, it offers a way to stabilize high-fidelity talking-head video without training a full video diffusion model.","feed_headline":"One high-frequency map makes talking avatars fully consistent","feed_subtitle":"Aligning a Fourier detail map of a coarse 3D proxy to real frames gives diffusion models the temporal stability they lack.","key_machinery":"The central object is the Temporally-Sensitive Detail (TSD) map: a per-frame image formed by Fourier transforming the coarse RGB output, suppressing all frequencies below $w=10$, and inverse-transforming, so that it retains contours, expression edges, and high-frequency details that shift between adjacent frames. The argument is carried by two diffusion stages: the temporal consistency diffusion module (TSDM), which uses extra IP-Adapter-style cross-attention layers to align the coarse TSD to the ground-truth TSD and thereby learn the temporal pattern; and the fully consistent diffusion module, which uses ControlNet to condition the final generation on the aligned TSD, the coarse head normal, and an emotion embedding chosen by nearest-neighbor matching of DECA expression vectors against MEAD. The aligned TSD is the load-bearing guidance; the normal and emotion conditions add 3D and expression control once temporal stability is in place.","core_discovery":"On its own terms, the paper establishes the following: a temporally-sensitive detail map, obtained by taking the high-frequency Fourier component (cutoff $w=10$) of a coarse RGB render from INSTA and of the ground-truth frame, can be aligned to real-frame detail through a small diffusion module, and that aligned map, when fed as a condition alongside head normals and a CLIP emotion embedding into a ControlNet-based diffusion renderer, yields avatars that are closer to ground truth and substantially more stable across time, pose, and expression than prior methods. The authors' core claim is that the aligned TSD 'represents the temporal patterns' and constrains the diffusion process to generate temporally stable talking heads, with this reliable guidance compensating for inaccuracies in the other conditions. Quantitative comparisons on three datasets report lower L2, higher PSNR/SSIM, lower LPIPS, lower pose error, and lower expression error than the compared baselines.","pith_inferences":["A natural extension is to test how sensitive the result is to the Fourier cutoff $w=10$; the paper reports no ablation on $w$, so a sweep over cutoffs would clarify whether the benefit comes from the specific band or from any high-frequency alignment.","The same two-stage recipe—align a cheap proxy's high-frequency residual to the target, then condition the generator on it—could transfer to other conditional generation tasks where the cheap condition is accurate in structure but noisy in detail, such as pose-conditioned human video or depth-conditioned scene rendering.","Because emotion labels are assigned by cosine similarity of DECA expression vectors, the method inherits DECA's expression ambiguities; an end-to-end learned emotion encoder or multiple hypothesis labels might sharpen expression control further.","The reported temporal metric is optical-flow magnitude between adjacent frames; complementing it with point-tracking or learned video-quality metrics would test whether the stability holds beyond flow-specific artifacts."],"forward_implications":["Generated talking-head videos should show adjacent-frame optical flow comparable to real video, rather than the high jitter seen in DiffusionRig and the baseline without aligned TSD.","Pose error (from DECA-estimated coefficients) and expression error should remain low across different viewpoints and expressions, not just on the training identities.","The method retains high image quality while running in about 10 denoising steps thanks to LCM, making it practical for interactive avatar generation.","Each condition plays a distinct role: removing aligned TSD hurts temporal and expression consistency, removing the normal condition hurts 3D consistency, and removing the emotion embedding leaves expressions less accurate."],"supporting_citations":[{"why":"INSTA supplies the coarse RGB and normal proxy whose TSD is extracted and aligned; the whole pipeline begins from its output.","marker":"[49]"},{"why":"DiffusionRig motivates the cross-attention conditioning design and serves as the main baseline that fails on temporal consistency.","marker":"[6]"},{"why":"The pretrained latent diffusion model is the backbone for both TSDM and the final portrait generator.","marker":"[33]"},{"why":"ControlNet integrates the normal and TSD conditions into the fully consistent diffusion module.","marker":"[44]"},{"why":"IP-Adapter-style additional cross-attention layers inject the TSD latent and pose/expression codes in the temporal module.","marker":"[43]"},{"why":"LCM is used to reduce denoising to about ten steps for the reported runtimes.","marker":"[23]"},{"why":"MEAD provides the emotion-labeled clips whose expression vectors form the database for emotion label assignment.","marker":"[40]"},{"why":"DECA computes expression vectors for emotion labeling and is used to estimate pose and expression error in evaluation.","marker":"[7]"}],"fun_headline_variants":["Fourier detail map aligns diffusion for consistent talking heads","Aligned high-frequency map stabilizes talking-head generation","Detail map alignment yields temporally consistent talking avatars","Aligning Fourier detail maps gives diffusion temporal stability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the high-frequency Fourier band at cutoff $w=10$ captures exactly the details that change between frames; if that band omits or distorts the relevant contours and expression edges, the alignment mechanism cannot stabilize generation.","fun_headline_variants_meta":{"raw":{"variants":["Fourier detail map aligns diffusion for consistent talking heads","Aligned high-frequency map stabilizes talking-head generation","Detail map alignment yields temporally consistent talking avatars","Aligning Fourier detail maps gives diffusion temporal stability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000462,"raw_usage":{"total_tokens":2333,"prompt_tokens":991,"completion_tokens":1342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":1279}},"tokens_in":607,"tokens_out":1342,"duration_ms":9946,"temperature":1.0,"reasoning_tokens":1279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:17:44.582840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline with several Fourier cutoffs (for instance, $w=5$, $10$, $20$, and a learned high-pass) on the same videos and measure adjacent-frame optical-flow error and DECA pose/expression error; if $w=10$ is not clearly the best or if the improvement over using unaligned TSD disappears for some cutoff, the claim that TSD defined by this cutoff carries the temporal pattern is not supported. A second decisive check is to evaluate on an unseen identity with large pose and expression changes: if temporal consistency degrades there, the alignment does not generalize beyond training conditions.","supporting_citations":[{"cited_title":"Bühler, Xu Chen, Michael J","cited_arxiv_id":null,"evidence_quote":"INSTA supplies the coarse RGB and normal proxy whose TSD is extracted and aligned; the whole pipeline begins from its output."},{"cited_title":"DiffusionRig: Learning Personalized Priors for Facial Appearance Editing","cited_arxiv_id":"2304.06711","evidence_quote":"DiffusionRig motivates the cross-attention conditioning design and serves as the main baseline that fails on temporal consistency."},{"cited_title":"GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraits","cited_arxiv_id":"2312.07669","evidence_quote":"IP-Adapter-style additional cross-attention layers inject the TSD latent and pose/expression codes in the temporal module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MEAD provides the emotion-labeled clips whose expression vectors form the database for emotion label assignment."},{"cited_title":"Learning an Animatable Detailed 3D Face Model from In-The-Wild Images","cited_arxiv_id":"2012.04012","evidence_quote":"DECA computes expression vectors for emotion labeling and is used to estimate pose and expression error in evaluation."}],"review_version":1}