{"id":"5edc11fc-49be-4b01-a79e-a8bf8f4f50a9","arxiv_id":"2507.12956","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A DiT-based portrait animation model transfers implicit facial expressions to one or more characters using a masked cross-attention mechanism, supported by a new multi-face dataset and benchmark.","lead":"FantasyPortrait animates static portraits using expression features pulled from a driving video, and extends this to scenes with several characters at once. It is a diffusion-transformer system, trained on a new video dataset, with a masked attention mechanism to keep each character's expressions independent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4's element-wise mask multiplication does not implement hard attention masking; masked tokens receive non-zero softmax mass, so the claimed 'preventing interference' mechanism is not supported by the stated math.","rationale":"Read in good faith, the paper is a solid empirical systems contribution: the ablation isolating masked cross-attention is dramatic, the new dataset and benchmark are useful, and the qualitative evidence is consistent with the claimed behavior. My concern is not with the plausibility of the system but with the formal description of the mechanism. Eq. 4 uses element-wise multiplication inside softmax, which is not equivalent to masking unless M is renormalizing in a way the paper never states. This is an internal correctness issue, not just a disagreement with community baselines, so it is more load-bearing than the absence of error bars or code release. The reader's weakest assumption (mask M must correctly localize each character's tokens) is related but does not identify this operation-level flaw; I therefore partially agree. The reader's CONDITIONAL verdict remains appropriate: the concern can be settled by a corrected equation, a code release, or the simple toy-softmax check, but until then the central 'prevents interference' claim is not pinned down.","tokens_in":12806,"tokens_out":6432,"duration_ms":73898,"concrete_test":"Run the exact Eq. 4 with a binary mask and the Wan2.1-14B backbone on a subset of ExprBench-Multi (or a toy two-token softmax, which already shows softmax([1,0])=[0.731,0.269] instead of [1,0]). Then replace the multiplication with an additive mask (M + QK^T, masked entries set to -inf) and rerun. If the additive version reproduces the reported AED ≈ 34.63 while the multiplicative version moves toward the ≈73.18 'w/o MCA' ablation, then Eq. 4 as written is not the mechanism responsible for the paper's central result and must be corrected or clarified with released code.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FantasyPortrait's central multi-character claim is carried by the masked cross-attention in Eq. 4: Z'_i = Z_i + softmax(M ⊙ Q_i K_i^T / sqrt(d_K)) V_i. As written, this does not produce a hard mask. In standard attention, masking is additive: logits are set to -inf (or a large negative) before softmax. Element-wise multiplication by a mask M sets disallowed logits to 0, and softmax then assigns those tokens the weight e^0, which is not zero. For a two-token case with allowed logit 1.0 and masked logit 0.0, the masked token receives 26.9% of attention; with N characters and many masked tokens, cross-character leakage grows. If M instead contains large negative values, multiplication is ill-defined because negative logits would be flipped to positive. The paper does not specify whether M is binary, soft, or additive, and releases no code. The Table 3 ablation (multi-character AED 34.63 with MCA vs 73.18 without) is the strongest evidence that the mechanism prevents interference, so this mathematical ambiguity is the most load-bearing weakness: either the implementation differs from Eq. 4 and the paper misdescribes its own mechanism, or the mask only attenuates rather than prevents interference, undercutting the stated contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FantasyPortrait, a diffusion-transformer-based portrait animation framework for single- and multi-character settings. It replaces explicit geometric priors with implicit expression features extracted by a pretrained encoder, adds an expression-augmented module for lip and emotion features, and introduces a masked cross-attention mechanism (Eq. 4) intended to prevent cross-character feature interference. The authors also contribute the Multi-Expr training dataset and the ExprBench evaluation benchmark. Experiments on ExprBench report state-of-the-art quantitative results, a user study, and ablations; the strongest single result is the multi-character AED improvement from 73.18 to 34.63 when the masked cross-attention is used (Table 3). The central claim is that the masked cross-attention provides independent yet coordinated multi-character expression control.","tokens_in":13069,"tokens_out":4681,"duration_ms":54912,"significance":"If the results hold, the paper addresses a real gap: most portrait animation methods are single-character, and multi-character animation suffers from expression leakage. The proposed Multi-Expr dataset and ExprBench benchmark are potentially useful community contributions, and the architecture is coherent with the DiT-based portrait animation trend. The Table 3 ablation suggests the masked cross-attention has a large effect, and the user study is consistent with the method's qualitative advantage. The paper is empirical and does not make parameter-free or theoretical claims; its contribution rests on the validity of the evaluation and the correctness of the mechanism description. These are not yet fully established, as detailed below.","major_comments":[{"comment":"This is a single comment.","section":"Sec. 3.3, Eq. (4)"},{"comment":"A second comment.","section":"Sec. 4.4, Tables 1 and 2"},{"comment":"A third comment.","section":"Sec. 3.2"}],"minor_comments":[{"comment":"The name 'Y onggang Qi' contains a spacing artifact and should read 'Yonggang Qi'.","section":"Author list, p. 1"},{"comment":"The phrase 'effectively soluting expression leakage' should be 'effectively solving expression leakage'.","section":"Conclusion, Sec. 5"},{"comment":"The text 'dathe Laplacian operator' should read 'the Laplacian operator'.","section":"Sec. 4.1"},{"comment":"The symbols z1 and z0 are used in the loss before they are defined; reorder the definitions for clarity.","section":"Sec. 3.1, Eq. (1)"},{"comment":"The caption should define all abbreviations, especially MAE, and state the units and direction of each metric; the current text only explains LMD, AED, and APD multipliers.","section":"Table 1"},{"comment":"The user study reports only mean scores from 32 participants without a description of the protocol, the number of samples per participant, or variance; please provide these details or cite the accompanying materials.","section":"Sec. 4.4, User Studies"},{"comment":"The text says face detection uses Huang et al. 2020 (CurricularFace), which is primarily a face recognition method; clarify whether the same model is used for detection and alignment or whether an additional detector is used.","section":"Sec. 3.3"},{"comment":"Figure 2 is dense and difficult to read at print size; enlarging the diagram and highlighting the mask construction and the Eq. (4) path would improve reproducibility.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the Eq. (4) discrepancy: if the actual implementation uses additive masking, the authors can easily fix the notation and the paper becomes much stronger; if the implementation truly is multiplicative binary masking, the central claim needs to be reframed and additional analysis is needed. The dataset and benchmark contributions are useful, but the self-constructed evaluation and the lack of error bars make the current 'significantly outperforms' claim premature. I would not reject on the current evidence, but the revision must directly address the mask implementation and the statistical robustness of the comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FantasyPortrait has a real contribution: it adapts a Wan-based DiT to multi-character portrait animation with an implicit expression-augmented encoder and a per-character masked cross-attention layer. The new Multi-Expr dataset and ExprBench benchmark are genuinely useful, and the ablation evidence—especially the AED jump from 34.63 to 73.18 when the mask is removed—does real work. If the method matches the description, it's a clear step forward for multi-character animation.\n\nThe main soft spot is Eq. 4. As written, M ⊙ QK^T is not a hard mask. Element-wise multiplication by a binary mask sets disallowed logits to 0, and softmax still gives those tokens weight e^0=1. Masked tokens do not get zero attention. Additive masking (adding -inf) would work, but the equation doesn't say that. The paper doesn't specify whether M is binary, soft, or additive, and no code is released to disambiguate. Since the central multi-character claim rests on this mechanism, this is load-bearing: either the implementation differs from Eq. 4 and the paper misdescribes itself, or the mask only attenuates interference rather than preventing it. This needs to be fixed before I'd trust the claim.\n\nOther issues are more typical for this subfield: no error bars on any table, baselines are off-the-shelf and not fine-tuned, only one multi-character baseline (LivePortrait), and the evaluation benchmark is self-created, with no data release. The identity-agnostic property of the implicit extractor is taken on faith from Wang et al. 2023a. None of these are fatal, but they make 'significantly outperforms' conditional on independent verification.\n\nI'd still send this to peer review. The contribution is real, the experiments are thorough for a systems paper, and the dataset/benchmark could standardize multi-character evaluation. I'd ask for code/data, a corrected masking equation, and ideally error bars. The paper deserves a serious referee.","headline":"Solid systems paper with a real multi-character contribution, but the masking equation is misdescribed and no code is released—fix those and it deserves acceptance.","tokens_in":13581,"tokens_out":3454,"would_cite":true,"duration_ms":35199,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A diffusion-transformer framework animates single- and multi-character portraits from one driving video using implicit expression features and latent-space attention masks.","keywords":["portrait animation","diffusion transformer","expression transfer","cross reenactment","multi-character animation","masked cross-attention","implicit expression representation","expression benchmark"],"falsifier":"Run the extractor on a scripted dataset in which many identities perform the same expression set; if the resulting embeddings cluster by identity rather than by expression, the identity-agnostic premise is false. Separately, retrain on ExprBench-Multi with the mask in Eq. 4 ablated; the paper's Table 3 predicts cross-reenactment AED rises from 34.63 to 73.18, so a markedly smaller change would indicate the mask is not the mechanism preventing inter-character interference.","tokens_in":12602,"feed_emoji":"🎭","tokens_out":12033,"duration_ms":119150,"temperature":0.7,"pith_summary":"FantasyPortrait is a video-generation method that animates a still portrait from a driving video and extends the same machinery to several portraits in one scene. The paper aims to show that explicit geometric controls such as facial landmarks and 3D morphable models are not necessary for high-fidelity cross-identity reenactment; instead, implicit, identity-agnostic expression features, selectively augmented for lips and emotions, transfer subtle facial dynamics more faithfully. For multi-character scenes, a masked cross-attention mechanism prevents one character's driving features from leaking into another's generation, enabling independent yet coordinated expressions. The authors also assemble a multi-character expression dataset and a benchmark, ExprBench, and report that FantasyPortrait outperforms existing methods on both quantitative metrics and user ratings, with the largest gains in cross reenactment and multi-character cases. If the claim holds, portrait animation moves from a per-character, geometry-dependent pipeline to a single diffusion-transformer model that handles single and multiple characters in one pass.","feed_headline":"Masked attention stops expression leakage between animated portraits","feed_subtitle":"Implicit expressions plus per-character attention masks drive several portraits from one video without identity leakage","key_machinery":"Two mechanisms carry the argument. First, the expression-augmented encoder $E_a$ takes the emotion and lip features ($e_{emo}$, $e_{lip}$) produced by the pretrained implicit extractor and decomposes them into learnable sub-feature tokens that attend to semantically aligned video tokens, capturing region-specific muscle and emotion dynamics; these are concatenated with head pose and eye features into a motion embedding $e_m$. Second, the masked cross-attention mechanism of Eq. 4 computes, in each DiT block, $Z'_i = Z_i + \\mathrm{softmax}((M \\odot Q_i K_i^\\top)/\\sqrt{d_K}) V_i$, where the latent mask $M$ is obtained by trilinear interpolation of the detected face mask into the DiT latent space and is multiplied inside the softmax so tokens of one portrait cannot attend to another portrait's motion tokens. The paper's strongest single piece of evidence for this mechanism is the ablation in Table 3, where removing the mask raises cross-reenactment AED on ExprBench-Multi from 34.63 to 73.18.","core_discovery":"The central claim is that a single diffusion-transformer framework can control both single- and multi-character portrait animation by decomposing facial dynamics into four implicit components, namely lip motion, eye gaze and blink, head pose, and emotion, and injecting them through cross-attention, with the lip and emotion components passed through an expression-augmented encoder that learns fine-grained sub-features. The authors argue that this avoids the identity leakage and alignment artifacts of landmark- or 3DMM-based controls. To keep multiple characters independent, the cross-attention in every diffusion-transformer block is masked: queries can only attend to motion tokens inside their own character's face mask, mapped into the latent grid by trilinear interpolation. On ExprBench, FantasyPortrait reports the best FID, FVD, LMD, MAE, AED, and APD among the compared methods, and the ablation supports the mask as the decisive component for multi-character control, with cross-reenactment AED worsening from 34.63 to 73.18 when the mask is removed.","pith_inferences":["If the extractor truly separates identity from expression, the same augmented encoder could serve other driving modalities such as audio, text, or synthesized motion; the paper does not test this.","The masked cross-attention design is not face-specific: any per-object mask could be plugged into Eq. 4, suggesting applications to multi-object or multi-pose animation beyond portraits; this is an extension, not a paper claim.","Because ExprBench is constructed by the same authors, an external multi-character benchmark would clarify how much of the gain comes from the architecture versus the new training data.","The reported metrics do not yet measure identity preservation; an identity-retrieval test on generated frames would separate expression fidelity from accidental identity drift."],"forward_implications":["Cross-identity reenactment no longer needs per-target keypoint adaptation or explicit geometry, so portrait animation can be driven across faces of different ethnicity, age, gender, or even stylized characters.","Multi-character scenes can be generated in one latent diffusion pass instead of segmenting and compositing faces in pixel space, which removes a class of boundary and discontinuity artifacts.","Selective augmentation of non-rigid expression features, namely lips and emotion, captures most of the fidelity gain; augmenting head pose and eye features adds little, so compute can be focused on the hard part.","The release of multi-character training data and a standard benchmark makes multi-character expression animation testable and comparable, which the paper argues was missing from the field."],"supporting_citations":[{"why":"Supplies the pretrained implicit expression extractor $E_e$ that outputs lip, eye, head-pose, and emotion features from aligned faces.","marker":"(Wang et al. 2023a)"},{"why":"Provides the pre-trained 40-layer video DiT backbone with causal 3D VAE, CLIP image conditioning, and masked first-frame training that FantasyPortrait adapts.","marker":"(Wan et al. 2025)"},{"why":"Gives the flow-matching objective used in Eq. 1, regressing velocity instead of noise during diffusion training.","marker":"(Lipman et al. 2022)"},{"why":"Establishes the latent diffusion formulation and VAE encoder-decoder that all latent-space operations presuppose.","marker":"(Rombach et al. 2022)"},{"why":"Face detection and recognition model used to locate and align each character before expression extraction and mask construction.","marker":"(Huang et al. 2020)"},{"why":"One of the two large video corpora filtered into the 30,000-clip Multi-Expr multi-character dataset.","marker":"(Nan et al. 2024)"},{"why":"The other source corpus for the Multi-Expr multi-character video clips.","marker":"(Li et al. 2025)"},{"why":"The single-portrait Hallo3 training set combined with Multi-Expr during training.","marker":"(Cui et al. 2025b)"},{"why":"The GAN-based LivePortrait baseline that the paper must beat in single- and multi-portrait comparisons.","marker":"(Guo et al. 2024)"},{"why":"Defines the AED and APD metrics used to evaluate cross-reenactment expression and pose accuracy in Tables 1 and 3.","marker":"(Siarohin et al. 2019)"}],"fun_headline_variants":["Masked attention stops expression leakage between portraits","Diffusion transformer animates multiple faces without identity seep","Expression-augmented model drives multi-character portrait motion","Per-character attention masks keep animated expressions independent","Implicit facial dynamics power multi-character animation without bleed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pretrained implicit expression extractor yields identity-agnostic dynamics and that the face mask, mapped into the DiT latent grid by trilinear interpolation, correctly localizes each character's tokens; if the extractor leaks identity or the mask misaligns, cross-identity and multi-character independence collapse.","fun_headline_variants_meta":{"raw":{"variants":["Masked attention stops expression leakage between portraits","Diffusion transformer animates multiple faces without identity seep","Expression-augmented model drives multi-character portrait motion","Per-character attention masks keep animated expressions independent","Implicit facial dynamics power multi-character animation without bleed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1297,"prompt_tokens":988,"completion_tokens":309,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":237}},"tokens_in":604,"tokens_out":309,"duration_ms":4305,"temperature":1.0,"reasoning_tokens":237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:34:02.369512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the extractor on a scripted dataset in which many identities perform the same expression set; if the resulting embeddings cluster by identity rather than by expression, the identity-agnostic premise is false. Separately, retrain on ExprBench-Multi with the mask in Eq. 4 ablated; the paper's Table 3 predicts cross-reenactment AED rises from 34.63 to 73.18, so a markedly smaller change would indicate the mask is not the mechanism preventing inter-character interference.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Face detection and recognition model used to locate and align each character before expression extraction and mask construction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the AED and APD metrics used to evaluate cross-reenactment expression and pose accuracy in Tables 1 and 3."}],"review_version":1}