{"id":"74e143c4-613b-4328-ab80-c4124c27426a","arxiv_id":"2411.18293","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"HiFiVFS applies SVD to video face swapping with identity-desensitized attribute features and detailed identity tokens, claiming state-of-the-art fidelity and temporal consistency.","lead":"This paper adapts Stable Video Diffusion for video face swapping, injecting identity from a source face and appearance attributes from a target video across multiple frames. It adds a fine-grained attribute module and a detailed identity tokenizer to preserve makeup, lighting, and temporal stability.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Undefined 'temporal co-denoising' in Sec. 5.1 may carry the reported temporal stability gains, making the SOTA video stability claim unverifiable without disclosure.","rationale":"The reader's weakest assumption focused on the FAL cycle's disentanglement guarantee (Sec. 4.2), which is a legitimate concern: if Eattr encodes identity in low-level features, the equality in Eq. (3) could hold without true disentanglement, undermining fine-grained attribute preservation. I find this concern real but somewhat indirect: the reported attribute metrics (Exp., Pose., Shape.) and the ablation study (Table 4) provide empirical support that FAL helps, even if the mechanism is not perfectly clean. In contrast, the temporal co-denoising issue is more directly load-bearing for the paper's headline SOTA claim. The paper's first contribution is explicitly about temporal stability, and the quantitative evidence for that claim (VIDD, FVD, and the user-study 'Stability' score) is obtained under an inference procedure that includes an undefined 'temporal co-denoising' step. If that step is a non-trivial component, the causal attribution of temporal stability to the proposed framework is unverifiable. This is a concrete, fixable omission: the authors can define the operation and provide an ablation. The verdict should remain CONDITIONAL because the concern is resolvable rather than fatal, and the paper's other elements (ablations, internal metric consistency, qualitative video claims) support a working system. I do not elevate to REJECT because the missing description is a documentation gap that can be closed, not a demonstrated logical flaw. My disagreement with the reader is partial: we both see the FAL disentanglement as a soft spot, but I would prioritize the co-denoising issue as the most load-bearing for the central temporal stability claim.","tokens_in":13029,"tokens_out":3146,"duration_ms":30275,"concrete_test":"Request the authors to release the exact algorithm for temporal co-denoising, then rerun the VFHQ-FS evaluation in two configurations: (1) with temporal co-denoising as used in the paper, and (2) with independent per-clip denoising (no co-denoising), keeping all other settings identical. If VIDD or FVD for 'Ours' shifts by more than the margin separating Ours from the best baseline (ΔVIDD=0.050 vs. Face-Adapter; ΔFVD=12.47 vs. SimSwap), the reported temporal stability is substantially attributable to an undocumented post-processing step, and the architectural contribution to temporal stability must be restated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of SOTA video face swapping rests on temporal stability, measured by VIDD and FVD in Table 2. In Sec. 5.1, Implementation Details, the authors state: 'the temporal co-denoising is used to weaken the detail discrepancies between different video clips,' but they never define this operation, specify its inputs, or ablate it. Since the model is trained on 16-frame clips and evaluated on 64-frame VFHQ-FS videos, inference must combine multiple clips; co-denoising appears to be the mechanism that aligns these clips. If this post-processing step, rather than the SVD temporal attention or the proposed FAL/DIL modules, is responsible for the VIDD/FVD improvements (e.g., 0.5041 vs. 0.5541 for SimSwap and 81.12 vs. 93.59 FVD), then the paper's assertion of being the 'first attempt to improve temporal stability within the face swapping framework' is not supported by the described architecture. The claim is also confused by the paper's own reference [42] (MobileFaceSwap), which is explicitly a video face swapping framework. Without code, a precise description of co-denoising, or an ablation that removes it, the temporal stability component of the SOTA claim cannot be independently checked, and the contribution attribution is unclear.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HiFiVFS, a video face swapping framework built on Stable Video Diffusion (SVD). A source image supplies identity and a target video supplies attributes; the model is trained with an inpainting-style data flow, a fine-grained attribute learning module (FAL) that uses an identity-replacement cycle plus adversarial learning, and a detailed identity learning module (DIL) that injects tokens from the last Res-Block layer of a face-recognition model. Training losses include denoising score matching, attribute-cycle, triplet identity, reconstruction, adversarial, and identity-cosine losses. The method is evaluated on FF++ and a new VFHQ-FS test set against GAN- and diffusion-based baselines, using identity retrieval/similarity, expression/shape/gaze/pose errors, VIDD, FVD, qualitative comparisons, and a user study. The paper claims state-of-the-art results in both fidelity and temporal stability.","tokens_in":13487,"tokens_out":6040,"duration_ms":52340,"significance":"If the claims hold, HiFiVFS is a useful advance: it adapts SVD to video-to-video face swapping, introduces a principled attribute-disentanglement objective, and achieves strong quantitative results on external benchmarks. The manuscript is commendable for giving explicit loss definitions, using external datasets and metrics, reporting ablations of FAL and DIL, and providing supplementary video comparisons. The main weakness is that the inference-time 'temporal co-denoising' procedure, which is central to the temporal-stability claim, is not defined or ablated, so the attribution of the reported VIDD/FVD gains to the architectural contributions is currently unverifiable.","major_comments":[{"comment":"The term 'temporal co-denoising' appears exactly once: 'the temporal co-denoising is used to weaken the detail discrepancies between different video clips.' It is never defined, no inputs or algorithm are given, and no ablation removes it. Because the model is trained on 16-frame clips but evaluated on 64-frame VFHQ-FS videos, multi-clip inference must be specified; if co-denoising is a test-time clip-chaining or post-processing step, it could be responsible for the VIDD/FVD improvements in Table 2 rather than the proposed FAL, DIL, or SVD temporal attention. Please provide a precise description and an ablation with co-denoising disabled.","section":"Sec. 5.1"},{"comment":"The FAL cycle loss Lattr = ||fattr - f'attr||^2 assumes that replacing the identity in Vt with a different identity and then re-extracting attributes yields identity-disentangled attribute features. This assumption is not tested: Eattr could store identity in low-level features that survive the cycle, or Dec could ignore fattr entirely, making the equality hold without true disentanglement. Since fine-grained attribute preservation and identity-leakage prevention are central contributions, the authors should provide a direct validation, for example a face-recognition/ID-retrieval test on fattr alone or an experiment showing that fattr does not recover the source identity.","section":"Sec. 4.2, Eq. (3)"},{"comment":"All headline numbers are point estimates without confidence intervals, repeated-run variance, or significance tests. The user study in Table 3 uses only 15 participants, yet the text states that the model 'significantly outperforms' the baselines, without reporting any statistical test. In addition, the DiffSwap row in Table 1 is based on 'official code' results that 'differ significantly from those in the paper,' as the table caption acknowledges; the authors should state explicitly which set of numbers is used and justify the choice, since a baseline evaluated under a different configuration may not be directly comparable.","section":"Sec. 5.2, Tables 1-3"},{"comment":"The claim that this is 'the first attempt to improve temporal stability within the face swapping framework' (Sec. 1) is contradicted by the paper's own reference [42], MobileFaceSwap, which is explicitly a video face swapping framework. The claim should be narrowed, for example to 'the first diffusion-based video face swapping framework with temporal attention,' or the authors should explain why prior video-based face swapping methods do not count as attempts at temporal stability.","section":"Sec. 1 and Sec. 2.2"}],"minor_comments":[{"comment":"The text says the overall loss has four components and refers to 'Denoising Score Matching (eq. 3),' but Eq. (3) is Lattr; the DSM loss is Eq. (2), and Eq. (8) actually lists three components (LDM, LFAL, Lid).","section":"Sec. 4.4"},{"comment":"The description of FAL says 'we randomly select a face' to obtain frid, but it is not stated whether this face comes from the same video Vt, another video, or a fixed identity bank; please clarify the sampling procedure.","section":"Sec. 4.2"},{"comment":"The text mentions 'four reshuffled face swapping results generated by SimSwap, HifiFace, InfoSwap, Face-Adapter, and our HiFiVFS,' which actually lists five methods; rephrase to avoid confusion.","section":"Sec. 5.2, Human Evaluation"},{"comment":"VIDD is defined only by a citation to FOS; because VIDD and FVD are central to the temporal-stability claims, the exact computation formula for VIDD should be given in the text or appendix.","section":"Sec. 5.1, Evaluation Metrics"},{"comment":"The phrase 'classifier guidance scale of 2' is unclear because SVD is a latent video diffusion model and no classifier is described; please specify what is being guided and how the guidance scale is applied.","section":"Sec. 5.1, Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to make a useful contribution once the temporal co-denoising mechanism is fully disclosed and ablated, the disentanglement assumption of FAL is validated, and the statistical basis for the SOTA claim is strengthened. I would also ask the authors to revisit the 'first attempt' claim against their own reference [42]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent, well-engineered video face swapping paper, and the SVD-based multi-frame design is genuinely new in this combination. The main thing to check before believing the SOTA claims is the undefined 'temporal co-denoising' in Sec. 5.1—it appears once, is never defined, and is not ablated, yet it may be doing much of the work for the VIDD/FVD numbers.\n\nWhat I liked: the losses are explicit, the ablations in Table 4 give real support for FAL and DIL, and the evaluation uses external datasets (FF++, VFHQ-FS) with external metrics (CosFace, 3DMM). They also honestly flag that DiffSwap's official code disagrees with its paper and report the official-code numbers separately. The idea of using SVD's temporal attention for video-to-video face swapping, with fine-grained attribute features fed by direct addition into the U-Net, is a reasonable contribution. The FAL cycle (swap identity, re-extract attributes, enforce consistency) is clever and the triplet identity loss is sensible for preserving target attributes.\n\nSoft spots, in order of importance:\n1. The temporal co-denoising. The model trains on 16-frame clips and evaluates on 64-frame videos, so inference must stitch clips. The paper says co-denoising 'weakens detail discrepancies between clips' but gives no definition, inputs, or ablation. Since the paper's headline claim is temporal stability, this is a load-bearing omission. This is fixable, but it has to be disclosed.\n2. The 'first video face swapping framework' claim is contradicted by their own reference [42], MobileFaceSwap, which is a video face swapping method. They likely mean 'first diffusion-based video face swapping with temporal attention,' but that is not what they wrote.\n3. The FAL disentanglement assumption is untested. The Lattr cycle can be satisfied via shortcuts (Eattr hiding identity in low-level features, or Dec ignoring fattr). A simple identity-retrieval test on fattr features would help.\n4. No code, models, or error bars. The quantitative tables are point estimates from a 15-person user study. For a paper making SOTA claims, that is thin, but not fatal.\n\nThe core architecture is credible and the ablations support the components. This paper deserves a serious referee, but the co-denoising step must be defined and ablated before the temporal stability claims can be accepted.","headline":"A plausible diffusion-based video face swapping system whose headline temporal stability claim rests partly on an undefined inference-time 'temporal co-denoising' step that needs to be specified and ablated.","tokens_in":13858,"tokens_out":1373,"would_cite":false,"duration_ms":14975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a video face swapping framework, built on Stable Video Diffusion, that preserves fine-grained target attributes and temporal stability through identity-disentangled attribute learning and detailed identity tokens.","keywords":["video face swapping","diffusion models","Stable Video Diffusion","temporal stability","fine-grained attributes","identity disentanglement","identity tokenizer","video-to-video generation"],"falsifier":"Take two source faces that are nearly identical in shape but differ in a fine attribute such as makeup or a skin mark, and use them in the FAL cycle: if the swapped output preserves the source's fine attribute instead of the target's, or if the attribute features $f_{\\text{attr}}$ and $f'_{\\text{attr}}$ differ for two targets with identical attributes but different identities, the identity-disentanglement claim fails.","tokens_in":12836,"feed_emoji":"🎬","tokens_out":9400,"duration_ms":74915,"temperature":0.7,"pith_summary":"The paper is trying to establish that video face swapping can be done directly on video rather than frame-by-frame, using the temporal prior of a pretrained video diffusion model. It claims that previous diffusion-based approaches, which rely on inpainting and global attribute features, lose fine-grained details such as lighting and makeup, and that this loss can be prevented by an identity-desensitization cycle that trains the attribute encoder to ignore who the face is. It also claims that identity similarity improves when the model uses high-resolution intermediate features from a face recognition network instead of the final compact identity vector. If these claims hold, video face swapping gains both temporal stability and the ability to preserve subtle target attributes, which are the two qualities that have limited earlier methods. A human evaluation and quantitative metrics on standard benchmarks are reported as evidence.","feed_headline":"Face-swap videos now keep lighting, makeup, and stability","feed_subtitle":"A diffusion-based pipeline swaps identity across frames while preserving fine target details that prior methods lose.","key_machinery":"The load-bearing mechanism is the FAL identity-desensitization cycle: the attribute encoder $E_{\\text{attr}}$ extracts attribute features $f_{\\text{attr}}$ from the target video; the decoder fuses $f_{\\text{attr}}$ with a randomly chosen identity vector to synthesize a modified video with a different identity; and $E_{\\text{attr}}$ is applied again to that modified video to get $f'_{\\text{attr}}$, with the loss $L_{\\text{attr}} = \\frac{1}{2}\\|f_{\\text{attr}} - f'_{\\text{attr}}\\|_2^2$ forcing the two to match. This trains the attribute features to be invariant to identity, and a triplet margin identity loss plus an adversarial loss push the cycle to produce realistic, identity-changed outputs. The second mechanism is DIL, which replaces the usual global 512-dimensional identity vector with 49 spatial tokens taken from the last Res-Block of a face recognition network, fed into cross- and temporal attention to retain fine identity detail.","core_discovery":"The central discovery is that a video-to-video face swapping pipeline can be built on Stable Video Diffusion by extending it to multi-frame input and conditioning it on identity features, while a Fine-grained Attributes Learning (FAL) module learns attribute features that are disentangled from identity through a reconstruction cycle: the same encoder is applied to the target video and to a synthesized video with a different identity, and the two attribute feature sets are forced to match. This, together with a Detailed Identity Learning (DIL) module that feeds spatial tokens from an intermediate recognition-layer into cross- and temporal attention, is what the authors claim enables both state-of-the-art identity similarity and preservation of fine details such as lighting, makeup, and occlusion handling. In their experiments, HiFiVFS outperforms GAN-based and diffusion-based baselines on FaceForensics++ and on a newly selected VFHQ-FS test set across identity, expression, pose, shape, gaze, video identity distance, and Fréchet Video Distance.","pith_inferences":["The identity-desensitization cycle is a general recipe: any video-to-video task that needs identity-free attribute features (reenactment, expression transfer, makeup transfer) could adopt the same reconstruction cycle with a different identity vector.","A stronger check of disentanglement than the reported metrics would be an explicit invariance test: measure how much the attribute features change when the identity is swapped while attributes are held fixed, across many pairs of identities.","The 16-frame training clips and sliding temporal co-denosing for long videos suggest a windowed extension to arbitrary-length footage, though boundary effects between windows are not analyzed in the paper.","Applying the FAL cycle in latent space rather than pixel space is a deliberate design choice; the paper's own ablation shows that the pixel-space variant drops across all metrics, implying the latent-space alignment with the denoising U-Net is doing real work."],"forward_implications":["Video face swapping can be treated as a video-to-video generation task, so temporal attention across frames replaces frame-by-frame processing and removes the per-frame jitter that image-based methods suffer.","Fine-grained target attributes like lighting, makeup, and occlusion are preserved by injecting low-level attribute features directly into the denoising U-Net input rather than through cross-attention.","Identity similarity in swapped videos improves when identity is represented by spatial tokens from an intermediate face-recognition layer instead of the final compact identity vector.","The EDM-style denoiser of Stable Video Diffusion lets identity losses be computed on decoded images during training, so the identity and attribute losses integrate directly into the diffusion objective.","The method inherits the sampling speed and memory costs of video diffusion models, and the authors point to diffusion distillation as the path to faster synthesis."],"supporting_citations":[{"why":"provides the pretrained Stable Video Diffusion backbone with temporal attention that HiFiVFS extends to multi-frame face swapping","marker":"[3]"},{"why":"baseline diffusion inpainting method whose attribute-feature limitations motivate the Fine-grained Attributes Learning module","marker":"[14]"},{"why":"baseline diffusion inpainting method whose data flow HiFiVFS adopts and compares against","marker":"[47]"},{"why":"supplies the adversarial objective used in Fine-grained Attributes Learning","marker":"[7]"},{"why":"motivates operating in latent space, which the FAL module relies on to align with the denoising U-Net","marker":"[26]"},{"why":"standard FaceForensics++ evaluation set used for quantitative comparisons","marker":"[29]"},{"why":"VFHQ dataset used for training and for the newly selected VFHQ-FS challenging test set","marker":"[38]"},{"why":"CosFace recognition model used to compute identity retrieval and identity similarity metrics","marker":"[34]"}],"fun_headline_variants":["Video face swap preserves lighting and makeup details","Stable Video Diffusion powers high-fidelity face swap","New method keeps fine attributes in video face swapping","Face swap in video with identity and detail disentanglement","HiFiVFS achieves SOTA video face swap with SVD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that forcing the attribute extractor to output the same features for a video and for the same video with a different face substituted in truly removes the person's identity from those features, rather than the model finding a shortcut where identity still leaks through.","fun_headline_variants_meta":{"raw":{"variants":["Video face swap preserves lighting and makeup details","Stable Video Diffusion powers high-fidelity face swap","New method keeps fine attributes in video face swapping","Face swap in video with identity and detail disentanglement","HiFiVFS achieves SOTA video face swap with SVD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000435,"raw_usage":{"total_tokens":2200,"prompt_tokens":919,"completion_tokens":1281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1205}},"tokens_in":535,"tokens_out":1281,"duration_ms":12472,"temperature":1.0,"reasoning_tokens":1205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:20:24.343041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two source faces that are nearly identical in shape but differ in a fine attribute such as makeup or a skin mark, and use them in the FAL cycle: if the swapped output preserves the source's fine attribute instead of the target's, or if the attribute features $f_{\\text{attr}}$ and $f'_{\\text{attr}}$ differ for two targets with identical attributes but different identities, the identity-disentanglement claim fails.","supporting_citations":[{"cited_title":"Diffswap: High-fidelity and con- trollable face swapping via 3d-aware masked diffusion","cited_arxiv_id":null,"evidence_quote":"baseline diffusion inpainting method whose data flow HiFiVFS adopts and compares against"},{"cited_title":"Stargan v2: Diverse image synthesis for multiple domains","cited_arxiv_id":null,"evidence_quote":"supplies the adversarial objective used in Fine-grained Attributes Learning"},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"motivates operating in latent space, which the FAL module relies on to align with the denoising U-Net"},{"cited_title":"Faceforen- sics++: Learning to detect manipulated facial images","cited_arxiv_id":null,"evidence_quote":"standard FaceForensics++ evaluation set used for quantitative comparisons"},{"cited_title":"Vfhq: A high-quality dataset and bench- mark for video face super-resolution","cited_arxiv_id":null,"evidence_quote":"VFHQ dataset used for training and for the newly selected VFHQ-FS challenging test set"},{"cited_title":"Cosface: Large margin cosine loss for deep face recognition","cited_arxiv_id":null,"evidence_quote":"CosFace recognition model used to compute identity retrieval and identity similarity metrics"}],"review_version":1}