Pith. sign in

REVIEW 3 major objections 6 minor 68 references

Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding optimized noise to a portrait can stop audio-driven talking-head generators from animating it, and the noise survives diffusion-based cleanup.

desk verdict Silencer-I's nullifying loss is a genuinely useful idea with solid sync-suppression results; Silencer-II's purification robustness claim is not yet supported because the paper never measures sync after purification. read the letter →

arxiv 2506.01591 v1 pith:4AFYBAUQ submitted 2025-06-02 cs.GR cs.CRcs.CVcs.SDeess.AS

Yuan Gan , Jiaxu Miao , Yunze Wang , Yi Yang This is my paper
classification cs.GRcs.CRcs.CVcs.SDeess.AS
keywords talking-headgenerationlatentdiffusionmodelsadversarialperturbationsprivacyprotectionaudio-visualsynchronizationpurificationrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Silencer is a two-stage method for protecting portrait privacy against latent-diffusion-model talking-head generators such as Hallo. The paper claims that a carefully optimized perturbation added to a portrait makes the generator's output ignore the driving audio, so the synthesized talking head stays silent and lip motion no longer matches the speech; SyncNet confidence drops from 6.4041 to 3.9685 on CelebA-HQ and from 5.4842 to 2.0017 on TalkingHead-1KH. The first stage introduces a nullifying loss that treats the reference portrait itself as the ground-truth frame, avoiding the need for synchronized video. The second stage optimizes a DDIM-inverted latent feature with an anti-purification objective so the perturbation survives diffusion-based purification better than prior image-space attacks. If the claim holds, users could proactively render their portraits unusable for unauthorized talking-head synthesis, though the paper itself concedes that purification still alters the perturbation and a perfectly silent post-purification portrait remains an open challenge.

What carries the argument

The load-bearing mechanism is the nullifying loss $L_N = \mathbb{E}\|\epsilon - \epsilon_\theta(\hat{z}_t, t, p, a_i)\|_2^2$, which replaces the synchronized ground-truth frame with the reference portrait as the target. Minimizing $L_N$ with gradient descent under a projected-gradient perturbation budget produces an adversarial portrait that the talking-head model tends to keep static. The second stage uses the anti-purification loss $L_{AP} = \lambda_1 L_N + \lambda_2 \|E(p'_0) - E(p_n)\|_2^2$, where $p'_0$ is the DDIM-inverted latent reconstructed image and $p_n$ is the stage-one adversarial example; this jointly drives synchronization failure and preserves identity while optimizing in latent space. A facial mask applied after the first 100 optimization steps limits distortion of recognizable facial features.

What would settle it

Run DiffPure or GrIDPure on a Silencer-II protected portrait, feed the purified portrait and the original audio into Hallo, and measure SyncNet confidence on the generated video. If the confidence returns to near the ground-truth level (about 6.4 on CelebA-HQ) while the purified image still shows the reported I-PSNR differences, the claimed purification resistance fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that an audio-conditioned talking-head generator can be made to ignore audio by changing the optimization target: instead of asking the model to produce a synchronized talking frame, the nullifying loss asks it to produce the reference portrait itself, so the perturbation that minimizes this loss keeps the mouth closed and the video out of sync. Silencer then re-optimizes the perturbation in the inverted latent space of a diffusion model, guided by the stage-one adversarial image and constrained by a face mask, producing perturbations that are substantially harder to remove by JPEG, AdvClean, DiffPure, and GrIDPure. The paper reports that this yields the lowest synchronization confidence among all compared protection methods and that the effect transfers to other LDM-based talking-head models built on the same diffusion backbone, such as EchoMimic and Hallo2.

Load-bearing premise

The paper measures purification robustness by how much the purified image differs from the original, not by whether a video generated from the purified portrait still fails to lip-sync the audio.

Editorial extensions

If this is right

  • If the claim holds, proactive privacy protection can be extended from image mimicry to audio-driven portrait animation, covering a realistic misuse scenario that previous perturbation methods did not address.
  • The reported transfer to EchoMimic and Hallo2 indicates that perturbations optimized on one LDM-based talking-head model may degrade synchronization across models sharing the Stable Diffusion backbone.
  • The nullifying loss could be adapted to other conditional generation tasks where the goal is to block a specific conditioning signal while preserving the input image.
  • The paper's benchmark with SyncNet confidence, M-LMD, and purification routines gives future proactive defenses a concrete evaluation protocol for talking-head privacy protection.
  • Because Silencer-II costs 241 seconds per portrait compared with 64 seconds for Silencer-I, practical deployment will likely prefer Silencer-I when purification resistance is not the primary threat.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial extension: because the paper measures purification robustness by I-PSNR and FID of the purified image rather than by SyncNet confidence of videos generated from that purified image, the claim of purification resistance is only as strong as the untested link between image-level metrics and audio-visual synchronization.
  • A testable extension would be to run a full pipeline evaluation where a purified Silencer-II portrait is animated and measured with SyncNet; if the purified portrait still produces synchronized speech, the anti-purification stage should be revised.
  • The transferability result suggests that the perturbation may attack the shared VAE or cross-attention mechanism rather than Hallo-specific modules, which could be probed by ablating individual model components.
  • A broader implication, if confirmed, is that users could routinely apply a silent perturbation to publicly posted portraits, but the method would need to survive social-media compression and face restoration to be practical.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Silencer, a two-stage adversarial method for protecting portrait privacy against audio-driven talking-head generation based on latent diffusion models. Stage I (Silencer-I) adds an l-infinity-bounded perturbation to the reference portrait by minimizing a nullifying loss that uses the reference portrait itself as the denoising target, with the goal of making the generated video ignore the audio and remain "silent" as measured by SyncNet confidence. Stage II (Silencer-II) further optimizes the DDIM-inverted latent using a combination of the nullifying loss and a VAE-feature constraint, aiming to produce perturbations that resist purification. The method is evaluated with Hallo on CelebA-HQ and TalkingHead-1KH, reporting lower SyncNet confidence than several prior protection methods, and with purification experiments using JPEG, AdvClean, DiffPure, and GrIDPure. The paper also includes component ablations and cross-model transfer experiments in the supplementary material.

Significance. If the anti-purification claim were fully supported, Silencer would be a useful contribution to proactive defenses against talking-head misuse, with a simple and effective nullifying loss and a plausible latent-space anti-purification strategy. The first-stage results are credible and directly measured: on two datasets, SyncNet confidence drops from 6.40 to 3.97 and from 5.48 to 2.00, well below prior baselines, and the component ablations support the contribution of the nullifying loss. The release of code and the cross-model transfer tests in the appendix strengthen reproducibility. However, the paper's second-stage claim of resistance to purification is currently evaluated only through image-level proxies and is explicitly conceded to be imperfect in Section 4.3, so the main advertised advantage over prior work is not yet demonstrated.

major comments (3)
  1. [Section 4.3, Table 3] The central claim that Silencer-II resists purification is not evaluated with the paper's own privacy metric. Table 3 reports only I-PSNR and FID after JPEG, AdvClean, DiffPure, and GrIDPure; no SyncNet confidence (or M-LMD) is reported for talking-head videos generated from the purified portraits. Since the objective is to nullify audio control, low I-PSNR or high FID can be caused by image corruption or visual distortion that does not prevent lip-sync, and a purified portrait can be "silent" or "talking" regardless of these image-level numbers. Please report SyncNet confidence for videos generated from purified portraits for Silencer-I, Silencer-II, and the strongest baselines across all four purification methods. The manuscript's own concession in Section 4.3 that purification alters the perturbations and prevents a "perfectly silent" portrait makes this quantification necessary; without it, the statement that Silencer-II "achieves the best anti-purification performance" is unsupported.
  2. [Section 3.2.2, Eq. (7)] The anti-purification loss L_AP does not incorporate any purification operator, noise-removal simulation, or adversarial training against purification; it is a combination of the nullifying loss and a VAE-feature constraint that keeps the optimized image close to the stage-I adversarial example. The robustness to purification is therefore an emergent property that must be demonstrated on the downstream outcome, not on image-level proxies. The current Table 3 evidence is insufficient for the reason stated above. I recommend adding direct SyncNet evaluations after purification or, alternatively, reformulating the anti-purification objective so that it explicitly optimizes against a purification procedure and then evaluating the resulting residual sync suppression.
  3. [Section 4.2, Table 1 and Appendix A.4] Report standard deviations or confidence intervals for the main comparisons. The supplementary material gives a standard deviation of 1.5607 for Silencer-II on CelebA-HQ, but no such values are given for the baselines or for Silencer-I. Without these, small differences such as the CelebA-HQ SyncNet gap between Silencer-I (4.0644) and Silencer-II (3.9685) cannot be interpreted, and the component-ablation conclusion in Table 4 that each component contributes to privacy protection is not fully supported. At a minimum, please provide per-method variance for Table 1 and Table 4.
minor comments (6)
  1. [Abstract and Section 1] There are several spacing typos, e.g., "proposeSilencer" and "ofSilencer"; please correct these throughout.
  2. [Section 3.2.2, Eq. (7)] Please clarify the relationship between the optimized latent p_t and the image p'_0 that enters Eq. (8); as written, p'_0 is described as the output of the inverted diffusion model, but the optimization variable is p_t, and the text should specify how one is obtained from the other.
  3. [Section 4.3] AdvClean is listed as a purification method but is neither defined nor cited; please add a reference or a description.
  4. [Section 4.1.2] The description of FID is ambiguous in the context of Table 3; please state explicitly that in Table 3, FID compares videos generated from protected/purified portraits against ground-truth videos generated from the original portraits.
  5. [Tables 1 and 4] The baseline label "SDTS(-)" is not defined in the text; if it is a variant of the SDS method from reference [59], please state this explicitly.
  6. [Figure 7] The ten timestep segments are not labeled directly on the x-axis; please add axis labels showing the timestep ranges so the reader can identify the [200, 300] interval without referring to the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the nullifying loss is a direct attack objective optimized through a surrogate denoising loss, while the claimed outcome is measured with an external SyncNet metric; the anti-purification limitation is an empirical gap, not a circular reduction.

full rationale

The paper does not derive its claimed privacy protection from the metric it reports by construction. Stage I defines a nullifying loss L_N = E||eps - eps_theta(z_hat_t, t, p, a_i)||^2 that treats the reference portrait as ground truth and optimizes the perturbation with PGD (Eq. 6); the claimed outcome, reduced SyncNet confidence in Table 1, is measured with an external SyncNet model on videos generated by Hallo, so the optimization and evaluation are distinct. Stage II's L_AP = lambda1 L_N + lambda2||E(p'_0) - E(p_n)||^2 does include L_N, so part of the sync reduction is by construction, but that is the intended attack objective rather than a hidden fitted-input-called-prediction: the authors do not claim to predict sync from L_N, and the stage-I result is independently verified by cross-model transfer to EchoMimic and Hallo2 in Table 5. No load-bearing claim rests on a self-citation: the cited work [15] by co-author Yuan Gan is background on talking-head generation and is not used to justify Silencer's mechanism or to forbid alternatives. The main gap is in Section 4.3, where purification robustness is evaluated only via I-PSNR/FID and the paper concedes that purification 'alter[s]' the perturbations and prevents a 'perfectly silent' portrait; this is an unsupported or weakened empirical claim, not circularity. Because all equations are stated, the attack objective is explicitly the thing being optimized, and the metric is external, the derivation chain is self-contained and non-circular.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The free parameters are standard attack hyperparameters, many of them chosen by ablation. The main reproducibility concern is the omission of lambda1/lambda2. The axioms are domain assumptions about differentiability, metric validity, and representativeness of the purification attacks.

free parameters (8)
  • Perturbation budget delta = 16/255
    l-infinity budget used in Table 1; ablation in Table 10 shows 8/255 weakens protection, so 16/255 is chosen by performance.
  • PGD iterations = 100
    Number of PGD steps in stage I; set equal to baselines.
  • AdamW learning rate = 0.01
    Used for latent optimization in stage II.
  • Latent optimization iterations = 200
    Stage II optimization steps.
  • Mask-free initial iterations s = 100
    Ablation Table 12 shows trade-off; s=100 chosen for balance.
  • Timestep range for nullifying loss = [200, 300]
    Ablation Fig. 7 selects this range for best trade-off between sync reduction and I-PSNR.
  • lambda1, lambda2 (weights in Eq. 7) = not reported
    Coefficients balancing nullifying and texture losses; values omitted from the paper, harming reproducibility.
  • DDIM inversion steps = 20
    Used for DDIM inversion in stage II; only the final timestep latent is optimized.
assumptions (5)
  • domain assumption Hallo's talking-head pipeline is differentiable end-to-end, allowing gradient-based optimization of the input portrait.
    The method applies PGD to the input image through the full generation loss; if the pipeline is not differentiable in practice (e.g., non-differentiable sampling), the attack fails. The paper does not verify this explicitly.
  • domain assumption SyncNet confidence is a faithful proxy for audio-visual synchronization and therefore for the success of the privacy protection.
    All main results are judged by lowering SyncNet confidence; the paper does not validate that a static video with low SyncNet confidence actually prevents misuse (e.g., the video could still be lip-synced by other methods).
  • ad hoc to paper Using the reference portrait as the denoising target in the nullifying loss forces the generated frame to stay silent rather than merely producing artifacts.
    The loss LN replaces the ground-truth frame with the reference portrait E(p). This is the core design choice of Silencer-I and is not derived from audio-visual theory; it is validated only empirically.
  • domain assumption DDIM inversion with 20 steps reconstructs the image faithfully enough that optimizing the inverted latent feature yields meaningful adversarial perturbations.
    Stage II relies on DDIM inversion accuracy; the paper does not analyze inversion error or its effect on the final perturbation.
  • domain assumption DiffPure and GrIDPure are representative of diffusion-based purification countermeasures.
    Robustness is demonstrated only against these two plus JPEG/AdvClean; other or stronger purification may remove the perturbation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation." pith.science (2026). https://pith.science/paper/4AFYBAUQ

@misc{pith2026250601591,
  author       = {Pith},
  title        = {Pith review of: Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4AFYBAUQ}},
  note         = {Machine review of arXiv:2506.01591}
}
read the original abstract

Advances in talking-head animation based on Latent Diffusion Models (LDM) enable the creation of highly realistic, synchronized videos. These fabricated videos are indistinguishable from real ones, increasing the risk of potential misuse for scams, political manipulation, and misinformation. Hence, addressing these ethical concerns has become a pressing issue in AI security. Recent proactive defense studies focused on countering LDM-based models by adding perturbations to portraits. However, these methods are ineffective at protecting reference portraits from advanced image-to-video animation. The limitations are twofold: 1) they fail to prevent images from being manipulated by audio signals, and 2) diffusion-based purification techniques can effectively eliminate protective perturbations. To address these challenges, we propose Silencer, a two-stage method designed to proactively protect the privacy of portraits. First, a nullifying loss is proposed to ignore audio control in talking-head generation. Second, we apply anti-purification loss in LDM to optimize the inverted latent feature to generate robust perturbations. Extensive experiments demonstrate the effectiveness of Silencer in proactively protecting portrait privacy. We hope this work will raise awareness among the AI security community regarding critical ethical issues related to talking-head generation techniques. Code: https://github.com/yuangan/Silencer.

Figures

Figures reproduced from arXiv: 2506.01591 by the authors.

Figure 1
Figure 1. Overview of Our Motivation. Given an audio input, talking-head animation models can be exploited to generate fabri￾cated videos using any portrait. To safeguard portrait privacy, we introduced Silencer, applying protective perturbations to ensure the portrait’s mouth remains closed in generated talking videos. ism of these animations. Consequently, this technological advancement increases the risks of misusing AI-Ge… view at source ↗
Figure 2
Figure 2. LDM and LDM-based Talking-head Generation Framework. (a) The inference process of latent diffusion mod￾els. Given random noise and text, LDM can generate a semanti￾cally coherent image through iterative denoising. (b) The talking￾head generation framework. Given a portrait and audio frame, the talking-head generation model can produce a lip-sync video frame with realistic facial expressions. • Semantic loss [30] is … view at source ↗
Figure 3
Figure 3. Silencer Framework. (a) In stage I, adversarial samples p n are generated through iterative PGD optimization using our proposed nullifying loss. These samples can avoid the influence of audio in the talking-head model. (b) In stage II, our anti-purification process is employed to optimize the inverted latent features, generating more robust perturbations capable of resisting purification. challenges: First, unlike i… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Visualization of ACA [4] and Silencer. The result of ACA is generated by optimizing latent feature with skip gradients using our nullifying loss. LN modifies the training target of talking-head genera￾tion from a synchronized frame to a still portrait. Then we treat LN…
Figure 5
Figure 5. Figure 5: Qualitative Comparison with Image Protection Methods. We visualize the protected portraits and their frame driven by audio. for recognition, while simultaneously maximizing the ro￾bustness of the perturbations against purification defenses. The optimization objective i…
Figure 6
Figure 6. Figure 6: Visual Comparison in Anti-purification. The third row is the animated talking frames with the portraits after GrIDPure [64]. the distribution distance between videos generated using original and protected portraits. Audio-visual synchronization. We evaluate the audio￾v…
Figure 7
Figure 7. Figure 7: Ablation Study on Timestep Ranges in Silencer-I. portraits. We select the following advanced purification methods to attack: JPEG [42], AdvClean, Diff-Pure [35] and GrIDPure [64].We evaluate anti-purification effective￾ness using I-PSNR, comparing original and purified…
Figure 8
Figure 8. Figure 8: Ablation Study on LT in Silencer-II. Without the as￾sistance of LT , the generated perturbation becomes highly notice￾able, significantly compromising the facial identity. 𝑠 = 50 𝑠 = 75 𝑠 = 100 𝑠 = 125 Clean 𝑠 = 200 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Visualization Results with Different Iteration s. The quality of the portrait decreases with the growth of s. s 50 75 100 125 200 I-SSIM↑ 0.7125 0.6998 0.6918 0.6844 0.6704 FID↑ 136.88 166.10 173.18 171.08 193.46 Sync↓ 5.5725 5.0413 4.0602 4.1832 4.0791 M-LMD↑ 1.8559 2…
Figure 10
Figure 10. Figure 10: Additional Visualization Comparison with Image Protection Methods in CelebA-HQ [25]. Protect Talking Protect Talking Method Clean AdvDM(+) PhotoGuard Mist SDS(+) SDS(-) SDTS(-) Silencer-I Silencer-II [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Additional Visualization Comparison with Image Protection Methods in TalkingHead-1KH [53]. gesting variations in inherent robustness. We intend to in￾vestigate the factors contributing to this variability in future research. Ablation Study on Perturbation Budget in Si…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 37 canonical work pages

  1. [1]

    A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt.arXiv preprint arXiv:2303.04226, 2023

    Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S Yu, and Lichao Sun. A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt.arXiv preprint arXiv:2303.04226, 2023. 2

    arXiv 2023

  2. [2]

    Diffusion models for impercepti- ble and transferable adversarial attack.IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–17,

    Jianqi Chen, Hao Chen, Keyan Chen, Yilan Zhang, Zhengxia Zou, and Zhenwei Shi. Diffusion models for impercepti- ble and transferable adversarial attack.IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–17,

    work page

  3. [3]

    Hierarchical cross-modal talking face generation with dynamic pixel-wise loss

    Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7832–7841, 2019. 1, 2, 7

    work page 2019

  4. [4]

    Content-based unrestricted ad- versarial attack

    Zhaoyu Chen, Bo Li, Shuang Wu, Kaixun Jiang, Shouhong Ding, and Wenqiang Zhang. Content-based unrestricted ad- versarial attack. InThirty-seventh Conference on Neural In- formation Processing Systems, 2023. 2, 5

    work page 2023

  5. [5]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditioning, 2024

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditioning, 2024. 1, 2, 12

    work page 2024

  6. [6]

    Out of time: auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: auto- mated lip sync in the wild. InAsian conference on computer vision, pages 251–263. Springer, 2016. 7

    work page 2016

  7. [7]

    You said that?BMVC, 2017

    Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that?BMVC, 2017. 2

    work page 2017

  8. [8]

    Hallo2: Long-duration and high-resolution audio-driven portrait im- age animation.arXiv preprint arXiv:2410.07718, 2024

    Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait im- age animation.arXiv preprint arXiv:2410.07718, 2024. 1, 12

    arXiv

Show all 68 references
  1. [9]

    On the detection of digital face manipulation

    Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain. On the detection of digital face manipulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 5781–5790, 2020. 1

    2020

  2. [10]

    Boosting adversarial at- tacks with momentum

    Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial at- tacks with momentum. InProceedings of the IEEE con- ference on computer vision and pattern recognition, pages 9185–9193, 2018. 2

    2018

  3. [11]

    Evading defenses to transferable adversarial examples by translation-invariant attacks

    Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4312–4321, 2019. 2

    2019

  4. [12]

    Implicit generation and model- ing with energy based models

    Yilun Du and Igor Mordatch. Implicit generation and model- ing with energy based models. InAdvances in Neural Infor- mation Processing Systems. Curran Associates, Inc., 2019. 2

    2019

  5. [13]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021. 3

    2021

  6. [14]

    Navi- gation instruction generation with bev perception and large language models

    Sheng Fan, Rui Liu, Wenguan Wang, and Yi Yang. Navi- gation instruction generation with bev perception and large language models. InECCV, 2024. 2

    2024

  7. [15]

    Efficient emotional adaptation for audio-driven talking-head generation

    Yuan Gan, Zongxin Yang, Xihang Yue, Lingyun Sun, and Yi Yang. Efficient emotional adaptation for audio-driven talking-head generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22634– 22645, 2023. 1, 2, 7

    2023

  8. [16]

    Patch-wise attack for fooling deep neu- ral network

    Lianli Gao, Qilong Zhang, Jingkuan Song, Xianglong Liu, and Heng Tao Shen. Patch-wise attack for fooling deep neu- ral network. InComputer Vision–ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23–28, 2020, Pro- ceedings, Part XXVIII 16, pages 307–322. Springer, 2020. 2

    2020

  9. [17]

    Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.Advances in neural information processing systems, 27, 2014. 2

    2014

  10. [18]

    Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572, 2014

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572, 2014. 2

    2014 arXiv

  11. [19]

    Your classifier is secretly an energy based model and you should treat it like one.arXiv preprint arXiv:1912.03263,

    Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one.arXiv preprint arXiv:1912.03263,

    1912 arXiv

  12. [20]

    Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis

    Yudong Guo, Keyu Chen, Sen Liang, YongJin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 5784–5794, 2021. 1, 2

    2021

  13. [21]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium.Advances in neural information processing systems, 30, 2017. 6

    2017

  14. [22]

    Stochas- tic security: Adversarial defense using long-run dynamics of energy-based models.arXiv preprint arXiv:2005.13525,

    Mitch Hill, Jonathan Mitchell, and Song-Chun Zhu. Stochas- tic security: Adversarial defense using long-run dynamics of energy-based models.arXiv preprint arXiv:2005.13525,

    2005 arXiv

  15. [23]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  16. [24]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 1

    2024

  17. [25]

    Progressive growing of gans for improved quality, stability, and variation

    Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. InInternational Conference on Learning Rep- resentations, 2018. 5, 6, 7, 14

    2018

  18. [26]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding vari- ational bayes. InICLR, 2014. 3

    2014

  19. [27]

    Ad- versarial examples in the physical world

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Ad- versarial examples in the physical world. InArtificial in- telligence safety and security, pages 99–112. Chapman and Hall/CRC, 2018. 2

    2018

  20. [28]

    Unganable: Defending against gan- based face manipulation

    Zheng Li, Ning Yu, Ahmed Salem, Michael Backes, Mario Fritz, and Yang Zhang. Unganable: Defending against gan- based face manipulation. In32nd USENIX Security Sympo- sium (USENIX Security 23), pages 7213–7230, 2023. 1

    2023

  21. [29]

    Mist: Towards improved adversarial examples for diffusion models.arXiv preprint arXiv:2305.12683, 2023

    Chumeng Liang and Xiaoyu Wu. Mist: Towards improved adversarial examples for diffusion models.arXiv preprint arXiv:2305.12683, 2023. 1, 2, 5, 6, 7

    2023 arXiv

  22. [30]

    Adversarial example does good: Preventing paint- ing imitation from diffusion models via adversarial exam- ples

    Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yim- ing Xue, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Adversarial example does good: Preventing paint- ing imitation from diffusion models via adversarial exam- ples. InInternational Conference on Machine Learning, pa...

    2023

  23. [31]

    Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022. 2

    2022 arXiv

  24. [32]

    Fre- quency domain model augmentation for adversarial attack

    Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Fre- quency domain model augmentation for adversarial attack. InEuropean conference on computer vision, pages 549–566. Springer, 2022. 2

    2022

  25. [33]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    I Loshchilov. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  26. [34]

    Towards deep learning models resis- tant to adversarial attacks.arXiv preprint arXiv:1706.06083,

    Aleksander Madry. Towards deep learning models resis- tant to adversarial attacks.arXiv preprint arXiv:1706.06083,

    arXiv

  27. [35]

    Diffusion models for adversarial purification.arXiv preprint arXiv:2205.07460,

    Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification.arXiv preprint arXiv:2205.07460,

    arXiv

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 3

    2022

  29. [37]

    Faceforen- sics++: Learning to detect manipulated facial images

    Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Chris- tian Riess, Justus Thies, and Matthias Nießner. Faceforen- sics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1–11, 2019. 1

    2019

  30. [38]

    Dis- rupting deepfakes: Adversarial attacks against conditional image translation networks and facial manipulation systems

    Nataniel Ruiz, Sarah Adel Bargal, and Stan Sclaroff. Dis- rupting deepfakes: Adversarial attacks against conditional image translation networks and facial manipulation systems. InComputer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 236–

    2020

  31. [39]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500–...

    2023

  32. [40]

    Raising the cost of malicious ai-powered image editing.arXiv preprint arXiv:2302.06588,

    Hadi Salman, Alaa Khaddaj, Guillaume Leclerc, Andrew Ilyas, and Aleksander Madry. Raising the cost of malicious ai-powered image editing.arXiv preprint arXiv:2302.06588,

    arXiv

  33. [41]

    Defense-gan: protecting classifiers against adversarial attacks using generative models.arXiv preprint arXiv:1805.06605, 2018

    P Samangouei. Defense-gan: protecting classifiers against adversarial attacks using generative models.arXiv preprint arXiv:1805.06605, 2018. 2

    2018 arXiv

  34. [42]

    Jpeg compressed images can bypass protections against ai editing.arXiv preprint arXiv:2304.02234, 2023

    Pedro Sandoval-Segura, Jonas Geiping, and Tom Goldstein. Jpeg compressed images can bypass protections against ai editing.arXiv preprint arXiv:2304.02234, 2023. 2, 7, 8

    2023 arXiv

  35. [43]

    Difftalk: Crafting diffusion models for generalized audio-driven portraits animation

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In CVPR, 2023. 2

    2023

  36. [44]

    Detecting deep- fakes with self-blended images

    Kaede Shiohara and Toshihiko Yamasaki. Detecting deep- fakes with self-blended images. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18720–18729, 2022. 1

    2022

  37. [45]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020. 2, 5

    2010 arXiv

  38. [46]

    Generative modeling by esti- mating gradients of the data distribution.Advances in neural information processing systems, 32, 2019

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution.Advances in neural information processing systems, 32, 2019. 2

    2019

  39. [47]

    Pixeldefend: Leveraging genera- tive models to understand and defend against adversarial ex- amples.arXiv preprint arXiv:1710.10766, 2017

    Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Er- mon, and Nate Kushman. Pixeldefend: Leveraging genera- tive models to understand and defend against adversarial ex- amples.arXiv preprint arXiv:1710.10766, 2017. 2

    2017 arXiv

  40. [48]

    Diffused heads: Diffusion models beat gans on talking-face genera- tion

    Michał Stypułkowski, Konstantinos V ougioukas, Sen He, Maciej Zi˛ eba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face genera- tion. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5091–510...

    2024

  41. [49]

    Rethinking the up-sampling op- erations in cnn-based generative network for generalizable deepfake detection

    Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Rethinking the up-sampling op- erations in cnn-based generative network for generalizable deepfake detection. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, ...

    2024

  42. [50]

    Edtalk: Effi- cient disentanglement for emotional talking head synthesis

    Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. Edtalk: Effi- cient disentanglement for emotional talking head synthesis. InEuropean Conference on Computer Vision, pages 398–

  43. [51]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024. 2

    2024 arXiv

  44. [52]

    Ex- ploring clip for assessing the look and feel of images

    Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Ex- ploring clip for assessing the look and feel of images. InPro- ceedings of the AAAI Conference on Artificial Intelligence, pages 2555–2563, 2023. 6

    2023

  45. [53]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. InCVPR, 2021. 5, 6, 14

    2021

  46. [54]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animations,

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animations,

  47. [55]

    Improving transferabil- ity of adversarial examples with input diversity

    Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferabil- ity of adversarial examples with input diversity. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2730–2739, 2019. 2

    2019

  48. [56]

    Hallo: Hierarchical audio-driven vi- sual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Li- wei Zhang, Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven vi- sual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024. 2, 3, 6

    2024 arXiv

  49. [57]

    Vasa-1: Lifelike audio-driven talking faces generated in real time.arXiv preprint arXiv:2404.10667,

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time.arXiv preprint arXiv:2404.10667,

    arXiv

  50. [58]

    Pixel is a barrier: Diffusion models are more adversarially robust than we think.arXiv preprint arXiv:2404.13320, 2024

    Haotian Xue and Yongxin Chen. Pixel is a barrier: Diffusion models are more adversarially robust than we think.arXiv preprint arXiv:2404.13320, 2024. 2

    2024 arXiv

  51. [59]

    Toward effective protection against diffusion-based mimicry through score distillation

    Haotian Xue, Chumeng Liang, Xiaoyu Wu, and Yongxin Chen. Toward effective protection against diffusion-based mimicry through score distillation. InThe Twelfth Interna- tional Conference on Learning Representations, 2023. 1, 2, 5, 6, 7

    2023

  52. [60]

    Ucf: Uncovering common features for generalizable deep- fake detection

    Zhiyuan Yan, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Ucf: Uncovering common features for generalizable deep- fake detection. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 22412–22423,

  53. [61]

    Adversar- ial purification with score-based generative models

    Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversar- ial purification with score-based generative models. InIn- ternational Conference on Machine Learning, pages 12062– 12072. PMLR, 2021. 2

    2021

  54. [62]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2023

  55. [63]

    To- wards large yet imperceptible adversarial image perturba- tions with perceptual color distance

    Zhengyu Zhao, Zhuoran Liu, and Martha Larson. To- wards large yet imperceptible adversarial image perturba- tions with perceptual color distance. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1039–1048, 2020. 2

    2020

  56. [64]

    Zhengyue Zhao, Jinhao Duan, Kaidi Xu, Chenan Wang, Rui Zhang, Zidong Du, Qi Guo, and Xing Hu. Can protective perturbation safeguard personal data from being exploited by stable diffusion? InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pa...

    2024

  57. [65]

    Talking face generation by adversarially disentan- gled audio-visual representation

    Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentan- gled audio-visual representation. InProceedings of the AAAI Conference on Artificial Intelligence, pages 9299– 9306, 2019. 2

    2019

  58. [66]

    Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation

    Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4176–4186,

  59. [67]

    DiffAudio

    Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevar- ria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation.ACM Transactions on Graphics (TOG), 39(6):1–15, 2020. 1, 2 A. More Experiments A.1. More Implementation Details The resolution of ...

    2020

  60. [416]

    Springer, 2024. 1, 2

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.