Pith. sign in

REVIEW 3 major objections 5 minor 40 references

Shielding faces from unified multimodal editing by breaking cross-branch agreement.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-04 04:12 UTC pith:VKV4KS35

load-bearing objection Genuine new mechanism for protecting facial identities against UMM editing, but the paper's own transfer results undercut the unqualified 'prevents UMMs' claim; the zero scores come from a small single-seed evaluation. the 3 major comments →

arxiv 2607.16898 v2 pith:VKV4KS35 submitted 2026-07-18 cs.CV cs.CLcs.CR

Cross-Branch Conflict as a Shield: Safeguarding Facial Identities in Unified Multimodal Image Editing

classification cs.CV cs.CLcs.CR
keywords adversarial protectionunified multimodal modelsfacial identityimage editingcross-branch conflictCKAprivacy defenseprompt-agnostic
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Unified multimodal models edit portraits by combining two visual pathways: a semantic ViT and a generative VAE. The paper argues that previous adversarial protections fail because they corrupt only one pathway, letting the other carry identity. It proposes CCS, which perturbs both pathways and adds a cross-branch conflict objective based on linear CKA to break their structural agreement. On the tested unified multimodal editor, this drives identity similarity to zero and near-zero instruction following, preventing unauthorized identity-preserving edits. The approach matters because it identifies a failure mode specific to unified multimodal editing and offers a prompt-agnostic protection.

Core claim

The central discovery is that in separate-encoder unified multimodal models, successful identity-preserving editing depends on the two visual branches agreeing on a consistent representation of the face. Single-branch attacks leave enough usable information in the other branch. CCS instead (1) pushes ViT and VAE features away from their clean counterparts and (2) uses a linear Centered Kernel Alignment objective to reduce the structural similarity between the two branches in the shared token space. With an 8/255 perturbation, the edited outputs no longer match the source identity (ISM=0.00) and satisfy almost none of the editing instructions across prompt sets, whereas single-branch baseline

What carries the argument

The load-bearing mechanism is a three-term adversarial objective applied to the shared token space of a UMM: a VAE-branch divergence loss, a ViT-branch divergence loss, and a cross-branch conflict loss computed with linear CKA after spatial interpolation of both feature maps to a common grid. The CKA term directly targets the structural agreement between the two branches, preventing the remaining information from being fused into a consistent facial representation. All parameters stay frozen; only the input perturbation is updated.

Load-bearing premise

The defender must have access to the exact image-encoding parts of the target model, or a surrogate model with nearly identical encoding parts; otherwise the perturbation may not transfer.

What would settle it

Take a protected image optimized against one UMM and run the same editing prompts on a different UMM with a different pair of visual encoders (no re-optimization). If the edited outputs retain the source identity at the same perturbation budget—as the paper itself reports for one transfer setting, where ISM stays 0.23—then the strong 'prevents identity-preserving edits' claim fails outside the grey-box setting. A second check: remove the CKA term and keep only the two divergence losses; the paper's own ablation shows ISM rises to 0.31 and Und-Score to 43.68, so a defender can verify that cross

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If CCS is correct, releasing a portrait with an 8/255 perturbation prevents a unified multimodal model from producing an identity-preserving edit, while leaving the image visually near-unchanged.
  • The result implies that protection methods for unified multimodal editing must target both visual branches and their interaction, not a single encoder.
  • Because CCS is prompt-agnostic, a single protected image defends against arbitrary future editing instructions without knowing what text will be used.
  • The finding that cross-branch structural agreement is necessary for editing suggests a general vulnerability surface: any downstream task relying on multi-branch fusion may be similarly disrupted by cross-branch conflict.
  • CCS also disrupts the understanding side of the model, so protected images resist both editing and malicious description.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's grey-box assumption is the main scope limit: the headline results require access to both visual encoders of the target or a close surrogate. On a different UMM architecture without re-optimization, identity preservation drops much less (ISM 0.23, Und-Score 81.23 at the same budget), so the strong claim should be read as applying to grey-box settings.
  • The cross-branch conflict idea could be tested as a general defense principle: any model that fuses two representation streams (e.g., audio-visual, depth-RGB) may be vulnerable to analogous CKA-based conflict attacks.
  • A testable extension is to measure whether the protection transfers to white-box access with a different token mapping or to models where the two branches share weights; the paper's mapping function and CKA computation may need adjustment.
  • The near-zero ISM and Und-Score values are on four hand-picked prompts plus 50 random prompts; an independent replication on a broader prompt distribution and multiple UMMs would clarify how universal the effect is.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CCS (Cross-Branch Conflict as a Shield), an adversarial perturbation method to protect facial images from unauthorized identity-preserving editing by unified multimodal models (UMMs). The motivation is that existing single-branch protections (VAE-only or ViT-only) fail because a UMM can recover identity information from the other visual pathway. CCS jointly optimizes three objectives: divergence of the VAE branch features, divergence of the ViT branch features, and a linear CKA-based loss that reduces structural agreement between the two branches. The method is evaluated on BAGEL-7B-MoT with four editing prompts on VGGFace2 and CelebA-HQ, reporting ISM=0.00, high FID, and low Und-Score relative to baselines. Ablations and robustness checks support the internal consistency of the design, and a transfer experiment to InternVL-U is reported in Appendix E.

Significance. The cross-branch protection idea is timely and sensible: unified editing models do rely on two visual encoders, and attacking both plus their agreement is a plausible way to prevent identity recovery. The paper includes a useful exploratory analysis of cross-branch CKA and shows that the two loss families are complementary (Table 5). If the results hold, CCS would be a strong grey-box protection for separate-encoder UMMs. The strengths are the clear problem formulation, the direct optimization scheme, and the inclusion of code in the supplementary material. However, the current evidence limits the scope of the claim: the strong quantitative results are obtained on a single architecture with encoder access, the evaluation uses a small single-seed set, and the Und-Score metric uses the same model family that is being attacked, which introduces a confound. The significance for general UMM protection is therefore conditional on additional validation.

major comments (3)
  1. [Defensive Assumptions; Abstract; Appendix E/Table 9] The abstract and introduction claim CCS 'effectively prevents UMMs' from recovering identity cues, but the evidence is grey-box and single-architecture. All main results are on BAGEL-7B-MoT, where the defender optimizes against both visual encoders of the target model. Appendix E/Table 9 shows the transfer result: at ε=8/255, perturbations optimized on BAGEL yield ISM=0.23 and Und-Score=81.23 on InternVL-U, meaning most transferred edits still preserve identity and follow instructions; at ε=16/255 the numbers remain weak (ISM=0.18, Und-Score=79.0). The paper's own data therefore contradicts an unqualified 'prevents UMMs' claim. The robustness analysis (Table 6) only tests image transformations on the same model, not architectural variation. Please qualify the claims to the grey-box, same-architecture setting, or add a cross-model experiment with re-optimization on the target encoders to
  2. [Experimental Settings; Tables 1–4; Appendix D.1] The headline results rest on a very small evaluation: 20 identities, 4 prompts, one surrogate model, and a single seed (Appendix D.1: 'All seeds are set to 2026'). No error bars, confidence intervals, or significance tests are reported. Because ISM=0.00 and Und-Score≈0 are extreme values, it is important to show variance across seeds and identity subsets. Without this, the reader cannot assess whether the gap over baselines is robust or an artifact of the particular seed/identity choice.
  3. [Appendix B (Und-Score); Tables 1–4] Und-Score is computed by using the understanding branch of BAGEL, the same model whose ViT branch is attacked, to judge whether the edited image satisfies each atomic instruction. This creates a confound: a low Und-Score may reflect that the judge's own visual encoder is degraded by the adversarial perturbation, rather than that the edited image objectively fails the instruction. Please validate Und-Score with an independent judge (e.g., a different VLM or human raters) or report agreement; as written, the instruction-following claim is entangled with the attack target.
minor comments (5)
  1. [Eq. (2) and Methodology] The mapping from branch names (generation/understanding) to symbols (F_z, F_v) is not defined explicitly before Eq. (2). Please state clearly that z denotes the VAE/generation branch and v denotes the ViT/understanding branch, and keep this consistent throughout.
  2. [Table 4] Table 4 has a formatting error: the PhotoGuard row prints '0.03 163.7843.4242.50' with no separators. Please fix the column alignment.
  3. [Appendix D.1] Please state how many random seeds were used and whether the reported numbers are averaged over seeds or taken from a single seed. The statement 'All seeds are set to 2026' suggests a single fix, which is not sufficient for the strong claims.
  4. [Table 5] The ablation row 'w/o branch-wise' removes both L_vae and L_vit. It would be informative to also report removing only one of the two branch-wise losses, to disentangle the contribution of each branch.
  5. [Figure 2] The x-axis label for panel (a) is not visible in the rendered figure; please add an explicit axis label and caption explanation for the probability density plot.

Circularity Check

0 steps flagged

No significant circularity: the optimized objectives and the reported protection metrics are distinct, and the only self-citation is non-load-bearing.

full rationale

CCS's derivation is empirical rather than formal: Eq. (11) defines L_total as a sum of branch-wise divergences and a CKA conflict term, and Eqs. (12)-(13) optimize only the input perturbation. The headline results (ISM=0.00, high FID/BRISQUE, near-zero Und-Score) are measured by external protocols: ISM via RetinaFace face detection and identity embeddings, FID via Inception features, BRISQUE as no-reference quality, and Und-Score by decomposing prompts into atomic questions answered by a UMM. None of these metrics is the optimized objective itself, and the ablation in Table 5 shows that removing the branch-wise or cross-branch terms leaves non-zero ISM, so the outcome is not forced by construction. The CKA statistic is used both to motivate the method and as one loss term, creating a mild confirmation bias but not an equivalence: reducing CKA does not by definition yield ISM=0. The only self-citation appears in 'features from two branches exhibit structural agreement (Li et al. 2026; Liu et al. 2026)', where Li et al. 2026 has overlapping authorship; however, Figure 2(a) independently shows the same phenomenon and no load-bearing uniqueness/forced-choice argument rests on the citation. Appendix E honestly reports limited transfer to InternVL-U (ISM=0.23, Und-Score=81.23 at epsilon=8/255), which narrows the generalization claim but is a scope/correctness issue, not circularity. Overall, the derivation chain is self-contained against external benchmarks and no step reduces to its own input.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim rests on tuned hyperparameters (loss weights, interpolation size, perturbation budget, and an unreported step size) and domain assumptions about UMM branch cooperation and the meaning of 'protection.' No new physical or architectural entities are introduced; the Und-Score is a new evaluation protocol, not an invented entity.

free parameters (4)
  • loss weights α, β, γ = α=10, β=0.2, γ=1.0
    Selected via hyperparameter analysis in Appendix G; Table 14 shows performance varies strongly with weights, so the central results depend on these tuned values.
  • CKA interpolation grid size = 16×16
    Used to align ViT and VAE tokens to 256 spatial tokens before computing CKA; Figure 8 shows low sensitivity, but it is still a hand-chosen hyperparameter.
  • perturbation budget ε = 8/255
    Standard adversarial budget, not derived; all headline results use this bound.
  • projected-gradient-ascent step size η = not reported
    Eq. (13) defines the update rule but never gives η, which affects convergence and reproducibility.
axioms (5)
  • domain assumption In separate-encoder UMMs, successful identity-preserving editing requires usable representations from both the ViT understanding branch and the VAE generation branch in the shared LLM-input token space.
    This motivates attacking both branches; it is supported only by the exploratory CKA/feature analysis in Figure 2, not by a derived law.
  • domain assumption Linear CKA computed on 16×16 bilinearly interpolated token grids is a causally adequate measure of cross-branch cooperation; minimizing it disrupts editing.
    Eqs. (4)-(7) and Appendix C use CKA as both the diagnostic and the objective; the causal link from CKA reduction to generation failure is assumed.
  • domain assumption The defender has grey-box access to the visual encoding modules of the target UMM or a close surrogate with similar visual encoding design.
    Stated in the Defensive Assumptions section; required to backpropagate into both branches. Appendix E shows transfer degrades substantially without this access.
  • domain assumption Treating output destruction (higher FID/BRISQUE, lower ISM/Und-Score) as successful protection is an acceptable operationalization.
    The Evaluation Metric section defines protection this way; it is a value choice about what defense means, not a mathematical necessity.
  • standard math Projected gradient ascent with ℓ∞ projection converges to a useful local maximizer of the composite loss.
    Eq. (13) relies on standard PGD behavior; no convergence certificate is given.

pith-pipeline@v1.3.0-alltime-deepseek · 15845 in / 16387 out tokens · 188817 ms · 2026-08-04T04:12:55.173383+00:00 · methodology

0 comments
read the original abstract

Unified multimodal models (UMMs) have recently demonstrated powerful instruction-based image editing capabilities, while also raising serious concerns about the unauthorized manipulation of personal portraits. We investigate a novel and practical problem: protecting facial identities against unauthorized editing of UMMs. Existing diffusion-based and VLM-based protection methods often become ineffective because they typically disrupt only a single visual branch. To understand this limitation, we conduct a feature-level analysis of the understanding and generation branches in unified image editing models. Our observations show that the structural agreement between these two branches is closely related to successful image editing. When only one branch is distorted, the model may still recover identity information from the other branch. Based on this, we propose Cross-Branch Conflict as a Shield (CCS), a unified adversarial protection framework. CCS jointly drives the ViT and VAE representations away from their clean counterparts. It also uses a linear Centered Kernel Alignment (CKA) objective to disrupt the structural consistency between the two branches. By degrading reliable identity information in both visual pathways and inducing incompatible cross-branch representations, CCS effectively prevents UMMs from recovering consistent facial identity cues during editing. Extensive experiments suggest that CCS consistently provides stronger protection in suppressing identity-preserving edits. Codes are in the supplementary material.

Figures

Figures reproduced from arXiv: 2607.16898 by Junxian Li, Rui Wang, Weiwei Tan, Yanjun Zhang, Yu Leo Zhang, Zhenhua Xu.

Figure 1
Figure 1. Figure 1: The overview of our proposed protection method [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The three sub figures showing results of exploratory experiments. (a) The probability density on CKA of the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of our method. Our method mainly consists of one cross-branch conflict part, and two branch-wise [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results of our method v.s. baselines. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Divergence induced by different adversarial protec [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Protection performance across diverse editing [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Effect of the interpolation size in the mapping [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative results of our method v.s. baselines. CCS demonstrates substantially stronger protection. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative results of our method on non-facial editing tasks. CCS also demonstrates stronger protection. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 7 linked inside Pith

  1. [1]

    Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pages=

    Disrupting diffusion-based inpainters with semantic digression , author=. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pages=

  2. [2]

    arXiv preprint arXiv:2505.17440 , year=

    Veattack: Downstream-agnostic vision encoder attack against large vision language models , author=. arXiv preprint arXiv:2505.17440 , year=

  3. [3]

    International Conference on Machine Learning , pages=

    Adversarial Example Does Good: Preventing Painting Imitation from Diffusion Models via Adversarial Examples , author=. International Conference on Machine Learning , pages=. 2023 , organization=

  4. [4]

    International Conference on Machine Learning , pages=

    PID: Prompt-Independent Data Protection Against Latent Diffusion Models , author=. International Conference on Machine Learning , pages=. 2024 , organization=

  5. [5]

    International Conference on Machine Learning , pages=

    Variance as a Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models , author=. International Conference on Machine Learning , pages=. 2025 , organization=

  6. [6]

    arXiv preprint arXiv:2305.12683 , year=

    Mist: Towards improved adversarial examples for diffusion models , author=. arXiv preprint arXiv:2305.12683 , year=

  7. [7]

    32nd USENIX Security Symposium (USENIX Security 23) , pages=

    Glaze: Protecting artists from style mimicry by \ Text-to-Image \ models , author=. 32nd USENIX Security Symposium (USENIX Security 23) , pages=

  8. [8]

    2024 IEEE symposium on security and privacy (SP) , pages=

    Nightshade: Prompt-specific poisoning attacks on text-to-image generative models , author=. 2024 IEEE symposium on security and privacy (SP) , pages=. 2024 , organization=

  9. [9]

    Advances in Neural Information Processing Systems , volume=

    Show-o2: Improved native unified multimodal models , author=. Advances in Neural Information Processing Systems , volume=

  10. [10]

    arXiv preprint arXiv:2501.17811 , year=

    Janus-pro: Unified multimodal understanding and generation with data and model scaling , author=. arXiv preprint arXiv:2501.17811 , year=

  11. [11]

    arXiv preprint arXiv:2505.14683 , year=

    Emerging properties in unified multimodal pretraining , author=. arXiv preprint arXiv:2505.14683 , year=

  12. [12]

    arXiv preprint arXiv:2603.09877 , year=

    Internvl-u: Democratizing unified multimodal models for understanding, reasoning, generation and editing , author=. arXiv preprint arXiv:2603.09877 , year=

  13. [13]

    arXiv preprint arXiv:2605.12500 , year=

    Sensenova-u1: Unifying multimodal understanding and generation with neo-unify architecture , author=. arXiv preprint arXiv:2605.12500 , year=

  14. [14]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Unigame: Turning a unified multimodal model into its own adversary , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  15. [15]

    arXiv preprint arXiv:2605.12309 , year=

    G ^2 TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models , author=. arXiv preprint arXiv:2605.12309 , year=

  16. [16]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Tuna: Taming unified visual representations for native unified multimodal models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  17. [17]

    2023 IEEE/CVF International Conference on Computer Vision (ICCV) , year=

    Anti-DreamBooth: Protecting users from personalized text-to-image synthesis , author=. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , year=

  18. [18]

    ArXiv , year=

    GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium , author=. ArXiv , year=

  19. [19]

    No-Reference Image Quality Assessment in the Spatial Domain , year=

    Mittal, Anish and Moorthy, Anush Krishna and Bovik, Alan Conrad , journal=. No-Reference Image Quality Assessment in the Spatial Domain , year=

  20. [20]

    2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) , year=

    VGGFace2: A Dataset for Recognising Faces across Pose and Age , author=. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) , year=

  21. [21]

    ArXiv , year=

    Progressive Growing of GANs for Improved Quality, Stability, and Variation , author=. ArXiv , year=

  22. [22]

    ArXiv , year=

    Visual Instruction Tuning , author=. ArXiv , year=

  23. [23]

    ArXiv , year=

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models , author=. ArXiv , year=

  24. [24]

    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    High-Resolution Image Synthesis with Latent Diffusion Models , author=. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

  25. [25]

    ArXiv , year=

    Denoising Diffusion Probabilistic Models , author=. ArXiv , year=

  26. [26]

    2025 IEEE Symposium on Security and Privacy (SP) , year=

    Towards Reliable Verification of Unauthorized Data Usage in Personalized Text-to-Image Diffusion Models , author=. 2025 IEEE Symposium on Security and Privacy (SP) , year=

  27. [27]

    CoRR , year=

    Intriguing properties of neural networks , author=. CoRR , year=

  28. [28]

    CoRR , year=

    Explaining and Harnessing Adversarial Examples , author=. CoRR , year=

  29. [29]

    ArXiv , year=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. ArXiv , year=

  30. [30]

    CoRR , year=

    Auto-Encoding Variational Bayes , author=. CoRR , year=

  31. [31]

    arXiv preprint arXiv:2409.12191 , year=

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , author=. arXiv preprint arXiv:2409.12191 , year=

  32. [32]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Break the visual perception: Adversarial attacks targeting encoded visual tokens of large vision-language models , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=

  33. [33]

    2024 IEEE Symposium on Security and Privacy (SP) , pages=

    Transferable multimodal attack on vision-language pre-training models , author=. 2024 IEEE Symposium on Security and Privacy (SP) , pages=. 2024 , organization=

  34. [34]

    Advances in Neural Information Processing Systems , volume=

    Vlattack: Multimodal adversarial attacks on vision-language tasks via pre-trained models , author=. Advances in Neural Information Processing Systems , volume=

  35. [35]

    The Journal of Machine Learning Research , volume=

    Algorithms for learning kernels based on centered alignment , author=. The Journal of Machine Learning Research , volume=. 2012 , publisher=

  36. [36]

    ArXiv , year=

    I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing , author=. ArXiv , year=

  37. [37]

    ArXiv , year=

    ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area , author=. ArXiv , year=

  38. [38]

    ArXiv , year=

    HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing , author=. ArXiv , year=

  39. [39]

    ArXiv , year=

    CLIPScore: A Reference-free Evaluation Metric for Image Captioning , author=. ArXiv , year=

  40. [40]

    arXiv preprint arXiv:2302.06588 , year=

    Raising the cost of malicious ai-powered image editing , author=. arXiv preprint arXiv:2302.06588 , year=