Pith. sign in

REVIEW 4 major objections 6 minor 61 references

A lightweight projection trained against a generative attacker can make CLIP features hard to invert into images while keeping classification and VLM performance nearly intact.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 18:47 UTC pith:W3NB55PO

load-bearing objection Practical adversarial projection against diffusion inversion of CLIP tokens; classification results are solid, VLM privacy gains for the utility-preserving setting are modest. the 4 major comments →

arxiv 2607.04484 v1 pith:W3NB55PO submitted 2026-07-05 cs.CV

TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction

Nikos Athanasiou , Ilya A. Petrov , Angela Yao , Shugao Ma , Eric Sauser , Edoardo Remelli , Shreyas Hampali , Johannes Sch\"onberger
show 2 more authors
Fadime Sener Bugra Tekin
This is my paper
classification cs.CV
keywords privacy-preserving representationsfeature inversionCLIPgenerative attacksadversarial projectionvision-language modelsIP-Adapterreconstruction privacy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CLIP-style visual features are routinely shared or cached, yet strong diffusion models can invert them into realistic images that expose faces, rooms, and other private content. TrustCLIP inserts a small residual projection between a frozen vision encoder and any downstream head, then trains that projection so a feature-conditioned image generator reconstructs poorly while the usual task loss stays low. The premise is that faithful reconstruction needs instance-level detail that ordinary recognition and multimodal reasoning do not fully require, so those cues can be suppressed without collapsing the semantic subspace. On scene classification the method keeps top-1 accuracy within about half a percent of unprotected CLIP while sharply raising perceptual distance of reconstructions; the same layer inside a multimodal LLM preserves competitive scores on coarse semantic benchmarks. Anyone who ships or stores visual embeddings has a concrete reason to care: the defense optimizes against the generative threat that standard privacy proxies miss.

Core claim

TrustCLIP establishes that a lightweight residual projection, trained jointly against a frozen generative reconstruction attacker and a downstream task loss, can selectively degrade the fidelity of images recovered from CLIP features while retaining the semantic signals needed for image classification and multimodal large language models.

What carries the argument

The privacy projection P_θ — a shallow token-wise residual MLP initialized to identity — optimized under the joint objective of task loss minus λ times reconstruction loss through a frozen IP-Adapter / Stable Diffusion attacker.

Load-bearing premise

The method assumes that the fine visual detail needed for faithful image reconstruction and the high-level semantics needed for ordinary tasks sit in sufficiently different parts of CLIP features that one small network can suppress the first without wrecking the second.

What would settle it

Retrain a strong generative attacker on TrustCLIP features and check whether reconstruction fidelity (DreamSim / LPIPS / PSNR) returns to unprotected-CLIP levels while task accuracy remains high; if it does, selective suppression has failed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Client-server and retrieval pipelines can transmit projected CLIP features with substantially lower risk of realistic image recovery.
  • Privacy evaluation for vision features should measure generative inversion fidelity, not only attribute-classifier accuracy.
  • The same residual projection can sit in front of other consumers of tokenized vision features without redesigning the encoder.
  • Identity-initialized residual projections give a controllable privacy–utility knob for multimodal LLMs.
  • Fine-grained tasks such as OCR and counting degrade more than coarse semantic tasks under the defense.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If reconstruction-critical and task-critical information remain separable in other encoders, the same projection pattern could protect cached embeddings in multi-tenant storage without full differential privacy.
  • Training against one strong diffusion attacker may already strip enough reconstructible content that other inverter families (higher-capacity adapters, feed-forward decoders) also fail.
  • Adaptive per-token or per-task suppression strength could recover more of the fine-grained utility currently sacrificed on OCR and counting.
  • Extending the objective to video or multi-view features would close privacy surfaces that temporal consistency currently leaves open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. TrustCLIP inserts a lightweight residual projection between a frozen CLIP encoder and downstream heads, trained jointly against a task loss and a negative reconstruction loss under a frozen IP-Adapter + Stable Diffusion attacker. The projection is intended to suppress reconstruction-critical detail while preserving task semantics. The paper evaluates privacy via PSNR/SSIM/LPIPS/DreamSim under both fixed and adaptive (re-trained) attackers, and utility on SUN397 classification and LLaVA-SP-style VLM benchmarks. Classification shows strong privacy–utility trade-offs (Top-1 within ~0.5% of unprotected CLIP; DSIM improved ~2.5×). VLM results span a spectrum: an identity-initialized projection keeps competitive scores with modest privacy gains (DSIM 0.32→~0.39–0.43), while a standard MLP yields stronger privacy at larger utility cost. Ablations include matched-ℓ2 Gaussian noise, transfer/non-diffusion attackers, and per-category VLM breakdowns.

Significance. If the result holds, the work is a useful contribution to privacy for foundation-model vision features: it reframes the threat as generative inversion (rather than attribute classifiers), gives a practical drop-in projection, and evaluates under adaptive attackers and matched-noise controls that prior defenses often omit. Classification evidence is the strongest part and would already be of interest for on-device/cloud embedding pipelines. The VLM instantiation is practically important but currently less decisive; a clearer, better-supported privacy–utility story there would make the paper substantially more impactful for multimodal systems.

major comments (4)
  1. §3.1 and Tab. 3 / App. Tabs. A.2–A.5: the load-bearing separability premise is only weakly supported for the utility-preserving VLM configuration that carries the practical claim. Identity-init TrustLLaVA moves DSIM only from 0.32 to ~0.39–0.43 while dropping several benchmarks (e.g., VQAv2 79.2→76.3, GQA 62.7→58.0, MMEP 1470.7→1390.8). The MLP that reaches DSIM 0.59–0.62 loses substantial utility (MM-Vet ~26.5–27.8). Tab. 4’s selective degradation is consistent with incomplete suppression rather than clean separability of reconstruction- vs task-critical information in VLM tokens. Please either strengthen privacy under the competitive-utility setting (or quantify when the trade-off is acceptable) and/or reframe the VLM claim to match the modest measured gains.
  2. §4.2 and Tab. 3 bottom panel: adaptive-attacker privacy gains for TrustLLaVA are small in absolute terms (PSNR 10.57→10.42; DSIM 0.32→0.39). Given that the paper advertises MLLM pipelines as a primary setting, these numbers need tighter statistical reporting (sample size, confidence intervals) and a clearer statement of what constitutes a meaningful privacy improvement under generative inversion. Without that, the VLM privacy claim is hard to assess relative to classification (Tab. 1), where the effect is large.
  3. §3.2–3.3 and Eq. (2): training uses a frozen attacker while evaluation uses an adaptive one (Eq. 1)—good practice—but the paper still trains against a single attacker family (IP-Adapter + SD). Appendix E.1’s transfer/CNN tests help, yet they are secondary. For the central claim that the projection reduces recoverable information itself, please make the multi-family evaluation primary (or train against a mixture / alternate attacker) and report whether joint optimization against more than one decoder changes the privacy–utility frontier, especially for VLMs.
  4. §4.1 baselines: the manuscript argues prior defenses (DP-CLIP, NinjaDesc, SPAct) are not comparable, then relies mainly on unprotected CLIP and Gaussian noise. That is reasonable for a first generative-defense paper, but a best-effort reimplementation or adapted proxy (e.g., adversarial feature projection / information-bottleneck style objective on the same CLIP tokens under the same generative attacker) would better locate TrustCLIP on the privacy–utility frontier. At minimum, state explicitly what would break if those methods were ported, and add one non-noise learned baseline on SUN397.
minor comments (6)
  1. Fig. 1 / abstract claim of “competitive” VLM performance should be qualified next to the actual drops in Tab. 3 (several points on VQAv2/GQA/SQA/MMEP) so the figure does not oversell the utility side.
  2. Notation: L_rec uses α∥x−x̃∥_p + (1−α)LPIPS; state the default p and α consistently in the main text (α=0.5 appears only in the appendix).
  3. Tab. 2 reports TrustCLIP Top-1 85.1 vs CLIP 83.9 on a 500-image subset under a fixed attacker—clarify why accuracy exceeds the full-test Tab. 1 baseline and keep subset vs full-test comparisons separate.
  4. §4.2 “DSIM improves 2.5×” should cite the exact cells (e.g., 0.215→0.514/0.556) and note variance; DreamSim scale interpretation would help non-specialists.
  5. Typos/clarity: “class semanticsi.e.” (Fig. 1 caption); “discriminativemetrics” / “generativeleakage” missing spaces in the introduction; arXiv id line shows 2607.04484 with a 2026 date—verify metadata.
  6. Project page and interactive gallery are welcome; ensure the camera-ready points to frozen artifact hashes or a DOI for the adaptive-attacker checkpoints used in Figs. 4–5.

Circularity Check

0 steps flagged

No circular derivation: TrustCLIP is an empirical adversarial-training method evaluated on external benchmarks and adaptive attackers, not a claim that reduces to its inputs by construction.

full rationale

The paper proposes a residual projection P_θ trained with L = L_task − λ_rec L_rec against a frozen generative attacker, then re-evaluates under an adaptive attacker retrained on the protected feature distribution (Eq. 1) and reports standard external metrics (SUN397 Top-1/Top-5, VQAv2, GQA, POPE, MME, SEED, MM-Vet, plus PSNR/SSIM/LPIPS/DSIM). Nothing is defined in terms of a quantity later reported as a prediction; LPIPS appears in both training and evaluation only as the usual train-and-measure pattern, not a self-definitional loop. The separability premise in §3.1 is an empirical hypothesis tested by the tables, not a uniqueness theorem or fitted parameter renamed as a result. Citations (CLIP, IP-Adapter, LLaVA-SP, etc.) are external prior art, not load-bearing self-citations of uniqueness or ansatz. No fitted-input-as-prediction, no renaming of a known closed-form result, and no circular reduction of the central claim to its own inputs. Circularity score is therefore 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central empirical claim rests on a separability premise about CLIP feature space, a handful of scalar trade-off knobs, and the modeling choice that IP-Adapter + Stable Diffusion is a sufficiently strong and representative generative adversary. No new physical entities are postulated; the privacy projection is an engineered module, not an ontological invention.

free parameters (3)
  • λ_rec (reconstruction loss weight) = 0.25 (best classification trade-off); 0.001 (VLM)
    Scalar that trades privacy against utility; set by hand (0.25–1.0 for classification, 0.001 for VLMs) and ablated but not derived.
  • residual weight r and init scale ε = r=1.0, ε=0.1 preferred
    Control strength of the identity-initialized residual; chosen by ablation (r∈{0.95,1.0}, ε∈{0.01,0.1}).
  • α (pixel vs LPIPS mix in L_rec) = 0.5
    Balances ℓp and perceptual reconstruction terms; fixed at 0.5.
axioms (4)
  • domain assumption Reconstruction-critical and task-critical information inside CLIP features are sufficiently non-overlapping that a residual MLP can suppress the former while preserving the latter.
    Stated as the premise of §3.1; supported by category-level VLM breakdowns but not proved.
  • domain assumption IP-Adapter conditioned on Stable Diffusion is a near-optimal practical generative attacker for CLIP features.
    Used both as training-time adversary and evaluation attacker; justified by public availability and empirical strength, not by a completeness proof.
  • ad hoc to paper Freezing the attacker during projection training yields a stable and meaningful adversarial gradient.
    Chosen to avoid min-max instability (§3.2); alternative joint training is not explored.
  • standard math Data-processing inequality implies that reducing information in the projected features degrades every possible downstream inverter.
    Invoked in §3.2 to argue transfer to unseen attackers.
invented entities (1)
  • TrustCLIP privacy projection P_θ independent evidence
    purpose: Lightweight residual map that transforms encoder tokens before any downstream consumer so that generative inversion is degraded.
    Engineered module, not a new physical or mathematical object; independent evidence is the empirical privacy–utility tables.

pith-pipeline@v1.1.0-grok45 · 29079 in / 2822 out tokens · 26846 ms · 2026-07-11T18:47:43.235886+00:00 · methodology

0 comments
read the original abstract

Vision and vision-language models rely on high-level visual representations that are increasingly used across recognition, retrieval, and multimodal reasoning pipelines. However, recent advances in generative modeling have shown that such features can often be inverted, enabling realistic reconstructions of the underlying image and raising significant privacy risks. We revisit this problem through the lens of reconstruction and propose TrustCLIP, a reconstruction-driven framework that treats a feature-conditioned generator as an explicit privacy adversary. TrustCLIP learns a projection between encoder features and downstream modules that is explicitly optimized to degrade the reconstructions produced by generative attackers while retaining the necessary signals for downstream tasks. Unlike prior defenses that rely on discriminative privacy metrics, TrustCLIP directly optimizes against a generative reconstruction attacker, targeting a threat not captured by standard evaluation protocols. We demonstrate its effectiveness in both conventional classification and multimodal large language model pipelines. Across these settings, TrustCLIP consistently reduces the fidelity of generative inversions while maintaining downstream task performance. Project page: https://atnikos.github.io/trustclip/

Figures

Figures reproduced from arXiv: 2607.04484 by Angela Yao, Bugra Tekin, Edoardo Remelli, Eric Sauser, Fadime Sener, Ilya A. Petrov, Johannes Sch\"onberger, Nikos Athanasiou, Shreyas Hampali, Shugao Ma.

Figure 1
Figure 1. Figure 1: TrustCLIP ensures privacy for visual understanding tasks while pre [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: CLIP features reveal privacy information. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of TrustCLIP. A frozen vision encoder extracts feature tokens from an input image. The privacy projection Pθ transforms these features before they are consumed by any downstream module. During training (red path), a frozen generative attacker Gϕ attempts to reconstruct the original image from the projected features; the reconstruction loss gradient is backpropagated through the attacker to update … view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison on SUN397 [54]: original (top), reconstructions from the vanilla CLIP attacker (middle), and from the TrustCLIP attacker (bottom) under the adaptive threat model (§3). Vanilla CLIP reveals faces, pets, textures, and distinctive color patterns; TrustCLIP obfuscates these while preserving scene semantics and class-level structure. (DSIM 0.88 vs. 0.80), and a non-diffusion CNN decoder f… view at source ↗
Figure 5
Figure 5. Figure 5: Privacy spectrum on VLMs. Each column shows the same COCO validation image across four settings. Row 1: Ground truth. Row 2 (LLaVA): Reconstructions from unprotected CLIP features—faces, identities, objects, and scene details are recov￾ered with high fidelity. Row 3 (Trust-LLaVA): Identity-initialized projection—facial features become unrecognizable and fine detail is suppressed, while scene-level seman￾ti… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 1 canonical work pages

  1. [1]

    arXiv preprint arXiv:1612.00410 (2016)

    Alemi, A.A., Fischer, I., Dillon, J.V., Murphy, K.: Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 (2016)

    Pith/arXiv arXiv 2016

  2. [2]

    arXiv preprint arXiv:2309.16609 (2023)

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    Pith/arXiv arXiv 2023

  3. [3]

    arXiv preprint arXiv:2308.12966 (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  4. [4]

    In: 32nd USENIX Security Symposium (USENIX Security ’23)

    Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramèr, F., Balle, B., Ippolito, D., Wallace, E.: Extracting training data from diffusion models. In: 32nd USENIX Security Symposium (USENIX Security ’23). pp. 5253–5270 (2023)

    2023

  5. [5]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Chen, C., Liu, D., Shah, M., Xu, C.: Enhancing privacy-utility trade-offs to mit- igate memorization in diffusion models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 8182–8191 (2025)

    2025

  6. [6]

    arXiv preprint arXiv:2508.00756 (2025),https://arxiv.org/abs/2508

    Chen, Y., Wang, X., Wang, S., Ma, X.: Leakyclip: Extracting training data from CLIP. arXiv preprint arXiv:2508.00756 (2025),https://arxiv.org/abs/2508. 00756

    Pith/arXiv arXiv 2025

  7. [7]

    In: CVPR

    Dave, I.R., Chen, C., Shah, M.: SPAct: Self-supervised privacy preservation for action recognition. In: CVPR. pp. 20164–20173 (2022)

    2022

  8. [8]

    Advances in neural information processing systems34, 8780–8794 (2021)

    Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems34, 8780–8794 (2021)

    2021

  9. [9]

    In: CVPR

    Dosovitskiy, A., Brox, T.: Inverting visual representations with convolutional net- works. In: CVPR. pp. 4829–4837 (2016)

    2016

  10. [10]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Dusmanu, M., Schonberger, J.L., Sinha, S.N., Pollefeys, M.: Privacy-preserving image features via adversarial affine subspace embeddings. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14267– 14277 (2021)

    2021

  11. [11]

    arXiv preprint arXiv:1511.05897 (2015)

    Edwards, H., Storkey, A.: Censoring representations with an adversary. arXiv preprint arXiv:1511.05897 (2015)

    Pith/arXiv arXiv 2015

  12. [12]

    Electronics14(7), 1282 (2025).https://doi.org/10.3390/ electronics14071282

    Elhenawy, M., Ashqar, H.I., Rakotonirainy, A., Alhadidi, T.I., Jaber, A., Tami, M.A.: Vision-language models for autonomous driving: CLIP-based dynamic scene understanding. Electronics14(7), 1282 (2025).https://doi.org/10.3390/ electronics14071282

    2025

  13. [13]

    In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS)

    Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit con- fidence information and basic countermeasures. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS). pp. 1322– 1333 (2015)

    2015

  14. [14]

    arXiv preprint arXiv:2306.13394 (2023),https://arxiv.org/abs/2306.13394

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., Ji, R., Shan, C., He, R.: MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 (2023),https://arxiv.org/abs/2306.13394

    Pith/arXiv arXiv 2023

  15. [15]

    In: Advances in Neural Information Processing Systems

    Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., Isola, P.: Dream- sim: Learning new dimensions of human visual similarity using synthetic data. In: Advances in Neural Information Processing Systems. vol. 36, pp. 50742–50768 (2023)

    2023

  16. [16]

    In: CVPR

    Geppert, M., Larsson, V., Speciale, P., Schönberger, J.L., Pollefeys, M.: Privacy preserving localization and mapping from uncalibrated cameras. In: CVPR. pp. 3316–3326 (2021) TrustCLIP 17

    2021

  17. [17]

    Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: Elevating the role of image understanding in visual question an- swering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017),https://openaccess.thecvf.com/content_cvpr_ 2017/papers/Goyal_Making_the_V_CVPR_2017_paper.pdf

    2017

  18. [18]

    Hintersdorf, D., Struppek, L., Brack, M., Friedrich, F., Schramowski, P., Kersting, K.: Does clip know my face? Journal of Artificial Intelligence Research80, 1033– 1062 (2024)

    2024

  19. [19]

    In: NeurIPS

    Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS. vol. 33, pp. 6840–6851 (2020)

    2020

  20. [20]

    In: International conference on machine learning

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Ges- mundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: International conference on machine learning. pp. 2790–2799. PMLR (2019)

    2019

  21. [21]

    arXiv preprint arXiv:2306.08173 (2023)

    Huang, A., Liu, P., Nakada, R., Zhang, L., Zhang, W.: Safeguarding data in multimodal ai: A differentially private approach to clip training. arXiv preprint arXiv:2306.08173 (2023)

    Pith/arXiv arXiv 2023

  22. [22]

    Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual rea- soning and compositional question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019),https: //openaccess.thecvf.com/content_CVPR_2019/papers/Hudson_GQA_A_New_ Dataset_for_Real- World_Visual_Reasoning_and_Compositional_Ques...

    2019

  23. [23]

    com/mlfoundations/open_clip(2021)

    Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., et al.: OpenCLIP.https://github. com/mlfoundations/open_clip(2021)

    2021

  24. [24]

    In: CVPR (2024)

    Ilic, S., Meier, L., Pollefeys, M., Oswald, M.R.: Selective, interpretable and motion consistent privacy attribute obfuscation for action recognition. In: CVPR (2024)

    2024

  25. [25]

    Kazemi, H., Chegini, A., Geiping, J., Feizi, S., Goldstein, T.: What do we learn from inverting clip models? arXiv preprint arXiv:2403.02580 (2024)

    Pith/arXiv arXiv 2024

  26. [26]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Kim, B., Kim, H., Kim, K., Kim, S., Kim, J.: Learning not to learn: Training deep neural networks with biased data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9012–9020 (2019)

    2019

  27. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., Shan, Y.: SEED- Bench: Benchmarking multimodal large language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13299–13308 (2024),https://openaccess.thecvf.com/content/CVPR2024/ html / Li _ SEED - Bench _ Benchmarking _ Multimodal _ Large _ Lang...

    2024

  28. [28]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Li, M., Xu, X., Fan, H., Zhou, P., Liu, J., Liu, J.W., Li, J., Keppo, J., Shou, M.Z., Yan, S.: Stprivacy: Spatio-temporal privacy-preserving action recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5106–5115 (2023)

    2023

  29. [29]

    In: Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing (EMNLP) (2023), https://arxiv.org/abs/2305.10355

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hallucination in large vision-language models. In: Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing (EMNLP) (2023), https://arxiv.org/abs/2305.10355

    Pith/arXiv arXiv 2023

  30. [30]

    arXiv:2310.03744 (2023)

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. arXiv:2310.03744 (2023)

    Pith/arXiv arXiv 2023

  31. [31]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tun- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26296–26306 (2024) 18 N. Athanasiou et al

    2024

  32. [32]

    arXiv preprint arXiv:2304.08485 (2023),https://arxiv.org/abs/2304.08485

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023),https://arxiv.org/abs/2304.08485

    Pith/arXiv arXiv 2023

  33. [33]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Lou, H., Fan, C., Liu, Z., Wu, Y., Wang, X.: Llava-sp: Enhancing visual repre- sentation with visual spatial tokens for mllms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 22014–22024 (October 2025)

    2025

  34. [34]

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for sciencequestionanswering.In:AdvancesinNeuralInformationProcessingSystems (NeurIPS) (2022),https://arxiv.org/abs/2209.09513

    Pith/arXiv arXiv 2022

  35. [35]

    In: CVPR

    Mahendran, A., Vedaldi, A.: Understanding deep image representations by invert- ing them. In: CVPR. pp. 5188–5196 (2015)

    2015

  36. [36]

    arXiv preprint arXiv:2108.01073 (2021)

    Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: SDedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 (2021)

    Pith/arXiv arXiv 2021

  37. [37]

    Mescheder, L.M., Geiger, A., Nowozin, S.: Which training methods for gans do actually converge? In: International Conference on Machine Learning (2018), https://api.semanticscholar.org/CorpusID:3345317

    2018

  38. [38]

    In: CVPR

    Moon, H., Lee, C., Hong, J.H.: Efficient privacy-preserving visual localization using 3d ray clouds. In: CVPR. pp. 9773–9783 (2024)

    2024

  39. [39]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Mou, C., Wang, X., Xie, L., Zhang, J., Qi, Z., Shan, Y., Qie, X.: T2I-Adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 1965–1973 (2024)

    1965

  40. [40]

    In: CVPR

    Ng, T., Kim, H.J., Lee, V.T., DeTone, D., Yang, T.Y., Shen, T., Ilg, E., Balntas, V., Mikolajczyk, K., Sweeney, C.: Ninjadesc: Content-concealing visual descriptors via adversarial learning. In: CVPR. pp. 12787–12797 (2022)

    2022

  41. [41]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    Pith/arXiv arXiv 2023

  42. [42]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition

    Pittaluga, F., Koppal, S.J., Kang, S.B., Sinha, S.N.: Revealing scenes by inverting structure from motion reconstructions. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pp. 145–154 (2019)

    2019

  43. [43]

    In: International Conference of Machine Learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference of Machine Learning. pp. 8748–8763. PMLR (2021)

    2021

  44. [44]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10684– 10695 (June 2022)

    2022

  45. [45]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Shamshad, F., Naseer, M., Nandakumar, K.: CLIP2Protect: Protecting facial pri- vacy using text-guided makeup via adversarial latent search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20595–20605 (June 2023).https://doi.org/10.1109/CVPR52729.2023.01973

    doi:10.1109/cvpr52729.2023.01973 2023

  46. [46]

    IEEE Internet of Things Journal (2025)

    Sharshar, A., Khan, L.U., Ullah, W., Guizani, M.: Vision-language models for edge networks: A comprehensive survey. IEEE Internet of Things Journal (2025). https://doi.org/10.1109/JIOT.2025.3579032

    doi:10.1109/jiot.2025.3579032 2025

  47. [47]

    In: Proceedings of the TrustCLIP 19 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019),https://arxiv.org/abs/1904.08920

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards VQA models that can read. In: Proceedings of the TrustCLIP 19 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019),https://arxiv.org/abs/1904.08920

    Pith/arXiv arXiv 2019

  48. [48]

    In: ICLR (2021)

    Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score- based generative modeling through stochastic differential equations. In: ICLR (2021)

    2021

  49. [49]

    In: CVPR

    Speciale, P., Schönberger, J.L., Kang, S.B., Sinha, S.N., Pollefeys, M.: Privacy preserving image-based localization. In: CVPR. pp. 5493–5503 (2019)

    2019

  50. [50]

    In: International Conference of Machine Learning

    Struppek, L., Hintersdorf, D., Correia, A., Adler, A., Kersting, K.: Plug & play attacks: Towards robust and flexible model inversion attacks. In: International Conference of Machine Learning. pp. 20522–20545. PMLR (2022)

    2022

  51. [51]

    In: Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security

    Truex, S., Baracaldo, N., Anwar, A., Steinke, T., Ludwig, H., Zhang, R., Zhou, Y.: A hybrid approach to privacy-preserving federated learning. In: Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security. pp. 1–11 (2019)

    2019

  52. [52]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Wang, Z., Wang, H., Jin, S., Zhang, W., Hu, J., Wang, Y., Sun, P., Yuan, W., Liu, K., Ren, K.: Privacy-preserving adversarial facial features. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8212–8221 (June 2023).https://doi.org/10.1109/CVPR52729.2023.00794

    doi:10.1109/cvpr52729.2023.00794 2023

  53. [53]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops

    Wang, Z.W., Vineet, V., Pittaluga, F., Sinha, S.N., Cossairt, O., Bing Kang, S.: Privacy-preserving action recognition using coded aperture videos. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 0–0 (2019)

    2019

  54. [54]

    2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition pp

    Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition pp. 3485–3492 (2010),https://api. semanticscholar.org/CorpusID:1309931

    2010

  55. [55]

    In: Proceedings of the 33rd ACM In- ternational Conference on Multimedia (MM ’25)

    Xiu, K., Zhang, S.Q.: CapRecover: A cross-modality feature inversion attack framework on vision language models. In: Proceedings of the 33rd ACM In- ternational Conference on Multimedia (MM ’25). ACM, Dublin, Ireland (2025). https://doi.org/10.1145/3746027.3755203

    work page doi:10.1145/3746027.3755203 2025

  56. [56]

    arXiv preprint arXiv:2308.06721 (2023)

    Ye, H., Zhang, J., Liu, S., Han, X., Yang, W.: Ip-adapter: Text compati- ble image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 (2023)

    Pith/arXiv arXiv 2023

  57. [57]

    arXiv preprint arXiv:2308.02490 (2023),https://arxiv.org/abs/2308.02490

    Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., Wang, L.: MM- Vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023),https://arxiv.org/abs/2308.02490

    Pith/arXiv arXiv 2023

  58. [58]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023)

    2023

  59. [59]

    In: ICCV (2023)

    Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV (2023)

    2023

  60. [60]

    In: CVPR (2018)

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)

    2018

  61. [61]

    Zhou,Z.,Zhu,J.,Yu,F.,Li,X.,Peng,X.,Liu,T.,Han,B.:Modelinversionattacks: A survey of approaches and countermeasures. arXiv preprint arXiv:2411.10023 (2024) A Overview TrustCLIP preserves the privacy of vision–language representations while main- taining strong performance across diverse benchmarks, as illustrated in Figure 2. 20 N. Athanasiou et al. Table ...

    arXiv 2024