{"id":"9459002b-8492-427c-989c-953c1d644581","arxiv_id":"2607.04484","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"A lightweight adversarially trained projection degrades generative inversion of CLIP features while preserving most classification and VLM utility.","lead":"TrustCLIP inserts a small learned projection after CLIP that is trained to wreck diffusion-based image reconstructions while keeping classification and VLM accuracy nearly intact. It matters because CLIP embeddings are widely shipped off-device and can currently be inverted into recognizable faces and private scenes.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The separability premise is only weakly tested for the utility-preserving VLM setting that carries the paper's practical claim.","rationale":"The reader correctly flags the non-overlap assumption in §3.1 as the weakest link and already notes that VLM privacy gains for the identity-init variant are modest. Classification evidence is solid and the adaptive/unseen-attacker checks (Tab. A.6, §4.2) reduce the risk of pure attacker overfitting. The remaining load-bearing uncertainty is whether the practical (utility-preserving) VLM configuration actually removes reconstructible content or merely makes the current attacker slightly worse. The proposed longer adaptive retrain is a direct, low-cost falsifier of that residual claim; if it fails, the paper’s strongest advertised trade-off (competitive VLM + meaningful generative privacy) weakens and the verdict stays CONDITIONAL pending either stronger privacy numbers or clearer guidance that only the high-privacy/low-utility MLP regime works. No mathematical error or circularity; the concern is empirical and already partially anticipated by the reader.","tokens_in":24891,"tokens_out":596,"duration_ms":6398,"concrete_test":"Freeze the identity-init TrustLLaVA projection after training, then retrain only the adaptive IP-Adapter (Eq. 1) for 2–3× longer / higher capacity on the protected features and re-measure DSIM/LPIPS on the same COCO/VLM images used for Tab. 3. If DSIM falls back toward the unprotected 0.32 baseline while VLM scores stay high, the residual projection has not removed reconstructible information and the separability claim fails for the utility-preserving regime.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on §3.1: reconstruction-critical detail (textures, identity cues) and task-critical semantics (categories, coarse layout) are sufficiently non-overlapping that a residual projection can suppress the former without destroying the latter. Classification (Tab. 1) supports this—Top-1 within 0.5%, DSIM ~2.5×. For VLMs the utility-preserving (identity-init) variant that keeps competitive scores only moves DSIM from 0.32 to 0.39–0.43 (Tab. 3 / A.5), while the MLP that reaches DSIM 0.59–0.62 loses substantial utility (MM-Vet 26.5–27.8). Tab. 4 already shows the expected selective degradation (fine-grained –11% vs semantic –2%), so the modest privacy numbers for the practical configuration may simply reflect incomplete suppression rather than clean separability. If the two information sets largely coincide inside CLIP tokens used by VLMs, the headline privacy–utility trade-off does not hold for the setting the paper advertises.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"TrustCLIP inserts a lightweight residual projection between a frozen CLIP encoder and downstream heads, trained jointly against a task loss and a negative reconstruction loss under a frozen IP-Adapter + Stable Diffusion attacker. The projection is intended to suppress reconstruction-critical detail while preserving task semantics. The paper evaluates privacy via PSNR/SSIM/LPIPS/DreamSim under both fixed and adaptive (re-trained) attackers, and utility on SUN397 classification and LLaVA-SP-style VLM benchmarks. Classification shows strong privacy–utility trade-offs (Top-1 within ~0.5% of unprotected CLIP; DSIM improved ~2.5×). VLM results span a spectrum: an identity-initialized projection keeps competitive scores with modest privacy gains (DSIM 0.32→~0.39–0.43), while a standard MLP yields stronger privacy at larger utility cost. Ablations include matched-ℓ2 Gaussian noise, transfer/non-diffusion attackers, and per-category VLM breakdowns.","tokens_in":25210,"tokens_out":1454,"duration_ms":19263,"significance":"If the result holds, the work is a useful contribution to privacy for foundation-model vision features: it reframes the threat as generative inversion (rather than attribute classifiers), gives a practical drop-in projection, and evaluates under adaptive attackers and matched-noise controls that prior defenses often omit. Classification evidence is the strongest part and would already be of interest for on-device/cloud embedding pipelines. The VLM instantiation is practically important but currently less decisive; a clearer, better-supported privacy–utility story there would make the paper substantially more impactful for multimodal systems.","major_comments":[{"comment":"§3.1 and Tab. 3 / App. Tabs. A.2–A.5: the load-bearing separability premise is only weakly supported for the utility-preserving VLM configuration that carries the practical claim. Identity-init TrustLLaVA moves DSIM only from 0.32 to ~0.39–0.43 while dropping several benchmarks (e.g., VQAv2 79.2→76.3, GQA 62.7→58.0, MMEP 1470.7→1390.8). The MLP that reaches DSIM 0.59–0.62 loses substantial utility (MM-Vet ~26.5–27.8). Tab. 4’s selective degradation is consistent with incomplete suppression rather than clean separability of reconstruction- vs task-critical information in VLM tokens. Please either strengthen privacy under the competitive-utility setting (or quantify when the trade-off is acceptable) and/or reframe the VLM claim to match the modest measured gains.","section":null},{"comment":"§4.2 and Tab. 3 bottom panel: adaptive-attacker privacy gains for TrustLLaVA are small in absolute terms (PSNR 10.57→10.42; DSIM 0.32→0.39). Given that the paper advertises MLLM pipelines as a primary setting, these numbers need tighter statistical reporting (sample size, confidence intervals) and a clearer statement of what constitutes a meaningful privacy improvement under generative inversion. Without that, the VLM privacy claim is hard to assess relative to classification (Tab. 1), where the effect is large.","section":null},{"comment":"§3.2–3.3 and Eq. (2): training uses a frozen attacker while evaluation uses an adaptive one (Eq. 1)—good practice—but the paper still trains against a single attacker family (IP-Adapter + SD). Appendix E.1’s transfer/CNN tests help, yet they are secondary. For the central claim that the projection reduces recoverable information itself, please make the multi-family evaluation primary (or train against a mixture / alternate attacker) and report whether joint optimization against more than one decoder changes the privacy–utility frontier, especially for VLMs.","section":null},{"comment":"§4.1 baselines: the manuscript argues prior defenses (DP-CLIP, NinjaDesc, SPAct) are not comparable, then relies mainly on unprotected CLIP and Gaussian noise. That is reasonable for a first generative-defense paper, but a best-effort reimplementation or adapted proxy (e.g., adversarial feature projection / information-bottleneck style objective on the same CLIP tokens under the same generative attacker) would better locate TrustCLIP on the privacy–utility frontier. At minimum, state explicitly what would break if those methods were ported, and add one non-noise learned baseline on SUN397.","section":null}],"minor_comments":[{"comment":"Fig. 1 / abstract claim of “competitive” VLM performance should be qualified next to the actual drops in Tab. 3 (several points on VQAv2/GQA/SQA/MMEP) so the figure does not oversell the utility side.","section":null},{"comment":"Notation: L_rec uses α∥x−x̃∥_p + (1−α)LPIPS; state the default p and α consistently in the main text (α=0.5 appears only in the appendix).","section":null},{"comment":"Tab. 2 reports TrustCLIP Top-1 85.1 vs CLIP 83.9 on a 500-image subset under a fixed attacker—clarify why accuracy exceeds the full-test Tab. 1 baseline and keep subset vs full-test comparisons separate.","section":null},{"comment":"§4.2 “DSIM improves 2.5×” should cite the exact cells (e.g., 0.215→0.514/0.556) and note variance; DreamSim scale interpretation would help non-specialists.","section":null},{"comment":"Typos/clarity: “class semanticsi.e.” (Fig. 1 caption); “discriminativemetrics” / “generativeleakage” missing spaces in the introduction; arXiv id line shows 2607.04484 with a 2026 date—verify metadata.","section":null},{"comment":"Project page and interactive gallery are welcome; ensure the camera-ready points to frozen artifact hashes or a DOI for the adaptive-attacker checkpoints used in Figs. 4–5.","section":null}],"recommendation":"major_revision","confidential_remarks":"The classification half is solid and publishable with modest polishing; the VLM half is the weak link relative to how prominently MLLMs are sold in the abstract and intro. I would not reject on novelty—the generative-adversary framing is timely—but I would not accept until the VLM privacy claim is either strengthened or honestly scoped. No integrity concerns; methodology (adaptive attackers, matched-ℓ2 noise) is above average for this area."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: they put a residual MLP between frozen CLIP and the downstream head, train it jointly against a frozen IP-Adapter reconstruction loss plus the task loss, and show that generative inversion gets worse while classification stays almost intact. That is a concrete recipe for a real systems problem (features leaving the device, retrieval caches, medical pipelines), and they evaluate it the right way—against an adaptive attacker retrained on the protected features, plus a matched-ℓ2 Gaussian control and a couple of transfer attackers.\n\nWhat is actually new is the adversary choice. Prior privacy projections and inversion-resistant descriptors mostly fight attribute classifiers or sparse geometric descriptors. Here the training-time adversary is a diffusion feature-conditioned generator on dense language-aligned tokens. Classification on SUN397 is the cleanest evidence: Top-1 within ~0.5% of unprotected CLIP, DSIM roughly 2.5× worse for the attacker, and noise baselines collapse utility long before they match the privacy numbers. The adaptive-attacker and CNN-decoder transfer checks help; they are not just overfitting one inverter. The identity-init residual design and the joint-optimization argument are sensible engineering.\n\nThe soft spot is the VLM claim that carries the practical pitch. The identity-initialized TrustLLaVA that keeps competitive scores only moves DSIM from 0.32 to ~0.39–0.43; the MLP that reaches 0.59–0.62 loses real utility (MM-Vet ~27). Table 4 already shows the expected pattern—semantic categories hold, fine-grained (count/OCR/loc) drop—so the modest privacy numbers for the useful configuration look more like incomplete suppression than clean separability of reconstruction-critical vs task-critical information inside VLM tokens. That does not kill the paper; it just means the headline privacy–utility story is stronger for classification than for the multimodal setting they advertise. Code is promised, not shipped; free parameters (λ_rec, residual weight, init scale) are ablated but still free.\n\nThis is for people building or auditing systems that ship CLIP embeddings. It deserves a serious referee. I would engage with it, cite the classification setup and the generative-adversary framing, and treat the VLM numbers as a first cut rather than settled.","headline":"Practical adversarial projection against diffusion inversion of CLIP tokens; classification results are solid, VLM privacy gains for the utility-preserving setting are modest.","tokens_in":25797,"tokens_out":558,"would_cite":true,"duration_ms":7288,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A lightweight projection trained against a generative attacker can make CLIP features hard to invert into images while keeping classification and VLM performance nearly intact.","keywords":["privacy-preserving representations","feature inversion","CLIP","generative attacks","adversarial projection","vision-language models","IP-Adapter","reconstruction privacy"],"falsifier":"Retrain a strong generative attacker on TrustCLIP features and check whether reconstruction fidelity (DreamSim / LPIPS / PSNR) returns to unprotected-CLIP levels while task accuracy remains high; if it does, selective suppression has failed.","tokens_in":25826,"feed_emoji":"🔒","tokens_out":815,"duration_ms":16895,"temperature":0.7,"pith_summary":"CLIP-style visual features are routinely shared or cached, yet strong diffusion models can invert them into realistic images that expose faces, rooms, and other private content. TrustCLIP inserts a small residual projection between a frozen vision encoder and any downstream head, then trains that projection so a feature-conditioned image generator reconstructs poorly while the usual task loss stays low. The premise is that faithful reconstruction needs instance-level detail that ordinary recognition and multimodal reasoning do not fully require, so those cues can be suppressed without collapsing the semantic subspace. On scene classification the method keeps top-1 accuracy within about half a percent of unprotected CLIP while sharply raising perceptual distance of reconstructions; the same layer inside a multimodal LLM preserves competitive scores on coarse semantic benchmarks. Anyone who ships or stores visual embeddings has a concrete reason to care: the defense optimizes against the generative threat that standard privacy proxies miss.","feed_headline":"Projection blocks image recovery from CLIP features","feed_subtitle":"Keeps classification and VLM scores high while diffusion attackers recover far less detail.","key_machinery":"The privacy projection P_θ — a shallow token-wise residual MLP initialized to identity — optimized under the joint objective of task loss minus λ times reconstruction loss through a frozen IP-Adapter / Stable Diffusion attacker.","core_discovery":"TrustCLIP establishes that a lightweight residual projection, trained jointly against a frozen generative reconstruction attacker and a downstream task loss, can selectively degrade the fidelity of images recovered from CLIP features while retaining the semantic signals needed for image classification and multimodal large language models.","pith_inferences":["If reconstruction-critical and task-critical information remain separable in other encoders, the same projection pattern could protect cached embeddings in multi-tenant storage without full differential privacy.","Training against one strong diffusion attacker may already strip enough reconstructible content that other inverter families (higher-capacity adapters, feed-forward decoders) also fail.","Adaptive per-token or per-task suppression strength could recover more of the fine-grained utility currently sacrificed on OCR and counting.","Extending the objective to video or multi-view features would close privacy surfaces that temporal consistency currently leaves open."],"forward_implications":["Client-server and retrieval pipelines can transmit projected CLIP features with substantially lower risk of realistic image recovery.","Privacy evaluation for vision features should measure generative inversion fidelity, not only attribute-classifier accuracy.","The same residual projection can sit in front of other consumers of tokenized vision features without redesigning the encoder.","Identity-initialized residual projections give a controllable privacy–utility knob for multimodal LLMs.","Fine-grained tasks such as OCR and counting degrade more than coarse semantic tasks under the defense."],"fun_headline_variants":["TrustCLIP projection blocks generative recovery of CLIP features","Residual projection degrades inversion of private visual features","Adversarial projection stops image recovery from CLIP embeddings","TrustCLIP foils generative attackers while keeping task accuracy","Lightweight projection reduces fidelity of CLIP feature inversions"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The method assumes that the fine visual detail needed for faithful image reconstruction and the high-level semantics needed for ordinary tasks sit in sufficiently different parts of CLIP features that one small network can suppress the first without wrecking the second.","fun_headline_variants_meta":{"raw":{"variants":["TrustCLIP projection blocks generative recovery of CLIP features","Residual projection degrades inversion of private visual features","Adversarial projection stops image recovery from CLIP embeddings","TrustCLIP foils generative attackers while keeping task accuracy","Lightweight projection reduces fidelity of CLIP feature inversions"]},"model":"grok-4.5","effort":"low","cost_usd":0.004934,"raw_usage":{"total_tokens":1281,"prompt_tokens":688,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":49340000,"prompt_tokens_details":{"text_tokens":688,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":518,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":688,"tokens_out":75,"duration_ms":4657,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T18:47:43.235886+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Retrain a strong generative attacker on TrustCLIP features and check whether reconstruction fidelity (DreamSim / LPIPS / PSNR) returns to unprotected-CLIP levels while task accuracy remains high; if it does, selective suppression has failed.","supporting_citations":[],"review_version":1}