{"id":"db4afe63-626e-4dae-817d-c86152fed7aa","arxiv_id":"2607.04339","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A modality-agnostic black-box MIA models embeddings of model-generated outputs and non-members as Gaussians and decides membership by likelihood-ratio test, outperforming single-modality baselines especially under zero knowledge.","lead":"The paper gives one black-box membership-inference method that works on text-to-text, text-to-image, and image-to-text generative models by comparing a sample’s embedding likelihood under model outputs versus non-member data. It matters because real systems mix modalities, so privacy audits need a single tool rather than three incompatible ones.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Gaussian embedding approximation with tiny samples is the soft spot, but ablations already bound its practical impact.","rationale":"The reader correctly isolates the distributional approximation (empirical mean/covariance + Gaussian LRT under tiny samples) as the weakest link supporting the strongest claim. The paper’s own theory (Section V) and the decision rule (Eq. 4) make this assumption explicit; the empirical tables show the method works, but do not isolate whether it works because of or in spite of the Gaussian-small-sample premise. Because the manuscript already contains extensive ablations on sample size, encoder choice, geometric median, distance metrics, and threshold, the concern is real yet already partially stress-tested; a single targeted non-parametric / whitened re-run would settle residual doubt without requiring a full re-evaluation. No stronger internal inconsistency or missing control appears. Hence the CONDITIONAL verdict and HIGH confidence remain appropriate; no adjustment is warranted.","tokens_in":26708,"tokens_out":630,"duration_ms":8749,"concrete_test":"For one representative setting (e.g., SD1.5+MSCOCO, partial-knowledge, BLIP embeddings), recompute the LRT after (i) PCA-whitening the embeddings to a well-conditioned subspace of rank r ≪ d chosen so that the sample covariance is invertible, and (ii) replacing the parametric Gaussian score with a non-parametric density estimate (k-NN or kernel density) on the same embeddings. If either change moves ASR/AUC by more than 5 points relative to Table I, the Gaussian-small-sample approximation is load-bearing and the claim weakens; otherwise the concern is largely academic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Steps 2–3 and Section V: that the empirical mean and covariance of only 1–3 % of |D_tune| embeddings under a fixed off-the-shelf encoder are statistically sufficient for the Neyman–Pearson LRT (Eq. 4, threshold 0) under a multivariate Gaussian model (Eq. 3) to remain near-optimal. The paper invokes LLN / CLT / Hoeffding for consistency of μ and Σ, yet never reports the embedding dimension d, the condition number of Σ, or any normality diagnostic. When d is large (typical for DistilBERT / BLIP / MiniLM), the sample covariance estimated from a few hundred points is singular or ill-conditioned, so the quadratic form in Eq. 3 is unstable. The zero-knowledge setting compounds the issue because D_aux is a noisy perturbation of D_syn; any residual correlation between the two clouds can shrink the KL gap that Section V claims drives separability. Ablations (Figs. 5–7, Tables III–V, geometric-median vs mean) show robustness to sample size and some encoders, but they do not directly test whether the Gaussian-plus-small-sample premise itself is the source of residual error versus the baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a unified black-box membership inference attack that applies across text-to-text (LLMs), text-to-image (diffusion), and image-to-text (VLM) generative models. The core idea is that likelihood-based training makes a model’s output distribution approximate its training (or fine-tuning) distribution; the adversary therefore estimates multivariate Gaussian distributions of embeddings of model-generated samples versus auxiliary non-member samples (real data under partial knowledge, or lightly perturbed synthetic samples under zero knowledge) and decides membership by a log-likelihood-ratio test with threshold zero (Eqs. 3–4). Extensive experiments under both threat models, on fine-tuning and pre-training data, and against modality-specific SOTA baselines (Tables I–II, XI) show competitive or superior ASR/AUC/TPR@1%FPR, with ablations on sample size, extractor choice, distribution estimator, distance metrics, threshold, classifier alternative, and DP-SGD defense.","tokens_in":27107,"tokens_out":1074,"duration_ms":9424,"significance":"If the empirical claims hold, the work supplies the first practical cross-modal MIA that does not require modality-specific signals, shadow models, or white-box access. The zero-knowledge setting and the extension to pre-training data are especially useful for real-world auditing of closed generative systems. The theoretical section correctly links likelihood training to KL minimization and shows that the zero-threshold LRT is Bayes-optimal under equal priors and Gaussian class-conditionals; the extensive ablation suite (dataset size, extractors, geometric median, distance vs likelihood, threshold, classifier, DP-SGD) gives the community concrete evidence of robustness and of the residual utility–privacy trade-off under DP-SGD. These contributions are of clear interest to the privacy and generative-model communities.","major_comments":[{"comment":"Section V and Steps 2–3 rest on the claim that the empirical mean and covariance of only 1–3 % of |D_tune| embeddings under a fixed off-the-shelf encoder are statistically sufficient for the Neyman–Pearson LRT (Eq. 4) under the multivariate Gaussian model (Eq. 3). The manuscript never reports the embedding dimension d, the condition number of Σ, or any normality diagnostic. For typical DistilBERT/BLIP/MiniLM dimensions the sample covariance estimated from a few hundred points is singular or ill-conditioned, rendering the quadratic form in Eq. 3 unstable. A short analysis (regularization of Σ, PCA projection, or a normality check) is needed to confirm that residual error versus the baselines is not an artifact of this premise.","section":"Section V, Steps 2–3, Eq. (3)"},{"comment":"In the zero-knowledge construction (Section IV, Step 1 and Eq. 5) D_aux is obtained by light noise or paraphrasing of D_syn. Any residual correlation between the two clouds shrinks the KL gap that Section V claims drives separability. The paper should quantify the empirical KL (or a proxy) between the estimated (μ_aux, Σ_aux) and (μ_syn, Σ_syn) and show that it remains large enough for the observed AUCs in Table II; otherwise the zero-knowledge gains may be overstated.","section":"Section IV, Step 1; Section V (ii)"}],"minor_comments":[{"comment":"Figure 4 (t-SNE) is only qualitative; a quantitative distance (e.g., MMD or estimated KL) between the four clouds would strengthen the claim that generated data hug the member distribution more tightly than test or external non-members.","section":"Figure 4"},{"comment":"Tables III–V show that extractor choice can drop ASR by 20–30 points (e.g., MPNet on LLaVA+COCO). A short practical guideline for selecting E would help readers reproduce the best-case numbers.","section":"Tables III–V"},{"comment":"Notation occasionally re-uses x for both the input–output pair and the ground-truth output (footnote after Eq. 3); a consistent convention would improve readability.","section":"Section III, Step 4"},{"comment":"The DP-SGD hyper-parameters (Table XII) produce quite different privacy budgets across modalities; a brief justification or a common-ε comparison would make the defense results easier to interpret.","section":"Table XII, Section IX"}],"recommendation":"minor_revision","confidential_remarks":"The central technical idea is sound and the experimental coverage is unusually thorough for a multi-modal MIA paper. The two major points are fixable with modest additional analysis and do not undermine the overall contribution; I would expect a clean accept after a short revision cycle. Fit for a solid ML/security venue is good."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know is that this is the first paper that actually ships a single black-box pipeline—output samples, embed, fit mean/cov, likelihood-ratio test—and shows it works across LLMs, diffusion models, and VLMs under both partial- and zero-knowledge. The numbers in Tables I–II are the real payload: comparable to the modality-specific SOTAs (SPV, CLiD, temperature, etc.) when you have auxiliary real data, and clearly better when you don’t.\n\nWhat’s new is not the observation that likelihood training drives p_θ toward the training distribution (that’s textbook KL minimization), nor the Neyman–Pearson zero-threshold rule under equal priors. What’s new is the demonstration that one decision rule, one embedding-space Gaussian model, and one set of steps transfers without modality-specific redesign. The theory section is clean and correctly derives the bias of the log-likelihood ratio from the KL gap. The experimental design is thorough: fine-tune and pre-train, three model pairs per modality, ablations on sample fraction (1–3 %), extractor choice, geometric median vs mean, distance metrics vs LRT, threshold, and even a neural classifier substitute. DP-SGD defense results are honest about the utility hit.\n\nThe soft spot is exactly the one the stress-test flags. They never report embedding dimension, condition number of Σ, or any normality check. With only a few hundred points in high-d space the sample covariance is almost surely ill-conditioned, so the quadratic form in Eq. 3 is unstable on paper. Zero-knowledge makes it worse because D_aux is just a noisy version of D_syn. The ablations (Figs. 5–7, Tables III–V) show the attack still works, so the practical impact is bounded, but the theoretical claim that the LRT remains near-optimal rests on an unvalidated approximation. No code, no error bars, free parameters around extractor and noise strength. None of that sinks the central empirical claim.\n\nThis is for people who care about privacy leakage in multi-modal generative systems and want a practical black-box baseline that doesn’t require training shadow models. It deserves a serious referee; the contribution is real and the evidence is extensive enough to evaluate. I’d engage with it.","headline":"First working unified black-box MIA across T2T/T2I/I2T; the embedding LRT is competitive or better under zero-knowledge, with the Gaussian small-sample step as the main soft underbelly.","tokens_in":27669,"tokens_out":582,"would_cite":true,"duration_ms":13040,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"One black-box likelihood-ratio test on embeddings works for membership inference across text-to-text, text-to-image, and image-to-text generative models.","keywords":["membership inference","generative models","cross-modal","likelihood ratio test","black-box attack","embedding space","fine-tuning privacy","zero-knowledge"],"falsifier":"Hold the attack fixed and replace the Gaussian likelihood-ratio rule with a pure distance-to-mean rule (cosine or Wasserstein) on the same embeddings; if the attack accuracy collapses across all three modalities, the claim that second-order distributional statistics are necessary is falsified. Conversely, if a non-Gaussian density estimator yields no further gain, the Gaussian approximation is sufficient.","tokens_in":27632,"feed_emoji":"🔍","tokens_out":641,"duration_ms":6940,"temperature":0.7,"pith_summary":"Generative models in three different modalities leak training membership through a shared statistical property: their outputs approximate the distribution of the data they were trained on. The authors turn that property into a single attack. They generate synthetic samples from the target model, collect (or synthesize) non-member samples, map both sets into a shared embedding space with an off-the-shelf encoder, fit simple Gaussians, and decide membership by a likelihood-ratio test. The same procedure works under partial knowledge of the data domain and under zero knowledge, and it works for both fine-tuning data and pre-training data. Experiments show the attack matches or beats modality-specific state-of-the-art methods while remaining strictly black-box. A sympathetic reader cares because modern AI systems already combine text and image generators; a privacy analysis that cannot travel across those components is incomplete.","feed_headline":"One likelihood test catches members across three generative modalities","feed_subtitle":"Black-box attack on embeddings works for text, images and vision-language models, even with zero external data","key_machinery":"The log-likelihood ratio under two empirical Gaussians fitted in embedding space (Eq. 3–4): a target sample is declared a member if its embedding is more likely under the synthetic distribution than under the real/auxiliary distribution.","core_discovery":"Likelihood-based training drives a generative model's output distribution toward its training distribution; once that fact is accepted, membership of a target sample can be decided by comparing its embedding likelihood under the model's synthetic output distribution versus an auxiliary non-member distribution, using a single Neyman–Pearson threshold of zero that is modality-agnostic and black-box.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["One likelihood ratio detects members across text image and vision-language models","Unified black-box MIA catches training data in three generative modalities","Embedding likelihood test unmasks members for text-to-text image and I2T models","Cross-modal membership inference via shared output distributions","Single Neyman-Pearson test works for fine-tune and pre-train data across modalities"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the mean and covariance of a modest number of embeddings, obtained from a fixed off-the-shelf encoder, are already good enough approximations of the true member and non-member distributions for the zero-threshold likelihood-ratio test to stay near-optimal.","fun_headline_variants_meta":{"raw":{"variants":["One likelihood ratio detects members across text image and vision-language models","Unified black-box MIA catches training data in three generative modalities","Embedding likelihood test unmasks members for text-to-text image and I2T models","Cross-modal membership inference via shared output distributions","Single Neyman-Pearson test works for fine-tune and pre-train data across modalities"]},"model":"grok-4.5","effort":"low","cost_usd":0.004918,"raw_usage":{"total_tokens":1407,"prompt_tokens":785,"num_sources_used":0,"completion_tokens":100,"cost_in_usd_ticks":49180000,"prompt_tokens_details":{"text_tokens":785,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":522,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":785,"tokens_out":100,"duration_ms":4874,"temperature":1.0,"reasoning_tokens":522,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T19:57:13.166849+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Hold the attack fixed and replace the Gaussian likelihood-ratio rule with a pure distance-to-mean rule (cosine or Wasserstein) on the same embeddings; if the attack accuracy collapses across all three modalities, the claim that second-order distributional statistics are necessary is falsified. Conversely, if a non-Gaussian density estimator yields no further gain, the Gaussian approximation is sufficient.","supporting_citations":[],"review_version":1}