FiMMIA: scaling semantic perturbation-based membership inference across modalities

Alena Fenogenova; Anton Emelyanov; Sergei Kudriashov

REVIEW 4 major objections 5 minor 8 references

A detector that perturbs only the text can identify whether an image, video, or audio sample was in a multimodal model's fine-tuning data, with average AUC-ROC of 88.7%, 88.4%, and 81.3%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:54 UTC pith:WYPBXN6M

load-bearing objection Worth reading for the benchmark audit and the open-source framework; the attack's headline numbers need a loss-only baseline and a more honest 'multimodal' label. the 4 major comments →

arxiv 2512.02786 v1 pith:WYPBXN6M submitted 2025-12-02 cs.LG

FiMMIA: scaling semantic perturbation-based membership inference across modalities

Anton Emelyanov , Sergei Kudriashov , Alena Fenogenova This is my paper

classification cs.LG

keywords membership inference attackmultimodal LLMdata contamination detectionperturbation-based MIAgrey-box attackfine-tuning leakagedistribution shiftcross-lingual transfer

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

FiMMIA claims that membership inference—deciding whether a given sample was in a model's training data—can be made to work for multimodal LLMs by perturbing only the text half of each sample and watching how the model's loss and embeddings react. The authors build a modular pipeline that generates 24 text perturbations per sample, computes loss and embedding differences between original and perturbed inputs, z-score normalizes them, and trains a small neural-network classifier to separate members from non-members. On fine-tuned multimodal models (3B-12B parameters), the detector averages 88.7% AUC-ROC for image, 88.4% for video, and 81.3% for audio, and it transfers across model families and from a Russian benchmark to English image datasets without retraining. The larger claim is practical: a grey-box attacker—or auditor—can detect contamination in multimodal benchmarks using a detector trained once, and many existing multimodal MIA benchmarks are too distribution-shifted to be trustworthy evaluation tools.

Core claim

The central discovery is that the membership signal of a multimodal sample (text plus an image, video, or audio clip) is readable from the text side alone. For each original sample, FiMMIA generates K=24 neighbors by masking-and-predicting, deleting, duplicating, and swapping tokens in the text while leaving the non-text modality untouched. It then measures, for the target model, the loss difference ΔL and a frozen text encoder's embedding difference Δe between original and neighbor, normalizes ΔL per model/dataset, and feeds these differences to an MLP that outputs a per-sample leakage probability. Across nine open multimodal models and three modalities the classifier separates fine-tuned-i

What carries the argument

The load-bearing object is the per-neighbor difference triple (ΔL_norm, Δe) formed by subtracting the target model's loss and a frozen text encoder's embedding on a text-perturbed neighbor from those on the original sample. The paper's premise is that this difference is a difficulty- and distribution-calibrated signature: members respond to semantic perturbation differently from non-members, and the difference generalizes across model families. The mechanism is the trained binary MLP that maps these K=24 differences to a single average leakage score, with z-score normalization of loss deltas computed on the evaluation split to keep the attack stable across datasets and models. The perturbati

Load-bearing premise

The load-bearing assumption is that a multimodal sample's membership signal is fully captured by perturbing only its text while the image, video, or audio stays fixed; if the stable signal lives in the non-text content, the reported 81-89% AUC scores may be text-side artifacts rather than evidence of multimodal leakage detection.

What would settle it

Run the identical FiMMIA training recipe twice on the same data splits and models: once perturbing only the text (as in the paper) and once perturbing only the image/video/audio while holding the text fixed. If the modality-only variant scores near 50% AUC while the text-only variant stays above 85%, the paper's across-modalities conclusion is refuted; if modality-only perturbations also separate members from non-members, the multimodal claim is verified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

An auditor with grey-box access (losses only) can flag which samples were used to fine-tune a multimodal LLM, without knowing the training data or retraining the detector on the target model.
A detector trained on one language's benchmark can be pointed at models trained on another language and still separate members from non-members, supporting contamination checks in settings where attack-specific training data is unavailable.
Most current image- and text-MIA benchmarks are not reliable testbeds: simple feature-based classifiers already separate their member/non-member splits, so high attack scores on them partly measure benchmark design flaws, not genuine leakage.
Because the framework supports neighbor generation for non-text modalities, the same pipeline can be extended to perturb images, video, or audio once appropriate neighbor generators exist.
Contamination auditing becomes feasible at scale for fine-tuned open models, which makes leaderboard-style evaluations harder to game through train/test leakage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper never perturbs the non-text modality, so its 'multimodal' claim is untested for the actual image/video/audio signal. A natural next experiment is to fix the text and perturb only the image/audio; if AUC collapses, the reported scores are likely driven by text-side regularities such as instruction wording or answer formatting.
The strong cross-lingual transfer hints that the learned signature is largely about the fine-tuning gap in the language head rather than about visual/audio understanding. If so, even stronger transfer to non-Russian, non-English languages should hold with no retraining, which would be a cheap confirmation.
Since z-score statistics are computed per dataset and per model, the detector's calibration depends on having a representative population of member and non-member samples at audit time. Deploying FiMMIA against an arbitrary closed model would require estimating these statistics blind, a regime the paper does not evaluate.
One could strip out the embedding-difference branch and train on loss deltas alone; if AUC stays above 90%, the expensive frozen encoder is unnecessary and the attack becomes a nearly closed-form threshold test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Worth reading for the benchmark audit and the open-source framework; the attack's headline numbers need a loss-only baseline and a more honest 'multimodal' label.

read the letter

First thing you should know: this paper is worth engaging with for its negative result, not for its headline attack. The distribution-shift audit is the cleanest contribution — they show that VL-MIA, WikiMIA and others separate members from non-members using hand-crafted features alone, with AUCs in the high 90s, which is a real problem for anyone using those benchmarks. The FiMMIA framework is also genuinely useful and open-sourced, with support for image, video, audio and text neighbors, even if the experiments only ever perturb the text side.

The main results, though, have a hole. In every experiment the 'leaked' model is a LoRA fine-tune and the 'non-leaked' model is the base. A trivial loss threshold could plausibly separate them with AUC close to what they report in Table 5. The paper never reports a loss-only baseline, so we don't know how much of the signal comes from the semantic perturbation and embedding pipeline versus simple overfitting. That is load-bearing: the contribution is precisely the perturbation-based mechanism, and without the control the numbers don't demonstrate it. Their distribution-shift baselines are not a substitute; those don't use model outputs at all.

Two smaller issues. The z-score normalization is described inconsistently: Section 3.4.3 says the statistics are calculated using the evaluation data, while Section 4.1 says they come from the train part. That matters, because using evaluation statistics would leak label information. The cross-lingual comparison with MM-DETECT on English datasets has no ground truth; agreement with a delta-score heuristic is weak evidence. And there are no error bars on the main AUCs, though the averages across models and the stability of the transfer tables make this a moderate concern, not a fatal one.

The 'multimodal' label is also stronger than what was tested. The perturbation is always applied to the text; images, video and audio are fixed. The pipeline may support other modalities, but the paper doesn't demonstrate it. That's a framing problem rather than a technical one, but it should be fixed.

Bottom line: the negative benchmark audit and the released code make this paper worth a serious referee. The FiMMIA effectiveness claim needs a loss-only baseline and ideally a perturbation on at least one non-text modality before it can be taken at face value. My recommendation is to send it to peer review with those requests, not to desk-reject it.

Referee Report

4 major / 5 minor

Summary. The paper addresses membership inference for multimodal LLMs. It first applies feature-based distribution-shift baselines to existing text/image MIA benchmarks and reports that most suffer from severe distribution shifts. It then proposes FiMMIA, a framework that (i) generates K=24 text perturbations of each sample, (ii) computes the target model's loss and the e5-mistral embedding differences between original and perturbed text, (iii) trains a binary neural classifier on these features to distinguish a LoRA-fine-tuned 'leaked' model from the base model, and (iv) averages the classifier output over neighbors at inference. Experiments on Russian MERA image/video/audio tasks across nine MLLMs report average AUC-ROC of 88.7/88.4/81.3, with transfer across model families and a cross-lingual comparison to MM-DETECT. The code and models are publicly released.

Significance. If the reported results survive the missing controls, FiMMIA would be a practically useful open-source tool and one of the first perturbation-based MIA frameworks evaluated on multimodal LLMs. The paper's distribution-shift analysis of existing MIA benchmarks is a useful cautionary contribution, and the release of a modular pipeline with reproducible neighbor generation is a concrete asset. However, the quantitative contribution currently rests on experiments that lack a loss-only control and contain a normalization inconsistency, so the significance of the semantic-perturbation mechanism itself is not yet established.

major comments (4)

[§3.4.2–3.4.3, Table 5] The reported AUCs do not include the simplest control. In every experiment, Mleak is a LoRA fine-tuned model and M is the unmodified base model, so members have been directly optimized to have low language-modeling loss. A threshold on the raw loss L(Mtest,t,s) — or on the average of ΔL over the K neighbors, without the e5-mistral embeddings — could already separate the two classes; the paper does not report this. Table 1 baselines use only dataset features and are not model-output baselines. Since the paper's contribution is the perturbation/embedding mechanism, this missing control is load-bearing: without it, the method may reduce to a loss-based MIA with extra machinery. Please report AUC for (i) raw loss threshold, (ii) mean ΔL over neighbors, and (iii) a classifier trained only on ΔL without Δe, under the same evaluation protocol.
[§3.4.3 vs §4.1] The normalization description is contradictory and potentially leaks test information. Section 3.4.3 says the mean μ and standard deviation σ for ΔL are calculated 'using the evaluation data,' while Section 4.1 says normalization parameters are calculated from the train part of each split. If evaluation (test) data statistics are used during training and at test time, the reported AUCs can be optimistically biased. Please specify exactly which data split is used and, if any test statistics are used, re-run the evaluation with train-only normalization or demonstrate that the results are unchanged.
[§3.3, §4.1, Table 1] The random-split MERA evaluation is not accompanied by the paper's own distribution-shift baseline. The authors argue that existing MIA benchmarks are unreliable because feature-only baselines achieve high AUC, but they do not run this baseline on their MERA splits. Without this control, one cannot rule out that FiMMIA's high AUC partly reflects unmodeled distribution shift in MERA (e.g., duplicates, ordering effects, or other artifacts). Please add the Table 1-style baseline AUC for each MERA modality/dataset to support the claim that the evaluation is clean.
[§4.3, Table 6] The cross-lingual transfer claim is not supported by the numbers. The text says that when MM-DETECT Δ<0, the fraction of samples FiMMIA flags as leaked is >0.1 'in most cases.' In Table 6, the negative-Δ rows are: COCO/Qwen-VL-Chat (0.00), COCO/LLaVA-1.5-7B (0.58), MMStar/fuyu-8b (0.011), and ScienceQA/fuyu-8b (0.19). Only two of four exceed 0.1, and the MMStar row is essentially 0.01. Moreover, Δ is a heuristic from MM-DETECT, not ground-truth leakage, so the comparison does not validate transfer. Please either provide a principled threshold with error bars or rephrase the claim to match the data.

minor comments (5)

[§3.4.1, title/abstract] The evaluated pipeline only perturbs text; every experiment fixes s=s'_k. The paper acknowledges this and says the pipeline can be modified, but the title's 'across modalities' wording is broader than the evidence. Please qualify the claim (e.g., 'text-perturbation-based MIA for multimodal LLMs') or add at least one experiment with a non-text perturbation to support the general formulation.
[Tables 2–5] No standard deviations, confidence intervals, or number of random seeds are reported. Given the computational cost and the stochasticity acknowledged in the Limitations, at least a variance estimate for a subset of the experiments would help interpret differences such as 53.6 vs 56.2 in Table 3.
[Appendix A.1] The concatenation size is described as '2 * projection_size,' but the loss component outputs projection_size and the embedding component outputs 512. This is only correct if projection_size=512, which is not stated. Please clarify.
[Abstract, §2.1] Typos and language issues: 'Although there are have been' in the abstract, 'intensionsal' in §2.1, and 'LLaV A' with a space in several tables. A careful proofread is needed.
[§4.3] The threshold of 0.1 for 'percentage leaked' and the sample sizes (2000/1000) appear arbitrary. Please justify them or report sensitivity of the cross-lingual conclusions to these choices.

Circularity Check

0 steps flagged

No significant circularity: FiMMIA trains a supervised attack model on known member/non-member loss differences and evaluates on held-out data and held-out models; no load-bearing step reduces to its own inputs.

full rationale

The derivation chain is not circular. FiMMIA's attack model is a supervised binary classifier trained on features (loss difference ΔL and embedding difference Δe) computed from a known leaked model Mleak (LoRA fine-tuned on the training split) versus a non-leaked base model M, with labels y∈{0,1} indicating which model produced the losses (Section 3.4.3). The classifier is then evaluated on a held-out 10% split of each dataset (Section 4.1) and, in transfer experiments, on entirely different target models (Tables 2-4). The membership label is not used to construct the input features at test time; it is only used to fit the classifier, which is the standard supervised MIA evaluation paradigm. The z-score normalization parameters are stated to be calculated from the train part of each split (Section 4.1); even under the looser wording in Section 3.4.3, using unlabeled evaluation statistics would be a methodological leakage concern, not definitional circularity. The lack of a loss-only baseline is a real experimental gap, but it does not make the reported AUCs equivalent to the method's inputs by construction; it only leaves open whether the perturbation/embedding components add signal beyond raw loss. The restriction to text-side perturbations while calling the method multimodal is a scope/validity limitation, not a circular derivation. Self-citations to MERA (Chervyakov et al. 2025) and FRED-T5 (Zmitrovich et al. 2024) are open-source resources used as benchmarks or tools, not load-bearing uniqueness theorems, and no central claim is justified only by a self-citation chain.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The method rests on supervised classification of loss/embedding differences; no new physical/mathematical entities are introduced. The main data-derived quantities are the z-score normalization statistics and the attack-model hyperparameters.

free parameters (4)

Number of perturbed neighbors K = 24
Hand-chosen; no sensitivity analysis reported. The attack averages the detector over K neighbors.
Z-score normalization statistics µ_D,M and σ_D,M = computed from train or evaluation data (description inconsistent)
Used to normalize loss differences before training/inference; if computed on evaluation data, it can bias AUC optimistically.
Attack model hyperparameters (learning rate, batch size, epochs, dropout, projection_size) = LR=2e-6, batch=64, epochs=10, dropout=0.2; projection_size not stated
Chosen without reported tuning/ablation; central results depend on this configuration.
Leakage threshold for cross-lingual 'percentage leaked' = not explicitly stated (implies probability threshold such that fraction leaked > 0.1 counts)
Table 6 reports percentages of samples predicted as leaked, but the decision threshold is not specified.

axioms (6)

domain assumption Random split of each MERA dataset into member/non-member sets creates i.i.d. distributions with no residual distribution shift.
Section 4.1 relies on random splitting to ensure high AUC reflects membership rather than confounds; no feature-based distribution-shift check is reported for these splits.
domain assumption Text-only perturbations (mask, delete, duplicate, swap) preserve the membership signal of the full multimodal sample.
Section 3.4.1 fixes the modality input s unchanged; the method's multimodal claim depends on the textual perturbation neighborhood being sufficient.
domain assumption The fixed text encoder e5-mistral-7b-instruct and Fred-T5 masking model provide useful embeddings/neighbors across Russian and English, and across image/video/audio tasks.
Sections 3.4.1 and 4.3 rely on these external models without ablating them.
domain assumption The attacker has gray-box access to per-sample loss (logprobs) of the target MLLM.
Stated in Section 3.4.3 and Limitations; if only black-box access is available, the method is not applicable.
domain assumption Fine-tuning with LoRA is a representative proxy for MLLM training for contamination scenarios.
Limitations state results may differ for pretraining and full fine-tuning; the central experiments use only LoRA adapters.
standard math Standard supervised classification (cross-entropy training, AUC-ROC evaluation) is a valid way to measure attack performance.
Used throughout Section 5 without proof; this is standard practice.

pith-pipeline@v1.3.0-alltime-deepseek · 18599 in / 16088 out tokens · 136425 ms · 2026-08-03T18:54:19.600889+00:00 · methodology

0 comments

read the original abstract

Membership Inference Attacks (MIAs) aim to determine whether a specific data point was included in the training set of a target model. Although there are have been numerous methods developed for detecting data contamination in large language models (LLMs), their performance on multimodal LLMs (MLLMs) falls short due to the instabilities introduced through multimodal component adaptation and possible distribution shifts across multiple inputs. In this work, we investigate multimodal membership inference and address two issues: first, by identifying distribution shifts in the existing datasets, and second, by releasing an extended baseline pipeline to detect them. We also generalize the perturbation-based membership inference methods to MLLMs and release \textbf{FiMMIA} -- a modular \textbf{F}ramework for \textbf{M}ultimodal \textbf{MIA}.\footnote{The source code and framework have been made publicly available under the MIT license via \href{https://github.com/ai-forever/data_leakage_detect}{link}.The video demonstration is available on \href{https://youtu.be/a9L4-H80aSg}{YouTube}.} Our approach trains a neural network to analyze the target model's behavior on perturbed inputs, capturing distributional differences between members and non-members. Comprehensive evaluations on various fine-tuned multimodal models demonstrate the effectiveness of our perturbation-based membership inference attacks in multimodal domains.

Figures

Figures reproduced from arXiv: 2512.02786 by Alena Fenogenova, Anton Emelyanov, Sergei Kudriashov.

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 5 linked inside Pith

[4]

why should I trust you?

Explicit tradeoffs between adversarial and nat- ural distributional robustness. InAdvances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexand...

Pith/arXiv arXiv 2022
[8]

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha

Benchmarking benchmark leakage in large language models.arXiv preprint arXiv:2404.18824. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In2018 IEEE 31st Computer Security Foundations Sympo- sium (CSF), pages 268–282. Jinhua Yin, Peiru Yang, Chen Yang, Huili...

Pith/arXiv arXiv 2018
[1144]

Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre

ACM. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre
[2020]

InPro- ceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343– 362

GAN-Leaks: A taxonomy of membership inference attacks against generative models. InPro- ceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343– 362. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and et al. 2024. Are we on the right way for evaluat...

Pith/arXiv arXiv 2020
[2022]

InProceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816–1826

An empirical analysis of memorization in fine- tuned autoregressive language models. InProceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816–1826. Ryoto Miyamoto, Xin Fan, Fuyuko Kido, Tsuneo Mat- sumoto, and Hayato Yamana. 2025. Openlvlm-mia: A controlled benchmark revealing the limits of mem- bership infere...

arXiv 2022
[2023]

InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore

NLP evaluation in trouble: On the need to mea- sure LLM data contamination for each benchmark. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore. Association for Computational Linguistics. Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. 2018. Ml-leaks: Model a...

Pith/arXiv arXiv 2023
[2024]

Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou

Qwen2-audio technical report.Preprint, arXiv:2407.10759. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio- language models.arXiv preprint arXiv:2311.07919. Debeshee Das, Jie Zhang, and Florian Tramèr. 2024. Blind...

Pith/arXiv arXiv 2023
[2025]

Poisoning attacks on llms require a near- constant number of poison samples.arXiv preprint arXiv: 2510.07192. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Per- rin, Tatiana Matejovicova, Alexandre Ramé, Mor- gane Rivière, Louis Rouillard, Thomas Mesnard, Ge- offrey Cideron, Jean bastien Grill, Sabela Ramo...

arXiv 2025

[1] [4]

why should I trust you?

Explicit tradeoffs between adversarial and nat- ural distributional robustness. InAdvances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexand...

Pith/arXiv arXiv 2022

[2] [8]

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha

Benchmarking benchmark leakage in large language models.arXiv preprint arXiv:2404.18824. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In2018 IEEE 31st Computer Security Foundations Sympo- sium (CSF), pages 268–282. Jinhua Yin, Peiru Yang, Chen Yang, Huili...

Pith/arXiv arXiv 2018

[3] [1144]

Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre

ACM. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre

[4] [2020]

InPro- ceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343– 362

GAN-Leaks: A taxonomy of membership inference attacks against generative models. InPro- ceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343– 362. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and et al. 2024. Are we on the right way for evaluat...

Pith/arXiv arXiv 2020

[5] [2022]

InProceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816–1826

An empirical analysis of memorization in fine- tuned autoregressive language models. InProceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816–1826. Ryoto Miyamoto, Xin Fan, Fuyuko Kido, Tsuneo Mat- sumoto, and Hayato Yamana. 2025. Openlvlm-mia: A controlled benchmark revealing the limits of mem- bership infere...

arXiv 2022

[6] [2023]

InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore

NLP evaluation in trouble: On the need to mea- sure LLM data contamination for each benchmark. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore. Association for Computational Linguistics. Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. 2018. Ml-leaks: Model a...

Pith/arXiv arXiv 2023

[7] [2024]

Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou

Qwen2-audio technical report.Preprint, arXiv:2407.10759. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio- language models.arXiv preprint arXiv:2311.07919. Debeshee Das, Jie Zhang, and Florian Tramèr. 2024. Blind...

Pith/arXiv arXiv 2023

[8] [2025]

Poisoning attacks on llms require a near- constant number of poison samples.arXiv preprint arXiv: 2510.07192. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Per- rin, Tatiana Matejovicova, Alexandre Ramé, Mor- gane Rivière, Louis Rouillard, Thomas Mesnard, Ge- offrey Cideron, Jean bastien Grill, Sabela Ramo...

arXiv 2025