{"id":"0a986c29-b940-43cc-a549-1dd94e85a048","arxiv_id":"2501.16671","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Generative AI can synthesize data that lets black-box attackers perform model extraction, membership inference, and model inversion without the target's training data.","lead":"This paper shows that attackers can use generative AI, such as Stable Diffusion and GPT-4, to create synthetic data for attacking machine learning models without needing the models' training data. The attacks include model stealing, membership inference, and model inversion, and work in a black-box setting with performance close to, though usually below, white-box attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central data-free claim depends on Section 3.1's assumption that a public generator covers the target's training distribution; the PET experiment does not test this because its dog/bird/cat classes are common and likely already in the generator's training data.","rationale":"The abstract's strongest claim is that adversaries can launch model-related attacks 'in a data-free and black-box manner' with performance comparable to white-box baselines. That claim is only as strong as the pipeline's first step: the public generator must produce data lying in the target model's training distribution. The paper states this as a likelihood, not a proven property, and all standard benchmarks are public datasets that may appear in the generator's training data. The PET attempt to address this is weakened by its choice of common classes and YouTube sources; it cannot falsify the concern. The reader's weakest assumption identified the same point, so I agree with the reader's conditional verdict. I do not see a reason to change the verdict: the empirical results on public datasets still support a narrower claim about data-free attacks on models trained on public distributions, and the paper should be accepted only after the unseen-domain assumption is tested with a genuinely private dataset. The invalid KL proof in Section 4 is a separate correctness issue but does not by itself decide the central empirical claim.","tokens_in":22932,"tokens_out":10937,"duration_ms":114521,"concrete_test":"Build a target classifier on a genuinely private image dataset whose categories are not publicly available and are semantically rare for public generators (e.g., self-recorded images of a specific proprietary instrument or a rare local species, with no online copies). Train the target with the same architecture as in Appendix A.1 (e.g., ResNet18), then run the full pipeline using Fast Stable Diffusion XL and GPT-4o with class-name prompts, the same augmentation and inter-class filtering from Section 3.2. Report model-extraction accuracy/agreement, MIA AUC, and inversion MSE on a held-out private test set, and compare with the random-search baselines in Section 5.4.1. If extraction accuracy and MIA AUC are not substantially above the random-search baseline, the Section 3.1 assumption fails for unseen domains and the data-free claim must be restricted to public distributions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states: 'the distribution space of training data for generative models is highly likely to encompass the distribution space of the training set for the target model if the target model shares a similar task with the generative model.' This is the load-bearing premise for all three attacks. If a public generator cannot produce samples near the target's training distribution, Step 1 yields unusable D_c, Step 2's boundary exploration starts from off-distribution points, and Step 3's filtering cannot invent a missing mode. The experiments use CIFAR10, MNIST, SkinCancer, BBCNews, and IMDB, all public datasets that are plausibly part of Stable Diffusion/GPT-4 training corpora, so they do not test the premise. The PET dataset (Section 5.1) was meant to be unseen, but it contains only cat/dog/bird; the dog and bird videos are YouTube links listed in the paper, and cats are one of the most common image categories online. A generator can produce a generic 'cat' even if the specific videos were never seen. Thus the paper demonstrates regeneration of public distributions, not data-free attacks on genuinely private domains. This external-validity gap is independent of the invalid KL proof in Section 4: even a correct proof of filtering benefits would not rescue the attack if the generator cannot cover the target domain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data-free, black-box framework for model-related attacks (model extraction, membership inference, and model inversion) that uses public generative models (Stable Diffusion and GPT-4) to synthesize a surrogate dataset, augments it by probing the target model's decision boundary, and then filters the data with an inter-class three-sigma rule. The authors evaluate the pipeline on CIFAR10, MNIST, SkinCancer, BBCNews, IMDB, and a newly collected PET dataset, and claim that the attacks achieve performance comparable to white-box baselines that use the target's training data and architecture. The paper also includes hyperparameter studies, ablations with random-search baselines and alternative generative models, a discussion of defenses, and a release of source code.","tokens_in":23226,"tokens_out":5509,"duration_ms":56815,"significance":"If validated, the paper would be a useful contribution to the security/privacy literature: it extends data-free model stealing and related attacks to a setting where the adversary uses only a public generative model and black-box query access, and it is one of the few studies to cover both image and text domains under one pipeline. The authors also provide ablations and a code link, which supports reproducibility. The central empirical claim, however, currently rests on an untested distribution-coverage assumption, and the evaluation protocol for membership inference and model extraction does not, as written, measure what the abstract claims it measures. The theoretical analysis in Section 4 is not sound. These issues preclude acceptance in the current form.","major_comments":[{"comment":"The proof that inter-class filtering reduces the KL divergence between the target training distribution and the generated distribution is invalid. The first integral in the second line is not itself a KL divergence and can be negative, so the statement 'any KL divergence value is larger than or equal to 0' does not justify dropping it. The subsequent claim that P'_gen(x)/P_gen(x) > 1 pointwise is not established; P'_gen is a distribution obtained by deleting samples and can be smaller than P_gen on some points or zero where P_gen is positive. The line 'Pgen(x < mu + 3sigma) < 1' also treats a probability distribution as a random variable. As written, the theoretical result is unproved. I recommend either supplying a correct proof or explicitly labeling the three-sigma filter as a heuristic supported only by the experiments.","section":"Section 4, Eq. (4)"},{"comment":"The membership inference attack is trained and evaluated using the auxiliary dataset's own train/test split (dDtrainaux versus dDtestaux), not using samples whose membership in the target model's training set is known. Tables 2 and 3 therefore report how well the attack model distinguishes the surrogate's generated training samples from its generated test samples. This does not establish that the attack transfers to the target model T. The paper does not describe a ground-truth membership test set for T or report how often samples from T's actual Dtrain are flagged as members. Please specify the evaluation protocol and report attack accuracy/F1/AUC on the target model's actual members versus non-members.","section":"Section 3.3 and Section 5.2"},{"comment":"Model extraction accuracy and agreement are computed on dDtestaux, a subset of the same generated, T-filtered dataset used to train the stolen model. Because both the target and stolen models are evaluated on samples that T has already classified consistently during the filtering step, the target accuracy and the agreement metric are inflated by construction. The paper should additionally evaluate both models on the original held-out test sets (e.g., the standard CIFAR10 test split, the original BBCNews/IMDB splits) or on an independently collected set, and report target versus stolen performance there.","section":"Section 5.2, Table 1"},{"comment":"The central premise of the paper is that a public generative model's training distribution 'is highly likely to encompass' the target model's training distribution. None of the main datasets tests this premise: CIFAR10, MNIST, SkinCancer, BBCNews, and IMDB are public and plausibly present in the training corpora of Stable Diffusion or GPT-4. The PET dataset does not resolve the issue: its dog and bird videos are listed as YouTube URLs in the paper, and 'cat' is a common image category, so the generator can produce in-domain samples without having seen the specific PET videos. Please test the pipeline on a genuinely private or niche domain where the generator cannot rely on memorized or near-memorized content, or otherwise quantify the distribution overlap between the generator's output and the target's training set. Without such a test, the data-free claim is only demonstrated for public-distribution targets.","section":"Section 3.1 and Section 5.1"},{"comment":"The experiments report averages over three runs but provide no error bars, confidence intervals, or statistical comparisons, yet the abstract and Section 5.5 make the strong claim of performance 'comparable' to white-box baselines. Some of the observed gaps are substantial (e.g., Table 2 CIFAR10 AUC 0.54 versus 0.67; Table 1 CIFAR10 agreement 83.7 versus 89.4). Please report standard deviations and either bootstrap confidence intervals or a paired significance test for the headline accuracy/agreement/AUC comparisons.","section":"Section 5.1 and Tables 1-4"}],"minor_comments":[{"comment":"The text says 'I is the unit vector' but the context requires the identity matrix; please correct the notation.","section":"Section 4, Eq. (1)"},{"comment":"In lines 2-3, the algorithm refers to samples xk_j in class i, but the centroid is computed over xi_j; the subscript should be i throughout for consistency.","section":"Algorithm 2"},{"comment":"The table header says 'AUC' but the values are TPR@1%FPR; the header should be updated to match the rows.","section":"Table 11"},{"comment":"There is a typo: 'utilitizing' should be 'utilizing'.","section":"Section 1, Contributions"},{"comment":"The label-only variants of membership inference and model inversion are described but not evaluated in the experiments; please clarify whether these variants are part of the claims or defer them to future work.","section":"Section 3.3"},{"comment":"The description of the comparison baselines would benefit from more detail on training hyperparameters and the amount of data used, since the baseline's access to the full target training set is a key point in the comparison.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the general idea is timely, but the evaluation currently overstates the strength of the evidence: the membership inference protocol may not measure target-model membership at all, the model extraction metric is partly circular, and the PET dataset does not provide the intended external-validity test. These are fixable with additional experiments and a corrected analysis, so I do not recommend rejection. The invalid KL proof in Section 4 should also be either corrected or downgraded to a heuristic; it should not remain as a formal theorem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know: this paper offers a genuinely new attack pipeline—using a public text-to-image or LLM generator to produce synthetic training data, then augmenting near decision boundaries and filtering via output vectors, to run model extraction, membership inference, and model inversion in a data-free black-box setting. That combination is new, and the experiments cover both image and text models. It beats random-search baselines by a wide margin and is competitive with, though usually below, a white-box shadow-model baseline.\n\nWhat's good: the pipeline is simple and plausible. The decision-boundary augmentation step is a nice trick: generate samples from a public model, then walk noise up to the target's boundary to collect harder examples. The inter-class filtering using target output vectors is also reasonable. The ablation with alternative generators (Gemini, Open-journey) and automated vs manual prompts shows the method is not tied to one generator. The defense evaluation, though brief, is a start. The PET dataset was a good idea, though as discussed below it doesn't really test what they claim.\n\nSoft spots. First, the theoretical proof in Section 4 is not just weak, it's invalid. The step \"Pgen(x < ∞)=1 and thus Pgen(x < μ + 3σ) < 1\" is nonsense, and treating the filtered distribution as pointwise larger than the original doesn't follow. Remove or fix it; the empirical results don't need it. Second, no error bars anywhere, despite saying results are averaged over three runs. For membership inference, AUC values are low (0.52–0.56), so variance matters. Third, \"comparable to white-box baseline\" is overstated. Look at Tables 1, 2, 4: our method is consistently a few points below baseline on accuracy, and noticeably worse on CIFAR10 membership inference (72.6 vs 79.4). It's better than random, but \"comparable\" is doing a lot of work. Fourth, and most important: the load-bearing assumption is that a public generator covers the target's training distribution. The PET experiment does not test this—cats, dogs, and birds are ubiquitous online, and the generator likely has seen plenty of them. So the paper demonstrates attacks on public distributions, not on genuinely private ones. That's a real external-validity gap, independent of the proof issue. Also, the abstract points to a Zenodo code link, but the ethics section says they will not release specific details; that tension should be resolved.\n\nWho this is for: security researchers working on model extraction/inversion and on the offensive uses of generative AI. It deserves a serious referee—the idea is timely and the pipeline is likely reproducible—but the authors should fix the proof, report error bars, soften the \"comparable\" language, and either find a genuinely private target domain or be honest that the assumption is untested. I'd send it to review with a request for major revision.","headline":"A timely, plausible data-free attack pipeline using public generators, but the theory is invalid, the 'comparable to white-box' claim is overstated, and the private-dataset test does not actually test the key distribution-coverage assumption.","tokens_in":23740,"tokens_out":2970,"would_cite":true,"duration_ms":27317,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that public generative models can supply synthetic data standing in for a target model's training set, enabling model extraction, membership inference, and model inversion attacks in a data-free, black-box setting with…","keywords":["model extraction","membership inference","model inversion","data-free attack","black-box attack","generative AI","synthetic data","diffusion models"],"falsifier":"Train a target classifier on a genuinely private dataset whose classes are absent from any public generator's training distribution (for example, specialized medical or industrial imagery), run the three-step pipeline, and compare the stolen model's accuracy and agreement to the random-search baseline. If they collapse to that baseline, the distribution-coverage premise is falsified.","tokens_in":22747,"feed_emoji":"🤖","tokens_out":6861,"duration_ms":65184,"temperature":0.7,"pith_summary":"Generative AI is usually studied as a victim of privacy attacks, but this paper argues it can be the weapon. The authors try to establish that an adversary who knows only the task and class labels of a deployed classifier, with no access to its training data, parameters, or architecture, can use a public diffusion model or large language model to synthesize a substitute dataset and then run model extraction, membership inference, and model inversion attacks that approach the performance of white-box baselines. The significance is that today's public generators may already lower the barrier for attacking real deployed models, turning a capability problem into a defense problem. The paper evaluates the pipeline on CIFAR10, MNIST, SkinCancer, BBCNews, and IMDB, plus a newly collected PET dataset, and shows that hand-crafted prompts, boundary augmentation, and inter-class filtering bring the synthetic data close enough to the target's training distribution.","feed_headline":"Generative AI enables data-free model attacks","feed_subtitle":"Synthetic images and text replace stolen training data for extraction, membership inference, and inversion attacks.","key_machinery":"The load-bearing mechanism is a three-stage data synthesis pipeline. First, prompted generation: the adversary asks a public generative model for samples of each target class, keeping only those the target classifies correctly. Second, boundary augmentation: starting from a kept sample, the adversary adds Gaussian noise in growing spherical shells and collects samples just inside the target's decision boundary, which expands coverage of the sample space. Third, inter-class filtering: samples are embedded through the target's output vectors, class centroids are computed, and any sample in class $k$ whose distance to another class's centroid exceeds three standard deviations is removed. The paper argues, via a Kullback-Leibler divergence inequality, that this filtering reduces the distribution shift between the generated set and the target's training distribution.","core_discovery":"On the paper's own terms, the central discovery is that a black-box adversary can replace the target model's private training set with generated data from a public generative model and still obtain a stolen model whose accuracy and agreement approach those of a thief who has the real training set and architecture. For example, on CIFAR10 the stolen model reaches 82.6% accuracy against a target at 88.7% and a white-box baseline at 85.3%. For membership inference, the adversary trains a shadow model on the generated data and transfers the resulting attack model to the target. For model inversion, the adversary trains a generative inversion model on pairs of target output vectors and generated inputs, then reconstructs inputs from confidence vectors. The paper also claims the same pipeline works in label-only settings, where boundary-crossing noise becomes a proxy for membership and class-representative samples stand in for exact inversion.","pith_inferences":["The PET experiment only partially tests the distribution-coverage assumption, because its cat/dog/bird classes are common and several videos were downloaded from YouTube, which may already be inside the generator's training data; a target trained on a genuinely novel domain remains the untested case.","The KL-divergence argument shows filtering reduces the distance to the target distribution only under the three-sigma model; in practice the same filtering could remove informative boundary samples, so the attack's edge may depend on how cleanly class centroids separate in output space.","As generators improve for non-image, non-text modalities such as audio, video, or structured medical records, the same three-step pipeline would likely carry over, making the threat broader than the two modalities tested here.","Because automated prompt generation already comes close to manual prompts in extraction accuracy, the attack appears automatable end-to-end, so the practical barrier for attackers is mostly compute and API access rather than expert prompt design."],"forward_implications":["A deployed image or text classifier that exposes output confidence vectors can be functionally copied with no access to its training data, provided a public generator exists for the same task.","Membership inference can be run without collecting a shadow dataset matching the target's domain: synthetic data plus a mimic model yields attack accuracy, F1, AUC, and TPR@1%FPR comparable to a white-box shadow-model baseline.","The same generated dataset supports model inversion by training an inversion model on pairs of target output vectors and generated inputs, with reconstruction quality close to the baseline that uses the real training set.","Label-only access does not stop the attacks: boundary-crossing noise can signal membership, and class-representative synthetic samples serve as inversion outputs.","Synthetic-data detection and output perturbation are only partial defenses: detection can be bypassed by generating more data, and perturbation leaves membership inference and inversion largely intact."],"supporting_citations":[{"why":"Fast Stable Diffusion XL on TPU v5e is the image generator actually used to synthesize attack data.","marker":"[1]"},{"why":"GPT-4.0 is the language generator used to synthesize text attack data.","marker":"[39]"},{"why":"DALL-E is cited as the example of a public image generator aligned with classifier tasks, motivating the distribution-coverage assumption.","marker":"[38]"},{"why":"Diffusion models supply the generation framework whose forward and reverse processes the distribution-shift analysis relies on.","marker":"[51]"},{"why":"Membership inference from first principles is the baseline method compared against for shadow-model membership attacks.","marker":"[5]"},{"why":"Label-only membership inference via perturbation threshold underpins the hard-label variant of the membership attack.","marker":"[31]"},{"why":"The model inversion baseline trains an inversion model on target outputs paired with inputs, the method the paper adapts to generated data.","marker":"[53]"},{"why":"Model extraction survey defines the threat model of functionally copying a target classifier.","marker":"[37]"}],"fun_headline_variants":["Data-free model attacks via generative AI","Model attacks go data-free with generative AI","Black-box model attacks using synthetic data","Steal models without data via generative AI","Generative AI powers model attacks without data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline only works if the public generative model's training data already covers the target model's task domain, because the generated samples are supposed to stand in for the target's private training data.","fun_headline_variants_meta":{"raw":{"variants":["Data-free model attacks via generative AI","Model attacks go data-free with generative AI","Black-box model attacks using synthetic data","Steal models without data via generative AI","Generative AI powers model attacks without data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000697,"raw_usage":{"total_tokens":3104,"prompt_tokens":854,"completion_tokens":2250,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":2186}},"tokens_in":470,"tokens_out":2250,"duration_ms":18759,"temperature":1.0,"reasoning_tokens":2186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T11:31:45.035774+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a target classifier on a genuinely private dataset whose classes are absent from any public generator's training distribution (for example, specialized medical or industrial imagery), run the three-step pipeline, and compare the stolen model's accuracy and agreement to the random-search baseline. If they collapse to that baseline, the distribution-coverage premise is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fast Stable Diffusion XL on TPU v5e is the image generator actually used to synthesize attack data."},{"cited_title":"DALL·E API Now Available in Public Beta, 2022","cited_arxiv_id":null,"evidence_quote":"DALL-E is cited as the example of a public image generator aligned with classifier tasks, motivating the distribution-coverage assumption."},{"cited_title":"Diffusion Models: A Comprehensive Survey of Methods and Applications","cited_arxiv_id":null,"evidence_quote":"Diffusion models supply the generation framework whose forward and reverse processes the distribution-shift analysis relies on."},{"cited_title":"Membership Inference Attacks From First Principles","cited_arxiv_id":null,"evidence_quote":"Membership inference from first principles is the baseline method compared against for shadow-model membership attacks."},{"cited_title":"Membership Leakage in Label-Only Exposures","cited_arxiv_id":null,"evidence_quote":"Label-only membership inference via perturbation threshold underpins the hard-label variant of the membership attack."},{"cited_title":"Neural Net- work Inversion in Adversarial Setting via Background Knowledge Alignment","cited_arxiv_id":null,"evidence_quote":"The model inversion baseline trains an inversion model on target outputs paired with inputs, the method the paper adapts to generated data."},{"cited_title":"I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences","cited_arxiv_id":null,"evidence_quote":"Model extraction survey defines the threat model of functionally copying a target classifier."}],"review_version":1}