{"id":"95539db6-3494-4cd8-aabe-d38b26fc882e","arxiv_id":"2506.05867","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"An attacker with one seed image per class can steal a black-box image classifier by genetically evolving text prompts, guided only by the victim's hard-label predictions.","lead":"Stealix is a model stealing attack that needs no written prompts or class names: the attacker provides one real image per class and the method automatically evolves text prompts to generate training images that fool the victim classifier into revealing its behavior. It matters because it shows open-source diffusion models can be turned against black-box APIs by non-experts, raising the bar for defenses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PC fitness proxy is validated only against the same victim model used to define it; the claimed 22.2% advantage is therefore not established for specialized domains with weak Stable Diffusion priors, exactly the scenarios the threat model emphasizes.","rationale":"The reader's weakest assumption is exactly the prompt consistency premise in Section 4.3, and my reading confirms that this is the load-bearing step. The concrete test would settle whether PC selects distribution-aligned prompts or merely boundary-aligned ones, and whether the reported 22.2% improvement generalizes beyond the four tested image domains. This does not change the reader's conditional verdict: the paper's empirical results are plausible on the tested datasets, but the central 'first prompt-agnostic attack' and 'up to 22.2% improvement' claims should be taken as conditional on the diffusion model being able to represent the target domain, and as requiring an independent validation of PC before the attack is presented as a general threat to specialized models. No ad hominem or code-release concerns enter this assessment; the single technical issue is the circularity of the PC validation and its observable collapse in the medical appendix.","tokens_in":177,"tokens_out":4570,"duration_ms":65553,"concrete_test":"Re-run the Section 5.2 correlation using an image feature space independent of the victim model (e.g., CLIP ViT-H features instead of the victim's penultimate layer) on an out-of-domain dataset such as PCAM or RetinaMNIST, and additionally track attacker test accuracy as a function of the PC of each evolved prompt. If the negative PC-to-feature-distance correlation disappears, or if attacker accuracy is not monotonic in PC, then PC is a victim-boundary-specific proxy and the claimed advantage should be restricted to domains with strong generative priors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Stealix 'removes the need for human-crafted prompts' and 'surpasses methods using class names or human-crafted prompts, improving attacker model accuracy by up to 22.2%' rests on the prompt consistency premise of Section 4.3: if a prompt captures the victim's latent class features, synthetic images are consistently classified as the target class. The validation in Section 5.2 computes Spearman's rho between PC and the L2 distance of mean feature vectors, but both quantities are defined through the same victim model's penultimate layer. This is not an independent test: a high-PC prompt may place generated images on the victim's side of the decision boundary while remaining far from the true data manifold. The failure mode is visible in Appendix L: on PCAM and RetinaMNIST, where Stable Diffusion has weak domain priors, Stealix's advantage over Real Guidance and DA-Fusion shrinks to 0.4-1.9 accuracy points. Thus the headline improvement is demonstrated only on datasets where Stable Diffusion can already depict the domain, and the paper does not establish that PC reliably tracks distributional similarity in the specialized, high-value settings its threat model is designed to cover. Since the attacker's final accuracy on real test data is the true target, a proxy that overfits the victim's boundary would make the reported gains non-transferable to domains where the generative prior and the victim data diverge.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Stealix, a model stealing attack that requires no human-crafted prompts or class names. The attacker starts from one seed image per class, uses a vision-language model to iteratively refine hard prompts with a contrastive loss, and employs a genetic algorithm whose fitness is a proposed 'prompt consistency' (PC) metric: the fraction of synthetic images, generated by a frozen Stable Diffusion model, that the victim classifies into the target class. The evolved prompts generate synthetic images that are pseudo-labeled by the victim and used to train a surrogate classifier. The authors validate the approach on EuroSAT, PASCAL VOC, CIFAR-10, and DomainNet, report improvements of up to 22.2 accuracy points over baselines under a 500-query-per-class budget, and include additional experiments on two medical datasets and a proprietary NSFW detector.","tokens_in":19104,"tokens_out":5212,"duration_ms":58230,"significance":"If the results hold, Stealix is a meaningful advance in model stealing: it demonstrates that hard-label black-box models can be stolen without prompt engineering or class-name knowledge, using only one seed image per class and a modest query budget. The paper ships a full algorithmic description, ablations, an analysis of the PC proxy, a diversity comparison, and a real-world NSFW-model case study. Strengths include the use of a public generative model and VLM, hard-label-only feedback, and explicit three-repeat runs. The main weaknesses are the endogenous validation of PC, the absence of variance reporting in the main tables, and the mismatch between the broad threat-model motivation (specialized high-value domains) and the fact that the largest gains appear on datasets where Stable Diffusion already has strong priors.","major_comments":[{"comment":"The validation of the prompt consistency (PC) proxy is circular and therefore does not establish that PC tracks distributional similarity to the victim data. PC (Eq. 5) is defined by querying the victim model, and the L2 feature distance is computed from the same victim model's penultimate layer. A prompt that places generated images on the victim's side of the decision boundary can achieve high PC without generating images close to the true data manifold. The paper's ultimate external check is the attacker's accuracy on the victim test set, but the paper does not separately validate that high-PC prompts are the ones that transfer. I ask the authors to validate PC against an independent feature space (e.g., a pre-trained foundation model not used anywhere in the attack) or to show that high-PC prompts outperform low-PC prompts on the victim test set in a domain-shifted setting. The concern is not merely theoretical: in Appendix L, on PCAM and RetinaMNIST, where Stable Diffusion has weak priors, Stealix's advantage over DA-Fusion shrinks to 0.7 and 1.3 accuracy points, consistent with the proxy overfitting the victim boundary.","section":"Section 5.2, Table 2"},{"comment":"All reported accuracies are means over only three repeated runs, and neither Table 1 nor Table 12 includes standard deviations, confidence intervals, or significance tests. Figure 4, cited as containing confidence intervals, is not shown with visible error bars in the manuscript. The absence of variance reporting is load-bearing for the comparisons that are close: in Table 12, Stealix's 62.2% versus DA-Fusion's 61.5% on PCAM, and 58.0% versus 56.7% on RetinaMNIST, could easily be within run-to-run noise. The authors should report per-run values or error bars and perform a paired significance test (or at least state the variance) for the headline claims, including the 22.2% improvement on CIFAR-10.","section":"Section 5.1, Tables 1 and 12, Figure 4"},{"comment":"The central claim that Stealix 'significantly outperforms other methods ... across diverse image domains' and 'improving attacker model accuracy by up to 22.2%' is overstated relative to the evidence. The 22.2% figure is from CIFAR-10, a dataset where class names are strong and Stable Diffusion has a good prior; on EuroSAT the improvement over the second-best baseline is 6.9 points, and in the medical domains in Appendix L it is below 1.5 points. Section 1 motivates the attack specifically with 'specialized fields, where high-value models are the most common', yet the specialized-domain results are relegated to an appendix and show only marginal gains. The contributions and conclusion should either scope the claim to domains with adequate generative priors or present the medical results prominently in the main text, with a clear statement of where the method does and does not help.","section":"Section 1 and Appendix L"}],"minor_comments":[{"comment":"The paper does not state the random seeds or the exact seed-image selection procedure, and the DomainNet experiment uses only 10 of 345 classes; please report which classes are used and how seed images are sampled across the three repeats.","section":"Section 5.1 and Appendix L"},{"comment":"Line 20 of Algorithm 3 ends with an incomplete set-union expression ('S t+1'); please fix the notation so that the update is fully specified.","section":"Appendix A, Algorithm 3"},{"comment":"The text should clarify whether the same M synthetic images used to compute PC are also added to the positive/negative sets and used for training the attacker, and whether this double use is intentional under the query budget.","section":"Section 4.3"},{"comment":"The NSFW experiment uses a dataset of only 200 images and reports a single point estimate (73.0% accuracy); please provide variation across repeats or a note about the small evaluation set.","section":"Section 5.3"},{"comment":"The project page is mentioned but the manuscript does not provide code or a data-release link; for an attack paper that claims reproducibility through three repeats, a code release or a detailed implementation appendix would substantially help the community.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a security-oriented ML venue, and the proposed method is a genuine step forward in automating model stealing. The main concern is that the validity of the PC proxy—the signal that drives the whole evolutionary search—is validated only through the same victim model that defines both quantities, and the paper's own appendix on medical domains shows that the advantage largely evaporates exactly in the specialized settings that the introduction uses to motivate the attack. With error bars, an independent proxy validation, and a scoped presentation of the claims, these issues are fixable within a revision. I would not reject, but the current version overclaims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a solid, well-executed attack paper, and the combination it puts together is genuinely new in the literature. It takes PEZ-style hard prompt optimization, DA-Fusion-style seed personalization, and a genetic algorithm with victim hard-label feedback, and makes them work for model stealing with no class names and no manual prompts. One seed image per class and 500 queries per class is a real lowering of the barrier. The paper does a lot right: the threat model is realistic and clearly stated; the ablation isolating the reproduction step is useful; the InstructBLIP comparison shows that human-readable prompts are not the same as useful prompts; and the qualitative examples of prompts evolving to drop misleading features are convincing.\n\nCredit where due: the authors also include medical datasets in Appendix L where Stable Diffusion has weak priors, and they honestly report that Stealix's advantage shrinks to under two accuracy points there. That is the right kind of transparency, and it directly bounds the scope of the main claim.\n\nThe soft spots are real but not fatal. The main table has no error bars even though the experiments were repeated three times; the figures show confidence intervals, so the data exists and should be in the table. There is no code or data release, which matters for a security paper claiming a new attack. And the PC proxy is validated by Spearman correlation against L2 feature distance computed through the same victim model. That is not an independent test, and the stress-test note is right that a high-PC prompt could sit on the victim's side of the boundary while staying far from the true data manifold. However, the final metric is attacker test accuracy, not PC, so the central result is not a tautology.\n\nMy overall read: the paper works where the generative prior is adequate, and the authors are honest that gains are modest where it is not. The claim of being \"first\" is a little strong given that all the components exist, but the construction is new and the empirical comparison is fair. I would send this to a serious referee, mainly to push for error bars in the main table, code release, and a clearer statement that the attack's benefit is contingent on the diffusion prior's coverage of the target domain. It deserves a round of revision, not a desk reject.","headline":"Solid empirical attack paper with a genuinely new combination, but the headline gains shrink exactly where the threat model matters most, and the PC proxy is validated only against the same victim model it is meant to approximate.","tokens_in":19638,"tokens_out":2424,"would_cite":true,"duration_ms":25980,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stealix steals hard-label image classifiers with no class names and no human-written prompts, beating prompt-based baselines by up to 22.2 points.","keywords":["model stealing","prompt evolution","genetic algorithm","diffusion models","hard-label black-box attacks","prompt consistency","vision-language models","intellectual property"],"falsifier":"Train a victim classifier that is overconfident on out-of-distribution synthetic images, so that prompts far from the victim's real data still get high prompt consistency. If a surrogate trained on those high-PC images fails to improve test accuracy relative to a random-prompt baseline, the paper's proxy-metric assumption is falsified.","tokens_in":18595,"feed_emoji":"🕵️","tokens_out":9005,"duration_ms":82857,"temperature":0.7,"pith_summary":"This paper argues that a hard-label black-box image classifier can be functionally stolen even when the attacker has no class names and no prompt-writing skill. Stealix starts from one real seed image per class and lets the victim model guide the search through text-prompt space: a genetic algorithm proposes prompts, a diffusion model draws images from each prompt, and the victim's top-1 label decides how good the prompt is. On EuroSAT, PASCAL VOC, CIFAR-10, and DomainNet the resulting proxy model beats methods that are given class names or human-crafted prompts, with the largest gain 22.2 percentage points over the class-name baseline on CIFAR-10 at a budget of 500 queries per class. The wider point is that open-source generative models turn an untrained attacker into a viable threat, so restricting APIs to hard labels slows the attack but does not stop it.","feed_headline":"Stealix steals black-box models without a single human-written prompt","feed_subtitle":"One seed image per class plus the victim's own labels beats hand-written prompts by up to 22.2%.","key_machinery":"The load-bearing object is prompt consistency (PC), the fraction of M = 10 synthesized images from a given prompt that the victim classifies as the target class; it serves as the fitness function in the genetic algorithm and as a proxy for how close the synthesized images are to the victim data. Around it, prompt refinement minimizes a triplet contrastive loss between the embedding of a prompt and the embeddings of the seed image, a positively classified image, and a negatively classified image: positive images are pulled toward the prompt and negative images are pushed away. The reproduction step uses tournament selection, one elite retained per generation, crossover on image triplets, and mutation by resampling from the accumulated seed, positive, and negative sets. These three mechanisms convert a single seed image per class into a growing, labelled pool of query images without ever asking the attacker to describe the class.","core_discovery":"The paper's central claim, stated sympathetically, is that the victim model's own predictions contain enough information to replace human prompt engineering for model stealing. A prompt earns a high score (called prompt consistency, or PC) when the images it synthesizes are consistently classified into the target class by the victim, and the authors show statistically that high PC correlates with small feature-space distance to the victim's real data, with Spearman rho between -0.63 and -0.88 across the four datasets. Around that score the method builds an evolutionary loop: contrastive prompt refinement keeps the prompt close to the seed and to positively classified synthetic images while pushing it away from negatively classified ones, while tournament selection, crossover, and mutation recombine the best image triplets and retain the best prompt as an elite. The discovery is thus that victim feedback, not human knowledge, is the workable source of prompt quality for data synthesis in model stealing.","pith_inferences":["The PC loop can be read as a black-box probe of what a model's decisions reveal about its training distribution; model stealing is one application, and measuring a model's exposure to synthetic data is another.","The medical-dataset results imply that the ceiling is set by the generative prior: when the diffusion model cannot draw the domain, PC loses its meaning, so an attacker with a domain-tuned generator could extend the attack to niche fields.","If a defender could detect query images whose rate of victim agreement is abnormally high relative to their distance from known data, that would break the PC signal; the current hard-label defense does not attempt this."],"forward_implications":["A hard-label-only API no longer blocks prompt-based model stealing: top-1 predictions alone drive the entire prompt evolution loop.","Attacker accuracy improves by up to 22.2 percentage points over class-name or human-prompt baselines under a 500-query-per-class budget.","One seed image per class suffices across satellite imagery, PASCAL VOC objects, CIFAR-10, and six DomainNet domains.","Higher prompt consistency translates into a more accurate stolen model, so the attack improves as the query budget grows.","Because synthesis and surrogate training are decoupled, the same synthetic query images can be reused to tune the attacker's architecture or hyperparameters without further queries."],"supporting_citations":[{"why":"Supplies the pre-trained text-to-image diffusion model that synthesizes all query images in Stealix.","marker":"Rombach et al. (2022)"},{"why":"Provides the hard-prompt optimization routine that Stealix extends with a triplet contrastive loss and victim feedback.","marker":"Wen et al. (2024)"},{"why":"Establishes the functionality-stealing paradigm and the attacker-accuracy evaluation metric used throughout.","marker":"Orekondy et al. (2019)"},{"why":"DA-Fusion is the textual-inversion baseline that Stealix is compared against and whose seed-image setup it shares.","marker":"Trabucco et al. (2024)"},{"why":"ASPKD is the closest diffusion-based hard-label stealing baseline that Stealix must outperform.","marker":"Hondru & Ionescu (2023)"},{"why":"DFME is the data-free GAN baseline whose failure under hard labels and quantization motivates Stealix's design.","marker":"Truong et al. (2021)"},{"why":"Knowledge distillation supplies the reference upper bound for an attacker with access to the victim's training data.","marker":"Hinton et al. (2015)"},{"why":"Supplies the genetic-algorithm operations, tournament selection, crossover, and mutation, used for prompt reproduction.","marker":"Zames (1981)"},{"why":"Simulates a human attacker by writing prompts from seed images, yielding the comparison that Stealix beats despite using no class names.","marker":"Dai et al. (2023)"}],"fun_headline_variants":["Stealix steals models with no prompts, using victim feedback alone","Model stealing without prompts: Stealix evolves its own","Stealix: Victim labels beat human prompts in model theft","Promptless model stealing: Stealix evolves better prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a prompt which makes the victim consistently output the target class on synthesized images produces images close to the victim's actual class data; the paper supports this only by a correlation computed on the same victim models that supply the labels.","fun_headline_variants_meta":{"raw":{"variants":["Stealix steals models with no prompts, using victim feedback alone","Model stealing without prompts: Stealix evolves its own","Stealix: Victim labels beat human prompts in model theft","Promptless model stealing: Stealix evolves better prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2731,"prompt_tokens":929,"completion_tokens":1802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1731}},"tokens_in":545,"tokens_out":1802,"duration_ms":14138,"temperature":1.0,"reasoning_tokens":1731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:12:45.946626+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a victim classifier that is overconfident on out-of-distribution synthetic images, so that prompts far from the victim's real data still get high prompt consistency. If a surrogate trained on those high-PC images fails to improve test accuracy relative to a random-prompt baseline, the paper's proxy-metric assumption is falsified.","supporting_citations":[{"cited_title":"Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery","cited_arxiv_id":null,"evidence_quote":"Provides the hard-prompt optimization routine that Stealix extends with a triplet contrastive loss and victim feedback."},{"cited_title":"Knockoff nets: Stealing functionality of black-box models","cited_arxiv_id":null,"evidence_quote":"Establishes the functionality-stealing paradigm and the attacker-accuracy evaluation metric used throughout."},{"cited_title":"Effective data augmentation with diffusion models","cited_arxiv_id":null,"evidence_quote":"DA-Fusion is the textual-inversion baseline that Stealix is compared against and whose seed-image setup it shares."},{"cited_title":"Towards Few-Call Model Stealing via Active Self-Paced Knowledge Distillation and Diffusion-Based Image Generation","cited_arxiv_id":"2310.00096","evidence_quote":"ASPKD is the closest diffusion-based hard-label stealing baseline that Stealix must outperform."},{"cited_title":"J., and Papernot, N","cited_arxiv_id":null,"evidence_quote":"DFME is the data-free GAN baseline whose failure under hard labels and quantization motivates Stealix's design."},{"cited_title":"Genetic algorithms in search, optimization and machine learning","cited_arxiv_id":null,"evidence_quote":"Supplies the genetic-algorithm operations, tournament selection, crossover, and mutation, used for prompt reproduction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Simulates a human attacker by writing prompts from seed images, yielding the comparison that Stealix beats despite using no class names."}],"review_version":1}