{"id":"41934fac-9f40-463f-92b5-08d39fd056a2","arxiv_id":"2505.03859","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Public model repositories host tens of thousands of easily downloadable deepfake generators, downloaded millions of times and mostly targeting women.","lead":"This paper counts and characterizes AI models that are built to generate images of real, identifiable people, mostly women, and finds tens of thousands of them publicly downloadable with millions of downloads. It matters because it shows the non-consensual deepfake problem is now a supply chain of ready-made generators, not just shared images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated 'Celebrity' tag proxy may inflate the headline count of 34,439 deepfake model variants; precision on a random sample is needed.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the validity of the 34,439 count depends on the 'Celebrity' tag as a proxy. This is the foundation of the paper's headline claim; if the tag has low precision, the scale of the phenomenon is overstated. The paper provides only an 'initial evaluation' without details, and its own limitations section flags potential false positives/negatives. My proposed test directly estimates the tag's precision on a random sample, which would settle whether the count is trustworthy. The non-consent characterization is also an inference from missing consent statements, but it is secondary to the count and the paper phrases it carefully. Therefore, the conditional verdict remains appropriate: the study is novel and likely directionally correct, but the central quantity needs validation before full acceptance. I agree with the reader's assessment.","tokens_in":20628,"tokens_out":3929,"duration_ms":36641,"concrete_test":"Randomly sample 400 model variants from the 34,439 'Celebrity'-tagged models, stratified by upload date and download quintile. Two independent annotators, blinded to the paper's conclusions, classify each model's card (name, description, example images, tags) as (1) intended to generate photorealistic images of a specific identifiable person, (2) intended to generate a specific identifiable person in non-photorealistic/stylized form, (3) generic 'celebrity' style or fictional character, or (4) not targeting an identifiable person. Compute the proportion in category 1 with 95% CI and inter-annotator agreement (Cohen's kappa). If the proportion is <90%, correct the 34,439 count and re-estimate downloads. Also sample 400 models NOT tagged 'Celebrity' from the same Civitai population to estimate false-negative rate and bound recall.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central number rests on Civitai's creator-assigned 'Celebrity' tag as a proxy for deepfake model variants (Section 3.1). The only validation is an 'initial evaluation' with no reported sample size, criteria, or inter-rater reliability. The tag could include false positives: stylized or cartoon celebrity likenesses, fictional characters, generic 'famous people' models, or models miscategorized for discoverability. The paper itself concedes reliance on 'accurate user tagging' could lead to false positives or negatives (Section 6). Part B's manual labeling found only ~13.6% of Flux/SD models (2,083/15,349) are deepfakes, but that sample is not drawn from the Celebrity-tagged population, so it does not validate the tag. If the tag's precision is materially below 100%, the 34,439 count and 14.9M download figure are overstates, weakening the 'rise of deepfakes' claim. The non-consent inference is secondary: absence of consent statements in model cards is suggestive but not conclusive, yet the headline count is the paper's most load-bearing and most fragile number.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical, three-part study of publicly downloadable text-to-image model variants that can generate images of identifiable people. Using metadata from Civitai and Hugging Face, the authors identify 34,439 Civitai models tagged 'Celebrity' (treated as deepfake model variants), with roughly 14.9 million cumulative downloads. They manually label 15,349 Stable Diffusion and Flux model variants across both platforms, finding 2,083 deepfake models, of which 96.4% target women. The paper also analyzes temporal trends, Terms of Service, and accessibility, arguing that LoRA fine-tuning enables the creation of deepfake models with as few as 20 images, 24GB VRAM, and about 15 minutes, and that platform enforcement and regulation lag behind the growth of the phenomenon.","tokens_in":20826,"tokens_out":2613,"duration_ms":27638,"significance":"If the headline estimates are correct, this is an important empirical contribution to the deepfake and NCII literature: it quantifies the supply side of non-consensual deepfake generators, documents a sharp temporal increase coinciding with Flux's release, and provides reproducible code and API-based data collection. The study's strengths include the independent measurement against platform APIs, the decision not to generate images, the large manually labeled sample in Part B, and the policy-relevant framing. However, the paper's central count rests on a creator-assigned tag whose precision is not systematically established, and the 'non-consensual' label is inferred from the absence of consent references in model cards. These issues affect the abstract's headline claims and need to be fixed before the results can be fully relied upon.","major_comments":[{"comment":"The headline count of 34,439 deepfake model variants rests entirely on Civitai's creator-assigned 'Celebrity' tag, but the validation reported in Section 3.1 is only described as an 'initial evaluation' with no sample size, criteria, or error rate. The manual labeling in Part B is not drawn from the Celebrity-tagged population, so it cannot validate the tag's precision. Since the tag is self-assigned by the same creators being studied, it may include stylized or fictional celebrity likenesses, 'famous people' compilations, or models miscategorized for discoverability, all of which would inflate the central count and the 14.9 million download figure. The authors should report a systematic validation: draw a random sample of Celebrity-tagged models, classify them independently against a pre-registered rubric that separates photorealistic identifiable-person models from other categories, and report precision, recall, and inter-rater reliability.","section":"Section 3.1, Table 2"},{"comment":"The manual labeling of 15,349 model variants appears to be a single-pass procedure without inter-rater reliability or dual coding. The criteria in Appendix C involve judgment calls, such as deciding when a character name implies an identifiable person and when example images are 'photorealistic' rather than cartoon depictions. Especially for the 2,083 models classified as deepfakes, the lack of reported inter-rater reliability makes the 96.4% female-targeting statistic and the Flux deepfake share in Table 7 harder to interpret. The authors should report the number of coders, a random subsample coded independently, and agreement statistics (e.g., Cohen's kappa), or otherwise justify why a single pass is reliable.","section":"Section 3.2, Appendix C"},{"comment":"The paper repeatedly refers to these models as 'non-consensual deepfake model variants' and as models 'without consent,' but the support for non-consent is the absence of consent references in model cards. Absence of a consent statement is not equivalent to evidence of non-consent, particularly because model cards on these platforms do not have a structured consent field. The claim would be strengthened by reporting how many model cards were examined, whether any explicit consent statements were found, and by softening the wording from 'non-consensual' to 'no evidence of consent identified' where the data only support the latter. This affects the title, abstract, and policy conclusions, so it should be addressed before publication.","section":"Section 4.3.1"}],"minor_comments":[{"comment":"Appendix A states 'the 34,440 Celebrity models' while the body and Table 2 consistently report 34,439; the count should be made consistent.","section":"Appendix A, Table 11"},{"comment":"The phrase 'posing a risk to public and non-public figures alika' appears to contain a typo ('alika' should likely be 'alike') and should be corrected.","section":"Section 4.3.2"},{"comment":"The limitation that Hugging Face monthly downloads are not directly comparable to Civitai lifetime downloads is acknowledged, but the abstract and discussion still present aggregate download figures without this caveat; one sentence noting the non-comparability when citing the 15 million figure would improve precision.","section":"Section 6, Limitations"},{"comment":"The paper does not state the date on which the Civitai API data were collected, beyond noting December 2024 in Figure 1; specifying the exact collection window would improve reproducibility.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The topic is squarely within FAccT's scope and the paper is likely to be influential, which raises the stakes for validating the main count. I would encourage the editor to require the tag-precision validation and inter-rater reliability as conditions for acceptance, rather than treating them as optional robustness checks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first big quantitative look at the public supply chain for deepfake image generators, and despite a soft spot in the headline number, the core finding—that thousands of downloadable models targeting identifiable people, mostly women, are a train ride away—holds up. Worth a serious referee.\n\nWhat is genuinely new: prior work has counted deepfake images or studied individual cases; this paper counts the generators themselves. It pulls 34k+ Civitai models tagged 'Celebrity', ~15M downloads, shows the Flux release accelerated uploads, and manually labels 2,083 SD/Flux models to find 96% target women. The LoRA/20-image/15-minute accessibility point is well documented with tutorials, and the ToS gap analysis is useful. They released code, did not generate any images for ethics reasons, and flag the main limitations themselves.\n\nSoft spots, in proportion: the 34,439 number rests entirely on Civitai's creator-assigned 'Celebrity' tag. The paper says there was an 'initial evaluation' but gives no sample size, criteria, or reliability check. The manual labeling in Part B is on a different population (SD/Flux model names, not Celebrity-tagged models), so it does not validate the tag. If the tag's precision is, say, 70–80%, the headline count and 14.9M downloads are overstates. The non-consent inference is also indirect: no consent statements found in model cards is evidence, but not proof. These are real concerns, but they do not sink the paper. Everything important is directional: even a 30–40% false positive rate leaves a very large, growing supply of deepfake models, and the gender imbalance is measured on the manually labeled set, not the tag. The paper itself concedes the tagging limitation.\n\nMinor: HF monthly vs Civitai lifetime downloads are not comparable (acknowledged); duplicates across platforms are retained (acknowledged); single-pass manual labeling without inter-rater reliability would matter more if the gender split were the headline, but the effect is huge (96/4) so a few mislabels won't flip it.\n\nWho it's for: AI governance, platform trust & safety, media studies, and anyone working on deepfake regulation. The paper deserves peer review; I'd ask the authors to validate the Celebrity tag on a random sample and report precision, and to add uncertainty or sensitivity analysis around the headline counts. That's a revision, not a rejection.","headline":"First large-scale count of the public deepfake-model supply chain; headline precision rests on an unvalidated tag, but the core finding is solid and deserves serious review.","tokens_in":21365,"tokens_out":2130,"would_cite":true,"duration_ms":20852,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that over 34,000 publicly downloadable deepfake model variants are available, with nearly 15 million cumulative downloads, and that 96% of a labelled sample target women.","keywords":["deepfakes","non-consensual intimate imagery","text-to-image models","LoRA fine-tuning","model-hosting platforms","Stable Diffusion","Flux","content moderation"],"falsifier":"Take a random sample of, say, 400 of the 34,439 models carrying the creator-assigned 'Celebrity' tag and have independent annotators, working without the study's labelling rubric, check whether each model's name, description, and example images actually target an identifiable real person and whether any consent statement appears. If more than about 10% of the sample turns out to target fictional characters, non-photorealistic subjects, or public figures with documented consent, the 34,000 count and the 'non-consensual' framing would need to be revised downward.","tokens_in":20397,"feed_emoji":"🖼️","tokens_out":6969,"duration_ms":64817,"temperature":0.7,"pith_summary":"This paper claims that the public supply of ready-made deepfake image generators is no longer a niche or marginal phenomenon. By reading metadata attached to model variants on two large public model-sharing platforms, it identifies more than 34,000 downloadable variants explicitly intended to produce images of identifiable people, with nearly 15 million cumulative downloads since late 2022. The paper further argues that the overwhelming majority of the carefully examined variants target women, that many signal intent to generate non-consensual intimate imagery, and that the cost of creating such a model has fallen to about 20 training images, 24GB of VRAM, and 15 minutes of fine-tuning time. If these numbers hold, the bottleneck for this kind of abuse is no longer technical skill or compute, but platform enforcement and legal intervention.","feed_headline":"34,000 deepfake image models online, 15 million downloads","feed_subtitle":"Ready-made generators mostly target women and can be built from 20 images on a consumer GPU.","key_machinery":"The load-bearing object is the model variant itself: a fine-tuned text-to-image model adapted to reproduce a specific person's likeness. The paper treats creator-assigned metadata tags as a lens onto this population, with one platform's 'Celebrity' tag marking variants intended to depict identifiable individuals and additional tags and description terms serving as red flags for sexualised intent. The named mechanism that makes the phenomenon scalable is low-rank adaptation (LoRA), a parameter-efficient fine-tuning technique that updates only small adapter matrices while keeping the base model frozen, letting a user create a likeness-specific model with as few as 20 images, 24GB of VRAM, and roughly 15 minutes of compute. The analysis is carried by comparing these metadata signals across two model families and two platforms, then manually labelling names and descriptions in a 15,349-model sample to separate deepfake variants from other fine-tunes.","core_discovery":"The paper's central discovery is that deepfake image generators have become a commodity available for direct download. Across the full platform census, 34,439 model variants carry a creator-assigned tag indicating they generate identifiable individuals, and those variants have been downloaded 14,908,183 times since November 2022. In a manually labelled sample of 2,083 deepfake variants from the Stable Diffusion and Flux model families, 96% target women, and 97 of the top 100 most downloaded variants target women. The authors interpret the absence of any consent statement in the examined model cards, together with tags and descriptions referencing sexualised or adult content, as evidence that many of these models are intended for non-consensual intimate imagery. The paper also finds that 80% of the tagged variants are LoRA adapters, that the release of Flux in August 2024 coincides with a sharp acceleration in uploads, and that by December 2024 deepfake-oriented LoRAs made up 44.3% of all Flux LoRA variants on the main platform examined.","pith_inferences":["Editorial inference: the public platform census almost certainly understates total deepfake model production, because locally trained models and models shared through private channels are invisible to metadata analysis; the paper itself says as much.","Editorial inference: the same metadata method could be rerun quarterly as an early-warning indicator for new base models, and it could be extended to video-generation models if comparable creator tags emerge.","Editorial inference: if platform terms were revised to require a consent statement before hosting any model depicting a real person, the paper's finding that no examined model card contains one suggests the public stock would shrink sharply, though production might move into less visible spaces.","Editorial inference: the 96% figure describes the manually labelled sample, not the full 34,000, so extrapolating it to the whole population would require validating the creator-assigned tag on the full set."],"forward_implications":["If the counts are right, the supply side of non-consensual deepfake imagery is already industrialised: each of the 34,000-plus downloadable variants can generate an effectively unlimited number of images.","Creation barriers are low enough that removing any individual model is unlikely to stop production, because the same small image set and consumer GPU can be reused locally without any public upload.","The concentration of uploads among a small number of prolific creators means platform-level user bans could reduce public availability more effectively than per-model takedowns.","Because 96% of the manually labelled deepfake models target women, the measurement implies that the abuse burden of this technology falls almost entirely on women, from celebrities to low-follower social media users.","The jump in uploads after the release of the Flux model family suggests that future high-quality open text-to-image base models are likely to produce another step-change in deepfake model creation unless access or fine-tuning is constrained."],"supporting_citations":[{"why":"Introduces low-rank adaptation, the parameter-efficient fine-tuning technique the paper identifies as the main creation method.","marker":"[22]"},{"why":"Announces the Flux model, whose August 2024 release coincides with the measured acceleration in deepfake model uploads.","marker":"[30]"},{"why":"Introduces the latent diffusion architecture underlying Stable Diffusion, the base model for most of the deepfake variants assessed.","marker":"[44]"},{"why":"Supplies the estimate that the main hosting platform carries over 100,000 fine-tuned Stable Diffusion variants, motivating the platform choice.","marker":"[31]"},{"why":"Platform training guide stating that Flux LoRA training needs only 20 to 30 images, used as evidence for how little data fine-tuning requires.","marker":"[56]"},{"why":"Tutorial advertising LoRA training in 15 minutes with 24GB of VRAM, cited for the accessibility of deepfake model creation.","marker":"[9]"},{"why":"Hosting platform content policy that the paper interprets as prohibiting non-consensual depictions or sexual content without consent.","marker":"[13]"},{"why":"Model-hub documentation cited for the download and variant counts showing the popularity of the base models.","marker":"[24]"}],"fun_headline_variants":["Deepfake generators now a consumer commodity: 35K models, 15M downloads","96% of deepfake models target women, 15M downloads since 2022","Easy deepfake: 20 images, 15 minutes, and a GPU","Flux LoRAs surge: 44% of all variants are deepfake tools","35,000 ready-made deepfake models, 15M downloads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The count depends on models being tagged 'Celebrity' by their uploaders, and the 'non-consensual' label depends on consent being absent from model cards; if the tag is used loosely or consent is simply not documented, the headline numbers overstate the problem.","fun_headline_variants_meta":{"raw":{"variants":["Deepfake generators now a consumer commodity: 35K models, 15M downloads","96% of deepfake models target women, 15M downloads since 2022","Easy deepfake: 20 images, 15 minutes, and a GPU","Flux LoRAs surge: 44% of all variants are deepfake tools","35,000 ready-made deepfake models, 15M downloads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2532,"prompt_tokens":1037,"completion_tokens":1495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":1390}},"tokens_in":653,"tokens_out":1495,"duration_ms":10669,"temperature":1.0,"reasoning_tokens":1390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:47:21.763480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 400 of the 34,439 models carrying the creator-assigned 'Celebrity' tag and have independent annotators, working without the study's labelling rubric, check whether each model's name, description, and example images actually target an identifiable real person and whether any consent statement appears. If more than about 10% of the sample turns out to target fictional characters, non-photorealistic subjects, or public figures with documented consent, the 34,000 count and the 'non-consensual' framing would need to be revised downward.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Announces the Flux model, whose August 2024 release coincides with the measured acceleration in deepfake model uploads."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Platform training guide stating that Flux LoRA training needs only 20 to 30 images, used as evidence for how little data fine-tuning requires."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Tutorial advertising LoRA training in 15 minutes with 24GB of VRAM, cited for the accessibility of deepfake model creation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Model-hub documentation cited for the download and variant counts showing the popularity of the base models."}],"review_version":1}