{"id":"5084f669-be45-4b09-911c-8e541921f94f","arxiv_id":"2508.12622","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using a knowledge graph of model and dataset relationships, this paper identifies more than 11,000 uncensored LLMs on Hugging Face and documents their use in cybercrime services.","lead":"Researchers mapped connections between open-source AI models and their training data on Hugging Face, then used graph-based machine learning to flag over 11,000 models they say can produce harmful content without guardrails. The study also finds these models are already embedded in hundreds of apps offering erotic role-play, malware code, and hacking help.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Derivation-edge inheritance fails for merges: Table 5 shows merged models are intermediate between censored and uncensored parents, so the 11,598 ULLM count is not fully supported by the graph-propagation assumption.","rationale":"I read the paper as a measurement study whose headline is the discovery of 11,598 ULLMs via a GNN over derivation relations. The mechanism must hold for this number to be meaningful. The reader's weakest assumption is correct and is the most load-bearing: Section 3.2 states a derived model inherits censorship features along derivation edges, but merging is a counterexample from the paper's own Table 5. The example of Konstanta-7B shows a merge of a high-RSR ULLM and a moderate-RSR model produces an intermediate RSR, so an uncensored parent does not guarantee an uncensored child. Since the GATv2 graph representation uses edges as plain directed pairs (Equation 1-2) without encoding the development method, the classifier must rely on node text features to distinguish merge semantics; this is not demonstrated. The 84-model validation, while valuable, only samples 12 merged models and all are predicted positives, so it cannot estimate false positives in the merge class or among predicted negatives. I therefore agree with the reader's identified weak assumption and recommend keeping the CONDITIONAL verdict, with the concrete merge-focused behavior test as the path to full acceptance.","tokens_in":33782,"tokens_out":7450,"duration_ms":78999,"concrete_test":"Behaviorally test the merge class: from the 744 merged ULLMs, identify those with at least one censored parent in the graph, randomly sample 50 such models (plus 50 merged models with only uncensored parents), and run the same CatQA 55-prompt RSR/AHS protocol used in Section 3.3. If mixed-parent merges show RSR near the censored baseline (23.12%) rather than the ULLM baseline (69.29%), the inheritance assumption fails and the 11,598 count needs downward revision; if they remain high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section 3.2's guilt-by-association: an edge from base to derived entity is used to propagate uncensored labels. This is not sound for all edge types. The paper's own Table 5 shows merged ULLMs have RSR 65.82% and AHS ~4.0, intermediate between their parents (example Konstanta-7B: 85.45% and 47.27% parents -> 58.18% merged), and prior work [122] shows merging can dilute or remove safety. Since the graph encodes edges only as (u,v) with no method or edge-type attribute in Equations (1)-(2), a merged model with one uncensored and one censored parent is aggregated like any other derivation; the model can be censored despite an uncensored parent. With 744 of 11,598 ULLMs reported as merges, a nontrivial false-positive rate in this class directly inflates the headline count. The 84-model behavioral validation includes only 12 merged models, all predicted positive, and does not sample mixed-parent merges or predicted negatives, so it does not bound this error. The finding that all 84 tested models are uncensored is real evidence, but it cannot certify the 11,598 population if the propagation rule fails on a substantial edge class.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents UFinder, a graph-based method to discover 'uncensored' large language models (ULLMs) on Hugging Face. The authors construct a directed knowledge graph whose nodes are LLMs and datasets and whose edges encode derivation relations (fine-tuning, merging, compression, abliteration, dataset generation) extracted from metadata. Starting from a small seed set of labeled ULLMs and uncensored datasets, they use a GATv2 classifier to propagate labels across the graph, reporting 11,598 discovered ULLMs from 17,104 collected LLMs. They validate a random sample of 84 predicted ULLMs behaviorally, finding high response success rates (RSR 69.29% vs 23.12% for censored models) and high average harmfulness scores. The paper then measures the ULLM ecosystem: development methods, malicious functionalities, downstream web and open-source ULLM applications, underground-forum promotion, and cross-platform spread. The authors conclude that an alarmingly large and exploitable ecosystem of ULLMs exists, and they propose mitigation strategies.","tokens_in":34048,"tokens_out":3997,"duration_ms":42311,"significance":"If the discovery method and the headline counts hold, this is the first systematic measurement of uncensored LLMs at platform scale and an important contribution to AI-security research. The external behavioral validation on 84 independently sampled models is a genuine strength: all 84 predicted ULLMs generated harmful content without refusal at substantially higher rates than censored models, giving a binomial 95% confidence lower bound of about 95.7% for precision on the validated sample. The paper also provides useful downstream measurements of web applications, GitHub projects, and underground-forum listings, and it will release artifacts including the ULLM catalog and an uncensored QA benchmark. The central finding—that a large ecosystem of ULLMs is readily available and actively exploited—is plausible and societally important, but the exact scale claim (11,598 ULLMs) rests on assumptions that the paper itself partially undermines, as detailed below.","major_comments":[{"comment":"The core propagation rule treats all derivation edges uniformly in the GATv2 formulation, but the paper's own Table 5 shows that merging produces intermediate uncensorship: merged models have RSR and AHS between those of their parents (e.g., Konstanta-7B gives 58.18% RSR from 85.45% and 47.27% parents). Since edges in Eqs. (1)-(2) carry no type attribute, a merged model with one uncensored and one censored parent is aggregated like any other derivation, so the label can propagate to a model that is effectively censored despite an uncensored ancestor. With 744 of the 11,598 reported ULLMs created by merging, a nontrivial false-positive rate in this edge class directly inflates the headline count. The 84-model behavioral validation includes only 12 merged models, all predicted positive, and does not sample mixed-parent merges or predicted negative models, so it cannot bound this error. I recommend making the model edge-type aware (e.g., separate attention per relation type) and validating a stratified sample that includes predicted negatives and mixed-parent merges.","section":"Section 3.2, Eqs. (1)-(2); Section 4.2, Table 5"},{"comment":"The count of 11,598 ULLMs treats ordinary base models lacking any alignment training (Mistral-7B-v0.1, gpt-j-6b, zephyr-7b-beta, etc.) and 8,647 compressed copies (mostly quantization variants of the same model) as distinct ULLMs. These are not independent uncensored models: a quantized GGUF copy of dolphin-2.8-mistral-7b is a duplicate artifact, not a new ULLM, and a base model without instruction tuning may respond without refusal simply because it has not learned to follow harmful instructions, rather than because it is 'uncensored' in the sense used for fine-tuned ULLMs. The 'alarming scale' narrative (e.g., '25 models downloaded over 1 million times') is thus overstated in terms of distinct exploitable models. The paper should report the number of distinct model families, separate base models from fine-tuned/abliterated ULLMs, and quantify how many of the 11,598 are unique versus duplicate quantizations.","section":"Section 4.1, Table 4; Section 4.2 (Compressing ULLMs)"},{"comment":"The five-fold cross-validation reported in Table 2 is performed on ground-truth nodes that are part of the same graph used for training, so the high accuracy (98.35%) may be inflated by transductive label leakage through graph structure; the independent behavioral validation is the more credible estimate. However, that validation samples only 84 predicted positives and 22 predicted negatives, with no confidence intervals and no stratification by derivation method or edge type. The 84/84 result gives a useful lower bound on precision for the sampled predicted-positive population, but it cannot certify the full population of 11,598 if the propagation rule fails on a substantial edge class such as merges. The paper should report confidence intervals for the validation metrics and explicitly stratify the validation sample by edge type and by predicted positive versus predicted negative status.","section":"Section 3.3, Table 2 and 'Validation' paragraph"}],"minor_comments":[{"comment":"The sets M_t (15,732) and M_m (1,566) sum to 17,298, yet the total is reported as 17,104; please clarify the overlap and how the union was computed.","section":"Section 3.3, Table 1"},{"comment":"The last row contains a typo: '98.33&' should be '98.33%'.","section":"Table 2"},{"comment":"Figure 4 renders as garbled '/uni/...' sequences rather than a readable model-type chart; please replace with a proper figure.","section":"Figure 4"},{"comment":"The dataset validation uses only 10 datasets (1 de-aligned, 2 toxic, 7 censored) with no confidence intervals, which is too small to support the general claim about dataset labels; please expand the sample or temper the conclusion.","section":"Appendix A"},{"comment":"The definition of ULLM is applied to base models that simply lack safety fine-tuning, such as Mistral-7B-v0.1; please clarify the distinction between 'no refusal training' and 'uncensored' and discuss how this affects the interpretation of RSR measurements for such models.","section":"Section 2.2 / Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to appeal to the security community and the empirical measurements are valuable. The main concern is not the direction of the finding but the precision of the scale claim: the uniform-edge propagation rule conflicts with the paper's own evidence on merges, and the count of 11,598 conflates duplicate quantization artifacts with distinct ULLMs. Both issues are addressable within the manuscript's scope. I would avoid the term 'first systematic study' being read as 'first to observe the phenomenon', since Malla [140] already documented ULLMs as backends; the novelty is in the large-scale discovery method and ecosystem measurement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives us something we didn't have: a systematic, large-scale picture of uncensored open-source LLMs on Hugging Face, plus a new method (derivation graph + GATv2) for discovering them. The ecosystem analysis is rich — 11,598 flagged models, download statistics, malicious apps, underground forum listings, cross-platform spread. The behavioral validation on 84 sampled models is real evidence: ULLMs refuse far less and produce more harmful content than censored models. That part holds up.\n\nWhere it gets soft is the count itself. The load-bearing step is guilt-by-association: if model B is derived from uncensored model A, B inherits the label. For fine-tuning on uncensored data, abliteration, and compression that is mostly sound. But the paper's own Table 5 shows merging produces models that are intermediate between an uncensored and a censored parent, and prior work (ref 122) agrees. The graph edges in Equations (1)-(2) carry no method or edge-type attribute, so a merged model with mixed parents is aggregated the same way as any other derivation. With 744 merges in the 11,598 total, the false-positive rate in that class directly inflates the headline number. The validation sample includes only 12 merged models, all predicted positive, and no predicted negatives, so it cannot bound this error. The stress-test note is right about this.\n\nThere are two other soft spots. First, the ground truth for both training and validation leans on metadata (tags, disclaimers, organizational claims) rather than behavior; the 84-model behavioral test is independent but small. Second, the paper counts ordinary base models like Mistral-7B-v0.1 as ULLMs. That is defensible under their definition (no safety guardrails), but it inflates the total in a way that could be separated out in the reporting. The code and data are promised but absent, so independent replication is not possible yet.\n\nStill, the central direction is credible. A large ecosystem of uncensored models exists, and it is being used. The precise number 11,598 is an estimate with nontrivial error, not a certified count. The authors are honest enough to include Table 5, which is exactly the evidence that undermines their propagation rule for merges.\n\nWho is this for? AI security researchers, Hugging Face and other platform moderators, and anyone measuring the open-model ecosystem. It should go to peer review, not be desk-rejected, but the revisions need to be substantive: release the artifacts, validate a sample of predicted negatives and mixed-parent merges, and report the count with merge-derived models excluded. I'd probably cite it once the data is out.","headline":"First real census of the uncensored-LLM ecosystem, with a graph-mining method that is worth knowing; the headline 11,598 count is probably inflated, but the paper deserves peer review.","tokens_in":34586,"tokens_out":1758,"would_cite":true,"duration_ms":20571,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By modeling model-to-model derivation, the paper identifies 11,598 uncensored LLMs on a major open-source platform from a small labeled seed set.","keywords":["uncensored large language models","graph attention networks","knowledge graph","model derivation","harmful content","cybercrime","model safety","open-source model ecosystems"],"falsifier":"Take the models UFinder labels uncensored and run the paper's 55-prompt refusal protocol on a random sample much larger than 84, say several hundred. If a substantial share of those models refuses most harmful prompts, or if a model built by merging two censored bases is labeled uncensored and then refuses, the guilt-by-association edge fails. The same test can be run adversarially: craft a model card that falsely claims fine-tuning from an uncensored model and check whether the graph labels it uncensored.","tokens_in":33584,"feed_emoji":"🕵️","tokens_out":8946,"duration_ms":89795,"temperature":0.7,"pith_summary":"The paper's central claim is that uncensored large language models can be discovered at scale not by testing each model, but by reading the derivation traces in their metadata. Models and datasets become nodes in a directed graph, with edges for fine-tuning, merging, compression, abliteration, and dataset creation; a graph attention network (GATv2) propagates \"uncensored\" labels from a few thousand hand-labeled seeds to the rest of the graph. On a collection of 17,104 models, this finds 11,598 uncensored models. A randomly sampled validation of 84 of them found all 84 generated harmful content without refusal, with an average response success rate of 69.29% compared with 23.12% for censored models. If the result holds, uncensored models are not a curiosity but a large, reusable infrastructure layer for malicious applications.","feed_headline":"Graph mining uncovers 11,598 uncensored AI models","feed_subtitle":"Derivation metadata plus a small seed of known unsafe models exposes a large ecosystem powering malicious services.","key_machinery":"The central mechanism is the derivation-relation knowledge graph, named UFinder. Nodes are LLMs or datasets; a directed edge from entity $u$ to entity $v$ means $v$ was built from $u$ by fine-tuning, training, merging, compression, abliteration, replication, or dataset generation. Node features combine a learned embedding of description and derivation text with a one-hot entity-category vector. GATv2 then re-embeds each node as an attention-weighted sum of its neighbors' transformed features, $h'_v = \\sum_{u \\in N(v)} \\alpha_{vu} W h_u$, and two fully-connected classifiers read censorship probabilities from the embeddings. The work it does is label propagation through inheritance: the seed labels flow along derivation edges, so an unlabeled model that inherits from an uncensored base becomes visible without being downloaded and queried.","core_discovery":"On the paper's own terms, the discovery is that a model's censorship status is legible through its heritage. Even when a model card says nothing about safety filters, the metadata usually says what the model was trained on, what it was merged from, what it was compressed from, or what dataset it generated. UFinder encodes this metadata into node features, builds a directed derivation graph, and applies GATv2 to classify each node. Starting from 1,218 labeled uncensored models plus labeled uncensored and censored datasets, it reports 11,598 uncensored models and 559 uncensored datasets from 17,104 models and 1,309 datasets. The validation result is the load-bearing number: all 84 randomly sampled ULLMs produced harmful content without refusal, while 22 censored models refused most of the same prompts. The paper also finds that compressed models account for 74.51% of the ULLMs, and that merging produces intermediate behavior, so inheritance is strongest for fine-tuning, abliteration, and compression and only partial for merges.","pith_inferences":["Editorial inference: if metadata can be forged, the method is gameable—an attacker could launder a censored model through a fake uncensored lineage, so deployment should combine graph labels with spot-checking.","Editorial inference: merging's intermediate behavior suggests censorship is graded rather than binary; a three-level label (censored, partially uncensored, uncensored) would better match the mechanism.","Editorial inference: the same derivation-graph approach should transfer to other safety-relevant properties, such as license violations, backdoors, or data-poisoning provenance, since it only needs honest or detectable parent-child metadata.","Editorial inference: the 84-model validation covers roughly 0.7% of the positive predictions, so the graph's precision at full scale is not yet measured; a much larger randomized validation is the natural next step."],"forward_implications":["Platforms that host or mirror open models inherit the exposure: the paper counts 3,826 of the identified ULLMs reappearing on five other hosting platforms.","The population is cheap to grow: most ULLMs are quantized or otherwise compressed copies of ULLMs, and publicly shared scripts automate fine-tuning, merging, and abliteration.","The models are commercially exploited: 52 web applications were confirmed to route through 98 open-source ULLMs, with some charging subscription fees, and 25.5% of those ULLMs were used against their license terms.","Because the census relied on metadata, it is a lower bound: developers who hide or falsify derivation information will be invisible to the method."],"supporting_citations":[{"why":"Supplies the prior case studies of malicious LLM applications and the notion of ULLM backends that this study extends.","marker":"[140]"},{"why":"Provides the GATv2 graph-attention architecture that UFinder uses to propagate labels over the derivation graph.","marker":"[120]"},{"why":"Introduces graph attention networks, the base attention mechanism behind the classifier.","marker":"[154]"},{"why":"Establishes that refusal behavior in aligned LLMs is mediated by a single direction, which underpins the abliteration derivation edge.","marker":"[117]"},{"why":"Supplies the harmful QA benchmark whose 55 prompts are used to validate that discovered ULLMs produce harmful content without refusal.","marker":"[119]"},{"why":"Documents the platform's hundreds of thousands of hosted LLMs, the scale that makes exhaustive per-model testing infeasible.","marker":"[141]"},{"why":"Defines the refusal-response and harmfulness scoring protocol used to compare ULLMs with censored models.","marker":"[147]"},{"why":"Supplies the standardized harmfulness evaluation rubric used in the validation scoring.","marker":"[142]"}],"fun_headline_variants":["11,598 uncensored LLMs found via model lineage","Graph AI exposes 11k+ uncensored LLMs","Model family tree reveals 11,598 unsafe AIs","Abliteration and merging spawn 11k unsafe LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that derivation metadata honestly reflects inheritance of censorship status: if a model card says a model was fine-tuned from, compressed from, or trained on an uncensored model or dataset, the derived model is assumed uncensored. The paper's own merging results show this is only approximate for merges, where a merged model can land between its censored and uncensored bases.","fun_headline_variants_meta":{"raw":{"variants":["11,598 uncensored LLMs found via model lineage","Graph AI exposes 11k+ uncensored LLMs","Model family tree reveals 11,598 unsafe AIs","Abliteration and merging spawn 11k unsafe LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000475,"raw_usage":{"total_tokens":2413,"prompt_tokens":1057,"completion_tokens":1356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":1286}},"tokens_in":673,"tokens_out":1356,"duration_ms":10339,"temperature":1.0,"reasoning_tokens":1286,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:20:06.981831+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the models UFinder labels uncensored and run the paper's 55-prompt refusal protocol on a random sample much larger than 84, say several hundred. If a substantial share of those models refuses most harmful prompts, or if a model built by merging two censored bases is labeled uncensored and then refuses, the guilt-by-association edge fails. The same test can be run adversarially: craft a model card that falsely claims fine-tuning from an uncensored model and check whether the graph labels it uncensored.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prior case studies of malicious LLM applications and the notion of ULLM backends that this study extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the harmful QA benchmark whose 55 prompts are used to validate that discovered ULLMs produce harmful content without refusal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the platform's hundreds of thousands of hosted LLMs, the scale that makes exhaustive per-model testing infeasible."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the refusal-response and harmfulness scoring protocol used to compare ULLMs with censored models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the standardized harmfulness evaluation rubric used in the validation scoring."}],"review_version":1}