Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs
Reviewed by Pith2026-06-30 05:19 UTCgrok-4.3pith:KYRUR2VCopen to challenge →
The pith
AI-Apps on model hubs exhibit broken access control, input injection, and credential leaks that enable code execution and backdoors.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that AI-Apps on leading platforms suffer from broken access control, insecure resource reuse, insufficient input validation, sensitive data exposure, and three novel architectural vulnerabilities inherent to the platform design; these issues are amplified from traditional web problems and appear at scale, with thousands of apps leaking credentials, hundreds enabling code execution, and tens harboring backdoors.
What carries the argument
The AI-App lifecycle mapped to OWASP-style risk taxonomies that surfaces five threat categories and ten attack vectors, including three novel architectural vulnerabilities.
If this is right
- Platforms must strengthen isolation and access controls for user-developed AI-Apps.
- Traditional issues such as world-readable logs become high-impact when combined with AI-App execution environments.
- Responsible disclosure of the identified issues can lead to platform-level fixes for the affected apps.
- The scale of findings indicates that untrusted third-party AI-Apps pose immediate risks to users performing inference or fine-tuning.
Where Pith is reading between the lines
- The amplification of generic web flaws in this ecosystem may require platform-specific security models rather than standard web hardening alone.
- Similar vulnerabilities could affect other AI hosting services that allow public custom apps with shared resources.
- Extending detection to private or fine-tuned apps on the same platforms would test whether the issues are limited to public listings.
Load-bearing premise
The custom analysis framework accurately detects credential leaks, input injection vulnerabilities, and embedded backdoors across 970,000 public AI-Apps at scale without significant false positives or missed cases.
What would settle it
A manual audit of a random sample of the apps flagged by the framework to verify the reported credential leaks, injection points, or backdoors, or confirmation that the platforms have not addressed the disclosed issues.
Figures
read the original abstract
AI-powered Applications (AI-Apps), hosted on platforms such as Hugging Face, are democratizing access to pre-trained models through online inference and fine-tuning services. While lowering AI adoption barriers, these platforms introduce an unexplored attack surface, as AI-Apps are often developed by untrusted parties with weak isolation and misconfigured security settings. In this paper, we present the first systematic security analysis of AI-Apps across three leading platforms. To structure our investigation, we map the AI-App lifecycle to established risk taxonomies (e.g., OWASP), identifying five threat categories and ten attack vectors ranging from generic web flaws to high-impact architectural issues. Our analysis reveals critical failures including broken access control, insecure resource reuse, insufficient input validation, and sensitive data exposure. Notably, we uncover three novel architectural vulnerabilities inherent to platform design and demonstrate how traditional issues (e.g., world-readable logs) are uniquely amplified in this ecosystem. To assess real-world impact, we develop an analysis framework Insightor and apply it to over 970,000 public AI-Apps. Alarmingly, we find thousands of apps leaking credentials, hundreds containing input injection vulnerabilities that allow arbitrary code execution, and tens harboring embedded backdoors -- indicating active exploitation. We have responsibly disclosed all findings to the affected platforms and developers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents the first systematic security analysis of AI-powered applications (AI-Apps) on pre-trained model hubs such as Hugging Face. It maps the AI-App lifecycle to OWASP risk taxonomies, identifying five threat categories and ten attack vectors. The authors develop the Insightor analysis framework, apply it to over 970,000 public AI-Apps, and report thousands of credential leaks, hundreds of input-injection vulnerabilities allowing arbitrary code execution, and tens of embedded backdoors. They also uncover three novel architectural vulnerabilities inherent to platform design and note amplification of traditional issues (e.g., world-readable logs). Responsible disclosure to affected platforms and developers is stated.
Significance. If the scanner-based findings hold after validation, the work would highlight substantial real-world security risks in the emerging AI-App ecosystem on model hubs, including novel platform-inherent issues and scaled traditional flaws. The large-scale empirical scan (970k apps) combined with OWASP structuring and responsible disclosure could inform platform hardening and developer practices. The absence of self-referential derivations or fitted parameters is a strength for direct observational claims.
major comments (2)
- [Insightor description and results sections] Section describing the Insightor framework and its application to the 970k-app corpus: the headline quantitative claims (thousands of credential leaks, hundreds of input-injection cases, tens of backdoors) rest entirely on Insightor detections, yet no false-positive rates, precision/recall figures, ground-truth sample validation, or manual confirmation of any detected issues are reported. This directly undermines the ability to distinguish reported counts from scanner artifacts.
- [Results section on empirical findings] Results reporting the scale findings: without any described validation procedure (e.g., manual review of a random subset or comparison against labeled AI-App code), the concrete counts cannot be assessed for reliability and are load-bearing for the central claim of 'active exploitation' and 'critical failures'.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on the validation of our empirical results. We agree that explicit validation is necessary to support the quantitative claims and will revise the manuscript accordingly.
read point-by-point responses
-
Referee: Section describing the Insightor framework and its application to the 970k-app corpus: the headline quantitative claims (thousands of credential leaks, hundreds of input-injection cases, tens of backdoors) rest entirely on Insightor detections, yet no false-positive rates, precision/recall figures, ground-truth sample validation, or manual confirmation of any detected issues are reported. This directly undermines the ability to distinguish reported counts from scanner artifacts.
Authors: We acknowledge that the submitted manuscript did not report false-positive rates or a formal validation procedure for Insightor. The framework employs rule-based and static analysis techniques (credential regex patterns, taint tracking for injection sinks, and signature matching for backdoors). In the revision we will add a dedicated validation subsection describing a manual review process on a random sample of detections across categories, including observed precision and any false positives encountered. This will be cross-referenced from the Insightor description. revision: yes
-
Referee: Results reporting the scale findings: without any described validation procedure (e.g., manual review of a random subset or comparison against labeled AI-App code), the concrete counts cannot be assessed for reliability and are load-bearing for the central claim of 'active exploitation' and 'critical failures'.
Authors: We agree and will incorporate the validation results directly into the results section. The revised text will qualify the reported counts with the sampling-based precision estimates and describe the validation methodology so that reliability can be evaluated. revision: yes
Circularity Check
No circularity: empirical mapping and scanning with no self-referential derivation
full rationale
The paper conducts an empirical security analysis by mapping the AI-App lifecycle to OWASP taxonomies, identifying threat categories and attack vectors through direct observation, and applying the custom scanner Insightor to public apps. No equations, fitted parameters, predictions, or self-citation chains appear in the provided text. Claims rest on standard external taxonomies and reported observations rather than any reduction of results to inputs by construction or prior author work. This is a standard non-circular empirical security study.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption OWASP risk taxonomies and established web vulnerability categories can be directly mapped to the AI-App lifecycle on model hubs
Reference graph
Works this paper leans on
-
[1]
https://docs.aws.amazon.com/secretsmanager/latest/user guide/intro.html
AWS Secrets Manage. https://docs.aws.amazon.com/secretsmanager/latest/user guide/intro.html
-
[2]
https://docs.gitlab.com/ci/variables/#mask-a-cicd-variab le
CI/CD Variable Mask. https://docs.gitlab.com/ci/variables/#mask-a-cicd-variab le
-
[3]
https://codeql.github.com/
CodeQL. https://codeql.github.com/
-
[4]
https://cog.run/
Cog. https://cog.run/
-
[5]
https://github.com/pooler/cpuminer/blob/master/minerd.1
cpuminer/minerd. https://github.com/pooler/cpuminer/blob/master/minerd.1
-
[6]
https://nvd.nist.gov/vuln/detail/cve-2023-6572
CVE-2023-6572. https://nvd.nist.gov/vuln/detail/cve-2023-6572
2023
-
[7]
https://nvd.nist.gov/vuln/detail/cve-2024-1540
CVE-2024-1540. https://nvd.nist.gov/vuln/detail/cve-2024-1540
2024
-
[8]
https://nvd.nist.gov/vuln/detail/cve-2024-1728
CVE-2024-1728. https://nvd.nist.gov/vuln/detail/cve-2024-1728
2024
-
[9]
https://nvd.nist.gov/vuln/detail/cve-2024-39236
CVE-2024-39236. https://nvd.nist.gov/vuln/detail/cve-2024-39236
2024
-
[10]
https://nvd.nist.gov/vuln/detail/cve-2024-47867
CVE-2024-47867. https://nvd.nist.gov/vuln/detail/cve-2024-47867
2024
-
[11]
https://cwe.mitre.org/data/definitions/78.html
CWE-78: Improper Neutralization of Special Elements used in an OS Command (‘OS Command Injection’) (4.17). https://cwe.mitre.org/data/definitions/78.html
-
[12]
https://github.com/DataDog/guarddog
DataDog/guarddog. https://github.com/DataDog/guarddog
-
[13]
https://trufflesecurity.c om/blog/travis-leaking-secrets-in-2023
Does Travis CI leak secrets in 2023? Truffle Security Co. https://trufflesecurity.c om/blog/travis-leaking-secrets-in-2023
2023
-
[14]
https://fastapi.tiangolo.com/
FastAPI. https://fastapi.tiangolo.com/
-
[15]
https://flask.palletsprojects.com/en/stable/
Flask. https://flask.palletsprojects.com/en/stable/
-
[16]
https://docs.github.com/en/account- and-profile/setting-up-and-managing-your-personal-account-on- github/managing-your-personal-account/deleting-your-personal-account
Github namespace reuse policy. https://docs.github.com/en/account- and-profile/setting-up-and-managing-your-personal-account-on- github/managing-your-personal-account/deleting-your-personal-account
-
[17]
https://gitstar-ranking.co m/
Gitstar ranking – Top GitHub users and repositories. https://gitstar-ranking.co m/
-
[18]
https://www.gradio.app/
Gradio. https://www.gradio.app/
-
[19]
https://developer.hashicorp.com/vault/docs
HashiCorp Vault. https://developer.hashicorp.com/vault/docs
-
[20]
https://huggingface.co/autotrain
Hugging Face AutoTrain . https://huggingface.co/autotrain
-
[21]
https://huggingface.co/docs/hugg ingface_hub/en/guides/cli
Hugging Face Command Line Interface (CLI). https://huggingface.co/docs/hugg ingface_hub/en/guides/cli
-
[22]
https://huggingface.co/docs/hub/en/api
Hugging Face Hub api endpoints. https://huggingface.co/docs/hub/en/api
-
[23]
https://huggingface.co/spaces
Hugging face spaces. https://huggingface.co/spaces
-
[24]
https://kubernetes.io/
Kubernetes. https://kubernetes.io/
-
[25]
https://huggingface.co/docs/huggingface_hub/en/guides/ manage-spaces
Manage your space. https://huggingface.co/docs/huggingface_hub/en/guides/ manage-spaces
-
[26]
https://replicate.com/meta
Meta - Replicate. https://replicate.com/meta
-
[27]
https://www.modelscope.cn/studios
ModelScope. https://www.modelscope.cn/studios
-
[28]
https://osv.dev/
OSV – Open Source Vulnerabilities. https://osv.dev/
-
[29]
https://owasp.org/Top10
OWASP Top 10 Web Application Security Risks. https://owasp.org/Top10
-
[30]
https://publicwww.com/
PublicWWW – Search Engine for Source Code. https://publicwww.com/
-
[31]
https://github.com/facebook/pyre-check
Pysa. https://github.com/facebook/pyre-check
-
[32]
https://replicate.com/
Replicate. https://replicate.com/
-
[33]
https://replicate.com/docs/topics/billing#public-models
Replicate billing. https://replicate.com/docs/topics/billing#public-models
-
[34]
https://datatracker.ietf.org/doc/html/rfc7519
RFC 7519: JSON Web Token (JWT). https://datatracker.ietf.org/doc/html/rfc7519
-
[35]
https://huggingface.co/docs/hub/en/s paces-run-with-docker
Run with docker – Hugging Face spaces. https://huggingface.co/docs/hub/en/s paces-run-with-docker
-
[36]
https://huggingface.co/docs/hub/en/security-secrets
Secrets Scanning. https://huggingface.co/docs/hub/en/security-secrets
-
[37]
https://huggingface.co/docs/hub/en/oauth
Sign in with Hugging Face. https://huggingface.co/docs/hub/en/oauth
-
[38]
https://huggingface.co/blog/space-secrets-disclo sure
Space secrets security update. https://huggingface.co/blog/space-secrets-disclo sure
-
[39]
https://huggingface.co/doc s/hub/en/spaces-dev-mode
Spaces Dev Mode: Seamless development in Spaces. https://huggingface.co/doc s/hub/en/spaces-dev-mode
-
[40]
https://huggingface.co/docs/hub/en/spaces-overview
Spaces Overview. https://huggingface.co/docs/hub/en/spaces-overview
-
[41]
https://streamlit.io/
Streamlit. https://streamlit.io/
-
[42]
https://www.tornadoweb.org/en/stable/
Tornado Web Server. https://www.tornadoweb.org/en/stable/
-
[43]
https://trufflesecurity.com/
Truffle Security Co. https://trufflesecurity.com/
-
[44]
https://www.uvicorn.org/
Uvicorn. https://www.uvicorn.org/
-
[45]
https://xmrig.com/
XMRig. https://xmrig.com/
-
[46]
Eihal Alowaisheq. 2019. Cracking wall of confinement: Understanding and analyzing malicious domain takedowns. InThe Network and Distributed System Security Symposium (NDSS)
2019
-
[47]
Michael Backes, Boris Köpf, and Andrey Rybalchenko. 2009. Automatic discovery and quantification of information leaks. In30th IEEE Symposium on Security and Privacy (S&P). 141–153
2009
-
[48]
Shared network vulnerability disclosure
Replicate blog. Shared network vulnerability disclosure. https://replicate.com/bl og/shared-network-vulnerability-disclosure
-
[49]
Konstantinos Chatzikokolakis, Tom Chothia, and Apratim Guha. 2010. Statistical measurement of information leakage. InInternational Conference on Tools and Algorithms for the Construction and Analysis of Systems. 390–404
2010
-
[50]
Anupam Das, Joseph Bonneau, Matthew Caesar, Nikita Borisov, and XiaoFeng Wang. 2014. The Tangled Web of Password Reuse. InThe Network and Distributed System Security Symposium (NDSS)
2014
-
[51]
AD Diego. 2017. Automatic extraction of API Keys from Android applications. PhD thesis, Ph. D. dissertation(2017)
2017
-
[52]
HTTP API
Replicate docs. HTTP API. https://replicate.com/docs/reference/http#models.lis t. 14 Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands
2026
-
[53]
Use a model as a training destination
Replicate docs. Use a model as a training destination. https://replicate.com/docs /topics/models/models-as-training-destinations/
-
[54]
Embed your Space in another website
Hugging Face documents. Embed your Space in another website. https://huggin gface.co/docs/hub/spaces-embed
-
[55]
Secret inputs for models
Replicate documents. Secret inputs for models. https://replicate.com/changelog/ 2024-06-07-secret-inputs-for-models
2024
-
[56]
Runhan Feng, Ziyang Yan, Shiyan Peng, and Yuanyuan Zhang. 2022. Automated detection of password leakage from public github repositories. InProceedings of the 44th International Conference on Software Engineering. 175–186
2022
-
[57]
What Is Cryptojacking? Definition and Explanation
Fortinet. What Is Cryptojacking? Definition and Explanation. https://www.fort inet.com/resources/cyberglossary/cryptojacking
-
[58]
Daniel Gruss, Michael Schwarz, Matthias Wübbeling, Simon Guggi, Timo Malderle, Stefan More, and Moritz Lipp. 2018. Use-after-freemail: Generalizing the Use-after-free Problem and Applying It to Email Services. InProceedings of the 2018 on Asia Conference on Computer and Communications Security (AsiaCCS). 297–311
2018
-
[59]
Yacong Gu, Lingyun Ying, Yingyuan Pu, Xiao Hu, Huajun Chai, Ruimin Wang, Xing Gao, and Haixin Duan. 2023. Investigating package related security threats in software registries. In2023 IEEE Symposium on Security and Privacy (S&P). 1578–1595
2023
-
[60]
Hijacking Safetensors Conversion on Hugging Face
HiddenLayer. Hijacking Safetensors Conversion on Hugging Face. https://hidd enlayer.com/innovation-hub/silent-sabotage/
-
[61]
Hang Hu, Peng Peng, and Gang Wang. 2019. Characterizing pixel tracking through the lens of disposable email services. In2019 IEEE Symposium on Security and Privacy (S&P). 365–379
2019
- [62]
-
[63]
Jason Jones, Wenxin Jiang, Nicholas Synovic, George Thiruvathukal, and James Davis. 2024. What do we know about Hugging Face? A systematic literature review and quantitative validation of qualitative claims. InProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 13–24
2024
-
[64]
Piergiorgio Ladisa, Henrik Plate, Matias Martinez, and Olivier Barais. 2023. SoK: Taxonomy of Attacks on Open-Source Software Supply Chains . In2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, Los Alamitos, CA, USA, 1509–1526. doi:10.1109/SP46215.2023.10179304
-
[65]
Tobias Lauinger, Abdelberi Chaabane, Ahmet Salih Buyukkayhan, Kaan Onarli- oglu, and William Robertson. 2017. Game of Registrars: An Empirical Analysis of Post-Expiration Domain Name Takeovers. In26th USENIX Security Symposium (USENIX Security 17). 865–880
2017
-
[66]
Kevin Lee and Arvind Narayanan. 2021. Security and Privacy Risks of Number Recycling at Mobile Carriers in the United States. In2021 APWG Symposium on Electronic Crime Research (eCrime). IEEE, 1–17
2021
-
[67]
2020.A Survey Study of Password Setting and Reuse
Qi Li. 2020.A Survey Study of Password Setting and Reuse. University of Delaware
2020
-
[68]
Daiping Liu, Shuai Hao, and Haining Wang. 2016. All your dns records point to us: Understanding the security threats of dangling dns records. InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS). 1414–1425
2016
-
[69]
Allison McDonald, Carlo Sugatan, Tamy Guberek, and Florian Schaub. 2021. The annoying, the disturbing, and the weird: challenges with phone numbers as identifiers and phone number recycling. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14
2021
-
[70]
Michael Meli, Matthew R McNiece, and Bradley Reaves. 2019. How bad can it git? characterizing secret leakage in public github repositories. InThe Network and Distributed System Security Symposium (NDSS)
2019
-
[71]
OWASP GenAI Security Project. 2025. Top 10 Risks & Mitigations for LLMs and GenAI Apps (2025). https://genai.owasp.org/llm-top-10/. Includes LLM01– LLM10 (2025) risk categories
2025
-
[72]
Joshua Avery Reed and JC Reed. 2020. Potential Email Compromise via Dangling DNS MX. (2020)
2020
-
[73]
Aakanksha Saha, Tamara Denning, Vivek Srikumar, and Sneha Kumar Kasera
-
[74]
In2020 International Conference on COMmunication Systems & NETworkS (COMSNETS)
Secrets in source code: Reducing false positives using machine learn- ing. In2020 International Conference on COMmunication Systems & NETworkS (COMSNETS). 168–175
-
[75]
Vibha Singhal Sinha, Diptikalyan Saha, Pankaj Dhoolia, Rohan Padhye, and Senthil Mani. 2015. Detecting and mitigating secret-key leaks in source code repositories. In2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. 396–400
2015
-
[76]
Marco Squarcina, Mauro Tempesta, Lorenzo Veronese, Stefano Calzavara, and Matteo Maffei. 2021. Can I Take Your Subdomain? Exploring Same-Site Attacks in the Modern Web. In30th USENIX Security Symposium (USENIX Security 21). 2917–2934
2021
-
[77]
Ke Coby Wang and Michael K Reiter. 2019. How to End Password Reuse on the Web. InThe Network and Distributed System Security Symposium (NDSS)
2019
-
[78]
Shenao Wang, Yanjie Zhao, Xinyi Hou, and Haoyu Wang. 2024. Large language model supply chain: A research agenda.ACM Transactions on Software Engineer- ing and Methodology(2024)
2024
-
[79]
Min Xu, Armin Namavari, David Cash, and Thomas Ristenpart. 2021. Search- ing encrypted data with size-locked indexes. InProceedings of USENIX Security Symposium. 4025–4042
2021
-
[80]
Zhou Yang, Jieke Shi, Prem Devanbu, and David Lo. 2024. Ecosystem of large language models for code.ACM Transactions on Software Engineering and Method- ology(2024)
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.