{"id":"d30083d7-2e49-44f4-a699-e4a13e9a2fca","arxiv_id":"2505.02502","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An internet-wide scan of 320,102 public-facing LLM services across 15 frameworks finds widespread plain-HTTP exposure, weak TLS, and unauthenticated access to model and system APIs.","lead":"This paper scanned the public internet and counted 320,102 self-hosted large language model services, and found that most are reachable without encryption or authentication. It is a broad early map of how exposed real-world LLM deployments are, useful to framework developers and security teams.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FOFA signature precision is unquantified; the Jupyter and FastAPI signatures in Table I are not LLM-specific, so the 320,102 census and all rates computed from it carry unknown bias.","rationale":"The reader's weakest assumption is the same one I find most load-bearing. The paper's strongest and most general claim, that 320,102 public LLM services exist and a large share are exposed, rests on FOFA queries whose precision is never quantified. For Ollama and llama.cpp, the fingerprints are fairly specific, and the Ollama API sampling is internally consistent, so the qualitative security findings for specific frameworks are credible. But Jupyter and FastAPI are broad-purpose tools; including 29,057 of them as 'LLM services' without validation can inflate the total and alter the mixed percentages. This is not a reason to reject the paper: the core qualitative message likely survives, and the authors provide artifacts. It is a reason to keep the verdict conditional until the census precision is measured and the reported totals are reconciled. I agree with the reader and recommend no change to the conditional verdict.","tokens_in":16721,"tokens_out":4561,"duration_ms":52695,"concrete_test":"Independently re-run the FOFA queries and draw a stratified random sample of at least 100 hits per framework signature (minimum 1,500 hosts). For each host, determine ground truth by connecting directly and checking: (1) server header/banner matches the framework; (2) the framework-specific API responds (e.g., /api/tags for Ollama, /api/kernels for Jupyter, /openapi.json for FastAPI); (3) responses expose model names or inference endpoints rather than generic web content. Compute precision with 95% confidence intervals per signature. If Jupyter or FastAPI precision is materially below 90%, remove those signatures or weight the totals by precision, then recompute the 320,102 census count and the HTTP/TLS exposure rates; if the revised rates shift by more than a few percentage points, the headline should be restricted to the framework-specific subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim depends on identifying every FOFA hit as an LLM service. In Table I, two of the largest non-Ollama contributions use generic fingerprints: Jupyter Notebook (24,531 hits via body=Jupyter Notebook) and FastAPI (4,526 via title/body FastAPI plus common ports). These are general-purpose developer tools, not LLM frameworks; a Jupyter server or FastAPI REST app can exist with no language model behind it. The paper states in Section III-B that queries were refined and 'a representative subset' manually validated, but it reports no per-signature precision, no sample size, and no criteria for confirming LLM identity. Consequently, the 320,102 'public-facing LLM services' total and derived percentages (e.g., 'over 40% plain HTTP', 'over 210,000 without valid TLS metadata') mix unknown numbers of non-LLM hosts into both numerator and denominator. The quantitative central claim is therefore not yet established, even though framework-specific findings such as Ollama's exposed API behavior are supported by narrower evidence. The numeric inconsistency between 320,102 (abstract/Table I) and 320,012 (Section IV-A) is secondary but reinforces that the dataset total is not stable enough for precise headline percentages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a large-scale internet measurement of publicly accessible LLM deployments. Using FOFA queries built from per-framework feature catalogs, the authors claim to have identified 320,102 public-facing LLM services across 15 frameworks, extracted 158 API endpoints in 12 functional categories, and probed these endpoints to characterize unauthenticated access, TLS usage, certificate hygiene, and security risk patterns. The central qualitative finding is that many self-hosted LLM deployments are reachable without authentication or transport encryption, with Ollama and ComfyUI highlighted as especially permissive. The paper concludes with secure-by-default recommendations for framework developers, operators, and the broader ecosystem.","tokens_in":16858,"tokens_out":3424,"duration_ms":40296,"significance":"If the quantitative claims survive scrutiny, this would be a useful and timely measurement study of a rapidly evolving ecosystem. The paper's strengths include a reproducible artifact repository, a transparent four-step methodology (framework selection, FOFA discovery, endpoint probing, risk analysis), and per-framework endpoint catalogs that can inform future audits. The Ollama-specific finding that a large share of public instances respond to unauthenticated model-management calls is credible and consistent with prior independent measurements. The novelty lies in the breadth of the census and the systematic risk taxonomy, rather than in a new attack or defensive technique. However, the significance is currently bounded by the unquantified precision of the discovery signatures and by unresolved numerical inconsistencies in the reported totals.","major_comments":[{"comment":"The census total of 320,102 is built on signatures that are not LLM-specific. Two of the largest non-Ollama contributions use generic fingerprints: Jupyter Notebook (24,531 hits via body=Jupyter Notebook) and FastAPI (4,526 hits via title/body FastAPI plus ports 8000/8080). These identify any Jupyter or FastAPI web application, not only LLM deployments. The manuscript states that queries were refined and 'a representative subset' was manually validated, but no per-signature precision, validation sample size, or confirmation criteria are reported. Because the headline percentages ('over 40% plain HTTP', 'over 210,000 without valid TLS metadata') are computed over this census, unknown numbers of non-LLM hosts enter both numerators and denominators. Please report per-signature precision and recall estimates, or restrict the census and all derived rates to frameworks with LLM-specific fingerprints.","section":"Section III-B and Table I"},{"comment":"The dataset totals are not reconciled: the abstract and Table I report 320,102, while Section IV-A and the RQ1 answer report 320,012. Table IV uses sample denominators that do not match Table I (e.g., Open WebUI 43,947 vs 37,242; Jan 380 vs 28,445; NextChat 20,000 vs 25,883), and Table VII reports 155,424 invocations while Table I lists 155,423 Ollama services. This makes it impossible to determine which denominator each response rate is computed against and undermines the quantitative precision of the main claims. Please reconcile the totals and state the sampling universe, sample size, and weighting procedure for every reported rate.","section":"Section IV-A and answer to RQ1"},{"comment":"The definition and detection of 'template responses' is not specified. The Ollama response rate changes from 35.23% to 15.28% when these are excluded, and the 14.69% full-dataset rate is used to validate that exclusion. Without a concrete criterion (e.g., exact-body matching or content hashing) and evidence that the same criterion was applied consistently to other frameworks, the reported framework-level responsiveness and the statement that template responses were not observed elsewhere are not verifiable. Please define the criterion, quantify how many responses per framework fall into it, and report rates both with and without such responses.","section":"Section III-C and Table IV"},{"comment":"The paper does not quantify the precision of the FOFA queries used to enumerate many of the frameworks in Table I, nor does it define what constitutes a distinct service. The high-traffic-domain analysis shows thousands of instances behind shared IPs and certificates (e.g., 6,206 instances under nellasushi.es served by two IPs), which raises the possibility that the same logical deployment is counted multiple times. Please state how unique services are defined, how duplicate records or multi-instance hosts are handled, and how this affects the geographic, organizational, and rate-based analyses.","section":"Section III-B and Section IV-A"}],"minor_comments":[{"comment":"The phrase 'over 210,000 endpoints lacked valid TLS metadata' conflates certificate validity with certificate subject naming; Section IV-A-5 actually analyzes subject CNs such as localhost and nan, which does not establish that TLS is absent or invalid. Please clarify the TLS measurement and use consistent terminology.","section":"Abstract and Section IV-A"},{"comment":"There is a typo in the first paragraph: 'OW ASP' should be 'OWASP'.","section":"Section II-A"},{"comment":"The cells reporting 'successful / total' are ambiguous because the denominator includes endpoints that were never probed for a given framework; please clarify whether 'total' denotes probed endpoints or all documented endpoints.","section":"Table V"},{"comment":"The FOFA feature catalog in Table I lists key features and counts but does not provide the exact query expressions used for each framework; making the full query set available in the artifact would improve reproducibility.","section":"Section III-B"},{"comment":"The note says 'Based on 155,424 sampled LLM service invocations', but an invocation count is not the same as a unique service count; please clarify whether the denominator is unique services or probe attempts.","section":"Table VII"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important topic, and the qualitative direction of the findings is plausible. The main obstacle is methodological: the census precision and internal count inconsistencies must be fixed before the quantitative claims can be used as evidence. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would also encourage the authors to make the per-signature validation data part of the artifact release."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of arXiv:2505.02502. The headline is right to be conditional.\n\nWhat's actually new: this is the broadest public comparison of self-hosted LLM exposure I know of — 15 frameworks, per-framework unauthenticated response rates, 158 API endpoints in 12 functional categories, and the Ollama 'template response' phenomenon. The authors ship an artifact repo, and the qualitative core — a lot of self-hosted LLM services are reachable without auth or TLS and leak model/system information — is consistent with prior measurements and with what operators see in the field. The framework-level differences (Ollama and Llamafile respond to ~10-15% of unauth requests; Open WebUI and Gradio effectively don't) are genuinely useful for prioritization.\n\nThe soft spots are real but not fatal. First, the census definition. Table I includes Jupyter Notebook (24,531) and FastAPI (4,526) based on generic fingerprints. Those are general developer tools, not necessarily LLM deployments. FOFA signature precision is never quantified, and 'a representative subset' manually validated is not enough to establish identity for those two framework buckets. So the 320,102 total and rates computed from it carry unknown bias. This doesn't sink the qualitative claim — strip out Jupyter and FastAPI and you still have close to 290k Ollama/Open WebUI/etc. hits — but the paper should present a precision analysis and a corrected range.\n\nSecond, the Ollama template-response exclusion. The abstract says 'over 35%' but the body says excluding generic template responses drops the rate to 15.28%, matching the full-dataset rate. The criterion is speculative ('we speculate...') and not operationalized. That's a significant swing in a headline number; it needs a concrete definition, examples, and inter-rater validation.\n\nThird, minor but telling: the abstract says 320,102, Section IV-A says 320,012, and 'over 210,000' appears three times with slightly different referents. That kind of sloppiness undercuts confidence in the precision of the rest.\n\nThe central argument holds up. It's a measurement study against an external ground truth, no circular derivation, and the related work already contains large-scale measurements of exposed LLM services, so the 'first large-scale' claim needs a softer wording and a direct comparison.\n\nWho benefits: security tooling folks, LLM framework maintainers, and anyone studying internet-scale misconfiguration. The paper deserves a serious referee — the comparative data and the template-response observation are worth publishing with revisions. I'd recommend accept with major revisions, contingent on precision analysis and reconciliation of the census numbers.\n\nBest.","headline":"Broad new map of exposed LLM services, but the census total over-counts via generic fingerprints; the qualitative findings hold and the paper merits review.","tokens_in":17484,"tokens_out":3030,"would_cite":true,"duration_ms":33456,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Internet-wide scanning finds 320,102 public-facing LLM services with widespread weak or missing security, including over 40% on plain HTTP and unauthenticated API leaks of model data.","keywords":["LLM deployment security","internet-wide measurement","FOFA","Ollama","API exposure","TLS misconfiguration","unauthenticated access","secure-by-default"],"falsifier":"Re-run the same FOFA queries on a fresh sample, fetch each matched host's landing page, and send one framework-specific API request per host; if many matched hosts do not actually serve LLM functionality, or if Ollama's 'template responses' turn out to be generic reverse-proxy pages rather than Ollama runtime answers, then the 320,102 census and the reported response rates overstate the true exposure.","tokens_in":16422,"feed_emoji":"🔓","tokens_out":6091,"duration_ms":64198,"temperature":0.7,"pith_summary":"The paper attempts to establish that self-hosted large language model deployments are widely exposed to the public internet with insecure defaults. Using the FOFA asset search engine, it enumerates 320,102 public-facing services across 15 deployment frameworks and, after probing 158 documented API endpoints, finds that more than 40% of endpoints use plain HTTP, over 210,000 endpoints lack valid TLS metadata, and frameworks such as Ollama respond to more than 35% of unauthenticated API requests, with about 15% leaking model or system information. The authors argue these problems are systemic, not isolated accidents, and that they produce concrete risks of model theft, resource abuse, and system compromise. The study matters because it gives the first large-scale quantitative map of the self-hosted LLM attack surface and points to specific defaults that turn local tools into public liabilities.","feed_headline":"320,102 public LLM services found open on the internet","feed_subtitle":"An internet-wide scan finds over 40% on plain HTTP and unauthenticated API leaks of model data.","key_machinery":"The central machinery is a four-step measurement pipeline. FOFA—an internet-wide asset search engine that indexes IPs, domains, ports, and service metadata—is queried using 15 hand-built feature catalogs (default ports, page titles, HTTP headers, favicon hashes) to enumerate candidate LLM services. The 158 official-documentation API endpoints are then probed unauthenticated over HTTP(S), and each response is mapped to a unified five-field schema (framework, category, endpoint, response type, security relevance) grouped into 12 functional categories. This pipeline converts the observation that many hosts exist into evidence that specific endpoints answer without credentials, which is the load-bearing support for the paper's security conclusions.","core_discovery":"The paper's central claim is that a large share of self-hosted LLM services on the public internet are reachable without authentication and often without encryption. The authors identify 320,102 public LLM services across 15 frameworks using hand-built FOFA signatures, extract 158 unique API endpoints grouped into 12 functional categories, and probe them with unauthenticated HTTP(S) requests. They report that over 40% of endpoints served plain HTTP, more than 210,000 endpoints lacked valid TLS metadata, and response rates varied widely by framework: Ollama and Llamafile answered over 10% of unauthenticated API calls, with Ollama leaking model or system information in roughly 15% of cases, while Open WebUI, NextChat, and Gradio answered below 0.1%. The paper interprets these findings as evidence that frameworks favor ease of deployment over secure defaults and that the resulting exposure enables model information disclosure, system configuration disclosure, unauthorized resource abuse, and potential system compromise.","pith_inferences":["A stricter active re-measurement that requires a successful LLM-specific API response from each host would likely shrink the raw census number, because the FastAPI and Jupyter Notebook signatures match generic tools; the ordering of frameworks by exposure is nonetheless likely to survive such a re-test.","The observed 'template responses' for Ollama suggest that part of the 35% response rate is produced by reverse proxies, load balancers, or defensive honeypots rather than by the LLM runtime itself, so the paper's 15.28% 'without template' rate is the safer upper bound for genuine leakage.","The 158 endpoints and 12 functional categories form a ready-made probe list; building a scanner around them and running it on new deployments would test whether the recommended secure-by-default changes actually reduce exposure over time."],"forward_implications":["If the census is right, the self-hosted LLM attack surface already numbers in the hundreds of thousands and is concentrated among a few cloud providers, so remediating high-density hosting domains and default images would shrink the exposed surface substantially.","If over 40% of endpoints use plain HTTP, passive network observers and on-path attackers can read or tamper with prompts and generated text for a large share of deployments.","If Ollama and Llamafile answer unauthenticated API calls at rates above 10%, then default installations of these frameworks effectively provide open model-listing, model-loading, and system-status endpoints to anyone on the internet.","If ComfyUI exposes endpoints in every risk category, then even a framework that looks like a niche UI tool becomes a systemic risk when its default bind address is 0.0.0.0 with no authentication, directly connecting model-hosting nodes to remote code execution and GPU abuse."],"supporting_citations":[{"why":"Supplies the FOFA index and query language used for the entire 320,102-service census.","marker":"[7]"},{"why":"Ollama is the framework whose default API exposure and response rates anchor the RQ2 and RQ3 findings.","marker":"[42]"},{"why":"ComfyUI provides the detailed per-endpoint risk table showing exposure in all five risk categories.","marker":"[10]"},{"why":"Jupyter Notebook is one of the 15 selected frameworks whose signature contributes 24,531 census hits.","marker":"[21]"},{"why":"FastAPI/Swagger UI is one of the 15 selected frameworks whose signature contributes 4,526 census hits.","marker":"[36]"},{"why":"OWASP Top 10 for LLM Applications 2025 supplies the risk taxonomy the paper maps its results onto.","marker":"[12]"},{"why":"The Ollama remote code execution CVE exemplifies the concrete exploitable consequence of unauthenticated exposed APIs.","marker":"[30]"},{"why":"The Open WebUI arbitrary file upload CVE exemplifies the vulnerability class behind unauthorized access in exposed LLM UIs.","marker":"[31]"}],"fun_headline_variants":["Public LLM services: 320k exposed, many without encryption","Ollama APIs leak model data to unauthenticated calls","Internet scan reveals 320k LLM services with insecure defaults","Over 40% of public LLM endpoints serve plain HTTP","LLM frameworks' insecure defaults expose services online"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The census counts every host matching a framework's FOFA signature as an LLM service, so the entire exposure statistics rest on the unquantified assumption that signatures such as 'title=FastAPI' or 'body=Jupyter Notebook' select LLM deployments rather than any other web application built with those tools.","fun_headline_variants_meta":{"raw":{"variants":["Public LLM services: 320k exposed, many without encryption","Ollama APIs leak model data to unauthenticated calls","Internet scan reveals 320k LLM services with insecure defaults","Over 40% of public LLM endpoints serve plain HTTP","LLM frameworks' insecure defaults expose services online"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2955,"prompt_tokens":993,"completion_tokens":1962,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1878}},"tokens_in":609,"tokens_out":1962,"duration_ms":18195,"temperature":1.0,"reasoning_tokens":1878,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:50:22.223680+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same FOFA queries on a fresh sample, fetch each matched host's landing page, and send one framework-specific API request per host; if many matched hosts do not actually serve LLM functionality, or if Ollama's 'template responses' turn out to be generic reverse-proxy pages rather than Ollama runtime answers, then the 320,102 census and the reported response rates overstate the true exposure.","supporting_citations":[{"cited_title":"Fofa: Cyberspace asset search engine,","cited_arxiv_id":null,"evidence_quote":"Supplies the FOFA index and query language used for the entire 320,102-service census."},{"cited_title":"Team, “Ollama,” https://ollama.com, 2024","cited_arxiv_id":null,"evidence_quote":"Ollama is the framework whose default API exposure and response rates anchor the RQ2 and RQ3 findings."},{"cited_title":"Comfyui,","cited_arxiv_id":null,"evidence_quote":"ComfyUI provides the detailed per-endpoint risk table showing exposure in all five risk categories."},{"cited_title":"Jupyter notebook,","cited_arxiv_id":null,"evidence_quote":"Jupyter Notebook is one of the 15 selected frameworks whose signature contributes 24,531 census hits."},{"cited_title":"Fastapi/swagger ui,","cited_arxiv_id":null,"evidence_quote":"FastAPI/Swagger UI is one of the 15 selected frameworks whose signature contributes 4,526 census hits."},{"cited_title":"Owasp top 10 for llm applications 2025,","cited_arxiv_id":null,"evidence_quote":"OWASP Top 10 for LLM Applications 2025 supplies the risk taxonomy the paper maps its results onto."},{"cited_title":"Cve-2024-37032: Ollama remote code execution vulnerability,","cited_arxiv_id":null,"evidence_quote":"The Ollama remote code execution CVE exemplifies the concrete exploitable consequence of unauthenticated exposed APIs."},{"cited_title":"Cve-2024-6707: Openwebui arbitrary file upload vulnerability,","cited_arxiv_id":null,"evidence_quote":"The Open WebUI arbitrary file upload CVE exemplifies the vulnerability class behind unauthorized access in exposed LLM UIs."}],"review_version":1}