{"id":"2c28a083-fac3-45a8-8a7b-806e08f1124a","arxiv_id":"2509.02411","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":0.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of privacy and security challenges for mobile large language models, summarizing known attack types and defenses without introducing new results.","lead":"This paper reviews the privacy and security risks of running large language models on phones and other mobile devices, and catalogs defenses like differential privacy, federated learning, and prompt encryption. It is a literature survey with no new experiments, useful mainly as an introductory map.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Side-channel defense section misapplies DP and FL and cites an attack paper as support, undermining a named contribution.","rationale":"The reader's weakest assumption (undocumented literature selection) is real but not the most acute problem. The central claim has two parts: comprehensive coverage and in-depth comparison of vulnerabilities. For the side-channel vulnerability category, the only defense section (§V.D) contains a technical error: DP does not mitigate physical side channels, and the cited work [80] is an attack paper. This is verifiable and directly contradicts the abstract's promise to compare effectiveness and limitations of defenses for side-channel attacks. Additionally, membership inference, model inversion, and side-channel attacks are never described as attacks before their defense subsections; only one-sentence definitions appear in the defense sections. So the paper's claim to 'analyze vulnerabilities unique to mobile LLMs' is systematically unfulfilled for three of the four named attack types. The strongest defensible verdict remains CONDITIONAL: the survey's structure and most summaries are usable, but the side-channel section and the missing attack taxonomies require revision. Therefore I keep the reader's conditional verdict (UNCHANGED), while identifying a different, more concrete locus of concern than the reader did.","tokens_in":14144,"tokens_out":9326,"duration_ms":102466,"concrete_test":"Analytically test the claim in §V.D using the DP definition: if M is an ε-DP mechanism for the LLM's output, show whether any bound on the mutual information I(X; S) follows for a side-channel measurement S (e.g., power trace) observed during inference. Since DP only constrains the output distribution and S depends on internal computation, no such bound follows without additional assumptions; therefore the claim is unsupported. In the same pass, read reference [80]: if it is an attack paper with no DP-defense content, the section's sole cited support for the DP-is-side-channel-defense claim is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V.D asserts that differential privacy is 'one of the most effective mitigations against the side-channel attacks' and cites [80]. However, [80] is 'LLM-FIN: Large Language Models Fingerprinting Attack on Edge Devices' (Nazari et al., ISQED 2024), an attack paper, not a DP defense. Conceptually, ε-DP bounds the change in output distribution given one input change; it does not constrain the physical side-channel signal (power, EM, timing, memory access patterns) observed during on-device inference. Adding noise to outputs or gradients does not obscure an adversary's direct physical measurements. Federated learning, also listed, only affects training-time data sharing and cannot prevent leakage during local inference. Since side-channel attacks are one of the four vulnerability classes explicitly named in the abstract, the survey fails to deliver the promised 'in-depth comparison of their effectiveness and limitations' for this category. This is a concrete correctness risk, not merely a missing search protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a survey of privacy and security issues for large language models deployed on mobile and edge platforms. It reviews background material on LLM architecture and mobile deployment, classifies privacy-preserving techniques (data anonymization, prompt encryption, differential privacy, federated learning), describes defenses against four attack categories (adversarial, membership inference, model inversion, side-channel), and discusses applications in healthcare, finance, and education. The abstract claims both a comprehensive overview and an in-depth comparison of the effectiveness and limitations of these solutions.","tokens_in":14386,"tokens_out":3434,"duration_ms":41342,"significance":"If the survey were comprehensive and accurate, it would be a useful reference for practitioners and researchers working on mobile LLM security, a timely topic for an IoT-oriented venue. The paper does bring together a number of relevant references on federated learning and privacy-preserving fine-tuning, and it sketches application-specific trade-offs. However, the value of the survey depends on the reliability of its categorizations and the support for its stated contributions. As detailed below, several load-bearing statements are inaccurate or unsupported, and the claimed comprehensiveness and comparative depth are not delivered.","major_comments":[{"comment":"The section states that differential privacy is 'one of the most effective mitigations against the side-channel attacks' and cites [80]. Reference [80] is 'LLM-FIN: Large Language Models Fingerprinting Attack on Edge Devices', an attack paper, not a DP defense. Conceptually, ε-DP bounds the change in an output distribution given a change in one input; it does not constrain physical side-channel signals such as power, EM, timing, or memory access patterns observed during on-device inference. Federated learning, also listed, affects training-time data sharing and cannot prevent leakage during local inference. Since side-channel attacks are one of the four vulnerability classes named in the abstract, this is a load-bearing correctness error. Please replace these claims with supported defenses (e.g., masking, constant-time execution, ORAM, TEE-based isolation) or explicitly note the lack of","section":"V.D (Defense on Side-Channel Attack)"},{"comment":"The abstract claims a 'comprehensive overview' and 'systematically categorizing existing solutions.' No search protocol, inclusion/exclusion criteria, database list, time span, or quality assessment is provided. The selection of roughly 90 references appears ad hoc, and several relevant areas for mobile LLM privacy/security are absent, including trusted execution environments for on-device inference, secure inference protocols, and model extraction defenses. Please either add a methodology subsection describing the literature search and selection process, or substantially qualify the comprehensiveness claim.","section":"Abstract and overall methodology"},{"comment":"The abstract promises an 'in-depth comparison of their effectiveness and limitations' for the analyzed attacks and defenses. The body does not deliver this. Each defense receives a short qualitative paragraph, with no comparative metrics, no tables contrasting privacy guarantees, utility costs, computational overhead, or applicable attack scenarios, and no explicit mapping between threats and defenses. For example, differential privacy is presented as a defense against adversarial, membership inference, model inversion, and side-channel attacks, but the conditions under which it is effective (e.g., training-time DP-SGD vs. inference-time noise) are not compared or delimited. Please add a comparative analysis (e.g., a threat-defense matrix) or revise the abstract to reflect the actual scope.","section":"Abstract and Sections IV-V"},{"comment":"Section VI opens by promising 'practical examples of successful LLM implementations that uphold data security and privacy.' The subsections that follow summarize general trade-offs and cite works that discuss privacy techniques; they do not document concrete deployed systems or evaluate their success. This overstatement is part of the paper's claimed contribution. Please either provide actual case studies with system details, or reframe the section as 'application areas and trade-offs' without the 'successful implementations' claim.","section":"VI (Trustworthy Mobile LLMs Practices)"}],"minor_comments":[{"comment":"Figure 6 is captioned 'Membership Inversion Attack' while the text refers to 'Model Inversion Attack'. The caption should be corrected.","section":"V.C / Fig. 6"},{"comment":"Reference [38] is about an IoTDB data analysis library and appears unrelated to the claim that mobile devices transmit input data to edge servers to reduce resource strain. Please check and replace with a relevant citation.","section":"II.B.2"},{"comment":"The sentence 'Differential Privacy applies controlled noise to datasets' is imprecise. Differential privacy is a formal definition; mechanisms such as DP-SGD add noise to gradients or outputs. Consider rewording to avoid conflating the definition with an algorithm.","section":"IV.C"},{"comment":"Minor grammatical issue: 'can help mitigate to many security risks' should be 'can help mitigate many security risks.'","section":"V.D"},{"comment":"Section II.B.2 states the survey 'focuses primarily on examining the privacy and security challenges associated with the collaborative edge computing of LLMs,' but the abstract and later sections also emphasize on-device deployment. This scope statement should be clarified or aligned with the actual coverage.","section":"II.B.2 / Abstract"},{"comment":"The text says 'methods like differential privacy protect data effectively, they can impair model performance' — the pronoun and sentence structure should be corrected for readability.","section":"VI.A"}],"recommendation":"major_revision","confidential_remarks":"The topic fits the journal's scope, but the manuscript's central claims of comprehensiveness and in-depth comparison are currently unsupported, and the side-channel defense section contains a concrete factual error that expert reviewers would likely catch. The paper could become acceptable after a thorough revision that adds methodology, corrects the unsupported claims, and either adds real comparative analysis or tempers the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a readable survey of privacy and security for mobile LLMs, but the abstract overclaims: there is no in-depth comparison of defenses, and one named defense is just wrong. The side-channel section (V.D) cites an attack paper [80] to support differential privacy as a mitigation, and treats DP and federated learning as side-channel defenses. Conceptually, DP bounds output-distribution change, not physical emissions, and FL only affects training-time data sharing; neither stops an adversary reading power or EM traces during local inference. That is a load-bearing error for a survey that advertises side-channel attacks as one of its four vulnerability classes.\n\nTo its credit, the survey organizes a broad literature—adversarial, membership inference, model inversion, plus privacy techniques like DP, FL, prompt encryption—and the high-level summaries of attacks and defenses are mostly accurate. The federated learning subsection with frameworks like OpenFedLLM and FedIT is a useful entry point. The application sections (health, finance, education) are straightforward and point to real trade-offs.\n\nBeyond the side-channel error, the comprehensiveness claim is fragile: no search protocol or inclusion criteria, roughly 90 citations, and no coverage of trusted execution environments for on-device inference. The in-depth comparison promised in the abstract never materializes; each defense gets a paragraph. Self-citations are minor (four references from the same group), not a problem per se.\n\nWho is this for? Practitioners new to mobile LLM security who want a quick map; researchers will find little new. The paper deserves a serious referee because the topic is timely and the errors are fixable, but it needs a rewrite of the side-channel section and a softening of the claims before publication.","headline":"A well-organized but shallow survey that botches its side-channel defense section and overclaims comprehensiveness.","tokens_in":14783,"tokens_out":1634,"would_cite":false,"duration_ms":17842,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey claims that the privacy and security problems of mobile large language models can be organized into four data-privacy defenses and four attack classes, and that mobile resource limits decide which defenses are practical.","keywords":["mobile large language models","LLM privacy","LLM security","differential privacy","federated learning","prompt encryption","membership inference attacks","adversarial attacks"],"falsifier":"Find a well-known area of on-device model protection—such as trusted execution environments or secure hardware enclaves for mobile inference—and check whether the survey covers it; a substantial absent literature would contradict the comprehensiveness claim. A reader could also rerun a structured literature search with explicit queries and compare the resulting set with the survey's references.","tokens_in":14108,"feed_emoji":"🔐","tokens_out":5838,"duration_ms":66025,"temperature":0.7,"pith_summary":"This paper is a survey that tries to establish a usable map of privacy and security issues for large language models running on phones and other mobile devices. It claims that existing defenses fall into four families—data anonymization, prompt encryption, differential privacy, and federated learning—and that the main attacks are adversarial, membership inference, model inversion, and side-channel. It compares each defense's effectiveness and limitations, and argues that mobile constraints such as limited compute, battery, and memory make server-style protections expensive or impractical. The value of the survey, if correct, is a single structured reference for developers, policymakers, and researchers who need to know what threats exist and which mitigations can realistically run on-device.","feed_headline":"Survey sorts mobile LLM threats and defenses into one map","feed_subtitle":"Four attack classes, four privacy defenses, and the resource limits that decide which one works on a phone.","key_machinery":"The load-bearing device is the paper's two-part taxonomy, introduced in Figure 1 and developed in Sections III through V. It separates data privacy preservation—anonymization, prompt encryption, differential privacy, and federated learning—from security countermeasures against adversarial, membership inference, model inversion, and side-channel attacks. The taxonomy works by framing each defense primarily through the mobile constraint: sensitive data is processed at the edge, where devices lack the compute, battery, and memory for heavyweight protections, so every method is evaluated on both its security effect and its feasibility on resource-limited hardware.","core_discovery":"In the paper's own terms, the central claim is that privacy and security issues in mobile LLMs can be systematically categorized into a small set of solution families and attack classes. The solution families are data anonymization, prompt encryption, differential privacy, and federated learning; the attack classes are adversarial attacks, membership inference attacks, model inversion attacks, and side-channel attacks. The paper asserts that each defense has known limitations under mobile resource constraints—differential privacy can degrade model accuracy, encryption and local processing strain battery and memory, and federated learning introduces communication and heterogeneity challenges—","pith_inferences":["Beyond the paper: the map is likely illustrative rather than exhaustive. A documented search protocol with inclusion and exclusion criteria would be needed to confirm the claimed comprehensiveness; the paper does not provide one.","Beyond the paper: the taxonomy suggests a natural fifth defense family for hardware-assisted isolation, such as trusted execution environments or secure enclaves for on-device inference, which the survey does not cover.","Beyond the paper: the qualitative trade-off discussion could be turned into a concrete benchmark measuring latency, energy, memory, and accuracy loss of each defense on representative mobile hardware; the paper reports no such measurements."],"forward_implications":["A developer choosing a privacy technique can use the survey's categories to see the main cost of each option: anonymization trades readability, encryption adds computation, differential privacy trades accuracy for noise, and federated learning spreads computation and data across devices.","Membership inference and model inversion are treated as serious risks for on-device LLMs because training data can include messages, voice commands, and location; the paper links defenses such as differential privacy, distillation, and adaptive regularization to countering overfitting.","The comparison implies that no single defense is sufficient, and that combinations—such as federated learning with differentially private parameter-efficient fine-tuning—are the current direction for balancing privacy and efficiency.","The open problems identified in the paper—mobile-friendly adversarial robustness, lightweight standardized privacy frameworks, and secure low-latency transmission—define a near-term research agenda for trustworthy mobile LLM deployment."],"supporting_citations":[{"why":"Frames on-device language models and defines the mobile deployment context the survey builds on.","marker":"[2]"},{"why":"Supplies the broader LLM privacy survey foundation and informs the privacy/security categories being extended.","marker":"[8]"},{"why":"Provides the adversarial attack definition and examples used in the security-vulnerability discussion.","marker":"[9]"},{"why":"Supplies the membership inference attack formulation that the defense section addresses.","marker":"[12]"},{"why":"Anchors the side-channel vulnerability discussion by linking LLM hardware and chip security to edge devices.","marker":"[13]"},{"why":"Provides differential-privacy fine-tuning as a concrete data-privacy preservation method.","marker":"[16]"},{"why":"Defines prompt encryption as a privacy-preserving communication scheme for LLMs.","marker":"[17]"},{"why":"Supplies a federated learning framework for training LLMs on decentralized private data.","marker":"[47]"},{"why":"Supplies the model inversion attack that Section V-C defends against.","marker":"[75]"}],"fun_headline_variants":["Survey maps mobile LLM attacks and defenses","Four defenses, four threats in mobile LLM security","Mobile LLM privacy: trade-offs under resource limits","Charting mobile LLM privacy threats and fixes","Mobile LLM security: a map of attacks and defenses"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The survey's claim of comprehensiveness rests on the assumption that the papers it chose to cite represent the whole field of mobile LLM privacy and security, since it does not document a search protocol or inclusion criteria.","fun_headline_variants_meta":{"raw":{"variants":["Survey maps mobile LLM attacks and defenses","Four defenses, four threats in mobile LLM security","Mobile LLM privacy: trade-offs under resource limits","Charting mobile LLM privacy threats and fixes","Mobile LLM security: a map of attacks and defenses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001156,"raw_usage":{"total_tokens":4591,"prompt_tokens":671,"completion_tokens":3920,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":415,"completion_tokens_details":{"reasoning_tokens":3846}},"tokens_in":415,"tokens_out":3920,"duration_ms":32768,"temperature":1.0,"reasoning_tokens":3846,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:34:19.903797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a well-known area of on-device model protection—such as trusted execution environments or secure hardware enclaves for mobile inference—and check whether the survey covers it; a substantial absent literature would contradict the comprehensiveness claim. A reader could also rerun a structured literature search with explicit queries and compare the resulting set with the survey's references.","supporting_citations":[{"cited_title":"LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trust","cited_arxiv_id":"2405.07061","evidence_quote":"Anchors the side-channel vulnerability discussion by linking LLM hardware and chip security to edge devices."},{"cited_title":"EmojiPrompt: Generative Prompt Obfuscation for Privacy-Preserving Communication with Cloud-based LLMs","cited_arxiv_id":"2402.05868","evidence_quote":"Defines prompt encryption as a privacy-preserving communication scheme for LLMs."},{"cited_title":"Openfedllm: Training large language models on decentralized private data via federated learning,","cited_arxiv_id":null,"evidence_quote":"Supplies a federated learning framework for training LLMs on decentralized private data."},{"cited_title":"Text embedding inversion security for multilingual language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the model inversion attack that Section V-C defends against."}],"review_version":1}