pith. sign in

arxiv: 2606.10173 · v1 · pith:625KEJMOnew · submitted 2026-06-08 · 💻 cs.CR · cs.AI

Local Is Not a Sufficient Privacy Boundary: Governing OS-Integrated On-Device AI

Jonghyun Chung , Sanket Badhe This is my paper

Pith reviewed 2026-06-27 15:56 UTC · model grok-4.3

classification 💻 cs.CR cs.AI
keywords on-device AIprivacyoperating systeminformation flowaudit rubricthreat modelgovernanceaccountability
0
0 comments X

The pith

Local inference does not suffice as a privacy boundary for AI integrated into operating systems.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that on-device AI privacy hinges on more than where computation happens. Even local models can gather extensive personal context from OS resources, retain derived data, authorize actions, and escalate to cloud services. The authors introduce a framework centered on the operating system that includes a threat model, risk taxonomy, architectural controls, and an audit rubric to enforce accountability. Applying this to major platforms reveals gaps in current implementations. This approach treats privacy as ongoing governance rather than a one-time deployment choice.

Core claim

Meaningful privacy in on-device AI depends on constrained information flow, bounded authority, visible user control, and auditable governance across the operating-system lifecycle. The framework specifies a threat model, a six-part privacy risk taxonomy, privacy-by-architecture controls, and a four-level audit rubric demonstrated on Apple Intelligence, Android AICore, and Microsoft Recall.

What carries the argument

The OS-centered privacy framework that includes a threat model, six-part risk taxonomy, privacy-by-architecture controls, and four-level audit rubric to evaluate institutional accountability.

If this is right

  • Local computation answers only where processing occurs, not who assembles context or what state persists.
  • System updates can alter authority without corresponding user visibility or consent.
  • The four-level audit rubric enables systematic comparison of different on-device AI implementations.
  • Privacy-by-architecture controls must address information flow and tool invocation at the OS level.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Regulators could require OS vendors to publish information flow diagrams for AI components.
  • Users might demand granular controls over what data the AI can access across apps.
  • Future work could extend the rubric to evaluate real-time behavior rather than documentation alone.

Load-bearing premise

Privacy in on-device AI is primarily an institutional accountability problem rather than determined by the location of computation.

What would settle it

A demonstration of a purely local on-device AI system that prevents unauthorized context assembly, persistent derived state, and unauthorized tool use without the framework's controls would challenge the claim.

Figures

Figures reproduced from arXiv: 2606.10173 by Jonghyun Chung, Sanket Badhe.

Figure 1
Figure 1. Figure 1: Conceptual privacy surface for OS-integrated on-device AI. Privacy risk arises from the full mediation layer, not only [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Operational workflow for applying the rubric to a [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
read the original abstract

As AI systems move into operating systems, privacy no longer turns only on whether a model runs locally. A local assistant may assemble email, calendar entries, files, screenshots, notifications, and app intents; retain embeddings or summaries; invoke tools; emit telemetry; or route difficult requests to cloud infrastructure. Local inference reduces some exposure, but it answers only one question: where computation occurs. It does not answer who may assemble context, what derived state persists, which actions are authorized, or how updates change the system's authority. We develop an OS-centered privacy framework for on-device AI that treats privacy as an institutional accountability problem rather than a deployment attribute. The framework specifies a threat model, a six-part privacy risk taxonomy, privacy-by-architecture controls, and a four-level audit rubric. We demonstrate the rubric through a documentation-bounded comparison of Apple Intelligence/Foundation Models, Android AICore/Gemini Nano, and Microsoft Recall. Meaningful privacy in on-device AI depends on constrained information flow, bounded authority, visible user control, and auditable governance across the operating-system lifecycle.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that privacy for OS-integrated on-device AI cannot be ensured by local inference alone, since such systems assemble personal context (email, files, screenshots), retain embeddings/summaries, invoke tools, emit telemetry, and may route to cloud. It develops an OS-centered framework treating privacy as institutional accountability rather than a deployment attribute, specifying a threat model, six-part privacy risk taxonomy, privacy-by-architecture controls, and four-level audit rubric. The rubric is demonstrated via documentation-bounded comparison of Apple Intelligence/Foundation Models, Android AICore/Gemini Nano, and Microsoft Recall, concluding that meaningful privacy requires constrained information flow, bounded authority, visible user control, and auditable governance across the OS lifecycle.

Significance. If the framework and its controls hold under scrutiny, the work could shift on-device AI privacy research and policy from locality-centric arguments toward lifecycle governance and accountability mechanisms. The structured taxonomy and rubric provide a concrete lens for auditing vendor claims, which is timely amid widespread deployment of OS-level AI features. Credit is due for the explicit framing of privacy as an institutional problem and the attempt to operationalize it via an audit rubric, though the documentation-only demonstration limits immediate applicability.

major comments (2)
  1. [Demonstration (documentation-bounded comparison)] Demonstration section: The four-level audit rubric is applied solely to vendor documentation for Apple Intelligence, Android AICore/Gemini Nano, and Microsoft Recall. Without runtime tracing, code inspection, or measurement of actual information flows (e.g., embedding retention or telemetry emission), the rubric's ability to detect real discrepancies between stated and implemented behaviors remains unshown. This is load-bearing for the central claim that the framework identifies shortfalls beyond what locality addresses.
  2. [Framework definition (threat model, taxonomy, controls)] Framework sections (threat model, taxonomy, controls): The six-part privacy risk taxonomy and four-level audit rubric are introduced as novel constructs without derivation from existing literature, empirical data, or falsifiable tests against system behavior. The claim that privacy depends on constrained flow, bounded authority, visible control, and auditable governance therefore rests on the framework's internal logic rather than demonstrated identification of actual privacy shortfalls in the three platforms.
minor comments (1)
  1. [Abstract] Abstract and introduction: The high-level description of the six-part taxonomy and four-level rubric does not enumerate their components or scoring criteria, which would improve clarity for readers evaluating the demonstration.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed review. The comments correctly identify the scope of our demonstration and the need for clearer grounding of the framework. We address each major comment below and outline targeted revisions.

read point-by-point responses

  1. Referee: Demonstration (documentation-bounded comparison): The four-level audit rubric is applied solely to vendor documentation for Apple Intelligence, Android AICore/Gemini Nano, and Microsoft Recall. Without runtime tracing, code inspection, or measurement of actual information flows (e.g., embedding retention or telemetry emission), the rubric's ability to detect real discrepancies between stated and implemented behaviors remains unshown. This is load-bearing for the central claim that the framework identifies shortfalls beyond what locality addresses.

    Authors: We agree that the demonstration relies exclusively on documentation analysis and does not include runtime verification or code inspection. This approach was selected because the framework is intended to offer an externally usable audit method based on publicly available information, given the closed-source nature of the evaluated systems. We recognize that this limits direct evidence of discrepancy detection. We will revise the manuscript to add an explicit limitations subsection in the demonstration, clarifying the documentation-bounded scope and outlining how the rubric could be extended with empirical methods in follow-on work. revision: partial

  2. Referee: Framework definition (threat model, taxonomy, controls): The six-part privacy risk taxonomy and four-level audit rubric are introduced as novel constructs without derivation from existing literature, empirical data, or falsifiable tests against system behavior. The claim that privacy depends on constrained flow, bounded authority, visible control, and auditable governance therefore rests on the framework's internal logic rather than demonstrated identification of actual privacy shortfalls in the three platforms.

    Authors: The taxonomy and rubric were constructed by applying the threat model to the concrete behaviors described in the three platforms' documentation, synthesizing elements from established privacy concepts such as information flow control and accountability. We will revise the framework sections to include an explicit derivation subsection that traces each taxonomy category and rubric level back to the threat model, with additional citations to relevant privacy-by-design and contextual integrity literature. This will make the logical grounding transparent. revision: yes

Circularity Check

0 steps flagged

Framework proposal is self-contained with no derivation chain

full rationale

The paper introduces an original OS-centered privacy framework consisting of a threat model, six-part taxonomy, privacy-by-architecture controls, and four-level audit rubric. It then applies the rubric in a documentation-bounded comparison of three commercial systems. No equations, fitted parameters, self-citations, or uniqueness theorems appear. The central claim that meaningful privacy requires constrained flow, bounded authority, visible control, and auditable governance is presented as a definitional premise of the new framework rather than derived from prior inputs or self-referential steps. The demonstration does not reduce to the framework by construction; it is an external application. This is a standard non-circular conceptual contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

Based solely on the abstract; no full text available to identify additional free parameters, axioms, or entities. The framework itself introduces new conceptual components without independent evidence cited in the abstract.

axioms (1)
  • domain assumption Privacy in on-device AI is best addressed as an institutional accountability problem rather than solely a deployment attribute.
    Explicitly stated as the approach in the abstract.
invented entities (2)
  • Six-part privacy risk taxonomy no independent evidence
    purpose: Categorize privacy risks specific to OS-integrated on-device AI
    Introduced as part of the new framework in the abstract.
  • Four-level audit rubric no independent evidence
    purpose: Evaluate privacy governance in on-device AI systems
    Introduced as part of the new framework and demonstrated in the abstract.

pith-pipeline@v0.9.1-grok · 5716 in / 1371 out tokens · 19312 ms · 2026-06-27T15:56:59.501297+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 7 canonical work pages · 1 internal anchor

  1. [1]

    M.; and Such, J

    Abdi, N.; Ramokapane, K. M.; and Such, J. M. 2019. More than Smart Speakers: Security and Privacy Perceptions of Smart Home Personal Assistants. In Proceedings of the Symposium on Usable Privacy and Security, 451--466

    2019

  2. [2]

    Android Developers Blog . 2024. An Introduction to Privacy and Safety for Gemini Nano. https://android-developers.googleblog.com/2024/10/introduction-to-privacy-and-safety-gemini-nano.html. Developer blog, accessed May 5, 2026

    2024

  3. [3]

    Apple Developer . 2026 a . Acceptable Use Requirements for the Foundation Models Framework. https://developer.apple.com/apple-intelligence/acceptable-use-requirements-for-the-foundation-models-framework/. Developer documentation, accessed May 5, 2026

    2026

  4. [4]

    Apple Developer . 2026 b . Foundation Models. https://developer.apple.com/documentation/FoundationModels. Developer documentation, accessed May 5, 2026

    2026

  5. [5]

    Apple Security Research . 2024. Private Cloud Compute: A New Frontier for AI Privacy in the Cloud. https://security.apple.com/blog/private-cloud-compute/. Security documentation, accessed May 5, 2026

    2024

  6. [6]

    Apple Support . 2026 a . Apple Intelligence and Privacy on iPhone. https://support.apple.com/guide/iphone/apple-intelligence-and-privacy-iphe3f499e0e/ios. Support documentation, accessed May 5, 2026

    2026

  7. [7]

    Apple Support . 2026 b . Review Device Management Restrictions for Apple Devices. https://support.apple.com/guide/deployment/review-device-management-restrictions-dep739685973/web. Accessed May 7, 2026

    2026

  8. [8]

    Cavoukian, A. 2011. Privacy by Design: The 7 Foundational Principles. Information and Privacy Commissioner of Ontario

    2011

  9. [9]

    Deng, M.; Wuyts, K.; Scandariato, R.; Preneel, B.; and Joosen, W. 2011. A Privacy Threat Analysis Framework: Supporting the Elicitation and Fulfillment of Privacy Requirements. Requirements Engineering, 16(1): 3--32

    2011

  10. [10]

    Diakopoulos, N. 2015. Algorithmic Accountability: Journalistic Investigation of Computational Power Structures. Digital Journalism, 3(3): 398--415

    2015

  11. [11]

    Doshi, A.; Hong, Y.; Xu, C.; Kang, E.; Kapravelos, A.; and Kastner, C. 2026. Towards Verifiably Safe Tool Use for LLM Agents. In Proceedings of ICSE-NIER. ArXiv:2601.08012

    work page arXiv 2026

  12. [12]

    P.; Jung, J.; McDaniel, P.; and Sheth, A

    Enck, W.; Gilbert, P.; Chun, B.-G.; Cox, L. P.; Jung, J.; McDaniel, P.; and Sheth, A. N. 2010. TaintDroid: An Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones. In Proceedings of OSDI

    2010

  13. [13]

    European Union . 2016. Regulation (EU) 2016/679, Article 35: Data Protection Impact Assessment . https://eur-lex.europa.eu/eli/reg/2016/679/oj. Accessed May 7, 2026

    2016

  14. [14]

    European Union . 2024. Regulation Laying Down Harmonised Rules on Artificial Intelligence. EU AI Act

    2024

  15. [15]

    P.; Ha, E.; Egelman, S.; Haney, A.; Chin, E.; and Wagner, D

    Felt, A. P.; Ha, E.; Egelman, S.; Haney, A.; Chin, E.; and Wagner, D. 2012. Android Permissions: User Attention, Comprehension, and Behavior. In Proceedings of the Symposium on Usable Privacy and Security

    2012

  16. [16]

    Google Android Developers . 2026. Gemini Nano. https://developer.android.com/ai/gemini-nano. Developer documentation, accessed May 5, 2026

    2026

  17. [17]

    Google Android Help . 2026. About Android AICore. https://support.google.com/android/answer/17065362. Accessed May 7, 2026

    work page arXiv 2026

  18. [18]

    Greshake, K.; Abdelnabi, S.; Mishra, S.; Endres, C.; Holz, T.; and Fritz, M. 2023. Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Proceedings of the ACM Workshop on Artificial Intelligence and Security

    2023

  19. [19]

    Information Commissioner's Office . 2023 a . Data Protection by Design and Default. Regulatory guidance

    2023

  20. [20]

    Information Commissioner's Office . 2023 b . Employment Practices and Data Protection: Monitoring Workers. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/employment/monitoring-workers/. Accessed May 7, 2026

    2023

  21. [21]

    Information Commissioner's Office . 2026. Data Protection Impact Assessments. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/accountability-and-governance/data-protection-impact-assessments-dpias/. Accessed May 7, 2026

    2026

  22. [22]

    Li, X.; Huang, D.; Li, J.; Cai, H.; Zhou, Z.; Dong, W.; Wang, X.; and Liu, Y. 2025. A Vision for Access Control in LLM-based Agent Systems. ArXiv:2510.11108

    work page arXiv 2025

  23. [23]

    Malkin, N.; Deatrick, J.; Tong, A.; Wijesekera, P.; Egelman, S.; and Wagner, D. 2019. Privacy Attitudes of Smart Speaker Users. Proceedings on Privacy Enhancing Technologies, 2019(4): 250--271

    2019

  24. [24]

    Marchiori, E.; de Haas, S.; Volnov, S.; Falcon, R.; Pinto, R.; and Zamarato, M. 2022. Android Private Compute Core Architecture. arXiv preprint arXiv:2209.10317

    work page arXiv 2022

  25. [25]

    Microsoft Learn . 2026 a . Manage Recall for Windows Clients. https://learn.microsoft.com/en-us/windows/client-management/manage-recall. Product documentation, accessed May 5, 2026

    2026

  26. [26]

    Microsoft Learn . 2026 b . Microsoft.Windows.AI Namespace: Phi Silica API Reference. https://learn.microsoft.com/en-us/windows/ai/apis/phi-silica-api-ref. Developer documentation, accessed May 5, 2026

    2026

  27. [27]

    Microsoft Learn . 2026 c . Platform Card: Microsoft Foundry on Windows -- Phi Silica (Language Model). https://learn.microsoft.com/en-us/windows/ai/cards/phi-silica-platform-card. Platform documentation, accessed May 5, 2026

    2026

  28. [28]

    Microsoft Support . 2026. Privacy and Control over Your Recall Experience. https://support.microsoft.com/windows/privacy-and-control-over-your-recall-experience-d404f672-7647-41e5-886c-a3c59680af15. Support documentation, accessed May 5, 2026

    2026

  29. [29]

    Microsoft Windows Experience Blog . 2024. Update on Recall Security and Privacy Architecture. https://blogs.windows.com/windowsexperience/2024/09/27/update-on-recall-security-and-privacy-architecture/. Accessed May 7, 2026

    2024

  30. [30]

    X.; Kuleshov, V.; Shmatikov, V.; and Rush, A

    Morris, J. X.; Kuleshov, V.; Shmatikov, V.; and Rush, A. M. 2023. Text Embeddings Reveal (Almost) As Much As Text. ArXiv:2310.06816

    work page arXiv 2023

  31. [31]

    National Institute of Standards and Technology . 2020. NIST Privacy Framework: A Tool for Improving Privacy through Enterprise Risk Management, Version 1.0 . https://www.nist.gov/privacy-framework. NIST CSWP 01162020, accessed May 7, 2026

    2020

  32. [32]

    National Institute of Standards and Technology . 2024. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1

    2024

  33. [33]

    Nissenbaum, H. 2004. Privacy as Contextual Integrity. Washington Law Review, 79(1): 119--158

    2004

  34. [34]

    A.; and Oeldorf-Hirsch, A

    Obar, J. A.; and Oeldorf-Hirsch, A. 2020. The Biggest Lie on the Internet: Ignoring the Privacy Policies and Terms of Service Policies of Social Networking Services. Information, Communication & Society, 23(1): 128--147

    2020

  35. [35]

    OECD . 2019. OECD Principles on Artificial Intelligence. Policy framework

    2019

  36. [36]

    N.; and Sandvig, C

    Plantin, J.-C.; Lagoze, C.; Edwards, P. N.; and Sandvig, C. 2018. Infrastructure Studies Meet Platform Studies in the Age of Google and Facebook. New Media & Society, 20(1): 293--310

    2018

  37. [37]

    D.; Smart, A.; White, R

    Raji, I. D.; Smart, A.; White, R. N.; Mitchell, M.; Gebru, T.; Hutchinson, B.; Smith-Loud, J.; Theron, D.; and Barnes, P. 2020. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. In Proceedings of FAT*, 33--44

    2020

  38. [38]

    J.; and Cowan, C

    Roesner, F.; Kohno, T.; Moshchuk, A.; Parno, B.; Wang, H. J.; and Cowan, C. 2012. User-Driven Access Control: Rethinking Permission Granting in Modern Operating Systems. In Proceedings of the IEEE Symposium on Security and Privacy, 224--238

    2012

  39. [39]

    L.; and Cranor, L

    Schaub, F.; Balebako, R.; Durity, A. L.; and Cranor, L. F. 2015. A Design Space for Effective Privacy Notices. In Proceedings of the Symposium on Usable Privacy and Security, 1--17

    2015

  40. [40]

    Seaver, N. 2017. Knowing Algorithms. Media in Action, 2(1): 412--422

    2017

  41. [41]

    D.; Boyd, D.; Friedler, S

    Selbst, A. D.; Boyd, D.; Friedler, S. A.; Venkatasubramanian, S.; and Vertesi, J. 2019. Fairness and Abstraction in Sociotechnical Systems. In Proceedings of FAT*, 59--68

    2019

  42. [42]

    Prompt Injection Attack to Tool Selection in LLM Agents

    Shi, J.; Yuan, Z.; Tie, G.; Zhou, P.; Gong, N. Z.; and Sun, L. 2026. Prompt Injection Attack to Tool Selection in LLM Agents. In Proceedings of NDSS. ArXiv:2504.19793

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  43. [43]

    Solove, D. J. 2006. A Taxonomy of Privacy. University of Pennsylvania Law Review, 154(3): 477--560

    2006

  44. [44]

    Wang, B.; He, W.; Zeng, S.; Xiang, Z.; Xing, Y.; Tang, J.; and He, P. 2025. Unveiling Privacy Risks in LLM Agent Memory. ArXiv:2502.13172

    work page arXiv 2025