Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Towards Computer-Using Personal Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes Computer-Using Personal Agents: browser-operating agents whose access to a user's personal data is routed through a controlled personal knowledge graph rather than granted directly.

desk verdict A coherent Dagstuhl position paper that names a useful architectural direction, but its central promise of 'better control' hinges on an enforcement boundary it never specifies. read the letter →

arxiv 2503.15515 v1 pith:C54FF73H submitted 2025-01-31 cs.HC cs.AIcs.MA

Piero A. Bonatti , John Domingue , Anna Lisa Gentile , Andreas Harth , Olaf Hartig , Aidan Hogan , Katja Hose , Ernesto Jimenez-Ruiz
show 4 more authors
Deborah L. McGuinness Chang Sun Ruben Verborgh Jesse Wright
This is my paper · ORCID
classification cs.HCcs.AIcs.MA
keywords Computer-UsingAgentsPersonalKnowledgeGraphdatacontrolprivacypoliciesODRLwebautomationmulti-agentcoordinationagentroadmap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Computer-using agents (CUAs) can operate browsers on a user's behalf, but they face a dilemma: give them personal data directly and risk misuse, or keep them away from it and limit what they can do. The paper proposes Computer-Using Personal Agents (CUPAs), defined as CUAs with controlled access to a structured repository of the user's private information. It instantiates that repository as a Personal Knowledge Graph (PKG) so that access and usage are governed by machine-readable policies rather than granted wholesale. If the proposal holds, users get finer-grained control over what agents may see and do, agents can automate more tasks that involve personal data, and multiple agents can cooperate on tasks that draw on several users' data.

What carries the argument

The load-bearing object is the Personal Knowledge Graph (PKG), a structured, user-side repository of personal data. It carries the argument because it separates data storage from the agent, allowing access to be mediated by formal policy mechanisms such as ODRL and policy engines with formal semantics. The paper also leans on a hybrid communication model—natural language for negotiation followed by structured data for committing to transactions—as the enabler for multi-agent coordination.

What would settle it

A user study in which people encode their own data-sharing preferences in the proposed policy language would settle the question: if participants routinely produce rules that permit a disclosure they later reject or block an action they wanted, the claimed control advantage of CUPAs collapses.

Watch

Extended reading notes

Core claim

The central claim is the definition offered in Section 1: a Computer-Using Personal Agent is a Computer-Using Agent that has controlled access to a structured repository of private information relating to a user. The paper further proposes to build that repository as a Personal Knowledge Graph, which makes the user's data queryable and integrable while enabling declarative policies over how the agent may use it. The paper argues this configuration is the right middle ground between giving agents no personal data and giving them unfettered access, and it lays out a three-stage roadmap—starting with today's CUAs augmented with a PKG, moving to agents that interact with websites over HTTP rather than through vision models, and finally to networks of cooperating CUPAs.

Load-bearing premise

The argument assumes that users can express their privacy intentions as machine-readable policies over a personal knowledge graph accurately enough to both permit useful automation and prevent misuse.

Editorial extensions

If this is right

  • CUPAs can safely automate form filling and other personal-data tasks that current CUAs push back to the user in take-over mode.
  • By combining the PKG with external knowledge sources, a CUPA can make context-aware choices, such as filtering recipes by health constraints or substituting unavailable ingredients.
  • CUPAs representing different users can exchange information and negotiate toward outcomes that respect each user's preferences and policies.
  • A staged deployment path exists: first augment existing CUAs with a PKG, then teach agents to interact with Web APIs directly, then connect CUPAs into networks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proposal implies that the main site of privacy governance shifts from websites' cookie banners and consent dialogs to a user-side policy file that travels with the agent; if standardised, this could make consent expressions portable across services.
  • A testable extension would be a benchmark of realistic user privacy intentions encoded as PKG policies, measuring whether the policy layer permits exactly the intended disclosures and blocks the unintended ones.
  • The 'discuss then transact' idea suggests that reliability of multi-agent deals may hinge less on making agents fully fluent in natural language and more on having a structured commitment layer underneath the conversation.
  • If CUPAs become widespread, the value of personal data may shift from companies that aggregate it to the individual, since a single user's PKG can be enriched and reused across many tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Computer-Using Personal Agents (CUPAs), defined in Section 1 as a Computer-Using Agent that has controlled access to a structured repository of private information relating to a user, instantiated as a Personal Knowledge Graph (PKG). It motivates the idea with a dinner-planning scenario (Section 2), surveys personal data management, PKGs, and LLM-based agents (Section 3), argues for added value such as multi-dimensional negotiation and auditing (Section 4), lists capabilities (Section 5), outlines technical challenges including security and trust (Section 6), and gives a three-stage roadmap from PKG-enhanced CUAs to networks of CUPAs (Section 7). The paper's central claim is that coupling browser-automating agents with a user-owned PKG and machine-readable policies yields better control, greater automation, better interoperability, and multi-user coordination than plain CUAs.

Significance. If the central architectural claim holds, the paper makes a useful conceptual contribution: it names a class of systems, connects it to existing technologies (Operator, Agent-E, ODRL, Solid/Charlie, FoodKG), and provides a concrete roadmap with intermediate milestones. The paper is clearly framed as a position/roadmap (Section 7), so the absence of empirical evaluation is appropriate. The main strength is that the proposal is specific enough to be falsifiable once an enforcement architecture is specified; one could then test whether policy-controlled PKG access improves user control relative to direct data sharing or takeover-mode CUAs. The main weakness is that the paper does not yet specify that enforcement architecture, and the claimed 'better control' advantage depends on it.

major comments (3)
  1. [Section 6; Section 4] The claim that CUPAs give users 'better control' over personal data is load-bearing (abstract, Section 4), but the paper never specifies an enforcement boundary. In the proposed architecture the same LLM-based agent that browses untrusted websites also reads the PKG and is responsible for honoring policies; the policies are just additional data in the agent's context. As a result, a website visited by the agent can inject instructions that cause the agent to ignore or misapply a policy, and a policy statement cannot by itself prevent the agent from emitting PKG-derived values in its own web requests. For 'controlled access' to be meaningful, the paper should commit to an external reference monitor that evaluates policies before any PKG data is exposed to the agent and before any agent-observable action is performed, and should discuss how prompt-injection resistance is achieved.
  2. [Section 7 (first roadmap step)] The first roadmap step says to extend CUAs 'such that they use a PKG in order to access knowledge personal to the user,' which places the LLM inside the trust boundary. This is in tension with the definition's 'controlled access' and with the abstract's control claim. The paper should either specify that in this stage all PKG reads are mediated by a separate policy engine (with the LLM seeing only permitted projections), or justify why the control property is intended to hold in a weaker sense. As written, the roadmap appears to defer the central control mechanism to an unspecified later stage.
  3. [Section 4; Section 6 (Security, Privacy, and Policies)] The proposal assumes that users can express privacy and delegation intentions as machine-readable policies over the PKG, citing ODRL and policy engines. It does not, however, address how non-expert users will author, review, and update these policies, nor what happens when policies are incomplete, inconsistent, or out of date. Since Section 4 argues that CUPAs reduce cognitive load relative to broad human-level policies, this usability and maintainability question is directly relevant to the 'better control' claim. A roadmap treatment should at least name this as an open problem and point to candidate mechanisms (e.g., policy templates, interactive policy elicitation, policy evolution from user corrections).
minor comments (5)
  1. [Section 2] The text 'invokes atakeover mode' should read 'invokes a takeover mode'.
  2. [Section 2] The sentence 'he requests that his CUPA to generates some suggestions' is ungrammatical; suggest 'his CUPA generate'.
  3. [Abstract; Section 4] The comparative claims such as 'better control' and 'better interoperability' are stated as facts, but the paper presents no evaluation; consider phrasing them as intended advantages or hypotheses to match the roadmap framing.
  4. [Section 7] The relationship between the three 'levels' and the three stages is not explicit; a sentence mapping each stage to the corresponding trust, accountability, and autonomy level would help.
  5. [Section 8] The phrase 'This report is a result of Dagstuhl Seminar 25051' should be 'This report resulted from Dagstuhl Seminar 25051'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CUPA is a definitional vision statement with no derivation chain; incidental self-citations are not load-bearing.

full rationale

The paper is a short position/roadmap essay, not a derivation or empirical study. Its central object, the CUPA, is introduced by definition ('a Computer-Using Agent (CUA) that has controlled access to a structured repository of private information relating to a user'), and the claimed benefits (better control, more automation, interoperability, coordination) are stated as design goals, not as consequences derived from fitted parameters or from prior results. No equation or prediction is constructed from data, so there is no fitted-input-called-prediction step. The citations to co-authored works (FoodKG [16], OWL-S [23], Wright's 'discuss then transact' model [31], and cookie governance [32]) appear only as illustrative examples or future-roadmap options; none is used to justify the central definition or to forbid alternatives. There is no uniqueness theorem invoked from the authors' prior work, and no ansatz is smuggled in via citation. The closest thing to a circularity concern is that 'better control' is asserted rather than proven, and the policy enforcement mechanism is not fully specified (Section 6); but this is an open technical risk, not a circular reduction. Accordingly, no circular step is exhibited, and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The central claim rests on real-world assumptions about LLM reliability, knowledge graph coverage, user policy management, and inter-agent interoperability; none are experimentally justified. No free parameters are fitted because no quantitative model is given. The CUPA is a newly named architecture, not an entity with external falsifiable evidence.

assumptions (5)
  • domain assumption LLM-based agents can perform computer tasks reliably enough for real user tasks
    The paper's Section 3 cites Kim et al. and OpenAI's Operator to support this; the entire CUA premise depends on this reliability, but no reliability evidence specific to the proposed scenario is given.
  • domain assumption A Personal Knowledge Graph can capture sufficient personal context (schedule, health, preferences) to support automation
    Section 2's scenario assumes the PKG holds schedule, allergy information, glucose thresholds, and preferences; the paper cites surveys but does not validate this coverage.
  • domain assumption Users can author and maintain machine-readable policies that accurately capture privacy and delegation intent
    Section 6 relies on ODRL and policy engines; the claimed user control advantage depends on this assumption, which is not tested.
  • domain assumption CUPAs of different users can communicate and negotiate with shared semantics
    Section 7's third stage depends on structured service descriptions and Wright's discuss-then-transact model; inter-agent semantics are assumed rather than demonstrated.
  • domain assumption Websites and Web APIs expose interfaces that permit agent interaction without breaking terms or structure
    Section 5 states CUPAs must interact with diverse websites and APIs; no evidence or access model is provided for this key operational premise.
invented entities (1)
  • CUPA (Computer-Using Personal Agent)
    purpose: Proposed architecture combining a computer-using agent with controlled access to a user's personal knowledge graph, intended to automate tasks with better user control and interoperability.
    The paper gives a scenario and a roadmap but no specification, API, schema, or measurable behavior that could confirm or falsify the architecture from outside.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Computer-Using Personal Agents." pith.science (2026). https://pith.science/paper/C54FF73H

@misc{pith2026250315515,
  author       = {Pith},
  title        = {Pith review of: Towards Computer-Using Personal Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C54FF73H}},
  note         = {Machine review of arXiv:2503.15515}
}
read the original abstract

Computer-Using Agents (CUA) enable users to automate increasingly-complex tasks using graphical interfaces such as browsers. As many potential tasks require personal data, we propose Computer-Using Personal Agents (CUPAs) that have access to an external repository of the user's personal data. Compared with CUAs, CUPAs offer users better control of their personal data, the potential to automate more tasks involving personal data, better interoperability with external sources of data, and better capabilities to coordinate with other CUPAs in order to solve collaborative tasks involving the personal data of multiple users.

Figures

Figures reproduced from arXiv: 2503.15515 by the authors.

Figure 1
Figure 1. Computer-Using Personal Agent state that Operator is “still learning, evolving and may make mistakes” [25]. There are thus many open questions relating to the use of personal user data by CUAs. Conversely, there are many potential benefits to users if CUAs are empowered with personal data. CUAs could autofill forms with personal data for users in a context-aware and potentially generative manner, automating a tediou… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 34 canonical work pages

  1. [1]

    Agent-e: From autonomous web navigation to foundational design principles in agentic systems

    Abuelsaad, T., Akkil, D., Dey, P., Jagmohan, A., Vempaty, A., and Kokku, R. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032 (2024)

    arXiv 2024

  2. [2]

    Trust in AI: progress, challenges, and future directions

    Afroogh, S., Akbari, A., Malone, E., Kargar, M., and Alambeigi, H. Trust in AI: progress, challenges, and future directions. Humanities and Social Sciences Communications 11 , 1 (2024), 1–30

    work page 2024

  3. [3]

    The science of managing our digital stuff

    Bergman, O., and Whittaker, S. The science of managing our digital stuff . MIT Press, 2016

    work page 2016

  4. [4]

    Charlie Works

    Berners-Lee, T. Charlie Works. Design Issues, https://www.w3.org/DesignIssues/Works.html, 2025

    work page 2025

  5. [5]

    The semantic web

    Berners-Lee, T., Hendler, J., and Lassila, O. The semantic web. Scientific American 284, 5 (2001), 34–43

    work page 2001

  6. [6]

    T., and W ard, C

    Birch, K., Cochrane, D. T., and W ard, C. Data as asset? the measurement, governance, and valuation of digital personal data by big tech. Big Data and Society 8 (2021)

    work page 2021

  7. [7]

    Chakraborty, P., Dutta, S., and Sanyal, D. K. Personal research knowledge graphs. In WWW 2022 - Companion Proceedings of the Web Conference 2022 (4 2022), Association for Computing Machinery, Inc, pp. 763–768

    work page 2022

  8. [8]

    Chakraborty, P., and Sanyal, D. K. A comprehensive survey of personal knowledge graphs, 11 2023

    work page 2023

Show all 36 references
  1. [9]

    Personal data: Thinking inside the box

    Chaudhry, A., Crowcroft, J., Howard, H., Madhavapeddy, A., Mortier, R., Haddadi, H., and McAuley, D. Personal data: Thinking inside the box. Aarhus Series on Human Centered Computing 1 (2015), 4

    2015

  2. [10]

    Trust-based service management for social internet of things systems

    Chen, R., Bao, F., and Guo, J. Trust-based service management for social internet of things systems. IEEE transactions on dependable and secure computing 13 , 6 (2015), 684–696

    2015

  3. [11]

    Cushing, A. L. PIM as a caring: using ethics of care to explore personal information management as a caring process. Journal of the Association for Information Science and Technology 74 , 11 (2023), 1282–1292

    2023

  4. [12]

    L., and Kerrigan, P

    Cushing, A. L., and Kerrigan, P. Personal information management burden: A framework for describing nonwork personal information management in the context of inequality. Journal of the Association for Information Science and Technology 73 (11 2022), 1543–1558

    2022

  5. [13]

    A., Mamede, H

    da Silva Costa, D. A., Mamede, H. S., and da Silva, M. M. Robotic Process Automation (RPA) adoption: a systematic literature review, 6 2022

    2022

  6. [14]

    R., and V atrapu, R

    Faber, B., Michelet, G., Weidmann, N., Mukkamala, R. R., and V atrapu, R. Bpdims:a blockchain-based personal data and identity management system. Proceedings of the Annual Hawaii International Conference on System Sciences 2019-Janua (2019), 6855–6864

    2019

  7. [15]

    Fornara, N., Rodríguez-Doncel, V., Esteves, B., Steyskal, S., and Smith, B. W. ODRL Formal Semantics, May 2024

    2024

  8. [16]

    L., and Zaki, M

    Haussmann, S., Seneviratne, O., Chen, Y., Ne’eman, Y., Codella, J., Chen, C.-H., McGuinness, D. L., and Zaki, M. J. FoodKG: A semantics-driven knowledge graph for food recommendation. In The Semantic Web – ISWC 2019 (Cham, 2019), C. Ghidini, O. Hartig, M. Maleshkova, V. Svátek...

    2019

  9. [17]

    ODRL Information Model 2.2, Feb 2023

    Iannella, R., and Villata, S. ODRL Information Model 2.2, Feb 2023

    2023

  10. [18]

    The future of personal information management, part 1: Our information, always and forever

    Jones, W. The future of personal information management, part 1: Our information, always and forever . Morgan & Claypool Publishers, 2012

    2012

  11. [19]

    P., and Teevan, J

    Jones, W. P., and Teevan, J. Personal information management, vol. 14. University of Washington Press Seattle, WA, 2007

    2007

  12. [20]

    YourDigitalSelf: a personal digital trace integration tool

    Kalokyri, V., Borgida, A., and lie Marian, A. YourDigitalSelf: a personal digital trace integration tool. International Conference on Information and Knowledge Management, Proceedings (2018), 1963–1966

    2018

  13. [21]

    Language Models can Solve Computer Tasks

    Kim, G., Baldi, P., and McAleer, S. Language Models can Solve Computer Tasks. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 (2023), A. Oh, T. ...

    2023

  14. [22]

    InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22), July 11â •fi15, 2022, Madrid, Spain (2022), vol

    Lin, J., and W ang, M.PKG : A Personal Knowledge Graph for Recommendation. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22), July 11â •fi15, 2022, Madrid, Spain (2022), vol. 1, Association for Computi...

    2022

  15. [23]

    Bringing semantics to web services: The OWL-S approach

    Martin, D., Paolucci, M., McIlraith, S., Burstein, M., McDermott, D., McGuinness, D., Parsia, B., Payne, T., Sabou, M., Solanki, M., Srinivasan, N., and Sycara, K. Bringing semantics to web services: The OWL-S approach. In Semantic Web Services and Web Process Composition (Ber...

    2005

  16. [24]

    Mortier, R., Zhao, J., Crowcroft, J., W ang, L., Li, Q., Haddadi, H., Amar, Y., Crabtree, A., Colley, J., Lodge, T., Brown, T., McAuley, D., and Greenhalgh, C. Personal data management with the databox: What’s inside the box? In Proceedings of the 2016 ACM Workshop on Cloud-As...

    2016

  17. [25]

    Introducing Operator

    OpenAI Team . Introducing Operator. OpenAI Blog https://openai.com/index/introducing-operator/, published 2025-01-23, accessed 2025-01-29, 2025

    2025

  18. [26]

    A Human-in-the-Loop Approach for Personal Knowledge Graph Construction from File Names

    Schröder, M., Jilek, C., and Dengel, A. A Human-in-the-Loop Approach for Personal Knowledge Graph Construction from File Names. In Knowledge Graph Construction (2022), vol. 3141, CEUR Workshop Proceedings

    2022

  19. [27]

    G., Balog, K., Bernard, N., Łajewska, W., and Linjordet, T

    Skjæveland, M. G., Balog, K., Bernard, N., Łajewska, W., and Linjordet, T. An ecosystem for personal knowledge graphs: A survey and research roadmap. AI Open 5 (2024), 55–69

    2024

  20. [28]

    D., Greenwood, D., Chan, A., and Pentland, A

    South, T., Marro, S., Hardjono, T., Mahari, R., Whitney, C. D., Greenwood, D., Chan, A., and Pentland, A. Authenticated delegation and authorized ai agents. arXiv preprint arXiv:2501.09674 (2025)

    2025 arXiv

  21. [29]

    M., Bichler, M., and Heinzl, A

    van der Aalst, W. M., Bichler, M., and Heinzl, A. Robotic process automation. Business and Information Systems Engineering 60 (8 2018), 269–272

    2018

  22. [30]

    An introduction to multiagent systems

    Wooldridge, M. An introduction to multiagent systems . Wiley, 2009

    2009

  23. [31]

    Here’s Charlie! Realising the semantic web vision of agents in the age of LLMs.CoRR abs/2409.04465 (2024)

    Wright, J. Here’s Charlie! Realising the semantic web vision of agents in the age of LLMs.CoRR abs/2409.04465 (2024)

    2024 arXiv

  24. [32]

    Me want cookie! Towards automated and transparent data governance on the Web, 2024

    Wright, J., Esteves, B., and Zhao, R. Me want cookie! Towards automated and transparent data governance on the Web, 2024

    2024

  25. [33]

    InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

    Y ang, J., Prabhakar, A., Narasimhan, K., and Y ao, S. InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, ...

    2023

  26. [34]

    M., Pina-Navarro, M., W ang, T., Li, X., Li, H., Fu, Y., W ang, R., Zhang, J., and Li, I

    Zhao, X., Blum, M., Y ang, R., Y ang, B., Carpintero, L. M., Pina-Navarro, M., W ang, T., Li, X., Li, H., Fu, Y., W ang, R., Zhang, J., and Li, I. AGENTiGraph: an interactive knowledge graph platform for LLM-based chatbots utilizing private data, 2024

    2024

  27. [35]

    Beyond preferences in ai alignment

    Zhi-Xuan, T., Carroll, M., Franklin, M., and Ashton, H. Beyond preferences in ai alignment. Philosophical Studies (Nov. 2024)

    2024

  28. [36]

    Zyskind, G., Nathan, O., and Pentland, A. S. Decentralizing privacy: Using blockchain to protect personal data. Proceedings - 2015 IEEE Security and Privacy Workshops, SPW 2015 (2015), 180–184

    2015

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.