Pith. sign in

REVIEW 4 major objections 4 minor 67 references

Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prototyping the user experience of computer-use agents is a two-phase, five-activity process that requires six specific tool capabilities, and a design probe shows non-experts can work within it.

desk verdict The taxonomy is the contribution; the validation claim is not supported by the data. read the letter →

arxiv 2510.04452 v3 pith:SPXV24C7 submitted 2025-10-06 cs.HC cs.AI

Jenny T. Liang , Titus Barik , Jeffrey Nichols , Eldon Schoop , Ruijia Cheng This is my paper
classification cs.HCcs.AI
keywords agentuserexperienceinterfaceagentsprototypingdesignrequirementsno-codetoolswebhuman-AIinteractionprobe
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prototyping the user experience of computer-use agents—AI that performs actions inside interfaces from natural-language commands—is a distinct activity from training or benchmarking such agents, and the paper argues it has been understudied. Drawing on interviews with 12 participants, it distills the activity into two highly iterative phases, designing the agent and inspecting it, then into five concrete activities: scoping the agent's tasks and user knowledge, designing its information display, designing its interactions with users, running the prototype, and understanding its runtime behavior. From these it derives six capabilities that any tool scaffolding agent UX prototyping should offer, from no-code interfaces to runtime debugging support. The authors instantiate the capabilities in a design probe and show that 14 participants with varied backgrounds could design and evaluate multiple distinct agent experiences with it. If correct, this provides the first systematic design requirements for agent UX prototyping tools and opens the door for non-engineers to shape agent behavior.

What carries the argument

The load-bearing mechanism is the Activity–Capability taxonomy itself: five prototyping Activities organized into two phases (Designing and Inspecting), each mapped to a Desired Capability (C1–C6) that a scaffolding tool should afford. AgentBuilder—a design probe combining a node-and-edge workflow editor, a structured natural-language prompt editor with bidirectional graph-prompt translation, and a browser execution environment with global agent controls and a detailed debug mode—serves as the concrete instantiation that makes the taxonomy testable. The two-way workflow-prompt translation is the key enabling device: it lets users alternate between graphical and textual notations while keepin

What would settle it

A controlled comparison where designers prototype an agent experience with a tool missing one of the six capabilities—say, no debugging support—and still complete the design task successfully would contradict the claim that all six are needed.

Watch

Extended reading notes

Core claim

The central claim is a validated design-requirements taxonomy for agent experience prototyping. Prototyping an interface agent's UX consists of two experimental, tightly coupled phases—Designing the Agent and Inspecting the Agent—which break down into five Activities: A1 scope and boundaries, A2 information display, A3 interactions between agent and user, A4 running the prototype, and A5 understanding runtime behavior. Each Activity yields a Desired Capability (C1–C6) that an agent prototyping system should support: no-code interfaces, constraining the agent's task space and user knowledge, defining the agent's UI in chat and environment, providing interaction components, providing a run/con

Load-bearing premise

The evidence for the six capabilities comes from watching people use a tool that was built to include all six, so the study may be confirming the tool rather than independently testing the requirements.

Editorial extensions

If this is right

  • People without AI engineering backgrounds can design and evaluate working agent experiences: all 14 study participants produced one to five distinct prototypes, ranging from fully autonomous to highly interactive.
  • Because the two phases are tightly coupled, agent prototyping tools need fast run-edit loops; pause/resume/cancel controls and debug replay are core requirements, not optional extras.
  • Tools must separate the information shown to end-users from the information shown to developers, since participants who hid execution details from end-users were less able to understand and debug their own agents.
  • Supporting both workflow-graph and natural-language-prompt authoring with bidirectional translation lets different developer styles (visual-first vs. text-first) use the same tool, as observed when participants generated prompts from graphs or graphs from prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the five activities generalize beyond web agents, the same C1–C6 list could scaffold prototyping for mobile and desktop computer-use agents; testing this is a straightforward extension.
  • Editorial inference: The end-user/developer information conflict hints that agent prototyping tools could offer an explicit role-switch mode; a study measuring debugging success with and without such a mode would test this.
  • Editorial inference: The single-site, 30-minute tasks likely under-represent long-horizon and multi-site agent workflows, so the taxonomy may be necessary but not sufficient for enterprise-scale agent UX; additional capabilities may emerge from longer-form studies.
  • Editorial inference: The finding that participants struggled to interpret multimodal runtime inputs (accessibility trees, screenshots) suggests a broader principle: debugging tools for agents need to translate between how the agent sees the interface and how humans see it, not just expose raw inputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The full text presents two qualitative studies on prototyping user experiences of (M)LLM-based interface agents. Study 1 is a requirements elicitation study with 12 participants (six experts, six non-experts) that yields five prototyping Activities (A1–A5) and six Desired Capabilities (C1–C6) for agent prototyping systems, organized into two phases: Designing the Agent and Inspecting the Agent. Study 2 is an in-situ study with 14 non-expert/less-expert developers who used AgentBuilder, a no-code design probe built to instantiate C1–C6, to prototype an agent for a coffee-chain website. The paper claims that Study 2 validates the design requirements and elicits tooling needs, culminating in design implications for agent UX prototyping tools. The supplied abstract, however, refers to a different artifact (AgentUXlab) and a 21-subcategory UX taxonomy that is not present in the full text.

Significance. If the proposed Activities and Desired Capabilities are accepted, they would be a useful early design-space articulation for agent UX prototyping tools, connecting prompt-prototyping literature to interface agents and supporting the democratization of agent design. The paper's strengths include a detailed design probe, rich qualitative observation, and appendices with concrete prototype walk-throughs and study materials. However, the validation evidence does not currently support the strength of the claims: the in-situ study is partly self-confirming, the coding is primarily single-author, and the manuscript explicitly concedes that the results should be treated as hypotheses. The contribution is therefore significant but conditional on reframing and, ideally, additional independent validation.

major comments (4)
  1. [Abstract; §1; §6.3] The abstract and full text are internally inconsistent. The abstract promises a 21-subcategory taxonomy and a probe called AgentUXlab, but the full text instead presents five Activities and six Desired Capabilities and a probe called AgentBuilder. Additionally, the first contribution claims 'validated design requirements,' while §6.3 states 'the results of this work should be treated as hypotheses for future work to validate.' These contradictory claims must be reconciled; as written, the central contribution is overstated.
  2. [§5.1.3; §5.2.2] The validation path is circular. AgentBuilder was built to implement C1–C6 (Section 4), and the in-situ study 'validates' each capability by observing participants use the corresponding affordance, with coding performed using the same set of Desired Capabilities (§5.1.3). Observing that participants can operate a tool that was designed to embody those capabilities is evidence of feasibility, not an independent test of whether those capabilities are the right design requirements. No baseline, alternative tool, or blind elicitation was used. The phrase 'We validate all the Desired Capabilities' in §5.2.2 is therefore too strong.
  3. [§3.1.3] The requirements elicitation was coded primarily by the author who conducted the interviews, with review by one other researcher. No inter-rater reliability or independent coding is reported. Since A1–A5 and C1–C6 form the paper's principal contribution, the lack of a second independent coder weakens the claim that these categories are 'validated' rather than emergent. The limitation should be disclosed in the limitations section, and the contribution wording should be calibrated accordingly.
  4. [Abstract vs. full text] If the 'taxonomy of design considerations for agent UX, comprising 21 subcategories' is a contribution of this work, it is absent from the full text; the Results section reports only the Activities and Desired Capabilities. This is not merely a presentational issue: readers cannot evaluate the central claim as advertised. The taxonomy must either be added to the manuscript or explicitly removed from the abstract and contribution list.
minor comments (4)
  1. [Throughout] There are notable template and typographical issues: §5.1.2 'sesesion'; §3.2.1 'prototying'; Appendix A.1 'abotu'; §4.1 'webapage'; §4.1 'Mel's cart' should be 'Mei's cart'. The ACM Reference Format placeholder dates (2018, 'Received 20 February 2007') should be updated before submission.
  2. [Fig. 1] The caption reads 'we implement these design requirements in AgentBuilder (Section 4) use the tool as a probe...'; this is ungrammatical and should be revised for clarity.
  3. [References] Reference [6] is listed as 'Anonymous Authors' with a PCS link; this is acceptable for anonymous submission but must be finalized and all anonymized references replaced in the camera-ready version.
  4. [§5.1.2] The protocol says participants 'remotely controlling the researcher's screen with the video conferencing software'; this is a significant usability constraint and should be acknowledged earlier, not only in the limitations, since it may affect how participants interacted with the tool.

Circularity Check

1 steps flagged · score 6.0 of 10

Circular validation: AgentBuilder is built from C1–C6, the in-situ study codes participants using that same category set, and then 'validates' the capabilities by observing use of those affordances.

  1. self definitional [Sections 4–5.2.2 (esp. §4 opening, §5.1.3, §5.2.2); claimed as contribution in §1]
    "To instantiate the Desired Capabilities of an agent prototyping system from Section 3, we develop a design probe, AgentBuilder. ... We used AgentBuilder as a design probe in an in situ study to validate the Activities and Desired Capabilities of agent prototyping scaffolds (Section 3). ... the author performed provisional coding [49] using the set of Desired Capabilities from the requirements elicitation study as well as open coding. ... We validate all the Desired Capabilities identified in Section 3.2 by observing participants engaged with the affordances of AgentBuilder that instantiate the"

    The C1–C6 set is both the specification used to construct AgentBuilder and the coding scheme used to evaluate it. Section 4 says the probe was built to instantiate the Section 3 Desired Capabilities; Section 5.1.3 says the analysis used the same Desired Capabilities as provisional codes; Section 5.2.2 then treats participants' use of the corresponding affordances as validating those capabilities. Because the tool was engineered to contain those affordances and the observations were coded with those categories, the in-situ study cannot fail to 'validate' the requirements merely by showing the affordances were used. The claimed validation therefore reduces to a self-fulfilling demonstration, not an independent test. The requirements-elicitation study itself is an independent grounding, so th

full rationale

The taxonomy's origin is independent: Section 3 elicits A1–A5 and C1–C6 from 12 interviews, so the design space is not purely invented by the probe. The circularity is in the validation step. AgentBuilder is built to instantiate C1–C6 (Section 4); the in-situ study's analysis codes with 'the set of Desired Capabilities from the requirements elicitation study' (Section 5.1.3); and the results section says the capabilities are 'validate[d] ... by observing participants engaged with the affordances of AgentBuilder that instantiate these capabilities' (Section 5.2.2). Observing use of affordances a tool was designed to provide, and coding behavior with the same categories, confirms availability/usability but not that these are the right requirements; it is a construction-consistent demonstration. Section 6.3 itself concedes: 'Therefore, the results of this work should be treated as hypotheses for future work to validate,' which undercuts the Abstract/Contributions phrasing 'validated design requirements.' There is no external benchmark, baseline tool, or blind elicitation to break the loop. Because the requirements elicitation is an independent source and the probe implementation is a reasonable instantiation, this is partial circularity rather than the whole derivation being definitional. Score 6 reflects one central 'prediction' (the validated requirements) reducing by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is a qualitative HCI paper with no numeric free parameters or invented physical/theoretical entities. The load-bearing assumptions are methodological: coding reliability, self-report validity, representativeness of the single agent implementation, and cross-domain generalizability.

assumptions (4)
  • domain assumption Single-coder qualitative analysis with one review produces reliable themes
    Section 3.1.3: analysis was performed by the author who conducted the interviews, with codes reviewed by one researcher; no inter-rater reliability metric is reported. The validity of the entire taxonomy depends on this assumption.
  • domain assumption Participants' self-reports and think-aloud reflections accurately represent the prototyping process
    Sections 3.1.2 and 5.1.2 rely on interviews, discussions of mock agents, and think-aloud protocols as evidence for activities and needs, assuming participants can articulate and do not distort their experiences.
  • domain assumption The CowPilot-based execution environment and its LLM are sufficiently representative of real computer use agents
    Section 4.2: AgentBuilder's execution interface reuses CowPilot's agent. The paper generalizes from this single agent implementation to 'computer use agents' broadly; an atypical or faulty agent would distort the observed prototyping needs.
  • domain assumption Findings generalize beyond the single company and single website studied
    All participants were recruited from one large technology company, and the in-situ study used only a Starbucks website with linked payment information (Sections 3.1.1, 5.1.2, B.2). The paper acknowledges this in Section 6.3 but still presents the design requirements as general.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes." pith.science (2026). https://pith.science/paper/SPXV24C7

@misc{pith2026251004452,
  author       = {Pith},
  title        = {Pith review of: Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPXV24C7}},
  note         = {Machine review of arXiv:2510.04452}
}
read the original abstract

Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands. Current research focuses on training and evaluating the underlying models, leaving these agents' user experience (UX) understudied. We conducted two studies to understand the design space of agent UX (RQ1) and the support required to prototype it (RQ2). First, we develop a taxonomy of design considerations for agent UX, comprising 21 subcategories of UX considerations. Then, through a requirements elicitation study with 12 participants---including six agent experts---we identify five Activities and six Desired Capabilities needed in tools prototyping agent UX. Informed by these results, we created AgentUXlab, a design probe that enables developers to design agents with different UX approaches for a website and evaluate those experiences by executing prototypes in a browser. From a user study with 14 participants, we elucidate tooling insights and derive design implications for agent UX prototyping tools.

Figures

Figures reproduced from arXiv: 2510.04452 by the authors.

Figure 1
Figure 1. An overview of the study. We first perform a requirements elicitation study (Section [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of the AgentBuilder prototyping interface in the Workflow tab. Developers define agent workflows (middle) using nodes and edges. Nodes ( A - F ) are selected from a library of actions the agent can take 1 . Edges define conditions and can be selected from a library or manually written in by the user G . Developers can further specify how the agent requests user input ( A , B ), and what information to di… view at source ↗
Figure 3
Figure 3. The AgentBuilder Prompt panel as part of the prototyping interface allows users to develop a structured prompt. Under the Edit tab 1 , users can manually or automatically generate a description of their workflow A to include in their prompt. In addition, the user can input information about the agent’s scope B , what knowledge it knows about the user C , and other specific instructions they want the agent to follow … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Overview of the AgentBuilder execution interface, an extension of the CowPilot agent [ [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Overview of the AgentBuilder execution interface’s [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Example agent prototypes 1 2 3 for the same task and user query, which is to order a cappuccino with the user query "Order me a coffee please!" Prototype 1 is interactive, and shows the most information about the agent’s UI actions; prototype 2 executes the task autono…
Figure 7
Figure 7. Figure 7: The agent prototype that P1 created during the [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]
Figure 8
Figure 8. Figure 8: The agent prototype that P4 created during the [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: The agent prototype that P6 created during the [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: The agent prototype that P14 created during the [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 6 canonical work pages

  1. [1]

    James E Allen, Curry I Guinn, and Eric Horvtz. 1999. Mixed-initiative interaction.IEEE Intelligent Systems and their Applications14, 5 (1999), 14–23. doi:10.1109/5254.796083

    arXiv 1999

  2. [2]

    Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software Engineering for Machine Learning: A Case Study. In2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 291–300. doi:10.1109/ICSE-SEIP.2...

    arXiv 2019

  3. [3]

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. InCHI Conference on Human Factors in Computing Systems (CHI). 1–13. doi:10.1145/ 3290605.3300233

    arXiv 2019

  4. [4]

    Anthropic. 2024. Architecture overview - Model Context Protocol — modelcontextprotocol.io. https://modelcontextprotocol.io/docs/learn/ architecture. [Accessed 11-09-2025]

    2024

  5. [5]

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena L Glassman. 2024. Chainforge: A visual toolkit for prompt engineering and LLM hypothesis testing. InCHI Conference on Human Factors in Computing Systems (CHI). 1–18. doi:10.1145/3613904.3642016

    arXiv 2024

  6. [6]

    An Exploratory Study on Prototyping User Experiences of Interface Agents

    Anonymous Authors. 2025. Supplemental Materials to "An Exploratory Study on Prototyping User Experiences of Interface Agents". The supplemental materials are available on PCS. We will make the supplemental materials publicly available upon acceptance. Manuscript submitted to ACM 24 Liang et al

    2025

  7. [7]

    Joshua Bloch. 2006. How to design a good API and why it matters. InCompanion to the ACM SIGPLAN Symposium on Object-Oriented Programming Systems, Languages, and Applications. 506–507. doi:10.1145/1176617.1176622

    arXiv 2006

  8. [8]

    Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, and Tat-Seng Chua. 2025. Large language models empowered personalized web agents. InACM on Web Conference (WWW). 198–215. doi:10.1145/3696410.3714842

    arXiv 2025

Show all 67 references
  1. [9]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

    2024

  2. [10]

    2015.Basics of qualitative research

    Juliet Corbin and Anselm Strauss. 2015.Basics of qualitative research. Vol. 14. Sage. doi:10.1177/1094428108324514

    2015 doi

  3. [11]

    Doris M Dehn and Susanne Van Mulken. 2000. The impact of animated interface agents: a review of empirical research.International Journal of Human-Computer Studies52, 1 (2000), 1–22. doi:10.1006/ijhc.1999.0325

    2000

  4. [12]

    Mateusz Dolata, Norbert Lange, and Gerhard Schwabe. 2024. Development in times of hype: How freelancers explore Generative AI?. InIEEE/ACM International Conference on Software Engineering (ICSE). 1–13. doi:10.1145/3597503.3639111

    2024

  5. [13]

    Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX design innovation: Challenges for working with machine learning as a design material. InCHI Conference on Human Factors in Computing Systems (CHI). 278–288. doi:10.1145/3025453.3025739

    2017

  6. [14]

    Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi. 2025. Interactive debugging and steering of multi-agent ai systems. InCHI Conference on Human Factors in Computing Systems. 1–15. doi:10.1145/3706598.3713581

    2025

  7. [16]

    Alexander J Fiannaca, Chinmay Kulkarni, Carrie J Cai, and Michael Terry. 2023. Programming without a programming language: Challenges and opportunities for designing developer tools for prompt programming. InCHI Conference on Human Factors in Computing Systems Extended Abstrac...

    2023

  8. [17]

    Gerhard Fischer. 2021. End-user development: Empowering stakeholders with artificial intelligence, meta-design, and cultures of participation. In International Symposium on End User Development. Springer, 3–16. doi:10.1007/978-3-030-79840-6_1

    2021 doi

  9. [18]

    Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. 2024. Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468(2024)

    2024 arXiv

  10. [19]

    Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K Kummerfeld, and Elena L Glassman. 2024. Supporting sensemaking of large language model outputs at scale. InCHI Conference on Human Factors in Computing Systems (CHI). 1–21. doi:10.1145/3613904.3642139

    2024

  11. [20]

    Pan Hao, Dongyeop Kang, Nicholas Hinds, and Qianwen Wang. 2025. FlowForge: Guiding the creation of multi-agent workflows with design space visualization as a thinking scaffold.arXiv preprint arXiv:2507.15559(2025)

    2025 arXiv

  12. [21]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. InAnnual Meeting of the Association for Computational Linguistics (ACL)(Volume 1: Long Papers)...

    2024 doi

  13. [22]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. InCHI Conference on Human Factors in Computing Systems (CHI). 159–166. doi:10.1145/302979.303030

    1999

  14. [23]

    Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P

    Faria Huq, Zora Zhiruo Wang, Frank F. Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P. Bigham, and Graham Neubig. 2025. CowPilot: A framework for autonomous and human-agent collaborative web navigation. InConference of the Nations of the Americas Chapter of the Association for Computat...

    2025 doi

  15. [24]

    Iat Long Iong, Xiao Liu, Yuxuan Chen, Hanyu Lai, Shuntian Yao, Pengbo Shen, Hao Yu, Yuxiao Dong, and Jie Tang. 2024. OpenWebAgent: An open toolkit to enable web agents on large language models. InAnnual Meeting of the Association for Computational Linguistics (ACL): System Dem...

    2024 doi

  16. [25]

    Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. Promptmaker: Prompt- based prototyping with large language models. InCHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI EA). 1–8. doi:10.11...

    2022

  17. [26]

    Ishika Joshi, Simra Shahid, Shreeya Manasvi Venneti, Manushree Vasu, Yantao Zheng, Yunyao Li, Balaji Krishnamurthy, and Gromit Yeuk-Yin Chan

  18. [27]

    Claire Kayacik, Sherol Chen, Signe Noerly, Jess Holbrook, Adam Roberts, and Douglas Eck. 2019. Identifying the intersections: User experience + research scientist collaboration in a generative machine learning interface. InExtended Abstracts of the CHI Conference on Human Fact...

    2019

  19. [28]

    Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. InCHI Conference on Human Factors in Computing Systems (CHI). 1–21. doi:10.1145/3613904.3642216

    2024

  20. [29]

    Scott R Klemmer, Anoop K Sinha, Jack Chen, James A Landay, Nadeem Aboobaker, and Annie Wang. 2000. Suede: A wizard of oz prototyping tool for speech user interfaces. InAnnual ACM symposium on User Interface Software and Technology (UIST). 1–10. doi:10.1145/354401.354406

    2000

  21. [30]

    Amy J Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, et al. 2011. The state of the art in end-user software engineering.ACM Computing Surveys (CSUR)43, 3 (2011), 1–44. doi:10.1145...

    2011

  22. [31]

    Amy J Ko, Brad A Myers, and Htet Htet Aung. 2004. Six learning barriers in end-user programming systems. InIEEE Symposium on Visual Languages-Human Centric Computing (VL/HCC). IEEE, 199–206. doi:10.1109/VLHCC.2004.47

    2004 doi

  23. [32]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. InProceedings of the 62nd Annual Meeting of the Asso...

    2024 doi

  24. [33]

    Robyn Kozierok and Pattie Maes. 1993. A learning interface agent for scheduling meetings. InInternational Conference on Intelligent User Interfaces. 81–88. doi:10.1145/169891.169908

    1993

  25. [34]

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2025. On the effects of data scale on UI control agents. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Cana...

    2025

  26. [35]

    Jenny T Liang, Aayush Kumar, Yasharth Bajpai, Sumit Gulwani, Vu Le, Chris Parnin, Arjun Radhakrishna, Ashish Tiwari, Emerson Murphy-Hill, and Guastavo Soares. 2025. TableTalk: Scaffolding spreadsheet development with a language agent.arXiv preprint arXiv:2502.09787(2025)

    2025 arXiv

  27. [36]

    Jenny T Liang, Melissa Lin, Nikitha Rao, and Brad A Myers. 2025. Prompts are programs too! Understanding how developers build software containing prompts.Proceedings of the ACM on Software Engineering2, FSE (2025), 1591–1614. doi:10.1145/3729342

    2025 doi

  28. [37]

    Jenny T Liang, Chenyang Yang, Agnia Sergeyuk, Travis D Breaux, and Brad A Myers. 2025. Understanding prompt programming tasks and questions. arXiv preprint arXiv:2507.17264(2025)

    2025 arXiv

  29. [38]

    Henry Lieberman. 1995. Letizia: an agent that assists web browsing. InProceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 1(Montreal, Quebec, Canada)(IJCAI’95). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 924–929

    1995

  30. [39]

    Henry Lieberman. 1997. Autonomous interface agents. InCHI Conference on Human Factors in Computing Systems (CHI). 67–74. doi:10.1145/258549. 258592

    1997 doi

  31. [40]

    Henry Lieberman and Ted Selker. 2003. Agents for the user interface.Handbook of Agent Technology(2003), 1–21

    2003

  32. [41]

    We need structured output

    Michael Xieyang Liu, Frederick Liu, Alexander J Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J Cai. 2024. "We need structured output": Towards user-centered constraints on large language model output. InExtended Abstracts of the CHI Conference on Human Factors i...

    2024

  33. [42]

    Pattie Maes and Robyn Kozierok. 1993. Learning interface agents. InAAAI, Vol. 93. 459–465

    1993

  34. [43]

    Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice.Proceedings of the ACM on Human-Computer Interaction3, CSCW (2019), 1–23. doi:10.1145/3359174

    2019 doi

  35. [44]

    Aditi Mishra, Bretho Danzy, Utkarsh Soni, Anjana Arunkumar, Jinbin Huang, Bum Chul Kwon, and Chris Bryan. 2025. PromptAid: Visual prompt exploration, perturbation, testing and iteration for large language models.IEEE Transactions on Visualization and Computer Graphics (TVCG)(2...

    2025

  36. [45]

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. 2024. Webcanvas: Benchmarking web agents in online environments.arXiv preprint arXiv:2406.12373(2024)

    2024 arXiv

  37. [46]

    Landay, and Beverly Harrison

    Kayur Patel, James Fogarty, James A. Landay, and Beverly Harrison. 2008. Investigating statistical machine learning as a tool for software development. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Florence, Italy)(CHI ’08). Association for Compu...

    2008

  38. [47]

    Savvas Petridis, Michael Terry, and Carrie J Cai. 2024. Promptinfuser: How tightly coupling AI and UI design impacts designers’ workflows. InACM Designing Interactive Systems Conference (DIS). 743–756. doi:10.1145/3643834.3661613

    2024

  39. [48]

    Kevin Pu, Daniel Lazaro, Ian Arawjo, Haijun Xia, Ziang Xiao, Tovi Grossman, and Yan Chen. 2025. Assistance or disruption? Exploring and evaluating the design and trade-offs of proactive AI programming support. InCHI Conference on Human Factors in Computing Systems (CHI). 1–21....

    2025

  40. [49]

    2009.The coding manual for qualitative researchers

    Johnny Saldaña. 2009.The coding manual for qualitative researchers. SAGE Publications

    2009

  41. [50]

    Silvia Schiaffino and Analıa Amandi. 2004. User-interface agent interaction: Personalization issues.International Journal of Human-Computer Studies60, 1 (2004), 129–148. doi:10.1016/j.ijhcs.2003.09.003

    2004 doi

  42. [51]

    Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents.interactions4, 6 (1997), 42–61. doi:10.1145/267505.267514

    1997

  43. [52]

    Hari Subramonyam, Roy Pea, Christopher Pondoc, Maneesh Agrawala, and Colleen Seifert. 2024. Bridging the gulf of envisioning: Cognitive challenges in prompt based interactions with llms. InCHI Conference on Human Factors in Computing Systems (CHI). 1–19. doi:10.1145/3613904.3642754

    2024

  44. [53]

    Hari Subramonyam, Divy Thakkar, Andrew Ku, Juergen Dieber, and Anoop K Sinha. 2025. Prototyping with prompts: Emerging approaches and challenges in generative ai design for collaborative software teams. InCHI Conference on Human Factors in Computing Systems (CHI). 1–22. doi:10...

    2025

  45. [54]

    Stephen Sutton and Ronald Cole. 1997. The CSLU Toolkit: Rapid prototyping of spoken language systems. InAnnual ACM Symposium on User Interface Software and Technology (UIST). 85–86. doi:10.1145/263407.263517

    1997

  46. [55]

    Kyle Thayer, Sarah E Chasins, and Amy J Ko. 2021. A theory of robust API knowledge.ACM Transactions on Computing Education (TOCE)21, 1 (2021), 1–32. doi:10.1145/3444945 Manuscript submitted to ACM 26 Liang et al

    2021 doi

  47. [56]

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception. InInternational Conference on Learning Representations Workshop on Large Language Model Agent...

    2024

  48. [57]

    Justin D Weisz, Jessica He, Michael Muller, Gabriela Hoefer, Rachel Miles, and Werner Geyer. 2024. Design principles for generative AI applications. InCHI Conference on Human Factors in Computing Systems (CHI). 1–22. doi:10.1145/3613904.364246

    2024

  49. [58]

    Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. Promptchainer: Chaining large language model prompts through visual programming. InCHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI EA). 1–10

    2022

  50. [59]

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. 2024. OS-Copilot: Towards generalist computer agents with self-improvement. InInternational Conference on Learning Representations Workshop on Large Language Model Age...

    2024

  51. [60]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking Multimodal Agents ...

    2024 arXiv

  52. [61]

    Chenyang Yang, Tesi Xiao, Michael Shavlovsky, Christian Kästner, and Tongshuang Wu. 2025. Orbit: A framework for designing and evaluating multi-objective rankers. InInternational Conference on Intelligent User Interfaces (IUI). 1093–1106. doi:10.1145/3708359.3712092

    2025

  53. [62]

    Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi T Iqbal, and Jaime Teevan. 2019. Sketching NLP: A case study of exploring the right things to design with language intelligence. InCHI Conference on Human Factors in Computing Systems (CHI). 1–12. doi:10.1145/3290605.3300415

    2019

  54. [63]

    Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating how experienced UX designers effectively work with machine learning. InACM Conference on Designing Interactive Systems Conference (DIS). 585–596. doi:10.1145/3196709.3196730

    2018

  55. [64]

    Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. InCHI Conference on Human Factors in Computing Systems (CHI). 1–13. doi:10.1145/3313831.3376301

    2020

  56. [65]

    J Diego Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. InCHI Conference on Human Factors in Computing Systems (CHI). 1–21. doi:10.1145/3544548.3581388

    2023

  57. [66]

    Zhuohao Zhang, Eldon Schoop, Jeffrey Nichols, Anuj Mahajan, and Amanda Swearngin. 2025. From Interaction to impact: Towards safer AI agent through understanding and evaluating mobile UI operation impacts. InInternational Conference on Intelligent User Interfaces (IUI). 727–744...

    2025

  58. [67]

    Boyuan Zheng, Boyu Gou, Scott Salisbury, Zheng Du, Huan Sun, and Yu Su. 2024. WebOlympus: An open platform for web agents on live websites. InConference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations. 187–197. doi:10.18653/v1/2024.emnlp-demo.20

    2024 doi

  59. [68]

    Order me a coffee please!

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Repr...

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.