Pith. sign in

REVIEW 3 major objections 3 minor 96 references

Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents

T0 review · 3 major / 3 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sidekick shows that a multimodal feedback layer can lift multitasking performance with computer-use agents beyond text-only interfaces.

desk verdict Well-run study of multimodal feedback for CUA multitasking, but the bundled auto-pause undercuts the causal claim that the feedback modality is what helps. read the letter →

arxiv 2607.17527 v1 pith:26RI3EFB submitted 2026-07-20 cs.HC

Ruei-Che Chang , Wenqian Xu , Dingzeyu Li , Bryan Wang , Anhong Guo This is my paper
classification cs.HC
keywords computeruseagentsmultimodalfeedbackambientdisplaymultitaskinghuman-agentcollaborationcontextresumptionerrordetectionprototypesystem
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the bottleneck in human–computer-use-agent collaboration is not agent capability but communication design. It introduces Sidekick, a prototype feedback layer that adapts to three phases: colored ambient cues when the agent runs in the background, spoken and visual summaries when the user returns, and real-time verbalized reasoning with on-screen annotations when the agent works in the foreground. In a 30-participant study with arithmetic as the primary task and spreadsheet filling delegated to an agent, Sidekick produced a higher spreadsheet score and fewer errors than a chat-only baseline or a peripheral-text baseline, without increasing the number of task switches or monitoring time, and without reducing arithmetic performance. The authors take this as evidence that feedback supporting peripheral awareness, rapid context resumption, and transparency is what lets users delegate and intervene at the right moments.

What carries the argument

The carry-the-argument object is the stage-based feedback pipeline: (1) a small always-on-top 'Sidekick display' mapping the agent's error count to an ambient color, (2) a VLM-generated multimodal resumption summary that uses SSML timestamps to sync TTS narration with screenshots, and (3) real-time verbalization and bounding-box annotations of agent reasoning in the foreground. The error detector — a VLM comparing before-and-after screenshots to the subgoal — is the gate that sets the color thresholds and the auto-pause on red; it determines when users are alerted. The study's arithmetic task acts as a continuous distractor to measure whether the feedback steals attention.

What would settle it

Run the same 30-participant study with the VLM error detector replaced by perfect ground-truth labels, or with the detector's judgments deliberately corrupted (e.g., random noise), and compare spreadsheet error counts. If corrupting detector reliability does not change Sidekick's advantage over the text baselines, the central claim about the communication layer collapses; if it does change, the mechanism is error detection, not multimodality. A simpler check: log the detector's decisions against the two injected errors per column to see whether it actually fires on real mistakes and stays sile

Watch

Extended reading notes

Core claim

The central claim is that multimodal, stage-adapted communication improves multitasking with computer-use agents. Sidekick condenses raw agent messages into one-sentence summaries, shows thumbnails, and uses color (green/yellow/orange/red) driven by a vision-language-model error detector that counts consecutive failed actions and pauses the agent on red. On returning, it presents replay-like summaries with speech synced to screenshots. In the foreground it verbalizes plans and actions with foley sounds and visual annotations. The controlled study found the biggest gains in the agent-assisted spreadsheet task: higher scores and roughly half the errors of the text baselines, with no significan

Load-bearing premise

Sidekick's color warnings and auto-pause are gated by a vision-language model that judges whether the agent's action succeeded; the paper reports no accuracy, false-positive, or false-negative rate for that judge, so the whole benefit rests on that judgment being reliable enough to alert people at the right time.

Editorial extensions

If this is right

  • Delegating work to an agent improves total performance only when feedback is designed for peripheral awareness; poorly designed text displays can erase the benefit.
  • Color-coded error thresholds let users catch mistakes without reading logs or increasing monitoring time.
  • Spoken and visual resumption summaries reduce the cost of switching back to a long-running agent.
  • Real-time verbalization and action annotation increase perceived transparency without raising cognitive load.
  • The same communication layer could extend to multi-agent workspaces and post-hoc trajectory analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the color cues and auto-pause depend on the VLM error detector, Sidekick's measured benefit is bounded by that detector's accuracy; measuring its false-positive and false-negative rates would reveal how much of the effect is communication design versus error-detection quality.
  • The three-scope structure (ambient, resumption, foreground) is a testable generalization: any long-horizon autonomous task may benefit from translating raw telemetry this way, even when the underlying agent is far more capable.
  • Participants' split preferences (some found rich cues distracting) suggest that adaptive, user-tunable modality selection could amplify the gains; the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper identifies communication gaps in human-CUA interaction through a formative study and introduces Sidekick, a prototype that augments a CUA chat interface with ambient color/text/thumbnails in a peripheral display, multimodal spoken/visual summaries on resumption, and real-time verbal/visual/auditory feedback during foreground operation. The contribution is evaluated in a 30-participant within-subjects study with an arithmetic primary task and a CUA-assisted spreadsheet-filling secondary task, under four conditions: manual (MN), chat baseline (BL), peripheral text (PT), and Sidekick (SK). The reported results show that SK yields significantly higher spreadsheet scores and lower spreadsheet error counts than BL and PT, with no significant increase in monitoring time or task switches and no arithmetic-task disruption, and with higher perceived helpfulness and timeliness of intervention. The authors also present three illustrative application scenarios and discuss design implications for long-horizon human-agent collaboration.

Significance. The topic is timely and the work makes a genuine empirical contribution to CUA feedback design. Strengths include a real prototype running with a current CUA/VLM stack, a counterbalanced within-subjects design, linear mixed-effects models with baseline-ability covariates, Holm-corrected pairwise tests, and mixed-method triangulation. If the central attribution is valid, the paper offers concrete design guidance for peripheral awareness, resumption support, and foreground transparency. However, the central claim that the multimodal communication layer drives the observed improvement is currently weakened by an unaddressed control-policy confound: in SK, the CUA automatically pauses during summary playback, while in BL and PT it does not. This asymmetry alone could account for part or all of the error-count reduction. In addition, the accuracy of the VLM error detector that gates the color/audio-visual signals and the pause is not reported, so the reliability and generalizability of the mechanism remain unclear. The application scenarios are appropriately labeled as unvalidated. Overall, the empirical foundation is promising but the paper needs a substantial experimental or analytic

major comments (3)
  1. [§4.2, §5.3, §5.6.2] The SK condition conflates feedback modality with agent control policy. Section 4.2 states that "The CUA remains paused during summary playback," and Section 5.3 lists SUMMARY as exclusive to SK. In BL and PT, the CUA continues executing unless the user manually clicks PAUSE. This gives SK users a free, low-cost window to let the agent stop, review completed actions, and intervene before further errors accumulate; BL/PT users must first notice a problem and take an extra action to pause. The reported lower spreadsheet error count for SK (1.31 vs. 2.51 BL and 2.32 PT) could therefore be caused by the pause itself rather than by multimodal feedback. The equal monitoring-time result does not rule this out, because auto-pause time is not counted as user monitoring. The Section 5.1 note that Sidekick "does not perform actions on the computer, but only provides feedback" makes the asymmetry st
  2. [§4.1, §5.6] The VLM error detector that gates the color states and the summary content is not validated. Section 4.1 states that a VLM (gemini-2.5-flash) assesses whether the post-execution state matches the user's query or the CUA's current subgoal, and that colors change at thresholds x=3, y=6, z=8 "based on error patterns observed during prototype development." No accuracy, false-positive/false-negative rates, or annotated-sample evaluation are reported. If the detector frequently mislabels correct actions as errors (or vice versa), Sidekick could misdirect attention or fail to alert users, undermining the mechanism the study attributes the improvement to. At a minimum, report VLM classification accuracy on a sample of the study's spreadsheet-filling actions, and provide a sensitivity analysis of the thresholds. This is needed for reproducibility and for interpreting the causal path from feedback
  3. [§5.6.1 / Figure 5] The paper does not separate the effect of the peripheral display from the effect of the modality. In PT, the peripheral display presents only text summaries and (as described in §5.6.1) a static color, while SK adds dynamic color changes, thumbnails, speech, and summary replay. The comparison between PT and SK therefore bundles display content, color dynamics, and modality. If the auto-pause confound is repaired, it would still strengthen the paper to add a PT+ dynamic color/thumbnail condition or to analyze which SK components (color vs. speech vs. summary) contribute to the spreadsheet-score advantage, since the qualitative data indicate that color cues were the most frequently mentioned mechanism.
minor comments (3)
  1. [§5.6.2 / Figure 5 caption] The panel references are swapped: the text says "number of task switches (Figure 5g)" and "time spent monitoring the spreadsheet task (Figure 5f)", but the Figure 5 caption lists (f) as average number of task switches and (g) as average monitoring time per switch. Please correct the cross-references.
  2. [§4.4] There is an inconsistency in the VLM model names: Section 4.1 and the beginning of Section 4.4 refer to gemini-2.5-flash, while later in Section 4.4 the object detection VLM is called gemini-3.5-flash. Please confirm which model was used for open-vocabulary detection.
  3. [§5.1 / §7] The scoring weights and the 12-point penalty for incorrect spreadsheet entries are described as calibrated through pilot studies. The authors note in Limitations that incentives may influence strategies, but the report would benefit from a sentence stating whether the conclusions are robust to alternative penalty values (e.g., a smaller penalty that makes error monitoring less critical).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Sidekick's comparative evaluation uses externally scored outcomes; internal thresholds are design parameters, not fits to the reported results.

full rationale

The central claim (Abstract: 'Sidekick significantly improved multitasking performance with CUAs compared to baseline systems') is supported by a between-condition user study with externally scored outcome measures: total/spreadsheet/arithmetic scores, spreadsheet error counts, task switches, monitoring time, and NASA-TLX. None of these are defined in terms of Sidekick's internal VLM detector or color thresholds. Section 4.1 says thresholds were 'set x=3, y=6, and z=8 based on the error patterns observed during prototype development,' but this is a system-design choice rather than a parameter fit to the 30-participant outcomes, and the reported spreadsheet error counts come from the task's ground truth, not from the VLM. The formative study motivates design goals but does not numerically imply the study's results. The strongest issue is an internal-validity confound, not circularity: Section 4.2 states 'The CUA remains paused during summary playback, allowing users to review the completed actions before execution resumes,' and Section 5.3 lists 'SUMMARY' as 'exclusive to SK,' so SK bundles an auto-pause control policy with its feedback channel. No per-session counts of summary plays or auto-pauses are reported, so the magnitude of this confound is unknown; this is a potentially serious correctness/validity risk (the error reduction may be partly caused by the pause rather than the multimodal channel), but it does not reduce the conclusion to its inputs by construction. Self-citation exists (e.g., ref [75], which includes co-author D. Li), but it is used only as background for high-stakes CUA intervention needs and is not load-bearing evidence for Sidekick's effectiveness. The unreported VLM accuracy (Section 4.1) is also a robustness limitation, not circularity. Application scenarios are explicitly acknowledged as not systematically evaluated (Section 6).

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The paper's central claim rests on calibrated design parameters (error thresholds, score weights) and empirical assumptions about CUA reliability and VLM accuracy. No new physical entities are introduced; Sidekick is a software artifact.

free parameters (4)
  • Error-threshold for yellow state (x) = 3
    In Section 4.1, the number of consecutive errors e triggers yellow when e > x, with x set to 3 based on error patterns during prototype development.
  • Error-threshold for orange state (y) = 6
    Set to 6 during development to balance alerting and false alarms.
  • Error-threshold for red state (z) = 8
    Set to 8, at which the CUA is automatically paused.
  • Spreadsheet scoring weights (correct/incorrect reward-penalty) = 3 / -12 / 0
    In Section 5.1, correct entries earn 3 points, incorrect incur a 12-point penalty, and blanks are 0; these values were 'carefully calibrated' via pilot studies to encourage error monitoring.
assumptions (3)
  • domain assumption CUAs are imperfect and make errors (e.g., about 60% accuracy on basic tasks)
    Motivates the need for error awareness; cited from OSWorld and other sources in Section 3.
  • domain assumption The vision-language model used for error detection is accurate enough to produce useful color cues
    No accuracy metrics are provided; this underlies the color and summary features in Section 4.1.
  • domain assumption The spreadsheet-filling task is representative of CUA multitasking
    Study uses this controlled task due to CUA reliability limits; the authors acknowledge limited generalizability in Section 7.
invented entities (1)
  • Sidekick prototype independent evidence
    purpose: A multimodal feedback system for CUAs that communicates status via ambient cues, summaries, and speech/visualization
    It is an engineered artifact, not a physically new entity; included for completeness but not analogous to the schema's 'graviton' category.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents." pith.science (2026). https://pith.science/paper/26RI3EFB

@misc{pith2026260717527,
  author       = {Pith},
  title        = {Pith review of: Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26RI3EFB}},
  note         = {Machine review of arXiv:2607.17527}
}
read the original abstract

Computer Use Agents (CUAs) can autonomously execute complex, multi-step tasks within GUIs, enhancing efficiency through parallel multitasking. However, our formative studies with CUA experts and GenAI users indicated that current feedback is primarily text-based, requiring sustained attention to monitor progress and offering limited visibility to trace past GUI interactions. Based on the findings, we developed a prototype system, Sidekick, for communicating CUAs' status with multimodal feedback across different stages of interaction: (i) When CUAs run in the background, Sidekick signals its execution state through ambient cues. (ii) Upon resuming interaction with CUAs, Sidekick provides multimodal summaries of completed actions to support rapid context resumption. (iii) When CUAs operate in the foreground, Sidekick enhances transparency by verbalizing and visualizing the agent's reasoning. A study with 30 participants demonstrated that Sidekick significantly improved multitasking performance with CUAs compared to baseline systems that presented textual feedback either in a typical chat or in an ambient display. Sidekick supported progress awareness, and error and action traceability more effectively. Finally, we demonstrate the promise of Sidekick through several example applications, and discuss implications for long-horizon human-agent collaboration.

Figures

Figures reproduced from arXiv: 2607.17527 by the authors.

Figure 1
Figure 1. Sidekick supports effective multitasking with computer-use agents (CUAs) by bridging communication gaps across [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Sidekick’s multimodal feedback across different stages of user interactions with CUAs. (a) When the CUA runs in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Study setup in a monitor. (a) Participants interacted [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Experimental procedure. Participants completed [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Results across conditions (MN, BL, PT, and SK) for task performance and interaction metrics. Panel (a) presents results from analyses treating MN as a fixed-effect condition alongside BL, PT, and SK. The remaining panels present results from analyses treating MN as a c…
Figure 6
Figure 6. Figure 6: NASA-TLX responses across conditions (MN, BL, PT, and SK) for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Results across conditions (BL, PT, and SK) for five Likert-scale measures (1–5): confidence in completing both tasks [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Sidekick’s potential application in slide editing. (a) [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 10
Figure 10. Figure 10: Sidekick’s potential application in collaboration [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

96 extracted references · 1 canonical work pages

  1. [1]

    Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

    2024. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. https://www.anthropic.com/news/3-5-models-and-computer-use

    2024

  2. [2]

    Anthropic

    2026. Anthropic. https://www.anthropic.com/

    2026

  3. [3]

    Claude Code

    2026. Claude Code. https://claude.com/product/claude-code

    2026

  4. [4]

    Claude Cowork by Anthropic

    2026. Claude Cowork by Anthropic. https://www.anthropic.com/product/claude- cowork

    2026

  5. [5]

    Computer use

    2026. Computer use. https://platform.openai.com/docs/guides/tools-computer- use

    2026

  6. [6]

    2026. Cua. https://cua.ai/

    2026

  7. [7]

    2026. Cursor. https://cursor.com/

    2026

  8. [8]

    Developing a computer use model

    2026. Developing a computer use model. https://www.anthropic.com/news/ developing-computer-use

    2026

Show all 96 references
  1. [9]

    Foley (sound design)

    2026. Foley (sound design). https://en.wikipedia.org/wiki/Foley_(sound_design)

    2026

  2. [10]

    Google Text-to-Speech AI

    2026. Google Text-to-Speech AI. https://cloud.google.com/text-to-speech

    2026

  3. [11]

    Introducing Docent

    2026. Introducing Docent. https://transluce.org/introducing-docent

    2026

  4. [12]

    Introducing Operator

    2026. Introducing Operator. https://openai.com/index/introducing-operator/

    2026

  5. [13]

    Introducing the Gemini 2.5 Computer Use model

    2026. Introducing the Gemini 2.5 Computer Use model. https://blog.google/ technology/google-deepmind/gemini-computer-use-model/ UIST ’26, November 02–05, 2026, Detroit, MI, USA Ruei-Che Chang, Wenqian Xu, Dingzeyu Li, Bryan Wang, and Anhong Guo

    2026

  6. [14]

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

    2026. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. https://os-world.github.io/

    2026

  7. [15]

    Speech Synthesis Markup Language (SSML)

    2026. Speech Synthesis Markup Language (SSML). https://docs.cloud.google. com/text-to-speech/docs/ssml

    2026

  8. [16]

    2026. Vercept. https://vercept.com/

    2026

  9. [17]

    What is Lume? https://cua.ai/docs/lume/guide/getting-started/ introduction

    2026. What is Lume? https://cua.ai/docs/lume/guide/getting-started/ introduction

    2026

  10. [18]

    Reyna Abhyankar, Qi Qi, and Yiying Zhang. 2025. Osworld-human: Bench- marking the efficiency of computer-use agents.arXiv preprint arXiv:2506.16042 (2025)

    2025 arXiv

  11. [19]

    Adamczyk and Brian P

    Piotr D. Adamczyk and Brian P. Bailey. 2004. If not now, when? the effects of interruption at different moments within task execution. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Vienna, Austria) (CHI ’04). Association for Computing Machinery, ...

    2004

  12. [20]

    Drucker, Bongshin Lee, Patrice Simard, and Jina Suh

    Saleema Amershi, Max Chickering, Steven M. Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. ModelTracker: Redesigning Performance Analysis Tools for Machine Learning. InProceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(Seoul, Republi...

    2015

  13. [21]

    Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human- AI Interaction. InProceedings of the 2019 CHI Conferen...

    2019

  14. [22]

    Ernesto Arroyo, Ted Selker, and Alexandre Stouffs. 2002. Interruptions as mul- timodal outputs: Which are the less disruptive?. InProceedings. Fourth IEEE International Conference on Multimodal Interfaces. IEEE, 479–482

    2002

  15. [24]

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al

  16. [25]

    Reiss, Suman Karumuri, William Cheung, Joshua Kaplan, Christopher Coleman, Ferdi Adeputra, and Jr

    Andrew Bragdon, Robert Zeleznik, Steven P. Reiss, Suman Karumuri, William Cheung, Joshua Kaplan, Christopher Coleman, Ferdi Adeputra, and Jr. LaViola, Joseph J. 2010. Code bubbles: a working set-based interface for code understand- ing and maintenance. InProceedings of the SIG...

    2010

  17. [26]

    Brewster, Peter C

    Stephen A. Brewster, Peter C. Wright, and Alistair D. N. Edwards. 1994. The design and evaluation of an auditory-enhanced scrollbar. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Boston, Massachusetts, USA) (CHI ’94). Association for Computing Ma...

    1994

  18. [27]

    Valdimar Briem and Leif R Hedman. 1995. Behavioural effects of mobile telephone use during simulated driving.Ergonomics38, 12 (1995), 2536–2562

    1995

  19. [28]

    J. J. Cadiz, Gina Venolia, Gavin Jancke, and Anoop Gupta. 2002. Designing and deploying an information awareness interface. InProceedings of the 2002 ACM Conference on Computer Supported Cooperative Work(New Orleans, Louisiana, USA)(CSCW ’02). Association for Computing Machine...

    2002

  20. [29]

    Joseph Chee Chang, Yongsung Kim, Victor Miller, Michael Xieyang Liu, Brad A Myers, and Aniket Kittur. 2021. Tabs.do: Task-Centric Browser Tab Management. InThe 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA)(UIST ’21). Association for C...

    2021

  21. [31]

    Ruei-Che Chang, Chia-Sheng Hung, Bing-Yu Chen, Dhruv Jain, and Anhong Guo

  22. [33]

    InProceedings of the 2024 ACM Designing Interactive Systems Confer- ence(Copenhagen, Denmark)(DIS ’24)

    SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness. InProceedings of the 2024 ACM Designing Interactive Systems Confer- ence(Copenhagen, Denmark)(DIS ’24). Association for Computing Machinery, New York, NY, USA, 116–132. doi:10.1145/3643834.3661556

    2024

  23. [34]

    Ruei-Che Chang, Rosiana Natalie, Wenqian Xu, Jovan Zheng Feng Yap, Tiange Luo, Venkatesh Potluri, and Anhong Guo. 2026. TouchScribe: Augmenting Non- Visual Hand-Object Interactions with Automated Live Visual Descriptions. In Proceedings of the 2026 CHI Conference on Human Fact...

    2026

  24. [35]

    Ruei-Che Chang, Yuxuan Liu, Lotus Zhang, and Anhong Guo. 2024. EditScribe: Non-Visual Image Editing with Natural Language Verification Loops. InPro- ceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility(St. John’s, NL, Canada)(ASSETS ’24). ...

    2024

  25. [36]

    Kuan-Wen Chen, Yung-Ju Chang, and Liwei Chan. 2022. Predicting Opportune Moments to Deliver Notifications in Virtual Reality. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems(New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, Ne...

    2022

  26. [37]

    Jessie YC Chen, Shan G Lakhmani, Kimberly Stowers, Anthony R Selkowitz, Julia L Wright, and Michael Barnes. 2018. Situation awareness-based agent transparency and human-autonomy teaming effectiveness.Theoretical issues in ergonomics science19, 3 (2018), 259–282

    2018

  27. [38]

    Heller, and David Lindlbauer

    Yi Fei Cheng, Jarod Bloch, Alexander Wang, Andrea Bianchi, Anusha Withana, Anhong Guo, Laurie M. Heller, and David Lindlbauer. 2026. Auditorily Embodied Conversational Agents: Effects of Spatialization and Situated Audio Cues on Presence and Social Perception. InProceedings of...

    2026 doi

  28. [39]

    Ruijia Cheng, Jenny T Liang, Eldon Schoop, and Jeffrey Nichols. 2026. Mapping the Design Space of User Experience for Computer Use Agents. InProceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26). ACM, 646–662. doi:10.1145/3742413.3789132

    2026

  29. [40]

    Dennis Collaris and Jarke J Van Wijk. 2020. ExplainExplore: Visual exploration of machine learning explanations. In2020 IEEE Pacific Visualization Symposium (PacificVis). IEEE, 26–35

    2020

  30. [41]

    Yi Fei Cheng, Hirokazu Shirado, and Shunichi Kasahara. 2025. Conversational Agents on Your Behalf: Opportunities and Challenges of Shared Autonomy in Voice Communication for Multitasking. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25)....

    2025 doi

  31. [42]

    Mary Czerwinski, Edward Cutrell, and Eric Horvitz. 2000. Instant messaging and interruption: Influence of task type on performance. InOZCHI 2000 conference proceedings, Vol. 356. 361–367

    2000

  32. [43]

    Sunny Consolvo and Jeffrey Towle. 2005. Evaluating an ambient display for the home. InCHI ’05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA)(CHI EA ’05). Association for Computing Machinery, New York, NY, USA, 1304–1307. doi:10.1145/1056808.1056902

    2005

  33. [44]

    Anindya Das Antar, Somayeh Molaei, Yan-Ying Chen, Matthew L Lee, and Nikola Banovic. 2024. VIME: Visual Interactive Model Explorer for Identifying Capabili- ties and Limitations of Machine Learning Models for Sequential Decision-Making. InProceedings of the 37th Annual ACM Sym...

    2024

  34. [46]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems36 (2023), 28091–28114

    2023

  35. [47]

    De Guzman, Margaret Yau, Anthony Gagliano, Austin Park, and Anind K

    Edward S. De Guzman, Margaret Yau, Anthony Gagliano, Austin Park, and Anind K. Dey. 2004. Exploring the design and use of peripheral displays of aware- ness information. InCHI ’04 Extended Abstracts on Human Factors in Computing Systems(Vienna, Austria)(CHI EA ’04). Associatio...

    2004

  36. [48]

    Jie Gao, Kaiser Sun, Jen tse Huang, Katherine Van Koevering, Sijie Ji, Heyuan Huang, Weiyan Shi, Zhuoran Lu, Ziang Xiao, Daniel Khashabi, and Mark Dredze

  37. [49]

    Dragunov, Thomas G

    Anton N. Dragunov, Thomas G. Dietterich, Kevin Johnsrude, Matthew McLaugh- lin, Lida Li, and Jonathan L. Herlocker. 2005. TaskTracer: a desktop environment to support multi-tasking knowledge workers. InProceedings of the 10th Inter- national Conference on Intelligent User Inte...

    2005

  38. [50]

    Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2010. Chronicle: capture, exploration, and playback of document workflow histories. InProceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology(New York, New York, USA)(UIST ’10). Association f...

    2010

  39. [51]

    Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908

    2006

  40. [52]

    Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, et al. 2025. Mind2web 2: Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents UIST ’26, November 02–05, 2026, ...

    2025 arXiv

  41. [53]

    Fred Hohman, Arjun Srinivasan, and Steven M Drucker. 2019. TeleGam: Combin- ing visualization and verbalization for interpretable machine learning. In2019 ieee visualization conference (VIS). IEEE, 151–155

    2019

  42. [54]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Pittsburgh, Pennsylvania, USA)(CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030

    1999

  43. [55]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...

    2024

  44. [56]

    Chilana, and Tovi Grossman

    Nathaniel Hudson, Benjamin Lafreniere, Parmit K. Chilana, and Tovi Grossman

  45. [57]

    Iqbal and Brian P

    Shamsi T. Iqbal and Brian P. Bailey. 2008. Effects of intelligent notification management on users and their tasks. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Florence, Italy)(CHI ’08). Association for Computing Machinery, New York, NY, USA, 9...

    2008

  46. [58]

    Eric Horvitz, Paul Koch, and Johnson Apacible. 2004. BusyBody: creating and fielding personalized models of the cost of interruption. InProceedings of the 2004 ACM Conference on Computer Supported Cooperative Work(Chicago, Illinois, USA)(CSCW ’04). Association for Computing Ma...

    2004

  47. [59]

    Peiling Jiang and Haijun Xia. 2025. Orca: Browsing at scale through user- driven and ai-facilitated orchestration across malleable webpages.arXiv preprint arXiv:2505.22831(2025)

    2025

  48. [60]

    Eser Kandogan and Ben Shneiderman. 1997. Elastic Windows: evaluation of multi- window operations. InProceedings of the ACM SIGCHI Conference on Human factors in computing systems. 250–257

    1997

  49. [61]

    Wonjoong Kim, Sangwu Park, Yeonjun In, Sein Kim, Dongha Lee, and Chanyoung Park. 2026. Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents. arXiv:2510.02837 [cs.AI] https://arxiv.org/abs/2510. 02837

    2026 arXiv

  50. [62]

    Iqbal and Brian P

    Shamsi T. Iqbal and Brian P. Bailey. 2011. Oasis: A framework for link- ing notification delivery to the perceptual structure of goal-directed tasks. ACM Trans. Comput.-Hum. Interact.17, 4, Article 15 (Dec. 2011), 28 pages. doi:10.1145/1879831.1879833

    2011

  51. [63]

    Ben Lafreniere, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2014. Investigating the feasibility of extracting tool demonstrations from in-situ video content. InProceedings of the SIGCHI Conference on Human Factors in Comput- ing Systems(Toronto, Ontario, Canada)(CHI...

    2014

  52. [64]

    Wei Li, Tovi Grossman, and George Fitzmaurice. 2012. GamiCAD: a gamified tutorial system for first time autocad users. InProceedings of the 25th Annual ACM Symposium on User Interface Software and Technology(Cambridge, Massachusetts, USA)(UIST ’12). Association for Computing M...

    2012

  53. [65]

    Mary J Lindstrom and Douglas M Bates. 1988. Newton—Raphson and EM al- gorithms for linear mixed-effects models for repeated-measures data.J. Amer. Statist. Assoc.83, 404 (1988), 1014–1022

    1988

  54. [66]

    Benjamin Lafreniere, Tovi Grossman, and George Fitzmaurice. 2013. Commu- nity enhanced tutorials: improving tutorials with multiple demonstrations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1779–1788

    2013

  55. [67]

    Maglio and Christopher S

    Paul P. Maglio and Christopher S. Campbell. 2000. Tradeoffs in displaying peripheral information. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(The Hague, The Netherlands)(CHI ’00). Association for Computing Machinery, New York, NY, USA, 241–248....

    2000

  56. [68]

    Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2011. Ambient help. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada)(CHI ’11). Association for Computing Machinery, New York, NY, USA, 2751–2760. doi:10.1145/1978942.1979349

    2011

  57. [69]

    Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2021. MeetingMate: an Ambient Interface for Improved Meeting Effectiveness and Corporate Knowl- edge Sharing. InGraphics Interface 2021. https://openreview.net/forum?id= ibaCpFUWVb9

    2021

  58. [70]

    Mynatt, Stephen Voida, Klaus M

    Blair MacIntyre, Elizabeth D. Mynatt, Stephen Voida, Klaus M. Hansen, Joe Tullio, and Gregory M. Corso. 2001. Support for multitasking and background awareness using interactive peripheral displays. InProceedings of the 14th Annual ACM Symposium on User Interface Software and ...

    2001

  59. [71]

    Daniel C McFarlane and Kara A Latorella. 2002. The scope and importance of human interruption in human-computer interaction design.Human-Computer Interaction17, 1 (2002), 1–61

    2002

  60. [72]

    Ananya Gubbi Mohanbabu, Rosiana Natalie, Brandon Kim, Anhong Guo, and Amy Pavel. 2026. A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents.arXiv preprint arXiv:2602.09310(2026)

    2026

  61. [73]

    Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. InProceedings of the SIGCHI conference on Human factors in computing systems. 249–256

    1990

  62. [74]

    Justin Matejka, Wei Li, Tovi Grossman, and George Fitzmaurice. 2009. Com- munityCommands: command recommendations for software applications. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 193–202

    2009

  63. [75]

    Yi-Hao Peng, Dingzeyu Li, Jeffrey P Bigham, and Amy Pavel. 2025. Morae: Proactively Pausing UI Agents for User Choices. InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). Association for Computing Machinery, New York, NY, USA, ...

    2025

  64. [76]

    Zachary Pousman and John Stasko. 2006. A taxonomy of ambient information systems: four patterns of design. InProceedings of the Working Conference on Advanced Visual Interfaces(Venezia, Italy)(A VI ’06). Association for Computing Machinery, New York, NY, USA, 67–74. doi:10.114...

    2006

  65. [77]

    Why Should I Trust You?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining(San Francisco, California, USA)(KDD ’16)....

    2016

  66. [78]

    Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, and Xiang Yue. 2025. AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Ivan Habernal, Pet...

    2025 doi

  67. [79]

    Pascal J Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F Grewe, and Thilo Stadelmann. 2026. A comprehensive survey of agents for computer use: Foundations, challenges, and future directions....

    2026

  68. [80]

    Nitin Sawhney and Chris Schmandt. 2000. Nomadic radio: speech and audio interaction for contextual messaging in nomadic environments.ACM transactions on Computer-Human interaction (TOCHI)7, 3 (2000), 353–383

    2000

  69. [81]

    Anthony R Selkowitz, Shan G Lakhmani, and Jessie YC Chen. 2017. Using agent transparency to support situation awareness of the Autonomous Squad Member. Cognitive Systems Research46 (2017), 13–25

    2017

  70. [82]

    Joshua S Rubinstein, David E Meyer, and Jeffrey E Evans. 2001. Executive control of cognitive processes in task switching.Journal of experimental psychology: human perception and performance27, 4 (2001), 763

    2001

  71. [83]

    Bigham, and Jeffrey Nichols

    Amanda Swearngin, Jason Wu, Xiaoyi Zhang, Esteban Gomez, Jen Coughenour, Rachel Stukenborg, Bhavya Garg, Greg Hughes, Adriana Hilliard, Jeffrey P. Bigham, and Jeffrey Nichols. 2024. Towards Automated Accessibility Report Generation for Mobile Apps.ACM Trans. Comput.-Hum. Inter...

    2024 doi

  72. [84]

    Maxwell Szymanski, Martijn Millecamp, and Katrien Verbert. 2021. Visual, textual or hybrid: the effect of user expertise on different explanations. InProceedings of the 26th International Conference on Intelligent User Interfaces(College Station, TX, USA)(IUI ’21). Association...

    2021

  73. [85]

    Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2024. AXNav: Replaying Accessibility Tests from Natural Lan- guage. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Associatio...

    2024

  74. [86]

    Jacob Somervell, CM Chewar, and D Scott McCrickard. 2002. Evaluating graphical vs. textual secondary displays for information notification. InProceedings of the ACM Southeast Conference, Raleigh NC. 153–160

    2002

  75. [87]

    Koen van de Merwe, Steven Mallam, and Salman Nazir. 2024. Agent transparency, situation awareness, mental workload, and operator performance: A systematic UIST ’26, November 02–05, 2026, Detroit, MI, USA Ruei-Che Chang, Wenqian Xu, Dingzeyu Li, Bryan Wang, and Anhong Guo liter...

    2024

  76. [88]

    Michael Vössing, Niklas Kühl, Matteo Lind, and Gerhard Satzger. 2022. Designing transparency for effective human-AI collaboration.Information Systems Frontiers 24, 3 (2022), 877–895

    2022

  77. [89]

    Bryan Wang, Zeyu Jin, and Gautham Mysore. 2022. Record Once, Post Every- where: Automatic Shortening of Audio Stories for Social Media. InProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA)(UIST ’22). Association for Computing...

    2022

  78. [90]

    Tan, Brian Meyers, and Mary Czerwinski

    Desney S. Tan, Brian Meyers, and Mary Czerwinski. 2004. WinCuts: manipulating arbitrary window regions for more effective use of screen space. InCHI ’04 Extended Abstracts on Human Factors in Computing Systems(Vienna, Austria)(CHI EA ’04). Association for Computing Machinery, ...

    2004

  79. [91]

    Zijie J Wang, Chudi Zhong, Rui Xin, Takuya Takagi, Zhi Chen, Duen Horng Chau, Cynthia Rudin, and Margo Seltzer. 2022. TimberTrek: exploring and curating sparse decision trees with interactive visualization. In2022 IEEE visualization and visual analytics (VIS). IEEE, 60–64

    2022

  80. [92]

    James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viégas, and Jimbo Wilson. 2019. The what-if tool: Interactive probing of machine learning models.IEEE transactions on visualization and computer graphics26, 1 (2019), 56–65

    2019

  81. [93]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al . 2024. Os- world: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Proc...

    2024

  82. [94]

    Xu Wang, Benjamin Lafreniere, and Tovi Grossman. 2018. Leveraging community- generated videos and command logs to classify and recommend software work- flows. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13

    2018

  83. [95]

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614(2024)

    2024 arXiv

  84. [96]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854(2023). Sidekick: Designing Communic...

    2023 arXiv

  85. [98]

    Yuhao Yang, Zhen Yang, Zi-Yi Dou, Anh Nguyen, Keen You, Omar Attia, Andrew Szot, Michael Feng, Ram Ramrakhya, Alexander Toshev, et al. 2025. Ultracua: A foundation model for computer use agents with hybrid action.arXiv preprint arXiv:2510.17790(2025)

    2025 arXiv

  86. [2018]

    InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada)(CHI ’18)

    Investigating How Online Help and Learning Resources Support Children’s Use of 3D Design Software. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada)(CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–14. doi:10...

    2018

  87. [2024]

    Windows agent arena: Evaluating multi-modal os agents at scale.arXiv preprint arXiv:2409.08264(2024)

    2024 arXiv

  88. [2026]

    arXiv:2605.13625 [cs.AI] https://arxiv

    How to Interpret Agent Behavior. arXiv:2605.13625 [cs.AI] https://arxiv. org/abs/2605.13625

    arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.