Pith. sign in

REVIEW 3 major objections 3 minor 96 references

Sidekick shows that a multimodal feedback layer can lift multitasking performance with computer-use agents beyond text-only interfaces.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-01 17:42 UTC pith:26RI3EFB

load-bearing objection Well-run study of multimodal feedback for CUA multitasking, but the bundled auto-pause undercuts the causal claim that the feedback modality is what helps. the 3 major comments →

arxiv 2607.17527 v1 pith:26RI3EFB submitted 2026-07-20 cs.HC

Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents

Ruei-Che Chang , Wenqian Xu , Dingzeyu Li , Bryan Wang , Anhong Guo This is my paper
classification cs.HC
keywords computer use agentsmultimodal feedbackambient displaymultitaskinghuman-agent collaborationcontext resumptionerror detectionprototype system
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the bottleneck in human–computer-use-agent collaboration is not agent capability but communication design. It introduces Sidekick, a prototype feedback layer that adapts to three phases: colored ambient cues when the agent runs in the background, spoken and visual summaries when the user returns, and real-time verbalized reasoning with on-screen annotations when the agent works in the foreground. In a 30-participant study with arithmetic as the primary task and spreadsheet filling delegated to an agent, Sidekick produced a higher spreadsheet score and fewer errors than a chat-only baseline or a peripheral-text baseline, without increasing the number of task switches or monitoring time, and without reducing arithmetic performance. The authors take this as evidence that feedback supporting peripheral awareness, rapid context resumption, and transparency is what lets users delegate and intervene at the right moments.

Core claim

The central claim is that multimodal, stage-adapted communication improves multitasking with computer-use agents. Sidekick condenses raw agent messages into one-sentence summaries, shows thumbnails, and uses color (green/yellow/orange/red) driven by a vision-language-model error detector that counts consecutive failed actions and pauses the agent on red. On returning, it presents replay-like summaries with speech synced to screenshots. In the foreground it verbalizes plans and actions with foley sounds and visual annotations. The controlled study found the biggest gains in the agent-assisted spreadsheet task: higher scores and roughly half the errors of the text baselines, with no significan

What carries the argument

The carry-the-argument object is the stage-based feedback pipeline: (1) a small always-on-top 'Sidekick display' mapping the agent's error count to an ambient color, (2) a VLM-generated multimodal resumption summary that uses SSML timestamps to sync TTS narration with screenshots, and (3) real-time verbalization and bounding-box annotations of agent reasoning in the foreground. The error detector — a VLM comparing before-and-after screenshots to the subgoal — is the gate that sets the color thresholds and the auto-pause on red; it determines when users are alerted. The study's arithmetic task acts as a continuous distractor to measure whether the feedback steals attention.

Load-bearing premise

Sidekick's color warnings and auto-pause are gated by a vision-language model that judges whether the agent's action succeeded; the paper reports no accuracy, false-positive, or false-negative rate for that judge, so the whole benefit rests on that judgment being reliable enough to alert people at the right time.

What would settle it

Run the same 30-participant study with the VLM error detector replaced by perfect ground-truth labels, or with the detector's judgments deliberately corrupted (e.g., random noise), and compare spreadsheet error counts. If corrupting detector reliability does not change Sidekick's advantage over the text baselines, the central claim about the communication layer collapses; if it does change, the mechanism is error detection, not multimodality. A simpler check: log the detector's decisions against the two injected errors per column to see whether it actually fires on real mistakes and stays sile

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Delegating work to an agent improves total performance only when feedback is designed for peripheral awareness; poorly designed text displays can erase the benefit.
  • Color-coded error thresholds let users catch mistakes without reading logs or increasing monitoring time.
  • Spoken and visual resumption summaries reduce the cost of switching back to a long-running agent.
  • Real-time verbalization and action annotation increase perceived transparency without raising cognitive load.
  • The same communication layer could extend to multi-agent workspaces and post-hoc trajectory analysis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the color cues and auto-pause depend on the VLM error detector, Sidekick's measured benefit is bounded by that detector's accuracy; measuring its false-positive and false-negative rates would reveal how much of the effect is communication design versus error-detection quality.
  • The three-scope structure (ambient, resumption, foreground) is a testable generalization: any long-horizon autonomous task may benefit from translating raw telemetry this way, even when the underlying agent is far more capable.
  • Participants' split preferences (some found rich cues distracting) suggest that adaptive, user-tunable modality selection could amplify the gains; the paper does not test this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper identifies communication gaps in human-CUA interaction through a formative study and introduces Sidekick, a prototype that augments a CUA chat interface with ambient color/text/thumbnails in a peripheral display, multimodal spoken/visual summaries on resumption, and real-time verbal/visual/auditory feedback during foreground operation. The contribution is evaluated in a 30-participant within-subjects study with an arithmetic primary task and a CUA-assisted spreadsheet-filling secondary task, under four conditions: manual (MN), chat baseline (BL), peripheral text (PT), and Sidekick (SK). The reported results show that SK yields significantly higher spreadsheet scores and lower spreadsheet error counts than BL and PT, with no significant increase in monitoring time or task switches and no arithmetic-task disruption, and with higher perceived helpfulness and timeliness of intervention. The authors also present three illustrative application scenarios and discuss design implications for long-horizon human-agent collaboration.

Significance. The topic is timely and the work makes a genuine empirical contribution to CUA feedback design. Strengths include a real prototype running with a current CUA/VLM stack, a counterbalanced within-subjects design, linear mixed-effects models with baseline-ability covariates, Holm-corrected pairwise tests, and mixed-method triangulation. If the central attribution is valid, the paper offers concrete design guidance for peripheral awareness, resumption support, and foreground transparency. However, the central claim that the multimodal communication layer drives the observed improvement is currently weakened by an unaddressed control-policy confound: in SK, the CUA automatically pauses during summary playback, while in BL and PT it does not. This asymmetry alone could account for part or all of the error-count reduction. In addition, the accuracy of the VLM error detector that gates the color/audio-visual signals and the pause is not reported, so the reliability and generalizability of the mechanism remain unclear. The application scenarios are appropriately labeled as unvalidated. Overall, the empirical foundation is promising but the paper needs a substantial experimental or analytic

major comments (3)
  1. [§4.2, §5.3, §5.6.2] The SK condition conflates feedback modality with agent control policy. Section 4.2 states that "The CUA remains paused during summary playback," and Section 5.3 lists SUMMARY as exclusive to SK. In BL and PT, the CUA continues executing unless the user manually clicks PAUSE. This gives SK users a free, low-cost window to let the agent stop, review completed actions, and intervene before further errors accumulate; BL/PT users must first notice a problem and take an extra action to pause. The reported lower spreadsheet error count for SK (1.31 vs. 2.51 BL and 2.32 PT) could therefore be caused by the pause itself rather than by multimodal feedback. The equal monitoring-time result does not rule this out, because auto-pause time is not counted as user monitoring. The Section 5.1 note that Sidekick "does not perform actions on the computer, but only provides feedback" makes the asymmetry st
  2. [§4.1, §5.6] The VLM error detector that gates the color states and the summary content is not validated. Section 4.1 states that a VLM (gemini-2.5-flash) assesses whether the post-execution state matches the user's query or the CUA's current subgoal, and that colors change at thresholds x=3, y=6, z=8 "based on error patterns observed during prototype development." No accuracy, false-positive/false-negative rates, or annotated-sample evaluation are reported. If the detector frequently mislabels correct actions as errors (or vice versa), Sidekick could misdirect attention or fail to alert users, undermining the mechanism the study attributes the improvement to. At a minimum, report VLM classification accuracy on a sample of the study's spreadsheet-filling actions, and provide a sensitivity analysis of the thresholds. This is needed for reproducibility and for interpreting the causal path from feedback
  3. [§5.6.1 / Figure 5] The paper does not separate the effect of the peripheral display from the effect of the modality. In PT, the peripheral display presents only text summaries and (as described in §5.6.1) a static color, while SK adds dynamic color changes, thumbnails, speech, and summary replay. The comparison between PT and SK therefore bundles display content, color dynamics, and modality. If the auto-pause confound is repaired, it would still strengthen the paper to add a PT+ dynamic color/thumbnail condition or to analyze which SK components (color vs. speech vs. summary) contribute to the spreadsheet-score advantage, since the qualitative data indicate that color cues were the most frequently mentioned mechanism.
minor comments (3)
  1. [§5.6.2 / Figure 5 caption] The panel references are swapped: the text says "number of task switches (Figure 5g)" and "time spent monitoring the spreadsheet task (Figure 5f)", but the Figure 5 caption lists (f) as average number of task switches and (g) as average monitoring time per switch. Please correct the cross-references.
  2. [§4.4] There is an inconsistency in the VLM model names: Section 4.1 and the beginning of Section 4.4 refer to gemini-2.5-flash, while later in Section 4.4 the object detection VLM is called gemini-3.5-flash. Please confirm which model was used for open-vocabulary detection.
  3. [§5.1 / §7] The scoring weights and the 12-point penalty for incorrect spreadsheet entries are described as calibrated through pilot studies. The authors note in Limitations that incentives may influence strategies, but the report would benefit from a sentence stating whether the conclusions are robust to alternative penalty values (e.g., a smaller penalty that makes error monitoring less critical).

Circularity Check

0 steps flagged

No significant circularity: Sidekick's comparative evaluation uses externally scored outcomes; internal thresholds are design parameters, not fits to the reported results.

full rationale

The central claim (Abstract: 'Sidekick significantly improved multitasking performance with CUAs compared to baseline systems') is supported by a between-condition user study with externally scored outcome measures: total/spreadsheet/arithmetic scores, spreadsheet error counts, task switches, monitoring time, and NASA-TLX. None of these are defined in terms of Sidekick's internal VLM detector or color thresholds. Section 4.1 says thresholds were 'set x=3, y=6, and z=8 based on the error patterns observed during prototype development,' but this is a system-design choice rather than a parameter fit to the 30-participant outcomes, and the reported spreadsheet error counts come from the task's ground truth, not from the VLM. The formative study motivates design goals but does not numerically imply the study's results. The strongest issue is an internal-validity confound, not circularity: Section 4.2 states 'The CUA remains paused during summary playback, allowing users to review the completed actions before execution resumes,' and Section 5.3 lists 'SUMMARY' as 'exclusive to SK,' so SK bundles an auto-pause control policy with its feedback channel. No per-session counts of summary plays or auto-pauses are reported, so the magnitude of this confound is unknown; this is a potentially serious correctness/validity risk (the error reduction may be partly caused by the pause rather than the multimodal channel), but it does not reduce the conclusion to its inputs by construction. Self-citation exists (e.g., ref [75], which includes co-author D. Li), but it is used only as background for high-stakes CUA intervention needs and is not load-bearing evidence for Sidekick's effectiveness. The unreported VLM accuracy (Section 4.1) is also a robustness limitation, not circularity. Application scenarios are explicitly acknowledged as not systematically evaluated (Section 6).

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 1 invented entities

The paper's central claim rests on calibrated design parameters (error thresholds, score weights) and empirical assumptions about CUA reliability and VLM accuracy. No new physical entities are introduced; Sidekick is a software artifact.

free parameters (4)
  • Error-threshold for yellow state (x) = 3
    In Section 4.1, the number of consecutive errors e triggers yellow when e > x, with x set to 3 based on error patterns during prototype development.
  • Error-threshold for orange state (y) = 6
    Set to 6 during development to balance alerting and false alarms.
  • Error-threshold for red state (z) = 8
    Set to 8, at which the CUA is automatically paused.
  • Spreadsheet scoring weights (correct/incorrect reward-penalty) = 3 / -12 / 0
    In Section 5.1, correct entries earn 3 points, incorrect incur a 12-point penalty, and blanks are 0; these values were 'carefully calibrated' via pilot studies to encourage error monitoring.
axioms (3)
  • domain assumption CUAs are imperfect and make errors (e.g., about 60% accuracy on basic tasks)
    Motivates the need for error awareness; cited from OSWorld and other sources in Section 3.
  • domain assumption The vision-language model used for error detection is accurate enough to produce useful color cues
    No accuracy metrics are provided; this underlies the color and summary features in Section 4.1.
  • domain assumption The spreadsheet-filling task is representative of CUA multitasking
    Study uses this controlled task due to CUA reliability limits; the authors acknowledge limited generalizability in Section 7.
invented entities (1)
  • Sidekick prototype independent evidence
    purpose: A multimodal feedback system for CUAs that communicates status via ambient cues, summaries, and speech/visualization
    It is an engineered artifact, not a physically new entity; included for completeness but not analogous to the schema's 'graviton' category.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents." pith.science (2026). https://pith.science/paper/26RI3EFB

@misc{pith2026260717527,
  author       = {Pith},
  title        = {Pith review of: Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26RI3EFB}},
  note         = {Machine review of arXiv:2607.17527}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Computer Use Agents (CUAs) can autonomously execute complex, multi-step tasks within GUIs, enhancing efficiency through parallel multitasking. However, our formative studies with CUA experts and GenAI users indicated that current feedback is primarily text-based, requiring sustained attention to monitor progress and offering limited visibility to trace past GUI interactions. Based on the findings, we developed a prototype system, Sidekick, for communicating CUAs' status with multimodal feedback across different stages of interaction: (i) When CUAs run in the background, Sidekick signals its execution state through ambient cues. (ii) Upon resuming interaction with CUAs, Sidekick provides multimodal summaries of completed actions to support rapid context resumption. (iii) When CUAs operate in the foreground, Sidekick enhances transparency by verbalizing and visualizing the agent's reasoning. A study with 30 participants demonstrated that Sidekick significantly improved multitasking performance with CUAs compared to baseline systems that presented textual feedback either in a typical chat or in an ambient display. Sidekick supported progress awareness, and error and action traceability more effectively. Finally, we demonstrate the promise of Sidekick through several example applications, and discuss implications for long-horizon human-agent collaboration.

Figures

Figures reproduced from arXiv: 2607.17527 by Anhong Guo, Bryan Wang, Dingzeyu Li, Ruei-Che Chang, Wenqian Xu.

Figure 1
Figure 1. Figure 1: Sidekick supports effective multitasking with computer-use agents (CUAs) by bridging communication gaps across [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Sidekick’s multimodal feedback across different stages of user interactions with CUAs. (a) When the CUA runs in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Study setup in a monitor. (a) Participants interacted [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Experimental procedure. Participants completed [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results across conditions (MN, BL, PT, and SK) for task performance and interaction metrics. Panel (a) presents results from analyses treating MN as a fixed-effect condition alongside BL, PT, and SK. The remaining panels present results from analyses treating MN as a covariate: (b) total score, (c) spreadsheet score, (d) arithmetic score, (e) average error count, (f) average number of task switches, and (g… view at source ↗
Figure 6
Figure 6. Figure 6: NASA-TLX responses across conditions (MN, BL, PT, and SK) for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Results across conditions (BL, PT, and SK) for five Likert-scale measures (1–5): confidence in completing both tasks [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Sidekick’s potential application in slide editing. (a) [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Sidekick’s potential application in collaboration [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

96 extracted references · 1 canonical work pages

  1. [1]

    Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

    2024. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. https://www.anthropic.com/news/3-5-models-and-computer-use

    2024

  2. [2]

    Anthropic

    2026. Anthropic. https://www.anthropic.com/

    2026

  3. [3]

    Claude Code

    2026. Claude Code. https://claude.com/product/claude-code

    2026

  4. [4]

    Claude Cowork by Anthropic

    2026. Claude Cowork by Anthropic. https://www.anthropic.com/product/claude- cowork

    2026

  5. [5]

    Computer use

    2026. Computer use. https://platform.openai.com/docs/guides/tools-computer- use

    2026

  6. [6]

    2026. Cua. https://cua.ai/

    2026

  7. [7]

    2026. Cursor. https://cursor.com/

    2026

  8. [8]

    Developing a computer use model

    2026. Developing a computer use model. https://www.anthropic.com/news/ developing-computer-use

    2026

  9. [9]

    Foley (sound design)

    2026. Foley (sound design). https://en.wikipedia.org/wiki/Foley_(sound_design)

    2026

  10. [10]

    Google Text-to-Speech AI

    2026. Google Text-to-Speech AI. https://cloud.google.com/text-to-speech

    2026

  11. [11]

    Introducing Docent

    2026. Introducing Docent. https://transluce.org/introducing-docent

    2026

  12. [12]

    Introducing Operator

    2026. Introducing Operator. https://openai.com/index/introducing-operator/

    2026

  13. [13]

    Introducing the Gemini 2.5 Computer Use model

    2026. Introducing the Gemini 2.5 Computer Use model. https://blog.google/ technology/google-deepmind/gemini-computer-use-model/ UIST ’26, November 02–05, 2026, Detroit, MI, USA Ruei-Che Chang, Wenqian Xu, Dingzeyu Li, Bryan Wang, and Anhong Guo

    2026

  14. [14]

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

    2026. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. https://os-world.github.io/

    2026

  15. [15]

    Speech Synthesis Markup Language (SSML)

    2026. Speech Synthesis Markup Language (SSML). https://docs.cloud.google. com/text-to-speech/docs/ssml

    2026

  16. [16]

    2026. Vercept. https://vercept.com/

    2026

  17. [17]

    What is Lume? https://cua.ai/docs/lume/guide/getting-started/ introduction

    2026. What is Lume? https://cua.ai/docs/lume/guide/getting-started/ introduction

    2026

  18. [18]

    Reyna Abhyankar, Qi Qi, and Yiying Zhang. 2025. Osworld-human: Bench- marking the efficiency of computer-use agents.arXiv preprint arXiv:2506.16042 (2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Adamczyk and Brian P

    Piotr D. Adamczyk and Brian P. Bailey. 2004. If not now, when? the effects of interruption at different moments within task execution. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Vienna, Austria) (CHI ’04). Association for Computing Machinery, New York, NY, USA, 271–278. doi:10.1145/985692.985727

    arXiv 2004

  20. [20]

    Drucker, Bongshin Lee, Patrice Simard, and Jina Suh

    Saleema Amershi, Max Chickering, Steven M. Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. ModelTracker: Redesigning Performance Analysis Tools for Machine Learning. InProceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(Seoul, Republic of Korea)(CHI ’15). Association for Computing Machinery, New York, NY, USA, 33...

    arXiv 2015

  21. [21]

    Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human- AI Interaction. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems(Glasgow, Scotland Uk)(CHI ’19). Associa...

    arXiv 2019

  22. [22]

    Ernesto Arroyo, Ted Selker, and Alexandre Stouffs. 2002. Interruptions as mul- timodal outputs: Which are the less disruptive?. InProceedings. Fourth IEEE International Conference on Multimodal Interfaces. IEEE, 479–482

    2002

  23. [24]

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al

  24. [25]

    Reiss, Suman Karumuri, William Cheung, Joshua Kaplan, Christopher Coleman, Ferdi Adeputra, and Jr

    Andrew Bragdon, Robert Zeleznik, Steven P. Reiss, Suman Karumuri, William Cheung, Joshua Kaplan, Christopher Coleman, Ferdi Adeputra, and Jr. LaViola, Joseph J. 2010. Code bubbles: a working set-based interface for code understand- ing and maintenance. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Atlanta, Georgia, USA)(CHI ...

    arXiv 2010

  25. [26]

    Brewster, Peter C

    Stephen A. Brewster, Peter C. Wright, and Alistair D. N. Edwards. 1994. The design and evaluation of an auditory-enhanced scrollbar. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Boston, Massachusetts, USA) (CHI ’94). Association for Computing Machinery, New York, NY, USA, 173–179. doi:10.1145/191666.191733

    arXiv 1994

  26. [27]

    Valdimar Briem and Leif R Hedman. 1995. Behavioural effects of mobile telephone use during simulated driving.Ergonomics38, 12 (1995), 2536–2562

    1995

  27. [28]

    J. J. Cadiz, Gina Venolia, Gavin Jancke, and Anoop Gupta. 2002. Designing and deploying an information awareness interface. InProceedings of the 2002 ACM Conference on Computer Supported Cooperative Work(New Orleans, Louisiana, USA)(CSCW ’02). Association for Computing Machinery, New York, NY, USA, 314–323. doi:10.1145/587078.587122

    arXiv 2002

  28. [29]

    Joseph Chee Chang, Yongsung Kim, Victor Miller, Michael Xieyang Liu, Brad A Myers, and Aniket Kittur. 2021. Tabs.do: Task-Centric Browser Tab Management. InThe 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA)(UIST ’21). Association for Computing Machinery, New York, NY, USA, 663–676. doi:10.1145/3472749.3474777

    arXiv 2021

  29. [31]

    Ruei-Che Chang, Chia-Sheng Hung, Bing-Yu Chen, Dhruv Jain, and Anhong Guo

  30. [33]

    InProceedings of the 2024 ACM Designing Interactive Systems Confer- ence(Copenhagen, Denmark)(DIS ’24)

    SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness. InProceedings of the 2024 ACM Designing Interactive Systems Confer- ence(Copenhagen, Denmark)(DIS ’24). Association for Computing Machinery, New York, NY, USA, 116–132. doi:10.1145/3643834.3661556

    arXiv 2024

  31. [34]

    Ruei-Che Chang, Rosiana Natalie, Wenqian Xu, Jovan Zheng Feng Yap, Tiange Luo, Venkatesh Potluri, and Anhong Guo. 2026. TouchScribe: Augmenting Non- Visual Hand-Object Interactions with Automated Live Visual Descriptions. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26). Association for Computing Machinery, New Yor...

    arXiv 2026

  32. [35]

    Ruei-Che Chang, Yuxuan Liu, Lotus Zhang, and Anhong Guo. 2024. EditScribe: Non-Visual Image Editing with Natural Language Verification Loops. InPro- ceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility(St. John’s, NL, Canada)(ASSETS ’24). Association for Computing Machinery, New York, NY, USA, Article 65, 19 pages. do...

    arXiv 2024

  33. [36]

    Kuan-Wen Chen, Yung-Ju Chang, and Liwei Chan. 2022. Predicting Opportune Moments to Deliver Notifications in Virtual Reality. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems(New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 186, 18 pages. doi:10.1145/3491102.3517529

    arXiv 2022

  34. [37]

    Jessie YC Chen, Shan G Lakhmani, Kimberly Stowers, Anthony R Selkowitz, Julia L Wright, and Michael Barnes. 2018. Situation awareness-based agent transparency and human-autonomy teaming effectiveness.Theoretical issues in ergonomics science19, 3 (2018), 259–282

    2018

  35. [38]

    Heller, and David Lindlbauer

    Yi Fei Cheng, Jarod Bloch, Alexander Wang, Andrea Bianchi, Anusha Withana, Anhong Guo, Laurie M. Heller, and David Lindlbauer. 2026. Auditorily Embodied Conversational Agents: Effects of Spatialization and Situated Audio Cues on Presence and Social Perception. InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26). Associ...

    doi:10.1145/3772318 2026

  36. [39]

    Ruijia Cheng, Jenny T Liang, Eldon Schoop, and Jeffrey Nichols. 2026. Mapping the Design Space of User Experience for Computer Use Agents. InProceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26). ACM, 646–662. doi:10.1145/3742413.3789132

    arXiv 2026

  37. [40]

    Dennis Collaris and Jarke J Van Wijk. 2020. ExplainExplore: Visual exploration of machine learning explanations. In2020 IEEE Pacific Visualization Symposium (PacificVis). IEEE, 26–35

    2020

  38. [41]

    Yi Fei Cheng, Hirokazu Shirado, and Shunichi Kasahara. 2025. Conversational Agents on Your Behalf: Opportunities and Challenges of Shared Autonomy in Voice Communication for Multitasking. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 160, 18 pages. ...

    doi:10.1145/3706598 2025

  39. [42]

    Mary Czerwinski, Edward Cutrell, and Eric Horvitz. 2000. Instant messaging and interruption: Influence of task type on performance. InOZCHI 2000 conference proceedings, Vol. 356. 361–367

    2000

  40. [43]

    Sunny Consolvo and Jeffrey Towle. 2005. Evaluating an ambient display for the home. InCHI ’05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA)(CHI EA ’05). Association for Computing Machinery, New York, NY, USA, 1304–1307. doi:10.1145/1056808.1056902

    arXiv 2005

  41. [44]

    Anindya Das Antar, Somayeh Molaei, Yan-Ying Chen, Matthew L Lee, and Nikola Banovic. 2024. VIME: Visual Interactive Model Explorer for Identifying Capabili- ties and Limitations of Machine Learning Models for Sequential Decision-Making. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology(Pittsburgh, PA, USA)(UIST ’24)....

    arXiv 2024

  42. [46]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems36 (2023), 28091–28114

    2023

  43. [47]

    De Guzman, Margaret Yau, Anthony Gagliano, Austin Park, and Anind K

    Edward S. De Guzman, Margaret Yau, Anthony Gagliano, Austin Park, and Anind K. Dey. 2004. Exploring the design and use of peripheral displays of aware- ness information. InCHI ’04 Extended Abstracts on Human Factors in Computing Systems(Vienna, Austria)(CHI EA ’04). Association for Computing Machinery, New York, NY, USA, 1247–1250. doi:10.1145/985921.986035

    arXiv 2004

  44. [48]

    Jie Gao, Kaiser Sun, Jen tse Huang, Katherine Van Koevering, Sijie Ji, Heyuan Huang, Weiyan Shi, Zhuoran Lu, Ziang Xiao, Daniel Khashabi, and Mark Dredze

  45. [49]

    Dragunov, Thomas G

    Anton N. Dragunov, Thomas G. Dietterich, Kevin Johnsrude, Matthew McLaugh- lin, Lida Li, and Jonathan L. Herlocker. 2005. TaskTracer: a desktop environment to support multi-tasking knowledge workers. InProceedings of the 10th Inter- national Conference on Intelligent User Interfaces(San Diego, California, USA) (IUI ’05). Association for Computing Machiner...

    arXiv 2005

  46. [50]

    Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2010. Chronicle: capture, exploration, and playback of document workflow histories. InProceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology(New York, New York, USA)(UIST ’10). Association for Computing Machinery, New York, NY, USA, 143–152. doi:10.1145/1866029.1866054

    arXiv 2010

  47. [51]

    Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908

    2006

  48. [52]

    Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, et al. 2025. Mind2web 2: Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents UIST ’26, November 02–05, 2026, Detroit, MI, USA Evaluating agentic search with agent-as-a-judge.arXiv preprint ...

    Pith/arXiv arXiv 2025

  49. [53]

    Fred Hohman, Arjun Srinivasan, and Steven M Drucker. 2019. TeleGam: Combin- ing visualization and verbalization for interpretable machine learning. In2019 ieee visualization conference (VIS). IEEE, 151–155

    2019

  50. [54]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Pittsburgh, Pennsylvania, USA)(CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030

    arXiv 1999

  51. [55]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6864–6890

    2024

  52. [56]

    Chilana, and Tovi Grossman

    Nathaniel Hudson, Benjamin Lafreniere, Parmit K. Chilana, and Tovi Grossman

  53. [57]

    Iqbal and Brian P

    Shamsi T. Iqbal and Brian P. Bailey. 2008. Effects of intelligent notification management on users and their tasks. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Florence, Italy)(CHI ’08). Association for Computing Machinery, New York, NY, USA, 93–102. doi:10.1145/1357054.1357070

    arXiv 2008

  54. [58]

    Eric Horvitz, Paul Koch, and Johnson Apacible. 2004. BusyBody: creating and fielding personalized models of the cost of interruption. InProceedings of the 2004 ACM Conference on Computer Supported Cooperative Work(Chicago, Illinois, USA)(CSCW ’04). Association for Computing Machinery, New York, NY, USA, 507–510. doi:10.1145/1031607.1031690

    arXiv 2004

  55. [59]

    Peiling Jiang and Haijun Xia. 2025. Orca: Browsing at scale through user- driven and ai-facilitated orchestration across malleable webpages.arXiv preprint arXiv:2505.22831(2025)

    arXiv 2025

  56. [60]

    Eser Kandogan and Ben Shneiderman. 1997. Elastic Windows: evaluation of multi- window operations. InProceedings of the ACM SIGCHI Conference on Human factors in computing systems. 250–257

    1997

  57. [61]

    Wonjoong Kim, Sangwu Park, Yeonjun In, Sein Kim, Dongha Lee, and Chanyoung Park. 2026. Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents. arXiv:2510.02837 [cs.AI] https://arxiv.org/abs/2510. 02837

    Pith/arXiv arXiv 2026

  58. [62]

    Iqbal and Brian P

    Shamsi T. Iqbal and Brian P. Bailey. 2011. Oasis: A framework for link- ing notification delivery to the perceptual structure of goal-directed tasks. ACM Trans. Comput.-Hum. Interact.17, 4, Article 15 (Dec. 2011), 28 pages. doi:10.1145/1879831.1879833

    arXiv 2011

  59. [63]

    Ben Lafreniere, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2014. Investigating the feasibility of extracting tool demonstrations from in-situ video content. InProceedings of the SIGCHI Conference on Human Factors in Comput- ing Systems(Toronto, Ontario, Canada)(CHI ’14). Association for Computing Machinery, New York, NY, USA, 4007–4016. doi:10...

    arXiv 2014

  60. [64]

    Wei Li, Tovi Grossman, and George Fitzmaurice. 2012. GamiCAD: a gamified tutorial system for first time autocad users. InProceedings of the 25th Annual ACM Symposium on User Interface Software and Technology(Cambridge, Massachusetts, USA)(UIST ’12). Association for Computing Machinery, New York, NY, USA, 103–112. doi:10.1145/2380116.2380131

    arXiv 2012

  61. [65]

    Mary J Lindstrom and Douglas M Bates. 1988. Newton—Raphson and EM al- gorithms for linear mixed-effects models for repeated-measures data.J. Amer. Statist. Assoc.83, 404 (1988), 1014–1022

    1988

  62. [66]

    Benjamin Lafreniere, Tovi Grossman, and George Fitzmaurice. 2013. Commu- nity enhanced tutorials: improving tutorials with multiple demonstrations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1779–1788

    2013

  63. [67]

    Maglio and Christopher S

    Paul P. Maglio and Christopher S. Campbell. 2000. Tradeoffs in displaying peripheral information. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(The Hague, The Netherlands)(CHI ’00). Association for Computing Machinery, New York, NY, USA, 241–248. doi:10.1145/332040.332438

    arXiv 2000

  64. [68]

    Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2011. Ambient help. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada)(CHI ’11). Association for Computing Machinery, New York, NY, USA, 2751–2760. doi:10.1145/1978942.1979349

    arXiv 2011

  65. [69]

    Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2021. MeetingMate: an Ambient Interface for Improved Meeting Effectiveness and Corporate Knowl- edge Sharing. InGraphics Interface 2021. https://openreview.net/forum?id= ibaCpFUWVb9

    2021

  66. [70]

    Mynatt, Stephen Voida, Klaus M

    Blair MacIntyre, Elizabeth D. Mynatt, Stephen Voida, Klaus M. Hansen, Joe Tullio, and Gregory M. Corso. 2001. Support for multitasking and background awareness using interactive peripheral displays. InProceedings of the 14th Annual ACM Symposium on User Interface Software and Technology(Orlando, Florida) (UIST ’01). Association for Computing Machinery, Ne...

    arXiv 2001

  67. [71]

    Daniel C McFarlane and Kara A Latorella. 2002. The scope and importance of human interruption in human-computer interaction design.Human-Computer Interaction17, 1 (2002), 1–61

    2002

  68. [72]

    Ananya Gubbi Mohanbabu, Rosiana Natalie, Brandon Kim, Anhong Guo, and Amy Pavel. 2026. A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents.arXiv preprint arXiv:2602.09310(2026)

    arXiv 2026

  69. [73]

    Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. InProceedings of the SIGCHI conference on Human factors in computing systems. 249–256

    1990

  70. [74]

    Justin Matejka, Wei Li, Tovi Grossman, and George Fitzmaurice. 2009. Com- munityCommands: command recommendations for software applications. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 193–202

    2009

  71. [75]

    Yi-Hao Peng, Dingzeyu Li, Jeffrey P Bigham, and Amy Pavel. 2025. Morae: Proactively Pausing UI Agents for User Choices. InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). Association for Computing Machinery, New York, NY, USA, Article 198, 14 pages. doi:10. 1145/3746059.3747797

    arXiv 2025

  72. [76]

    Zachary Pousman and John Stasko. 2006. A taxonomy of ambient information systems: four patterns of design. InProceedings of the Working Conference on Advanced Visual Interfaces(Venezia, Italy)(A VI ’06). Association for Computing Machinery, New York, NY, USA, 67–74. doi:10.1145/1133265.1133277

    arXiv 2006

  73. [77]

    Why Should I Trust You?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining(San Francisco, California, USA)(KDD ’16). Association for Computing Machinery, New York, NY, USA, 1135–1144. doi:10.1145/...

    arXiv 2016

  74. [78]

    Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, and Xiang Yue. 2025. AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Ivan Habernal, Peter Schulam, and Jörg Tiede- mann (Eds.). Association for Computational Linguisti...

    doi:10.18653/v1/2025.emnlp-demos.15 2025

  75. [79]

    Pascal J Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F Grewe, and Thilo Stadelmann. 2026. A comprehensive survey of agents for computer use: Foundations, challenges, and future directions.Journal of Artificial Intelligence Research85 (2026)

    2026

  76. [80]

    Nitin Sawhney and Chris Schmandt. 2000. Nomadic radio: speech and audio interaction for contextual messaging in nomadic environments.ACM transactions on Computer-Human interaction (TOCHI)7, 3 (2000), 353–383

    2000

  77. [81]

    Anthony R Selkowitz, Shan G Lakhmani, and Jessie YC Chen. 2017. Using agent transparency to support situation awareness of the Autonomous Squad Member. Cognitive Systems Research46 (2017), 13–25

    2017

  78. [82]

    Joshua S Rubinstein, David E Meyer, and Jeffrey E Evans. 2001. Executive control of cognitive processes in task switching.Journal of experimental psychology: human perception and performance27, 4 (2001), 763

    2001

  79. [83]

    Bigham, and Jeffrey Nichols

    Amanda Swearngin, Jason Wu, Xiaoyi Zhang, Esteban Gomez, Jen Coughenour, Rachel Stukenborg, Bhavya Garg, Greg Hughes, Adriana Hilliard, Jeffrey P. Bigham, and Jeffrey Nichols. 2024. Towards Automated Accessibility Report Generation for Mobile Apps.ACM Trans. Comput.-Hum. Interact.31, 4, Article 54 (Sept. 2024), 44 pages. doi:10.1145/3674967

    work page doi:10.1145/3674967 2024

  80. [84]

    Maxwell Szymanski, Martijn Millecamp, and Katrien Verbert. 2021. Visual, textual or hybrid: the effect of user expertise on different explanations. InProceedings of the 26th International Conference on Intelligent User Interfaces(College Station, TX, USA)(IUI ’21). Association for Computing Machinery, New York, NY, USA, 109–119. doi:10.1145/3397481.3450662

    arXiv 2021

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.