Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Predicting user interventions makes web agents 26.5% more useful

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:09 UTC pith:ELN5DILK

load-bearing objection Useful new dataset and plausible offline prediction, but headline numbers are inconsistent and the §5 user study does not support the causal claim. the 3 major comments →

arxiv 2602.17588 v4 pith:ELN5DILK submitted 2026-02-19 cs.CL cs.HC

Modeling Distinct Human Interaction in Web Agents

Faria Huq , Zora Zhiruo Wang , Zhanqiu Guo , Venu Arvind Arangarajan , Tianyue Ou , Frank Xu , Shuyan Zhou , Graham Neubig
show 1 more author
Jeffrey P. Bigham
This is my paper
classification cs.CL cs.HC
keywords human intervention modelingweb navigation agentshuman-AI collaborationintervention predictioncollaboration stylesmultimodal LLM fine-tuninguser studyPerfect Timing Score
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that human intervention during web navigation is not random noise but a structured, learnable pattern, and that agents which anticipate when a user will step in can collaborate more effectively. To support this, the authors collect 400 real-user trajectories of human-agent web tasks, identify four consistent collaboration styles (hands-off, hands-on, collaborative, takeover) from the data, and fine-tune LMs to predict at each agent step whether the user will intervene. On this prediction task, the fine-tuned models improve intervention-prediction accuracy by 61.4–63.4% over base LMs and outperform proprietary generalist models on the timing metric. When the predictor is deployed in a live web agent, the four participants who returned for a second study rate it 26.5% higher on perceived usefulness than their earlier ratings of the baseline agent. If these results hold, the work would establish intervention modeling as a practical component of agent design, not just an analytical tool.

Core claim

The paper establishes that human intervention in collaborative web agents can be framed as a stepwise binary prediction problem p(y_t=1 | o_t, a-hat_t, tau_{t-1}), estimated by fine-tuning a multimodal LM to output <ask_user> or <agent_continue>. The authors build CowCorpus from 400 trajectories, cluster users into four stable styles using intervention frequency, intensity, normalized position, and handback rate, and show that style-conditioned models trained on corresponding clusters outperform generic models, except for the sparsest cluster. Fine-tuned open-weight LMs reach Perfect Timing Scores that beat closed-source models (0.303 vs 0.293 for the best proprietary model). In a live deplo

What carries the argument

The central object is the intervention-prediction model: a multimodal language model fine-tuned to classify each agent step as <ask_user> or <agent_continue>, conditioned on the trajectory history, current screenshot, accessibility tree, and proposed action. The empirical scaffold is CowCorpus, 400 real-user trajectories with step-level annotations of pause, resume, and override events, plus a four-style taxonomy (hands-off, hands-on, collaborative, takeover) derived from k-means clustering of four per-user features: intervention frequency, intensity, normalized position, and handback rate. The supporting evaluation machinery is the Perfect Timing Score (PTS), which penalizes mistimed predic

Load-bearing premise

The paper's live-usefulness claim assumes that the higher ratings given by the four returning participants reflect the intervention-prediction module's effect, not order, novelty, or the model having seen those participants' earlier interaction data.

What would settle it

Run a controlled cross-over study with fresh participants, comparing the same web agent with and without the intervention predictor, with order counterbalanced and logs of interruption timing; if the predictor neither prompts at moments closer to real interventions nor yields higher usefulness ratings than the no-predictor agent, the deployment claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If intervention prediction works, a web agent can decide to stay silent or ask based on learned user style, reducing unnecessary confirmations while preserving opportunities for correction.
  • Fine-tuned small open-weight models can match or exceed generalist large language models on this timing task, suggesting specialized interaction data matters more than model scale for collaboration.
  • Style-conditioned models are useful: users who intervene rarely (takeover style) benefit from models trained on heavier-intervening groups, indicating a transferable error signal.
  • Including the full history of human actions improves prediction accuracy (81.4% vs 76.3% step accuracy), so agents should attend to the interaction trajectory, not only the current page.
  • The reported 26.5% gain in user-rated usefulness comes from changing only the prompting policy, not the execution engine, making the intervention model a separable addition to existing agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the intervention-prediction signal is calibrated as a continuous cost, agents could trade off interruption cost against expected error, extending the binary ask/continue decision to a principled autonomy knob across domains like robotics or document editing.
  • The four-style taxonomy may serve as a compact user model: a testable prediction is that a user's style measured in one session predicts their intervention behavior with different agents and task types, enabling lifelong personalization.
  • The finding that explicit reasoning degrades timing suggests that rapid, reactive predictors may be preferable for deciding when to pause, potentially guiding a division of labor between a slow deliberative planner and a fast interruption detector.
  • A faithful extension of the paper's own result is that the predictor's value could be measured more directly by tracking interruption counts and task completion times in a larger deployment, rather than only self-reported usefulness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces CowCorpus, a dataset of 400 human-agent web-navigation trajectories with interleaved actions and intervention annotations. It clusters users into four collaboration styles, formulates human intervention modeling as stepwise binary prediction, and trains general and style-conditioned LMs. The authors report that fine-tuned models improve intervention prediction over base LMs and over proprietary models, and that deploying a prediction-enhanced agent (PlowPilot) increases user-rated usefulness by 26.5% in a four-participant follow-up study. The paper concludes that structured modeling of human intervention leads to more adaptive, collaborative agents.

Significance. If the results hold, the paper would make a solid empirical contribution: a real-user corpus of collaborative web tasks, a well-motivated step-level prediction task, and evidence that small fine-tuned open-weight models can beat large proprietary models at intervention timing. The trajectory-level held-out split and the time-aware PTS metric are reasonable design choices, and the planned release of models and code is a genuine strength. The main weakness is the live user study, which is too confounded to support the causal claim as stated; the paper's lasting value is therefore likely to rest on the dataset and the offline prediction results rather than on the deployment evaluation.

major comments (3)
  1. [§5; Figure 6; Abstract] The live-evaluation claim is not supported by the reported design. The four participants are a self-selected subset of the same 20 annotators whose first-round trajectories were used to define clusters and train style-conditioned models. PlowPilot ratings are compared against the participants' own earlier CowPilot ratings, with no control condition, counterbalancing, or statistical test. The sentence in §5 claiming that 'the observed gains therefore arise solely from proactively modeling human intervention' is an unsupported causal attribution; familiarity, order effects, and overfitting to those four users are equally plausible explanations. The paper itself labels the results 'preliminary results' and 'initial evidence'; this caveat should be reflected in the abstract and conclusion. Additionally, the header abstract reports a 36.8% increase while the full text and Figure 6 report 26.5
  2. [§4.2; Tables 4 and 6; Abstract] The abstract's claim of '61.4–63.4% improvement in intervention prediction accuracy' does not match any 'accuracy' column in the tables. Gemma-27B step accuracy improves from 0.239 to 0.853 (relative +257%), while Gemma PTS improves from 0.187 to 0.303 (relative +62%); LLaVA PTS improves from 0.017 to 0.201. The reported range appears to be the PTS relative improvement for Gemma, not an accuracy figure. The paper should specify exactly which metric the headline number refers to and report it consistently. Since PTS is a paper-defined metric with a free penalty parameter alpha, the value of alpha used for the headline number should be stated wherever the number appears.
  3. [§4.3; Eqs. (2)–(5)] The style labels used for the personalized models are derived from user-level features computed over each user's full trajectory set. Because the train/test split is at the trajectory level, test trajectories from a user contribute to the cluster assignment that determines which style-conditioned model is trained and evaluated for that user. This creates a form of user-level leakage, so the diagonal dominance in Figure 5 is optimistically biased for generalization to new users. A user-level (leave-one-user-out) split—or at least recomputing cluster labels using only training trajectories—is needed before claiming that style-conditioned models adapt to a user's collaboration style.
minor comments (4)
  1. [Table 3; §1; §4.1] The paper states 'over 4,200 interleaved human and agent actions' and gives 2,748 agent + 1,476 human = 4,224 actions. However, Table 3's per-trajectory averages (8.7 for standard, 7.0 for free-form) over 200 standard and 200 free-form trajectories imply roughly 3,140 total actions. The processed dataset described in §4.1 contains only 1,247 training + 251 test steps. Please clarify the filtering/preprocessing that connects these numbers.
  2. [§5] PLOWPILOT is introduced with citation to Allen et al. (2007), which is the PLOW system. Since PLOWPILOT is described as CowPilot plus the intervention module, the base-system citation should be Huq et al. (2025). Also clarify whether PLOWPILOT is a renamed fork of CowPilot or a separate extension.
  3. [§3.3; Figure 3] The choice k=4 is stated as producing 'distinct and stable' groups, but no cluster validity or stability analysis is reported. With only 20 users, a silhouette score, bootstrap stability check, or leave-one-user-out clustering would strengthen the claim of four consistent interaction styles.
  4. [§4.1; Tables 4, 6] The test set contains only 251 steps, with roughly 31 positive intervention steps given the 1:7 ratio. The headline comparisons (e.g., Gemma-SFT vs. Claude) would benefit from confidence intervals or a significance test; in their absence, the differences may not be robust.

Circularity Check

2 steps flagged

Personalization evaluation is partially circular: cluster labels for style-conditioned models are derived from the same trajectories being tested, and the live user study rates the model on the same users whose data defined the clusters; the main offline prediction result remains independent.

specific steps
  1. self definitional [§3.2.1 (Eqs. 2–5), §3.3 (k-means), §4.1 (split), §4.3 (style-conditioned models)]
    "Using the four participant-level measures in §3.2.1, we cluster users by interaction behavior with k-means (k=4). ... We split CO WCO R P U Sdata into train and test sets at the trajectory level to avoid leakage."

    Equations (2)–(5) compute per-user frequency, intensity, normalized position, and handback as sums over D_u, the user's full trajectory set. The k-means cluster labels are therefore functions of every trajectory, including trajectories later placed in the test split. A style-conditioned model is selected/trained by that cluster and evaluated on the held-out trajectory, so the trajectory's own intervention steps have already contributed to the conditioning variable. The claimed 'trajectory-level split' does not remove the test label from the cluster input; the diagonal dominance in Fig. 5 is partly an in-sample fit.

  2. fitted input called prediction [§5, Figure 6]
    "We assigned the customized interactive prediction model from §4.3 based on the cluster they belonged to. We compare their post-session ratings for the intervention-aware agent against their earlier ratings for the baseline agent to assess changes in user satisfaction. ... The observed gains therefore arise solely from proactively modeling human intervention."

    The four raters are the same users whose first-round trajectories were clustered (§3.3) and used to fine-tune the cluster-specific models (§4.3). Their PlowPilot ratings are compared with their own earlier CowPilot ratings, with no control condition, counterbalancing, or statistical test. The 26.5% gain is therefore measured on the model's fitted population: the raters are the inputs used to build the personalized model, so the result cannot establish that intervention modeling, rather than overfitting or order effects, causes the perceived usefulness.

full rationale

Most of the paper is self-contained. The offline intervention-prediction experiment (§4.1–§4.2) uses a trajectory-level train/test split, reports separate baselines, and does not fit the test labels; the reported 61.4–63.4% improvement is not constructed from the evaluation target. The k-means taxonomy is descriptive, and no load-bearing self-citation chain or 'uniqueness theorem' forces the central derivation. The circularity is confined to the personalization thread. First, the user-level cluster features (Eqs. 2–5) are computed over all trajectories before the split, so the cluster label that selects a style-conditioned model for a held-out trajectory is itself a function of that trajectory's intervention steps; Fig. 5's diagonal dominance is thus partly in-sample. Second, the live study (§5) evaluates the cluster-conditioned model on the same four users whose data defined the clusters and compares it with their own earlier baseline ratings, so the 26.5% usefulness gain is not independent evidence of the causal claim. The abstract's '36.8%' versus §5's '26.5%' discrepancy and the lack of statistical testing are additional non-circular validity concerns. The headline offline result remains an independent, non-circular contribution.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central claims rest on a small, self-selected participant pool, a hand-chosen cluster count, a hand-set evaluation metric weight, and the assumption that binary step-level intervention labels capture the full dynamics of human intervention.

free parameters (2)
  • k (number of interaction clusters) = 4
    Chosen a priori for k-means; no cluster validation or stability analysis is reported.
  • alpha (PTS temporal penalty) = 0.2
    Set by hand in §2 with a sensitivity sweep in §B.5; the metric and its normalization are defined by this constant.
axioms (4)
  • domain assumption Human intervention is a binary step-level variable y_t ∈ {0,1} determined at each agent action.
    Formulation in §2; interventions that happen during an agent action or outside the logged step are not modeled.
  • domain assumption Four collaboration styles recovered from 20 participants generalize to other users.
    Clustering in §3.3 on four hand-designed features with k=4; no evidence of stability beyond this sample.
  • domain assumption PTS with squared-distance penalty and α=0.2 reflects the quality of intervention timing.
    Metric introduced in §2; sensitivity analysis in §B.5 shows ranking stability but not validity.
  • domain assumption Self-reported 7-point Likert ratings measure agent usefulness.
    Used in §5 user study; no validation or connection to objective task success/time.

pith-pipeline@v1.3.0-alltime-deepseek · 17227 in / 11243 out tokens · 112408 ms · 2026-08-02T22:09:47.743191+00:00 · methodology

0 comments
read the original abstract

Despite rapid progress in autonomous web agents, human involvement remains essential for shaping preferences and correcting agent behavior as tasks unfold. However, current agentic systems lack a principled understanding of when and why humans intervene, often proceeding autonomously past critical decision points or requesting unnecessary confirmation. In this work, we introduce the task of modeling human intervention to support collaborative web task execution. We collect CowCorpus, a dataset of 400 real-user web navigation trajectories containing over 4,200 interleaved human and agent actions. We identify four distinct patterns of user interaction with agents -- hands-off supervision, hands-on oversight, collaborative task-solving, and full user takeover. Leveraging these insights, we train language models (LMs) to anticipate when users are likely to intervene based on their interaction styles, yielding a 61.4-63.4% improvement in intervention prediction accuracy over base LMs. Finally, we deploy these intervention-aware models in live web navigation agents and evaluate them in a user study, finding a 36.8% increase in user-rated agent usefulness. Together, our results show structured modeling of human intervention leads to more adaptive, collaborative agents.

Figures

Figures reproduced from arXiv: 2602.17588 by Faria Huq, Frank Xu, Graham Neubig, Jeffrey P. Bigham, Shuyan Zhou, Tianyue Ou, Venu Arvind Arangarajan, Zhanqiu Guo, Zora Zhiruo Wang.

Figure 1
Figure 1. Figure 1: In this paper, we present COWCOR￾P US, a dataset of 400 real-user collaborative web trajectories that captures when and how humans intervene during execution, enabling intervention-aware agents that engage users only when needed. This motivates a central question: Can agents proactively anticipate when human inter￾vention is likely and adapt their behavior accord￾ingly? To answer this, we introduce COW￾COR… view at source ↗
Figure 2
Figure 2. Figure 2: Visual Illustration of how PTS is calcu [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Four distinct types of human-agent interaction patterns: Takeover, Hands-on, Hands-off, and Collaborative. We visualize the user groups using PCA (left), and describe the interaction mechanism of each group (right). 3.3. Task-Level Interaction Patterns We analyze when human interventions occur during collaborative task execution and how such temporal patterns vary across users. We summarize each participan… view at source ↗
Figure 4
Figure 4. Figure 4: Perfect Timing Score on COWCORP US. Out of the proprietary models, Claude outper￾forms GPT-4o and Gemini-2.5. On the finetuned model, Gemma 27B significantly boosts the per￾formance when finetuned on COWCOR P US. Proprietary Models remain overly con￾servative: We evaluate three families of closed-source LMs (Claude 4 Sonnet (Anthropic, 2025), GPT-4o (Hurst et al., 2024), and Gemini 2.5 Pro (Comanici et al.… view at source ↗
Figure 5
Figure 5. Figure 5: The heatmap shows the PTS score on the cluster-wise trained models for each of the [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: User response to the Likert scale questionnaire after the study. On average, user reports [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Three example tasks from top three free-form task categories. (all identifiable informa [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Overview of the collaborative AI agent, COWPI LOT Huq et al. (2025) used in our data collection. 1) Before the task is initiated, the user gives a textual task description as input. 2) During task execution, the system tracks the actions performed by the user and the agent. 3) After the task is executed, the user can download the task log. task, participants are shown a summary of their annotation, where t… view at source ↗
Figure 9
Figure 9. Figure 9: Time log across participants for the same task [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Zero-shot models maintain consis￾tent PTS rankings across 𝛼. We sweep 𝛼 over a fixed grid while holding all inputs constant and recompute PTS for each model under zero-shot setting. Kendall’s W significant test reveals that PTS preserves relative ordering under different temporal penalties [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Motif: Discovering and Automating Personal Web Workflows

    cs.HC 2026-07 conditional novelty 7.0

    Motif discovers more automatable personal web workflows from passive browser logs than users self-identify, and generates useful deterministic programs for most of them.

  2. Hedwig: Dynamic Autonomy for Coding Agents Under Local Oversight

    cs.HC 2026-05 unverdicted novelty 6.0

    Hedwig is a coding agent that dynamically adjusts its autonomy by learning behavioral guidelines from developer decisions and feedback over time.

  3. Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability

    cs.CL 2026-05 unverdicted novelty 4.0

    The paper develops a unified framework that organizes computer-use agent reliability around perception-decision-execution layers and creation-deployment-operation-maintenance stages to map security and alignment inter...

Reference graph

Works this paper leans on

30 extracted references · 1 canonical work pages · cited by 3 Pith papers

  1. [3]

    task, participants are shown a summary of their annotation, where they can download the data log form if they wish

    After the task is executed, the user can download the task log. task, participants are shown a summary of their annotation, where they can download the data log form if they wish. Such self-initiated data collection ensuresthe user has full control over which data they want to share with us and which they do not. A.2.CO WPI L O T: T ask Annotation Framewo...

    2025

  2. [5]

    Comanici, E

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reason- ing, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  3. [6]

    K. Feng, K. Pu, M. Latzke, T. August, P . Siangliulue, J. Bragg, D. S. Weld, A. X. Zhang, and J. C. Chang. Cocoa: Co-planning and co-execution with ai agents. arXiv preprint arXiv:2412.10999,

    arXiv

  4. [7]

    URLhttps://arxiv.org/abs/2402.03578. W. Hua, X. Yang, M. Jin, Z. Li, W. Cheng, R. Tang, and Y. Zhang. Trustagent: Towards safe and trustworthy llm-based agents,

    Pith/arXiv arXiv

  5. [8]

    URLhttps://arxiv.org/abs/2402.01586. F. Huq, Z. Z. Wang, F. F. Xu, T. Ou, S. Zhou, J. P . Bigham, and G. Neubig. CowPilot: A framework for autonomous and human-agent collaborative web navigation. In N. Dziri, S. X. Ren, and S. Diao, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Ling...

    Pith/arXiv arXiv 2025

  6. [9]

    ISBN 979-8-89176-191-9

    Association for Computational Linguistics. ISBN 979-8-89176-191-9. doi: 10.18653/v1/2025.naacl-demo.17. URLhttps://aclanthology.org/2025.naacl-demo.17/. A. Hurst, A. Lerer, A. P . Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv 2025

  7. [10]

    doi: 10.18653/v1/2024.acl-demos.8

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-demos.8. URL https://aclanthology.org/2024.acl-demos.8. P . Khadpe, R. Krishna, L. Fei-Fei, J. T. Hancock, and M. S. Bernstein. Conceptual metaphors im- pact perceptions of human-ai collaboration. Proc. ACM Hum.-Comput. Interact., 4(CSCW2), Oct

    work page doi:10.18653/v1/2024.acl-demos.8 2024

  8. [13]

    URL https://arxiv.or g/abs/2409.11295. D. Misra, J. Langford, and Y. Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795,

    Pith/arXiv arXiv

  9. [14]

    URLhttps://arxiv.org/abs/2502.02649. H. Mozannar, G. Bansal, C. Tan, A. Fourney, V . Dibia, J. Chen, J. Gerrits, T. Payne, M. K. Maldaner, M. Grunde-McLaughlin, et al. Magentic-ui: Towards human-in-the-loop agentic systems. arXiv preprint arXiv:2507.22358,

    arXiv

  10. [15]

    Y. Pan, D. Kong, S. Zhou, C. Cui, Y. Leng, B. Jiang, H. Liu, Y. Shang, S. Zhou, T. Wu, et al. Web- canvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373,

    Pith/arXiv arXiv

  11. [16]

    URL https: //arxiv.org/abs/2504.00907. Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto. Identifying the risks of lm agents with an lm-emulated sandbox,

    arXiv

  12. [17]

    org/abs/2309.15817

    URL https://arxiv. org/abs/2309.15817. W. Saunders, G. Sastry, A. Stuhlmueller, and O. Evans. Trial without error: Towards safe reinforcement learning via human intervention. arXiv preprint arXiv:1707.05173,

    Pith/arXiv arXiv

  13. [20]

    doi: 10.18653/v1/2024.naacl-long.347

    Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.347. URL https://aclanthology.org/2024.naacl-long. 347/. Y. Shao, V . Samuel, Y. Jiang, J. Yang, and D. Yang. Collaborative gym: A framework for enabling and evaluating human-agent collaboration,

    doi:10.18653/v1/2024.naacl-long.347 2024

  14. [21]

    Tariq, M

    S. Tariq, M. B. Chhetri, S. Nepal, and C. Paris. A2c: A modular multi-stage collaborative decision framework for human-ai teams. ArXiv, abs/2401.14432,

    Pith/arXiv arXiv

  15. [22]

    URLhttps://taxy.ai/. D. Wang, E. Churchill, P . Maes, X. Fan, B. Shneiderman, Y. Shi, and Q. Wang. From human- human collaboration to human-ai collaboration: Designing ai systems that can work to- gether with people. In Extended abstracts of the 2020 CHI conference on human factors in computing systems, pages 1–6,

    2020

  16. [23]

    Z. Z. Wang, Y. Shao, O. Shaikh, D. Fried, G. Neubig, and D. Yang. How do ai agents do human work? comparing ai and human workflows across diverse occupations. arXiv preprint arXiv:2510.22780,

    arXiv

  17. [26]

    ISBN 9798400713064

    Association for Computing Machinery. ISBN 9798400713064. doi: 10.1145/3708359.3712153. URL https://doi.org/10.1145/370835 9.3712153. B. Zheng, B. Gou, S. Salisbury, Z. Du, H. Sun, and Y. Su. WebOlympus: An open platform for web agents on live websites. In D. I. Hernandez Farias, T. Hope, and M. Li, editors, Proceedings of the 2024 Conference on Empirical ...

    arXiv 2024

  18. [27]

    doi: 10.18653/v1/2024.emnlp-demo.20

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-demo.20. URL https://aclant hology.org/2024.emnlp-demo.20. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854,

    Pith/arXiv arXiv 2024

  19. [28]

    CO WCO R P U S: Human-Agent Collaborative Web Corpus A.1

    15 A. CO WCO R P U S: Human-Agent Collaborative Web Corpus A.1. T ask Annotation Setup and Participant Information P12:Find most viewed runner review video since 2025The most viewed runner review video since 2025 is 'Best Daily Running Shoes 2025: Nike, Brooks, ASICS, Saucony, etcby The Running Channel' with 101,074 views 1 3 5 (a) Information Access P16:...

    2025

  20. [30]

    • Personalized Interests & Lifestyle : Engagement with entertainment or lifestyle content

    18 • Information Access : Users are looking for specific information, such as facts, news, academic papers, and definitions. • Personalized Interests & Lifestyle : Engagement with entertainment or lifestyle content. • Product & Service Interaction : Users shop for specific products, compare prices, or book services like flights or rentals. • Content Gener...

    arXiv 2024

  21. [2015]

    doi: 10.1109/SECON.2015.7132964. S. Chen, S. Wiseman, and B. Dhingra. Chatshop: Interactive information seeking with language agents. arXiv preprint arXiv:2404.09911,

    arXiv 2015

  22. [2017]

    Schick, J

    T. Schick, J. Dwivedi-Yu, Z. Jiang, F. Petroni, P . Lewis, G. Izacard, Q. You, C. Nalmpantis, E. Grave, and S. Riedel. Peer: A collaborative language model. ArXiv, abs/2208.11663,

    Pith/arXiv arXiv

  23. [2018]

    doi: 10.1007/s10514-017-9677-2

    ISSN 0929-5593. doi: 10.1007/s10514-017-9677-2. URL https://doi.org/10.1007/s105 14-017-9677-2. J. Allen, N. Chambers, G. Ferguson, L. Galescu, H. Jung, M. Swift, and W. Taysom. Plow: a col- laborative task learning agent. In Proceedings of the 22nd National Conference on Artificial Intelligence - Volume 2, AAAI’07, page 1514–1519. AAAI Press,

    doi:10.1007/s10514-017-9677-2

  24. [2019]

    ISBN 9781450368162

    Association for Computing Machinery. ISBN 9781450368162. doi: 10.1145/3332165.3347899. URL https://doi.org/10.1145/3332165.3347899. Z. Liao, L. Mo, C. Xu, M. Kang, J. Zhang, C. Xiao, Y. Tian, B. Li, and H. Sun. Eia: Environmental injection attack on generalist web agents for privacy leakage,

    arXiv

  25. [2020]

    URLhttps://doi.org/10.1145/3415234

    doi: 10.1145/3415234. URLhttps://doi.org/10.1145/3415234. M. Lee, P . Liang, and Q. Yang. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. Proceedings of the 2022 CHI Conference on Human 13 Factors in Computing Systems,

    doi:10.1145/3415234 2022

  26. [2021]

    URLhttps://doi.org/10.1145/3432945

    doi: 10.1145/3432945. URLhttps://doi.org/10.1145/3432945. Z. Zhang, J. Gao, R. S. Dhaliwal, and T. J.-J. Li. Visar: A human-ai argumentative writing assistant with visual programming and rapid draft prototyping. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA,

    doi:10.1145/3432945

  27. [2022]

    URLhttps://api.semanticscholar.org/CorpusID:251765117. Y. Shao, Y. Jiang, T. Kanell, P . Xu, O. Khattab, and M. Lam. Assisting in writing Wikipedia-like articles from scratch with large language models. In K. Duh, H. Gomez, and S. Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui...

    2024

  28. [2023]

    ISBN 9798400701320

    Association for Computing Machinery. ISBN 9798400701320. doi: 10.1145/35 86183.3606800. URLhttps://doi.org/10.1145/3586183.3606800. Z. J. Zhang, E. Schoop, J. Nichols, A. Mahajan, and A. Swearngin. From interaction to impact: Towards safer ai agent through understanding and evaluating mobile ui operation impacts. In Proceedings of the 30th International C...

    arXiv

  29. [2024]

    Chandrasekaran and J

    B. Chandrasekaran and J. M. Conrad. Human-robot collaboration: A survey. In SoutheastCon 2015, pages 1–8,

    2015

  30. [2025]

    Accessed: 2026-01-29

    URL https://www.anth ropic.com/claude-4-system-card. Accessed: 2026-01-29. 12 G. Bansal, J. W. Vaughan, S. Amershi, E. Horvitz, A. Fourney, H. Mozannar, V . Dibia, and D. S. Weld. Challenges in human-agent communication. arXiv preprint arXiv:2412.10380,

    Pith/arXiv arXiv 2026