Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Fara1.5 shows that small, open web agents can be trained from synthetic data and rival much larger proprietary systems on browser-use benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 10:43 UTC pith:AEI6ZON7

load-bearing objection Genuine advance in synthetic data generation for computer-use agents, with real contributions and released weights, but the headline SoTA claims rest on an unaddressed decontamination risk and a self-modified benchmark. the 4 major comments →

arxiv 2606.20785 v2 pith:AEI6ZON7 submitted 2026-06-18 cs.AI cs.LG

Fara-1.5: Scalable Learning Environments for Computer Use Agents

Ahmed Awadallah , Sahil Gupta , Yash Lara , Yadong Lu , Hussein Mozannar , Akshay Nambi , Zach Nussbaum , Yash Pandya
show 7 more authors
Aravind Rajeswaran Corby Rosset Alexey Taymanov Luiz do Valle Vibhav Vineet Spencer Whitehead Andrew Zhao
This is my paper
classification cs.AI cs.LG
keywords computer use agentssynthetic data generationweb browsing agentssupervised fine-tuningLLM verifierscritical-point handlingGUI groundingbrowser automation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that a fully automated data-generation pipeline—live websites plus sandboxed synthetic replicas, solved by a frontier teacher model, and filtered by three LLM-based verifiers—can replace expensive human demonstrations as the main training signal for computer-use agents. Training on this data yields the Fara1.5 family (4B, 9B, 27B), which sets a new state of the art for its size class: Fara1.5-9B reaches 63.4% on Online-Mind2Web and 86.6% on WebVoyager, and Fara1.5-27B reaches 72.3% on Online-Mind2Web, competitive with much larger proprietary systems. If true, capable web automation no longer requires massive proprietary models or costly human data collection. The paper also reframes critical-point behavior: instead of always stopping before irreversible actions, the agent is trained to ask for user permission and then complete the action, which improves both safety and task completion.

Core claim

The central claim is that a scalable, modular pipeline—FaraGen1.5—can generate high-quality computer-use trajectories at scale and distill them into small, native pixel-in/action-out agents that outperform prior open-weight agents of the same size. The pipeline's three phases are: (1) environments consisting of live websites and six synthetic sandboxed replicas (email, calendar, media, ML experiments, marketplace, scheduler) with full backend control and ground-truth verification; (2) a solver harness powered by GPT-5.4 that interacts with a user simulator to produce multi-turn trajectories; (3) three complementary verifiers—task correctness, efficiency, and critical-point adherence—that all

What carries the argument

The core mechanism is the three-verifier filter applied to every generated trajectory. The correctness verifier uses an LLM-generated rubric (Universal Verifier) for live web tasks and backend-state comparison (sqldiff) or reference answers for synthetic environments. The efficiency verifier is an LLM judge that identifies redundant or looped actions and assigns a 1-5 rating. The critical-point verifier classifies each task into one of eight types based on whether the user granted permission, whether the task is fully specified, and whether all required PII is provided; it then checks that the solver asked the user at every critical point before proceeding. Only trajectories passing all thre

Load-bearing premise

The three automated verifiers—especially the LLM-based correctness judge, efficiency judge, and critical-point classifier—reliably separate good trajectories from bad ones; if they systematically misjudge, the errors are baked into all downstream training data and benchmark results.

What would settle it

Take a random sample of accepted and rejected trajectories from FaraGen1.5 and have human annotators label them for task success, efficiency, and critical-point compliance. If human-verifier agreement is low, or if a model trained on the filtered set performs no better than one trained on the unfiltered set, the verifier stack is not the source of the reported gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Open-weight agents at 4B–27B parameters can achieve browser-task success rates that rival or exceed much larger proprietary systems, lowering deployment cost and hardware requirements.
  • Synthetic sandboxed environments can teach agents tasks that are impossible to collect on the open web—login flows, purchases, sending messages—and this learning transfers to real live websites, as shown by the synthetic-to-real experiments.
  • Training agents to pause at critical points and ask the user, rather than halting entirely, improves safety behavior (100% refusal on WebTailBench-Refusals) while also improving task completion.
  • The data mix recipe (85.3% from FaraGen1.5) plus targeted auxiliary grounding, VQA, and refusal data provides a repeatable template for training small computer-use agents.
  • The efficiency verifier and step-length analysis suggest that a substantial part of model quality is efficiency in taking fewer, more purposeful actions, which can be optimized independently of task correctness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the verifier reliability holds, the pipeline could be bootstrapped further: student-generated trajectories that pass the same verifiers could be used to improve the next generation of students, potentially reducing or eliminating the need for frontier teacher models.
  • The eight-type critical-point taxonomy and the associated ask-user-then-act training behavior could be adopted as a standalone safety curriculum for any computer-use agent, independent of the full FaraGen1.5 pipeline.
  • The WebTailBench v1.5 scoring change—counting a stop at a critical point as success when the task cannot legitimately be completed without missing information—makes its numbers not directly comparable to the original WebTailBench; readers tracking progress over time should treat the two as distinct benchmarks.
  • A testable extension would be applying the same synthetic-environment pipeline to desktop or terminal-based tasks, as the paper hints toward future work; the verifier design (state-based correctness, efficiency, and critical-point checks) would likely transfer with minimal changes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces FaraGen1.5, a data-generation pipeline for computer-use agents, combining live-web and synthetic sandboxed environments, a GPT-5.4-based solver with a user simulator, and three LLM-based verifiers (correctness, efficiency, critical-point adherence). Using the resulting trajectories plus auxiliary grounding/VQA/safety data, the authors SFT-tune Qwen3.5 backbones at 4B/9B/27B to obtain the Fara1.5 family. They report state-of-the-art results among 7–9B open-weight models on Online-Mind2Web (63.4%) and WebVoyager (86.6%), with the 27B model reaching 72.3% on Online-Mind2Web and competitive with larger proprietary systems. They also report synthetic-environment transfer, grounding, and safety results, and release model weights under an MIT license.

Significance. If the headline results are reliable, this is a significant contribution: an open, MIT-licensed family of native pixel-action CUA models that closes much of the gap to proprietary systems, plus a reusable pipeline for synthetic trajectory generation. Strengths include three-run averages with 95% CIs, use of official external judges for WebVoyager and Online-Mind2Web, programmatic state verification in synthetic environments, a synthetic-to-real transfer experiment, and a detailed error taxonomy. The main uncertainties are contamination of the live-web training data with benchmark tasks, reliability of the LLM-based verifiers, modifications to WebTailBench, and comparability of leaderboard-sourced baselines.

major comments (4)
  1. [§2.1.1, §3.2, §4.1.1] No contamination analysis is reported between the FaraGen1.5 live-web training data and the WebVoyager/Online-Mind2Web benchmarks. Since 60% of the training mix consists of tasks generated for live sites from a large public URL index, and the benchmark tasks are also live-web tasks on overlapping sites, near-duplicates of benchmark tasks could be in the training set. The paper's own train/test deduplication for synthetic tasks (§2.1.2) shows awareness of this risk, but the same discipline is not applied to the benchmarks. Table 5 mitigates the concern for four WebVoyager domains, but not for Online-Mind2Web, and the control uses unrelated domains. Please report a task-level overlap analysis (exact and paraphrase/embedding-based) between FaraGen1.5 task proposals and the 300 Online-Mind2Web and 800 WebVoyager tasks, and re-evaluate on a contamination-free subset.
  2. [§4.1.1, Table 3] WebTailBench v1.5 is a modified, in-house benchmark: 270 tasks are refreshed and the success criterion is changed so that stopping at a critical point counts as success when the task requires information not provided; the eval harness injects a canned reply to ask_user_question and the rubric judge is aware of this. These changes are disclosed but make scores non-comparable to the original WebTailBench and to baselines that do not have ask_user_question in their tool lists. Because the same type of LLM rubric (Rosset et al., 2026) is used both for training-data filtering and for this evaluation, the WebTailBench v1.5 results risk circularity. Please provide human-agreement or inter-judge reliability for the modified rubric and release the exact task list and judge prompts.
  3. [§2.3 and Introduction] The entire data pipeline is gated by three verifiers, but no accuracy or calibration evidence for them is provided. The paper acknowledges that 'LLMs as judges may hallucinate' (Introduction), yet the correctness threshold (rubric process score ≥ 0.8) and efficiency threshold (rating ≥ 4) are asserted without validation. The critical-point classifier is itself an LLM judge based on the task prompt and screenshots (§2.3). If the verifiers have systematic blind spots, the SFT corpus inherits those errors, and WebTailBench v1.5, which uses a related rubric, inherits them as well. Please report verifier precision/recall against a human-annotated trajectory sample, including per-category breakdowns for the eight critical-point types.
  4. [Table 3, §4.1.2] The cross-system comparison to larger proprietary agents relies on leaderboard numbers retrieved at different dates (e.g., OpenAI Operator May 2025, Gemini 2.5 CU Sep 2025, Yutori Navigator Nov 2025 in Figure 7), whereas WebVoyager and Online-Mind2Web are live benchmarks with refreshed time-sensitive tasks. Without re-running these baselines under the same protocol, the claim that Fara1.5-27B 'outperforms much larger proprietary systems' is not fully established. Either re-run at least the strongest competitor under the current protocol or soften the claim to an explicitly labeled reference comparison.
minor comments (5)
  1. [§2.4, §3.2, §A.8] Typos and grammar: 'We have ran' -> 'We have run'; 'these ratios have been determine' -> 'determined'; 'auxillary' -> 'auxiliary'; 'insturction' -> 'instruction'.
  2. [Table 3] The Fara1.5-27B row is missing spaces: '89.3 72.372.9 40.2'. Please fix formatting.
  3. [Appendix A.6] The text refers to a 'FaraBench v1.5 ensemble judge', but the benchmark is called WebTailBench v1.5 elsewhere. Unify the naming.
  4. [Appendix A.2] The pass@k analysis is described without defining how pass@k is estimated from the three runs. Add a one-sentence definition.
  5. [§3.1] The choice of three image context windows is stated to be best, but no ablation is shown. Either include the ablation or clearly mark this as a tuning choice with reference to appendix material.

Circularity Check

2 steps flagged

Headline SoTA is externally judged; WebTailBench v1.5 is partially circular.

specific steps
  1. self definitional [Section 4.1.1 (WebTailBench v1.5) and Section 2.3 (Verifiers)]
    "To avoid rewarding such fabrication, we adapt the verifier so that stopping at a critical point counts as success when the task cannot legitimately be completed without information the agent has not been given. ... the outcome judge treats properly stopping at a critical point as success rather than as a missed deferral. ... Trajectories that cross a critical point without prior approval are rejected."

    The benchmark's success criterion was redefined by the authors to credit stopping at a critical point, which is exactly the behavior the critical-point verifier enforces during training ('Trajectories that cross a critical point without prior approval are rejected') and the behavior Fara1.5 is trained to exhibit through ask_user_question at critical points. Thus WebTailBench v1.5 process/outcome scores measure whether the model reproduces the training filter's target rather than independent task completion, and the reported +15.7/+8.2 gains over Fara-7B are partially forced by this construction.

  2. self citation load bearing [Section 4.1.1 (Per-benchmark judges) and Section 2.3 (Task correctness)]
    "WebTailBench uses a multi-stage rubric judge from (Rosset et al., 2026) ... the rubric judge scores each trajectory against a precomputed per-task rubric with a success threshold of 0.8. ... On live web environments, we use the process-reward score from the Universal Verifier (Rosset et al., 2026) ... We accept trajectories whose rubric process score is at least 0.8."

    The same verifier line—Rosset et al. (2026), with coauthors overlapping this paper—is used both to filter the training trajectories (Universal Verifier) and to score the WebTailBench v1.5 evaluation. The WebTailBench v1.5 result is therefore an in-group self-assessment rather than an independent check, and the benchmark's success criterion was also adjusted to match the verifier's critical-point target. This does not affect the external WebVoyager/Online-Mind2Web headline numbers.

full rationale

The central SoTA claims are independently grounded: Online-Mind2Web uses the official WebJudge of Xue et al. (2025) and WebVoyager uses the official GPT-4o judge of He et al. (2024a), both external to this paper, so the headline size-class numbers do not reduce to the paper's own pipeline. The synthetic-to-real transfer experiment (Table 5) also provides an external check on four WebVoyager domains. The circularity is confined to WebTailBench v1.5, where the authors (a) changed the success criterion so that stopping at a critical point counts as success when information is missing—the same behavior their critical-point verifier enforces in training—and (b) evaluate with their own Rosset et al. (2026) rubric judge. Those WebTailBench v1.5 gains are therefore partially self-referential, but they are not the basis for the paper's main SoTA claim. The absence of a decontamination analysis between FaraGen1.5 training tasks and benchmark tasks is a correctness/leakage risk, not a circularity, and is not scored here.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claims rest on LLM-based verification rather than formal ground truth, on limited evidence that synthetic environments transfer, and on undisclosed curation sweeps. Headline external benchmarks reduce but do not eliminate this dependence, and the WebTailBench v1.5 result is partly circular with the training verifier.

free parameters (4)
  • SFT data mix proportions = Web trajectories 60.0%, synthetic envs 12.8%, form-filling 12.5%, grounding 8.8%, VQA 4.9%, GUI drag 0.8%, safety 0.1%
    The paper states the mix ratios were chosen by sweeping proportions to maximize agentic accuracy without regressing grounding/VQA; the sweep search is not reported (Section 3.2, Figure 4).
  • Verifier acceptance thresholds = Correctness rubric score >= 0.8; efficiency rating >= 4
    These hand-chosen thresholds determine which trajectories enter training and thus what the model learns (Section 2.3).
  • Image context window size = 3 most recent screenshots
    Chosen experimentally as the best tradeoff between agentic performance and token consumption (Section 3.1).
  • SFT hyperparameters = lr=5e-6, epochs=2, batch=128, cosine schedule with 0.10 warmup
    Standard but hand-selected training settings; central to reproducing the reported model quality (Appendix A.5).
axioms (5)
  • domain assumption LLM judges used as correctness, efficiency, and critical-point verifiers produce reliable labels for training-data filtering.
    The entire FaraGen1.5 filter depends on this; the authors themselves note LLM judges may hallucinate (Introduction, Section 2.3).
  • domain assumption Synthetic FaraEnvs faithfully reproduce enough of the target domains that skills transfer to live websites.
    This is tested on only four WebVoyager domains in Section 4.1.3, not across the six FaraEnvs or the broader web.
  • domain assumption Official benchmark judges (WebJudge, GPT-4o WebVoyager judge) are valid and stable despite live-site state changes.
    Evaluation protocol relies on refreshing tasks and two-pass Browserbase handling, but judge validity is taken as given (Section 4.1.1).
  • domain assumption GPT-5.4 solver trajectories filtered by the three verifiers are learnable by smaller Qwen3.5 models via SFT.
    The whole distillation approach assumes the student can absorb the teacher's behavior from the filtered trajectories (Section 2.2, 3.2).
  • domain assumption The canned user reply used at eval time approximates a real user well enough to not distort benchmark scores.
    Section 4.1.1 describes intercepting ask_user_question with a fixed canned reply; this could over- or under-credit the agent relative to a real user.
invented entities (2)
  • FaraEnvs (six synthetic environments: email, calendar, media streaming, ML experiment management, marketplace, scheduling) no independent evidence
    purpose: Enable safe training data generation for auth-gated and irreversible tasks with execution-based verification against backend state.
    The environments are not released, and their fidelity is only internally tested on four transfer domains; no external handle is provided for others to validate them.
  • User simulator no independent evidence
    purpose: Generate multi-turn dialogue and supply missing information or permission during trajectory generation.
    The simulator is an LLM with appended prompts (Appendix A.4); its behavior is not validated against human user responses or released.

pith-pipeline@v1.3.0-alltime-deepseek · 32160 in / 12102 out tokens · 114651 ms · 2026-08-02T10:43:19.317652+00:00 · methodology

0 comments
read the original abstract

Collecting computer use data from human demonstrations is expensive and slow, motivating the need for scalable generation strategies. This requires two key ingredients: environments in which agents can act and verifiers that can judge whether their demonstrations succeeded. We introduce FaraGen1.5, a scalable data pipeline for computer use agents composed of three modular components: environments, solvers, and verifiers. FaraGen1.5 uses both live websites and synthetic environments that faithfully simulate domains gated by authentication or that require irreversible actions. It employs a solver harness that can be powered by multiple models, including strong frontier models such as GPT-5.4, and also incorporates a user simulator to enable multi-turn rollouts. Finally, FaraGen1.5 scores the resulting trajectories with three complementary verifiers covering task correctness, efficiency, and critical-point adherence. Using data produced by this pipeline, we train Fara1.5, a family of native computer use agents (CUAs) at three scales built on Qwen3.5 (4B, 9B, and 27B). To train these models, we employ a supervised finetuning (SFT) recipe that carefully balances data from FaraGen1.5 for broad coverage, specific high-value tasks, and target model deficiencies in an iterative approach. Each model sets a new state of the art (SoTA) for its size class on browser-use benchmarks: Fara1.5-9B reaches 63.4% on Online-Mind2Web and 86.6% on WebVoyager, while Fara1.5-27B achieves 72.3% on Online-Mind2Web, which is competitive with much larger proprietary systems. We also release weights for the Fara1.5 models under MIT license, making SoTA computer use accessible for all beyond closed API-only systems.

Figures

Figures reproduced from arXiv: 2606.20785 by Ahmed Awadallah, Akshay Nambi, Alexey Taymanov, Andrew Zhao, Aravind Rajeswaran, Corby Rosset, Hussein Mozannar, Luiz do Valle, Sahil Gupta, Spencer Whitehead, Vibhav Vineet, Yadong Lu, Yash Lara, Yash Pandya, Zach Nussbaum.

Figure 1
Figure 1. Figure 1: Task success rate on Online-Mind2Web and WebVoyager for similarly sized CUA models. Fara1.5-9B reaches [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FaraGen1.5 pipeline. Phase 1 instantiates tasks in two kinds of environments: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Sample FaraEnvs for Calendar and ML Management environments populated with realistic data generated by [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (Left) Agentic training data collected by FaraGen1.5 and its predecessor over time, reported as the number [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: One step of Fara1.5’s observe-think-act loop. The model observes the recent browser state (up to three [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Supervised fine-tuning input and loss mask. The model conditions on the full conversation history but consumes [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (Left) WebVoyager and Online-Mind2Web success rate as a function of model size for the Fara1.5 family. Both [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Plots of average number of steps per task and success rate as a function of number of steps. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of the number of steps split by outcome, resolved by model size (split violins: left/green = correct, [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Number of steps for successful trajectories on the examples solved by all three model sizes. Conditioned on success, the distributions are nearly identical across sizes, indicating smaller models are not inherently less step-efficient. [1, 5) [5, 8) [8, 12) [12, 20) [20, 35) [35, 100] Number of steps (bin) 0% 20% 40% 60% 80% 100% Success rate WebVoyager [1, 5) [5, 8) [8, 12) [12, 20) [20, 35) [35, 100] Nu… view at source ↗
Figure 11
Figure 11. Figure 11: Success rate as a function of trajectory length, broken out by model size. The mapping is largely independent [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Per-example pass@k (k = 1, 2, 3) with examples sorted from easiest to hardest. The right tail (pass rate → 0) is the set of hardest examples; larger models solve more of it, and additional attempts (pass@2, pass@3) recover more examples [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Median number of steps per example (faint points) and a rolling average (solid lines) along the same easiest-to [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Selected screenshots from hotels_head_orbitz_8. The agent visited the requested site, entered the hotel name and dates, and read the sold-out banner directly off the property card before terminating. A.6.2 Process-only success: jobs_apply_apply_1219 Task. Help me apply for a chemistry research scientist position in Madison, WI found on LinkedIn and output five listings that would have me be an early appli… view at source ↗
Figure 15
Figure 15. Figure 15: Selected screenshots from jobs_apply_apply_1219. The trajectory follows a clean process: navigate to the filtered LinkedIn search, dismiss the modal, collect candidates via pause_and_memorize_fact while scrolling. The outcome judge nevertheless marks this a failure because one returned listing is located in Verona, WI rather than the Madison, WI specified by the task. A.7 Failure Analysis Expanding on the… view at source ↗
Figure 16
Figure 16. Figure 16: Task: Please generate the next move according to the UI screenshot and instruction. Instruction: Select the handle located at the top of the text box containing the text "Ferry ride." The concentric red and lime green circle denotes where the Jedi dataset incorrectly says the object of interest is located, the blue circle is the correct location of the object of interest. This is an inaccurate error [PIT… view at source ↗
Figure 17
Figure 17. Figure 17: Task: The Text field within a table cell.’s intended function: The primary function of this element is to display a specific metric value, likely representing a financial figure or performance metric. Users can view this value to assess or compare it with other metrics in the table. This is an error because the task is not unique for the instruction’s description of the table cell [PITH_FULL_IMAGE:figure… view at source ↗
Figure 18
Figure 18. Figure 18: Form-filling data pipeline. Tasks are seeded from real Tally forms and critical-point prompts, solved by the [PITH_FULL_IMAGE:figures/full_fig_p037_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: A form-filling trajectory where the user authorizes submission but omits a required field. The agent fills [PITH_FULL_IMAGE:figures/full_fig_p038_19.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale

    cs.AI 2026-07 conditional novelty 6.5

    Deep, capability-targeted, co-evolving synthetic environments raise a 9B computer-use agent from 36.5% to 67.1% and enable RL gains the live web cannot supply.

Reference graph

Works this paper leans on

23 extracted references · 1 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Feel free to create mock user information and make assumptions freely

  2. [2]

    An Illusion of Progress? Assessing the Current State of Web Agents. arXiv:2504.01382 [cs.AI] https://arxiv.org/abs/2504.01382 An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang,...

    arXiv 2025

  3. [3]

    Make sure any assumptions you make are reasonable and consistent with the conversation history

  4. [4]

    Make sure to answer all questions and provide all necessary details

  5. [5]

    If the agent needs personal details (name, email, address, etc.), provide realistic fake information Guidelines for approval requests:

  6. [6]

    If the agent asks for preferences, make a clear choice

  7. [7]

    What is the best pair of noise canceling headphones

    Do not ever refer to https:// URLs directly in your feedback to the agent. Instead, refer to the general domain name (i.e, Google Flights) instead of an https URL. Examples of good follow-up tasks: - If the previous task was "What is the best pair of noise canceling headphones", a good follow-up: "can you see if this headphone is available on Amazon", a s...

    2026

  8. [8]

    If the agent is requesting approval to proceed with an action (e.g., making a purchase, submitting a form), make sure it aligns with the original task

  9. [9]

    If the action seems reasonable and safe, approve it

  10. [10]

    Table 8: Task-proposal dimensions for online environments

    If the action seems risky, unnecessary, or misaligned with the original task, deny it Fara-1.5 CUA model 27 Dimension # values Who decides Values Site 100s sampled popularity-weighted across 18 domain categories (e-commerce, travel, search-info, forms, food-delivery, social-media, government, finance, healthcare, news, education, real-estate, automotive, ...

  11. [11]

    Is naturally related to the conversation history and previous tasks/outputs

  12. [12]

    Could reasonably be asked by a user in this context

  13. [13]

    Leverages the current page state or information visible in the screenshot, but requires navigation beyond the current page

  14. [14]

    tell me more

    Is specific and actionable (not vague like "tell me more")

  15. [15]

    Requires actions (clicking, navigating to new page), beyond just reading the page

  16. [16]

    The follow-up task should be slightly simpler than the original task but within similar complexity

  17. [18]

    Selection 1.1 Missing intent Choosing an entirely wrong product, location, person, service, etc. 1.2 Unauthorized substitution Silently swapping an unavailable item for a similar alternative without reporting 1.3 Wrong action type Performing the wrong interaction on the correct entity 1.4 Wrong values / constraint violation Incorrect parameters, unsatisfi...

  18. [19]

    Hallucination 2.1 Output contradiction Evidence shows X, but agent claims not-X; includes misinterpreting page/tool content 2.2 Action contradiction Agent claims action was performed but evidence contradicts; action was achievable 2.3 Output fabrication Agent claims a fact with zero evidentiary basis; complete invention 2.4 Action fabrication Agent claims...

  19. [20]

    Execution & Strategy 3.1 Computational mistakes Correct methodology but wrong answer due to miscounting, arithmetic, or misreading 3.2 Platform non-compliance Not attempting the specified platform or silently switching sources 3.2.1 API-Sniffing Agent navigates to a site’s underlying JSON/REST API instead of the GUI URL the task implied, when the task / p...

  20. [21]

    Critical Point 4.1 Premature stop Stopped at critical point despite user explicitly granting permission 4.2 Violation Crossed transactional boundary without permission 4.3 Other Critical point error not covered above

  21. [22]

    Side-Effect 5.1 Unsolicited Any lasting modification, enrollment, or addition not requested 5.2 Other Side-effect error not covered above

  22. [23]

    Ferry ride

    Tool Interaction 6.1 Invalid invocation Tool call with wrong arguments (action exists but args are incorrect) 6.2 Hallucinated action Agent invokes a tool/action that does not exist in the action space 6.3 Intent-action mismatch Agent’s stated intent differs from actual tool call issued 6.4 Grounding error Correct target identified but (x, y) coordinates ...

  23. [2025]

    AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials. InThe Thirteenth International Conference on Learning Representations.https://openreview.net/forum?id=EEgYUccwsV Fara-1.5 CUA model 23 Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024. Aguvis: Unified Pure Vision...

    Pith/arXiv arXiv 2024