Pith. sign in

REVIEW 3 major objections 4 minor 56 references

Benchmarking LLM Judges for Mobile Agent Evaluation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Judge quality scores predict how well LLM judges evaluate mobile agents.

desk verdict A solid mobile-agent judge benchmark with an overreaching abstract: the evaluation-side meta-correlation is convincing, but the RL validation is too underpowered to support the paper's headline claim. read the letter →

arxiv 2608.11434 v1 pith:UTOCIAGE submitted 2026-08-11 cs.AI cs.CLcs.CV

Ziqiang Wan , Li Gu , Zhixiang Chi , Zhi Liu , Seyed Mehdi Ayyoubzadeh , Yuanhao Yu , Yang Wang This is my paper · ORCID
classification cs.AIcs.CLcs.CV
keywords LLM-as-a-judgemobileagentsGUIagentevaluationbenchmarkrewardmodelreinforcementlearningjudgereliabilitymultimodaltrajectories
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mobile agent benchmarks increasingly trust LLM judges to decide whether a phone task was completed, and this paper asks whether those judges can be trusted and whether their quality can be predicted cheaply. To answer, it builds MobileJudgeBench, 931 human-annotated trajectories drawn from six existing benchmarks, four agent models, and 68 apps, and evaluates six judge methods across five LLM backbones. Its central thesis is that benchmark quality metrics—especially F1 and balanced accuracy—predict real-world judge utility: they track how reliably a judge ranks agents (F1 vs. ranking fidelity, Spearman $\rho_s=0.90$) and how well the judge behaves as a reward signal in on-policy reinforcement learning. A second thesis is that elaborate judge pipelines are not worth much: a simple baseline that samples screenshots matches or beats purpose-built methods, with the LLM backbone driving most of the remaining difference. If these claims hold, judge evaluation becomes a first-class benchmarking concern rather than an assumed background detail.

What carries the argument

The load-bearing object is MobileJudgeBench itself: 931 human-annotated success/failure labels over trajectories from six mobile benchmarks, four agent models, and 68 apps, with every trajectory stored as a sequence of screenshots, actions, UI trees, and agent reasoning. Its mechanism is a two-stage evaluation: each judge variant is first scored with trajectory-level classification metrics (accuracy, precision, recall, F1, balanced accuracy), and then those metrics are validated against two downstream uses of judges—ranking fidelity and success-rate error for agent evaluation, and success rate of a GRPO-trained agent when the judge is the reward. A meta-correlation across the 30 variants is what turns the benchmark from a static dataset into a predictive instrument.

What would settle it

Train an agent with a fifth reward signal that scores high on F1 but is deliberately biased toward one agent class, and check whether ranking fidelity or RL success still tracks F1; if a low-F1, high-precision judge outperforms a high-F1 judge in training, the paper's ordering claim collapses. A lighter check: add another reward condition to the RL study and see whether the easy-set success-rate ordering still follows judge accuracy.

Watch

Extended reading notes

Core claim

The paper's central claim is that MobileJudgeBench scores predict how useful a judge is in actual mobile-agent evaluation and training. Across 30 judge variants (6 methods × 5 backbones), F1 correlates with agent-ranking fidelity at 0.90 and balanced accuracy correlates with success-rate estimation error at −0.79, while precision alone has no predictive power. In an on-policy RL study with four reward sources on AndroidWorld's easy tasks, judge accuracy orders the final agent performance: the rule-based checker reaches 54.6% success, followed by a GPT-5-mini judge at 45.4%, a GPT-5.2 judge at 42.6%, and a Qwen judge at 39.9%. The paper also claims that judge failure modes are backbone-specific rather than method-specific: GPT-based judges over-reject (false-negative-heavy, driven by last-frame anchoring and unfamiliar success states), while Qwen-based judges over-accept (false-positive-heavy, driven by constraint violations and partial completion), with visual surface matching a shared weakness.

Load-bearing premise

The claim that judge accuracy predicts training outcomes rests on only four reward conditions, three of them the same judge method with different backbones, in a single Android environment, so the ordering could change with a different judge method or app suite.

Editorial extensions

If this is right

  • Benchmarks should report judge reliability alongside agent scores, since judge accuracy between 76% and 91% can shift agent rankings by up to 13 positions.
  • Simple, cheap judges—uniformly sampled screenshots with a strong backbone—are a defensible default for mobile-agent evaluation; elaborate pipelines demonstrated no consistent advantage.
  • F1 and balanced accuracy, not precision alone, are the metrics to report for judge quality in evaluation settings.
  • When judges are used as RL rewards, false-positive control matters: the equal-accuracy judge with higher precision yielded higher best-checkpoint training success.
  • Because fault profiles depend on the backbone, a judge's precision-recall character should be matched to its application, not chosen by method alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the meta-correlation generalizes, judge benchmarks like this one could be used as a cheap filter for reward-model selection: score candidate judges on a small annotated set before spending GPU-hours on RL, an extension the paper does not run.
  • The opposite failure profiles suggest a testable calibration strategy: choose or adjust judges by precision-recall target depending on whether the use is leaderboard ranking (balanced) or reward signaling (false-positive averse); the paper points this way but does not test it directly.
  • The RL validation compares only one judge method across three backbones plus an oracle in one environment; a natural next step would vary judge methods and environments to see whether the accuracy-to-training-success ordering is stable.
  • The shared 'surface UI match' failure suggests that verification-style judges that check actual system state, rather than screenshots alone, could fix a common failure mode; this is implied by the taxonomy but not implemented here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces MobileJudgeBench, a benchmark of 931 human-annotated mobile-agent trajectories drawn from six existing benchmarks, four agent models, and 68 apps. It evaluates six LLM-as-judge methods (five adapted from prior benchmarks plus a simple baseline) across five LLM backbones, producing 30 judge variants. The main findings are: (1) a simple screenshot-based baseline is competitive with or better than purpose-built judge pipelines; (2) judge quality metrics, especially F1 and balanced accuracy, correlate with agent-ranking fidelity in a meta-correlation across 30 variants; (3) an on-policy RL experiment on AndroidWorld with four reward conditions suggests that judge accuracy orders downstream training performance; and (4) a failure analysis identifies contrasting conservative and permissive failure profiles for GPT- and Qwen-based judges. The authors release trajectories, annotations, and code.

Significance. If the central claims hold, this is a valuable contribution: it addresses an underexamined but important issue, provides a reusable judge-evaluation benchmark, and offers an unusually careful evaluation-side validation with task-cluster bootstrap confidence intervals, leave-one-out checks over backbones and methods, and a nondeterminism sensitivity analysis. The failure taxonomy is also well grounded, with high double-coded inter-rater agreement (Cohen's kappa = 0.957). The evaluation-side meta-correlation (Section 5.2.1, Table 4) is the strongest part of the paper and is not circular: judges are scored against human labels, and downstream evaluation reliability is measured with a rule-based checker. The training-side claim in the abstract, however, is substantially stronger than the evidence in Section 5.2.2, which rests on only four reward conditions, one environment, one judge method, and a best-checkpoint convention that collapses at the fixed final checkpoint.

major comments (3)
  1. [Abstract and §5.2.2] The abstract states that benchmark quality metrics 'reliably predict real-world judge utility' including 'downstream performance when judges serve as reward signals for on-policy reinforcement learning.' Section 5.2.2 supports this with only four reward conditions (one rule-based oracle and three backends of the same simple-baseline judge), all on a single environment (AndroidWorld easy set). With N=4, a monotone ordering is weak evidence, and the paper's own Limitations section concedes that training-time comparisons across judge methods are narrower than the offline study. Either add additional reward conditions or judge methods/environments, or reword the abstract and conclusion to describe the training-side evidence as suggestive and preliminary.
  2. [§7 and Appendix E, Table 10] The conclusion states that 'the judge with stronger false-positive control reached a higher best-checkpoint success rate.' This is internally inconsistent with the appendix: at the fixed final checkpoint (step 300), GPT-5.2 and Qwen tie at 25.9% on the full 116-task suite. The separation of 2.7pp on the easy set exists only under the best-checkpoint convention, which the paper itself labels 'suggestive rather than conclusive.' The conclusion and abstract should carry this caveat explicitly, and the best-checkpoint convention should be stated wherever the precision-based ordering is discussed.
  3. [§5.2.2 and Appendix E] The rule-based reward condition coincides with the evaluation signal used to measure training success, so its top ranking is an upper-bound control rather than evidence that judge benchmark metrics predict training outcomes. Among the three LLM conditions, all use the same simple baseline judge and differ only in backbone. This design cannot support a general claim that judge quality metrics predict downstream training performance across judge methods. The paper acknowledges some of this in the Limitations, but the abstract's unqualified wording should be revised to reflect the narrow scope of the RL validation.
minor comments (4)
  1. [§4.2 and Table 3] The ablation table states that 16 screenshots at 1/4 resolution, 48 at 1/16, and 192 at 1/64 are 'approximately equal cost,' but the cost proxy is not defined. If the proxy is total input pixels or tokens, that should be stated explicitly; otherwise the iso-cost comparison is hard to interpret.
  2. [Appendix F.2] The hard-core failure definition requires at least 4 of 5 judge methods to err, but no sensitivity analysis is given for this threshold. Because the failure sets are small (48 and 78 cases), reporting the taxonomy under a 3-of-5 threshold would help establish that the qualitative conclusions are not threshold artifacts.
  3. [§5.2.1 and Figure 2] The text reports the best judge's agreement with human success rates as 'rho = 0.97' without consistently distinguishing Spearman rho from the rho_s used elsewhere. Please define the correlation coefficient in the caption or text.
  4. [Figures 8 and 9] The labels 'GT:SuccessPred:Failure' are missing a space after 'Success'; minor formatting issue that should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: judge quality is scored against independent human labels and downstream utility is measured by a rule-based checker, not by the judges themselves.

full rationale

The paper's central claims are empirical rather than derivational. Judge methods are scored on 931 human-annotated trajectories with pairwise annotator agreement 88.4%, and the meta-correlation in §5.2.1 is computed across 30 judge variants between trajectory-level quality metrics and agent-level reliability metrics; the two sets of metrics are computed from the same judge predictions but are not definitionally linked (F1 does not equal Spearman rho or MAE by construction). The RL validation in §5.2.2 compares four reward conditions on AndroidWorld and evaluates all conditions with the ground-truth rule-based checker ('All conditions are evaluated using the ground-truth rule-based checker (not the LLM judge), ensuring fair comparison'), so the judge is not scoring its own training outcomes. The paper explicitly flags the equal-accuracy GPT-5.2 vs Qwen comparison as convention-dependent and 'suggestive rather than conclusive,' which is a statistical-power caveat, not a circular step. No load-bearing self-citations or imported uniqueness theorems appear; the adapted judge methods are drawn from external benchmarks and the simple baseline is designed and ablated in this paper. The main limitation (N=4 training conditions) concerns strength of evidence, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical benchmark paper with no theoretical derivation and no invented physical entities. The central claims rest on the quality of human labels, the fidelity of adapted judge methods, the representativeness of the sampled benchmarks and agents, and the validity of the AndroidWorld rule-based checker for RL evaluation. Design choices such as the baseline input budget, the hard-core failure threshold, and the best-checkpoint convention act as hand-selected parameters that affect specific reported numbers.

free parameters (3)
  • Baseline judge default input budget = 48 uniformly sampled screenshots, max long edge 600px
    Chosen by hand as the default for the simple baseline judge; the ablation in §5.1 shows 16 screenshots at 1/4 resolution performs better, so this is not an optimized setting.
  • Hard-core failure threshold = at least 4 of 5 judge methods must err
    Chosen to define 'hard-core failures' in §5.3.1; directly determines the failure counts (48 GPT vs 78 Qwen) and the taxonomy distribution.
  • Best-checkpoint selection for RL comparison = checkpoint with highest seed-averaged easy-set success rate
    Used in §5.2.2 to order GPT-5.2 above Qwen; at the fixed final checkpoint they tie on the full task set (Appendix E), so the precision-ordering result depends on this choice.
assumptions (4)
  • domain assumption Human annotations (88.4% pairwise agreement, discussed to consensus) are treated as ground truth for trajectory success.
    The entire benchmark and all accuracy metrics rest on the reliability of the 9 graduate-student annotators; noise in labels propagates into every reported metric. See §3.2.
  • domain assumption The adapted judge methods faithfully represent their original benchmark judges.
    SPA-Bench is evaluated only with its fine-stage MLLM judge because the coarse stage needs task-specific annotations; AgentRewardBench, A3 essential states, etc. are adapted to mobile trajectories. If these adaptations are unfaithful, the comparison between simple baseline and purpose-built methods is biased. See §4.1.
  • domain assumption The six selected benchmarks and four agent models are representative of mobile-agent evaluation practice.
    The 931 trajectories come from 289 tasks in 6 benchmarks generated by 4 agents; generalization to other tasks, apps, and judges is assumed. See §3.1.
  • domain assumption The AndroidWorld rule-based checker provides ground truth for RL evaluation.
    Training outcomes are measured with the built-in state checker, not human labels; the rule-based reward condition is thus an upper bound. See §5.2.2 and Appendix E.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking LLM Judges for Mobile Agent Evaluation." pith.science (2026). https://pith.science/paper/UTOCIAGE

@misc{pith2026260811434,
  author       = {Pith},
  title        = {Pith review of: Benchmarking LLM Judges for Mobile Agent Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UTOCIAGE}},
  note         = {Machine review of arXiv:2608.11434}
}
read the original abstract

Mobile agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile agent trajectories remains largely unexamined. We introduce MobileJudgeBench, a benchmark for systematically evaluating LLM-as-judge methods on mobile agent trajectories. Our benchmark comprises 931 human-annotated trajectories spanning 6 mobile agent benchmarks, 4 agent models, and 68 apps. We evaluate 6 judge methods (five adapted from SPA-Bench, A3 with two modes, AndroidArena, and AgentRewardBench, plus a simple baseline we design) across multiple LLM backends. Our experiments reveal three key findings. First, a simple baseline judge with sampled screenshots is competitive with, and often exceeds, purpose-built methods, indicating that more elaborate judge pipelines do not consistently improve judge quality; among competitive methods, the LLM backbone is the primary driver. Second, benchmark quality metrics reliably predict real-world judge utility: they correlate with both agent ranking fidelity for evaluation and downstream performance when judges serve as reward signals for on-policy reinforcement learning. Third, failure analysis across two LLM backends uncovers qualitatively opposite failure profiles, one conservative and the other permissive, linked to the backbone's precision-recall characteristics.

Figures

Figures reproduced from arXiv: 2608.11434 by the authors.

Figure 1
Figure 1. Overview of MOBILEJUDGEBENCH. We construct a judge benchmark from 931 human-annotated mobile agent trajectories spanning 6 benchmarks (left, §3), evaluate 6 LLM-as-judge methods across 5 backends (middle, §5.1), and validate that benchmark metrics predict judge utility for agent evaluation (§5.2.1) and on-policy training (§5.2.2), complemented by a root-cause failure analysis (§5.3) (right). Automatic evaluation of … view at source ↗
Figure 2
Figure 2. Judge quality predicts evaluation reliability (30 method [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. On-policy RL training curves with different judge reward signals (mean [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Root cause taxonomy for hard-core failures. The GPT set is dominated by false negatives (last-frame [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Annotation platform. Left: trajectory video view with task description, agent reasoning, and playback [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Task-cluster bootstrap 95% CIs for the meta-correlation across the 30 judge variants. Green: interval [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Per-seed easy-set accuracy for each reward condition. Seed variance is moderate; the ranking Rule-based [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Representative false negative failure examples (4 categories). Judges incorrectly predict failure despite [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Representative false positive failure examples (3 categories). Judges incorrectly predict success despite [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 10 canonical work pages

  1. [1]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  2. [2]

    arXiv preprint arXiv:2411.15594 , year=

    A survey on llm-as-a-judge , author=. arXiv preprint arXiv:2411.15594 , year=

    arXiv

  3. [3]

    arXiv preprint arXiv:2412.05579 , year=

    Llms-as-judges: a comprehensive survey on llm-based evaluation methods , author=. arXiv preprint arXiv:2412.05579 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2410.10934 , year=

    Agent-as-a-judge: Evaluate agents with agents , author=. arXiv preprint arXiv:2410.10934 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2407.18370 , year=

    Trust or escalate: Llm judges with provable guarantees for human agreement , author=. arXiv preprint arXiv:2407.18370 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2502.01534 , year=

    Preference leakage: A contamination problem in llm-as-a-judge , author=. arXiv preprint arXiv:2502.01534 , year=

    arXiv

  7. [7]

    arXiv preprint arXiv:2310.17631 , year=

    Judgelm: Fine-tuned large language models are scalable judges , author=. arXiv preprint arXiv:2310.17631 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2407.00215 , year=

    Llm critics help catch llm bugs , author=. arXiv preprint arXiv:2407.00215 , year=

    arXiv

Show all 56 references
  1. [9]

    arXiv preprint arXiv:2504.08942 , year=

    Agentrewardbench: Evaluating automatic evaluations of web agent trajectories , author=. arXiv preprint arXiv:2504.08942 , year=

  2. [10]

    arXiv preprint arXiv:2404.06474 , year=

    Autonomous evaluation and refinement of digital agents , author=. arXiv preprint arXiv:2404.06474 , year=

    arXiv

  3. [11]

    arXiv preprint arXiv:2503.02403 , year=

    Autoeval: A practical framework for autonomous evaluation of mobile agents , author=. arXiv preprint arXiv:2503.02403 , year=

  4. [12]

    arXiv preprint arXiv:2504.01382 , year=

    An illusion of progress? assessing the current state of web agents , author=. arXiv preprint arXiv:2504.01382 , year=

  5. [13]

    Agentic Reward Modeling: Verifying

    Cui, Chaoqun and Huang, Jing and Wang, Shijing and Zheng, Liming and Kong, Qingchao and Zeng, Zhixiong , journal=. Agentic Reward Modeling: Verifying

  6. [14]

    arXiv preprint arXiv:2509.21823 , year=

    ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration , author=. arXiv preprint arXiv:2509.21823 , year=

    arXiv

  7. [15]

    The Thirteenth International Conference on Learning Representations , year=

    AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents , author=. The Thirteenth International Conference on Learning Representations , year=

  8. [16]

    The Thirteenth International Conference on Learning Representations , year=

    SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation , author=. The Thirteenth International Conference on Learning Representations , year=

  9. [17]

    2025 , eprint=

    A3: Android Agent Arena for Mobile GUI Agents , author=. 2025 , eprint=

    2025

  10. [18]

    Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining , pages=

    Understanding the weakness of large language model agents within a complex android environment , author=. Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining , pages=

  11. [19]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Androidlab: Training and systematic benchmarking of android autonomous agents , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  12. [20]

    arXiv preprint arXiv:2404.16660 , year=

    Benchmarking Mobile Device Control Agents Across Diverse Configurations , author=. arXiv preprint arXiv:2404.16660 , year=

    arXiv

  13. [21]

    Advances in Neural Information Processing Systems , volume=

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments , author=. Advances in Neural Information Processing Systems , volume=

  14. [22]

    The Twelfth International Conference on Learning Representations , year=

    WebArena: A Realistic Web Environment for Building Autonomous Agents , author=. The Twelfth International Conference on Learning Representations , year=

  15. [23]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  16. [24]

    Advances in Neural Information Processing Systems , volume=

    Mind2web: Towards a generalist agent for the web , author=. Advances in Neural Information Processing Systems , volume=

  17. [25]

    Advances in Neural Information Processing Systems , volume=

    Androidinthewild: A large-scale dataset for android device control , author=. Advances in Neural Information Processing Systems , volume=

  18. [26]

    AgentBench: Evaluating

    Xiao Liu and Hao Yu and Hanchen Zhang and Yifan Xu and Xuanyu Lei and Hanyu Lai and Yu Gu and Hangliang Ding and Kaiwen Men and Kejuan Yang and Shudan Zhang and Xiang Deng and Aohan Zeng and Zhengxiao Du and Chenhui Zhang and Sheng Shen and Tianjun Zhang and Yu Su and Huan Sun...

    2024

  19. [27]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Mobile-bench: An evaluation benchmark for llm-based mobile agents , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  20. [28]

    Advances in Neural Information Processing Systems , volume=

    On the effects of data scale on ui control agents , author=. Advances in Neural Information Processing Systems , volume=

  21. [29]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  22. [30]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

  23. [31]

    arXiv preprint arXiv:2212.08073 , year=

    Constitutional ai: Harmlessness from ai feedback , author=. arXiv preprint arXiv:2212.08073 , year=

    arXiv

  24. [32]

    Advances in Neural Information Processing Systems , volume=

    Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  25. [33]

    Zehan Qi and Xiao Liu and Iat Long Iong and Hanyu Lai and Xueqiao Sun and Jiadai Sun and Xinyue Yang and Yu Yang and Shuntian Yao and Wei Xu and Jie Tang and Yuxiao Dong , booktitle=. Web. 2025 , url=

    2025

  26. [34]

    arXiv preprint arXiv:2509.18119 , year=

    Mobilerl: Online agentic reinforcement learning for mobile gui agents , author=. arXiv preprint arXiv:2509.18119 , year=

  27. [35]

    arXiv preprint arXiv:2507.05720 , year=

    Mobilegui-rl: Advancing mobile gui agent through reinforcement learning in online environment , author=. arXiv preprint arXiv:2507.05720 , year=

    arXiv

  28. [36]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Rewardbench: Evaluating reward models for language modeling , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    2025

  29. [37]

    International Conference on Machine Learning , pages=

    Scaling laws for reward model overoptimization , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  30. [38]

    arXiv preprint arXiv:2409.15922 , year=

    The dark side of rich rewards: Understanding and mitigating noise in vlm rewards , author=. arXiv preprint arXiv:2409.15922 , year=

  31. [39]

    The Eleventh International Conference on Learning Representations , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. The Eleventh International Conference on Learning Representations , year=

  32. [40]

    2023 , eprint=

    AppAgent: Multimodal Agents as Smartphone Users , author=. 2023 , eprint=

    2023

  33. [41]

    2023 , eprint=

    CogAgent: A Visual Language Model for GUI Agents , author=. 2023 , eprint=

    2023

  34. [42]

    arXiv preprint arXiv:2501.12326 , year=

    Ui-tars: Pioneering automated gui interaction with native agents , author=. arXiv preprint arXiv:2501.12326 , year=

    arXiv

  35. [43]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  36. [44]

    arXiv preprint arXiv:2403.07718 , year=

    Workarena: How capable are web agents at solving common knowledge work tasks? , author=. arXiv preprint arXiv:2403.07718 , year=

    arXiv

  37. [45]

    arXiv preprint arXiv:2410.02907 , year=

    Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild , author=. arXiv preprint arXiv:2410.02907 , year=

    arXiv

  38. [46]

    2025 , eprint=

    Qwen2.5-VL Technical Report , author=. 2025 , eprint=

    2025

  39. [47]

    arXiv preprint arXiv:2406.12793 , year=

    Chatglm: A family of large language models from glm-130b to glm-4 all tools , author=. arXiv preprint arXiv:2406.12793 , year=

    arXiv

  40. [48]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  41. [49]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    arXiv

  42. [50]

    Shi, Wentao and Wang, Yu and Zhao, Yuyang and Chen, Yuxin and Feng, Fuli and Hao, Xueyuan and Su, Xi and Gu, Qi and Su, Hui and Cai, Xunliang and He, Xiangnan , journal=

  43. [51]

    Sui, Yifan and Huang, Xin and Li, Hongbing and Xu, Fang and Lv, Jiahe and Yan, Haolong and Shen, Yeqing and Liu, Litao and Fan, Zhimin and Meng, Ziyang and Wang, Jia and Qi, Junbo and Tan, Kaijun and Ge, Zheng and Zhang, Xiangyu and Jiang, Daxin and Yoshie, Osamu , journal=

  44. [52]

    2025 , note=

    Gemini 3 Flash Model Card , author=. 2025 , note=

    2025

  45. [53]

    2025 , note=

    Claude Sonnet 4.5 System Card , author=. 2025 , note=

    2025

  46. [54]

    arXiv preprint arXiv:2507.01006 , year=

    arXiv

  47. [55]

    Zheng, Yaowei and Lu, Junting and Wang, Shenzhi and Feng, Zhangchi and Kuang, Dongdong and Xiong, Yuwen and Zhang, Richong , year =

  48. [56]

    Sheng, Guangming and Zhang, Chi and Ye, Zilingfeng and Wu, Xibin and Zhang, Wang and Zhang, Ru and Peng, Yanghua and Lin, Haibin and Wu, Chuan , journal =

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.