Pith. sign in

REVIEW 2 major objections 5 minor 63 references

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Few human demonstrations adapt web agents to new sites, lifting success by up to 7.21 percentage points.

desk verdict The ICL results are solid and useful, but the meta-learning claim is confounded by an unequal adaptation protocol and should not be taken as supported in the current draft. read the letter →

arxiv 2411.13451 v1 pith:RZZ7AAYD submitted 2024-11-20 cs.AI cs.CLcs.LG

Gaurav Verma , Rachneet Kaur , Nishan Srishankar , Zhen Zeng , Tucker Balch , Manuela Veloso This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords webagentsmultimodallargelanguagemodelsfew-shotlearninghumandemonstrationsin-contextmeta-learningMind2WebVisualArena
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that web agents can be adapted to unseen websites and domains with just one or two human demonstrations, rather than requiring more large-scale pre-training or fine-tuning. On the Mind2Web and VisualWebArena benchmarks, adding 1 or 2 multimodal demonstrations improves task success rate by 3.36 to 7.21 percentage points, a relative increase of 21.03% to 65.75% over baseline agents. The AdaptAgent framework packages demonstrations in two ways: in-context examples for proprietary multimodal models like GPT-4o used through SeeAct, and meta-learning for open-weights models like CogAgent. The broader claim is that few-shot adaptability is a practical complement to scale-based generalization for proprietary and unfamiliar web platforms.

What carries the argument

The central objects are the human demonstration and the meta-learning update that turns it into adaptation. A demonstration is decomposed into (visual snapshot, filtered top-50 HTML elements, action) triples and, for proprietary models, inserted into the SeeAct prompt as an in-context example. For open-weights models, the machinery is first-order model-agnostic meta-learning: an inner loop fine-tunes the agent on two tasks per website, an outer loop updates the original parameters using loss on held-out tasks from that website, and training on 292 such tasks produces a prior that adapts to a new website with two demonstrations. The paper also varies how inner-loop and outer-loop tasks are paired—intra-website, inter-website, or hybrid—to steer whether adaptation generalizes within a site or across a domain.

What would settle it

Take CogAgent-FT (DE), the conventionally fine-tuned model trained on the same 292 examples, and give it the identical two-demonstration fine-tuning step on each Mind2Web cross-website and cross-domain test site before evaluation. If it matches or exceeds CogAgent-FOMAML's success rates, the paper's central attribution to meta-learning is unsupported.

Watch

Extended reading notes

Core claim

At its core, the paper reports that a small number of multimodal human demonstrations—each recorded as a sequence of page snapshots, filtered HTML elements, and the human's chosen action—can shift a web agent's success on unseen sites and domains. For proprietary agents, AdaptAgent inserts one such demonstration into the SeeAct prompt for GPT-4o and observes consistent gains across Mind2Web's cross-task, cross-website, and cross-domain sets as well as VisualWebArena. For open-weights agents, it applies first-order model-agnostic meta-learning (FOMAML) over tasks sampled from 73 websites, then adapts the meta-learned model with two demonstrations per new website or domain; the resulting CogAgent-FOMAML beats a conventionally fine-tuned CogAgent trained on the same amount of data and, in most settings, the larger fine-tuned baseline. Additional ablations show that multimodal demonstrations outperform text-only ones, that the pairing of inner-loop and outer-loop tasks during meta-learning changes whether generalization favors cross-website or cross-domain settings, and that adding more demonstrations yields diminishing returns.

Load-bearing premise

The open-weights result assumes that the improvement attributed to meta-learning is not actually just the effect of fine-tuning on two demonstration tasks before evaluation, because the conventionally fine-tuned baseline receives no such adaptation.

Editorial extensions

If this is right

  • If the gains hold, web agents can be tailored to proprietary or enterprise websites with a single recorded human session, reducing the need for task-specific annotation campaigns.
  • Multimodal demonstrations, not just text transcripts, are the effective unit of adaptation, so recording screenshots along with clicks is worth the extra prompt cost.
  • Meta-learning's data-selection strategy becomes a tunable choice: intra-website pairing favors cross-website generalization, inter-website pairing favors cross-domain, and hybrid balances the two.
  • The saturating gains with more demonstrations suggest a practical operating point of one to two demonstrations per target site, balancing accuracy against inference cost.
  • Even with adaptation, best task success remains below 25%, so few-shot adaptation is an improvement within a still-limited range, not a full solution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is to select demonstrations automatically—by coverage of common actions or by difficulty—since the paper's meta-learning data-selection ablation shows that which tasks are paired changes the kind of generalization achieved.
  • The same demonstration format could encode compliance or accessibility rules for enterprise sites by recording one example of the desired behavior, an application the paper leaves implicit.
  • The live-environment gains on VisualWebArena suggest adaptation helps not just offline trajectory matching but interactive success; deploying adapted agents on additional live sites would test that directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes AdaptAgent, a framework for adapting multimodal web agents with few-shot human demonstrations. For proprietary models (SeeAct with GPT-4o), it adds one multimodal in-context demonstration to the prompt; for open-weights models (CogAgent), it meta-trains with FOMAML and then adapts on two demonstrations from the target website or domain. The authors evaluate on Mind2Web and VisualWebArena, reporting absolute gains of 3.36 to 7.21 percentage points in task success rate over non-adapted baselines, and they provide ablations on demonstration modality, meta-learning data-selection strategies, and the number of demonstrations.

Significance. If the results hold, the in-context-learning component is a practically useful and inexpensive way to adapt proprietary web agents to new websites and domains, and the multimodal-versus-text demonstration comparison is informative. The paper also provides useful implementation details, including prompts and LoRA hyperparameters. However, the claim that FOMAML meta-learning produces a better starting point than conventional fine-tuning for open-weights agents is not supported by the current evaluation because the comparison is confounded by unequal adaptation protocols. The central meta-learning contribution therefore needs additional experimental work before the headline claims can be accepted.

major comments (2)
  1. [Section 4, Table 1] The comparison between CogAgent-FOMAML and the CogAgent-FT baselines is confounded by the adaptation protocol. Section 4 states that after meta-learning, CogAgent-FOMAML is fine-tuned on 2 tasks from each target website (or domain) before evaluation, while CogAgent-FT and CogAgent-FT (DE) receive no target-site adaptation. The gains in Table 1 (e.g., cross-website Overall SR 16.96 vs. 12.42) could therefore be caused entirely by the two adaptation demonstrations rather than by the meta-learned prior. The required control is a baseline, ideally CogAgent-FT and CogAgent-FT (DE), fine-tuned on the same two target-site tasks and then evaluated on the remaining tasks. This is not a minor nuance: on the cross-task split, where no target adaptation is applied, CogAgent-FOMAML (11.89) does not beat CogAgent-FT (13.36), so the claim that the meta-learned prior itself is superior to conventional fine-tuning is unsupported.
  2. [Section 5, footnote 2 and Table 3] The statement that CogAgent-FOMAML outperforms CogAgent-FT 'across all tests except for Mind2Web cross-task' is misleading because every non-cross-task comparison includes the two target-site adaptation demonstrations for FOMAML but none for FT. The only unconfounded evidence about the learned prior is the cross-task row, where FOMAML is worse. The data-selection-strategy analysis in Table 3 has the same issue: all FOMAML variants receive adaptation data on the target website or domain, while the FT baselines do not, so the differences among intra-website, inter-website, and hybrid strategies cannot be attributed to meta-learning without the matching fine-tuned-adaptation control.
minor comments (5)
  1. [Section 5, first paragraph] The phrase 'date-equivalence' appears to be a typo for 'data-equivalence'; please correct it.
  2. [Section 4, Experimental Protocol] The sentence 'For VisualWebArena, we adapt the meta-trained model on the Mind2Web training set, using 2 tasks from each of the 3 websites' is confusing; it should presumably say that adaptation is performed on the VisualWebArena websites, not on the Mind2Web training set.
  3. [Table 1 footnote] The claim that GPT-4o has 175B parameters is not supported by the cited GPT-4 technical report; the parameter count should be removed or attributed to an official source.
  4. [Appendix A.5] The visual-difficulty labels are generated by GPT-4o with chain-of-thought and then validated by humans; since the same model family is then evaluated, this is a potential source of bias in the difficulty-stratified analysis and should be acknowledged as a limitation or replaced with independent labels.
  5. [Figure 2 (right)] The numbers inset in the bars are difficult to read; consider adding a table or larger annotations for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ICL and FOMAML branches are standard few-shot procedures evaluated on external benchmarks, and no claimed gain is fitted into the evaluation by construction.

full rationale

The paper's derivation chain is self-contained. In the proprietary branch (Section 4), a single multimodal in-context demonstration is drawn from the website/domain under evaluation and the agent is then scored on the remaining tasks of that website/domain; the demonstration is a prompt, not a fitted parameter, and the remaining tasks are not the demonstration, so the reported gain is not an identity. In the open-weights branch (Section 3 and Appendix A.2), the FOMAML update is the standard first-order truncation of MAML's meta-gradient; it is not defined in terms of the reported success rates. The cross-website and cross-domain evaluations do adapt CogAgent-FOMAML on two target-site tasks before testing while the CogAgent-FT and CogAgent-FT(DE) baselines are not adapted, which is an experimental confound for attributing the gain to meta-learning rather than to the adaptation data itself; this is a correctness/control concern, not circularity, because the gain is not forced by construction. The auxiliary visual-difficulty stratification in Appendix A.5 uses GPT-4o to label difficulty for the same GPT-4o-based SeeAct agent, but those labels were human-validated (fewer than 5% changed) and this analysis is not load-bearing for the main few-shot claim. Self-citations to prior work by the same research group (e.g., [3], [45], [55]) are motivational and do not carry the derivation. No equation or prediction reduces to its own input.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on standard training hyperparameters (LoRA rank, learning rate, meta-batch size, inner steps) and on domain assumptions about FOMAML fidelity, cross-benchmark transfer, and the sufficiency of a few demonstrations. No new theoretical entities are introduced.

free parameters (5)
  • Number of in-context demonstrations = 1 for ICL; 2 for meta-adaptation
    The core few-shot setting; chosen based on a cost/accuracy trade-off, and the paper notes gains saturate with more examples.
  • LoRA rank = 20
    Hyperparameter for all CogAgent fine-tuning and meta-learning; controls the capacity of the adaptation.
  • Learning rate = 1e-5
    Used for CogAgent fine-tuning, meta-learning, and adaptation.
  • Inner-loop gradient steps = 1
    Number of update steps in the meta-learning inner loop and during post-meta adaptation.
  • Meta-batch size = 1
    One website is used per meta-update during meta-learning.
assumptions (4)
  • domain assumption First-order approximation of MAML (FOMAML) is valid for this model and data; second-order gradients are negligible.
    Invoked in Appendix A.2 to justify using FOMAML instead of MAML, relying on cited results that FOMAML performs on par with MAML.
  • domain assumption Meta-learning on the Mind2Web training set transfers to VisualWebArena's live websites.
    The agent is meta-trained on Mind2Web and then adapted to VisualWebArena without any target-site pre-training; the paper does not demonstrate transferability of the learned prior across benchmarks beyond the reported results.
  • domain assumption Human demonstrations sampled from the target website or domain are sufficient to induce adaptation for unseen tasks.
    The core few-shot premise. The paper tests at most 10 demonstrations for ICL and 2 for meta-adaptation, and does not characterize how representative a single demonstration is.
  • domain assumption The amended Mind2Web cross-task set preserves the benchmark's intended difficulty.
    Appendix A.3.1 describes a re-splitting of train and cross-task data using Jaccard similarity. This deviates from the standard benchmark, and the new split is not published, so the evaluation is not directly comparable to prior results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations." pith.science (2026). https://pith.science/paper/RZZ7AAYD

@misc{pith2026241113451,
  author       = {Pith},
  title        = {Pith review of: AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RZZ7AAYD}},
  note         = {Machine review of arXiv:2411.13451}
}
read the original abstract

State-of-the-art multimodal web agents, powered by Multimodal Large Language Models (MLLMs), can autonomously execute many web tasks by processing user instructions and interacting with graphical user interfaces (GUIs). Current strategies for building web agents rely on (i) the generalizability of underlying MLLMs and their steerability via prompting, and (ii) large-scale fine-tuning of MLLMs on web-related tasks. However, web agents still struggle to automate tasks on unseen websites and domains, limiting their applicability to enterprise-specific and proprietary platforms. Beyond generalization from large-scale pre-training and fine-tuning, we propose building agents for few-shot adaptability using human demonstrations. We introduce the AdaptAgent framework that enables both proprietary and open-weights multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2). Our experiments on two popular benchmarks -- Mind2Web & VisualWebArena -- show that using in-context demonstrations (for proprietary models) or meta-adaptation demonstrations (for meta-learned open-weights models) boosts task success rate by 3.36% to 7.21% over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03% to 65.75%. Furthermore, our additional analyses (a) show the effectiveness of multimodal demonstrations over text-only ones, (b) shed light on the influence of different data selection strategies during meta-learning on the generalization of the agent, and (c) demonstrate the effect of number of few-shot examples on the web agent's success rate. Overall, our results unlock a complementary axis for developing widely applicable multimodal web agents beyond large-scale pre-training and fine-tuning, emphasizing few-shot adaptability.

Figures

Figures reproduced from arXiv: 2411.13451 by the authors.

Figure 1
Figure 1. AdaptAgent for few-shot adaptation of web agents that are based on proprietary and open-weights multimodal LLMs. Left: For proprietary MLLM-based web agents, we include the multimodal human demonstration as in-context examples. Right: For web agents based on open￾weights MLLMs, we first learn a better prior using meta-learning and then use few-shot human demonstrations for faster adaptation. could avoid costly retra… view at source ↗
Figure 2
Figure 2. Additional analyses. Left: Ablation study on demonstration modality in SeeAct*. Center: Comparison of overall SR across meta-learning adaptation strategies in CogAgent. Right: Variation in performance with different numbers of in-context demonstrations; numbers are inset in the bars. Type Model Cross-Task Cross-Website Cross-Domain Ele. Acc. Op. F1 Step SR Overall SR Ele. Acc. Op. F1 Step SR Overall SR Ele. Acc. Op.… view at source ↗
Figure 3
Figure 3. Visual depiction of the protocol used for meta-learning using the Mind2Web train set (left), [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 22 canonical work pages

  1. [1]

    Apprenticeship learning via inverse reinforcement learning

    Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004

    work page 2004

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    A survey of robot learning from demonstration

    Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469–483, 2009

    work page 2009

  4. [4]

    In-context learning with long-context models: An in-depth exploration

    Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200, 2024

    arXiv 2024

  5. [5]

    Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks

    Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, De Chezelles, Thibault Le Sellier, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks. arXiv preprint arXiv:2407.05291, 2024

    arXiv 2024

  6. [6]

    Robots that imitate humans

    Cynthia Breazeal and Brian Scassellati. Robots that imitate humans. Trends in cognitive sciences, 6(11):481–487, 2002

    work page 2002

  7. [7]

    Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations

    Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783–792. PMLR, 2019. 9

    work page 2019

  8. [8]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

Show all 63 references
  1. [9]

    Learning and reproduction of gestures by imitation

    Sylvain Calinon, Florent D’halluin, Eric L Sauser, Darwin G Caldwell, and Aude G Billard. Learning and reproduction of gestures by imitation. IEEE Robotics & Automation Magazine, 17(2):44–54, 2010

    2010

  2. [10]

    On learning, representing, and generalizing a task in a humanoid robot

    Sylvain Calinon, Florent Guenter, and Aude Billard. On learning, representing, and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 37(2):286–298, 2007

    2007

  3. [11]

    Learning from suboptimal demonstration via self-supervised reward regression

    Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning , pages 1262–1277. PMLR, 2021

    2021

  4. [12]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

    2024 arXiv

  5. [13]

    Model-based inverse reinforcement learning from visual demonstrations

    Neha Das, Sarah Bechtle, Todor Davchev, Dinesh Jayaraman, Akshara Rai, and Franziska Meier. Model-based inverse reinforcement learning from visual demonstrations. In Conference on Robot Learning, pages 1930–1942. PMLR, 2021

    1930

  6. [14]

    Mind2web: Towards a generalist agent for the web, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023

    2023

  7. [15]

    Inverse kkt: Learning cost functions of manipulation tasks from demonstrations

    Peter Englert, Ngo Anh Vien, and Marc Toussaint. Inverse kkt: Learning cost functions of manipulation tasks from demonstrations. The International Journal of Robotics Research , 36(13-14):1474–1488, 2017

    2017

  8. [16]

    Model-agnostic meta-learning for fast adap- tation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  9. [17]

    One-shot visual imitation learning via meta-learning

    Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on robot learning, pages 357–368. PMLR, 2017

    2017

  10. [18]

    Explaining and harnessing adversar- ial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6572, 2014

    2014 arXiv

  11. [19]

    A real-world webagent with planning, long context understanding, and program synthesis, 2024

    Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis, 2024

    2024

  12. [20]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  13. [21]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016

    2016

  14. [22]

    Cogagent: A visual language model for gui agents, 2023

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023

    2023

  15. [23]

    A data-driven approach for learning to control computers

    Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. InInternational Conference on Machine Learning, p...

    2022

  16. [24]

    Imitation learning: A survey of learning methods

    Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35, 2017

    2017

  17. [25]

    Do berts learn to use browser user interface? exploring multi-step tasks with unified vision-and-language berts

    Taichi Iki and Akiko Aizawa. Do berts learn to use browser user interface? exploring multi-step tasks with unified vision-and-language berts. arXiv preprint arXiv:2203.07828, 2022. 10

    2022 arXiv

  18. [26]

    Data-efficient alignment of large language models with human feedback through natural language, 2023

    Di Jin, Shikib Mehri, Devamanyu Hazarika, Aishwarya Padmakumar, Sungjin Lee, Yang Liu, and Mahdi Namazifar. Data-efficient alignment of large language models with human feedback through natural language, 2023

    2023

  19. [27]

    Dspy: Compiling declarative language model calls into self-improving pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023

    arXiv

  20. [28]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  21. [29]

    Learning driving styles for autonomous vehicles from demonstration

    Markus Kuderer, Shilpa Gulati, and Wolfram Burgard. Learning driving styles for autonomous vehicles from demonstration. In2015 IEEE international conference on robotics and automation (ICRA), pages 2641–2646. IEEE, 2015

    2015

  22. [30]

    Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent. arXiv preprint arXiv:2404.03648, 2024

    2024 arXiv

  23. [31]

    Reinforcement learning on web interfaces using workflow-guided exploration.arXiv preprint arXiv:1802.08802, 2018

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration.arXiv preprint arXiv:1802.08802, 2018

    2018 arXiv

  24. [32]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning, 2024

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning, 2024

    2024

  25. [33]

    Algorithms for inverse reinforcement learning

    Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2, 2000

    2000

  26. [34]

    On first-order meta-learning algorithms

    A Nichol. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018

    2018 arXiv

  27. [35]

    Experimental evidence on the productivity effects of generative artificial intelligence

    Shakked Noy and Whitney Zhang. Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654):187–192, 2023

    2023

  28. [36]

    Oracle AI agents help organizations achieve new levels of productivity, Sep 2024

    Oracle. Oracle AI agents help organizations achieve new levels of productivity, Sep 2024. Online press release

    2024

  29. [37]

    Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale

    Tianyue Ou, Frank F Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale. arXiv preprint arXiv:2409.15637, 2024

    2024 arXiv

  30. [38]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  31. [39]

    Alvinn: An autonomous land vehicle in a neural network

    Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988

    1988

  32. [40]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  33. [41]

    Recent advances in robot learning from demonstration

    Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3:297–330, 2020

    2020

  34. [42]

    Your transformer is secretly linear.arXiv preprint arXiv:2405.12250, 2024

    Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. Your transformer is secretly linear.arXiv preprint arXiv:2405.12250, 2024. 11

    2024 arXiv

  35. [43]

    Generalization guarantees for imitation learning

    Allen Ren, Sushant Veer, and Anirudha Majumdar. Generalization guarantees for imitation learning. In Conference on Robot Learning, pages 1426–1442. PMLR, 2021

    2021

  36. [44]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and C...

    2011

  37. [45]

    Interactive robot task training through dialog and demonstration

    Paul E Rybski, Kevin Yoon, Jeremy Stolarz, and Manuela M Veloso. Interactive robot task training through dialog and demonstration. In Proceedings of the ACM/IEEE international conference on Human-robot interaction, pages 49–56, 2007

    2007

  38. [46]

    Learning from demonstration

    Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996

    1996

  39. [47]

    Dynamic movement primitives-a framework for motor control in humans and humanoid robotics

    Stefan Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. In Adaptive motion of animals and machines, pages 261–280. Springer, 2006

    2006

  40. [48]

    Evolutionary principles in self-referential learning

    Jurgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The meta-meta-... hook.) Diploma thesis, Institut f. Informatik, Tech. Univ. Munich, 1(2):48, 1987

    1987

  41. [49]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324, 2023

    arXiv

  42. [50]

    Show, don’t tell: Aligning language models with demonstrated feedback

    Omar Shaikh, Michelle Lam, Joey Hejna, Yijia Shao, Michael Bernstein, and Diyi Yang. Show, don’t tell: Aligning language models with demonstrated feedback. arXiv preprint arXiv:2406.00888, 2024

    2024 arXiv

  43. [51]

    From pixels to ui actions: Learning to follow instructions via graphical user interfaces, 2023

    Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces, 2023

    2023

  44. [52]

    World of bits: An open-domain platform for web-based agents

    Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135–3144. PMLR, 2017

    2017

  45. [53]

    Mas- tering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016

    2016

  46. [54]

    Beyond browsing: Api-based web agents

    Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api-based web agents. arXiv preprint arXiv:2410.16464, 2024

    2024 arXiv

  47. [55]

    Perception, cognition, and action in teams of robots

    Manuela Veloso. Perception, cognition, and action in teams of robots. Colloquium at the Department of Computer Science, Princeton University, September 28 2005

    2005

  48. [56]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023

    2023 arXiv

  49. [57]

    Transformers: State- of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in n...

    2020

  50. [58]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023. 12

    2023 arXiv

  51. [59]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  52. [60]

    Selfd: self-learning large-scale driving policies from the web

    Jimuyang Zhang, Ruizhao Zhu, and Eshed Ohn-Bar. Selfd: self-learning large-scale driving policies from the web. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17316–17326, 2022

    2022

  53. [61]

    Group preference optimization: Few-shot alignment of large language models

    Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.11523, 2023

    2023 arXiv

  54. [62]

    Gpt-4v(ision) is a generalist web agent, if grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024

    2024

  55. [63]

    ideal answers

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433–1438. Chicago, IL, USA, 2008. A Appendix A.1 Detailed Related Work A.1.1 UI/Web Agents AI-enabled digital device control [52, 23]...

    2008

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.