Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

Large Action Models: From Inception to Implementation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A four-phase training pipeline converts a 7B language model into a Large Action Model that the paper reports surpasses GPT-4o on Windows GUI task completion, reaching 81.2% offline task success versus 67.2% for GPT-4o, and 71.0% online…

desk verdict A genuinely useful engineering recipe for turning a 7B model into a Windows GUI agent, but the headline performance claim rests on an unverified test-set split and a circular GPT-4o-as-teacher-and-judge setup. read the letter →

arxiv 2412.10047 v2 pith:LMEYHKIK submitted 2024-12-13 cs.AI

Lu Wang , Fangkai Yang , Chaoyun Zhang , Junting Lu , Jiaxu Qian , Shilin He , Pu Zhao , Bo Qiao
show 10 more authors
Ray Huang Si Qin Qisheng Su Jiayi Ye Yudi Zhang Jian-Guang Lou Qingwei Lin Saravan Rajmohan Dongmei Zhang Qi Zhang
This is my paper · ORCID
classification cs.AI
keywords LargeActionModelsGUIagentssupervisedfine-tuningself-boostingexplorationrewardmodelofflinereinforcementlearningWindowsOSautomationtask-plandata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a general-purpose language model can be turned into a Large Action Model—a model that emits executable actions in a real environment—through a carefully staged training pipeline. Using a 7-billion-parameter Mistral model fine-tuned for Windows GUI tasks, it reports that the resulting LAM4 completes 81.2% of offline test tasks, ahead of GPT-4o's 67.2%, and 71.0% of online text-only tasks, ahead of GPT-4o's 63.0%, while also finishing tasks in about a third of the time. The authors frame the contribution as a generalizable blueprint: task-plan pretraining, imitation of expert trajectories, self-boosting exploration on tasks the expert failed, and reward-model-based policy optimization. A sympathetic reader would take the central point to be that a small specialized action model can outperform much larger general-purpose LLMs within a constrained environment, and that the same workflow can be ported to other domains.

What carries the argument

The central object is the four-phase LAM training pipeline. Phase 1, task-plan pretraining, uses supervised fine-tuning on task-plan pairs drawn from documentation, WikiHow, search queries, and evolved data to teach the model to decompose requests into structured plans. Phase 2, learning from experts, fine-tunes on GPT-4o-generated action trajectories where each step maps a UI state to a concrete action, grounding the plans in an executable form. Phase 3, self-boosting exploration, uses the ReAct mechanism to let the model attempt tasks that GPT-4o failed, collecting its own successful trajectories as new training data. Phase 4, learning from a reward model, trains a scalar reward model with uniform +1/−1 labels on successful and failed trajectories and uses offline PPO to optimize the policy on failure trajectories. This staged progression—from planning, to imitation, to self-exploration, to reward-based refinement—is what carries the argument, along with the automated data-collection pipeline that converts abstract task-plan data into concrete, execution-validated action sequences.

What would settle it

Re-run the offline and online evaluation on a disjoint, human-annotated set of Windows GUI tasks never touched by any training phase, scoring success by document-state verification such as XML diff rather than GPT-4o judgment; if the reported gap over text-only GPT-4o disappears or inverts, the paper's core claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that its four-phase training recipe yields a specialized action model whose performance exceeds the teacher model, GPT-4o, that supplied most of its training data, in both offline and online evaluation. The phase-wise trajectory is concrete: LAM1, trained only on 76,672 task-plan pairs, reaches 82.2% planning task success; LAM2, which imitates 2,192 GPT-4o-generated action trajectories, jumps to 76.8% decision-making task success; LAM3, which adds 496 self-generated successes from tasks GPT-4o failed, reaches 79.3%; and LAM4, refined with a reward model and offline PPO on 1,788 failure trajectories, reaches 81.2%. Online, LAM's text-only task success rate of 71.0% surpasses GPT-4o's text-only 63.0%, while GPT-4o with visual input reaches 75.5% but takes 96.48 seconds versus LAM's 30.42 seconds. The paper presents this as evidence that a domain-specialized LAM trained from scratch can be both more accurate and far more efficient than a state-of-the-art generalist model, and that the workflow transfers beyond the Windows/Word case study.

Load-bearing premise

The evaluation treats the same 435 test requests used during training as the online test set and relies on GPT-4o to judge whether tasks succeeded, so the claim that LAM4 beats GPT-4o depends on those test tasks not leaking into the self-boosting and reward-model training data and on the judge being unbiased.

Editorial extensions

If this is right

  • A 7-billion-parameter specialized model can outperform a much larger generalist LLM on bounded GUI tasks, both in task success rate and in execution speed, when trained with environment-grounded data.
  • The jump from 76.8% to 79.3% offline task success shows that a model can learn from its own successes on tasks its teacher failed, making self-boosting a viable data source beyond expert demonstrations.
  • Adding reward-model-guided offline RL on failure trajectories yields a further improvement to 81.2%, indicating that mistakes, when properly scored, are useful training signal rather than noise to discard.
  • The authors claim the same data-collection and training workflow carries over to other environments and applications, offering a template for building LAMs beyond the Windows and Word testbed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence of the pipeline's reliance on environment-specific action grounding is that performance will likely drop more sharply on Windows applications not seen in training (e.g., Excel or PowerPoint) than on unseen Word tasks; the paper reports no cross-application generalization results.
  • Because the 435 online test tasks are the same set used for training, the reported online TSR of 71.0% is best read as an upper bound; a held-out task set scored by human verification or document-state checks would give a cleaner, likely smaller, estimate of the pipeline's true benefit.
  • The reward model assigns the same +1 to every step of a successful trajectory and −1 to every step of a failed one, ignoring intermediate progress; a step-level or progress-aware reward could plausibly extract more from the failure trajectories than the reported 1.9-point gain.
  • The efficiency edge of a text-only, 7B-parameter LAM suggests the approach could extend to resource-constrained or edge deployment, but the pipeline's dependence on GPT-4o for data annotation and evaluation leaves open how it would bootstrap in a domain without a strong teacher model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a four-phase pipeline for converting Mistral-7B into a Windows GUI agent (LAM4), consisting of task-plan pretraining, supervised fine-tuning on GPT-4o trajectories, self-boosting exploration on GPT-4o failure cases, and offline PPO with a learned reward model. Using Microsoft Word as a testbed and the UFO agent for grounding, the authors report offline TSR of 81.2% versus 67.2% for GPT-4o (Table 3) and online TSR of 71.0% versus 63.0% for text-only GPT-4o (Table 4), with substantially lower latency. The paper also positions the workflow as a generalizable blueprint for LAM development and releases the data collection code.

Significance. The staged data collection and training recipe, detailed prompts, and public dataflow code are useful engineering contributions, and the paper is clearly organized. The headline result—a 7B text-only model beating GPT-4o on Windows GUI task completion with roughly 2–3x lower latency—would be practically significant if the evaluation is valid. However, the current evidence does not establish that result, because the test split may have been used in Phases 3 and 4 and because GPT-4o serves as data generator, filter, and judge throughout. The manuscript's strengths are its explicit phasewise data counts, the concrete description of grounding in the UFO agent, and the reproducibility of the data collection pipeline.

major comments (4)
  1. [§4.2, §4.3, §4.4, §7.1] The test set appears to contaminate Phases 3 and 4. Section 4.2 splits the 2,192 GPT-4o successes into 1,757 training and 435 test trajectories, but Section 4.3 states that LAM3 is fine-tuned on the original 2,192 GPT-4o successes plus 496 new successes, and Section 4.4.1 trains the reward model on GPT-4o's successful trajectories (2,192) as well as 1,788 LAM failures. Section 7.1 then says online evaluation uses the same set of 435 test requests used during LAM training. Unless the authors provide a task-level audit showing that the 435 test task IDs were excluded from the Phase 3 training pool, the Phase 3 failure pool, and the Phase 4 RM/PPO pools, Tables 3 and 4 can be explained by memorization rather than generalization. This is the most load-bearing issue and must be fixed by re-running Phases 3 and 4 with an explicit held-out split and reporting the resulting numbers.
  2. [§3.1.3, §3.2.1, §3.2.3, §5.2.1, §7.4] GPT-4o is used both to generate the data the model is trained on and to judge whether tasks are successfully completed. Specifically, GPT-4o constructs task-plan data (Section 3.1.3), instantiates task-action trajectories (Section 3.2.1), filters which execution trajectories are retained (Section 3.2.3), computes the planning metrics for LAM1 (Section 5.2.1), and determines online task success (Section 7.4). Because LAM is trained to imitate GPT-4o's style and then evaluated by a GPT-4o judge, the judge may systematically favor LAM's outputs over GPT-4o's own zero-shot outputs. The authors should validate at least a sample of the online evaluations with human judgment or with an independent deterministic success criterion (e.g., final document state), and report agreement or disagreement with the GPT-4o judge.
  3. [§5.3, §7.4 (Tables 3 and 4)] All headline comparisons are single point estimates over 435 tasks with no confidence intervals, significance tests, or multiple runs. Given the relatively small test set and the additional variance in online execution, the claim that LAM4 'outperforms' GPT-4o requires more than point estimates; please report bootstrap confidence intervals or per-task breakdowns for the TSR differences in Tables 3 and 4.
  4. [§5.3.1] The offline decision-making metrics compare predicted actions against a ground-truth action history that was itself generated by GPT-4o. This exact-match style of evaluation penalizes valid alternative solutions and therefore favors models fine-tuned on that exact distribution. The offline TSR should be interpreted as a measure of imitation fidelity rather than task-completion quality, and this distinction should be stated explicitly in the paper.
minor comments (5)
  1. [Figure 4 caption] The caption says 'two-phrase data collection'; this should be 'two-phase data collection.'
  2. [§3.1.4] The data augmentation numbers are not fully consistent: 29,182 original pairs plus 47,490 evolved pairs gives 76,672, but 'increased by 150%' suggests 43,773 additional pairs; please clarify the exact augmentation ratio and how the 47,490 figure is derived.
  3. [§4.4] The sentence beginning 'we first The reward model' is incomplete and should be rewritten.
  4. [Appendix B.2 heading] The heading says 'The instantiation prompt used in the evaluation phase,' but the prompt is the evaluation prompt, not the instantiation prompt; please correct the heading.
  5. [§5.2.2 and Table 2] The evaluation in Table 2 is reported on 15,334 tasks, which is not reconciled with the 76,672 training pairs described in Section 3.1.4; please state the split and whether any evolution-derived tasks appear in the evaluation set.

Circularity Check

2 steps flagged · score 7.0 of 10

Reported test set was included in Phases 3-4 training, so the TSR advantage over GPT-4o is a fit, not a prediction.

  1. fitted input called prediction [Sections 4.2-4.4 and 7.1 (test split reused in later training; online test on the same requests)]
    "We split these 2,192 trajectories into a training set of 1,757 and a test set of 435 trajectories... These self-labeled successes, combined with the original 2,192 GPT-4o successes, form an augmented dataset... we train a reward model (RM) on ... GPT-4o’s successful trajectories (2192) gathered in previous phases... The online performance of LAM is evaluated on the same set of 435 test requests used during LAM training."

    The 435 'test' trajectories are a subset of the 2,192 GPT-4o successes defined in Section 4.2. Phase 3 fine-tuning of LAM3 uses all 2,192 GPT-4o successes, and Phase 4 reward-model training uses the same 2,192 plus LAM trajectories, without excluding the 435 test IDs. Consequently the offline TSR of 81.2% (Table 3) and the online TSR of 71.0% (Table 4) are computed on tasks whose trajectories were training inputs for LAM3 and LAM4. The 'prediction' reduces by construction to a re-measurement on fitted data: the model was optimized on the test tasks, so the reported margin over GPT-4o is not an out-of-sample generalization result.

  2. self definitional [Sections 3.2.3, 5.2.1, and 7.4 (GPT-4o as labeler, filter, and judge)]
    "we prompt GPT-4o to evaluate whether the executed task aligns with the original task description and achieves successful completion... To compute these metrics, we leverage GPT-4o to compare each step of the LAM1 output with the corresponding ground truth steps... Task success is determined by an evaluation agent using GPT-4o, which assesses the full task completion trajectory, including plans, action sequences, and screenshots, to verify task completion."

    GPT-4o constructs the task-plan and task-action data (Sections 3.1.3 and 3.2), decides which trajectories are retained as 'successful' during data collection (Section 3.2.3), scores planning metrics (Section 5.2.1), and finally judges online task completion (Section 7.4). The model is therefore trained to satisfy a GPT-4o-generated notion of success and then evaluated by the same model family using that same notion. This makes the headline comparison 'LAM surpasses GPT-4o' a measurement against a criterion manufactured by GPT-4o itself, rather than an independent ground truth; the evaluation loop is closed by the same system that produced the training labels.

full rationale

The decisive circularity is in the evaluation protocol, not in the training equations. Section 4.2 splits the 2,192 GPT-4o trajectories into 1,757 training and 435 test trajectories, but Sections 4.3-4.4 then use 'the original 2,192 GPT-4o successes' to fine-tune LAM3 and to train the Phase 4 reward model, with no exclusion of the 435 test IDs. Since the 435 test trajectories are a subset of those 2,192, the offline TSR of 81.2% (Table 3) and the online TSR of 71.0% (Table 4) are measured on trajectories that were training inputs for LAM3/LAM4. The reported 'prediction' is therefore a fitted-input evaluation: the model was optimized on the test tasks, so its superiority over GPT-4o is not an out-of-sample result. A secondary self-referential loop compounds this: GPT-4o generates the task-plan and task-action labels, filters trajectories as successful in Section 3.2.3, computes plan metrics in Section 5.2.1, and judges online task completion in Section 7.4, so the success criterion is defined by the same model family that produced the training targets. I do not find load-bearing self-citation or imported uniqueness; the four-phase training pipeline itself is not circular. The score of 7 reflects that the central empirical claim reduces, at the evaluation stage, to a fit on the reported test set rather than an independent generalization result.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a pipeline whose data and labels come from GPT-4o, on an assumption of test-set integrity, and on several hand-chosen training data sizes and reward assignments. There are no invented physical entities.

free parameters (3)
  • Uniform binary reward assignment = +1 for each step in successful trajectories, -1 for each step in failed trajectories
    Chosen by hand in Section 4.4.1; assumes every step in a failed trajectory is equally bad, which shapes the reward model and PPO training.
  • Data evolution augmentation ratio = 150% increase, yielding 47,490 evolved samples
    Chosen to enlarge the task-plan dataset in Section 3.1.4; no principle is given for this specific amount.
  • Phase 3 exploration sample size = 2,284 GPT-4o failed tasks sampled, 496 successes kept
    Arbitrary sampling choice in Section 4.3 that determines LAM3 and LAM4 training data and therefore the reported performance.
assumptions (5)
  • domain assumption GPT-4o-generated trajectories are valid expert demonstrations.
    Used throughout Section 3.2; the entire training signal in Phases 2-4 is built from GPT-4o instantiations and evaluations.
  • domain assumption GPT-4o evaluation of task completion is a reliable ground truth.
    Used in Sections 3.2.3, 5.2.1, and 7.4; all reported success rates depend on this judge.
  • domain assumption The 435 test tasks were not used in Phases 3-4 training.
    Section 7.1 says online evaluation uses the same 435 test requests used during LAM training; the paper does not state that these tasks were excluded from self-boosting or PPO data.
  • domain assumption UIA API provides complete and accurate control metadata for grounding actions.
    Section 6.2: the agent relies on UIA to enumerate actionable controls; errors here would break action grounding.
  • domain assumption Task success can be inferred from screenshots, XML canvas states, and control states.
    Section 3.2.3 and the evaluation prompt in Appendix B.2 assume these signals are sufficient for judging completion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Action Models: From Inception to Implementation." pith.science (2026). https://pith.science/paper/LMEYHKIK

@misc{pith2026241210047,
  author       = {Pith},
  title        = {Pith review of: Large Action Models: From Inception to Implementation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMEYHKIK}},
  note         = {Machine review of arXiv:2412.10047}
}
read the original abstract

As AI continues to advance, there is a growing demand for systems that go beyond language-based assistance and move toward intelligent agents capable of performing real-world actions. This evolution requires the transition from traditional Large Language Models (LLMs), which excel at generating textual responses, to Large Action Models (LAMs), designed for action generation and execution within dynamic environments. Enabled by agent systems, LAMs hold the potential to transform AI from passive language understanding to active task completion, marking a significant milestone in the progression toward artificial general intelligence. In this paper, we present a comprehensive framework for developing LAMs, offering a systematic approach to their creation, from inception to deployment. We begin with an overview of LAMs, highlighting their unique characteristics and delineating their differences from LLMs. Using a Windows OS-based agent as a case study, we provide a detailed, step-by-step guide on the key stages of LAM development, including data collection, model training, environment integration, grounding, and evaluation. This generalizable workflow can serve as a blueprint for creating functional LAMs in various application domains. We conclude by identifying the current limitations of LAMs and discussing directions for future research and industrial deployment, emphasizing the challenges and opportunities that lie ahead in realizing the full potential of LAMs in real-world applications. The code for the data collection process utilized in this paper is publicly available at: https://github.com/microsoft/UFO/tree/main/dataflow, and comprehensive documentation can be found at https://microsoft.github.io/UFO/dataflow/overview/.

Figures

Figures reproduced from arXiv: 2412.10047 by the authors.

Figure 1
Figure 1. The transition from LLMs to LAMs. [28]. Current LLMs may excel at understanding and planning in tex￾tual form but often fall short when required to produce actionable outputs. This is particularly true in scenarios that demand precise task decomposition, long-term planning [12, 88], and the coordi￾nation of multi-step actions [63]. Furthermore, LLMs are generally optimized for broad, general-purpose tasks rather tha… view at source ↗
Figure 2
Figure 2. The objective difference between LLMs and LAMs. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The process pipeline for LAM development and implementation. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The two-phrase data collection and preparation process. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The pipeline to construct the task plan data. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The pipeline of task-action data conversion and [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: The overview of LAM training pipeline. "yes," "no," or "unsure." If the task is evaluated as "yes", the tra￾jectory is deemed successful; otherwise, it is classified as a failure. The detailed prompt used for this evaluation is provided in Ap￾pendix B.2. This evaluatio…
Figure 9
Figure 9. Figure 9: The overall architecture of the AppAgent employed [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: A word template file with the description “A doc [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: A word template file with the description “A doc [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: A word template file with the description “A doc [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Topological sum rule for geometric phases of quantum gates

    quant-ph 2026-03 unverdicted novelty 6.0 of 10

    Geometric phases of a two-qubit gate over a complete basis sum to a multiple of the Hamiltonian winding number, so topology is necessary for entanglement generation.

  2. GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.

  3. Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A LoRA fine-tuned Molmo-7B model, trained on synthetic automotive UI data with reasoning and pass/fail evaluation labels, improves visual grounding on a new automotive benchmark and on the external ScreenSpot test.

  4. UFO2: The Desktop AgentOS

    cs.AI 2025-04 reject novelty 6.0 of 10

    UFO2 reports that deep Windows integration, hybrid vision and UIA control detection, and GUI-plus-API actions lift desktop automation success rates above prior CUAs, but the evaluation is partly contaminated by benchm...

  5. Initial Steps in Integrating Large Reasoning and Action Models for Service Composition

    cs.AI 2025-07 conditional novelty 5.0 of 10

    A conceptual framework integrating LRMs and LAMs for end-to-end automated service composition, with no empirical validation.

  6. A Lightweight Incentive-Based Privacy-Preserving Smart Metering Protocol for Value-Added Services

    cs.CR 2025-08 unverdicted novelty 4.0 of 10

    A layered protocol of local differential privacy, blind signatures, pseudonyms, temporal aggregation, and anonymous routing is claimed to keep smart-meter readings private while still enabling reward token redemption.

  7. Large Language Models as Computable Approximations to Solomonoff Induction

    cs.LG 2025-05 reject novelty 2.0 of 10

    The paper argues LLMs are computable approximations of Solomonoff induction, but its central derivation recovers the model's own probabilities by construction.

Reference graph

Works this paper leans on

116 extracted references · 23 canonical work pages · cited by 7 Pith papers

  1. [1]

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin, Jindong Chen, and Abhanshu Sharma

  2. [2]

    Anjanava Biswas and Wrick Talukdar. 2023. Guardrails for trust, safety, and ethical development and deployment of Large Language Models (LLM). Journal of Science & Technology 4, 6 (2023), 55–82

    2023

  3. [3]

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and 16 Zack Hui. 2024. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. arXiv preprint arXiv:2409.08264 (2024)

    arXiv 2024

  4. [4]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

    arXiv 2020

  5. [5]

    Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning. PMLR, 3676–3713

    2023

  6. [6]

    Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, et al. 2024. When large language models meet personalization: Perspectives of challenges and opportunities.World Wide Web 27, 4 (2024), 42

    2024

  7. [7]

    Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. 2024. GUICourse: From General Vision Language Models to Versatile GUI Agents. arXiv preprint arXiv:2406.11317 (2024)

    arXiv 2024

  8. [8]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935 (2024)

    arXiv 2024

Show all 116 references
  1. [9]

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. 2024. Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles. IEEE Intelligent Transportation Systems Magazine (2024)

    2024

  2. [10]

    Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051 (2022)

    2022 arXiv

  3. [11]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems 36 (2024)

    2024

  4. [12]

    Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.04254 (2023)

    2023 arXiv

  5. [13]

    Duong Tran Dinh, Pham Ngoc Hung, and Tung Nguyen Duy. 2018. A method for automated user interface testing of windows-based applications. InProceedings of the 9th International Symposium on Information and Communication Technology . 337–343

    2018

  6. [14]

    Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. [n. d.]. How Far Are We From AGI: Are LLMs All We Need? Transactions on Machine Learning Research ([n. d.])

  7. [15]

    Emilio Ferrara. 2024. GenAI against humanity: Nefarious applications of genera- tive artificial intelligence and large language models. Journal of Computational Social Science (2024), 1–21

    2024

  8. [16]

    Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. 2024. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 12462–12469

    2024

  9. [17]

    William J Gehring, Brian Goss, Michael GH Coles, David E Meyer, and Emanuel Donchin. 1993. A neural system for error detection and compensation. Psycho- logical science 4, 6 (1993), 385–390

    1993

  10. [18]

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. 2023. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296 (2023)

    2023 arXiv

  11. [19]

    Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati

  12. [20]

    Jianliang He, Siyu Chen, Fengzhuo Zhang, and Zhuoran Yang. 2024. From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems. arXiv preprint arXiv:2405.19883 (2024)

    2024 arXiv

  13. [21]

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2024. CogAgent: A visual language model for GUI agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14281–14290

    2024

  14. [22]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  15. [23]

    Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, Saravan Rajmohan, and Dongmei Zhang. 2024. AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation. CoRR (2024)

    2024

  16. [24]

    Yucheng Hu and Yuxing Lu. 2024. Rag and rau: A survey on retrieval-augmented language model in natural language processing. arXiv preprint arXiv:2404.19543 (2024)

    2024 arXiv

  17. [25]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  18. [26]

    Yuxuan Jiang, Chaoyun Zhang, Shilin He, Zhihao Yang, Minghua Ma, Si Qin, Yu Kang, Yingnong Dang, Saravan Rajmohan, Qingwei Lin, et al. 2024. Xpert: Em- powering incident management with query recommendations via large language models. In Proceedings of the IEEE/ACM 46th Intern...

    2024

  19. [27]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics 9 (2021), 962–977

    2021

  20. [28]

    Sai Shashank Kalakonda, Shubh Maheshwari, and Ravi Kiran Sarvadevabhatla

  21. [29]

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. 2024. OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web. arXiv preprint arXiv:2402.17553 (2024)

    2024 arXiv

  22. [30]

    Mert Karabacak and Konstantinos Margetis. 2023. Embracing large language models for medical applications: opportunities and challenges. Cureus 15, 5 (2023)

    2023

  23. [31]

    In 2023 IEEE International Conference on Multimedia and Expo (ICME)

    Action-gpt: Leveraging large-scale language models for improved and gen- eralized action generation. In 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 31–36

    2023

  24. [32]

    Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2024. Language models can solve computer tasks. Advances in Neural Information Processing Systems 36 (2024)

    2024

  25. [33]

    Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. 2020. Calibrated language model fine-tuning for in-and out-of-distribution data. arXiv preprint arXiv:2010.11506 (2020)

    2020 arXiv

  26. [34]

    Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning an...

    2023

  27. [35]

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024. On the Effects of Data Scale on Computer Control Agents. arXiv preprint arXiv:2406.03679 (2024)

    2024 arXiv

  28. [36]

    Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2023. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149 (2023)

    2023 arXiv

  29. [37]

    Richard Koo and Sam Toueg. 1987. Checkpointing and rollback-recovery for distributed systems. IEEE Transactions on software Engineering 1 (1987), 23–31

    1987

  30. [38]

    Andreas Liesenfeld, Alianda Lopez, and Mark Dingemanse. 2023. Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction- tuned text generators. In Proceedings of the 5th international conference on con- versational user interfaces. 1–6

    2023

  31. [39]

    Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. 2023. Do- main specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint arXiv:2305.18703 (2023)

    2023 arXiv

  32. [40]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance . 374–382

    2023

  33. [41]

    Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. 2024. Towards safer large language models through machine unlearning.arXiv preprint arXiv:2402.10058 (2024)

    2024 arXiv

  34. [42]

    Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qing- wei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. 2024. Turn every application into an agent: Towards efficient human-agent-computer interaction with api-first llm-based agents. arXiv preprint ar...

    2024 arXiv

  35. [43]

    Jun Liu, Chaoyun Zhang, Jiaxu Qian, Minghua Ma, Si Qin, Chetan Bansal, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2024. Large Language Models can Deliver Accurate and Interpretable Time Series Anomaly Detection. arXiv preprint arXiv:2405.15370 (2024)

    2024 arXiv

  36. [44]

    Bertalan Meskó and Eric J Topol. 2023. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ digital medicine 6, 1 (2023), 120

    2023

  37. [45]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024)

    2024 arXiv

  38. [46]

    Zilin Ma, Yiyang Mei, and Zhaoyuan Su. 2023. Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support. In AMIA Annual Symposium Proceedings, Vol. 2023. American Medical Informatics Association, 1105

    2023

  39. [47]

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems 36 (2024). 17

    2024

  40. [48]

    Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality 15, 2 (2023), 1–21

    2023

  41. [49]

    Timo Minssen, Effy Vayena, and I Glenn Cohen. 2023. The challenges for regu- lating medical use of ChatGPT and other large language models. Jama (2023)

    2023

  42. [50]

    Alastair Pennycook. 1985. Actions speak louder than words: Paralanguage, communication, and education. Tesol Quarterly 19, 2 (1985), 259–282

    1985

  43. [51]

    Andrés Piñeiro-Martín, Carmen García-Mateo, Laura Docío-Fernández, and Maria Del Carmen Lopez-Perez. 2023. Ethical challenges in the development of virtual assistants powered by large language models. Electronics 12, 14 (2023), 3170

    2023

  44. [52]

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. 2024. Screenagent: A vision language model-driven computer control agent. arXiv preprint arXiv:2402.07945 (2024)

    2024 arXiv

  45. [53]

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lilli- crap. 2024. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems 36 (2024)

    2024

  46. [54]

    Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, et al. 2023. Tptu: Task planning and tool usage of large language model-based ai agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop

    2023

  47. [55]

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al . 2024. AndroidWorld: A dynamic benchmarking environment for au- tonomous agents. arXiv preprint arXiv:2405.145...

    2024 arXiv

  48. [56]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  49. [57]

    Chirag Shah, Ryen W White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Snigdha Sarathi Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Xi- aochuan Ni, et al. 2023. Using large language models to generate, validate, and apply user intent taxonomies. arXiv prepri...

    2023 arXiv

  50. [58]

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925 (2023)

    2023 arXiv

  51. [59]

    Zirui Song, Yaohang Li, Meng Fang, Zhenhao Chen, Zecheng Shi, and Yuan Huang. 2024. MMAC-Copilot: Multi-modal Agent Collaboration Operating Sys- tem Copilot. arXiv preprint arXiv:2404.18074 (2024)

    2024 arXiv

  52. [60]

    Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, et al. 2024. Towards general computer control: A multimodal agent for red dead redemption ii as a case study. arXiv preprint arXiv:2403.03186 (2024)

    2024 arXiv

  53. [61]

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine 29, 8 (2023), 1930–1940

    2023

  54. [62]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36 (2024)

    2024

  55. [63]

    Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambham- pati. 2022. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop

    2022

  56. [64]

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. arXiv preprint arXiv:2401.16158 (2024)

    2024 arXiv

  57. [65]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345

    2024

  58. [66]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large language models can accurately predict searcher preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1930–1940

    2024

  59. [67]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024)

    2024 arXiv

  60. [68]

    Zige Wang, Wanjun Zhong, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Data management for large language models: A survey. arXiv e-prints (2023), arXiv–2312

    2023

  61. [69]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)

    2021 arXiv

  62. [70]

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al . 2024. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems 36 (2024)

    2024

  63. [71]

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. 2024. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456 (2024)

    2024 arXiv

  64. [72]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 (2023)

    2023 arXiv

  65. [73]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al . 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972 (2024)

    2024 arXiv

  66. [74]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  67. [75]

    Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming . 1–10

    2022

  68. [76]

    Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. 2024. Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Tech...

    2024

  69. [77]

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 9, 1 (2023), 1

    2023 arXiv

  70. [78]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 (2023)

    2023 arXiv

  71. [79]

    Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep calm and explore: Language models for action generation in text-based games. arXiv preprint arXiv:2010.02903 (2020)

    2020 arXiv

  72. [80]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022)

    2022 arXiv

  73. [81]

    Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, and Eray Tüzün. 2023. Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. arXiv preprint arXiv:2304.10778 (2023)

    2023 arXiv

  74. [82]

    Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. Appagent: Multimodal agents as smartphone users.arXiv preprint arXiv:2312.13771 (2023)

    2023 arXiv

  75. [83]

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang

  76. [84]

    Chaoyun Zhang, Zicheng Ma, Yuhao Wu, Shilin He, Si Qin, Minghua Ma, Xiaoting Qin, Yu Kang, Yuyi Liang, Xiaoyu Gou, et al. 2024. AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models. arXiv preprint arXiv:2403.15157 (2024)

    2024 arXiv

  77. [85]

    Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. 2024. xLAM: A Family of Large Action Models to Empower AI Agent Systems. arXiv preprint arXiv:2409.03215 (2024)

    2024 arXiv

  78. [86]

    Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S Yu. 2023. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07226 (2023)

    2023

  79. [87]

    Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. 2024. On the out-of-distribution gener- alization of multimodal large language models. arXiv preprint arXiv:2402.06599 (2024)

    2024 arXiv

  80. [88]

    arXiv preprint arXiv:2402.07939 (2024)

    UFO: A UI-Focused Agent for Windows OS Interaction. arXiv preprint arXiv:2402.07939 (2024)

    2024 arXiv

  81. [89]

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.13501 (2024)

    2024 arXiv

  82. [90]

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. Safetybench: Evaluating the safety of large language models with multiple choice questions.arXiv preprint arXiv:2309.07045 (2023)

    2023 arXiv

  83. [91]

    Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B Tenenbaum, and Chuang Gan. 2023. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510 (2023)

    2023 arXiv

  84. [92]

    A doc with a rectangle shape

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey.arXiv preprint arXiv:2308.07107 (2023). 18 A TEMPLATE WORD FILES Figure 10, 11, and 12 sh...

    2023

  85. [93]

    Yudi Zhang, Pei Xiao, Lu Wang, Chaoyun Zhang, Meng Fang, Yali Du, Yev- geniy Puzyrev, Randolph Yao, Si Qin, Qingwei Lin, et al . 2024. RuAG: Learned-rule-augmented Generation for Large Language Models. arXiv preprint arXiv:2411.03349 (2024)

    2024 arXiv

  86. [96]

    Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Making harmful behaviors unlearnable for large language models. arXiv preprint arXiv:2311.02105 (2023)

    2023 arXiv

  87. [98]

    The < New Task > must based on the given task

  88. [99]

    The < New Task > must be able to be completed step - by - step by a Windows Operating System or an Application on Windows platform . 19

  89. [100]

    You should try your best not to make the < New Task > become verbose , < New Task > can only add up to 50 words into # Given Task #

  90. [101]

    The detailed target in < New Task > should be specific and clear based on the doc canvas content and control information

  91. [102]

    # # The requirements for < Action Plan >

    The < New Task > should be able to implemented by the available controls and actions . # # The requirements for < Action Plan >

  92. [103]

    The < Action Plan > should be step - by - step actions to take in the doc file environment

  93. [104]

    Each action should be in the available actions from < Available Actions >

  94. [105]

    observation

    Each action should be generated with a " step " description which is the function description of the action . # # Response Format - You are required to response in a JSON format , consisting of several distinct parts with the following keys and corresponding content : {{ " obs...

  95. [106]

    yes " ,

    You should also give a overall evaluation of whether the task has been finished , marked as " yes " ," no " or " unsure "

  96. [107]

    ambiguous

    You should also give a overall evaluation of the quality of task , marked as " ambiguous " ," over - detailed " or " good " . Criteria for evaluation of the task completion :

  97. [108]

    If the controls or canvas content expected to be changed are not changed , the task is not completed

    The < Final Control State : > and < Final Env Status : > should be consistent with the task requirements . If the controls or canvas content expected to be changed are not changed , the task is not completed

  98. [109]

    If the agent actions are not consistent with the task requirements , the task is not completed

    The < Execution Trajectory > should be consistent with the task requirements . If the agent actions are not consistent with the task requirements , the task is not completed

  99. [110]

    Criteria for evaluation of the task quality :

    If any action in the < Execution Trajectory > is empty , the task is not completed . Criteria for evaluation of the task quality :

  100. [111]

    selection

    The description of the < Original Request : > should be clear and unambiguous , without the meaning of " selection "

  101. [112]

    task_quality

    The description of the < Original Request : > should not be too detailed like step - by - step actions . # # Response Format You must strictly follow the below JSON format for your reply , and do not change the format nor output additional information . {{ " task_quality " : T...

  102. [113]

    Yes " or

    Can < Answer1 > solve the user s question ? - Your answer should be " Yes " or " No "

  103. [114]

    Yes " or

    Can < Answer2 > solve the user s question ? - Your answer should be " Yes " or " No "

  104. [115]

    similar action item from action item set1 / similar action item from action item set2

    Both two answers contain a list of steps marked by numbers . Your task is to extract action items from the provided steps in both answers . The action item is defined like a combination of action and element . Compare the action items to identify similarities . Output the simi...

  105. [116]

    1" or "2

    Which assistant provides a more helpful response ? - Your answer should be "1" or "2" , where "1" represents < Answer1 > and "2" represents < Answer2 >. - Your answer should contain the reason ( s ) for your choice . You should not focus on the length of the answer or the deta...

  106. [2017]

    arXiv preprint arXiv:1707.06347 (2017)

    Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  107. [2023]

    Advances in Neural Information Processing Systems 36 (2023), 79081–79094

    Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems 36 (2023), 79081–79094

    2023

  108. [2024]

    arXiv preprint arXiv:2402.04615 (2024)

    Screenai: A vision-language model for ui and infographics understanding. arXiv preprint arXiv:2402.04615 (2024)

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.