REVIEW 4 cited by
LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The emergent large language/multimodal models facilitate the evolution of mobile agents, especially in mobile UI task automation. However, existing evaluation approaches, which rely on human validation or established datasets to compare agent-predicted actions with predefined action sequences, are unscalable and unfaithful. To overcome these limitations, this paper presents LlamaTouch, a testbed for on-device mobile UI task execution and faithful, scalable task evaluation. By observing that the task execution process only transfers UI states, LlamaTouch employs a novel evaluation approach that only assesses whether an agent traverses all manually annotated, essential application/system states. LlamaTouch comprises three key techniques: (1) On-device task execution that enables mobile agents to interact with realistic mobile environments for task execution. (2) Fine-grained UI component annotation that merges pixel-level screenshots and textual screen hierarchies to explicitly identify and precisely annotate essential UI components with a rich set of designed annotation primitives. (3) A multi-level application state matching algorithm that utilizes exact and fuzzy matching to accurately detect critical information in each screen, even with unpredictable UI layout/content dynamics. LlamaTouch currently incorporates four mobile agents and 496 tasks, encompassing both tasks in the widely-used datasets and our self-constructed ones to cover more diverse mobile applications. Evaluation results demonstrate LlamaTouch's high faithfulness of evaluation in real-world mobile environments and its better scalability than human validation. LlamaTouch also enables easy task annotation and integration of new mobile agents. Code and dataset are publicly available at https://github.com/LlamaTouch/LlamaTouch.
Forward citations
Cited by 4 Pith papers
-
Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation
An LLM prompted to conduct heuristic evaluation reported more usability issues on two apps than five human experts, but the ground truth included the LLM's own findings.
-
LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark
Few-shot human demonstrations, parsed into structured descriptions and retrieved by instruction similarity, substantially raise mobile GUI agent accuracy and success rates across multiple models.
-
Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction
On the OEDD benchmark, LLM agents select the worse of two actions below chance when the correct choice requires combining two earlier facts and ignoring a recent distractor in contexts over 1,615 tokens.
-
Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents
The paper calls for privacy and security to be core evaluation criteria for LLM-powered GUI agents through human-centered assessment, in-context consent, and built-in safeguards.
Discussion (0). Continue with ORCID to comment.