REVIEW 6 cited by
MobileFlow: A Multimodal LLM For Mobile GUI Agent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Currently, the integration of mobile Graphical User Interfaces (GUIs) is ubiquitous in most people's daily lives. And the ongoing evolution of multimodal large-scale models, such as GPT-4v, Qwen-VL-Max, has significantly bolstered the capabilities of GUI comprehension and user action analysis, showcasing the potentiality of intelligent GUI assistants. However, current GUI Agents often need to access page layout information through calling system APIs, which may pose privacy risks. Fixing GUI (such as mobile interfaces) to a certain low resolution might result in the loss of fine-grained image details. At the same time, the multimodal large models built for GUI Agents currently have poor understanding and decision-making abilities for Chinese GUI interfaces, making them difficult to apply to a large number of Chinese apps. This paper introduces MobileFlow, a multimodal large language model meticulously crafted for mobile GUI agents. Transforming from the open-source model Qwen-VL-Chat into GUI domain, MobileFlow contains approximately 21 billion parameters and is equipped with novel hybrid visual encoders, making it possible for variable resolutions of image inputs and good support for multilingual GUI. By incorporating Mixture of Experts (MoE) expansions and pioneering alignment training strategies, MobileFlow has the capacity to fully interpret image data and comprehend user instructions for GUI interaction tasks. Finally, MobileFlow outperforms Qwen-VL-Max and GPT-4v in terms of task execution by GUI agents on both public and our proposed evaluation metrics, and has been successfully deployed in real-world business contexts, proving its effectiveness for practical applications.
Forward citations
Cited by 6 Pith papers
-
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning
A tuning-free method that aggregates selected attention maps in a pretrained multimodal LLM achieves GUI grounding accuracy comparable to fine-tuned systems, especially for text.
-
MageBench: Bridging Large Multimodal Models to Agents
MageBench introduces a 483-scenario benchmark showing current large multimodal models are far weaker than humans at agent tasks requiring continuous visual feedback and planning.
-
Software Engineering for and with GUI Agent
A survey of 336 GUI-agent papers finds rapid growth alongside weak engineering support for recovery, human oversight, maintainability, and privacy, and calls for lifecycle-centered testing and governance.
-
Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation
ASR replaces the vision encoder of a multimodal LLM with graph-derived structural features to generate UI layouts, reporting better overlap and relation metrics than four prior methods.
-
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
A 2B multimodal agent trained with two-stage supervised fine-tuning and synthesized hierarchical/reflection reasoning achieves competitive results on ScreenSpot and AndroidWorld.
-
Generative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability
A survey of generative AI in multimodal user interfaces, recommending hybrid interface designs and lightweight on-device frameworks.
Discussion (0). Continue with ORCID to comment.