REVIEW 16 cited by
ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Screen user interfaces (UIs) and infographics, sharing similar visual language and design principles, play important roles in human communication and human-machine interaction. We introduce ScreenAI, a vision-language model that specializes in UI and infographics understanding. Our model improves upon the PaLI architecture with the flexible patching strategy of pix2struct and is trained on a unique mixture of datasets. At the heart of this mixture is a novel screen annotation task in which the model has to identify the type and location of UI elements. We use these text annotations to describe screens to Large Language Models and automatically generate question-answering (QA), UI navigation, and summarization training datasets at scale. We run ablation studies to demonstrate the impact of these design choices. At only 5B parameters, ScreenAI achieves new state-of-the-artresults on UI- and infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and InfographicVQA) compared to models of similar size. Finally, we release three new datasets: one focused on the screen annotation task and two others focused on question answering.
Forward citations
Cited by 16 Pith papers
-
Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering
FRAME augments CLIP and BERT embeddings with a UI component graph, spectral propagation, and Vietoris-Rips triangle weighting, improving screen retrieval and clustering on three Android UI benchmarks.
-
VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model with Structured Visual Reasoning and Native Tool Use
A sub-2B Spanish cybersecurity vision-language model is released with a negative visual-grounding result and an ablation design for the NoPE positional-encoding question.
-
GUI-AC: Enhancing Continual Learning in GUI Agents
GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.
-
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation
WebMMU introduces a multilingual, three-task benchmark for website understanding and code generation, and finds current MLLMs underperform on reasoning, grounding, and functional code editing.
-
Task Mode: Dynamic Filtering for Task-Specific Web Navigation using LLMs
An LLM-powered browser extension that filters webpages to task-relevant content reduced screen reader users' task completion time by about half in a 12-participant study.
-
Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation
Explicitly storing short-term and long-term text memories, instead of raw screenshots, improves GUI agent accuracy on cross-app tasks, and a new annotated dataset helps 7B models approach 72B-level memory use.
-
AndroidGen: Building an Android Language Agent under Data Scarcity
AndroidGen combines retrieval, reflection, operation checking, and step-by-step evaluation to build Android agents that outperform prior systems on AndroidWorld and AitW without human-labeled trajectories.
-
WEPO: Web Element Preference Optimization for LLM-based Web Navigation
Applying DPO with DOM-distance-based negative samples improves LLM web navigation on Mind2Web.
-
Large Action Models: From Inception to Implementation
A four-phase training pipeline converts a 7B language model into a Windows GUI action model that reaches 81.2% offline and 71.0% online task success on the authors' Word test set, beating text-only GPT-4o.
-
DocVLM: Make Your VLM an Efficient Reader
DocVLM compresses OCR text and layout into 64 learned queries that are injected into frozen VLMs, boosting DocVQA accuracy at low image-token budgets across three model families.
-
Software Engineering for and with GUI Agent
A survey of 336 GUI-agent papers finds rapid growth alongside weak engineering support for recovery, human oversight, maintainability, and privacy, and calls for lifecycle-centered testing and governance.
-
IDEA: Augmenting Design Intelligence through Design Space Exploration
IDEA combines LLM-generated constraints with Monte Carlo Tree Search over a formal design space to automate design decision-making in data storytelling and pictorial visualization.
-
Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development
Fine-tuning GPT-4o on UI metadata and icon images generates icon alt-text at development time that outperforms prior deep-learning and vision-language baselines on the WC20 icon benchmark.
-
HAUR: Human Annotation Understanding and Recognition Through Text-Heavy Images
A new task and synthetic dataset for recognizing human annotations in text-heavy images, with an OCR-fused Pix2Struct model that reports state-of-the-art results on the new benchmark.
-
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
A 2B multimodal agent trained with two-stage supervised fine-tuning and synthesized hierarchical/reflection reasoning achieves competitive results on ScreenSpot and AndroidWorld.
-
MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
A broad survey that organizes MLLM evaluation benchmarks into capability categories, explains benchmark construction and scoring methods, and identifies gaps in current evaluation practice.
Discussion (0). Continue with ORCID to comment.