REVIEW 40 cited by
CogAgent: A Visual Language Model for GUI Agents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
People are spending an enormous amount of time on digital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks like writing emails, but struggle to understand and interact with GUIs, thus limiting their potential to increase automation levels. In this paper, we introduce CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation. By utilizing both low-resolution and high-resolution image encoders, CogAgent supports input at a resolution of 1120*1120, enabling it to recognize tiny page elements and text. As a generalist visual language model, CogAgent achieves the state of the art on five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using only screenshots as input, outperforms LLM-based methods that consume extracted HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW, advancing the state of the art. The model and codes are available at https://github.com/THUDM/CogVLM, with a new version of CogAgent-9B-20241220 available at https://github.com/THUDM/CogAgent.
Forward citations
Cited by 40 Pith papers
-
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.
-
Grounding Computer Use Agents on Human Demonstrations
GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.
-
LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation
A new benchmark with 58,500 AI videos and 2.6 million human annotations, plus an LMM-based metric that predicts perceptual quality, prompt-video correspondence, and task-specific correctness.
-
Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting
EC-Bench finds best MLLMs score 29.98% enumeration F1 and 23.74% counting accuracy on 152 hour-scale videos, with errors driven by instance identification and temporal grounding rather than arithmetic.
-
AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents
AppDeltaWorld predicts mobile GUI transitions as code updates retrieved under action constraints, and its generated trajectories improve an 8B mobile agent on several benchmarks.
-
GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
Action-weighted SFT plus KL-regularized GRPO on a curated 81K reasoning dataset lifts open-source GUI agents by 11–22 points on online task-completion benchmarks.
-
UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
Training computer-use agents on a hybrid GUI-plus-programmatic-tool action space, with SFT and online RL, improves OSWorld success by ~23% relative over GUI-only baselines and transfers to Windows without Windows-spec...
-
MobiAgent: A Systematic Framework for Customizable Mobile Agents
A full-stack mobile agent framework reports state-of-the-art task completion on its own DAG-based benchmark and 2-3x speedups from replaying recorded trajectories.
-
CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning
A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.
-
SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
A self-evolving computer-use agent trained with full-trajectory state judging and curriculum task generation goes from 11.3% to 34.5% average success on five OSWorld apps, and a specialist-to-generalist variant beats ...
-
Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation
Screen2AX generates hierarchical macOS accessibility metadata from a screenshot and reports improved GPT-4 UI task success compared with native accessibility and OmniParser V2.
-
3DGS-IEval-15K: A Large-scale Image Quality Evaluation Database for 3D Gaussian-Splatting
A human-rated dataset of 15,200 compressed 3D Gaussian Splatting images from 10 scenes and 6 algorithms, benchmarked against 30 quality metrics.
-
Exploring the Potential of Metacognitive Support Agents for Human-AI Co-Creation
Metacognitive support agents, simulated by human wizards, improved the feasibility of AI-generated mechanical designs in a 20-participant formative study.
-
AndroidGen: Building an Android Language Agent under Data Scarcity
AndroidGen combines retrieval, reflection, operation checking, and step-by-step evaluation to build Android agents that outperform prior systems on AndroidWorld and AitW without human-labeled trajectories.
-
TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents
A training-free agentic framework, combining SAM-based region parsing with spatially enhanced element descriptions, improves LVLM action grounding and GUI referring on multiple benchmarks.
-
InSTA: Towards Internet-Scale Training For Agents
Automated LLM task generation, agent execution, and judge filtering at 150k-site scale lets a 1.7B model match much larger web agents.
-
Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
Eagle2-9B matches or outperforms much larger vision-language models on many benchmarks through a carefully constructed post-training data strategy.
-
LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer
A hierarchical window transformer that injects image details into upsampled CLIP features and compresses them with cross-scale window attention improves MLLM fine-grained perception by 3.7% on average over LLaVA-UHD.
-
Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval
A one-stage, training-free method using GPT-4o with reflective chain-of-thought prompting sets new state-of-the-art benchmark numbers for composed image retrieval, though code and a full baseline comparison are not ye...
-
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning
A tuning-free method that aggregates selected attention maps in a pretrained multimodal LLM achieves GUI grounding accuracy comparable to fine-tuned systems, especially for text.
-
WEPO: Web Element Preference Optimization for LLM-based Web Navigation
Applying DPO with DOM-distance-based negative samples improves LLM web navigation on Mind2Web.
-
Learning to Correction: Explainable Feedback Generation for Visual Commonsense Reasoning Distractor
A new GPT-4-generated dataset of distractors and corrective feedback for visual commonsense reasoning, plus a compact LMM (PEIFG) that produces explainable corrections and beats existing baselines in automatic and hum...
-
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
ShowUI is a lightweight 2B vision-language-action model that uses UI-guided token pruning and a curated 256K dataset to reach 75.1% zero-shot screenshot grounding accuracy.
-
Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
Tracking cross-layer attention momentum lets training-free pruning reactivate late-rising visual tokens, cutting final-layer tokens to ~23 at 77.8% prune while keeping near-baseline multimodal accuracy.
-
Qwen-Audio-VAE Technical Report
A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.
-
TiP4GEN: Text to Immersive Panorama 4D Scene Generation
TiP4GEN generates motion-rich, geometry-consistent 360-degree 4D scenes from a global text prompt plus four local perspective prompts, using a dual-branch video diffusion model with bidirectional cross-attention and a...
-
DFBench: Benchmarking Deepfake Image Detection Capability of Large Multimodal Models
DFBench adds a 540,000-image benchmark with 12 modern generators, partial edits, and distorted real images, and its three-model LMM ensemble, MoA-DF, reaches near-perfect recall on its own test split.
-
Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation
ASR replaces the vision encoder of a multimodal LLM with graph-derived structural features to generate UI layouts, reporting better overlap and relation metrics than four prior methods.
-
Visual Instruction Tuning with Chain of Region-of-Interest
CoRoI injects a chain of language-guided image regions into LLM hidden layers and reports improved MLLM benchmark scores at 7B-34B scale.
-
Digi-Q: Learning Q-Value Functions for Training Device-Control Agents
An offline RL method learns a Q-function from frozen VLM features and extracts a device-control policy by imitating the best of several actions ranked by that Q-function.
-
EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
An encoder-free vision-language model using separate attention, normalization, and feed-forward weights for image versus text tokens outperforms earlier encoder-free models and narrows the gap to encoder-based VLMs wi...
-
Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding
PIIP is a multi-branch, parameter-inverted image pyramid that uses smaller pretrained networks for high-resolution inputs and larger networks for low-resolution inputs, improving efficiency across perception and multi...
-
Cybernaut: Towards Reliable Web Automation
A demonstration-to-SOP framework plus robust element identification and a trace similarity metric improves enterprise web automation success rates on an internal benchmark, with a fine-tuned consistency classifier rea...
-
Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement
A GUI agent that trims UI input with a recommendation module and asks users when decisions are ambiguous reports state-of-the-art success rates, though the interaction module is not benchmarked.
-
GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding
Modeling GUI elements as Gaussian distributions instead of binary targets yields 92.0% (ScreenSpot), 93.3% (ScreenSpot-v2), and 47.5% (ScreenSpot-Pro) for a 7B model, outperforming UI-TARS-72B by a relative 24.7% on t...
-
ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding
A 3B vision-language model fine-tuned with LoRA on a 24K example dataset with a two-stage schedule reaches the best reported accuracy among sub-4B models on GUI grounding benchmarks.
-
Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models
The MDI benchmark evaluates large multimodal models on age-stratified, real-world multiple-choice questions and finds GPT-4o leading at about 79 percent average accuracy.
-
Improved GUI Grounding via Iterative Narrowing
Iteratively narrowing the view around a VLM's predicted location improves GUI grounding accuracy on ScreenSpot, but the improvement is unreliable for fine-tuned models.
-
Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends
A structured overview of question answering over visually rich documents, comparing encoding, vision-only, and multi-page methods, and highlighting comparability issues in existing benchmarks.
-
Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms
A survey that proposes the Generalist Virtual Agent concept and taxonomies for agent environments, tasks, perceptions, actions, models, and evaluation, concluding that real-world-like environments favor human-like int...
Discussion (0). Continue with ORCID to comment.