REVIEW 55 cited by
VIMA: General Robot Manipulation with Multimodal Prompts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and a four-level evaluation protocol for systematic generalization. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. VIMA features a recipe that achieves strong model scalability and data efficiency. It outperforms alternative designs in the hardest zero-shot generalization setting by up to $2.9\times$ task success rate given the same training data. With $10\times$ less training data, VIMA still performs $2.7\times$ better than the best competing variant. Code and video demos are available at https://vimalabs.github.io/
Forward citations
Cited by 55 Pith papers
-
RoboTTT: Context Scaling for Robot Policies
A robot policy that updates its own weights during deployment can use 8,000 steps of history, steadily improving as context grows and enabling one-shot imitation from human videos.
-
TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
TeamCraft presents a large multi-modal, multi-agent Minecraft benchmark and shows that current models generalize poorly to novel goals, scenes, and team sizes.
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
Diagnosing Compositional Generalization in Sequential Robot Tasks
Exhaustive combination coverage is unnecessary; a structured subset covering instruction pairs can match full-set OOD performance in simulated sequential manipulation.
-
RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation
Dense per-frame intermediate representations (traces, masks, grasp poses, subtasks) improve embodied VQA, VLA action generation, and world-model video prediction in the new 230k-episode RoboInter-Data suite.
-
Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition
VLA skills that score 77-100% in isolation stall from chained states in BEHAVIOR-1K; failures trace to next-skill readiness, target grounding, and control execution.
-
Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation
Sparse retry keypoints plus pairwise preference learning yield mistake-sensitive values that reweight mixed-quality demos and raise real-robot imitation success over progress-based baselines.
-
TempoVLA: Learning Speed-Controllable Vision-Language-Action Policies
TempoVLA learns a single VLA policy with controllable execution speed via variable-speed trajectory augmentation and explicit speed conditioning.
-
PrimitiveVLA: Learning Reusable Motion Primitives for Efficient and Generalizable Robotic Manipulation
PrimitiveVLA introduces a primitive-centric framework that disassembles demonstrations into reusable motion primitives during fine-tuning and assembles them at inference via VLM planner and LLM switch for improved dat...
-
Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models
Adding a real-world supervised loss to simulation reinforcement learning improves real-robot success and data efficiency for VLA co-training.
-
LLaDA-VLA: Vision Language Diffusion Action Models
LLaDA-VLA applies a masked diffusion vision-language model to robot control with localized action-token classification and hierarchical decoding, achieving SOTA success rates on SimplerEnv, CALVIN, and real-robot tasks.
-
Spatial Traces: Enhancing VLA Models with Spatial-Temporal Understanding
ST-VLA combines depth maps with projected keypoint traces to enhance spatial-temporal understanding in vision-language-action models, reporting a 4 point success rate gain over SpatialVLA in SimplerEnv.
-
GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming
A streaming XML function-token interface with multi-channel scheduling lets robots execute concurrent speech and motion while the LLM is still generating, reportedly beating native function calling 15/15 vs 6/15 on co...
-
DEMONSTRATE: Zero-shot Language to Robotic Control via Multi-task Demonstration Learning
DEMONSTRATE learns a zero-shot mapping from natural-language embeddings to MPC cost parameters from demonstrations, achieving tabletop manipulation success rates comparable to prior LLM-based pipelines.
-
RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
RoboMonkey shows that test-time sampling with Gaussian perturbation and a VLM-based action verifier improves the success rate of vision-language-action models on manipulation tasks.
-
ROSA: Harnessing Robot States for Vision-Language and Action Alignment
ROSA trains a VLA model jointly on expert actions and automatically recorded robot states, improving success rates and generalization, particularly with few demonstrations.
-
GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation
GenManip is a benchmark and simulation platform with LLM-generated scene graphs for testing how robot policies generalize to new instructions, layouts, and objects.
-
Hierarchical Instruction-aware Embodied Visual Tracking
HIEVT uses an LLM to convert natural language instructions into spatial goals (bounding boxes) and an offline RL policy to track targets to those goals, claiming strong generalization across environments.
-
Rethinking Agent Design: From Top-Down Workflows to Bottom-Up Skill Evolution
Agents that start with no game knowledge can build a reusable skill library through trial-and-error and visual feedback, then progress further in two complex games than baseline agents given extra hints.
-
CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding
A contrastive pre-training framework and dataset that map tactile 3D point clouds to contact-state language descriptions, enabling zero-shot and LLM-based tactile understanding.
-
TesserAct: Learning 4D Embodied World Models
A 4D embodied world model that generates RGB-depth-normal videos from an image and instruction, reconstructs the scene as point clouds, and uses those point clouds to train better robot manipulation policies.
-
RL-Driven Data Generation for Robust Vision-Based Dexterous Grasping
A vision-based dexterous grasping policy trained on RL-generated simulated trajectories mixed with 40 real demonstrations achieves 100% real-world success on all tested objects.
-
A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards
IKER uses VLM-generated keypoint rewards to train manipulation policies in simulation that transfer to a real robot, enabling multi-step tasks and replanning.
-
MuST: Multi-Head Skill Transformer for Long-Horizon Dexterous Manipulation with Skill Progress
MuST adds per-skill action heads and a progress-guided skill selector to the Octo robot policy, improving long-horizon pick-and-pack success from about 32% to 90% in one simulated setting.
-
AutoLife: Automatic Life Journaling with Smartphones and LLMs
AutoLife, a layered pipeline that fuses motion, time, GPS, and WiFi contexts, generates life journal text from smartphone sensors with BERTScore F1 near 0.70.
-
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
A single MLLM-based agent, finetuned with cross-domain supervision and online RL, achieves strong zero-shot generalization across manipulation, navigation, games, UI control, and planning.
-
CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction
A coarse-to-fine autoregressive policy with multi-scale action tokenization matches or beats diffusion policies on robot manipulation benchmarks at roughly 10x lower inference cost.
-
Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoning
A robot policy generates its own language reasoning before acting and injects it into a diffusion action decoder, outperforming several VLA baselines on real-robot manipulation.
-
Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning
Tra-MoE shows that a sparsely-gated MoE version of the ATM trajectory model improves with out-of-domain video data, while the dense baseline degrades.
-
Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics
Adversarial patches optimized against OpenVLA raise manipulation failure rates from about 23% to 100% in LIBERO simulation and disrupt a physical robot arm in 43% of trials.
-
Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulation
LifelongVLA pairs dual-timescale LoRA gating with stochastic cached-prefix replay to cut catastrophic forgetting in VLA policies, reporting 83.2% average success and 11.4% forgetting on a 10-task LIBERO stream.
-
Qwen-Audio-VAE Technical Report
A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.
-
TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making
TCPO uses stepwise preference optimization and an action consistency constraint to raise ALFWorld average success from 20.0% to 26.7%.
-
Towards Deeper Understanding of Natural User Interactions in Virtual Reality Based Assembly Tasks
Untrained VR users adapt their multimodal instruction strategies to the spatial clarity of the task, using explicit speech with concrete anchors and implicit speech with prolonged pointing in ambiguous settings.
-
LLMs-guided adaptive compensator: Bringing Adaptivity to Automatic Control Systems with Large Language Models
An LLM-generated adaptive compensator, refined through iterative prompting, outperformed classical adaptive controllers on soft and humanoid robots in simulation and prototype tests.
-
ACTLLM: Action Consistency Tuned Large Language Model
ACTLLM trains an LLM to jointly produce structured scene descriptions and actions, and reports improved compositional and zero-shot generalization on CLIPORT and VIMA.
-
CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation
CrayonRobo trains a vision-language-action model to read colored 2D prompt overlays (contact point, end-effector axes, movement direction) and output SE(3) contact poses, enabling step-by-step and long-horizon robotic...
-
Few-Shot Vision-Language Action-Incremental Policy Learning
TOPIC adds task-specific prompts and task-similarity-based weight interpolation to transformer policies, improving few-shot action-incremental learning in simulation and on a real robot.
-
Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
A DPO-trained VLM critic that critiques and iteratively refines a reasoning VLM improves accuracy on several multimodal benchmarks, with large gains on MathVista and RealWorldQA.
-
Simple-to-Complex Structured Demonstrations for Vision-Language-Action Learning
Simple-to-complex staged demonstration collection (task decomposition, environment standardization, progressive complexity) substantially improves π0.5 VLA success on dual-arm block sorting and towel folding versus en...
-
Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
A review that categorizes large-model-empowered embodied AI into hierarchical and end-to-end decision-making, imitation and reinforcement learning, and world models.
-
HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and Decision in Embodied Agents
HyCodePolicy closes the loop between generated robot code, visual checkpoint monitoring, and iterative repair, raising average success rates on 10 simulated manipulation tasks by up to 16.5 points over one-shot code-a...
-
Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning
A benchmark of 15 multimodal LLMs on grid path planning reports modest success on 8x8 grids and near-failure on 20x20 grids, but its visual-vs-text comparison is confounded by prompt differences.
-
LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Tasks
A unified vision-language-action model that emits a sub-task description followed by a discrete action token outperforms modular and action-only baselines on simulated long-horizon tabletop tasks.
-
Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agents
An agentic framework using a GPT-4o planner, an OpenVLA executor, and a LoRA-fine-tuned Qwen2.5-VL verifier achieves 79.6% average success on LIBERO by decomposing and verifying subgoals.
-
Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models
A survey organizes multimodal reasoning research into a staged roadmap and proposes native large multimodal reasoning models that unify perception, generation, and agentic planning.
-
A Survey of Interactive Generative Video
A survey that divides interactive generative video research into five modules: generation, control, memory, dynamics, and intelligence.
-
When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis
SeeUnsafe uses off-the-shelf multimodal LLMs with visual prompts and severity-based aggregation to classify traffic videos and identify the road users involved in critical events.
-
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.
-
RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-World
A skill-centric hierarchical framework with a unified vision-language-action model executes new tasks by recombining eight meta-skills, reporting up to 50 percentage points higher success than task-centric baselines.
-
Data Pyramid for Embodied Manipulation: A Survey
Embodied training data form a five-layer pyramid—real-robot, UMI, ego/exo, simulation, general V–L—ordered by the trade-off between scale and robot alignment, and model capabilities track how those layers are mixed.
-
Vision Generalist Model: A Survey
A structured review of vision generalist models, classifying them into encoding-based and sequence-to-sequence frameworks and summarizing datasets, benchmarks, techniques, and open problems.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
-
A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
A survey that catalogs large vision-language models, their alignment methods, benchmarks, and challenges, but is compromised by inconsistent counts and misclassified entries.
-
Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions
This is a broad survey of medical multimodal foundation models that categorizes datasets, proxy tasks, vision-language models, and clinical applications without introducing new results.
Discussion (0). Continue with ORCID to comment.