{"work":{"id":"7b5f6cce-bbaa-40ed-8b09-7330832dd736","openalex_id":"https://openalex.org/W4303648971","doi":"10.48550/arxiv.2210.03094","arxiv_id":"2210.03094","raw_key":null,"title":"VIMA: General Robot Manipulation with Multimodal Prompts","authors":null,"authors_text":"Jiang, Y","year":2022,"venue":"cs.RO","abstract":"Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and a four-level evaluation protocol for systematic generalization. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. VIMA features a recipe that achieves strong model scalability and data efficiency. It outperforms alternative designs in the hardest zero-shot generalization setting by up to $2.9\\times$ task success rate given the same training data. With $10\\times$ less training data, VIMA still performs $2.7\\times$ better than the best competing variant. Code and video demos are available at https://vimalabs.github.io/","external_url":"https://arxiv.org/abs/2210.03094","cited_by_count":65,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2210.03094","created_at":"2026-05-10T22:29:29.901668+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Jiang, A","render_title":"Jiang, A"},"hub":{"state":{"work_id":"7b5f6cce-bbaa-40ed-8b09-7330832dd736","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":26,"external_cited_by_count":65,"distinct_field_count":4,"first_pith_cited_at":"2023-02-22T18:47:51+00:00","last_pith_cited_at":"2026-07-07T13:24:37+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-20T07:19:42.545849+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":6},{"context_role":"dataset","n":1}],"polarity_counts":[{"context_polarity":"background","n":7}],"runs":{},"summary":{},"graph":{},"authors":[]}}