REVIEW 13 cited by
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., "chopping trees" to long-horizon tasks, e.g., "obtaining a diamond pickaxe". JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task of $\texttt{ObtainDiamondPickaxe}$, JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. The project page is available at https://craftjarvis.org/JARVIS-1
Forward citations
Cited by 13 Pith papers
-
TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
TeamCraft presents a large multi-modal, multi-agent Minecraft benchmark and shows that current models generalize poorly to novel goals, scenes, and team sizes.
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control
A three-layer embodied memory (Executor, Sentry, Planner) with cross-modal Add/Update/Delete operations raises long-horizon VLA task progress to ~90% and reduces planner API calls versus flat memory.
-
CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter
CrafterDojo provides CrafterVPT, CrafterCLIP, and CrafterSteve-1 models plus datasets and benchmarks, making Crafter a usable testbed for general-purpose embodied agents.
-
Conditional Multi-Stage Failure Recovery for Embodied Agents
A conditional four-stage chain-prompting method for failure recovery improves success on the TEACH embodied-agent benchmark from 24.9% to 36.5% with the same plan and executor.
-
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
Embodied VideoAgent augments an LLM-based video agent with persistent object memory built from egocentric video, depth, and pose, plus VLM-based memory updates, reporting gains on Ego4D-VQ3D, OpenEQA, and EnvQA.
-
Plancraft: an evaluation dataset for planning with LLM agents
A new Minecraft crafting benchmark shows current LLM agents solve only a minority of planning tasks, and vision-language models almost never succeed on image observations.
-
MineStudio: A Streamlined Package for Minecraft AI Agent Development
A software framework that integrates seven engineering components to streamline Minecraft AI agent development.
-
GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents
A weakly supervised latent-variable agent improves multimodal instruction following by combining VAE self-imitating on unlabeled data with a likelihood-based alignment of labeled and video latents.
-
Visual Language Models as Operator Agents in the Space Domain
Vision-language models can act as spacecraft operators in the KSPDG simulator from screenshots, and fine-tuning OpenVLA on ten episodes shows preliminary promise for robotic satellite inspection.
-
FaGeL: Fabric LLMs Agent empowered Embodied Intelligence Evolution with Autonomous Human-Machine Collaboration
A smart-fabric LLM agent that learns from textual feedback via a token-level DualCUT alignment method, with an 11.3% Overcooked-AI score gain claimed.
-
WiS Platform: Enhancing Evaluation of LLM-Based Multi-Agent Systems Through Game-Based Analysis
An open platform and benchmark for evaluating LLM-based multi-agent systems through the 'Who is Spy?' game, including a leaderboard and behavioral analysis of ten models.
-
STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft
An audio-conditioned STEVE-1 agent, built with a new Minecraft audio-video CLIP model and a learned prior, matches or beats text- and video-conditioned versions on most short-horizon collection tasks.
Discussion (0). Continue with ORCID to comment.