REVIEW 8 cited by
MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
It is a long-lasting goal to design a generalist-embodied agent that can follow diverse instructions in human-like ways. However, existing approaches often fail to steadily follow instructions due to difficulties in understanding abstract and sequential natural language instructions. To this end, we introduce MineDreamer, an open-ended embodied agent built upon the challenging Minecraft simulator with an innovative paradigm that enhances instruction-following ability in low-level control signal generation. Specifically, MineDreamer is developed on top of recent advances in Multimodal Large Language Models (MLLMs) and diffusion models, and we employ a Chain-of-Imagination (CoI) mechanism to envision the step-by-step process of executing instructions and translating imaginations into more precise visual prompts tailored to the current state; subsequently, the agent generates keyboard-and-mouse actions to efficiently achieve these imaginations, steadily following the instructions at each step. Extensive experiments demonstrate that MineDreamer follows single and multi-step instructions steadily, significantly outperforming the best generalist agent baseline and nearly doubling its performance. Moreover, qualitative analysis of the agent's imaginative ability reveals its generalization and comprehension of the open world.
Forward citations
Cited by 8 Pith papers
-
Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics
A 3D-aware VLM, RoboTracer, generates metric-grounded spatial traces for robot manipulation using scale supervision and metric-sensitive reinforcement rewards.
-
CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusion
Causal Diffusion Policy adds historical action conditioning and attention cache sharing to diffusion-based robot policies, improving success rates on most tested manipulation tasks under degraded observations.
-
BAR: A Backward Reasoning based Agent for Complex Minecraft Tasks
A backward-reasoning LLM agent with recursive goal decomposition, state-consistency repair, and stage memory outperforms forward-planning baselines on Minecraft task planning.
-
Solving New Tasks by Adapting Internet Video Knowledge
Inverse Probabilistic Adaptation, a score-composition variant that keeps the large video model as the base and consults a small in-domain model, achieves 68.3% average success on MetaWorld policy supervision and stays...
-
Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection
A VLM-generated code monitor tracks geometric elements and evaluates spatio-temporal constraints, enabling real-time reactive and proactive failure detection for robotic manipulation.
-
Number it: Temporal Grounding Videos like Flipping Manga
Number-Prompt overlays frame numbers on video frames, improving temporal grounding in video LLMs and setting new state-of-the-art results on moment retrieval and highlight detection.
-
A Survey of Interactive Generative Video
A survey that divides interactive generative video research into five modules: generation, control, memory, dynamics, and intelligence.
-
STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft
An audio-conditioned STEVE-1 agent, built with a new Minecraft audio-video CLIP model and a learned prior, matches or beats text- and video-conditioned versions on most short-horizon collection tasks.
Discussion (0). Continue with ORCID to comment.