REVIEW 23 cited by
Embodied Task Planning with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Equipping embodied agents with commonsense is important for robots to successfully complete complex human instructions in general environments. Recent large language models (LLM) can embed rich semantic knowledge for agents in plan generation of complex tasks, while they lack the information about the realistic world and usually yield infeasible action sequences. In this paper, we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint, where the agent generates executable plans according to the existed objects in the scene by aligning LLMs with the visual perception models. Specifically, we first construct a multimodal dataset containing triplets of indoor scenes, instructions and action plans, where we provide the designed prompts and the list of existing objects in the scene for GPT-3.5 to generate a large number of instructions and corresponding planned actions. The generated data is leveraged for grounded plan tuning of pre-trained LLMs. During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations. Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin, which indicates the practicality of embodied task planning in general and complex environments.
Forward citations
Cited by 23 Pith papers
-
Attention at Rest Stays at Rest: Breaking Visual Inertia for Cognitive Hallucination Mitigation
Visual attention in MLLMs shows inertia that hinders cognitive inference on object relations, addressed by a training-free Inertia-aware Visual Excitation method that selects dynamically emerging tokens and applies an...
-
CookBench: A Long-Horizon Embodied Planning Benchmark for Complex Cooking Scenarios
CookBench is a new long-horizon cooking benchmark with 14,394 bilingual instructions and over 4,500 embodied cooking tasks that current AI models cannot yet solve autonomously.
-
PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation
PartInstruct is a new large-scale simulated benchmark with part-level language instructions and training demonstrations; current robot policies achieve at most 31.72% average success on it.
-
RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback
RFTF trains a value model on temporal state orderings to supply dense rewards for reinforcement fine-tuning of vision-language-action models, achieving an average success length of 4.296 on CALVIN ABC-D with the Seer-...
-
ConceptTree: Bringing Semantic Transparency to Black-Box Decision Making for Robotic Manipulation
ConceptTree trains a decision tree over VLM-supervised, human-interpretable visual concepts to select robotic manipulation skills, enabling traceable and intervenable high-level decisions.
-
APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts
A VLM planner that adaptively inserts latent visual thoughts of future states into its reasoning trace beats language-only and prior VLM planners on long-horizon kitchen tasks, especially under tight free space.
-
Any House Any Task: Scalable Long-Horizon Planning for Abstract Human Tasks
A reinforcement-trained LLM that decomposes abstract household requests into PDDL subgoals and solves them with a symbolic planner outperforms prompting and end-to-end planning baselines on long-horizon tasks.
-
Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following
ExRAP couples LLM planning with a temporal knowledge-graph memory and information-based exploration, improving success and efficiency for continual embodied instruction following.
-
ANNIE: Be Careful of Your Robots
The authors build a safety-centered benchmark and attack method that induces vision-language-action robot policies to violate ISO-based safety rules in a majority of tested episodes.
-
From reactive to cognitive: brain-inspired spatial intelligence for embodied agents
A brain-inspired navigation system stores landmarks, routes, and map-like voxel features in structured spatial memory and uses MLLM-powered retrieval to achieve strong results across object, instance, instruction, and...
-
Reinforced Reasoning for Embodied Planning
An SFT-plus-GRPO recipe lifts a 7B VLM to 35.6 percent success on EB-ALFRED versus 22.0 for GPT-4o-mini and 33.7 for Qwen2.5-VL-72B, with smaller but consistent gains on unseen EB-Habitat.
-
Learning Compositional Behaviors from Demonstration and Language
BLADE learns structured, planable action representations from language-annotated demonstrations and composes them with a symbolic planner, outperforming latent and LLM/VLM baselines on new manipulation tasks.
-
Political Actor Agent: Simulating Legislative System for Roll Call Votes Prediction with Large Language Models
PAA, a role-playing LLM agent with multi-view planning and leader-follower influence, reports 91.8-92.1% accuracy on U.S. House roll-call prediction.
-
Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data
OVM3D-Det automatically labels 3D boxes from RGB images and depth estimates, then trains a monocular 3D detector that beats naive baseline matching on novel classes in indoor and outdoor datasets.
-
RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning
A 3B VLM trained with SFT plus GRPO and an LCS-based reward reaches 55.3% on EmbodiedBench's EB-ALFRED, beating GPT-4o-mini and the 7B REBP planner.
-
Learn as Individuals, Evolve as a Team: Multi-agent LLMs Adaptation in Embodied Environments
LIET combines a finetuned local utility function with an iteratively updated shared knowledge list to improve multi-agent LLM planning, and it outperforms CoELA, ProAgent, and RoCo on C-WAH and TDW-MAT benchmarks.
-
Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions
Using robot actions and LLM visual reasoning, the system identifies deformability properties of unseen objects with up to 78.57% accuracy, which helps plan bin-packing at over 96% success after replanning.
-
Embodied CoT Distillation From LLM To Off-the-shelf Agents
DeDer distills LLM chain-of-thought reasoning into a two-tier small-language-model policy (rationale writer plus planner) and reports state-of-the-art ALFRED success rates for small-model embodied agents.
-
Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
A review that categorizes large-model-empowered embodied AI into hierarchical and end-to-end decision-making, imitation and reinforcement learning, and world models.
-
A Spatial Relationship Aware Dataset for Robotics
A new robot-acquired, spatial-relationship-labelled dataset is released and benchmarked, with qualitative evidence that explicit spatial cues improve ChatGPT 4o robot planning.
-
LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Tasks
A unified vision-language-action model that emits a sub-task description followed by a discrete action token outperforms modular and action-only baselines on simulated long-horizon tabletop tasks.
-
From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents
A structured survey that categorizes LLM-based social simulation into individual, scenario, and society simulation, with associated methods, benchmarks, and observed trends.
- TANGO: Training-free Embodied AI Agents for Open-world Tasks
Discussion (0). Continue with ORCID to comment.