REVIEW 19 cited by
GenSim: Generating Robotic Simulation Tasks via Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data. However, existing methods for data generation have generally focused on scene-level diversity (e.g., object instances and poses) rather than task-level diversity, due to the human effort required to come up with and verify novel tasks. This has made it challenging for policies trained on simulation data to demonstrate significant task-level generalization. In this paper, we propose to automatically generate rich simulation environments and expert demonstrations by exploiting a large language models' (LLM) grounding and coding ability. Our approach, dubbed GenSim, has two modes: goal-directed generation, wherein a target task is given to the LLM and the LLM proposes a task curriculum to solve the target task, and exploratory generation, wherein the LLM bootstraps from previous tasks and iteratively proposes novel tasks that would be helpful in solving more complex tasks. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks, on which we conduct supervised finetuning and evaluate several LLMs including finetuned GPTs and Code Llama on code generation for robotic simulation tasks. Furthermore, we observe that LLMs-generated simulation programs can enhance task-level generalization significantly when used for multitask policy training. We further find that with minimal sim-to-real adaptation, the multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen long-horizon tasks in the real world and outperform baselines by 25%. See the project website (https://liruiw.github.io/gensim) for code, demos, and videos.
Forward citations
Cited by 19 Pith papers
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation
Dense per-frame intermediate representations (traces, masks, grasp poses, subtasks) improve embodied VQA, VLA action generation, and world-model video prediction in the new 230k-episode RoboInter-Data suite.
-
PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution
A file-based operating-system layer with a session verifier and persistent memory improves embodied-agent task completion on game, simulated, and real-robot platforms without retraining policies.
-
LLM Trainer: Automated Robotic Data Generation via Demonstration Augmentation using LLMs
An LLM-based pipeline automatically augments one human demonstration into a large imitation-learning dataset, using Thompson sampling to pick the best annotation and beating expert-annotated baselines on most tasks.
-
GhostShell: Streaming LLM Function Calls for Concurrent Embodied Programming
A streaming XML function-token interface with multi-channel scheduling lets robots execute concurrent speech and motion while the LLM is still generating, reportedly beating native function calling 15/15 vs 6/15 on co...
-
RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills
RobotSmith autonomously designs, 3D-prints, and uses task-specific tools for robotic manipulation, raising task success from 2.8% (no tool) to 50% in simulation.
-
GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation
GenManip is a benchmark and simulation platform with LLM-generated scene graphs for testing how robot policies generalize to new instructions, layouts, and objects.
-
ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
ManiTaskGen automatically generates diverse, feasible mobile manipulation tasks from any input scene, and uses them to benchmark and improve vision-language robot agents.
-
Real-Time Verification of Embodied Reasoning for Generative Skill Acquisition
VERGSA trains a process reward model on MCTS-labeled subtask outcomes and uses it to select scene configurations and subtask supervisions, improving simulated task success rates.
-
Steerable Scene Generation with Post Training and Inference-Time Search
A single diffusion scene prior over SE(3) object sets can be steered by RL post training, text conditioning, and MCTS search, backed by a new 44 million scene dataset.
-
RoboGround: Robotic Manipulation with Grounded Vision-Language Priors
RoboGround uses object and placement-area masks from a grounded vision-language model as intermediate guidance, significantly improving simulated robots' generalization to novel objects, categories, and instructions.
-
VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion
A framework that reconstructs real scenes as interactive 3D Gaussian simulations and trains RGB-only navigation policies for legged robots that transfer to the real world without retraining.
-
MobileH2R: Learning Generalizable Human to Mobile Robot Handover Exclusively from Scalable and Diverse Synthetic Data
A pipeline generates 100K+ synthetic human handover scenes and safe demonstrations to train a vision-based mobile robot handover policy that transfers to the real world.
-
EvoCurr: Self-evolving Curriculum with Behavior Code Generation for Complex Decision-making
EvoCurr couples an LLM curriculum designer with an LLM code-generating solver, but its only reported success is 1 of 5 runs and no direct baseline is shown.
-
SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training
Simulation-pretrained policies, with digital-twin demos for critic bootstrapping and action proposals, cut real-world RL training time while reaching near-perfect success on three manipulation tasks.
-
LLM-based Property-based Test Generation for Guardrailing Cyber-Physical Systems
An LLM-based tool, ChekProp, extracts safety properties from CPS code and documentation and generates property-based tests that can serve as design-time checks and proposed runtime guardrails.
-
RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning
RoboVerse unifies seven simulators, 15 benchmarks, and 510.5k migrated trajectories into one platform with a four-level generalization benchmark, claiming better robot learning and sim-to-real transfer.
-
Multi-agent Embodied AI: Advances and Future Directions
A survey that maps multi-agent embodied AI methods and benchmarks across control, learning, and generative-model categories, and lists open challenges.
-
Foundation Model Driven Robotics: A Comprehensive Review
A review of foundation-model-driven robotics that synthesizes recent work across perception, planning, control, HRI, simulation, and sim-to-real transfer, and highlights open challenges.
Discussion (0). Continue with ORCID to comment.