REVIEW 29 cited by
Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation
read the original abstract
Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/
Forward citations
Cited by 29 Pith papers
-
HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning
HARBOR is a new agentic harness framework that automates robot RL workflows end-to-end across 16 tasks in manipulation, locomotion, and dexterous control, matching or exceeding default configurations while enabling si...
-
RoboBPP: Benchmarking Robotic Online Bin Packing with Physics-based Simulation
RoboBPP is a physics-based simulation benchmark for robotic online bin packing that supplies three real-world industrial datasets, extended stability and safety metrics, and an open leaderboard.
-
SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
An automated real-to-sim pipeline builds digital twins and affordance-preserving cousins from video, yielding sim evaluations that correlate with real robot policy success and zero-shot sim-to-real gains.
-
SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation
SimFoundry automates zero-shot real-to-sim scene generation from video, producing digital twins and cousins that enable policy training with 0.911 mean Pearson correlation to real-world results and 17-40% success gain...
-
Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience
SCORE constrains sim RL to the support of a real-data policy via flow steering, raising average success on eight dexterous tasks from 37.8% to 89.9%.
-
iMaC: Translating Actions into Motion and Contact Images for Embodied World Models
iMaC introduces image-based action tokens in a dual-branch architecture to improve future state prediction and control in embodied world models over vector-based baselines.
-
Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video
Video2Sim2Real turns a single human video into a deployable robot manipulation skill by reconstructing a digital twin, anchoring motions to object-centric simulator configurations, and bridging sim-to-real gaps with i...
-
Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
PACT uses demo-trained progress localization plus intervention preference pairs to correct inflated Bellman targets and align the actor, raising average real-robot success by 24.5% over HIL-SERL.
-
Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
PACT calibrates credit assignment in HIL-RL by penalizing Bellman targets on suboptimal segments using counterfactual advantages from human-policy preference pairs, yielding 24.5% higher success rates and 1.3x faster ...
-
Creative Robot Tool Use by Counterfactual Reasoning
Robots discover causal tool features through VLM suggestions and physics-based counterfactual perturbations in simulation, then transfer manipulation skills via conditioned keypoint matching.
-
A Principled Approach for Creating High-fidelity Synthetic Demonstrations for Imitation Learning
DMP retargeting within 3DGS scenes preserves expert motion shape and phase to create diverse yet high-fidelity demonstrations, yielding lower deviation, fewer collisions, and higher downstream policy success than plan...
-
Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations
RecGen achieves state-of-the-art 3D multi-object scene reconstruction from sparse RGB-D views by combining compositional synthetic scene generation with strong 3D shape priors, outperforming SAM3D by 30%+ in shape qua...
-
SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds
SIM1 converts sparse real demonstrations into high-fidelity synthetic data through physics-aligned simulation, yielding policies that match real-data performance at a 1:15 ratio with 90% zero-shot success on deformabl...
-
Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models
Adding a real-world supervised loss to simulation reinforcement learning improves real-robot success and data efficiency for VLA co-training.
-
TwinRL: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation
TwinRL expands RL exploration via digital twin reconstruction and twin RL warm-up to guide real-world learning, reaching near-100% success with 20 minutes of on-robot time across four tasks.
-
TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation
A training-free pipeline generates instance-level, physically interactive 3D tabletop scenes from text or one image, with a differentiable rotation optimizer and top-view spatial alignment for collision-free layouts.
-
Diffusion Policy Policy Optimization
DPPO fine-tunes diffusion policies via policy gradients and outperforms prior RL approaches for diffusion policies and PG-tuned alternatives on robot benchmarks while enabling stable training and hardware deployment.
-
Shared Voxel-Map-Based Cooperative Indoor UAV Guidance with a Multi-Agent Soft Actor-Critic Controller
A multi-agent SAC controller using a shared voxel-map BEV representation achieves 90.3% simulated corridor success and 100% success across 50 real two-drone indoor trials after A*-based imitation fine-tuning.
-
Active Real-World Factor-Based Evaluation for Generalist Robot Policies
An active evaluation framework selects the most informative task configurations for real-robot tests, matching random testing's accuracy in 20-40% fewer trials.
-
ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos
ArtiTwinSplat creates interactable digital twins of articulated objects from RGB-D videos via Gaussian Splatting and automatic part and joint discovery.
-
Scalable Multi-Task Data Generation via Reinforcement Learning for Language-Conditioned Bimanual Dexterous Manipulation
An RL data generation pipeline with generalizable rewards and language annotations produces diverse synthetic datasets that improve multi-task policy generalization on three bimanual manipulation tasks.
-
ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation
Compositional Simulation generates scalable real-world robot training data by combining classical simulation with neural simulation in a closed-loop real-sim-real augmentation pipeline.
-
ObjSplat: Geometry-Aware Gaussian Surfels for Active Object Reconstruction
Coupling Gaussian-surfel reconstruction with back-face-aware uncertainty and next-best-path lookahead yields object scans that are more complete and photorealistic while reducing path length about 4–5× versus greedy planners.
-
Robust and Safe Multi-Agent Reinforcement Learning with Communication for Autonomous Vehicles: From Simulation to Hardware
RSR-RSMARL is a robust safe MARL framework with V2V communication and CBF safety shields that supports zero-shot sim-to-real transfer and improves coordination on 1/10-scale vehicle hardware.
-
What Matters in Building Vision-Language-Action Models for Generalist Robots
Systematic tests of VLM backbones, policy architectures, and cross-embodiment data yield RoboVLMs that set new SOTA on robot manipulation benchmarks while requiring few manual designs.
-
Scalable Multi-Task Data Generation via Reinforcement Learning for Language-Conditioned Bimanual Dexterous Manipulation
RL pipeline with generalizable rewards and domain randomization generates datasets that improve generalization of language-conditioned bimanual policies on three manipulation tasks.
-
A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation
Authors perform a cross-simulator, cross-policy empirical study of sim-to-real correlation for VLA policies and distill guidance on using simulation for policy improvement.
-
JoyAI-Sim: A Simulation-Enabled Interconversion Toolchain for the Embodied Data Pyramid
JoyAI-Sim provides bidirectional Robot-Simulation-Human pathways for aligned model evaluation and data generation in robotics using the JoySim simulator as an evaluation layer and physical consistency filter.
-
Redefining End-of-Life: Intelligent Automation for Electronics Remanufacturing Systems
A literature review of intelligent automation approaches using robotics, AI, and control for disassembly, inspection, sorting, and reprocessing of end-of-life electronics.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.