Pith. sign in

REVIEW 10 cited by

Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixels

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.22634 v2 pith:DY4YTZY6 submitted 2025-03-28 cs.RO cs.AI

Adam Wei , Abhinav Agarwal , Boyuan Chen , Rohan Bosworth , Nicholas Pfaff , Russ Tedrake This is my paper
classification cs.ROcs.AI
keywords cotrainingdatapoliciessim-and-realsimulatedperformancerealevaluated
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Cotraining with demonstration data generated both in simulation and on real hardware has emerged as a promising recipe for scaling imitation learning in robotics. This work seeks to elucidate basic principles of this sim-and-real cotraining to inform simulation design, sim-and-real dataset creation, and policy training. Our experiments confirm that cotraining with simulated data can dramatically improve performance, especially when real data is limited. We show that these performance gains scale with additional simulated data up to a plateau; adding more real-world data increases this performance ceiling. The results also suggest that reducing physical domain gaps may be more impactful than visual fidelity for non-prehensile or contact-rich tasks. Perhaps surprisingly, we find that some visual gap can help cotraining -- binary probes reveal that high-performing policies must learn to distinguish simulated domains from real. We conclude by investigating this nuance and mechanisms that facilitate positive transfer between sim-and-real. Focusing narrowly on the canonical task of planar pushing from pixels allows us to be thorough in our study. In total, our experiments span 50+ real-world policies (evaluated on 1000+ trials) and 250 simulated policies (evaluated on 50,000+ trials). Videos and code can be found at https://sim-and-real-cotraining.github.io/.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Robotics

    cs.RO 2026-06 unverdicted novelty 7.0 of 10

    Ambient Diffusion Policy enables better imitation learning from suboptimal robot data by leveraging spectral properties to restrict data usage to specific diffusion times.

  2. SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

    cs.RO 2026-06 conditional novelty 6.0 of 10

    An automated real-to-sim pipeline builds digital twins and affordance-preserving cousins from video, yielding sim evaluations that correlate with real robot policy success and zero-shot sim-to-real gains.

  3. SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    SimFoundry automates zero-shot real-to-sim scene generation from video, producing digital twins and cousins that enable policy training with 0.911 mean Pearson correlation to real-world results and 17-40% success gain...

  4. Training and Evaluating Diffusion Policies with Long Context Lengths

    cs.RO 2026-06 conditional novelty 6.0 of 10

    Naive long-context Diffusion Policies succeed with UNet+Cross-Attention and sufficient data; variable-history training cuts sample complexity in the low-data regime.

  5. A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies

    cs.RO 2026-04 unverdicted novelty 6.0 of 10

    Sim-and-real co-training for robot policies is driven primarily by balanced cross-domain representation alignment and secondarily by domain-dependent action reweighting.

  6. Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models

    cs.RO 2026-02 conditional novelty 6.0 of 10

    Adding a real-world supervised loss to simulation reinforcement learning improves real-robot success and data efficiency for VLA co-training.

  7. BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfer

    cs.RO 2026-07 unverdicted novelty 5.0 of 10

    BIFROST learns invariant latent states via cross-domain bisimulation on paired data to enable zero-shot sim2real policy transfer for visual navigation, contact-rich manipulation, and visual servoing.

  8. HumanoidMimicGen: Data Generation for Loco-Manipulation via Whole-Body Planning

    cs.RO 2026-05 unverdicted novelty 5.0 of 10

    HumanoidMimicGen automatically generates large loco-manipulation datasets from few source demonstrations using whole-body planning, enabling visuomotor policies that outperform real-data-only training by 20% on a new ...

  9. HyperSim: A Holistic Sim-To-Real Framework For Robust Robotic Manipulation

    cs.RO 2026-05 unverdicted novelty 5.0 of 10

    HyperSim reports 80% and 95% sim-to-real success on two manipulation policies across 400 real executions by combining synthetic environment synthesis, adversarial trajectories, and co-training.

  10. A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation

    cs.RO 2025-07 accept novelty 5.0 of 10

    Multi-task pretraining of diffusion policies on diverse robot data produces more successful, robust, and data-efficient policies for dexterous manipulation than single-task baselines, with performance scaling with pre...

Pith tools