Pith. sign in

REVIEW 25 cited by

Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.05996 v1 pith:2RAJHYNJ submitted 2024-07-08 cs.RO

Moritz Reuss , Ömer Erdinç Yağmurlu , Fabian Wenzel , Rudolf Lioutikov This is my paper
classification cs.RO
keywords multimodallanguagegoallearningtransformerannotationsbehaviordiffusion
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

This work introduces the Multimodal Diffusion Transformer (MDT), a novel diffusion policy framework, that excels at learning versatile behavior from multimodal goal specifications with few language annotations. MDT leverages a diffusion-based multimodal transformer backbone and two self-supervised auxiliary objectives to master long-horizon manipulation tasks based on multimodal goals. The vast majority of imitation learning methods only learn from individual goal modalities, e.g. either language or goal images. However, existing large-scale imitation learning datasets are only partially labeled with language annotations, which prohibits current methods from learning language conditioned behavior from these datasets. MDT addresses this challenge by introducing a latent goal-conditioned state representation that is simultaneously trained on multimodal goal instructions. This state representation aligns image and language based goal embeddings and encodes sufficient information to predict future states. The representation is trained via two self-supervised auxiliary objectives, enhancing the performance of the presented transformer backbone. MDT shows exceptional performance on 164 tasks provided by the challenging CALVIN and LIBERO benchmarks, including a LIBERO version that contains less than $2\%$ language annotations. Furthermore, MDT establishes a new record on the CALVIN manipulation challenge, demonstrating an absolute performance improvement of $15\%$ over prior state-of-the-art methods that require large-scale pretraining and contain $10\times$ more learnable parameters. MDT shows its ability to solve long-horizon manipulation from sparsely annotated data in both simulated and real-world environments. Demonstrations and Code are available at https://intuitive-robots.github.io/mdt_policy/.

Discussion (0). Sign in to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

    cs.RO 2026-05 unverdicted novelty 7.0 of 10

    AwareVLN introduces a structural reasoning module and automatic data engine with progress division to equip VLN agents with self-awareness of agent state and task progress, outperforming prior methods on Habitat datasets.

  2. CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policies

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    CF-VLA uses a coarse initialization over endpoint velocity followed by single-step refinement to achieve strong performance with low inference steps on CALVIN, LIBERO, and real-robot tasks.

  3. Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills

    cs.RO 2026-08 conditional novelty 6.0 of 10

    A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.

  4. Inference-Time Robot Behavior Steering through Physically-Aware Reconfiguration of Task-Structure

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    ReStruct steers robot policies at inference time by reconfiguring task structure with neural automata and synchronous products, claiming up to 25% gains over VLA models in success and preference adherence.

  5. UniviewVLA: A Unified Multiview Vision-Language-Action Model with World Modeling

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    UniviewVLA generates multiview future views from two cameras via world modeling, plus token compression and view selection, to boost occlusion handling in robot manipulation while matching standard benchmark performance.

  6. What Are We Actually Benchmarking in Robot Manipulation?

    cs.RO 2026-06 conditional novelty 6.0 of 10

    LIBERO and CALVIN fail multiple proposed diagnostics for shortcut solvability, statistical significance, overfitting, and data dependence, while a tiny 0.09B probe reaches near-SOTA on LIBERO.

  7. A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Model

    cs.RO 2026-04 unverdicted novelty 6.0 of 10

    A1 is a transparent VLA framework achieving state-of-the-art robot manipulation success with up to 72% lower latency via adaptive layer truncation and inter-layer flow matching.

  8. Emergent Neural Automaton Policies: Learning Symbolic Structure from Visuomotor Trajectories

    cs.RO 2026-03 unverdicted novelty 6.0 of 10

    ENAP extracts an emergent Mealy automaton from visuomotor trajectories to act as a high-level planner for a low-level residual policy, yielding up to 27% higher success than end-to-end VLA policies in low-data regimes.

  9. VolumeDP: Modeling Volumetric Representation for Manipulation Policy Learning

    cs.RO 2026-03 conditional novelty 6.0 of 10

    An RGB-only diffusion policy that builds an explicit volumetric representation, distills it into spatial tokens, and conditions a multi-token decoder reaches 88.8% average success on LIBERO.

  10. VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A transformer-based compressor that turns old observations into fixed-size memory tokens improves non-Markovian imitation-learning robot policies, with large gains on memory-intensive simulated tasks.

  11. Robot-DIFT: Correspondence-Sensitive Diffusion Features for Contact-Rich Robot Manipulation

    cs.RO 2026-02 conditional novelty 6.0 of 10

    Distilling Stable Diffusion's decoder features into a deterministic student backbone with a multi-scale fusion network improves contact-rich manipulation success in simulation and on a real robot.

  12. Think Proprioceptively: State-Grounded Visual Token Selection for VLA Policies

    cs.RO 2026-02 conditional novelty 6.0 of 10

    Using tokenized proprioception plus instruction to select ~15% of visual patches matches or beats full-token VLA baselines and cuts latency by ~58%.

  13. PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation

    cs.RO 2026-01 unverdicted novelty 6.0 of 10

    PALM improves long-horizon robotic manipulation success by distilling affordance representations for object interaction and predicting within-subtask progress in a VLA model.

  14. MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation

    cs.RO 2025-08 conditional novelty 6.0 of 10

    MemoryVLA introduces a perceptual-cognitive memory bank and working-memory retrieval mechanism into VLA models, raising success rates on long-horizon robotic tasks by up to 26 points over prior baselines.

  15. Interactive Post-Training for Vision-Language-Action Models

    cs.LG 2025-05 unverdicted novelty 6.0 of 10

    RIPT-VLA applies RL with dynamic rollout sampling and leave-one-out advantage estimation to fine-tune VLA models, achieving up to 97.5% success rates and recovering from 4% to 97% success with one demonstration in 15 ...

  16. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success

    cs.RO 2025-02 accept novelty 6.0 of 10

    OpenVLA-OFT fine-tuning boosts LIBERO success rate from 76.5% to 97.1%, speeds action generation 26x, and outperforms baselines on real bimanual dexterous tasks.

  17. Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations

    cs.CV 2024-12 unverdicted novelty 6.0 of 10

    Video Prediction Policy conditions robot action learning on future-frame predictions inside fine-tuned video diffusion models, yielding 18.6% relative gains on Calvin ABC-D and 31.6% higher real-world success rates.

  18. A Survey on Vision-Language-Action Models for Embodied AI

    cs.RO 2024-05 unverdicted novelty 6.0 of 10

    This is the first survey on vision-language-action models, providing a taxonomy across three lines, plus summaries of datasets, simulators, benchmarks, challenges, and future directions in embodied AI.

  19. TS-Mask VLA: 2D Temporal-Spatial Masking for Vision-Language-Action Model with Effective Bridging

    cs.RO 2026-07 conditional novelty 5.0 of 10

    A 0.5B VLA with bridge-conditioned discrete diffusion and 2D temporal–spatial action masking reaches 95.7% LIBERO success and 4.19 CALVIN average length.

  20. AdaptiveLoad: Towards Efficient Video Diffusion Transformer Training

    cs.DC 2026-05 unverdicted novelty 5.0 of 10

    AdaptiveLoad cuts computational imbalance in video DiT training from 39% to 18.9% and raises throughput 27.2% via memory-compute constraints and a custom LayerNorm-Modulate kernel.

  21. RynnVLA-002: A Unified Vision-Language-Action and World Model

    cs.RO 2025-11 conditional novelty 5.0 of 10

    A single model that jointly predicts robot actions and future images outperforms separate action-only and video-only models on LIBERO and real SO100 manipulation tasks.

  22. Reflection-Based Task Adaptation for Self-Improving VLA

    cs.RO 2025-10 unverdicted novelty 5.0 of 10

    Reflective Self-Adaptation combines failure-reflective reinforcement learning with success-guided imitation learning to enable faster and more reliable task adaptation for pre-trained Vision-Language-Action models.

  23. Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models

    cs.RO 2025-09 conditional novelty 5.0 of 10

    ARFM adaptively adjusts a scaling factor in the flow-matching loss so that offline RL advantage signals are preserved while gradient variance is controlled, improving VLA robot policy fine-tuning.

  24. GR-3 Technical Report

    cs.RO 2025-07 unverdicted novelty 5.0 of 10

    GR-3 is a VLA model that generalizes to novel objects, environments, and abstract instructions, outperforms the π0 baseline, and integrates with the new ByteMini bi-manual mobile robot.

  25. Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges

    cs.RO 2025-08 conditional novelty 4.0 of 10

    A survey that taxonomizes robotic manipulation policies trained by imitation learning, traces their evolution, and compiles benchmark comparisons.

Pith tools