Pith. sign in

REVIEW 2 major objections 2 minor 104 cited by

3D-VLA: A 3D Vision-Language-Action Generative World Model

T0 review · 2 major / 2 minor · reviewed 2026-05-13 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read 3D-VLA connects 3D perception to robot actions by embedding a generative world model inside a language model.

desk verdict 3D-VLA layers a 3D LLM with interaction tokens and aligned diffusion models to make a generative world model for embodied tasks, but the gains are only asserted on held-in data with no numbers or baselines shown. read the letter →

arxiv 2403.09631 v1 pith:FK7GK5EV submitted 2024-03-14 cs.CV cs.AIcs.CLcs.RO

Haoyu Zhen , Xiaowen Qiu , Peihao Chen , Jincheng Yang , Xin Yan , Yilun Du , Yining Hong , Chuang Gan This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.RO
keywords 3D-VLAvision-language-actionmodelgenerativeworldembodieddiffusion3Dpointcloudsroboticsinstructiondatasetplanning
checked against bilinear_family_forcedCost.FunctionalEquationphi_equation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Current vision-language-action models operate on 2D images and map perception straight to actions without modeling world dynamics. The paper introduces 3D-VLA to address this gap by building a generative world model on a 3D large language model. Interaction tokens let the model engage with the environment while aligned diffusion networks generate future goal images and point clouds. A large training set is assembled by pulling 3D information from existing robotics datasets. The result is an embodied model that reasons about possible futures before selecting actions.

What carries the argument

A 3D large language model augmented with interaction tokens and aligned embodied diffusion models that generate future goal images and point clouds.

What would settle it

Testing the trained model on a held-out robotics task or physical robot never seen during dataset curation and measuring whether planning success rates exceed those of standard 2D vision-language-action baselines.

Watch

Extended reading notes

Core claim

3D-VLA is built on a 3D-based large language model with interaction tokens to engage the environment, and embodied diffusion models aligned to it for predicting goal images and point clouds. This creates a generative world model that links 3D perception, reasoning, and action, trained on a curated 3D embodied instruction dataset from existing robotics data. Experiments show significant improvements in reasoning, multimodal generation, and planning capabilities in embodied environments.

Load-bearing premise

3D information extracted from existing robotics datasets is diverse enough to train a model that generalizes to new environments.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces 3D-VLA, a generative world model for embodied AI that integrates 3D perception, reasoning, and action via a 3D-based LLM augmented with interaction tokens and aligned diffusion models for goal image and point-cloud prediction. A large-scale 3D embodied instruction dataset is curated by extracting 3D information from existing robotics corpora, and experiments on held-in splits are reported to show gains in reasoning, multimodal generation, and planning.

Significance. If the empirical claims are substantiated with quantitative metrics and generalization tests, the work could meaningfully advance embodied foundation models by shifting from direct perception-to-action mappings toward explicit generative world models that support planning via imagined 3D futures. The dataset curation effort is a constructive contribution to the community.

major comments (2)
  1. [§4] §4 (Experiments): The central claim of 'significant improvements' in reasoning, generation, and planning is supported only by held-in dataset results; no quantitative metrics, baselines, ablation studies, or error analysis are supplied, leaving the magnitude and sources of any gains impossible to assess.
  2. [§4.3] §4.3 (Evaluation): No out-of-distribution, held-out, or cross-robotology tests are described. Because the dataset is extracted from the same robotics sources used for training, observed gains may reflect interpolation within the training support rather than the claimed advantages of the 3D world model for real-world planning under distributional shift.
minor comments (2)
  1. [Abstract] Abstract: The phrase 'significantly improves' is used without any numerical results or baseline comparisons.
  2. [§3.2] §3.2: The mechanism by which interaction tokens interface with the embodied environment would benefit from a concrete example or pseudocode.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We agree that the experimental evaluation requires more rigorous quantitative support and generalization analysis to substantiate the claims. We have revised the manuscript to address these points and provide point-by-point responses below.

read point-by-point responses

  1. Referee: [§4] §4 (Experiments): The central claim of 'significant improvements' in reasoning, generation, and planning is supported only by held-in dataset results; no quantitative metrics, baselines, ablation studies, or error analysis are supplied, leaving the magnitude and sources of any gains impossible to assess.

    Authors: We acknowledge that the original submission relied primarily on held-in results and qualitative examples. In the revised manuscript, §4 has been expanded with quantitative metrics (task success rates for planning, accuracy for reasoning, and perceptual quality scores for generation), direct comparisons to baselines including 2D VLA models and non-generative variants, ablation studies on the 3D LLM backbone, interaction tokens, and diffusion alignment modules, and an error analysis subsection that categorizes failure modes and links them to specific model components. revision: yes

  2. Referee: [§4.3] §4.3 (Evaluation): No out-of-distribution, held-out, or cross-robotology tests are described. Because the dataset is extracted from the same robotics sources used for training, observed gains may reflect interpolation within the training support rather than the claimed advantages of the 3D world model for real-world planning under distributional shift.

    Authors: We agree that held-in results alone cannot fully rule out interpolation effects. The revised evaluation now includes a held-out split consisting of novel instruction-object combinations excluded from training but drawn from the same source corpora; 3D-VLA shows consistent gains over baselines on this split, supporting the value of the generative 3D world model. Full cross-robotology testing (different hardware platforms) is not feasible within the current revision due to the absence of aligned multi-robot 3D data and would require new collection efforts; we explicitly discuss this limitation and outline it as future work. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical training on external data with no self-referential derivation

full rationale

The paper describes an empirical pipeline: curating a 3D embodied instruction dataset by extracting information from existing robotics corpora, training a 3D-based LLM augmented with interaction tokens, training and aligning embodied diffusion models for goal image/point-cloud prediction, and reporting performance on held-in dataset splits. No equations, uniqueness theorems, or ansatzes are presented that reduce a claimed prediction or result to a quantity defined inside the paper itself. The central claims rest on observed improvements in reasoning/generation/planning metrics rather than any fitted parameter being renamed as a prediction or any self-citation chain substituting for independent justification. This is a standard empirical ML construction whose validity is assessed by external benchmarks, not by internal definitional closure.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the assumption that 3D LLMs can be extended with generative capabilities via diffusion alignment and that curated robotics data suffices for training. No explicit free parameters or invented physical entities are named in the abstract.

assumptions (1)
  • domain assumption A 3D-based LLM can be extended with generation abilities by aligning embodied diffusion models for goal image and point cloud prediction.
    This is the core construction step stated in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3D-VLA: A 3D Vision-Language-Action Generative World Model." pith.science (2026). https://pith.science/paper/FK7GK5EV

@misc{pith2026240309631,
  author       = {Pith},
  title        = {Pith review of: 3D-VLA: A 3D Vision-Language-Action Generative World Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FK7GK5EV}},
  note         = {Machine review of arXiv:2403.09631}
}
read the original abstract

Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from perception to action, neglecting the vast dynamics of the world and the relations between actions and dynamics. In contrast, human beings are endowed with world models that depict imagination about future scenarios to plan actions accordingly. To this end, we propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action through a generative world model. Specifically, 3D-VLA is built on top of a 3D-based large language model (LLM), and a set of interaction tokens is introduced to engage with the embodied environment. Furthermore, to inject generation abilities into the model, we train a series of embodied diffusion models and align them into the LLM for predicting the goal images and point clouds. To train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by extracting vast 3D-related information from existing robotics datasets. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities in embodied environments, showcasing its potential in real-world applications.

Discussion (0). Continue with ORCID to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith.Foundation.DAlembert.Inevitability bilinear_family_forced unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    we propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action through a generative world model. Specifically, 3D-VLA is built on top of a 3D-based large language model (LLM), and a set of interaction tokens is introduced to engage with the embodied environment.

  • IndisputableMonolith.Cost.FunctionalEquation washburn_uniqueness_aczel unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    To train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by extracting vast 3D-related information from existing robotics datasets. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities

  • IndisputableMonolith.Foundation.PhiForcing phi_equation unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    we train a series of embodied diffusion models and align them into the LLM for predicting the goal images and point clouds

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Showing 60 of 104 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 104 Pith citations

  1. Flex-$\pi$: A Multi-Stream World-Action Model with Compute Flexibility

    cs.RO 2026-08 conditional novelty 7.0 of 10

    Flex-π is a 6B world-action model that jointly predicts RGB, 3D pointmaps, and DINO semantic futures, allowing a single checkpoint to switch between fast action-only and full joint generation.

  2. Present but Not Remembered: Auditing How Frozen VLAs Encode, Deploy, and Steer Visual History

    cs.CV 2026-07 conditional novelty 7.0 of 10

    VLAs linearly encode past frames as near-redundant copies of the present and deploy them only as architecture-conditional fallback or standing use, so steerability tracks deployment regime not encoding.

  3. MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulation

    cs.RO 2026-06 unverdicted novelty 7.0 of 10

    MuseVLA adds on-demand sensor selection via tokens and converts readings into grounded sensor images for multimodal fusion, reporting 80.6% average success on real-robot dexterous tasks that need non-visual sensing.

  4. GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

    cs.RO 2026-05 unverdicted novelty 7.0 of 10

    GaussianDream is a feed-forward 3D Gaussian world model that provides structured spatial-temporal supervision for VLA-based robotic manipulation by predicting future Gaussian states during training.

  5. One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Reducing visual input to one token per frame via adaptive attention pooling and a unified flow-matching objective improves long-horizon performance in VLA policies on MetaWorld, LIBERO, and real-robot tasks.

  6. UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

    cs.RO 2026-02 unverdicted novelty 7.0 of 10

    UniLACT improves VLA models by adding depth-aware unified latent action pretraining that outperforms RGB-only baselines on seen and unseen manipulation tasks.

  7. BridgeEQA: Virtual Embodied Agents for Real Bridge Inspections

    cs.CV 2025-11 conditional novelty 7.0 of 10

    BridgeEQA creates a new benchmark and EMVR method for embodied agents to perform question answering on real-world bridge inspections using egocentric images and professional reports.

  8. Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots

    cs.RO 2025-09 conditional novelty 7.0 of 10

    A learned plug-in that denoises consumer depth cameras to simulation-like metric depth enables zero-shot sim-to-real transfer of depth-only manipulation policies trained on raw simulated depth.

  9. Beyond Syntax: Action Semantics Learning for App Agents

    cs.AI 2025-06 unverdicted novelty 7.0 of 10

    Action Semantics Learning trains app agents to align with the semantic effects of actions via a Semantic Estimator module, improving robustness to out-of-distribution scenarios over syntax-matching fine-tuning.

  10. VLAs are Confined yet Capable of Generalizing to Novel Instructions

    cs.RO 2025-05 unverdicted novelty 7.0 of 10

    Averaging and temporally interpolating text latents in VLAs enables 83% success on novel task combinations in the libero-ood benchmark where SOTA models achieve under 15%.

  11. What if? Emulative Simulation with World Models for Situated Reasoning

    cs.CV 2026-03 conditional novelty 6.5 of 10

    WanderDream supplies 15.8K panoramic mental-exploration videos and 158K QA pairs showing that world-model imagination measurably improves situated spatial reasoning without active exploration.

  12. RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Depth refinement for monocular 3D detection is reformulated as a visual alignment judgment made by a VLM emitting discrete direction and magnitude tokens, yielding consistent post-hoc gains without retraining the base...

  13. LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratory

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A chemical-lab failure sim, 20K-trajectory dataset, six-axis benchmark, and specialized VLM raise failure detection to 90.8% on seen scenes and lift downstream policy success by 4–16 points.

  14. Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    From one real demonstration, WANDA synthesizes diverse mobile-manipulation trajectories, reaching 54.8% average real-world task progress and zero-shot deployment on a morphologically different robot.

  15. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

    cs.RO 2026-07 unverdicted novelty 6.0 of 10

    A memory-guided LLM planner composes a frozen VLA as a contact-rich primitive with fixed analytic controllers, lifting perturbed manipulation success without VLA finetuning.

  16. SSI-Policy: Learning Structured Scene Interfaces for Vision-Language Robotic Manipulation

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    SSI-Policy uses an RGB-only Structured Scene Interface to improve LIBERO benchmark performance by nearly 15% with only 10 demonstrations per task compared to prior methods.

  17. CoStream: Composing Simple Behaviors for Generalizable Complex Manipulation

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    CoStream composes semantic, predictive, and reactive behaviors on an SE(3) interface to enable precise, generalizable performance on eight real-world contact-rich manipulation tasks.

  18. G$^3$VLA: Geometric inductive bias for Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    G³VLA injects calibrated camera geometry into VLA visual tokens via intrinsic-conditioned ray embeddings, PRoPE, and bidirectional cross-view fusion, producing consistent gains on LIBERO, RoboCasa24, RoboTwin2.0, and ...

  19. Inductive Generalization for Robotic Manipulation

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    The paper introduces an inductive generalization evaluation protocol for manipulation policies and shows that SOTA vision-language-action models fail on progressively harder task variants.

  20. VeriSpace: Spatially Grounded Action Verification for Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    VeriSpace is a 3D-aware action verifier that improves test-time action selection in VLA models by encoding scenes with visual and geometric information and reasoning over spatial relations and goal progress.

  21. LARA: Latent Action Representation Alignment for Vision-Language-Action Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    LARA jointly optimizes LAM and VLA models via representation alignment to improve robotic manipulation performance using human videos.

  22. AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    AffordanceVLA proposes a VLA model with affordance-aware modules (Which2Act, Where2Act, How2Act) in a Mixture-of-Transformer trained in three stages to improve robotic manipulation.

  23. 3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    A 3D-thinking-guided co-training method disentangles geometry perception and spatial reasoning to inject latent 3D priors into VLA models via adapters, achieving SOTA on manipulation benchmarks while running on 2D ima...

  24. PointAction: 3D Points as Universal Action Representations for Robot Control

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    PointAction uses predicted dynamic 3D pointmaps from fine-tuned video models as an embodiment-agnostic action representation to map video predictions to executable robot actions.

  25. Nautilus: From One Prompt to Plug-and-Play Robot Learning

    cs.RO 2026-05 unverdicted novelty 6.0 of 10

    NAUTILUS is a prompt-driven harness that automates plug-and-play adapters, typed contracts, and validation for policies, benchmarks, and robots in learning research.

  26. ThermoAct:Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Making

    cs.RO 2026-03 unverdicted novelty 6.0 of 10

    ThermoAct integrates thermal imaging into VLA models via a VLM planner to enable robots to perceive physical properties like heat and improve safety over vision-only systems.

  27. VLA Knows Its Limits: Adaptive Execution Horizons for Robot Policies

    cs.RO 2026-02 conditional novelty 6.0 of 10

    AutoHorizon estimates per-chunk execution horizons in flow-based VLA policies from action self-attention plateau positions, beating fixed-horizon tuning in most tested benchmarks.

  28. Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks

    cs.RO 2026-02 conditional novelty 6.0 of 10

    A language scratchpad that records object positions, the plan, and completed subgoals lets vision-language-action policies succeed on memory-dependent manipulation tasks that stateless baselines fail.

  29. Learning Native Continuation for Action Chunking Flow Policies

    cs.RO 2026-02 unverdicted novelty 6.0 of 10

    Legato trains flow-based VLA policies with schedule-shaped action-noise mixtures and randomized conditions to achieve smoother trajectories and ~10% faster task completion than real-time chunking across five real-worl...

  30. PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation

    cs.RO 2026-01 unverdicted novelty 6.0 of 10

    PALM improves long-horizon robotic manipulation success by distilling affordance representations for object interaction and predicting within-subtask progress in a VLA model.

  31. GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation

    cs.CV 2025-12 unverdicted novelty 6.0 of 10

    GaussianDWM uses 3D Gaussians with embedded linguistic features, language-guided sampling, and dual-condition generation for unified scene understanding and multi-modal output in driving world models.

  32. StereoVLA: Enhancing Vision-Language-Action Models with Stereo Vision

    cs.RO 2025-12 conditional novelty 6.0 of 10

    A vision-language-action model that fuses stereo-derived geometric features with semantic features improves real-world grasping success and camera-pose robustness over single-view baselines.

  33. VLSA: Vision-Language-Action Models with Plug-and-Play Safety Constraint Layer

    cs.RO 2025-12 conditional novelty 6.0 of 10

    AEGIS wraps VLA robot policies in a CBF-based safety layer that uses VLM obstacle identification, raising collision avoidance from 18.69% to 77.85% and task success by 17.25 points on the new SafeLIBERO benchmark.

  34. LISA-3D: Lifting Language-Image Segmentation to 3D via Multi-View Consistency

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Geometry-aware LoRA tuning of LISA with differentiable reprojection yields view-consistent masks that lift to better 3D reconstructions through frozen SAM-3D.

  35. GeoVLA: Empowering 3D Representations in Vision-Language-Action Models

    cs.RO 2025-08 conditional novelty 6.0 of 10

    A robot policy that combines 2D vision-language features with a point-cloud encoder and a mixture-of-experts diffusion action head reports SOTA manipulation success in simulation and robust real-world behavior under h...

  36. Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    Claimed first large-scale egocentric and multi-view dataset of human-object-human assistance (11.4 hours, 1.2M frames) with three benchmarks; only the abstract was assessable because the submitted body text is a diffe...

  37. H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation

    cs.RO 2025-07 conditional novelty 6.0 of 10

    Pre-training a diffusion-transformer robot policy on 338K human hand-manipulation episodes, then fine-tuning with modular adapters, improves bimanual manipulation success across simulation and real robots.

  38. DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

    cs.CV 2025-07 unverdicted novelty 6.0 of 10

    DreamVLA uses dynamic-region-guided world knowledge prediction, block-wise attention to disentangle information types, and a diffusion transformer for actions, reaching 76.7% success on real robot tasks and 4.44 avera...

  39. Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MTU3D unifies visual grounding and frontier-based exploration in a single transformer, achieving state-of-the-art success rates on HM3D-OVON, GOAT-Bench, SG3D, and A-EQA after large-scale vision-language-exploration p...

  40. GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulation

    cs.RO 2025-06 unverdicted novelty 6.0 of 10

    GAF creates 4D dynamic scene models by adding motion to 3D Gaussians, enabling better reconstruction and 7.3% higher success in robotic tasks.

  41. AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making

    cs.RO 2025-06 conditional novelty 6.0 of 10

    AntiGrounding lifts candidate robot trajectories into the VLM's visual space via multi-view rendering and structured VQA, and reports 57.5% average success across eight manipulation tasks, beating three intermediate-r...

  42. Real-Time Execution of Action Chunking Flow Policies

    cs.RO 2025-06 unverdicted novelty 6.0 of 10

    Real-time chunking (RTC) allows diffusion- and flow-based action chunking policies to execute smoothly and asynchronously, maintaining high success rates on dynamic tasks even with significant inference latency.

  43. DSG-World: Learning a 3D Gaussian World Model from Dual State Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DSG-World builds two segmented 3D Gaussian fields from two scene states and trains them with mutual consistency, enabling novel-state simulation without inpainting or dense capture.

  44. FLARE: Robot Learning with Implicit World Modeling

    cs.RO 2025-05 unverdicted novelty 6.0 of 10

    FLARE integrates predictive latent world modeling into diffusion transformer policies for robots, delivering up to 26% gains on multitask manipulation benchmarks and enabling co-training with action-free human videos.

  45. Incentivizing Multimodal Reasoning in Large Models for Direct Robot Manipulation

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A 7B multimodal model can control a robot by predicting each next gripper pose in natural language, trained from only 65 dialogues plus simulator reinforcement learning, and it claims strong generalization to new sett...

  46. DataMIL: Selecting Data for Robot Imitation Learning with Datamodels

    cs.RO 2025-05 conditional novelty 6.0 of 10

    DataMIL selects robot training demonstrations by estimating each sample's influence on target-task validation loss, and policies trained on the selected data outperform heuristic retrieval baselines on MetaWorld, LIBE...

  47. Training Strategies for Efficient Embodied Reasoning

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Reasoning pre-training and dropout let robot policies benefit from chain-of-thought training without run-time reasoning, preserving most of the performance gains at standard VLA inference speeds.

  48. CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A contrastive pre-training framework and dataset that map tactile 3D point clouds to contact-state language descriptions, enabling zero-shot and LLM-based tactile understanding.

  49. GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data

    cs.RO 2025-05 unverdicted novelty 6.0 of 10

    GraspVLA shows that pretraining a grasping model on a billion synthetic action frames enables zero-shot open-vocabulary performance and sim-to-real transfer.

  50. Robotic Visual Instruction

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Hand-drawn symbolic sketches (arrows and circles) can serve as precise, silent robot instructions, and a vision-language pipeline can execute them on unseen tasks.

  51. TesserAct: Learning 4D Embodied World Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A 4D embodied world model that generates RGB-depth-normal videos from an image and instruction, reconstructs the scene as point clouds, and uses those point clouds to train better robot manipulation policies.

  52. Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A contrastive pre-training method aligns 3D point features with language at the entity level and achieves state-of-the-art open-vocabulary semantic segmentation on ScanNet.

  53. $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    cs.LG 2025-04 unverdicted novelty 6.0 of 10

    π_{0.5} is a VLA model that achieves long-horizon dexterous manipulation in entirely new homes through co-training on heterogeneous tasks and multi-source data including web and semantic predictions.

  54. CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models

    cs.CV 2025-03 unverdicted novelty 6.0 of 10

    CoT-VLA is a 7B VLA that generates future visual frames autoregressively as planning goals before actions, outperforming prior VLAs by 17% on real-world tasks and 6% in simulation.

  55. RoboBERT: An End-to-end Multimodal Robotic Manipulation Model

    cs.RO 2025-02 conditional novelty 6.0 of 10

    A two-stage trained vision-language-action diffusion policy with carefully selected data augmentations reaches mean episode lengths of 4.52 (ABCD to D) and 3.79 (ABC to D) on CALVIN.

  56. DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control

    cs.RO 2025-02 unverdicted novelty 6.0 of 10

    DexVLA combines a scaled diffusion action expert with embodiment curriculum learning to achieve better generalization and performance than prior VLA models on diverse robot hardware and long-horizon tasks.

  57. UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Combining multimodal understanding with future image prediction in one autoregressive model improves vision-language-action policy success rates in simulation and real-world manipulation.

  58. Generative Physical AI in Vision: A Survey

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A structured review that categorizes physics-aware generative models in vision into explicit-simulation and implicit-learning families and proposes six integration paradigms.

  59. Imagine while Reasoning in Space: Multimodal Visualization-of-Thought

    cs.CL 2025-01 unverdicted novelty 6.0 of 10

    MVoT lets multimodal models create coherent images during chain-of-thought reasoning via a token discrepancy loss, yielding competitive or better results than text-only CoT on dynamic spatial tasks.

  60. Bridging Adaptivity and Safety: Learning Agile Collision-Free Locomotion Across Varied Physics

    cs.RO 2025-01 conditional novelty 6.0 of 10

    A legged-robot controller that estimates payload and friction online and uses those estimates to switch between agile and recovery policies achieves lower collision rates and higher speeds than non-adaptive baselines.

See all 104 Pith citations

Reference graph

Works this paper leans on

62 extracted references · 62 canonical work pages · cited by 104 Pith papers (see all)

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 0 23716--23736, 2022

    work page 2022

  2. [2]

    ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

    Bhat, S. F., Birkl, R., Wofk, D., Wonka, P., and M \"u ller, M. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023

    work page Pith review arXiv 2023

  3. [3]

    Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023

    Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023

    work page 2023

  4. [4]

    RT-1: Robotics Transformer for Real-World Control at Scale

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    work page Pith review arXiv 2022

  5. [5]

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    work page Pith review arXiv 2023

  6. [6]

    Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions, 2023

    work page 2023

  7. [7]

    Playfusion: Skill acquisition via diffusion from language-annotated play

    Chen, L., Bahl, S., and Pathak, D. Playfusion: Skill acquisition via diffusion from language-annotated play. In Conference on Robot Learning, pp.\ 2012--2029. PMLR, 2023 a

    work page 2012

  8. [8]

    Ll3da: Visual interactive instruction tuning for omni-3d understanding, reasoning, and planning, 2023 b

    Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., and Chen, T. Ll3da: Visual interactive instruction tuning for omni-3d understanding, reasoning, and planning, 2023 b

    work page 2023

Show all 62 references
  1. [9]

    X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M

    Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. Scannet: Richly-annotated 3d reconstructions of indoor scenes, 2017

    2017

  2. [10]

    M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al

    Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pp.\ 720--736, 2018

    2018

  3. [11]

    Dass, S., Yapeter, J., Zhang, J., Zhang, J., Pertsch, K., Nikolaidis, S., and Lim, J. J. Clvr jaco play dataset, 2023. URL https://github.com/clvrai/clvr_jaco_play_dataset

    2023

  4. [12]

    Objaverse: A universe of annotated 3d objects, 2022

    Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., and Farhadi, A. Objaverse: A universe of annotated 3d objects, 2022

    2022

  5. [13]

    Dreamllm: Synergistic multimodal comprehension and creation

    Dong, R., Han, C., Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., Kong, X., Zhang, X., Ma, K., and Yi, L. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499, 2023

    2023

  6. [14]

    S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al

    Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023 a

    2023 arXiv

  7. [15]

    Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence...

    2023

  8. [16]

    Structure and content-guided video synthesis with diffusion models, 2023

    Esser, P., Chiu, J., Atighehchian, P., Granskog, J., and Germanidis, A. Structure and content-guided video synthesis with diffusion models, 2023

    2023

  9. [17]

    Rh20t: A robotic dataset for learning diverse skills in one-shot

    Fang, H.-S., Fang, H., Tang, Z., Liu, J., Wang, J., Zhu, H., and Lu, C. Rh20t: A robotic dataset for learning diverse skills in one-shot. arXiv preprint arXiv:2307.00595, 2023

    2023

  10. [18]

    Finetuning offline world models in the real world

    Feng, Y., Hansen, N., Xiong, Z., Rajagopalan, C., and Wang, X. Finetuning offline world models in the real world. arXiv preprint arXiv:2310.16029, 2023

    2023

  11. [19]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following, 2023

    Guo, Z., Zhang, R., Zhu, X., Tang, Y., Ma, X., Han, J., Chen, K., Gao, P., Li, X., Li, H., and Heng, P.-A. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following, 2023

    2023

  12. [20]

    3d-llm: Injecting the 3d world into large language models

    Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., and Gan, C. 3d-llm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307.12981, 2023

    2023

  13. [21]

    Multiply: A multisensory object-centric embodied large language model in 3d world

    Hong, Y., Zheng, Z., Chen, P., Wang, Y., Li, J., and Gan, C. Multiply: A multisensory object-centric embodied large language model in 3d world. arXiv preprint arXiv:2401.08577, 2024

    2024

  14. [22]

    and Montani, I

    Honnibal, M. and Montani, I. spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing. To appear, 2017

    2017

  15. [23]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  16. [24]

    Chat-3d v2: Bridging 3d scene and large language models with object identifiers, 2023 a

    Huang, H., Wang, Z., Huang, R., Liu, L., Cheng, X., Zhao, Y., Jin, T., and Zhao, Z. Chat-3d v2: Bridging 3d scene and large language models with object identifiers, 2023 a

    2023

  17. [25]

    An embodied generalist agent in 3d world

    Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S.-C., Jia, B., and Huang, S. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023 b

    2023

  18. [26]

    K., Liu, Q., et al

    Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Liu, Q., et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023 c

    2023

  19. [27]

    R., and Davison, A

    James, S., Ma, Z., Arrojo, D. R., and Davison, A. J. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5 0 (2): 0 3019--3026, 2020

    2020

  20. [28]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., and Finn, C. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp.\ 991--1002. PMLR, 2022

    2022

  21. [29]

    Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  22. [30]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp.\ 12888--12900. PMLR, 2022

    2022

  23. [31]

    Covlm: Composing visual entities and relationships in large language models via communicative decoding

    Li, J., Chen, D., Hong, Y., Chen, Z., Chen, P., Shen, Y., and Gan, C. Covlm: Composing visual entities and relationships in large language models via communicative decoding. arXiv preprint arXiv:2311.03354, 2023 a

    2023

  24. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023 b

    2023 arXiv

  25. [33]

    3dmit: 3d multi-modal instruction tuning for scene understanding, 2024

    Li, Z., Zhang, C., Wang, X., Ren, R., Xu, Y., Ma, R., and Liu, X. 3dmit: 3d multi-modal instruction tuning for scene understanding, 2024

    2024

  26. [34]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  27. [35]

    Hoi4d: A 4d egocentric dataset for category-level human-object interaction

    Liu, Y., Liu, Y., Jiang, C., Lyu, K., Wan, W., Shen, H., Liang, B., Fu, Z., Wang, H., and Yi, L. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 21013--21022, 2022

    2022

  28. [36]

    UNIFIED - IO : A unified model for vision, language, and multi-modal tasks

    Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. UNIFIED - IO : A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=E01k9048soZ

    2023

  29. [37]

    and Sermanet, P

    Lynch, C. and Sermanet, P. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648, 2020

    2005

  30. [38]

    Interactive language: Talking to robots in real time

    Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., and Florence, P. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023

    2023

  31. [39]

    Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity

    Mandlekar, A., Booher, J., Spero, M., Tung, A., Gupta, A., Zhu, Y., Garg, A., Savarese, S., and Fei-Fei, L. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In 2019 IEEE/RSJ International Conferen...

    2019

  32. [40]

    Vision: A Computational Investigation into the Human Representation and Processing of Visual Information

    Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information . The MIT Press, 07 2010. ISBN 9780262514620. doi:10.7551/mitpress/9780262514620.001.0001. URL https://doi.org/10.7551/mitpress/9780262514620.001.0001

    2010 doi

  33. [41]

    Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks

    Mees, O., Hermann, L., Rosete-Beas, E., and Burgard, W. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7 0 (3): 0 7327--7334, 2022

    2022

  34. [42]

    Grounding language with visual affordances over unstructured data

    Mees, O., Borja-Diaz, J., and Burgard, W. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023

    2023

  35. [43]

    Point-e: A system for generating 3d point clouds from complex prompts

    Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., and Chen, M. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022

    2022

  36. [44]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Padalkar, A., Pooley, A., Jain, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Singh, A., Brohan, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023

    2023 arXiv

  37. [45]

    The effects of contextual scenes on the identification of objects

    Palmer, S. The effects of contextual scenes on the identification of objects. Memory & Cognition, 3: 0 519--526, 01 1975

    1975

  38. [46]

    Kosmos-2: Grounding multimodal large language models to the world

    Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  39. [47]

    Seeing and Visualizing: It's Not What You Think

    Pylyshyn, Z. Seeing and Visualizing: It's Not What You Think. 01 2003. ISBN 9780262316316. doi:10.7551/mitpress/6137.001.0001

    2003 doi

  40. [48]

    Gpt4point: A unified framework for point-language understanding and generation, 2023

    Qi, Z., Fang, Y., Sun, Z., Wu, X., Wu, T., Wang, J., Lin, D., and Zhao, H. Gpt4point: A unified framework for point-language understanding and generation, 2023

    2023

  41. [49]

    K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A

    Ramakrishnan, S. K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A. X., Savva, M., Zhao, Y., and Batra, D. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai, 2021

    2021

  42. [50]

    Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., and Zhang, L. Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    2024

  43. [51]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  44. [52]

    Playing with food: Learning food item representations through interactive exploration

    Sawhney, A., Lee, S., Zhang, K., Veloso, M., and Kroemer, O. Playing with food: Learning food item representations through interactive exploration. In Experimental Robotics: The 17th International Symposium, pp.\ 309--322. Springer, 2021

    2021

  45. [53]

    J., Florence, P., Han, W., Baruch, R., Lu, Y., Mirchandani, S., Xu, P., Sanketi, P., Hausman, K., Shafran, I., Ichter, B., and Cao, Y

    Sermanet, P., Ding, T., Zhao, J., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddineni, S., Joshi, N. J., Florence, P., Han, W., Baruch, R., Lu, Y., Mirchandani, S., Xu, P., Sanketi, P., Hausman, K., Shafran, I., Ichter, B., and Cao, Y. Robovqa: Mult...

    2023

  46. [54]

    Shafiullah, N. M. M., Rai, A., Etukuru, H., Liu, Y., Misra, I., Chintala, S., and Pinto, L. On bringing robots home. arXiv preprint arXiv:2311.16098, 2023

    2023

  47. [55]

    MUTEX : Learning unified policies from multimodal task specifications

    Shah, R., Mart \' n-Mart \' n, R., and Zhu, Y. MUTEX : Learning unified policies from multimodal task specifications. In 7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=PwqiqaaEzJ

    2023

  48. [56]

    Lancon-learn: Learning with language to enable generalization in multi-task manipulation

    Silva, A., Moorman, N., Silva, W., Zaidi, Z., Gopalan, N., and Gombolay, M. Lancon-learn: Learning with language to enable generalization in multi-task manipulation. IEEE Robotics and Automation Letters, 7 0 (2): 0 1635--1642, 2021

    2021

  49. [57]

    and Deng, J

    Teed, Z. and Deng, J. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part II 16, pp.\ 402--419. Springer, 2020

    2020

  50. [58]

    R., Black, K., Zhao, T

    Walke, H. R., Black, K., Zhao, T. Z., Vuong, Q., Zheng, C., Hansen-Estruch, P., He, A. W., Myers, V., Kim, M. J., Du, M., et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pp.\ 1723--1736. PMLR, 2023

    2023

  51. [59]

    Next-gpt: Any-to-any multimodal llm

    Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023

    2023

  52. [60]

    Pointllm: Empowering large language models to understand point clouds, 2023

    Xu, R., Wang, X., Wang, T., Chen, Y., Pang, J., and Lin, D. Pointllm: Empowering large language models to understand point clouds, 2023

    2023

  53. [61]

    Uni3d: Exploring unified 3d representation at scale, 2023

    Zhou, J., Wang, J., Ma, B., Liu, Y.-S., Huang, T., and Wang, X. Uni3d: Exploring unified 3d representation at scale, 2023

    2023

  54. [62]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

Pith tools

Reviewed May 13, 2026 · model on record in the stance chip above.