Real2Code: Reconstruct Articulated Objects via Code Generation
read the original abstract
We present Real2Code, a novel approach to reconstructing articulated objects via code generation. Given visual observations of an object, we first reconstruct its part geometry using an image segmentation model and a shape completion model. We then represent the object parts with oriented bounding boxes, which are input to a fine-tuned large language model (LLM) to predict joint articulation as code. By leveraging pre-trained vision and language models, our approach scales elegantly with the number of articulated parts, and generalizes from synthetic training data to real world objects in unstructured environments. Experimental results demonstrate that Real2Code significantly outperforms previous state-of-the-art in reconstruction accuracy, and is the first approach to extrapolate beyond objects' structural complexity in the training set, and reconstructs objects with up to 10 articulated parts. When incorporated with a stereo reconstruction model, Real2Code also generalizes to real world objects from a handful of multi-view RGB images, without the need for depth or camera information.
This paper has not been read by Pith yet.
Forward citations
Cited by 11 Pith papers
-
3D Generation for Embodied AI and Robotic Simulation: A Survey
3D generation for embodied AI is shifting from visual realism toward interaction readiness, organized into data generation, simulation environments, and sim-to-real bridging roles.
-
Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects
Sketch2Arti is the first category-agnostic sketch-based system that infers movable parts and motion parameters from 2D user sketches on CAD objects and supports sketch-guided internal structure completion for shell models.
-
Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects
Sketch2Arti is the first sketch-based system that automatically finds movable parts in CAD objects and predicts their motion parameters from 2D user sketches, trained without object category labels and supporting inte...
-
ART: Articulated Reconstruction Transformer
ART is a category-agnostic transformer that maps sparse multi-state RGB images to per-part 3D geometry, texture, and articulation parameters via learnable part slots.
-
NeuROK: Generative 4D Neural Object Kinematics
NeuROK learns a data-driven latent kinematic parameterization on a large 4D dataset to generate realistic object deformations by simulating dynamics only in low-dimensional latent space via Lagrangian mechanics.
-
PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World
PhysForge generates physics-grounded 3D assets via a VLM-planned Hierarchical Physical Blueprint and a KineVoxel Injection diffusion model, backed by the new PhysDB dataset of 150,000 annotated assets.
-
DailyArt: Discovering Articulation from Single Static Images via Latent Dynamics
DailyArt recovers full joint parameters of articulated objects from a single static image by synthesizing an opened state and comparing discrepancies, supporting downstream part-level novel state synthesis.
-
ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos
ArtiTwinSplat creates interactable digital twins of articulated objects from RGB-D videos via Gaussian Splatting and automatic part and joint discovery.
-
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
A structured survey of multimodal code intelligence that formulates the field by code roles and organizes work into four domains while proposing verification-centered research directions.
-
3D Generation for Embodied AI and Robotic Simulation: A Survey
The survey organizes 3D generation for embodied AI into data generators for assets, simulation environments for interaction, and sim-to-real bridges, noting a shift toward interaction readiness and listing bottlenecks...
-
3D Generation for Embodied AI and Robotic Simulation: A Survey
The paper surveys 3D generation techniques for embodied AI and robotics, categorizing them into data generation, simulation environments, and sim-to-real bridging while identifying bottlenecks in physical validity and...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.