REVIEW 7 cited by
Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We address the problem of multi-object 3D pose control in image diffusion models. Instead of conditioning on a sequence of text tokens, we propose to use a set of per-object representations, Neural Assets, to control the 3D pose of individual objects in a scene. Neural Assets are obtained by pooling visual representations of objects from a reference image, such as a frame in a video, and are trained to reconstruct the respective objects in a different image, e.g., a later frame in the video. Importantly, we encode object visuals from the reference image while conditioning on object poses from the target frame. This enables learning disentangled appearance and pose features. Combining visual and 3D pose representations in a sequence-of-tokens format allows us to keep the text-to-image architecture of existing models, with Neural Assets in place of text tokens. By fine-tuning a pre-trained text-to-image diffusion model with this information, our approach enables fine-grained 3D pose and placement control of individual objects in a scene. We further demonstrate that Neural Assets can be transferred and recomposed across different scenes. Our model achieves state-of-the-art multi-object editing results on both synthetic 3D scene datasets, as well as two real-world video datasets (Objectron, Waymo Open).
Forward citations
Cited by 7 Pith papers
-
Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation
A single reference image guides a diffusion model to insert coherent objects into camera-plus-lidar driving scenes and to insert mammographic anomalies into new scans.
-
Controllable 3D Placement of Objects with Scene-Aware Diffusion Models
Projecting a color-coded 3D bounding box into a ControlNet conditioning map gives diffusion inpainting models precise control over vehicle orientation and placement in driving scenes.
-
Gaussian Splatting is an Effective Data Generator for 3D Object Detection
Inserting Gaussian-splat reconstructed 3D objects into reconstructed driving scenes is a more effective augmentation for camera-based 3D object detection than diffusion-based image synthesis.
-
MObI: Multimodal Object Inpainting Using Diffusion Models
MObI jointly inpaints camera and lidar views of driving scenes, inserting objects from a single reference image at a user-specified 3D bounding box.
-
Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imagination
DREMA creates an object-centric Gaussian Splatting plus PyBullet world model and generates equivariant-transformed demonstrations, improving imitation learning from a handful of real demonstrations.
-
Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles
A finetuned pose-conditioned diffusion model, Drive-1-to-3, synthesizes photorealistic novel views of real vehicles from a single image on Waymo and other driving datasets, beating prior methods on FID and LPIPS.
-
Causal Information Prioritization for Efficient Reinforcement Learning
CIP combines DirectLiNGAM-style causal masks for state-reward and action-reward links with counterfactual data augmentation and an empowerment objective to improve RL sample efficiency.
Discussion (0). Continue with ORCID to comment.