REVIEW 14 cited by
SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view-agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi-view inconsistency problem. We find that this problem primarily stems from geometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem in the final outputs. Therefore, we improve the consistency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-aware and to produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This "coarse" alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality objects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the-art pipelines, obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem. Our method represents a new state-of-the-art performance with an 85+% consistency rate by human evaluation, while many previous methods are around 30%. Our project page is https://sweetdreamer3d.github.io/
Forward citations
Cited by 14 Pith papers
-
ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation
ELSA3D introduces elastic semantic anchoring via sparse anchor tokens and a scale-aware octree tokenizer to unify 3D generation and captioning at reduced computational cost.
-
PixGS: Pixel-Space Diffusion for Direct 3D Gaussian Splat Generation
PixGS is a single-stage pixel-space diffusion model that directly produces high-quality 3D Gaussian Splats from text or images in ~1s, outperforming multi-stage latent methods on standard benchmarks.
-
CharacterShot: Controllable and Consistent 4D Character Animation
A new pipeline generates pose-controlled, view-consistent 4D character animations from one reference image and a 2D pose sequence, backed by a new 13,115-character dataset and benchmark.
-
Apply Hierarchical-Chain-of-Generation to Complex Attributes Text-to-3D Generation
HCoG uses an LLM to sort object parts from inside out and sequentially optimizes 3D Gaussian splats, improving attribute binding for complex text-to-3D prompts.
-
MeshGen: Generating PBR Textured Mesh with Render-Enhanced Auto-Encoder and Generative Data Augmentation
A single photo is converted into a 3D mesh with PBR textures using a render-enhanced auto-encoder, two data-augmentation schemes, and a multi-view texturing pipeline, with the claimed result being the best quality amo...
-
Direct and Explicit 3D Generation from a Single Image
A modified Stable Diffusion model generates six views of depth, color, and 3D Gaussian features from one image, then lifts them into a textured mesh or splatted scene in 15 to 25 seconds.
-
Few-step Flow for 3D Generation via Marginal-Data Transport Distillation
MDT-dist distills a pretrained 3D flow model into a 1-2 step generator using velocity matching plus velocity distillation, cutting TRELLIS inference from 6.1s to 0.68s while approximately preserving generation quality.
-
XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding
XSpecMesh speeds up auto-regressive mesh generation by about 1.7x using multi-head speculative decoding with cross-attention heads and a probability threshold verification, while keeping output quality close to the ba...
-
ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding
ShapeLLM-Omni unifies text, image, and 3D generation and understanding in one autoregressive LLM using discrete 3D tokens and a new 3D-Alpaca training dataset.
-
UniTEX: Universal High Fidelity Generative Texturing for 3D Shapes
UniTEX generates textures for 3D shapes by predicting continuous volumetric texture functions, bypassing UV maps.
-
MV-Adapter: Multi-view Consistent Image Generation Made Easy
An adapter bolts multi-view generation onto frozen text-to-image diffusion models, producing consistent views at up to 768 resolution on SDXL.
-
DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation
A pipeline that generates editable 3D scenes from natural language by combining LLM-based layout planning, multi-timestep diffusion distillation, and staged camera sampling.
-
DreamComposer++: Empowering Diffusion Models with Multi-View Conditions for 3D Content Generation
A multi-view conditioning framework that improves controllable novel view synthesis and 3D reconstruction by injecting fused 3D latents into frozen image and video diffusion models.
-
Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assets
An open-source-oriented image-to-3D system combines a VAE-DiT geometry generator and a diffusion texture module, claiming state-of-the-art quality over open-source rivals and near-proprietary performance.
Discussion (0). Continue with ORCID to comment.