REVIEW 10 cited by
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.
Forward citations
Cited by 10 Pith papers
-
PoseAlign: Sculpting Pose-Consistent Meshes via Text-Guided Deformation
Two-stage text-guided mesh deformation (Laplacian CLIP scaling + attention-shared SDS Jacobian sculpting) better preserves source pose while aligning to text than TextDeformer or MeshUp.
-
PixGS: Pixel-Space Diffusion for Direct 3D Gaussian Splat Generation
PixGS is a single-stage pixel-space diffusion model that directly produces high-quality 3D Gaussian Splats from text or images in ~1s, outperforming multi-stage latent methods on standard benchmarks.
-
TInR: Exploring Tool-Internalized Reasoning in Large Language Models
TInR-U internalizes tool knowledge into LLMs via bidirectional alignment, supervised fine-tuning, and reinforcement learning, outperforming standard tool-integrated reasoning in both in-domain and out-of-domain evaluations.
-
Unifi3D: A Study on 3D Representations for Generation and Reconstruction in a Common Framework
SDF grids reconstruct best, Dual Octrees score best on automatic generation metrics, but users prefer SDF output, and reconstruction plus compression errors make up a large share of generation error.
-
Benefit from Reference: Retrieval-Augmented Cross-modal Point Cloud Completion
A retrieval-augmented cross-modal framework with structural shared encoding and gated reference priors is claimed to achieve state-of-the-art point cloud completion, although the evaluation may be tainted by same-obje...
-
Efficient Part-level 3D Object Generation via Dual Volume Packing
From a single image, a 3D latent diffusion model generates all parts of an object at once by packing the part structure into two non-overlapping volumes.
-
IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations
A diffusion model that decomposes arbitrary numbers of images of the same object, under varying lighting, into albedo, normal, metallic, and roughness maps with multi-view consistency.
-
PRM: Photometric Stereo based Large Reconstruction Model
PRM uses photometric-stereo-style rendered images as both input and supervision, with mesh-based differentiable PBR, to reconstruct 3D meshes with finer local details and more robustness to complex appearances.
-
PhyCAGE: Physically Plausible Compositional 3D Asset Generation from a Single Image
A single-image pipeline that generates physically plausible compositional 3D Gaussian Splatting assets by using a physics simulator as a gradient-driven optimizer.
-
TwoSquared: 4D Generation from 2D Image Pairs
A method that generates a temporally consistent, textured 4D mesh sequence from only two RGB images showing an object's initial and final poses.
Discussion (0). Continue with ORCID to comment.