REVIEW 14 cited by
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.
Forward citations
Cited by 14 Pith papers
-
PoseAlign: Sculpting Pose-Consistent Meshes via Text-Guided Deformation
Two-stage text-guided mesh deformation (Laplacian CLIP scaling + attention-shared SDS Jacobian sculpting) better preserves source pose while aligning to text than TextDeformer or MeshUp.
-
PixGS: Pixel-Space Diffusion for Direct 3D Gaussian Splat Generation
PixGS is a single-stage pixel-space diffusion model that directly produces high-quality 3D Gaussian Splats from text or images in ~1s, outperforming multi-stage latent methods on standard benchmarks.
-
PixGS: Pixel-Space Diffusion for Direct 3D Gaussian Splat Generation
A single-stage pixel-space diffusion model for direct 3D Gaussian Splat generation that bypasses latent compression and adds geometric supervisions to outperform prior multi-stage methods.
-
Toward Visually Realistic Simulation: A Benchmark for Evaluating Robot Manipulation in Simulation
VISER is a new visually realistic simulation benchmark for robot manipulation tasks that uses PBR materials and MLLM-assisted asset generation, achieving 0.92 Pearson correlation with real-world policy performance.
-
ReplicateAnyScene: Zero-Shot Video-to-3D Composition via Textual-Visual-Spatial Alignment
ReplicateAnyScene performs fully automated zero-shot video-to-compositional-3D reconstruction by cascading alignments of generic priors from vision foundation models across textual, visual, and spatial dimensions.
-
TInR: Exploring Tool-Internalized Reasoning in Large Language Models
TInR-U internalizes tool knowledge into LLMs via bidirectional alignment, supervised fine-tuning, and reinforcement learning, outperforming standard tool-integrated reasoning in both in-domain and out-of-domain evaluations.
-
Unifi3D: A Study on 3D Representations for Generation and Reconstruction in a Common Framework
SDF grids reconstruct best, Dual Octrees score best on automatic generation metrics, but users prefer SDF output, and reconstruction plus compression errors make up a large share of generation error.
-
Benefit from Reference: Retrieval-Augmented Cross-modal Point Cloud Completion
A retrieval-augmented cross-modal framework with structural shared encoding and gated reference priors is claimed to achieve state-of-the-art point cloud completion, although the evaluation may be tainted by same-obje...
-
PoseAlign: Sculpting Pose-Consistent Meshes via Text-Guided Deformation
PoseAlign splits text-guided mesh deformation into Laplacian-based global pose scaling and attention-sharing SDS local sculpting to keep pose while matching text.
-
ReScene: Structured Indoor Scene Reconstruction from Multi-View Captures
ReScene introduces HierView for view prioritization and Relation-Aware Assembly for scene graph fusion, reporting 17% lower Chamfer Distance and 26% lower LPIPS than prior baselines on ScanNet while running faster.
-
ISAP-3D: Identity-Slot Aligned Part-Aware 3D Generation
ISAP-3D proposes identity-slot aligned modeling with semantic identity tokens and one-to-one layout prediction to achieve stable part-aware 3D generation.
-
CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models
CG-MLLM is a multimodal LLM using a Mixture-of-Transformer architecture with separate TokenAR and BlockAR components integrated with a pre-trained vision-language backbone and 3D VAE to enable 3D captioning and high-f...
-
Dual-Stream EEG Decoding for 3D Visual Perception
Dual-stream EEG decoder separates identity and orientation to support 3D reconstruction from neural signals via circular regression and conditioned diffusion.
-
TInR: Exploring Tool-Internalized Reasoning in Large Language Models
TInR-U internalizes tool knowledge into an LLM via bidirectional alignment, SFT warm-up, and RL, claiming better in- and out-of-domain tool reasoning without external docs.
Discussion (0). Sign in to comment.