Diffusion models as plug-and-play priors
Reviewed by Pithpith:F2HCGQSJopen to challenge →
read the original abstract
We consider the problem of inferring high-dimensional data $\mathbf{x}$ in a model that consists of a prior $p(\mathbf{x})$ and an auxiliary differentiable constraint $c(\mathbf{x},\mathbf{y})$ on $x$ given some additional information $\mathbf{y}$. In this paper, the prior is an independently trained denoising diffusion generative model. The auxiliary constraint is expected to have a differentiable form, but can come from diverse sources. The possibility of such inference turns diffusion models into plug-and-play modules, thereby allowing a range of potential applications in adapting models to new domains and tasks, such as conditional generation or image segmentation. The structure of diffusion models allows us to perform approximate inference by iterating differentiation through the fixed denoising network enriched with different amounts of noise at each step. Considering many noised versions of $\mathbf{x}$ in evaluation of its fitness is a novel search mechanism that may lead to new algorithms for solving combinatorial optimization problems.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Visual Diffusion Models are Geometric Solvers
Standard visual diffusion models operating in pixel space can approximate solutions to the inscribed square, Steiner tree, and simple polygon problems.
-
DreamFusion: Text-to-3D using 2D Diffusion
Optimizes a Neural Radiance Field via probability density distillation from a 2D diffusion model to produce text-conditioned 3D scenes viewable from any angle.
-
Predicting 3D structure by latent posterior sampling
A two-stage latent-variable model uses diffusion-based score matching to sample 3D scenes from posteriors conditioned on varied observations via volumetric rendering likelihoods.
-
Predicting 3D structure by latent posterior sampling
A latent-variable approach uses diffusion models on NeRF-encoded scene representations to perform posterior sampling for 3D reconstruction from single-view, multi-view, noisy, sparse-pixel, or sparse-depth inputs.
-
Predicting 3D structure by latent posterior sampling
A two-stage method trains NeRF latents then a diffusion prior to sample posteriors for 3D reconstruction from varied observations including single-view, multi-view, noisy, sparse pixels, and sparse depth.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.