Neural Volumes: Learning Dynamic Renderable Volumes from Images
read the original abstract
Modeling and rendering of dynamic scenes is challenging, as natural scenes often contain complex phenomena such as thin structures, evolving topology, translucency, scattering, occlusion, and biological motion. Mesh-based reconstruction and tracking often fail in these cases, and other approaches (e.g., light field video) typically rely on constrained viewing conditions, which limit interactivity. We circumvent these difficulties by presenting a learning-based approach to representing dynamic objects inspired by the integral projection model used in tomographic imaging. The approach is supervised directly from 2D images in a multi-view capture setting and does not require explicit reconstruction or tracking of the object. Our method has two primary components: an encoder-decoder network that transforms input images into a 3D volume representation, and a differentiable ray-marching operation that enables end-to-end training. By virtue of its 3D representation, our construction extrapolates better to novel viewpoints compared to screen-space rendering techniques. The encoder-decoder architecture learns a latent representation of a dynamic scene that enables us to produce novel content sequences not seen during training. To overcome memory limitations of voxel-based representations, we learn a dynamic irregular grid structure implemented with a warp field during ray-marching. This structure greatly improves the apparent resolution and reduces grid-like artifacts and jagged motion. Finally, we demonstrate how to incorporate surface-based representations into our volumetric-learning framework for applications where the highest resolution is required, using facial performance capture as a case in point.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
ClipGStream: Clip-Stream Gaussian Splatting for Any Length and Any Motion Multi-View Dynamic Scene Reconstruction
ClipGStream enables scalable flicker-free reconstruction of long dynamic multi-view videos by performing stream optimization at the clip level with clip-independent spatio-temporal fields, residual anchor compensation...
-
Learning Efficient 4D Gaussian Representations from Monocular Videos with Flow Splatting
Flow Splatting extends 4D Gaussian volumes with time-varying means and covariances, approximates a velocity field, and splats it to render optical flow for supervising dynamic reconstruction from monocular video.
-
Wispy to Voluminous: Prior-free Multi-view Capture of Strand-level Facial Hair
Four-stage pipeline reconstructs explicit facial hair strands from multi-view images via 3D Gaussian optimization and curve tracing, claimed as first such method.
-
VEPHand: View-Efficient Photometric Hand Performance Capture at Scale
End-to-end neural pipeline extracts hand geometry from unmasked limited-view images and registers it to a personalized tetrahedral model via volumetric offsets, achieving SOTA on over 12,000 sequences.
-
Long-LRM++: Preserving Fine Details in Feed-Forward Wide-Coverage Reconstruction
Long-LRM++ achieves real-time 14 FPS high-fidelity 360-degree scene reconstruction from 32-64 views by using semi-explicit Gaussians plus a light decoder, matching LaCT quality on DL3DV and improving depth prediction.
-
Improving 3D Gaussian Splatting Compression by Scene-Adaptive Lattice Vector Quantization
Scene-adaptive lattice vector quantization improves rate-distortion performance of 3DGS compression over uniform scalar quantization while adding little overhead and supporting multiple bit rates from one trained model.
-
ODE-GS: Latent ODEs for Dynamic Scene Extrapolation with 3D Gaussian Splatting
ODE-GS uses latent neural ODEs on Gaussian parameters to extrapolate dynamic 3D scenes, reporting 19.8% metric gains over baselines on D-NeRF, NVFi, and HyperNeRF.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.