Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

How to Use Diffusion Priors under Sparse Views?

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Score Distillation Sampling can provide effective visual guidance for sparse-view 3D reconstruction if the rendered image distribution is first rectified by warping the sparse views into pseudo viewpoints.

desk verdict Strong empirical method built on an invalid gradient decomposition; worth a careful revision, not a desk reject. read the letter →

arxiv 2412.02225 v1 pith:ZWUYIVMD submitted 2024-12-03 cs.CV

Qisen Wang , Yifan Zhao , Jiawei Ma , Jia Li This is my paper · ORCID
classification cs.CV
keywords scoredistillationsamplingsparse-viewnovelviewsynthesisdiffusionpriors3DGaussianSplattinginlinemodedeviationinpaintingmodeldepthregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Score Distillation Sampling (SDS) fails for sparse-view 3D reconstruction because it drives the rendered image distribution toward the nearest mode of a pre-trained diffusion prior, which is often a failure mode inconsistent with the given views. It proposes Inline Prior Guided Score Matching (IPSM), which first warps a seen view into pseudo viewpoints using the current rendered depth and masks the result for consistency, forming an inline visual prior, then rectifies the rendered distribution with this prior through an inpainting diffusion model. The original SDS objective is decomposed into two sub-objectives: pulling the rendered distribution toward the rectified distribution and pulling the rectified distribution toward the diffusion prior. On top of 3D Gaussian Splatting, the IPSM-Gaussian pipeline adds monocular depth and geometry consistency regularizations and reports state-of-the-art reconstruction quality on LLFF and DTU under three input views. A sympathetic reader would take away that diffusion visual guidance can be harvested under sparse views without fine-tuning or pre-training, provided the geometry used to build the inline prior is trustworthy.

What carries the argument

The rectified distribution, built by inverse warping a seen viewpoint image to pseudo viewpoints with the rendered depth and masked by a depth-consistency threshold (Eqs. 8-10), is the central object. This distribution is the bridge: it decomposes the SDS objective into two KL divergences (Eq. 11) and is realized in practice by an inpainting diffusion model that shares a VAE feature space with the base diffusion model. The two gradient terms of Eq. 12 implement the decomposition: one matches the rendered image to the rectified image, the other matches the rectified image to the diffusion prior.

What would settle it

Replace the rendered depth in Eqs. 7-10 with increasingly corrupted depth (or remove monocular depth regularization entirely) on a scene with thin structures and repeated texture, and track reconstruction quality: if quality does not degrade, the inline prior is not playing the claimed corrective role; if even mild corruption collapses the gain over SDS, then the external depth prior, not the diffusion rectification, is the load-bearing component.

Watch

Extended reading notes

Core claim

The central claim is that the mode deviation of SDS under sparse views can be corrected by a rectified distribution constructed from inline priors: sampling pseudo viewpoints around a seen view, inverse-warping the seen image with rendered depth, masking by depth-consistency, and using the masked warped image as an inpainting condition. IPSM then replaces the single SDS gradient with two matched gradients: one between the rendered noisy image and the rectified noisy image, and one between the rectified noisy image and the pre-trained diffusion prior. The rectified distribution acts as an intermediate state that amplifies the distance between target and failure modes, steering optimization toward the target mode. The authors additionally support the inline prior with monocular depth regularization and a pixel-level geometry consistency loss, and show that direct SDS guidance degrades the 3D Gaussian Splatting baseline performance while IPSM improves it.

Load-bearing premise

The method assumes the inverse-warped, depth-masked image from a seen view is a trustworthy visual condition for the inpainting diffusion model; if the rendered depth is wrong, the inline prior is wrong and the diffusion guidance will reinforce incorrect geometry rather than correct it.

Editorial extensions

If this is right

  • IPSM provides effective diffusion visual guidance for sparse-view reconstruction without fine-tuning or pre-training, improving 3D Gaussian Splatting baselines on LLFF and DTU under 3 views.
  • Direct SDS with both low and high CFG values degrades or gives unstable reconstruction, whereas IPSM reduces cross-run variance in SSIM, LPIPS, and PSNR.
  • The method scales to 6 and 9 input views with the same hyperparameters, outperforming prior methods including those that require pre-training.
  • Depth and geometry consistency regularization are necessary supports: monocular depth keeps the warped inline prior accurate, and masked L1 narrows the pixel-level gap between rendered and rectified images.
  • View-conditioned 3D diffusion priors used with vanilla SDS still fail to boost sparse-view reconstruction, suggesting that the inline rectification, not the dimensionality of the prior, is what matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The need for the rectified distribution to live in the same VAE feature space as the diffusion prior couples the method to a specific inpainting model; a natural extension is to learn a lightweight adapter that maps other inpainting or generation priors into that space.
  • Because the inline prior is constructed from rendered depth, the method inherits the failure modes of monocular depth estimation; scenes with repeated texture, thin structures, or large occlusions could be tested as stress cases, and one would predict quality drops tied to depth accuracy.
  • The two-term decomposition suggests a general recipe: any task where a cheap geometric prior can pin down part of the target distribution could use the same rectified-bridge trick to make a text-conditioned generative prior obey scene-specific constraints.
  • The experiments with opposite and extrapolated views show large absolute gaps remain; the method's benefit is currently largest in near-novel views close to the seen viewpoints, so extending the pseudo-view sampling strategy to cover large baselines is a testable next step.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses sparse-view novel view synthesis by combining 3D Gaussian Splatting with a diffusion prior. The authors observe that vanilla Score Distillation Sampling (SDS) degrades reconstruction quality under sparse views, and they attribute this to mode deviation. They propose Inline Prior Guided Score Matching (IPSM), which warps a visible input image to pseudo viewpoints using the current rendered depth, masks the result with a geometric consistency check, and uses this masked warped image as an inpainting condition for a diffusion model. The claimed theoretical contribution is that the SDS objective can be decomposed into two KL divergences, leading to two sub-objectives whose gradients are given in Eq. (12) and whose losses are given in Eq. (13). The full pipeline, IPSM-Gaussian, adds monocular depth regularization and pixel-level geometry consistency regularization. Experiments on LLFF and DTU report state-of-the-art metrics, with ablations and multiple repeated runs. The central theoretical derivation, however, has gaps that bear directly on the paper's main claim of decomposing the SDS objective; my recommendation is major revision.

Significance. If the empirical results hold, the paper offers a practical recipe: using an inpainting diffusion model conditioned on inline geometric priors, together with depth and geometry regularizers, improves sparse-view 3DGS reconstruction without fine-tuning the diffusion model. The paper is commendable for reporting repeated experiments with standard deviations, for including ablations of each loss component, and for releasing code. The additional experiments with 6 and 9 views, with two-view opposite and extrapolation settings, and with view-conditioned diffusion priors, are useful for situating the method. However, the paper's headline claim is that IPSM 'decomposes the original optimization objective of SDS,' and this claim is the basis for the method's motivation and for its distinction from a purely heuristic loss. Because the derivation in Eqs. (11)-(13) is not mathematically established, the contribution currently rests on an empirical heuristic whose theoretical framing needs substantial rework. The empirical improvements are suggestive but do not, by themselves, validate the decomposition claim.

major comments (4)
  1. [Sec. 3.2, Eq. (12)] The claimed gradient of the second KL term is incomplete. The rectified distribution q̃^{θ,φ} is conditioned on the warped masked image I^{i→j}_0 and the consistency mask M^{i→j} (Eqs. 8-10), and both depend on the rendered depth D_j, which is a function of θ. Therefore the derivative of D_KL(q̃^{θ,φ} || p*) with respect to θ includes additional terms through the conditioning variable, not just the rendering Jacobian ∂g/∂θ. A stop-gradient on the inline prior is nowhere stated. Without such a device, Eq. (12) is not the gradient of Eq. (11), and the central claim that IPSM 'decomposes the original optimization objective of SDS' is not established. Please derive the missing terms, or explicitly state and justify a stop-gradient or other relaxation that makes Eq. (12) valid.
  2. [Sec. 3.2, Eq. (13)] The squared-score objective is not a decomposition of the SDS loss. Writing a = ω(ε_φ − ε) and b = ω(ε* − ε_φ), one has ||a||² + ||b||² = ||a+b||² − 2a·b, so the cross term is generally non-zero; the objective in Eq. (13) therefore differs from the squared SDS residual by a generally non-vanishing term. In addition, with η_r = 0.1 the two terms are not even on equal footing, so the expression is not an algebraic split of the SDS gradient. Moreover, the gradient of Eq. (13) includes ∂ε_φ/∂θ terms through x_t and through the conditioning image, so Eq. (13) is not equivalent to Eq. (12) under the usual Jacobian-free SDS approximation. Please state precisely what objective is optimized in practice and how it relates to Eq. (11); if the method is a heuristic combination of score-matching terms, that should be stated explicitly rather than presented as a decomposition.
  3. [Sec. 3.2 and Sec. 3.3, Eqs. (8)-(10) vs. Eq. (11)] The optimization target is not fixed but is generated from the current θ: the warped image I^{i→j}_0 and the mask M^{i→j} use the rendered depth D_j, which is the very quantity being optimized. This makes the KL decomposition in Eq. (11) circular in the sense that the model is pulled toward a distribution that depends on its own estimate. The monocular depth regularization in Eqs. (14)-(15) mitigates the risk but does not eliminate it, and the paper does not analyze how sensitive IPSM is to inaccurate D_j beyond the illustrative Fig. 6(a). Please provide a quantitative study of this dependence, for example by comparing IPSM with rendered depth versus fixed monocular depth, or by corrupting D_j with controlled errors, to demonstrate that the method is not dominated by self-confirmation.
  4. [Sec. 4.2 and Appendix A.4] The SOTA comparison is only partially controlled. Several baselines are imported from prior papers rather than re-run under the same schedule, initialization, and codebase (e.g., FSGS on LLFF, DNGaussian on DTU), and the FSGS baseline on DTU is reproduced with modified hyperparameters plus an added background-mask prior. These choices are described in Appendix A.4, but the reader cannot tell from Table 1 which numbers are exactly comparable. Please mark clearly which baselines are reproduced under identical conditions, which are taken from published sources, and what effect the added prior has on the reproduced FSGS numbers; ideally, re-run the most competitive baselines under the paper's training schedule and SfM initialization.
minor comments (5)
  1. [Sec. 3.2, Eq. (11)] In the second KL term, the distribution p*_0(x^j_t) appears to mix an image-level distribution with a noisy variable x^j_t; this should presumably be p*_t(x^j_t).
  2. [Abstract and Sec. 1] The phrase 'without any fine-tuning or pre-training' is stronger than what is implemented, since the method uses a pre-trained Stable Diffusion inpainting model and a pre-trained monocular depth estimator; please clarify that the claim refers to no fine-tuning of the 3D representation or the diffusion backbone.
  3. [Fig. 6(b)] The caption of Fig. 6(b) is difficult to parse; please rewrite it to label clearly which panels correspond to Base, IPSM, depth regularization, and the combined method.
  4. [Sec. 5] The phrase 'comprehended analysis' in the Conclusion should be 'comprehensive analysis.'
  5. [Appendix A.4] The statement that the FSGS DTU reproduction used MipNeRF-360 hyperparameters because 'we observe that the selected hyper-parameters are more suitable' should be accompanied by the actual values or a small sensitivity study, since this choice directly affects the comparison in Table 1.

Circularity Check

2 steps flagged · score 6.0 of 10

IPSM's rectified distribution is built from the model's own rendered depth, and the claimed SDS decomposition is a gradient-level identity renamed as a loss-level decomposition; external diffusion and depth priors and independent benchmarks keep the result from being fully tautological.

  1. self definitional [Sec. 3.2 ('Inline Prior' and 'Inline Prior Guided Score Matching'), Eqs. 7-12]
    "we formulate the transforming function ψ(Ii0; Dj, Rj→i) which inversely warps image Ii0 from viewpoint vi to vj. ... we utilize the warped masked image Ii→j 0 from the seen viewpoints to guide the sampling trajectory of ˆxj;ϕ 0 ∼ ˜qθ,ϕ 0 (xj 0|Mi→j ⊙ Ii→j 0 , Mi→j)"

    The rectified distribution \tilde q^{θ,ϕ}_0 is conditioned on the warped masked image I^{i→j}_0 and mask M^{i→j}; both are computed by inverse-warping through the rendered depth D_j (Eqs. 8-10), which is a function of the optimized θ. Therefore the KL target in Eq. 11 is not an external distribution but a target generated from the model's own current depth estimate. Eq. 12's gradient contains only ∂g/∂θ and drops the ∂I^{i→j}_0/∂D_j · ∂D_j/∂θ chain that a true gradient of Eq. 11 requires; the paper never states a stop-gradient on the inline prior. The added monocular depth term (Eqs. 14-15) is independent grounding, but the claimed decomposition itself is self-referential.

  2. renaming known result [Abstract; Sec. 3.2, Eq. 13]
    "decomposes the original optimization objective of SDS ... Consequently, the IPSM regularization can be represented as LIPSM = ηr Et,ϵ,vj ||ω(t)(ϵϕ − ϵ)||2 + Et,ϵ,vj ||ω(t)(ϵ∗ − ϵϕ)||2."

    The transition from Eq. 12 to Eq. 13 treats the algebraic identity (ϵ*−ϵ) = (ϵϕ−ϵ)+(ϵ*−ϵϕ) as a decomposition of the SDS objective. But Eq. 12 is a gradient split, not a loss identity. Since ||a+b||² = ||a||²+||b||²+2a·b, the sum in Eq. 13 equals ||ω(ϵ*−ϵ)||² plus a generally nonzero cross term when ηr=1; it is not the SDS KL objective of Eq. 5. The 'decomposition' is therefore introduced by defining L_IPSM as that sum and renaming a gradient identity as a theoretical derivation, rather than deriving the loss from Eq. 5.

full rationale

Score 6: the central theoretical claim partially reduces by construction. The rectified distribution that IPSM uses as its optimization target is built by inverse-warping a seen image through the current rendered depth (Eqs. 7-10), so the model is pushed toward a target that depends on its own estimate. Moreover, the advertised 'decomposition' of the SDS objective is only an algebraic split of the SDS gradient; the squared-error loss in Eq. 13 differs from the SDS loss by a cross term, so the theoretical framing is a new heuristic objective presented as a derivation. The method is not a tautology: the frozen inpainting diffusion model, the external monocular depth prior, and the LLFF/DTU benchmarks provide independent content, and no load-bearing self-citation was found. If an implicit stop-gradient on the inline prior was intended, the first issue would become a formal-derivation gap rather than full circularity, but as written the claimed decomposition is not established.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a cluster of tuned hyperparameters (7+ hand-set weights/schedules) and on several unproved domain assumptions: the frozen diffusion and inpainting models, the compatibility of their VAE spaces, the accuracy of DPT monocular depth, and the validity of the rectified-distribution decomposition. The method is not a derivation from first principles; it is a carefully tuned recipe whose performance depends on external pretrained models.

free parameters (7)
  • lambda_IPSM = 2.0
    Weight of the IPSM regularization loss in Eq. 17, fixed for all datasets.
  • eta_r = 0.1
    Balance between the two IPSM sub-objectives LG1 and LG2 in Eq. 13.
  • eta_d = 0.1
    Balance between seen-view and pseudo-view depth supervision in Eq. 15.
  • lambda_depth = 0.5 on LLFF; 0.05 on DTU
    Weight of depth loss; reduced on DTU for background masking.
  • lambda_geo = 2.0 on LLFF; 0.2 on DTU
    Weight of geometry consistency loss in Eq. 16.
  • mask_threshold_tau = 0.3 for IPSM, 0.1 for Lgeo
    Threshold for the consistency mask in Eq. 10.
  • lambda_1 / lambda_ssim = 0.8/0.2 on LLFF; 0.4/0.6 on DTU
    Color and SSIM weights inherited from 3DGS, adjusted per dataset following DNGaussian.
assumptions (5)
  • domain assumption The pretrained Stable Diffusion v1.5 and Stable Diffusion Inpainting v1.5 models provide accurate estimates of the real-data score p*_t (Eq. 5-6).
    The entire score distillation framework treats frozen diffusion denoisers as a proxy for the true noisy image distribution.
  • domain assumption The VAE feature spaces of the base diffusion model and the inpainting model are compatible enough to combine their noise predictions in Eq. 12-13.
    The paper states they share the same VAE-feature domain; the method fails if this compatibility breaks.
  • domain assumption Monocular depth estimates from DPT are sufficiently accurate to regularize rendered depth and make the inverse warping (Eq. 8-10) reliable.
    IPSM depends on warped images; the paper itself identifies depth error as a failure mode (Sec. 4.3).
  • ad hoc to paper The rectified distribution sampled by the inpainting model is a valid intermediate distribution for KL decomposition (Eq. 11).
    This is the paper's central construction; its validity is assumed rather than proved, and the gradient in Eq. 12 drops d(epsilon_phi)/dtheta terms.
  • domain assumption Camera poses are known and SfM initialization is available for target scenes (LLFF all scenes; DTU except scans 30 and 110).
    Inverse warping requires accurate relative poses; SfM failure on two DTU scenes forces random initialization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How to Use Diffusion Priors under Sparse Views?." pith.science (2026). https://pith.science/paper/ZWUYIVMD

@misc{pith2026241202225,
  author       = {Pith},
  title        = {Pith review of: How to Use Diffusion Priors under Sparse Views?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZWUYIVMD}},
  note         = {Machine review of arXiv:2412.02225}
}
read the original abstract

Novel view synthesis under sparse views has been a long-term important challenge in 3D reconstruction. Existing works mainly rely on introducing external semantic or depth priors to supervise the optimization of 3D representations. However, the diffusion model, as an external prior that can directly provide visual supervision, has always underperformed in sparse-view 3D reconstruction using Score Distillation Sampling (SDS) due to the low information entropy of sparse views compared to text, leading to optimization challenges caused by mode deviation. To this end, we present a thorough analysis of SDS from the mode-seeking perspective and propose Inline Prior Guided Score Matching (IPSM), which leverages visual inline priors provided by pose relationships between viewpoints to rectify the rendered image distribution and decomposes the original optimization objective of SDS, thereby offering effective diffusion visual guidance without any fine-tuning or pre-training. Furthermore, we propose the IPSM-Gaussian pipeline, which adopts 3D Gaussian Splatting as the backbone and supplements depth and geometry consistency regularization based on IPSM to further improve inline priors and rectified distribution. Experimental results on different public datasets show that our method achieves state-of-the-art reconstruction quality. The code is released at https://github.com/iCVTEAM/IPSM.

Figures

Figures reproduced from arXiv: 2412.02225 by the authors.

Figure 1
Figure 1. Dilemma of SDS. Average PSNR↑, SSIM↑, and LPIPS↓ of each iteration on the LLFF test dataset [22] with Base (without SDS), SDS (CFG=7.5), and SDS (CFG=100). The prior-added period starts from the 2K iteration and ends at the 9.5K iteration. The opacity is also reset at 2K. The details and final training results of SDS are shown in Sec. 4.4. distinctive optimization characteristics across sparse input views. Consequen… view at source ↗
Figure 2
Figure 2. Comparison of SDS and IPSM. Left: Tending to seek nearest mode, causing mode deviation. Right: Rectifying distribution to seek the target mode. Rendering. For the 3D representation θ = {µn, Σn, cn, αn}, we can optimize the trainable parame￾ters θ through the following differentiable rendering function x0(p) = X N n=1 cnα˜n nY−1 m=1 (1 − α˜m), (3) where x0(p) is the rendering color at pixel p of rendered image x0, an… view at source ↗
Figure 3
Figure 3. IPSM-Gaussian obtains the inline prior within sparse views through inversely warping seen views to unseen pseudo views, thus modifying the rendered image distribution to the rectified distri￾bution. Consequently taking the rectified distribution as the intermediate state, two sub-optimization objectives are utilized for controlling the optimization direction. distribution of rendered images is not constrained to see… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on the LLFF dataset. [62] text prompts. Background priors are introduced on DTU for accurately reconstructing the object-of-interest. All experimental results are obtained on a single RTX 3090. More training details and experimental environments …
Figure 5
Figure 5. Figure 5: Qualitative comparison on DTU. DTU. Similar performances of the quantitative results on the DTU dataset [54] are shown in Tab. 1. The AVGE of our method is improved by +23.76% compared to FSGS [58] and +21.43% compared to FreeNeRF [8]. Note that DNGaus￾sian [9] does no…
Figure 6
Figure 6. Figure 6: (a) Impact of depth error on the inline prior. (b) Ablation of IPSM and depth regularizations. 4.3 Ablation Study We conduct detailed ablations of regularization terms on the LLFF dataset [22] shown in Tab. 2. We can notice that the first two regularization terms, i.e.…
Figure 7
Figure 7. Figure 7: Qualitative comparison with SDS [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Intuitive explanation of the inline priors. scan8 scan30 scan34 scan38 scan55 scan63 scan103 scan114 [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Examples of novel view synthesis from our method with 3 input views on the DTU dataset. image; the second column includes the rendering images of the pseudo unseen view which is sampled around the seen view (Eq. 3); the third column presents the rendering depths corres…
Figure 10
Figure 10. Figure 10: Examples of novel view synthesis from our method with 3 input views on the LLFF dataset. B.8 More Qualitative Results We present additional examples of rendered images in the test set shown in [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sparse-View 3D Reconstruction: Recent Advances and Open Challenges

    cs.CV 2025-07 conditional novelty 3.0 of 10

    A comprehensive survey that organizes sparse-view 3D reconstruction methods into geometry-based, NeRF, 3DGS, and diffusion-based categories, with benchmarks and open challenges.

Reference graph

Works this paper leans on

74 extracted references · 66 canonical work pages · cited by 1 Pith paper

  1. [1]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021

    2021

  2. [2]

    3d gaussian splatting for real-time radiance field rendering,

    B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Transactions on Graphics (TOG), vol. 42, no. 4, pp. 1–14, 2023

    work page 2023

  3. [3]

    Beyondpixels: A comprehensive review of the evolution of neural radiance fields,

    A. Rabby and C. Zhang, “Beyondpixels: A comprehensive review of the evolution of neural radiance fields,” arXiv preprint arXiv:2306.03000, 2023

    arXiv 2023

  4. [4]

    Nerf: Neural radiance field in 3d vision, a comprehensive review,

    K. Gao, Y . Gao, H. He, D. Lu, L. Xu, and J. Li, “Nerf: Neural radiance field in 3d vision, a comprehensive review,”arXiv preprint arXiv:2210.00379, 2022

    arXiv 2022

  5. [5]

    A survey on 3d gaussian splatting,

    G. Chen and W. Wang, “A survey on 3d gaussian splatting,”arXiv preprint arXiv:2401.03890, 2024

    arXiv 2024

  6. [6]

    3d gaussian as a new vision era: A survey,

    B. Fei, J. Xu, R. Zhang, Q. Zhou, W. Yang, and Y . He, “3d gaussian as a new vision era: A survey,”arXiv preprint arXiv:2402.07181, 2024

    arXiv 2024

  7. [7]

    Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs,

    M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan, “Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5480–5490, 2022

    work page 2022

  8. [8]

    Freenerf: Improving few-shot neural rendering with free frequency regularization,

    J. Yang, M. Pavone, and Y . Wang, “Freenerf: Improving few-shot neural rendering with free frequency regularization,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8254–8263, 2023

    work page 2023

Show all 74 references
  1. [9]

    Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization,

    J. Li, J. Zhang, X. Bai, J. Zheng, X. Ning, J. Zhou, and L. Gu, “Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20775–20785, 2024

    2024

  2. [10]

    Putting nerf on a diet: Semantically consistent few-shot view synthesis,

    A. Jain, M. Tancik, and P. Abbeel, “Putting nerf on a diet: Semantically consistent few-shot view synthesis,” in International Conference on Computer Vision (ICCV), pp. 5885–5894, 2021

    2021

  3. [11]

    Depth-supervised nerf: Fewer views and faster training for free,

    K. Deng, A. Liu, J.-Y . Zhu, and D. Ramanan, “Depth-supervised nerf: Fewer views and faster training for free,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12882–12891, 2022

    2022

  4. [12]

    Sparsenerf: Distilling depth ranking for few-shot novel view synthesis,

    G. Wang, Z. Chen, C. C. Loy, and Z. Liu, “Sparsenerf: Distilling depth ranking for few-shot novel view synthesis,” in International Conference on Computer Vision (ICCV), pp. 9065–9076, 2023

    2023

  5. [13]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervi- sion,” in International Conference on Machine Learning (ICML), pp. 8748–8763, 2021

    2021

  6. [14]

    Sparsegs: Real-time 360 {\deg} sparse view synthesis using gaussian splatting,

    H. Xiong, S. Muttukuru, R. Upadhyay, P. Chari, and A. Kadambi, “Sparsegs: Real-time 360 {\deg} sparse view synthesis using gaussian splatting,” arXiv preprint arXiv:2312.00206, 2023

    2023 arXiv

  7. [15]

    Reconfusion: 3d reconstruction with diffusion priors,

    R. Wu, B. Mildenhall, P. Henzler, K. Park, R. Gao, D. Watson, P. P. Srinivasan, D. Verbin, J. T. Barron, B. Poole, et al., “Reconfusion: 3d reconstruction with diffusion priors,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21551–21561, 2024

    2024

  8. [16]

    Deceptive-nerf/3dgs: Diffusion- generated pseudo-observations for high-quality sparse-view reconstruction,

    X. Liu, J. Chen, S.-H. Kao, Y .-W. Tai, and C.-K. Tang, “Deceptive-nerf/3dgs: Diffusion- generated pseudo-observations for high-quality sparse-view reconstruction,” in ECCV 2024 Workshop on Wild 3D: 3D Modeling, Reconstruction, and Generation in the Wild, 2024

    2024

  9. [17]

    Deep unsupervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International Conference on Machine Learning (ICML), pp. 2256–2265, 2015

    2015

  10. [18]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 6840–6851, 2020. 11

    2020

  11. [19]

    Score-based generative modeling through stochastic differential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” in International Conference on Learning Representations (ICLR), 2021

    2021

  12. [20]

    Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models,

    F. Bao, C. Li, J. Zhu, and B. Zhang, “Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models,” inInternational Conference on Learning Representations (ICLR), 2021

    2021

  13. [21]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in International Conference on Learning Representations (ICLR), 2021

    2021

  14. [22]

    Local light field fusion: Practical view synthesis with prescriptive sampling guidelines,

    B. Mildenhall, P. P. Srinivasan, R. Ortiz-Cayon, N. K. Kalantari, R. Ramamoorthi, R. Ng, and A. Kar, “Local light field fusion: Practical view synthesis with prescriptive sampling guidelines,” ACM Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1–14, 2019

    2019

  15. [23]

    Dreamfusion: Text-to-3d using 2d diffusion,

    B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text-to-3d using 2d diffusion,” in International Conference on Learning Representations (ICLR), 2023

    2023

  16. [24]

    Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors,

    G. Qian, J. Mai, A. Hamdi, J. Ren, A. Siarohin, B. Li, H.-Y . Lee, I. Skorokhodov, P. Wonka, S. Tulyakov, et al., “Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors,” in International Conference on Learning Representations (ICLR), 2024

    2024

  17. [25]

    Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,

    Z. Wang, C. Lu, Y . Wang, F. Bao, C. Li, H. Su, and J. Zhu, “Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,”Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023

    2023

  18. [26]

    Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching,

    Y . Liang, X. Yang, J. Lin, H. Li, X. Xu, and Y . Chen, “Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6517–6526, 2024

    2024

  19. [27]

    Novel view synthesis in tensor space,

    S. Avidan and A. Shashua, “Novel view synthesis in tensor space,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1034–1040, IEEE, 1997

    1997

  20. [28]

    Instant neural graphics primitives with a multiresolution hash encoding,

    T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Transactions on Graphics (TOG), vol. 41, no. 4, pp. 1–15, 2022

    2022

  21. [29]

    Plenoxels: Radiance fields without neural networks,

    S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5501–5510, 2022

    2022

  22. [30]

    Tensorf: Tensorial radiance fields,

    A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” inEuropean Conference on Computer Vision (ECCV), pp. 333–350, 2022

    2022

  23. [31]

    D-nerf: Neural radiance fields for dynamic scenes,

    A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, “D-nerf: Neural radiance fields for dynamic scenes,” in Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 10318–10327, 2021

    2021

  24. [32]

    Nerf in the wild: Neural radiance fields for unconstrained photo collections,

    R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, “Nerf in the wild: Neural radiance fields for unconstrained photo collections,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7210–7219, 2021

    2021

  25. [33]

    Block-nerf: Scalable large scene neural view synthesis,

    M. Tancik, V . Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-nerf: Scalable large scene neural view synthesis,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8248–8258, 2022

    2022

  26. [34]

    Ref-nerf: Structured view-dependent appearance for neural radiance fields,

    D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P. Srinivasan, “Ref-nerf: Structured view-dependent appearance for neural radiance fields,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5481–5490, 2022

    2022

  27. [35]

    Mip-nerf 360: Unbounded anti-aliased neural radiance fields,

    J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5470–5479, 2022. 12

    2022

  28. [36]

    Barf: Bundle-adjusting neural radiance fields,

    C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey, “Barf: Bundle-adjusting neural radiance fields,” in International Conference on Computer Vision (ICCV), pp. 5741–5751, 2021

    2021

  29. [37]

    Neural sparse voxel fields,

    L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt, “Neural sparse voxel fields,”Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 15651–15663, 2020

    2020

  30. [38]

    Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes,

    X. Zhou, Z. Lin, X. Shan, Y . Wang, D. Sun, and M.-H. Yang, “Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21634–21643, 2024

    2024

  31. [39]

    Splattingavatar: Realistic real-time human avatars with mesh-embedded gaussian splatting,

    Z. Shao, Z. Wang, Z. Li, D. Wang, X. Lin, Y . Zhang, M. Fan, and Z. Wang, “Splattingavatar: Realistic real-time human avatars with mesh-embedded gaussian splatting,” inConference on Computer Vision and Pattern Recognition (CVPR), pp. 1606–1616, 2024

    2024

  32. [40]

    Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling,

    Z. Li, Z. Zheng, L. Wang, and Y . Liu, “Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19711–19722, 2024

    2024

  33. [41]

    Hifi4g: High-fidelity human performance rendering via compact gaussian splatting,

    Y . Jiang, Z. Shen, P. Wang, Z. Su, Y . Hong, Y . Zhang, J. Yu, and L. Xu, “Hifi4g: High-fidelity human performance rendering via compact gaussian splatting,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19734–19745, 2024

    2024

  34. [42]

    Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,

    J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” in International Conference on Learning Representations (ICLR), 2023

    2023

  35. [43]

    Flipnerf: Flipped reflection rays for few-shot novel view synthesis,

    S. Seo, Y . Chang, and N. Kwak, “Flipnerf: Flipped reflection rays for few-shot novel view synthesis,” in International Conference on Computer Vision (ICCV), pp. 22883–22893, 2023

    2023

  36. [44]

    Därf: Boosting radiance fields from sparse input views with monocular depth adaptation,

    J. Song, S. Park, H. An, S. Cho, M.-S. Kwak, S. Cho, and S. Kim, “Därf: Boosting radiance fields from sparse input views with monocular depth adaptation,” Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023

    2023

  37. [45]

    Simplenerf: Regularizing sparse input neural radiance fields with simpler solutions,

    N. Somraj, A. Karanayil, and R. Soundararajan, “Simplenerf: Regularizing sparse input neural radiance fields with simpler solutions,” in SIGGRAPH Asia 2023 Conference Papers, pp. 1–11, 2023

    2023

  38. [46]

    Geconerf: Few-shot neural radiance fields via geometric consistency,

    M.-S. Kwak, J. Song, and S. Kim, “Geconerf: Few-shot neural radiance fields via geometric consistency,” in International Conference on Machine Learning (ICML) , pp. 18023–18036, 2023

    2023

  39. [47]

    Geoaug: Data augmentation for few-shot nerf with geometry constraints,

    D. Chen, Y . Liu, L. Huang, B. Wang, and P. Pan, “Geoaug: Data augmentation for few-shot nerf with geometry constraints,” in European Conference on Computer Vision (ECCV), pp. 322–337, 2022

    2022

  40. [48]

    Vision transformers for dense prediction,

    R. Ranftl, A. Bochkovskiy, and V . Koltun, “Vision transformers for dense prediction,” in International Conference on Computer Vision (ICCV), pp. 12179–12188, 2021

    2021

  41. [49]

    pixelnerf: Neural radiance fields from one or few images,

    A. Yu, V . Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields from one or few images,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4578–4587, 2021

    2021

  42. [50]

    Zero-1-to-3: Zero-shot one image to 3d object,

    R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. V ondrick, “Zero-1-to-3: Zero-shot one image to 3d object,” in International Conference on Computer Vision (ICCV), pp. 9298–9309, 2023

    2023

  43. [51]

    Zeronvs: Zero-shot 360-degree view synthesis from a single image,

    K. Sargent, Z. Li, T. Shah, C. Herrmann, H.-X. Yu, Y . Zhang, E. R. Chan, D. Lagun, L. Fei- Fei, D. Sun, et al., “Zeronvs: Zero-shot 360-degree view synthesis from a single image,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9420–9429, 2024

    2024

  44. [52]

    invs: Repurposing diffusion inpainters for novel view synthesis,

    Y . Kant, A. Siarohin, M. Vasilkovsky, R. A. Guler, J. Ren, S. Tulyakov, and I. Gilitschenski, “invs: Repurposing diffusion inpainters for novel view synthesis,” in SIGGRAPH Asia 2023 Conference Papers, pp. 1–12, 2023. 13

    2023

  45. [53]

    Stable score distillation for high-quality 3d generation,

    B. Tang, J. Wang, Z. Wu, and L. Zhang, “Stable score distillation for high-quality 3d generation,” arXiv preprint arXiv:2312.09305, 2023

    2023 arXiv

  46. [54]

    Large scale multi-view stereopsis evaluation,

    R. Jensen, A. Dahl, G. V ogiatzis, E. Tola, and H. Aanæs, “Large scale multi-view stereopsis evaluation,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 406–413, 2014

    2014

  47. [55]

    Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes,

    J. Chibane, A. Bansal, V . Lazova, and G. Pons-Moll, “Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes,” inConference on Computer Vision and Pattern Recognition (CVPR), pp. 7911–7920, 2021

    2021

  48. [56]

    Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo,

    A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, “Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo,” in International Conference on Computer Vision (ICCV), pp. 14124–14133, 2021

    2021

  49. [57]

    Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,

    J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” in International Conference on Computer Vision (ICCV), pp. 5855–5864, 2021

    2021

  50. [58]

    Fsgs: Real-time few-shot view synthesis using gaussian splatting,

    Z. Zhu, Z. Fan, Y . Jiang, and Z. Wang, “Fsgs: Real-time few-shot view synthesis using gaussian splatting,” in European Conference on Computer Vision (ECCV), pp. 145–163, 2025

    2025

  51. [59]

    Image quality assessment: from error visibility to structural similarity,

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004

    2004

  52. [60]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” inConference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595, 2018

    2018

  53. [61]

    Structure-from-motion revisited,

    J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4104–4113, 2016

    2016

  54. [62]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inInternational Conference on Machine Learning (ICML), pp. 12888–12900, 2022

    2022

  55. [63]

    High-resolution image synthe- sis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthe- sis with latent diffusion models,” in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695, 2022

    2022

  56. [64]

    Cor-gs: sparse-view 3d gaussian splatting via co-regularization,

    J. Zhang, J. Li, X. Yu, L. Huang, L. Gu, J. Zheng, and X. Bai, “Cor-gs: sparse-view 3d gaussian splatting via co-regularization,” in European Conference on Computer Vision (ECCV), pp. 335–352, 2025

    2025

  57. [65]

    Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction,

    J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny, “Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction,” in International Conference on Computer Vision (ICCV), pp. 10901–10911, 2021

    2021

  58. [66]

    Development of an image data set of construction machines for deep learning object detection,

    B. Xiao and S.-C. Kang, “Development of an image data set of construction machines for deep learning object detection,” Journal of Computing in Civil Engineering , vol. 35, no. 2, p. 05020005, 2021

    2021

  59. [67]

    Deep learning image captioning in construction manage- ment: a feasibility study,

    B. Xiao, Y . Wang, and S.-C. Kang, “Deep learning image captioning in construction manage- ment: a feasibility study,” Journal of Construction Engineering and Management , vol. 148, no. 7, p. 04022049, 2022

    2022

  60. [68]

    Stereo magnification: learning view synthesis using multiplane images,

    T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, “Stereo magnification: learning view synthesis using multiplane images,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1–12, 2018

    2018

  61. [69]

    Laion-5b: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al., “Laion-5b: An open large-scale dataset for training next generation image-text models,” Advances in Neural Information Processing Systems (NeurIPS), vo...

    2022

  62. [70]

    Following RegNeRF [7], FreeNeRF [8], and DNGaussian [9], we only use the selected 15 testing scenes for optimization

    split the DTU dataset [ 54] into 88 training scenes for pre-training and 15 testing scenes for per-scene fine-tuning. Following RegNeRF [7], FreeNeRF [8], and DNGaussian [9], we only use the selected 15 testing scenes for optimization. The IDs of testing scenes are: 8, 21, 30,...

  63. [71]

    Meanwhile, the reported results of 3DGS [2] are also obtained with the SfM [61] initialization which is the same as ours

    to ensure fairness, although this rasterizer has the same function as the rasterizer of 3DGS [2]. Meanwhile, the reported results of 3DGS [2] are also obtained with the SfM [61] initialization which is the same as ours. FSGS. We use the official code of FSGS [58] to reproduce ...

  64. [72]

    A.5 Experimental Environments and Computing Resources All the experiments are conducted on a single RTX 3090 with CUDA 11.3

    initialization on the DTU dataset [54]. A.5 Experimental Environments and Computing Resources All the experiments are conducted on a single RTX 3090 with CUDA 11.3. The training time of IPSM-Gaussian is about 1 hour on the RTX 3090, which is mainly due to the inference time of...

  65. [73]

    and our model achieves improvements of 27.27%, 22.57% on average PSNR and A VGE scores respectively. We can notice that although our method is improved compared to DNGaussian, in fact, current sparse-view reconstruction methods (including our method) cannot successfully recons...

  66. [74]

    is superior compared to using Zero-1-to-3 [50] since the former utilizes 3D annotated scene data for fine-tuning while Zero-1-to-3 only uses 3D objects dataset for fine-tuning. However, although ZeroNVS [51] as 3D prior can achieve stunning results in single-view reconstructio...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.