Mind the Gap: Standard 3DGS Evaluation Primarily Measures Near-Trajectory Interpolation
Pith reviewed 2026-07-03 00:48 UTC · model grok-4.3
The pith
Standard 3D Gaussian Splatting evaluation measures near-trajectory interpolation rather than spatial generalization.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Standard MipNeRF360-style 3D Gaussian Splatting evaluation holds out every N-th frame, but these frames have trained neighbors on both sides, so the metric measures near-trajectory interpolation rather than spatial generalization. A matched-count protocol isolating this effect by training on the same number of images but differing in holdout distribution finds a large consistent interpolation-extrapolation gap of 3~12dB that exceeds typical method differences, is robust to training noise, can flip rankings, and persists across representation families including NeRF.
What carries the argument
The matched-count paired holdout protocol, where one arm uses evenly spread holdouts for interpolation and the other uses a contiguous spatial sector for extrapolation, while keeping training image count identical.
If this is right
- The performance gap is dominated by a diffuse/geometry-proxy component.
- The gap tracks each view's angular distance to its nearest training view, serving as a zero-cost signal for capture planning.
- Loss-side regularization produces only marginal gains.
- Standard holdouts remain useful for near-trajectory rendering but should not alone be interpreted as evidence of spatial generalization.
Where Pith is reading between the lines
- Future benchmarks could routinely report both interpolation and extrapolation metrics to distinguish these capabilities.
- The angular distance to nearest training view could be incorporated into training objectives to encourage better spatial coverage.
- This evaluation gap may affect conclusions in other novel view synthesis tasks where camera trajectories are limited.
- Capture strategies that maximize minimum angular separation might reduce the observed extrapolation penalty.
Load-bearing premise
The contiguous spatial sector holdout and the evenly spread holdout differ only in interpolation versus extrapolation properties and introduce no other systematic differences in scene coverage, lighting, or reconstruction difficulty when the image count is matched.
What would settle it
Finding that performance on the contiguous sector holdout matches the even-spread holdout performance when the number of training images is identical, or that controlling for angular distance eliminates the gap.
Figures
read the original abstract
Standard MipNeRF360-style 3D Gaussian Splatting (3DGS) evaluation holds out every N-th frame -- but these frames have trained neighbors on both sides, so the metric measures near-trajectory interpolation rather than spatial generalization. We introduce a fair matched-count protocol that isolates this effect: both arms train on the same number of images and differ only in whether the holdout is spread evenly (interpolation) or forms a contiguous spatial sector (extrapolation). Our primary finding is a large, consistent interpolation-extrapolation gap of 3~12dB -- several times the differences typically reported between competing methods. The gap is robust to training noise, is in two cases large enough to flip a method ranking under multi-seed confirmation, and -- crucially -- persists across three representation families, including a non-Gaussian volumetric neural radiance field (NeRF), so it reflects spatial coverage rather than any one representation. Diagnostically, it is dominated by a diffuse/geometry-proxy component and tracks each view's angular distance to its nearest training view, a zero-cost signal that also guides capture planning; loss-side regularization yields only marginal gains. Standard holdouts remain useful for near-trajectory rendering but should not, alone, be read as evidence of spatial generalization. Prior work notes protocol sensitivity; ours is, to our knowledge, the first to combine matched-count paired holdout, cross-representation quantification, and a diagnostic analysis Table 1. We describe a spatial-holdout benchmark toolkit with standardized splits and baselines for 16 scenes, which we are preparing for public release.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that standard MipNeRF360-style 3DGS evaluation (every-Nth-frame holdouts) primarily measures near-trajectory interpolation rather than spatial generalization. It introduces a matched-count protocol that compares evenly-spread holdouts (interpolation) against contiguous spatial-sector holdouts (extrapolation) while fixing training image count, reports a consistent 3-12 dB PSNR gap across three representation families (including NeRF), shows the gap can flip method rankings under multi-seed evaluation, and provides an angular-distance diagnostic that tracks per-view difficulty.
Significance. If the reported gap is attributable solely to interpolation vs. extrapolation properties rather than training-set confounds, the result would be significant for novel-view synthesis evaluation: the gap magnitude exceeds typical inter-method differences, the cross-representation persistence (including non-Gaussian NeRF) strengthens the spatial-coverage interpretation, and the zero-cost angular-distance diagnostic offers a practical capture-planning signal. The multi-seed confirmation and preparation of a 16-scene benchmark toolkit are additional strengths.
major comments (3)
- [matched-count protocol (abstract and §3)] The matched-count protocol description does not demonstrate that angular statistics or spatial density of the training views are equalized between the evenly-spread and contiguous-sector arms; removing a contiguous block necessarily creates larger gaps in the training trajectory itself, which could alter optimization dynamics, regularization, or learned proxies independently of the test-set interpolation/extrapolation distinction.
- [Table 1 and multi-seed experiments] Table 1 and the multi-seed results report consistent gaps and occasional ranking flips but provide no error bars, standard deviations, or statistical significance tests, weakening the claim that the gap is robust to training noise and large enough to reliably flip rankings.
- [diagnostic analysis and §4] While the angular-distance diagnostic is shown to track per-view error, the manuscript does not include a control experiment that holds training-view angular coverage fixed while varying only test-set properties, leaving the central isolation of interpolation vs. extrapolation unverified.
minor comments (2)
- [abstract] The abstract states the toolkit is 'preparing for public release' but does not specify the exact split definitions or baseline implementations that will be released, which would aid reproducibility.
- [throughout] Notation for the two holdout arms (e.g., 'evenly spread' vs. 'contiguous sector') should be defined once with a short acronym or label for consistent use in figures and tables.
Simulated Author's Rebuttal
We thank the referee for their constructive comments and for recognizing the potential impact of our work. We provide point-by-point responses to the major comments below. We will make revisions to address the concerns raised.
read point-by-point responses
-
Referee: [matched-count protocol (abstract and §3)] The matched-count protocol description does not demonstrate that angular statistics or spatial density of the training views are equalized between the evenly-spread and contiguous-sector arms; removing a contiguous block necessarily creates larger gaps in the training trajectory itself, which could alter optimization dynamics, regularization, or learned proxies independently of the test-set interpolation/extrapolation distinction.
Authors: We agree that the manuscript does not explicitly demonstrate equalization of angular statistics or spatial density for the training views. The protocol is designed to fix the training image count while varying the holdout configuration to isolate test-set properties. However, as noted, the contiguous holdout introduces a larger gap in the training trajectory. In the revision, we will add quantitative comparisons of the angular coverage and gap statistics for the training sets in both protocols (e.g., histograms of angular distances between training views). This will clarify the extent to which training dynamics may differ and allow readers to assess the contribution of test-set extrapolation. We will also discuss this as a potential confounding factor. revision: yes
-
Referee: [Table 1 and multi-seed experiments] Table 1 and the multi-seed results report consistent gaps and occasional ranking flips but provide no error bars, standard deviations, or statistical significance tests, weakening the claim that the gap is robust to training noise and large enough to reliably flip rankings.
Authors: We acknowledge the absence of error bars and statistical tests in the current manuscript. To strengthen these claims, we will expand the multi-seed experiments, report standard deviations across seeds, and include statistical significance tests (e.g., paired t-tests) for the observed gaps and ranking flips in the revised Table 1 and associated text. This will provide a more rigorous quantification of robustness to training noise. revision: yes
-
Referee: [diagnostic analysis and §4] While the angular-distance diagnostic is shown to track per-view error, the manuscript does not include a control experiment that holds training-view angular coverage fixed while varying only test-set properties, leaving the central isolation of interpolation vs. extrapolation unverified.
Authors: The matched-count protocol aims to isolate the effect by fixing training count, but we recognize that a stricter control holding training angular coverage exactly fixed while varying only the test set would provide stronger isolation. Constructing such a control is non-trivial because the holdout selection directly affects both. In the revision, we will add a discussion of this limitation and explore whether additional experiments (e.g., using synthetic view selection to match angular stats) can be included. The persistence of the gap across representations and the diagnostic's correlation with angular distance provide supporting evidence, but we agree a dedicated control would be valuable. revision: partial
Circularity Check
No circularity: purely empirical measurement of holdout gap
full rationale
The paper reports an empirical comparison of two matched-count holdout protocols (evenly-spread vs. contiguous-sector) across multiple scenes and representations, measuring PSNR/SSIM differences directly from rendered outputs. No equations, derivations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text or abstract. The central claim is the observed 3-12 dB gap and its correlation with angular distance; these are data-driven observations, not quantities constructed by definition from the inputs or prior self-work. The protocol description and diagnostic analysis stand independently without reducing to any of the enumerated circular patterns.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption The contiguous spatial sector holdout and the evenly spread holdout differ solely in spatial distribution and introduce no other systematic differences in reconstruction difficulty when the number of training images is identical.
Reference graph
Works this paper leans on
- [1]
-
[2]
HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds
HunyuanWorld Team. HY-World 2.0: A multi-modal world model for reconstructing, generating, and simulating 3D worlds.arXiv preprint arXiv:2604.14268, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[3]
H.-X. Yu, H. Duan, C. Herrmann, W. T. Freeman, and J. Wu. WonderWorld: Interactive 3D scene generation from a single image. InCVPR, 2025
work page 2025
-
[4]
Gao,et al.CAT3D: Create anything in 3D with multi- view diffusion models
R. Gao,et al.CAT3D: Create anything in 3D with multi- view diffusion models. InNeurIPS, 2024
work page 2024
-
[5]
W. Yu,et al.ViewCrafter: Taming video diffusion models for high-fidelity novel view synthesis.IEEE TPAMI, 2025
work page 2025
-
[6]
J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. InCVPR, 2022
work page 2022
-
[7]
A. Knapitsch, J. Park, Q.-Y . Zhou, and V . Koltun. Tanks and Temples: Benchmarking large-scale scene reconstruc- tion.ACM Trans. Graph., 36(4), 2017
work page 2017
-
[8]
R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi,et al.NeRF in the Wild: Neural radiance fields for unconstrained photo collections. InCVPR, 2021
work page 2021
-
[9]
V . Ye, R. Li, J. Kerr, M. Turkulainen,et al.gsplat: An open- source library for Gaussian Splatting.JMLR, 26(34), 2025
work page 2025
-
[10]
J. Kulhanek and T. Sattler. NerfBaselines: Consistent and reproducible evaluation of novel view synthesis methods. In NeurIPS Datasets & Benchmarks, 2025
work page 2025
-
[11]
Ye,et al.AbsGS: Recovering fine details for 3D Gaussian Splatting
Z. Ye,et al.AbsGS: Recovering fine details for 3D Gaussian Splatting. InACM MM, 2024
work page 2024
-
[12]
Yu,et al.Mip-Splatting: Alias-free 3D Gaussian Splat- ting
Z. Yu,et al.Mip-Splatting: Alias-free 3D Gaussian Splat- ting. InCVPR, 2024
work page 2024
-
[13]
J. C. Lee, D. Rho, X. Sun, J. H. Ko, and E. Park. Compact 3D Gaussian representation for radiance field. InCVPR, 2024
work page 2024
- [14]
- [15]
-
[16]
K. Sargent, Z. Li, T. Shah, C. Herrmann, H.-X. Yu, Y . Zhang, et al.ZeroNVS: Zero-shot 360-degree view synthesis from a single image. InCVPR, 2024
work page 2024
-
[17]
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. InECCV, 2020
work page 2020
-
[18]
F. Warburg, E. Weber, M. Tancik, A. Holynski, and A. Kanazawa. Nerfbusters: Removing ghostly artifacts from casually captured NeRFs. InICCV, 2023
work page 2023
-
[19]
X. Chen, Q. Zhang, X. Li, Y . Chen, Y . Feng, X. Wang, and J. Wang. Hallucinated neural radiance fields in the wild. In CVPR, 2022
work page 2022
-
[20]
D. Kotovenko, O. Grebenkova, and B. Ommer. EDGS: Elim- inating densification for efficient convergence of 3DGS. In CVPR, 2026
work page 2026
-
[21]
Han,et al.Extrapolated urban view synthesis benchmark
X. Han,et al.Extrapolated urban view synthesis benchmark. InICCV, 2025
work page 2025
-
[22]
S. Hwang, M.-J. Kim, T. Kang, J. Kang, and J. Choo. VEGS: View Extrapolation of Urban Scenes in 3D Gaussian Splat- ting using Learned Priors. InECCV, 2024
work page 2024
-
[23]
SLAM&Render: A Benchmark for the Intersection Between Neural Rendering, Gaussian Splatting and SLAM
S. Cerezo, G. Meli, T. Berriel Martins, K. Safronov, and J. Civera. SLAM&Render: A benchmark for the inter- section of neural rendering and SLAM.arXiv preprint arXiv:2504.13713, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[24]
K. Xiong,et al.ICO-GS: Intrinsic geometry-appearance consistency optimization for sparse-view Gaussian Splatting. arXiv preprint arXiv:2603.02893, 2026
-
[25]
Xu,et al.DropoutGS: Dropping out Gaussians for better sparse-view rendering
X. Xu,et al.DropoutGS: Dropping out Gaussians for better sparse-view rendering. InCVPR, 2025
work page 2025
-
[26]
Xing,et al.3DGS-IEval-15K: A large-scale image quality evaluation database for 3D Gaussian Splatting
Y . Xing,et al.3DGS-IEval-15K: A large-scale image quality evaluation database for 3D Gaussian Splatting. InACM MM, 2025
work page 2025
-
[27]
Geo-EVS: Geometry-Conditioned Extrapolative View Synthesis for Autonomous Driving
Y . Lan,et al.Geo-EVS: Geometry-conditioned extrapola- tive view synthesis for autonomous driving.arXiv preprint arXiv:2604.07250, 2026
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[28]
K. Tan,et al.ExtraGS: Geometric-aware trajectory ex- trapolation with uncertainty-guided generative priors.arXiv preprint arXiv:2508.15529, 2025. 9
-
[29]
Paliwal,et al.CoherentGS: Sparse novel view synthesis with coherent 3D Gaussians
A. Paliwal,et al.CoherentGS: Sparse novel view synthesis with coherent 3D Gaussians. InECCV, 2024
work page 2024
-
[30]
S. Kheradmand, D. Rebain, G. S. Sharma, W. Sun, Y . C. Tseng, H. Isack, A. Kar, A. Tagliasacchi, and K. M. Yi. 3D Gaussian Splatting as Markov chain Monte Carlo. In NeurIPS, 2024
work page 2024
-
[31]
T. Lu, M. Yu, L. Xu, Y . Xiangli, L. Wang, D. Lin, and B. Dai. Scaffold-GS: Structured 3D Gaussians for view- adaptive rendering. InCVPR, 2024
work page 2024
-
[32]
Fang,et al.Mini-Splatting: Representing scenes with a constrained number of Gaussians
G. Fang,et al.Mini-Splatting: Representing scenes with a constrained number of Gaussians. InECCV, 2024
work page 2024
-
[33]
Cheng,et al.GaussianPro: 3D Gaussian Splatting with progressive propagation
K. Cheng,et al.GaussianPro: 3D Gaussian Splatting with progressive propagation. InICML, 2024
work page 2024
-
[34]
Y . Wang,et al.FreeSplat: Generalizable 3D Gaussian Splatting towards free-view synthesis of indoor scenes. In NeurIPS, 2024
work page 2024
-
[35]
D. Charatan, S. Li, A. Tagliasacchi, and V . Sitzmann. pix- elSplat: 3D Gaussian Splats from image pairs for scalable generalizable 3D reconstruction. InCVPR, 2024
work page 2024
-
[36]
Y . Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T. H. Cai, and J. Liang. MVSplat: Efficient 3D Gaussian Splatting from sparse multi-view images. In ECCV, 2024
work page 2024
-
[37]
H. Xu, B. Peng, Z. Cai, H. Zhou, B. Zhuang, and S. Soatto. DepthSplat: Connecting Gaussian Splatting and Depth. In CVPR, 2025
work page 2025
-
[38]
arXiv preprint arXiv:2403.20309 (2024) 2, 3
Z. Fan,et al.InstantSplat: Unbounded sparse-view pose- free Gaussian Splatting in 40 seconds.arXiv preprint arXiv:2403.20309, 2024
-
[39]
Turkulainen,et al.DN-Splatter: Depth and normal priors for Gaussian Splatting and meshing
M. Turkulainen,et al.DN-Splatter: Depth and normal priors for Gaussian Splatting and meshing. InWACV, 2025
work page 2025
- [40]
-
[41]
J. Kulhanek, S. Peng, Z. Kukelova, M. Pollefeys, and T. Sat- tler. WildGaussians: 3D Gaussian Splatting in the wild. In NeurIPS, 2024
work page 2024
-
[42]
Q. Wang, L. Fan, Y . Wang, Y . Chen, and Z. Zhang. FreeVS: Generative view synthesis on free driving trajectory. In ICLR, 2025
work page 2025
-
[43]
W. Xiao, R. Santa Cruz, D. Ahmedt-Aristizabal, O. Salvado, C. Fookes, and L. Lebrat. NeRF Director: Revisiting view selection in neural volume rendering. InCVPR, 2024
work page 2024
-
[44]
arXiv preprint arXiv:2411.14208 (2024)
K. Liu,et al.ViewExtrapolator: Novel view ex- trapolation with video diffusion priors.arXiv preprint arXiv:2411.14208, 2024
-
[45]
M. Shih, W. Ma,et al.ExtraNeRF: Visibility-aware view extrapolation of neural radiance fields with diffusion models. InCVPR, 2024
work page 2024
-
[46]
M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. M. Sajjadi, A. Geiger, and N. Radwan. RegNeRF: Regularizing neu- ral radiance fields for view synthesis from sparse inputs. In CVPR, 2022
work page 2022
-
[47]
H. Park, G. Ryu, and W. Kim. DropGaussian: Structural regularization for sparse-view Gaussian Splatting. InCVPR, 2025
work page 2025
-
[48]
Y . Yun,et al.FASR: Frequency-adaptive sharpness regular- ization for improving 3D Gaussian Splatting generalization. arXiv preprint arXiv:2511.17918, 2025
-
[49]
K. Chen, Y . Zhong, Z. Li, J. Lin, Y . Chen, M. Qin, and H. Wang. Quantifying and alleviating co-adaptation in sparse-view 3D Gaussian Splatting. InNeurIPS, 2025
work page 2025
-
[50]
J. Ke, Q. Wang, Y . Wang, P. Milanfar, and F. Yang. MUSIQ: Multi-scale image quality transformer. InICCV, 2021
work page 2021
-
[51]
L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao. Depth Anything V2. InNeurIPS, 2024
work page 2024
-
[52]
Fang,et al.Dropping Anchor and Spherical Harmonics for Sparse-view Gaussian Splatting
G. Fang,et al.Dropping Anchor and Spherical Harmonics for Sparse-view Gaussian Splatting. InCVPR, 2026
work page 2026
-
[53]
T. Gottwald,et al.PRIMU: Uncertainty estimation for novel views in Gaussian Splatting from primitive-based representations of error and coverage.arXiv preprint arXiv:2508.02443, 2025
-
[54]
J. Jiang,et al.FisherRF: Active view selection and uncer- tainty quantification for radiance fields using Fisher infor- mation. InECCV, 2024
work page 2024
-
[55]
Chen,et al.COVER: Coverage optimization for camera view selection
J. Chen,et al.COVER: Coverage optimization for camera view selection. InCVPR, 2026
work page 2026
-
[56]
T. M ¨uller, A. Evans, C. Schied, and A. Keller. Instant neu- ral graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, 2022
work page 2022
-
[57]
R. Li, M. Tancik, and A. Kanazawa. NerfAcc: A general NeRF acceleration toolbox.arXiv preprint arXiv:2210.04847, 2022. 10 Appendix: Supplementary Material This appendix provides multi-metric results, multi-seed val- idation details, method configurations, and an artifact mani- fest. All results use thefair matched-count protocol(both arms train onN−Kimage...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.