REVIEW 4 major objections 6 minor 32 references
LeC$^2$O-NeRF: Learning Continuous and Compact Large-Scale Occupancy for Urban Scenes
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A 0.15M-parameter occupancy network can replace the discrete occupancy grid for large-scale NeRF, speeding up training while improving PSNR, SSIM, and LPIPS.
desk verdict Credible engineering result with a real compute-accounting gap: report GPU-hours and add error bars before this is ready. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
LeC$^2$O-NeRF's central object is the occupancy network: a 4-layer MLP that returns normalized occupancy scores over $n$ scene sub-networks and one empty-space network, acting as a continuous, trainable replacement for the discrete occupancy grid. It is driven by three losses: the imbalanced occupancy loss $L_o = (n+v)\left(\frac{f_e p_e}{v} + \sum_{i=1}^n f_i p_i\right)$, which generalizes the balanced load loss of GShard by treating the empty-space network as $v$ virtual experts so that roughly $v/(n+v)$ of points are dispatched as unoccupied; the density loss $L_d = \sigma_e/\sigma_s$, computed with detached densities, which forces the mean density of empty-space points below that of scene points; and the standard volume-rendering loss. Together these let the MLP learn occupancy from the scene's own radiance field without external geometry supervision.
What would settle it
Train the same architecture on one large-scale scene but swap the density ratio $L_d$ for its inverse, so the loss prefers empty-space points to have higher density than scene points; if the occupancy network still converges to a useful surface-near binary occupancy, then the density loss is not the mechanism driving the dispatch, and the paper's explanation would need revision. A second check: evaluate occupancy accuracy against independent geometry from LiDAR or COLMAP instead of the depth maps of a fully-trained Switch-NeRF.
Extended reading notes
Core claim
The central claim is that occupancy for large-scale NeRF can be encoded as a continuous function learned by a compact MLP, rather than maintained as a discrete momentum-density grid. The occupancy network $O(x)$ outputs $n+1$ normalized values that dispatch each 3D point into one of $n$ scene sub-networks (occupied) or a tiny empty-space network (unoccupied). It is trained end-to-end with the radiance field using the rendering loss plus an imbalanced occupancy loss that controls the occupied/unoccupied ratio and a density loss $L_d = \sigma_e/\sigma_s$ that pushes unoccupied points to smaller density. The paper reports that this 0.15M-parameter network classifies occupancy more accurately than a 2.0M-parameter $128^3$ grid, retains only about 13–16% of points as occupied, and when frozen and used as a sampler it improves the training of Instant-NGP and Switch-NeRF at matched training time, including the occupancy-training time itself.
Load-bearing premise
The occupancy network trusts the density predictions of the very radiance field it is helping to train: if the density field is initially poor, the density loss can reinforce wrong routing decisions instead of correcting them.
Editorial extensions
If this is right
- Existing grid-based accelerators, including the NerfAcc occupancy grid and Instant-NGP's multi-scale grids, can be replaced by a frozen 0.15M-parameter MLP that needs no per-scene grid-resolution tuning.
- Occupancy converges quickly (within about 10k–40k training steps), so the network can be frozen early and then used purely as a sampler to accelerate the remaining NeRF training.
- At aligned wall-clock training time, LeC2O-guided Instant-NGP and Switch-NeRF attain higher PSNR, SSIM, and lower LPIPS than the same methods with an occupancy grid, meaning the speed-up does not sacrifice accuracy.
- Because the occupancy function is continuous, it represents irregular large-scale urban scenes without the bounding-box and resolution constraints of grids.
Reading between the lines
- If the same occupancy network can be pretrained on a city and then transferred to a new scene, the per-scene 1.6–1.8h occupancy training could be skipped entirely, making the acceleration nearly free at inference time.
- The trick of treating empty space as $v$ virtual experts in the load-balancing loss could transfer to other sparse 3D representations, such as pruning Gaussians in splatting or sampling in robotic occupancy mapping, wherever most of the volume is empty.
- The density ratio $L_d$ provides a parameter-free, supervision-free diagnostic for how well an occupancy predictor separates surface from free space, potentially useful in settings where ground-truth occupancy is unavailable.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes LeC2O-NeRF, a method for learning a continuous, compact occupancy field as a small MLP (0.15M parameters) for large-scale NeRF. The occupancy network dispatches each 3D point either to one of n scene sub-networks or to a tiny empty-space network; the model is trained end-to-end with a rendering loss plus an imbalanced occupancy loss (Eq. 1) and a density loss (Eq. 2) that uses detached densities. After pretraining, the frozen occupancy network is used to discard unoccupied points during point sampling in Instant-NGP and Switch-NeRF. Experiments on Mega-NeRF and Block-NeRF report occupancy classification metrics against a depth-derived ground truth and image reconstruction metrics with wall-clock training time aligned between methods.
Significance. If the claims hold, the paper provides a practical alternative to occupancy grids for large-scale NeRF: a compact continuous predictor that retains roughly 13--16% of points, improves recall substantially over the grid, and improves PSNR/SSIM/LPIPS on most scenes under wall-clock alignment. The core derivation of the imbalanced loss is sound, the ablations are informative, and the paper gives useful occupancy diagnostics (point proportions, density/alpha ratios, memory usage). The main limitation is that the speed claim is made on wall-clock time where GPU counts differ between occupancy pretraining and target NeRF training; until total GPU-hours are reported, the 'speed up without sacrificing accuracy' conclusion is not established. The occupancy ground-truth proxy is generated by the same model family used to train the occupancy network, which weakens the absolute accuracy claims.
major comments (4)
- [§4.3, §4.4, Table 3] The aligned-time comparison does not control for total GPU compute. Occupancy pretraining is reported as 1.6--1.8 h in §4.3 without specifying the number of GPUs, while §4.4 states that INGP is trained on 2 RTX 3090 GPUs and Switch-NeRF on 8. Since the occupancy network is fitted with 8 scene sub-networks and then reused for both target methods, it is plausible that pretraining uses 8 GPUs; under that assumption INGP+Ours costs roughly 8×1.7 + 2×(20−1.7) ≈ 50.2 GPU-hours versus 40 GPU-hours for vanilla INGP. The reported wall-clock tie therefore does not imply a compute tie. Please report GPU-hours for every configuration (including occupancy pretraining) or run all comparisons on identical GPU counts, and state the alignment protocol explicitly. Without this, the central claim that the method can speed up state-of-the-art NeRF methods without sacrificing accuracy is not supported.
- [§4.2, Table 1] The occupancy accuracy ground truth is derived from depth maps of a fully trained Switch-NeRF, the same model family that the occupancy network is trained with. A learned occupancy model trained jointly with a Switch-NeRF-like radiance field may be systematically favored by this reference, so the reported Accuracy/Precision/Recall/F1 improvements over the grid are not an independent measure of geometric quality. Please corroborate the occupancy evaluation with an external depth or LiDAR source, or at least report how the metrics change with a different reference model, and state this limitation explicitly in the paper.
- [§3.3, Eq. (2)] The density loss supervises occupancy with the model's own density predictions, detached, and this signal is assumed to be reliable. Early in training, when the radiance field is poor, these densities may be unreliable and the loss could reinforce routing errors rather than correct them. The paper should include an experiment in which Ld is applied only after a warm-up period or in which the detached densities come from a separate frozen reference, and should report the ratio σe/σs at very early steps; the current Figure 8d starts at 1k steps and does not show the noisiest regime.
- [Table 3, Residence row] The statement in §4.4 that the method consistently outperforms Switch, NGP, and Switch+Grid is contradicted on the Residence scene, where Switch+Ours has PSNR 22.10 versus 22.18 for Switch+Grid, even though SSIM and LPIPS are better. No error bars or multiple seeds are reported anywhere in the tables. Please report standard deviations over at least three seeds and either soften the consistency claim or identify the metric (e.g., LPIPS or an aggregate) on which consistency is claimed.
minor comments (6)
- [§3.3] The sentence 'Our empty space network O should secure more 3D points' should refer to the occupancy network, not the empty space network; the empty space network is the one whose fraction should increase.
- [§3.3, Eq. (2)] The ratio σe/σs can have a near-zero denominator during early training; add a small positive constant to the denominator for numerical stability and state this in the text.
- [§4.3] The sampling counts are ambiguous: the text says '512 points for each ray' for occupancy training and later 'usually sample 128 samples along a ray' for guided training; clarify which stage each number applies to.
- [Table 1] The 'Para. Number' row compares only the occupancy representation parameters; state whether the scene and empty sub-network parameters are excluded, and give the grid resolution (e.g., 128^3 cells) in the caption for context.
- [§4.4] There is a typo in 'Instan-NGP' in the paragraph describing Table 3; it should be 'Instant-NGP'.
- [Figure 5] The caption label 'GridOurs' should be written 'Grid / Ours' to avoid confusion between the two columns.
Circularity Check
No significant circularity: the rendering benchmarks are independent of the occupancy network's fitted parameters, and the occupancy ratio is an explicitly stated design target, not a hidden prediction.
full rationale
The paper's central claims are validated with held-out novel-view metrics (PSNR/SSIM/LPIPS) on Instant-NGP and Switch-NeRF, both of which are trained after the occupancy network is frozen and are not used to fit the occupancy network. Those external benchmarks make the main speed/accuracy claim independently testable. The occupancy-ratio numbers in Table 1 are indeed a direct consequence of the imbalanced occupancy loss: with n=8 and v=80, the loss targets an empty-space share of v/(n+v), about 91%, and the paper says so explicitly in Section 3.3: 'These values make the occupancy network dispatch about 85% points to the empty space network.' Reporting the resulting 13-16% occupied ratio is therefore reporting a designed constraint, not a prediction derived from hidden fitted inputs. The nontrivial finding is that render quality improves at this enforced sparsity, which is checked against held-out images and against the occupancy grid at matched time. The density loss Ld = sigma_e / sigma_s uses the same network's detached density outputs as a soft training signal; this is a self-supervised objective and can pose a training-stability risk if early densities are poor, but it is not a circular derivation because no reported result is defined as the minimizer of that loss. The occupancy-accuracy evaluation uses depth maps from a fully-trained Switch-NeRF as a geometric reference; this is a self-citation by the same authors, but the reference model was trained independently of the proposed occupancy network and is not used to fit the occupancy parameters, so it functions as an external benchmark rather than as an input-output identity. The skeptic's GPU-hour alignment concern is a legitimate fairness/correctness question about whether wall-clock comparisons hide extra parallel compute, but it is not a circularity of the derivation chain. No equation in the paper sets a reported result equal to its own input by construction, and no load-bearing premise is justified solely by an unverified self-citation.
Assumptions & free parameters
free parameters (4)
- virtual empty sub-network count v =
80 for Mega-NeRF, 40 for Block-NeRF
- scene sub-network count n =
8
- loss weights wo, wd =
wo=0.0005, wd=0.1 (Mega-NeRF), wd=0.005 (Block-NeRF)
- occupancy network width =
256
assumptions (4)
- domain assumption Most 3D points in a large-scale scene are unoccupied (sparsity prior).
- domain assumption Density of unoccupied points is much smaller than that of occupied points.
- standard math The balanced loss from GShard (Lepikhin et al., 2021) transfers to the occupancy-dispatch setting.
- domain assumption Depth maps predicted by a fully-trained Switch-NeRF provide a good reference for ground-truth occupancy.
invented entities (2)
-
Occupancy network O (4-layer MLP)
-
Empty space network Ee (identity layer + prediction head)
Cite this review
Pith. "Pith review of LeC$^2$O-NeRF: Learning Continuous and Compact Large-Scale Occupancy for Urban Scenes." pith.science (2026). https://pith.science/paper/V6RNMYQ5
@misc{pith2026241111374,
author = {Pith},
title = {Pith review of: LeC$^2$O-NeRF: Learning Continuous and Compact Large-Scale Occupancy for Urban Scenes},
year = {2026},
howpublished = {\url{https://pith.science/paper/V6RNMYQ5}},
note = {Machine review of arXiv:2411.11374}
}
read the original abstract
In NeRF, a critical problem is to effectively estimate the occupancy to guide empty-space skipping and point sampling. Grid-based methods work well for small-scale scenes. However, on large-scale scenes, they are limited by predefined bounding boxes, grid resolutions, and high memory usage for grid updates, and thus struggle to speed up training for large-scale, irregularly bounded and complex urban scenes without sacrificing accuracy. In this paper, we propose to learn a continuous and compact large-scale occupancy network, which can classify 3D points as occupied or unoccupied points. We train this occupancy network end-to-end together with the radiance field in a self-supervised manner by three designs. First, we propose a novel imbalanced occupancy loss to regularize the occupancy network. It makes the occupancy network effectively control the ratio of unoccupied and occupied points, motivated by the prior that most of 3D scene points are unoccupied. Second, we design an imbalanced architecture containing a large scene network and a small empty space network to separately encode occupied and unoccupied points classified by the occupancy network. This imbalanced structure can effectively model the imbalanced nature of occupied and unoccupied regions. Third, we design an explicit density loss to guide the occupancy network, making the density of unoccupied points smaller. As far as we know, we are the first to learn a continuous and compact occupancy of large-scale NeRF by a network. In our experiments, our occupancy network can quickly learn more compact, accurate and smooth occupancy compared to the occupancy grid. With our learned occupancy as guidance for empty space skipping on challenging large-scale benchmarks, our method consistently obtains higher accuracy compared to the occupancy grid, and our method can speed up state-of-the-art NeRF methods without sacrificing accuracy.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021
2021
-
[3]
Barron, Ben Mildenhall, Dor Verbin, Pratul P
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022
2022
-
[4]
Chan, Connor Z
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022
2022
-
[5]
Tensorf: Tensorial radiance fields
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022
2022
-
[6]
Plenoxels: Radiance fields without neural networks
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 5501--5510, June 2022
work page 2022
-
[7]
Efficientnerf efficient neural radiance fields
Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientnerf efficient neural radiance fields. In CVPR, 2022
work page 2022
-
[8]
Motion-oriented compositional neural radiance fields for monocular dynamic human modeling
Kim Jaehyeok, Wee Dongyoon, and Dan Xu. Motion-oriented compositional neural radiance fields for monocular dynamic human modeling. In ECCV, 2024
work page 2024
Show all 32 references
-
[9]
3d gaussian splatting for real-time radiance field rendering
Bernhard Kerbl, Georgios Kopanas, Thomas Leimk \"u hler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42 0 (4), July 2023. URL https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
2023
-
[10]
Gshard: Scaling giant models with conditional computation and automatic sharding
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In ICLR, 2021
2021
-
[11]
Nerfacc: A general nerf accleration toolbox
Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: A general nerf accleration toolbox. arXiv preprint arXiv:2210.04847, 2022
2022 arXiv
-
[12]
Neural sparse voxel fields
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurIPS, 2020
2020
-
[13]
Nerf in the wild: Neural radiance fields for unconstrained photo collections
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021
2021
-
[14]
Switch-ne RF : Learning scene decomposition with mixture of experts for large-scale neural radiance fields
Zhenxing MI and Dan Xu. Switch-ne RF : Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=PQ2zoIZqvm
2023
-
[15]
Srinivasan, Matthew Tancik, Jonathan T
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020
2020
-
[16]
Instant neural graphics primitives with a multiresolution hash encoding
Thomas M\"uller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41 0 (4): 0 102:1--102:15, 2022
2022
-
[17]
Implicit event-rgbd neural slam
Delin Qu, Chi Yan, Dong Wang, Jie Yin, Qizhi Chen, Dan Xu, Yiting Zhang, Bin Zhao, and Xuelong Li. Implicit event-rgbd neural slam. In CVPR, 2024
2024
-
[18]
Srinivasan, Jonathan T
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022
2022
-
[19]
Sign-agnostic conet: Learning implicit surface reconstructions by sign-agnostic optimization of convolutional occupancy networks
Jiapeng Tang, Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang. Sign-agnostic conet: Learning implicit surface reconstructions by sign-agnostic optimization of convolutional occupancy networks. In ICCV, 2021
2021
-
[20]
Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In CVPR, 2022
2022
-
[21]
Learning unified decompositional and compositional nerf for editable novel view synthesis
Yuxin Wang, Wayne Wu, and Dan Xu. Learning unified decompositional and compositional nerf for editable novel view synthesis. In ICCV, 2023
2023
-
[22]
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 2004
2004
-
[23]
Pygs: Large-scale scene representation with pyramidal 3d gaussian splatting
Zipeng Wang and Dan Xu. Pygs: Large-scale scene representation with pyramidal 3d gaussian splatting. arXiv preprint arXiv:2405.16829, 2024
2024 arXiv
-
[24]
Grid-guided neural radiance fields for large urban scenes
Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin. Grid-guided neural radiance fields for large urban scenes. In CVPR, 2023
2023
-
[25]
Point-nerf: Point-based neural radiance fields
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5438--5448, 2022
2022
-
[26]
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018
2018
-
[27]
Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision
Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. CVPR, 2023 a
2023
-
[28]
Efficient large-scale scene representation with a hybrid of high-resolution grid and plane features
Yuqi Zhang, Guanying Chen, and Shuguang Cui. Efficient large-scale scene representation with a hybrid of high-resolution grid and plane features. arXiv preprint arXiv:2303.03003, 2023 b
2023 arXiv
-
[29]
Cvt-xrf: Contrastive in-voxel transformer for 3d consistent radiance fields from sparse inputs
Yingji Zhong, Lanqing Hong, Zhenguo Li, and Dan Xu. Cvt-xrf: Contrastive in-voxel transformer for 3d consistent radiance fields from sparse inputs. In CVPR, 2024
2024
-
[30]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[31]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[32]
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.