REVIEW 4 major objections 5 minor 30 references
Neural shape reconstruction from multiple views with static pattern projection
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Reconstructing a consistent 3D shape from multiple structured-light views is possible while both the camera and the projector move freely, if the neural surface and all device poses are refined together by differential rendering.
desk verdict A plausible free-motion active-stereo extension of NeuS that is undermined by missing synthetic experiments and inconsistent quantitative reporting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the projector-coordinate function c(p), a 2D affine map from a world point expressed in projector coordinates to projector pixel coordinates. It converts the neural SDF surface into two types of rendered images: coordinate images that directly encode correspondences, and pattern images obtained by a bilinear texture lookup T(c(p)) of the fixed projected pattern. Because c(p) depends on the projector pose and the rendering weights depend on the camera ray and the SDF, gradients of the losses flow to the SDF, the camera poses, and the projector poses simultaneously. The rendering follows NeuS's opacity formulation, and optimization uses a multi-resolution hash encoding feeding a small MLP that outputs scalar SDF values.
What would settle it
Capture a synthetic or real scene of a specular (glossy) sphere under the same static pattern and run the optimization; if the pattern-image loss forces the surface to warp to explain the highlight, the reconstructed sphere will deviate from ground truth. A controlled test would compare the recovered SDF against a known CAD model for materials with progressively higher specularity.
Extended reading notes
Core claim
The central claim is that a consistent 3D shape can be reconstructed from multiple images taken under a static structured-light pattern even while both the camera and the projector move freely, by optimizing a neural SDF, camera poses, and projector poses jointly through volumetric differential rendering. Following the NeuS volume-rendering scheme, the surface is the zero level set of an SDF; unlike NeuS, the color network is replaced by a fixed mapping c(p) that sends a 3D point to its projector coordinate, which is used both to render projector-coordinate images and to look up the projected pattern intensity T(c(p)). The loss combines an L1 term on projector coordinates, a cosine term on pattern images, and an Eikonal regularizer, and backpropagates through the renderer to update the SDF and all poses. The authors report that with this scheme, calibration parameters and inter-frame alignments are refined consistently across frames, yielding a global reconstruction.
Load-bearing premise
The pattern-appearance loss assumes the surface is Lambertian and that the projected pattern's brightness depends only on surface position, so glossy highlights or spatially varying albedo would be read as geometric error.
Editorial extensions
If this is right
- Handheld or freely manipulated camera-projector systems can be used for full-object scanning without a factory or offline rig calibration step.
- The joint pose-and-shape refinement corrects misalignments that per-frame auto-calibration and ICP introduce, so multi-frame integration becomes consistent.
- Because the pattern-image loss is essential when projector-coordinate decoding is incomplete, future active-stereo methods should not rely solely on decoded correspondences.
- The method inherits the sharp-edge softening of SDF representations; the cone vertex example shows very fine geometric detail is partially smoothed.
- All parameters are refined by a differentiable renderer, so the same framework can in principle absorb additional sensor constraints or priors.
Reading between the lines
- Because the pattern appearance is modeled as a view-independent lookup, the method implicitly assumes Lambertian, albedo-uniform surfaces; a natural next test is to scan a glossy or texture-painted object, where the pattern loss would likely be misinterpreted as geometry error.
- The renderer treats the projected pattern purely as a texture; extending it with a small per-point albedo field or a BRDF term would broaden the method to materials with spatially varying reflectance.
- Since each captured frame is decoded independently and only a static SDF is optimized, the approach as presented reconstructs static scenes; making the SDF time-dependent would let the same auto-calibration idea apply to deforming or moving targets under one-shot projection.
- The method could be adapted to other coded-pattern families, because the renderer only requires a coordinate mapping and a pattern texture; the grid-pattern decoder in the paper is one particular front end.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a method for reconstructing a consistent 3D shape from multiple images captured while a camera and a structured-light projector move freely. The core idea is to represent the scene as a neural signed distance field and jointly optimize the SDF, camera poses, and projector poses by volumetric differential rendering of two image modalities: projector-coordinate images and pattern images. The projector-coordinate images come from a U-Net decoder of the captured pattern images, and the pattern images are rendered as a view-independent lookup of the projected pattern. Experiments are reported on two real objects, with an ablation study and a comparison to KinectFusion.
Significance. If validated, the method would address a real usability bottleneck in active-stereo scanning by removing the need for rigid camera-projector calibration and enabling free motion during capture. The differential-rendering formulation is a natural extension of NeuS to structured-light input and could be impactful for endoscopy and industrial inspection. However, the current manuscript provides weak and partly inconsistent empirical support, omits synthetic experiments that the abstract promises, and relies on an unstated Lambertian appearance assumption without testing its limits.
major comments (4)
- [Abstract and Section 4] The abstract states that the method is evaluated by performing 3D reconstruction using both synthetic and real images, but Section 4 contains only real-object experiments (cone-and-cylinder and head). No synthetic dataset, synthetic rendering, or synthetic results appear anywhere in the manuscript. This discrepancy is load-bearing because the synthetic experiments are claimed as part of the validation; either add synthetic experiments with known ground truth or revise the abstract and the contribution statement to describe only real-object validation.
- [Section 4.2 and Table 1] The reported accuracy numbers are internally inconsistent. Section 4.2 says the proposed method achieved an RMSE of 0.619 mm compared to 0.97 mm by KinectFusion, while Table 1 reports an ICP error of 5.94 mm for the full method ('all'). If both numbers measure the same reconstruction error relative to ground truth, they are contradictory by an order of magnitude. If they measure different quantities (e.g., RMSE over a region vs. global ICP distance, or distance to KinectFusion vs. to a different GT), the definitions are not given. The manuscript must define each metric precisely and reconcile the headline 0.619 mm value with the Table 1 value.
- [Eq. (7) and Eq. (8)] The pattern rendering in Eq. (7) treats the observed pattern image as a view-independent bilinear lookup T(c(p)) of the projected pattern, and the pattern loss in Eq. (8) compares this to a locally normalized observed image. This implicitly assumes that the surface is Lambertian with uniform albedo and that projector visibility, foreshortening, and camera response do not affect pattern appearance. For shiny or spatially varying surfaces, the optimizer would interpret appearance changes as geometric error. The paper neither states nor tests this assumption; the two demonstration objects appear matte and relatively uniform. Since the pattern loss is one of only two data terms, this unmodeled physical effect is central to the claimed generality of free-motion active stereo. Please either add an appearance model or explicitly restrict the claims and validate on objects with non-Lambertian or textured surfaces.
- [Section 4.3 and Table 1] The ablation study is described in text and summarized in Table 1, but the table's column headers are not defined in the caption or body. The reader cannot determine which column corresponds to which ablation configuration ('w/o pat loss', 'w/o pose opt.', 'w/o scale'). The text states that without pattern loss the shape became 'much worse', but the absolute magnitude of the effect is not clearly tied to the table entries. Please provide a properly labeled table with explicit configuration definitions so that the contribution of each component is quantitatively interpretable.
minor comments (5)
- [Section 4.1] The text says 'In each iteration, 2024 pixels were sampled'; this is almost certainly a typo for '2048 pixels' (or a non-standard batch size that should be explained).
- [Section 4.2] The phrase 'treated as ground truth' for KinectFusion is confusing: the same sentence reports RMSE against KinectFusion, but a comparison method cannot simultaneously be ground truth and a baseline. Clarify whether the RMSE is computed against a separate high-precision scan or against the KinectFusion result.
- [Section 3.4 and 4.1] The sigmoid scale parameter s and its annealing schedule are not described. Section 4.3 mentions 'decreasing scaling parameters while optimization' but gives no schedule; this is a free parameter that significantly affects NeuS-style optimization and should be specified for reproducibility.
- [Throughout] The terms 'NeuralSDF' and 'NeuS' are used interchangeably; please use one consistent name and define the relationship explicitly.
- [Related Work and Experiments] Several closely related methods (ActiveNeus [10], Qiao et al. [20], Shandilya et al. [22]) are cited but never quantitatively compared. Given that the paper's central claim is about the advantage of joint pose optimization with both correspondence and pattern losses, a comparison to at least one of these would strengthen the evaluation.
Circularity Check
No significant circularity: the optimization targets are independently observed/decoded inputs, and the fitted SDF and poses are not used to define them.
full rationale
The paper's central claim is that jointly optimizing a neural SDF and camera/projector poses against (i) U-Net-decoded projector-coordinate images and (ii) observed normalized pattern images yields a consistent reconstruction. Neither target is defined in terms of the fitted variables. The projector-coordinate targets are produced by the external decoder of Furukawa et al. [7] from captured images; the pattern targets are observed images after local intensity normalization. The rendered quantities are functions of the SDF, the poses, and the known projected pattern via Eqs. (5)-(7), and the loss in Eq. (8) compares these rendered quantities to the fixed targets. This is a forward-model consistency objective, not a fit-then-predict loop. The pattern appearance model in Eq. (7) assumes a view-independent lookup T(c(p)) and ignores BRDF, albedo, and shadowing, but that is an unmodeled physical assumption and a correctness risk, not a circular step; it does not make the target equal to the rendered output by construction. The paper relies on several self-authored prior works, but they supply inputs or baselines (correspondence decoding, active-stereo SDF variants, differential-rendering auto-calibration) rather than the final claim. In particular, [7] is load-bearing as the source of dense correspondences, but those correspondences are independent observed/decoded data external to the optimized SDF and poses; the citation is real evidence of an existing method, not an unverified uniqueness theorem invoked to forbid alternatives. Therefore, no equation reduces to its own input, and no fitted parameter is renamed as a prediction.
Assumptions & free parameters
free parameters (3)
- Loss weights wc, wp, we =
1000, 0.05, 0.01
- Sigmoid scale s and annealing schedule =
not reported
- Neural network architecture hyperparameters =
HashGrid 2^4..2^8, 40-D feature, MLP 128x2
assumptions (4)
- standard math NeuS volume rendering equations (Eq. 5-6) accurately approximate the surface integral
- domain assumption Camera and projector intrinsics, including alpha and beta in Eq. (2), are known and fixed
- domain assumption Observed pattern brightness depends only on projector coordinate, a view-independent Lambertian-like appearance
- domain assumption U-Net predicted projector-coordinate images are accurate enough to initialize and supervise optimization
Cite this review
Pith. "Pith review of Neural shape reconstruction from multiple views with static pattern projection." pith.science (2026). https://pith.science/paper/H7LELB7T
@misc{pith2026250601389,
author = {Pith},
title = {Pith review of: Neural shape reconstruction from multiple views with static pattern projection},
year = {2026},
howpublished = {\url{https://pith.science/paper/H7LELB7T}},
note = {Machine review of arXiv:2506.01389}
}
read the original abstract
Active-stereo-based 3D shape measurement is crucial for various purposes, such as industrial inspection, reverse engineering, and medical systems, due to its strong ability to accurately acquire the shape of textureless objects. Active stereo systems typically consist of a camera and a pattern projector, tightly fixed to each other, and precise calibration between a camera and a projector is required, which in turn decreases the usability of the system. If a camera and a projector can be freely moved during shape scanning process, it will drastically increase the convenience of the usability of the system. To realize it, we propose a technique to recover the shape of the target object by capturing multiple images while both the camera and the projector are in motion, and their relative poses are auto-calibrated by our neural signed-distance-field (NeuralSDF) using novel volumetric differential rendering technique. In the experiment, the proposed method is evaluated by performing 3D reconstruction using both synthetic and real images.
Figures
Reference graph
Works this paper leans on
-
[1]
A user-friendly method to geometrically calibrate projector-camera systems
Samuel Audet and Masatoshi Okutomi. A user-friendly method to geometrically calibrate projector-camera systems. InComputer Vision and Pattern Recognition Workshops,
-
[2]
Method for registration of 3-d shapes
Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. InRobotics-DL tentative, pages 586–606. Inter- national Society for Optics and Photonics, 1992
work page 1992
-
[3]
Sofien Bouaziz, Andrea Tagliasacchi, and Mark Pauly. Sparse iterative closest point.Computer Graphics Forum (Symposium on Geometry Processing), 32(5):1–11, 2013
work page 2013
-
[4]
Long Chen, Wen Tang, Nigel W John, Tao Ruan Wan, and Jian Jun Zhang. Slam-based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality.Computer methods and programs in biomedicine, 158:135–146, 2018
work page 2018
-
[5]
Geometric video projector auto-calibration
Jamil Drar ´eni, S ´ebastien Roy, and Peter Sturm. Geometric video projector auto-calibration. InCVPR Workshops, pages 39–46. IEEE, 2009
work page 2009
-
[6]
Ryo Furukawa, Masaki Mizomori, Shinsaku Hiura, Shiro Oka, Shinji Tanaka, and Hiroshi Kawasaki. Wide-area shape reconstruction by 3d endoscopic system based on cnn decod- ing, shape registration and fusion. InOR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis, pages 139–150...
work page 2018
-
[7]
Ryo Furukawa, Shiro Oka, Takahiro Kotachi, Yuki Okamoto, Shinji Tanaka, Ryusuke Sagawa, and Hiroshi Kawasaki. Fully auto-calibrated active-stereo-based 3d endoscopic sys- tem using correspondence estimation with graph convolu- tional network. InEMBC, pages 4357–4360. IEEE, 2020
work page 2020
-
[8]
Single and multi-frame auto- calibration for 3d endoscopywith differential rendering
Ryo Furukawa, Ryusuke Sagawa, Shiro Oka, Shinji Tanaka, and Hiroshi Kawasaki. Single and multi-frame auto- calibration for 3d endoscopywith differential rendering. InInternational Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2023
work page 2023
Show all 30 references
-
[9]
Implicit geometric regularization for learning shapes.arXiv preprint arXiv:2002.10099, 2020
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes.arXiv preprint arXiv:2002.10099, 2020
2002 arXiv
-
[10]
Activeneus: Neural signed distance fields for active stereo
Ichimaru Kazuto, Takaki Ikeda, Diego Thomas, Takafumi Iwaguchi, and Hiroshi Kawasaki. Activeneus: Neural signed distance fields for active stereo. InInternational Conference on 3D Vision (3DV), 2024
2024
-
[11]
Defslam: Tracking and mapping of deforming scenes from monocular sequences.IEEE Transactions on robotics, 37(1):291–303, 2020
Jose Lamarca, Shaifali Parashar, Adrien Bartoli, and JMM Montiel. Defslam: Tracking and mapping of deforming scenes from monocular sequences.IEEE Transactions on robotics, 37(1):291–303, 2020
2020
-
[12]
Multi-view neural surface reconstruction with structured light
Chunyu Li, Taisuke Hashimoto, Eiichi Matsumoto, and Hi- roharu Kato. Multi-view neural surface reconstruction with structured light. InThe British Machine Vision Conference (BMVC), 2022
2022
-
[13]
A calibration method for un- coupling projector and camera of a structured light system
Jiarui Liao and Lilong Cai. A calibration method for un- coupling projector and camera of a structured light system. In2008 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pages 770–774. IEEE, 2008
2008
-
[14]
Tissue surface reconstruction aided by local normal in- formation using a self-calibrated endoscopic structured light system
Jianyu Lin, Neil T Clancy, Danail Stoyanov, and Daniel S El- son. Tissue surface reconstruction aided by local normal in- formation using a self-calibrated endoscopic structured light system. InInternational Conference on Medical Image Com- puting and Computer-Assisted Interve...
2015
-
[15]
Live tracking and dense reconstruction for handheld monocular endoscopy.IEEE transactions on medical imag- ing, 38(1):79–89, 2018
Nader Mahmoud, Toby Collins, Alexandre Hostettler, Luc Soler, Christophe Doignon, and Jose Maria Martinez Mon- tiel. Live tracking and dense reconstruction for handheld monocular endoscopy.IEEE transactions on medical imag- ing, 38(1):79–89, 2018
2018
-
[16]
Nerf: Representing scenes as neural radiance fields for view syn- thesis.Communications of the ACM, 65(1):99–106, 2021
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis.Communications of the ACM, 65(1):99–106, 2021
2021
-
[17]
tiny-cuda-nn, 4 2021
Thomas M ¨uller. tiny-cuda-nn, 4 2021
2021
-
[18]
Newcombe, Andrew J
Richard A. Newcombe, Andrew J. Davison, Shahram Izadi, Pushmeet Kohli, Otmar Hilliges, Jamie Shotton, David Molyneaux, Steve Hodges, David Kim, and Andrew Fitzgib- bon. KinectFusion: Real-time dense surface mapping and tracking. InIEEEISMAR, pages 127–136, 2011
2011
-
[19]
Dtam: Dense tracking and mapping in real-time
Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. Dtam: Dense tracking and mapping in real-time. In2011 international conference on computer vision, pages 2320–2327. IEEE, 2011
2011
-
[20]
Depth reconstruction with neural signed distance fields in struc- tured lightsystems
Rukun Qiao, Hiroshi Kawasaki, and Hongbin Zha. Depth reconstruction with neural signed distance fields in struc- tured lightsystems. InInternational Conference on 3D Vision (3DV), 2024
2024
-
[21]
Salas-Moreno, Richard A
Renato F. Salas-Moreno, Richard A. Newcombe, Hauke Strasdat, Paul H.J. Kelly, and Andrew J. Davison. Slam++: Simultaneous localisation and mapping at the level of ob- jects. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013
2013
-
[22]
Neural fields for structured lighting
Aarrushi Shandilya, Benjamin Attal, Christian Richardt, James Tompkin, and Matthew O’Toole. Neural fields for structured lighting. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023
2023
-
[23]
Jingwei Song, Jun Wang, Liang Zhao, Shoudong Huang, and Gamini Dissanayake. Mis-slam: Real-time large-scale dense deformable slam system in minimal invasive surgery based on heterogeneous computing.IEEE Robotics and Automa- tion Letters, 3(4):4068–4075, 2018
2018
-
[24]
Nerfstudio: A modular framework for neural radiance field development
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Nerfstudio: A modular framework for neural radiance field development. InACM SIGGRAPH 2023 Conference Proceedings, pages 1–12, 2023
2023
-
[25]
Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021
2021
-
[26]
Flexible cali- bration of a portable structured light system through surface plane.Acta Automatica Sinica, 34(11):1358–1362, 2008
GAO Wei, W ANG Liang, and HU Zhan-Yi. Flexible cali- bration of a portable structured light system through surface plane.Acta Automatica Sinica, 34(11):1358–1362, 2008
2008
-
[27]
Calibra- tion of a structured light system by observing planar object from unknown viewpoints
Koichiro Yamauchi, Hideo Saito, and Yukio Sato. Calibra- tion of a structured light system by observing planar object from unknown viewpoints. InICPR, pages 1–4. IEEE, 2008
2008
-
[28]
Go-icp: Solv- ing 3d registration efficiently and globally optimally
Jiaolong Yang, Hongdong Li, and Yunde Jia. Go-icp: Solv- ing 3d registration efficiently and globally optimally. InThe IEEE International Conference on Computer Vision (ICCV), December 2013
2013
-
[29]
Emdq-slam: Real- time high-resolution reconstruction of soft tissue surface from stereo laparoscopy videos
Haoyin Zhou and Jagadeesan Jayender. Emdq-slam: Real- time high-resolution reconstruction of soft tissue surface from stereo laparoscopy videos. InMICCAI, pages 331–340. Springer, 2021
2021
-
[2009]
IEEE Computer Society Con- ference on, pages 47–54
CVPR Workshops 2009. IEEE Computer Society Con- ference on, pages 47–54. IEEE, 2009
2009
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.