REVIEW 3 major objections 6 minor 1 cited by
PRM: Photometric Stereo based Large Reconstruction Model
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read PRM shows that training a large reconstruction model on photometric stereo images—with varied materials and lighting—yields meshes with finer local detail and strong robustness to glossy or dark input appearances.
desk verdict Solid LRM variant with a real renderer-transfer problem: the photometric-stereo gains are large but measured inside the paper's own split-sum pipeline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the photometric stereo image, produced online during training by combining mesh rasterization with the split-sum approximation to the rendering equation. The split-sum approximation rewrites the specular term as a precomputed BRDF integral times a mipmap-sampled lighting term, so the model can render many material/lighting combinations in real time; the mesh representation (FlexiCubes) makes this rendering differentiable and yields albedo, normal, depth, specular-light, and diffuse-light maps. These maps give extra supervision terms—especially the lighting maps, which are functions of the surface normal alone—so the network is solving the photometric stereo system of equations rather than just predicting colors.
What would settle it
Render a held-out set of objects with both the split-sum pipeline and a full Monte Carlo path tracer, then run PRM on the path-traced images: if Chamfer Distance or F-Score degrades markedly relative to split-sum renderings, the photometric cues are partly renderer artifacts. Alternatively, evaluate PRM on calibrated real captures with known surface geometry, where input appearance includes uncontrolled specularities and shadows, and compare error against the synthetic-test numbers reported in the paper.
Extended reading notes
Core claim
The paper's central claim is that photometric stereo images—rendered by varying camera pose, metallic, roughness, and environment lighting—are a better training distribution for large reconstruction models than the fixed-lighting images used by predecessors. By using these images as both input and supervision, PRM reduces the geometry/texture ambiguity that makes glossy surfaces come out bumpy or wrong, and it recovers fine local relief that albedo-only supervision misses. The authors report that on GSO, Chamfer Distance falls from 0.076 (InstantMesh) to 0.050 and F-Score@0.1 rises from 0.931 to 0.981, with comparable gains on OmniObject3D, and that the model also predicts albedo, allowing relighting and material editing. The mechanism behind the gain is that shading color and the intermediate specular and diffuse lighting maps carry photometric cues that depend only on surface normals, so supervising on them acts like photometric stereo and sharpens the geometry.
Load-bearing premise
The whole photometric-stereo benefit rests on the real-time split-sum approximation used to render both the training inputs and the supervision images: if that rendering shortcut is systematically biased, the photometric cues become renderer-specific artifacts and the robustness gains may not carry over to real photographs.
Editorial extensions
If this is right
- Training on photometric stereo images should make a reconstruction model generalize to input photos with specular highlights, cast shadows, and dark appearances, because the model has seen those shading patterns during training.
- Supervising on specular and diffuse lighting maps, not just final color, directly constrains surface normals and should improve fine local relief on smooth or textureless surfaces.
- Predicting albedo rather than shading color makes the output separable into geometry and material, enabling relighting and material editing as post-hoc operations.
- Rendering training data online with rasterization and split-sum approximation removes the offline rendering bottleneck, so the training distribution can be expanded to varied camera poses, fields of view, and material combinations cheaply.
Reading between the lines
- A testable extension: feeding PRM a set of images of the same object under several lighting conditions should let it output normals and albedo directly, effectively turning the reconstruction model into a fast multiview photometric stereo solver—something the paper does not evaluate.
- Because the split-sum approximation is shared between training and supervision, an internal consistency check (comparing PRM's reconstructions when trained with full Monte Carlo PBR as supervision) would tell whether the reported gains come from true photometric cues or from the network exploiting the renderer's shortcut, a comparison the paper does not run.
- The same online rendering pipeline could be used to ablate the value of other appearance variations—camera field-of-view, environment map intensity, or cast shadows—to isolate which photometric cue contributes most to the detail improvement.
- The model's robustness to background clutter is limited by its white-background training; combining photometric stereo rendering with background augmentation is a natural direction the paper mentions only in passing.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PRM, a feed-forward large reconstruction model that reconstructs meshes from single or multi-view images by leveraging photometric stereo images rendered with randomly varied materials and lighting under a real-time split-sum PBR approximation. The model uses FlexiCubes as an explicit mesh representation, enabling differentiable PBR to render shading variables such as albedo, specular/diffuse light maps, normals, and depth for multi-term supervision (Eq. 12). Experiments on GSO and OmniObject3D report significant improvements over prior methods (e.g., GSO CD 0.050 vs 0.076 for InstantMesh; FS@0.1 0.981 vs 0.931; PSNR 25.1 vs 20.0), along with a controlled comparison in Table 2 using ground-truth multi-view inputs. The paper also demonstrates qualitative robustness on real images and supports relighting and material editing applications.
Significance. If the reported gains are reproducible and transfer beyond the paper's own rendering pipeline, PRM is a meaningful advance: it introduces a practical way to inject photometric stereo cues into large reconstruction models, with a real-time split-sum rendering pipeline that enables online data generation, and an explicit mesh representation with differentiable PBR supporting multiple photometric supervisions and downstream relighting/editing. The paper's strengths include consistent quantitative improvements across two datasets, a controlled comparison with ground-truth multi-view input (Table 2), and a systematic ablation study (Table 3). The qualitative results on real images (Figs. 13 and 18) are encouraging. However, the evaluation is currently closed-loop with respect to the rendering approximation, which tempers the strength of the central claims and requires additional validation.
major comments (3)
- [Sec. 3.2 (Eqs. 7-9) and Sec. 4.1] The quantitative evaluation is conducted entirely within the same rendering pipeline used for training. Photometric stereo images, the supervision maps (L_spec, L_diff, normal, etc.), and the test-set ground-truth images are all produced with the split-sum approximation (Eqs. 7-9). Consequently, the reported improvements over prior methods (e.g., CD 0.050 vs 0.076 on GSO) may reflect that PRM learns the systematic errors of this particular approximation rather than physically generalizable photometric cues. The paper does not compare the split-sum output against Monte Carlo path tracing, and real-image results appear only qualitatively (Figs. 13 and 18). To substantiate the claim of robustness to real appearances, I request either a quantitative evaluation on a test set rendered with an independent path tracer, or a real-capture benchmark with geometric metrics, or at minimum an analysis of the split-sum approximation error on representative materials.
- [Sec. 4.1, Evaluation Protocol] The protocol for 2D metrics (PSNR/SSIM/LPIPS) is under-specified. The authors state that novel views are rendered from the reconstructed meshes and compared to ground truth, but they do not state which renderer is used for this step. If PRM's predicted meshes are rendered with the same split-sum PBR as its training (which appears to be the case from Sec. 3.3), while competing methods are rendered with a different standard renderer (e.g., Blender Cycles), the comparison is not fair and the PSNR gains could be inflated. The authors must specify a single common rendering pipeline for all evaluated methods, or otherwise demonstrate that the results are insensitive to the renderer choice.
- [Table 3 (Appendix A.3)] The ablation study is trained on a reduced subset of 10k objects, whereas the full model is trained on a larger filtered Objaverse set (size not disclosed). As a result, the 'Full model' row in Table 3 reports CD 0.066, which is notably worse than the main evaluation's CD 0.050 for PRM. The relative order of ablations is informative, but the quantitative magnitudes do not directly transfer to the full-scale model. Please state this limitation clearly, and if feasible, provide full-scale ablations for at least the most important components (e.g., w/o lighting supervision).
minor comments (6)
- [Sec. 3.1] There is a typo: 'reconstrcut' should be 'reconstruct'.
- [Sec. 3.3] The text says 'mean squaree error loss'; this should be 'mean squared error loss'.
- [Figure 5 caption] The caption uses 'w/o PRB' where it should be 'w/o PBR'.
- [Appendix A.9] The phrase 'while background' appears twice and should be 'white background'.
- [Sec. 3.2, Discussion] The photometric-stereo formulation assumes known lighting, metallic, and roughness per view, but at inference the network only receives RGB images. The paper should clarify that the model does not perform classical photometric stereo at inference; rather, it learns a mapping from multi-view appearance to shape and albedo, with the PS formalism serving as a training-data design principle.
- [Tables 1-3] No error bars or statistical significance tests are reported for any quantitative result. Given the 300-object evaluation set, reporting results over a few random evaluation subsets or training runs would strengthen the claims.
Circularity Check
No circularity: photometric-stereo supervision and external benchmarks keep the evaluation non-tautological.
full rationale
PRM's derivation is a supervised-learning pipeline rather than a closed-form derivation, and none of its reported results reduce to its own inputs by construction. Training renders ground-truth Objaverse meshes with the split-sum PBR approximation (Eqs. 7–9) and optimizes the loss in Eq. (12) against those renderer outputs; this is standard training supervision, not a fitted parameter renamed as a prediction. The quantitative evaluation on GSO and OmniObject3D compares Chamfer Distance and F-Score against external mesh geometry, and the 2D metrics compare rendered novel views to ground-truth views; these benchmarks do not use trainable constants, so the numeric gains over InstantMesh and other baselines are not forced by the model's definition. The photometric-stereo equations (Eqs. 3–6) are classical external results used as motivation, not author-specific uniqueness theorems, and the paper does not invoke any self-citation to forbid alternative designs or to define its representation. The few self-citations (Ref-NeuS, X-ray, FlexGen) appear in related-work or motivation contexts and are not load-bearing premises of the method. The only substantive limitation is that training, supervision, and quantitative evaluation all use the same split-sum renderer, so the appearance-robustness claim is not yet quantitatively validated on Monte Carlo rendering or real captures; real-image results are qualitative. That is an external-validity and correctness risk, not circularity, because external object geometry and independent baseline models still make the benchmark non-tautological.
Assumptions & free parameters
free parameters (2)
- loss weights in Eq. (12) =
lambda_LPIPS=2.0, lambda_normal=0.2, lambda_depth=0.5, lambda_mask=1.0, lambda_reg=0.01
- per-view consistency threshold for materials and lighting =
50 percent probability of changing materials and lighting per view
assumptions (4)
- domain assumption Split-sum approximation gives a sufficiently accurate and differentiable approximation to Cook-Torrance specular reflection.
- domain assumption The sampled Polyhaven environment maps and 121 metallic and roughness combinations cover the appearance variation needed for robustness.
- domain assumption The pretrained multi-view diffusion model produces multi-view images consistent enough for single-image reconstruction.
- domain assumption FlexiCubes mesh extraction and mesh rasterization provide unbiased gradients for geometry optimization.
Cite this review
Pith. "Pith review of PRM: Photometric Stereo based Large Reconstruction Model." pith.science (2026). https://pith.science/paper/YH73F3AI
@misc{pith2026241207371,
author = {Pith},
title = {Pith review of: PRM: Photometric Stereo based Large Reconstruction Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/YH73F3AI}},
note = {Machine review of arXiv:2412.07371}
}
read the original abstract
We propose PRM, a novel photometric stereo based large reconstruction model to reconstruct high-quality meshes with fine-grained local details. Unlike previous large reconstruction models that prepare images under fixed and simple lighting as both input and supervision, PRM renders photometric stereo images by varying materials and lighting for the purposes, which not only improves the precise local details by providing rich photometric cues but also increases the model robustness to variations in the appearance of input images. To offer enhanced flexibility of images rendering, we incorporate a real-time physically-based rendering (PBR) method and mesh rasterization for online images rendering. Moreover, in employing an explicit mesh as our 3D representation, PRM ensures the application of differentiable PBR, which supports the utilization of multiple photometric supervisions and better models the specular color for high-quality geometry optimization. Our PRM leverages photometric stereo images to achieve high-quality reconstructions with fine-grained local details, even amidst sophisticated image appearances. Extensive experiments demonstrate that PRM significantly outperforms other models.
Figures
Figures from the paper (15 more)
Forward citations
Cited by 1 Pith paper
-
A Hybrid Neural-Microfacet BRDF Model for Real-Time Rendering
A hybrid BRDF model, combining a GGX analytical term with a tiny learned residual and gating network, fits measured materials more accurately than fully neural models at equal memory cost.
Reference graph
Works this paper leans on
-
[3]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
-
[5]
Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching
Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint arXiv:2311.11284,
-
[6]
Uncalibrated photo- metric stereo under natural illumination
Zhipeng Mo, Boxin Shi, Feng Lu, Sai-Kit Yeung, and Yasuyuki Matsushita. Uncalibrated photo- metric stereo under natural illumination. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
work page 2018
-
[9]
Score-based generative modeling through stochastic differential equations
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456,
arXiv 2011
-
[10]
Lgm: Large multi-view gaussian model for high-resolution 3d content creation
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054,
-
[14]
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024a. Xinli Xu, Wenhang Ge, Jiantao Lin, Jiawei Feng, Lie Xu, HanFeng Zhao, Shunsi Zhang, and Ying- Cong Chen. Flexgen: Flexible multi-vi...
-
[15]
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.09414,
-
[16]
Clay: A controllable large-scale generative model for creating high-quality 3d assets
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. arXiv preprint arXiv:2406.13897,
Show all 17 references
-
[17]
3.1 we introduce the D, F and G term of the specular component of BRDF property
13 PRM: Photometric Stereo based Large Reconstruction Model A A PPENDIX A.1 BRDF P ARAMETERIZATION In Sec. 3.1 we introduce the D, F and G term of the specular component of BRDF property. We implement the Cook-Torrance BRDF model (Cook & Torrance, 1982). The basic specular alb...
1982
-
[1987]
Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model. arXiv preprint arXiv:2311.06214,
-
[2013]
Dreamfusion: Text-to-3d using 2d diffusion
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988,
-
[2018]
Dual marching cubes
Gregory M Nielson. Dual marching cubes. In IEEE visualization 2004,
2004
-
[2020]
Lrm: Large reconstruction model for single image to 3d
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400,
-
[2021]
Crm: Single image to 3d textured mesh with convolutional reconstruction model
Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.05034,
-
[2022]
Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689,
-
[2023]
Google scanned objects: A high-quality dataset of 3d scanned household items
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA),
2022
-
[2024]
Triposr: Fast 3d object reconstruction from a single image
12 PRM: Photometric Stereo based Large Reconstruction Model Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv p...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.