{"id":"01d5b36e-ec36-4aef-bdfd-9ef43bbd902a","arxiv_id":"2412.07371","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PRM uses photometric-stereo-style rendered images as both input and supervision, with mesh-based differentiable PBR, to reconstruct 3D meshes with finer local details and more robustness to complex appearances.","lead":"A new 3D reconstruction model trains on images rendered under many lighting and material conditions, so it can recover fine surface details and shiny objects from a single photo. It reports large gains over prior large reconstruction models on two standard datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative gains may be confined to the paper's own split-sum renderer: training and evaluation share the same approximate PBR, so the photometric-stereo benefit and appearance robustness are not yet shown to transfer to Monte Carlo or real captures.","rationale":"The reader identifies the same weakest assumption, and I agree that it is the most load-bearing concern. The strongest quantitative claims—34% CD improvement over InstantMesh and near-perfect F-Score on GSO—are measured on renderings produced by the same split-sum approximation used to create training data and to differentiate through the predicted mesh. This shared-renderer setup cannot distinguish 'better photometric reconstruction' from 'better inversion of this particular approximate renderer.' If the split-sum approximation is biased for specular materials or for the high-roughness/low-roughness extremes sampled during training, the model may exploit those biases; robustness to real captures would then be coincidental. The paper's qualitative real-image examples are encouraging but not quantitative and are not compared against baselines. A Monte Carlo re-rendering test is feasible and would settle the issue. Since the concern is unaddressed but addressable, the reader's CONDITIONAL verdict is appropriate; no verdict change is needed.","tokens_in":15111,"tokens_out":6883,"duration_ms":73953,"concrete_test":"Render the 300 GSO and 300 Omni3D evaluation objects with a high-quality Monte Carlo path tracer (e.g., Blender Cycles with at least 512 samples per pixel) using the same random material grid and Polyhaven environment maps as the paper. Feed the path-traced multi-view images to the trained PRM and recompute the Table 1 metrics. If CD increases by more than ~15% relative to the reported 0.050/0.053, or if PRM no longer outperforms InstantMesh under the same MC-rendered inputs, the photometric-stereo benefit and robustness claim are renderer-specific rather than physical.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that photometric stereo images—renderings with randomly varied materials and lighting—provide rich shading cues that improve local geometry and appearance robustness. However, the evidence for this claim is generated in a closed loop. Training renderings (Sec. 3.2) and every quantitative evaluation (Tables 1-3) use the same real-time split-sum approximation (Eqs. 7-9), and the model's own differentiable PBR uses the same approximation. Thus the network is trained to invert a specific approximate renderer and is scored on images from that same renderer; reported gains (e.g., GSO CD 0.050 vs InstantMesh 0.076) may reflect the model learning the approximation's systematic errors (e.g., the GGX pre-filtered specular mipmap L_spec and the roughness-dependent D term) rather than genuine photometric cues. The paper provides no comparison against a Monte Carlo path tracer and no quantitative evaluation on real captures; real-image results are qualitative only (Figs. 13, 18). The photometric-stereo formalism in Sec. 3.2 also assumes known lighting, metallic, and roughness per view, but at inference the network sees only RGB, so the robustness claim rests entirely on the training distribution. The single most load-bearing unvalidated step is therefore the use of split-sum PBR for both data generation and supervision without any check that the resulting shading variations are physically representative.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PRM, a feed-forward large reconstruction model that reconstructs meshes from single or multi-view images by leveraging photometric stereo images rendered with randomly varied materials and lighting under a real-time split-sum PBR approximation. The model uses FlexiCubes as an explicit mesh representation, enabling differentiable PBR to render shading variables such as albedo, specular/diffuse light maps, normals, and depth for multi-term supervision (Eq. 12). Experiments on GSO and OmniObject3D report significant improvements over prior methods (e.g., GSO CD 0.050 vs 0.076 for InstantMesh; FS@0.1 0.981 vs 0.931; PSNR 25.1 vs 20.0), along with a controlled comparison in Table 2 using ground-truth multi-view inputs. The paper also demonstrates qualitative robustness on real images and supports relighting and material editing applications.","tokens_in":15401,"tokens_out":8083,"duration_ms":77807,"significance":"If the reported gains are reproducible and transfer beyond the paper's own rendering pipeline, PRM is a meaningful advance: it introduces a practical way to inject photometric stereo cues into large reconstruction models, with a real-time split-sum rendering pipeline that enables online data generation, and an explicit mesh representation with differentiable PBR supporting multiple photometric supervisions and downstream relighting/editing. The paper's strengths include consistent quantitative improvements across two datasets, a controlled comparison with ground-truth multi-view input (Table 2), and a systematic ablation study (Table 3). The qualitative results on real images (Figs. 13 and 18) are encouraging. However, the evaluation is currently closed-loop with respect to the rendering approximation, which tempers the strength of the central claims and requires additional validation.","major_comments":[{"comment":"The quantitative evaluation is conducted entirely within the same rendering pipeline used for training. Photometric stereo images, the supervision maps (L_spec, L_diff, normal, etc.), and the test-set ground-truth images are all produced with the split-sum approximation (Eqs. 7-9). Consequently, the reported improvements over prior methods (e.g., CD 0.050 vs 0.076 on GSO) may reflect that PRM learns the systematic errors of this particular approximation rather than physically generalizable photometric cues. The paper does not compare the split-sum output against Monte Carlo path tracing, and real-image results appear only qualitatively (Figs. 13 and 18). To substantiate the claim of robustness to real appearances, I request either a quantitative evaluation on a test set rendered with an independent path tracer, or a real-capture benchmark with geometric metrics, or at minimum an analysis of the split-sum approximation error on representative materials.","section":"Sec. 3.2 (Eqs. 7-9) and Sec. 4.1"},{"comment":"The protocol for 2D metrics (PSNR/SSIM/LPIPS) is under-specified. The authors state that novel views are rendered from the reconstructed meshes and compared to ground truth, but they do not state which renderer is used for this step. If PRM's predicted meshes are rendered with the same split-sum PBR as its training (which appears to be the case from Sec. 3.3), while competing methods are rendered with a different standard renderer (e.g., Blender Cycles), the comparison is not fair and the PSNR gains could be inflated. The authors must specify a single common rendering pipeline for all evaluated methods, or otherwise demonstrate that the results are insensitive to the renderer choice.","section":"Sec. 4.1, Evaluation Protocol"},{"comment":"The ablation study is trained on a reduced subset of 10k objects, whereas the full model is trained on a larger filtered Objaverse set (size not disclosed). As a result, the 'Full model' row in Table 3 reports CD 0.066, which is notably worse than the main evaluation's CD 0.050 for PRM. The relative order of ablations is informative, but the quantitative magnitudes do not directly transfer to the full-scale model. Please state this limitation clearly, and if feasible, provide full-scale ablations for at least the most important components (e.g., w/o lighting supervision).","section":"Table 3 (Appendix A.3)"}],"minor_comments":[{"comment":"There is a typo: 'reconstrcut' should be 'reconstruct'.","section":"Sec. 3.1"},{"comment":"The text says 'mean squaree error loss'; this should be 'mean squared error loss'.","section":"Sec. 3.3"},{"comment":"The caption uses 'w/o PRB' where it should be 'w/o PBR'.","section":"Figure 5 caption"},{"comment":"The phrase 'while background' appears twice and should be 'white background'.","section":"Appendix A.9"},{"comment":"The photometric-stereo formulation assumes known lighting, metallic, and roughness per view, but at inference the network only receives RGB images. The paper should clarify that the model does not perform classical photometric stereo at inference; rather, it learns a mapping from multi-view appearance to shape and albedo, with the PS formalism serving as a training-data design principle.","section":"Sec. 3.2, Discussion"},{"comment":"No error bars or statistical significance tests are reported for any quantitative result. Given the 300-object evaluation set, reporting results over a few random evaluation subsets or training runs would strengthen the claims.","section":"Tables 1-3"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the closed-loop evaluation: all quantitative evidence is generated with the same split-sum renderer used in training. The authors should be asked to either evaluate with an independent renderer or real data, or to temper the generalization claims. The paper is otherwise well-executed and within the scope of the conference. If the evaluation gap is addressed, the work could be a strong paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you work on single-image 3D. The paper's trick: train an LRM on multi-view images rendered with varied materials and lighting (photometric stereo style), and supervise with split-sum PBR outputs, including specular and diffuse lighting maps. The reported gains are large and consistent across two datasets: GSO CD drops from 0.076 to 0.050, FS@0.1 goes from 0.931 to 0.981, PSNR from 20.0 to 25.1. The ablations show that the added supervisions and the material variation each matter. That part is credible.\n\nThe architecture is basically InstantMesh with a different data pipeline and loss; the novelty is the integration of online split-sum rendering and photometric-style supervision into an LRM, not a new theoretical idea. That is still a reasonable contribution. The paper is clearly written and the limitations are stated.\n\nThe soft spot is exactly what the stress-test note flags, and I think it lands. Training and evaluation images come from the same split-sum approximation (Eqs. 7–9). There is no Monte Carlo path-traced sanity check, and no quantitative real-capture evaluation. So large parts of the gains could be the model learning the approximation's systematic errors rather than true photometric cues. The authors admit albedo is still entangled with lighting, which is consistent. Also no error bars, code, or weights.\n\nHowever, the geometry metrics compare the predicted mesh against ground-truth meshes that are not rendered, so it is not entirely circular. And the controlled Table 2, where both methods get ground-truth multi-view input, still favors PRM. So I would not call this a deal-breaker, just an unvalidated transfer claim.\n\nMy take: the paper deserves a serious referee. A reviewer should push for a transfer test: retrain or fine-tune on a different renderer, or evaluate on real captures, or at least compare split-sum against full Monte Carlo on a few assets. If that holds, this is a solid empirical contribution. Without it, the head-to-head numbers are only meaningful inside the same renderer. I would accept this for peer review and bring it to reading group, with the caveat to everyone that the evaluation is closed-loop.","headline":"Solid LRM variant with a real renderer-transfer problem: the photometric-stereo gains are large but measured inside the paper's own split-sum pipeline.","tokens_in":15959,"tokens_out":3677,"would_cite":true,"duration_ms":37126,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PRM shows that training a large reconstruction model on photometric stereo images—with varied materials and lighting—yields meshes with finer local detail and strong robustness to glossy or dark input appearances.","keywords":["photometric stereo","large reconstruction model","physically-based rendering","3D mesh reconstruction","albedo prediction","specular robustness","triplane representation","split-sum approximation"],"falsifier":"Render a held-out set of objects with both the split-sum pipeline and a full Monte Carlo path tracer, then run PRM on the path-traced images: if Chamfer Distance or F-Score degrades markedly relative to split-sum renderings, the photometric cues are partly renderer artifacts. Alternatively, evaluate PRM on calibrated real captures with known surface geometry, where input appearance includes uncontrolled specularities and shadows, and compare error against the synthetic-test numbers reported in the paper.","tokens_in":14898,"feed_emoji":"💡","tokens_out":5643,"duration_ms":51279,"temperature":0.7,"pith_summary":"This paper proposes PRM, a feed-forward model that reconstructs a 3D mesh from one or a few images. Its central claim is that if the training images—both the inputs and the supervision—are rendered as photometric stereo images, with randomly varied material and lighting conditions, the model learns finer local surface detail and becomes robust to glossy, dark, or highlight-heavy input appearances. The authors argue that previous large reconstruction models trained with fixed, simple lighting entangle texture and geometry, and that replacing those images with photometric stereo images supplies the shading equations needed to pin down surface normals. If correct, the approach makes single-image 3D reconstruction more accurate on everyday objects with varied appearances, and it enables relighting and material editing from a single input.","feed_headline":"Photometric stereo images sharpen 3D reconstruction by a third","feed_subtitle":"Varying lighting and materials in training yields finer mesh detail and handles glossy inputs better than fixed-light models.","key_machinery":"The load-bearing object is the photometric stereo image, produced online during training by combining mesh rasterization with the split-sum approximation to the rendering equation. The split-sum approximation rewrites the specular term as a precomputed BRDF integral times a mipmap-sampled lighting term, so the model can render many material/lighting combinations in real time; the mesh representation (FlexiCubes) makes this rendering differentiable and yields albedo, normal, depth, specular-light, and diffuse-light maps. These maps give extra supervision terms—especially the lighting maps, which are functions of the surface normal alone—so the network is solving the photometric stereo system of equations rather than just predicting colors.","core_discovery":"The paper's central claim is that photometric stereo images—rendered by varying camera pose, metallic, roughness, and environment lighting—are a better training distribution for large reconstruction models than the fixed-lighting images used by predecessors. By using these images as both input and supervision, PRM reduces the geometry/texture ambiguity that makes glossy surfaces come out bumpy or wrong, and it recovers fine local relief that albedo-only supervision misses. The authors report that on GSO, Chamfer Distance falls from 0.076 (InstantMesh) to 0.050 and F-Score@0.1 rises from 0.931 to 0.981, with comparable gains on OmniObject3D, and that the model also predicts albedo, allowing relighting and material editing. The mechanism behind the gain is that shading color and the intermediate specular and diffuse lighting maps carry photometric cues that depend only on surface normals, so supervising on them acts like photometric stereo and sharpens the geometry.","pith_inferences":["A testable extension: feeding PRM a set of images of the same object under several lighting conditions should let it output normals and albedo directly, effectively turning the reconstruction model into a fast multiview photometric stereo solver—something the paper does not evaluate.","Because the split-sum approximation is shared between training and supervision, an internal consistency check (comparing PRM's reconstructions when trained with full Monte Carlo PBR as supervision) would tell whether the reported gains come from true photometric cues or from the network exploiting the renderer's shortcut, a comparison the paper does not run.","The same online rendering pipeline could be used to ablate the value of other appearance variations—camera field-of-view, environment map intensity, or cast shadows—to isolate which photometric cue contributes most to the detail improvement.","The model's robustness to background clutter is limited by its white-background training; combining photometric stereo rendering with background augmentation is a natural direction the paper mentions only in passing."],"forward_implications":["Training on photometric stereo images should make a reconstruction model generalize to input photos with specular highlights, cast shadows, and dark appearances, because the model has seen those shading patterns during training.","Supervising on specular and diffuse lighting maps, not just final color, directly constrains surface normals and should improve fine local relief on smooth or textureless surfaces.","Predicting albedo rather than shading color makes the output separable into geometry and material, enabling relighting and material editing as post-hoc operations.","Rendering training data online with rasterization and split-sum approximation removes the offline rendering bottleneck, so the training distribution can be expanded to varied camera poses, fields of view, and material combinations cheaply."],"supporting_citations":[{"why":"Establishes the LRM baseline and transformer triplane architecture that PRM extends; the predecessor whose fixed-lighting data PRM argues is insufficient.","marker":"Hong et al., 2023"},{"why":"InstantMesh, the two-stage triplane+FlexiCubes pipeline and pretrained weights PRM is built on and the main baseline in comparisons.","marker":"Xu et al., 2024a"},{"why":"Supplies the split-sum approximation, the real-time PBR method used for online photometric stereo rendering.","marker":"Karis & Games, 2013"},{"why":"FlexiCubes, the differentiable iso-surface extraction that lets PRM render a mesh with PBR and lighting maps.","marker":"Shen et al., 2023"},{"why":"Classic photometric stereo formulation; motivates that varying lighting yields equations that constrain normals.","marker":"Woodham, 1980"},{"why":"Objaverse dataset used for training; supplies the 3D assets with albedo maps.","marker":"Deitke et al., 2023"},{"why":"GSO dataset used for evaluation; the quantitative benchmark where the main 0.050 vs 0.076 Chamfer Distance result is reported.","marker":"Downs et al., 2022"},{"why":"LPIPS loss used in the supervision objective; one of the losses that carries the photometric supervision.","marker":"Zhang et al., 2018"}],"fun_headline_variants":["Varying lighting in training sharpens 3D reconstruction","Photometric stereo cues refine mesh details and gloss","PRM uses varied lighting to recover fine 3D features","Glossy surfaces handled better with photometric stereo","Large reconstruction model gains from lighting diversity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole photometric-stereo benefit rests on the real-time split-sum approximation used to render both the training inputs and the supervision images: if that rendering shortcut is systematically biased, the photometric cues become renderer-specific artifacts and the robustness gains may not carry over to real photographs.","fun_headline_variants_meta":{"raw":{"variants":["Varying lighting in training sharpens 3D reconstruction","Photometric stereo cues refine mesh details and gloss","PRM uses varied lighting to recover fine 3D features","Glossy surfaces handled better with photometric stereo","Large reconstruction model gains from lighting diversity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000324,"raw_usage":{"total_tokens":1805,"prompt_tokens":919,"completion_tokens":886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":811}},"tokens_in":535,"tokens_out":886,"duration_ms":79369,"temperature":1.0,"reasoning_tokens":811,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:09.154753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a held-out set of objects with both the split-sum pipeline and a full Monte Carlo path tracer, then run PRM on the path-traced images: if Chamfer Distance or F-Score degrades markedly relative to split-sum renderings, the photometric cues are partly renderer artifacts. Alternatively, evaluate PRM on calibrated real captures with known surface geometry, where input appearance includes uncontrolled specularities and shadows, and compare error against the synthetic-test numbers reported in the paper.","supporting_citations":[{"cited_title":"Google scanned objects: A high-quality dataset of 3d scanned household items","cited_arxiv_id":null,"evidence_quote":"GSO dataset used for evaluation; the quantitative benchmark where the main 0.050 vs 0.076 Chamfer Distance result is reported."}],"review_version":1}