Pith. sign in

REVIEW 4 major objections 7 minor 15 references

Differentiable Mobile Display Photometric Stereo

T0 review · 4 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a consumer mobile phone, using its own display as the light source and its front camera as the sensor, can reconstruct surface normals and albedos of matte objects in ordinary rooms rather than a dark room, reaching…

desk verdict A promising mobile photometric stereo demo whose main experimental evidence is partially circular and whose natural-object claims lack ground truth; the engineering is real, the evaluation needs work. read the letter →

arxiv 2502.05055 v1 pith:Q34UV3G7 submitted 2025-02-07 cs.CV cs.AIcs.GRcs.LG

Gawoon Ban , Hyeongjun Kim , Seokjun Choi , Seungwoo Yoon , Seung-Hwan Baek This is my paper · ORCID
classification cs.CVcs.AIcs.GRcs.LG
keywords photometricstereomobilephonedifferentiablepatternlearningHDRimagingsurfacenormalreconstructiondisplayilluminationfallenleavesdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that display photometric stereo, which normally requires a fixed desktop monitor, a polarization camera, and a dark room, can be carried out with a single mobile phone. The authors build a custom app that displays learned illumination patterns on the phone screen while the front camera captures RAW and HDR images, then reconstruct normals with a differentiable pattern-learning pipeline. If correct, high-quality surface-orientation capture becomes portable and works under general indoor lighting, and it can be applied in the wild to objects such as fallen leaves. The reported reconstruction error is roughly 0.01 higher than the desktop DDPS baseline, and the method fails on glossy objects.

What carries the argument

The load-bearing mechanism is a learned display-pattern matrix $\mathbf{P}$ optimized through a differentiable pipeline: a differentiable image formation function $f_I(\mathbf{P}_i, \mathbf{B}) = \sum_j \mathbf{B}_j \mathbf{P}_{i,j}$ synthesizes captured images from basis images $\mathbf{B}$, and a differentiable photometric stereo function $f_n$ estimates normals, with loss $(1 - \mathbf{N} \cdot \mathbf{N}_{GT})/2$ penalizing angular error. Optimization uses Adam, with a sigmoid constraint keeping patterns in $[0,1]$. Reconstruction at test time assumes a Lambertian, planar surface at a fixed 10 cm distance and solves $\mathbf{I} = \boldsymbol{\rho} \odot \mathbf{P} \mathbf{l} \mathbf{N}$ by pseudo-inverse, with albedo set to the maximum observed intensity. Supporting image processing includes Gaussian filtering, camera undistortion, and HDR merging from exposure-bracketed RAW captures.

What would settle it

Photograph a curved, non-planar object at distances other than 10 cm, or a matte object under strong ambient light, and compare the reconstructed normal map to a ground-truth 3D scan; if the angular error rises sharply, the planar Lambertian and fixed-distance assumptions are the cause.

Watch

Extended reading notes

Core claim

The central claim is that a mobile phone with a screen and a front camera is a sufficient imaging system for practical physics-based photometric stereo. The paper introduces differentiable mobile display photometric stereo (DMDPS), which learns display patterns end-to-end using a differentiable image formation model and a differentiable photometric stereo solver, then uses those learned patterns to reconstruct surface normals on real objects. The key measurement is that, for 3D-printed training objects, DMDPS achieves average angular reconstruction error about 0.01 higher than the desktop DDPS baseline, despite using a weaker, non-polarized light source and capturing in general environments instead of a dark room. The authors also produce a first dataset of fallen leaves with reconstructed surface normals and albedos.

Load-bearing premise

The reconstruction assumes each surface patch is matte, flat, and fixed at a 10 cm distance, with all non-display light treated as removable noise; if ambient light, specular reflection, or curved geometry violates that model, the pseudo-inverse normal estimate is biased.

Editorial extensions

If this is right

  • Surface normal and albedo capture no longer requires a dedicated desktop setup, enabling photometric stereo in homes, offices, or field conditions.
  • The method's error being only about 0.01 above the desktop baseline suggests that learned display patterns remain effective when the light source is much smaller and weaker.
  • The fallen-leaves dataset demonstrates that the approach can be applied to unstructured natural objects, yielding normals that capture veins and surface undulations.
  • A phone-based system makes it straightforward to build new photometric-stereo datasets by photographing objects as they are encountered, rather than transporting them to a lab.
  • The glossy-object failure indicates that extending the method to reflective materials would require diffuse-specular separation, which the current phone camera cannot perform.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed 10 cm planar assumption could be relaxed by estimating scene depth per pixel or by capturing at multiple distances, which would extend the method to curved objects; the paper does not explore this.
  • Because the learned patterns are optimized for one phone's display and camera, transferring them to another phone model would likely require recalibration or re-learning, a step the paper leaves implicit.
  • The HDR approach suggests that carefully chosen exposure brackets could also compensate for mild object motion or varying ambient flicker, though the paper does not test these conditions.
  • A natural extension is to attach a polarizing film to the front camera and use a polarized display mode, which the authors mention as future work and which would directly address their main error source.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Differentiable Mobile Display Photometric Stereo (DMDPS), a photometric stereo system that uses a mobile phone's display as a programmable light source and its front camera for capture. The authors develop a custom mobile app for synchronized pattern display and RAW/HDR image capture, apply Gaussian filtering and camera undistortion, and adapt the differentiable pattern-learning framework of DDPS to the mobile setting. Experiments on 3D-printed objects show that learned patterns reduce reconstruction error relative to initial patterns, and the authors introduce a new dataset of fallen leaves with reconstructed surface normals and albedos.

Significance. If the central claims hold, DMDPS would be a practical and portable photometric stereo solution requiring only a commodity smartphone, enabling capture in general environments rather than a dark room and without polarization hardware. The contributions include the mobile app, the adaptation of learned illumination patterns, and a new natural-object dataset. However, the quantitative evaluation is compromised by a likely train/test overlap, and the natural-object validation is strictly qualitative; these issues must be resolved before the paper's main claims are convincing.

major comments (4)
  1. [Section 6.1 / Table 2; Section 5.1] The paper does not state whether the 3D-printed objects used to compute the reconstruction errors in Table 2 are the same as those used in the pattern-learning procedure described in Section 5.1. Since Section 5.1 refers to a "3D-printed training dataset" and Table 2 reports errors for "the 3D-printed database," it appears that the evaluation may be performed on the training set itself. If so, the improvements in Table 2 (e.g., from 0.4109 to 0.05754 for the Flat gray pattern) could be inflated by overfitting, and the claim that learned patterns generalize is not supported. The authors should either report results on a clearly disjoint held-out set of 3D-printed objects or explicitly state that Table 2 is a training-set evaluation and provide additional test-set numbers.
  2. [Section 5.2, Eq. (1); Section 6.3] The reconstruction model I = ρ ⊙ P l N assumes a planar surface at a fixed distance of 10 cm, and the illumination directions l are computed under this assumption. Fallen leaves are non-planar and exhibit curvature, so the true illumination directions at each scene point differ from l, causing a systematic bias in the pseudo-inverse normal estimate. Section 6.3 claims that the method "handle[s] variations in curvature" and achieves "high angular accuracy" on leaves, but no quantitative evaluation with ground-truth normals is provided; the supporting evidence is limited to qualitative images in Figure 7. This is a load-bearing gap because the fallen-leaf dataset is one of the paper's headline contributions. The authors should provide either ground-truth normal maps for a set of leaves (e.g., obtained by 3D scanning) or a quantitative sensitivity analysis that estimates the error introduced by the planar assumption on curved surfaces.
  3. [Section 4.3 / Section 3.3] The paper claims that HDR and RAW imaging enable capture in general environments instead of a dark room, but no controlled experiment is reported to substantiate this claim. Specifically, there is no comparison of reconstruction error with and without HDR, nor a comparison of captures in a standard room versus a dark room. Without such an ablation, it is unclear whether the claimed advantage of "general environments" is actually attributable to the HDR/RAW pipeline, and whether the method would work without these components. The authors should add an experiment that isolates the effect of HDR/RAW on reconstruction accuracy, or tone down the claim accordingly.
  4. [Section 5.1, Eq. (2) and pattern-learning objective] The differentiable image formation model f_I(P_i, B) = Σ_j B_j P_{i,j} is a linear function of the basis image B and the pattern P_i. This model does not account for external ambient light, specular reflections, or the nonlinear merging used in HDR imaging, all of which are present in the actual capture pipeline. The paper does not explain how B is captured (e.g., whether it includes external light components), nor why this linear simulation is adequate for learning patterns that are then used on real HDR captures. The authors should describe the acquisition of B and discuss the potential domain gap between the simulated training images and real captured images, or provide an experiment that tests whether patterns learned with the linear model transfer to real captures with HDR.
minor comments (7)
  1. [Table 1] The label "OLA T" is a typo and should be "OLAT."
  2. [Section 4.3, tangential distortion formula] The equation for tangential distortion appears incorrect: x_distorted = x(2p1 y + p2(r^2 + 2x^2)) is missing the additive identity term. The standard model is x_distorted = x + [2p1 xy + p2(r^2 + 2x^2)] (and similarly for y). Please correct the formula.
  3. [Section 4.3, HDR merging equation] The weight function W(I_ti) in the HDR merging equation is not defined. Please specify the weighting scheme used.
  4. [Section 5.1 / Table 2] The paper does not report the number of 3D-printed training objects, the number of fallen leaves, or the number of patterns K used in the experiments. These details are necessary for reproducibility and for assessing the scale of the evaluation.
  5. [Section 6.1] The statement that the average reconstruction error is "approximately 0.01 higher than that of DDPS" would be more informative if the DDPS error on a comparable object set were reported explicitly, preferably in the same table.
  6. [Figure 7] The leaf reconstructions in Figure 7 are qualitative. Adding quantitative metrics or a comparison with a baseline photometric stereo method would strengthen the claims in Section 6.3.
  7. [Throughout] There are several typos, e.g., "capture s" in the abstract, "normal s" in the abstract, "sperate" in Section 6.1, and inconsistent use of spacing. A careful proofread is needed.

Circularity Check

1 steps flagged · score 6.0 of 10

Table 2's quantitative validation is measured on the same 3D-printed training set used to optimize the display patterns, so the reported improvement is a fitting result rather than a prediction.

  1. fitted input called prediction [Section 5.1 (Pattern Learning) and Section 6.1 / Table 2]
    "To learn display patterns, we utilize a 3D -printed training dataset containing ground-truth normal maps NGT and a basis image B. ... Table 2 summarizes the reconstruction errors of each pattern for the 3D-printed database."

    The display patterns P are the optimization variables in Section 5.1, minimized against the same 3D-printed ground-truth normals NGT via the loss (1 - N·NGT)/2. Section 6.1 then reports Table 2 'reconstruction error for each initial pattern' on the 3D-printed database and compares it to DDPS, with no held-out test split described. Thus the central quantitative accuracy claim measures training-set fit, not generalization; the learned-pattern improvement over initial patterns is forced by the training objective. The fallen-leaf results are qualitative only, so no independent quantitative validation of the learned patterns is provided.

full rationale

The only quantitative evaluation of DMDPS's central claim is Table 2, computed on the same 3D-printed database used for pattern learning; no train/test split is described. This makes the reported reconstruction error a fitted training objective rather than a prediction. The method does contain independent components: a custom capture app, HDR/RAW processing, undistortion, and an analytic pseudo-inverse normal reconstruction with no trainable parameters. The natural-object leaf demonstration is an application to unseen data, but lacks ground-truth normals, so it cannot quantitatively redeem the circularity. The DDPS comparison is based on the authors' own prior work, but this is a normal baseline citation and not load-bearing to the derivation. Therefore score 6, partial circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central reconstruction relies on a linear Lambertian image formation model, a fixed planar geometry at 10 cm, and linear superposition of display patterns; these are domain assumptions. Hyperparameters such as the Gaussian sigma and pattern-learning step size are tuned by hand without final values. No new physical entities are introduced.

free parameters (3)
  • Gaussian filter sigma = not reported
    Fine-tuned for the mobile camera in Sections 2 and 4.3, but the value and tuning criterion are not stated; it controls how much noise smoothing is applied before pattern learning and reconstruction.
  • Pattern learning step size alpha and decay factor = alpha tested at 5, 10, 15, 20; decay factor 0.3; final choice unspecified
    Table 1 sweeps alpha to evaluate reconstruction error, and the decay factor is set to 0.3, but the paper never states which configuration was used for the final learned patterns.
  • Albedo estimate rho = per-pixel maximum intensity across captures
    Section 5.2 sets albedo to the maximum observed intensity to make the pseudo-inverse well posed; this is an ad hoc estimate that biases normals for pixels with interreflections or specularity.
assumptions (5)
  • domain assumption Lambertian linear image formation, I = rho ⊙ P l N, with no ambient or specular term.
    Section 5.2 uses this equation to invert intensities for normals; the paper's glossy-object results in Section 6.2 show the assumption fails for specular surfaces.
  • domain assumption Planar scene at fixed 10 cm distance for computing per-superpixel light directions l.
    Section 5.2 assumes a planar surface at 10 cm to define l_j; real 3D-printed objects and leaves have depth variation, making this an approximation.
  • domain assumption Linear superposition of display patterns and basis image B in fI(Pi, B) = sum_j B_j P_ij.
    Section 5.1 models captured intensity as a weighted sum of superpixel intensities, ignoring interreflections, display nonlinearities, and external light.
  • domain assumption HDR merging of bracketed RAW frames removes or reduces ambient light interference.
    Section 4.3 assumes that weighting and summing multiple exposure times recovers a clean scene-radiance image; the paper provides no quantitative test of how well ambient light is rejected.
  • standard math Adam optimizer with sigmoid parameterization converges to useful patterns.
    Section 5.1 uses Adam as a black-box optimizer and a sigmoid to keep patterns in [0,1]; this is standard practice, not a new mathematical claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Differentiable Mobile Display Photometric Stereo." pith.science (2026). https://pith.science/paper/Q34UV3G7

@misc{pith2026250205055,
  author       = {Pith},
  title        = {Pith review of: Differentiable Mobile Display Photometric Stereo},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q34UV3G7}},
  note         = {Machine review of arXiv:2502.05055}
}
read the original abstract

Display photometric stereo uses a display as a programmable light source to illuminate a scene with diverse illumination conditions. Recently, differentiable display photometric stereo (DDPS) demonstrated improved normal reconstruction accuracy by using learned display patterns. However, DDPS faced limitations in practicality, requiring a fixed desktop imaging setup using a polarization camera and a desktop-scale monitor. In this paper, we propose a more practical physics-based photometric stereo, differentiable mobile display photometric stereo (DMDPS), that leverages a mobile phone consisting of a display and a camera. We overcome the limitations of using a mobile device by developing a mobile app and method that simultaneously displays patterns and captures high-quality HDR images. Using this technique, we capture real-world 3D-printed objects and learn display patterns via a differentiable learning process. We demonstrate the effectiveness of DMDPS on both a 3D printed dataset and a first dataset of fallen leaves. The leaf dataset contains reconstructed surface normals and albedos of fallen leaves that may enable future research beyond computer graphics and vision. We believe that DMDPS takes a step forward for practical physics-based photometric stereo.

Figures

Figures reproduced from arXiv: 2502.05055 by the authors.

Figure 3
Figure 3. Photos Captured with Different Settings. These images show objects photographed with varying ISO and exposure time values. To address this issue, a dummy image is captured before displaying the pattern, effectively reducing the green dot’s impact on the final image. 4.3. Image Processing To achieve high-quality surface normal reconstruction, followings were considered. Gaussian Filter Images captured with a mobile p… view at source ↗
Figure 6
Figure 6. Training sets. Raw images from the training dataset captured using the DMDPS app. Illumination patterns step_size (𝛼 = 5) step_size (𝛼 = 10) step_size (𝛼 = 15) step_size (𝛼 = 20) Mono-gradient 0.06439 0.063 0.06189 0.063 OLAT 0.0684 0.07576 0.07749 0.07673 Mono-random 0.07387 0.06616 0.06567 0.06488 Average 0.06888 0.0683 0.06835 0.0682 [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 8
Figure 8. Initial & Learned patterns. The left side of each row represents the initial patterns, while the right side shows the learned patterns. patterns, and the surface normals and albedos restored with DMDPS [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

  1. [1]

    Seokjun Choi, Seungwoo Yoon, Giljoo Nam, Seungyong Lee, Seung-Hwan Baek, Differentiable Display Photometric Stereo, In CVPR, pp. 3,2023

    work page 2023

  2. [2]

    4-6,2023

    Seokjun Choi, Seungwoo Yoon, Giljoo Nam, Seungyong Lee, Seung-Hwan Baek, Differentiable Display Photometric Stereo, In CVPR, pp. 4-6,2023

    work page 2023

  3. [3]

    Near-instant capture of high-resolution facial geometry and reflectance

    Graham Fyffe, Paul Graham, Borom Tunwattanapong, Abhijeet Ghosh, and Paul Debevec. Near-instant capture of high-resolution facial geometry and reflectance. In Comput. Graph. Forum, volume 35, pages 353–363. Wiley Online Library, 2016. 2

    work page 2016

  4. [4]

    Deep polarization imaging for 3d shape and svbrdf acquisition

    Valentin Deschaintre, Yiming Lin, and Abhijeet Ghosh. Deep polarization imaging for 3d shape and svbrdf acquisition. In IEEE Conf. Comput. Vis. Pattern Recog., pages 15567–15576, 2021. 2

    work page 2021

  5. [5]

    Mobile surface reflectometry

    J Riviere, P Peers, and A Ghosh. Mobile surface reflectometry. In Comput. Graph. Forum, volume 1, pages 191–202, 2016. 2

    work page 2016

  6. [6]

    Reflectance capture using univariate sampling of brdfs

    Zhuo Hui, Kalyan Sunkavalli, Joon-Y oung Lee, Sunil Hadap, Jian Wang, and Aswin C Sankaranarayanan. Reflectance capture using univariate sampling of brdfs. In Int. Conf. Comput. Vis., pages 5362–5370, 2017. 2

    work page 2017

  7. [7]

    Neural light transport for relighting and view synthesis

    Xiuming Zhang, Sean Fanello, Y un-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, et al. Neural light transport for relighting and view synthesis. ACM Trans. Graph., 40(1):1–17,

    work page

  8. [8]

    Light stage superresolution: continuous high -frequency relighting

    Tiancheng Sun, Zexiang Xu, Xiuming Zhang, Sean Fanello, Christoph Rhemann, Paul Debevec, Y un-Ta Tsai, Jonathan T Barron, and Ravi Ramamoorthi. Light stage superresolution: continuous high -frequency relighting. ACM Trans. Graph., 39(6):1–12, 2020. 2, 5

    work page 2020

Show all 15 references
  1. [9]

    Acquiring the reflectance field of a human face

    Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145–156,

  2. [10]

    Qian ZHENG, Boxin SHI, Gang PAN, Summary study of data- driven photometric stereo methods, In 2020 Beijing Zhongke Journal, pages 2-3

    2020

  3. [11]

    Kingma and Jimmy Ba

    Diederik P . Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 5

    2015

  4. [12]

    Photometric stereo with non-parametric and spatially-varying reflectance

    Neil Alldrin, Todd Zickler, and David Kriegman. Photometric stereo with non-parametric and spatially-varying reflectance. In 2008 IEEE Conference on Computer

    2008

  5. [13]

    Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials

    Min Li, Zhenglong Zhou, Zhe Wu, Boxin Shi, Changyu Diao, and Ping Tan. Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials. IEEE Trans. Image Process., 29:4159– 4173, 2020. 2, 3

    2020

  6. [14]

    Luces: A dataset for near -field point light source photometric stereo

    Roberto Mecca, Fotios Logothetis, Ignas Budvytis, and Roberto Cipolla. Luces: A dataset for near -field point light source photometric stereo. arXiv preprint arXiv:2104.13135, 2021. 2

    2021 arXiv

  7. [15]

    What is learned in deep uncalibrated photometric stereo? In Eur

    Guanying Chen, Michael Waechter, Boxin Shi, Kwan-Y ee K Wong, and Y asuyuki Matsushita. What is learned in deep uncalibrated photometric stereo? In Eur. Conf. Comput. Vis., pages 745–762. Springer, 2020

    2020

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.