REVIEW 3 major objections 5 minor 40 references
CompenNet++: End-to-end Full Projector Compensation
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper proposes CompenNet++, the first end-to-end network that performs full projector compensation—geometric correction plus photometric compensation—jointly, learning the geometric warp directly from photometric sampling images…
desk verdict First end-to-end full projector compensation, but the headline comparison runs on a proxy metric; worth reviewing with a demand for a direct project-and-capture test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is WarpingNet, the geometric correction subnet: a cascaded coarse-to-fine sampling grid predictor. It stacks (1) a learnable $2\times3$ affine matrix that maps the projector field of view toward a front view, (2) a thin-plate-spline grid with 76 learnable parameters for a smooth nonlinear warp, and (3) a grid-refinement network, built on a U-Net-like encoder-decoder, that adds residual displacements to the grid. Instead of resampling the image three times, the finer grids sample the coarser grids in coordinate space, so the input image is warped once with a single bilinear interpolation; this keeps the result sharp and reduces computation. After training, the whole geometric stage collapses to one sampling grid, and the photometric surface-feature branch collapses to additive biases.
What would settle it
Measure the learned geometric warp against an independent structured-light ground truth on a surface with a deep concavity or strong interreflection; if the compensation in the occluded region stays plain gray or the reprojection error exceeds that of a two-step SL baseline, the joint-training claim fails. Alternatively, ablate the grid refinement network while keeping photometric compensation fixed on a specular surface; if the end-to-end advantage disappears, the mechanism is the refinement network rather than the joint formulation.
Extended reading notes
Core claim
The central claim is that a deep network can learn the composite geometric warp and photometric transfer of a projector-camera-surface system directly from photometric sampling images, without building a pixel-correspondence map by structured light. The paper models a camera capture as $\tilde{x} = \mathcal{T}(\mathcal{F}(x;g,s))$ and trains the network to invert both factors at once, $x^* = \mathcal{F}^{\dagger}(\mathcal{T}^{-1}(x);\mathcal{T}^{-1}(\tilde{s}))$. The geometric part is a cascaded coarse-to-fine warp: a learnable affine transform, a thin-plate-spline grid, and a residual grid-refinement network, all composed in 2D coordinate space so the image is resampled only once. The photometric part is an improved CompenNet initialized as a linear channel-independent model to avoid degenerate solutions such as plain-gray outputs. At test time both subnets simplify to a single sampling grid plus a bias-augmented photometric backbone, with no measured performance drop.
Load-bearing premise
The method assumes that jointly training the warp and the photometric network converges to a useful solution from the proposed initialization; if the photometric loss gives insufficient gradient signal toward the correct warp, the output degrades to a plain gray image, and the paper also assumes every surface patch is directly illuminated by the projector, so self-occluded regions fail.
Editorial extensions
If this is right
- Projector systems no longer need a structured-light scan: the geometric warp is learned from the same photometric sampling images used for color compensation, saving the 42 extra projected patterns used in the comparisons.
- Errors stop propagating from geometric mapping into photometric compensation; joint training can tolerate specular highlights and reflections that corrupt structured-light decoding.
- At test time the trained system reduces to one image resampling and one pass through the photometric backbone, which is faster and uses less memory than the training-time network.
- Even with only 48 training images and shortened training, the method can keep a structural-similarity advantage over two-step baselines trained on 500 images.
- The joint formulation provides a template for other projector-camera tasks where geometry and appearance are entangled and traditionally handled in sequence.
Reading between the lines
- The cascaded coordinate-space grid composition could be reused for projector-camera calibration or multi-projector blending, where geometric warps are currently estimated from markers or structured light.
- The initialization recipe—starting the photometric subnet as a linear channel-independent subtractor—is a general defense against degenerate vacuous solutions in end-to-end image-to-image training, and could be tested on other compensation networks.
- On surfaces with strong interreflections or global light transport, conditioning only on a single surface image may be insufficient; a testable extension would feed multiple surface images captured under different projector inputs.
- A direct geometric evaluation against structured-light ground truth (for example reprojection error) would isolate how much of the end-to-end gain comes from better warping versus better photometric modeling, since the paper reports only indirect compensation quality.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CompenNet++, an end-to-end neural network for full projector compensation on nonplanar textured surfaces. It extends the authors' earlier CompenNet by adding a WarpingNet subnet that learns a cascaded coarse-to-fine geometric warp directly from photometric sampling images, concatenating it with a photometric compensation subnet, and training both jointly. A benchmark with 20 setups is introduced, and experiments compare against TPS-, Pix2pix-, and CompenNet-based two-step baselines. The paper also describes task-specific weight initialization, network simplification for test-time efficiency, and reports qualitative and quantitative results claimed to outperform prior methods. The source code and benchmark are released.
Significance. If the empirical claims are confirmed, this would be a useful step: the first end-to-end trainable full compensation pipeline, demonstrated on a public benchmark with 20 setups, with code release. The joint-training idea is principled, and the coarse-to-fine grid refinement and test-time simplification are sensible engineering contributions. The paper explicitly ships reproducible code and a new benchmark, which are strengths. However, the evaluation protocol in Section 3.8 does not directly test the production pipeline described in Section 3.6, and the reported improvements over CompenNet with structured light are small and reported without error bars. The central comparative claim therefore needs additional evidence before the paper can be accepted.
major comments (3)
- [Section 3.8, Eq. (9), Table 1, Fig. 5] Section 3.8 defines the reported metric as \hat{y}_i = \pi^\dagger_\theta(\tilde{y}_i; \tilde{s}) compared with y_i, where \tilde{y}_i is the camera capture obtained by projecting y_i. This evaluates the inverse map on the range of the forward imaging process, i.e., reconstruction of the projector input from its capture. The production pipeline in Section 3.6 and Fig. 5 instead feeds the desired viewer image z (after affine fit z' = A z) into the simplified network and projects the output. A desired image z is not generally a camera capture \tilde{y}_i, so it lies outside the input distribution used in Eq. (9). Table 1 therefore does not directly measure full compensation quality for arbitrary desired images; it measures inverse consistency on training-distribution captures. A direct project-and-capture experiment is required: for held-out desired images z, compute z* = \pi'^\dagger_\theta(z'), project z*, capture the result, and compare it with z, for all baselines. Without that, the abstract's claim of outperforming two-step methods is not supported by the reported numbers.
- [Section 3.5] Section 3.5 states that the test-time simplification is made 'without any sacrifice in performance quality,' and Section 3.6 uses the simplified \pi'^\dagger_\theta at deployment, but no experiment compares \pi'^\dagger_\theta with the un-simplified \pi^\dagger_\theta under identical training. The rows CompenNet++ fast/faster in Table 1 use different training-set sizes and iteration counts, so they are not an ablation of the simplification. Please add an ablation that trains once and evaluates both the full and simplified networks on the same data.
- [Table 1] Table 1 reports averages over 20 setups but no per-setup variances or significance tests. At #Train=500, CompenNet++ attains PSNR 21.11 versus 20.96 for CompenNet w/ SL and SSIM 0.727 versus 0.712; these differences are small relative to likely cross-setup variation. Without error bars, paired tests, or per-setup results, the claim of 'clear advantages' is not quantitatively established. Reporting the standard deviation over the 20 setups, or a per-setup table or scatter plot, would address this.
minor comments (5)
- [Section 2.2] There is a likely typo: 'context-aware ones typically assume pro-cam pixels one-to-one mapping and context-aware ones also consider neighborhood/global information' should read 'context-independent ones also consider neighborhood/global information.'
- [Sections 3.1 and 3.2] The symbol x is used both for a projector input image and for a desired perceived image (e.g., Eq. (3) versus Eq. (7)). Please use separate symbols or state explicitly that the same symbol is intentionally reused.
- [Fig. 5] Figure 5 is underspecified: it should state which tensor is the input to \pi'^\dagger_\theta, in which coordinate frame z' lives, and how the affine transform A is applied.
- [Section 3.4, Eq. (13)] The initialization in Eq. (13) uses a synthetic surface image \dot{s} that 'mimics' the warped surface image, but the paper does not describe how \dot{s} is generated; a sentence or reference is needed for reproducibility.
- [Section 4.2] The statement 'we use 42 less images than two-step SL-based method' should be qualified: at #Train=500 the total is 500 versus 542, while at #Train=48 the difference is 42, so the claim depends on the training-set size.
Circularity Check
No circular derivation found: the core claim is empirically tested against external baselines, and self-citations to CompenNet are not load-bearing.
full rationale
The paper's derivation chain is a supervised inverse-modeling construction, not a self-referential one. Equations (1)-(6) define the forward projection/capture process and formally write the compensation image as F^†(T^{-1}(x); T^{-1}(~s)); Eqs. (7)-(9) then train a network π^†_θ on sampled pairs (~x_i, x_i) by minimizing a loss between the network output and the known projector input x_i. The evaluation in Section 3.8 uses held-out pairs (~y_i, y_i) under the same setup, so the reported PSNR/RMSE/SSIM measure generalization of the learned inverse, not a quantity that equals the training target by construction. No parameter is fitted to the validation metric and then reported as a prediction. CompenNet [15] is cited extensively as the photometric backbone and as the source of the evaluation protocol, and it is by the same authors. However, the central claim of this paper—that jointly learning geometry and photometry outperforms two-step structured-light methods—is supported by comparisons with TPS [10], Pix2pix [17], and CompenNet w/ SL, all trained and evaluated under the same protocol. The reused CompenNet is a published component, not an unverified theorem imported to force the conclusion, so the self-citations are routine prior work rather than load-bearing circularity. The only notable concern is an evaluation/deployment mismatch: Section 3.8 evaluates ŷ_i = π^†_θ(~y_i; ~s) against y_i, whereas the test-time pipeline in Fig. 5 feeds a desired image z through the simplified network. This is a validity or correctness issue about whether the headline 'outperforms' is directly supported, not a circularity: the reported metric is not equal to the training loss by construction, and all compared methods are subject to the same protocol. The limitations section also explicitly identifies self-occlusion as a failure mode, which is an external, falsifiable scope condition rather than a circular defensive move.
Assumptions & free parameters
free parameters (4)
- Network weights θ = {θF, θT} =
Learned on each setup
- TPS control point grid (6x6, 76 parameters) =
6x6 grid
- Loss weighting (L1 + SSIM, equal weights) =
1 and 1
- Training hyperparameters (learning rate, batch size, iterations) =
1e-3, 48, 1500
assumptions (4)
- domain assumption The image formation process (Eq. 1) can be disentangled into a geometric warp T and a photometric transfer F (Eq. 2).
- domain assumption The camera-captured surface image \tilde{s} under a plain gray input captures the spectral interaction needed for compensation (Eq. 5).
- domain assumption The inverse mappings F† and T^{-1} are representable by the chosen CNN architectures and learnable from the sampled pairs.
- domain assumption Each surface patch is directly illuminated by the projector (no self-occlusion).
Cite this review
Pith. "Pith review of CompenNet++: End-to-end Full Projector Compensation." pith.science (2026). https://pith.science/paper/TIV55S3A
@misc{pith2026190806246,
author = {Pith},
title = {Pith review of: CompenNet++: End-to-end Full Projector Compensation},
year = {2026},
howpublished = {\url{https://pith.science/paper/TIV55S3A}},
note = {Machine review of arXiv:1908.06246}
}
read the original abstract
Full projector compensation aims to modify a projector input image such that it can compensate for both geometric and photometric disturbance of the projection surface. Traditional methods usually solve the two parts separately, although they are known to correlate with each other. In this paper, we propose the first end-to-end solution, named CompenNet++, to solve the two problems jointly. Our work non-trivially extends CompenNet, which was recently proposed for photometric compensation with promising performance. First, we propose a novel geometric correction subnet, which is designed with a cascaded coarse-to-fine structure to learn the sampling grid directly from photometric sampling images. Second, by concatenating the geometric correction subset with CompenNet, CompenNet++ accomplishes full projector compensation and is end-to-end trainable. Third, after training, we significantly simplify both geometric and photometric compensation parts, and hence largely improves the running time efficiency. Moreover, we construct the first setup-independent full compensation benchmark to facilitate the study on this topic. In our thorough experiments, our method shows clear advantages over previous arts with promising compensation quality and meanwhile being practically convenient.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Fast high-resolution appear- ance editing using superimposed projections
Daniel G Aliaga, Yu Hong Yeung, Alvin Law, Behzad Sa- jadi, and Aditi Majumder. Fast high-resolution appear- ance editing using superimposed projections. ACM Tran. on Graphics, 2012. 1, 2
work page 2012
-
[2]
Fabri- cating diminishable visual markers for geometric registration in projection mapping
Hirotaka Asayama, Daisuke Iwai, and Kosuke Sato. Fabri- cating diminishable visual markers for geometric registration in projection mapping. IEEE TVCG, 2018. 1, 2
work page 2018
-
[3]
Robust content-dependent photometric projector com- pensation
Mark Ashdown, Takahiro Okabe, Imari Sato, and Yoichi Sato. Robust content-dependent photometric projector com- pensation. In CVPRW PROCAMS, 2006. 1, 2
work page 2006
-
[4]
Embedded entertainment with smart projectors
Oliver Bimber, Andreas Emmerling, and Thomas Klemmer. Embedded entertainment with smart projectors. Computer,
-
[5]
Saliency-guided pro- jection geometric correction using a projector-camera sys- tem
Ameneh Boroomand, Hicham Sekkati, Mark Lamm, David A Clausi, and Alexander Wong. Saliency-guided pro- jection geometric correction using a projector-camera sys- tem. In ICIP, 2016. 1, 2
work page 2016
-
[6]
Approximate thin plate spline mappings
Gianluca Donato and Serge Belongie. Approximate thin plate spline mappings. In ECCV, 2002. 4, 6
work page 2002
-
[7]
A projector-camera system with real-time photometric adapta- tion for dynamic environments
Kensaku Fujii, Michael D Grossberg, and Shree K Nayar. A projector-camera system with real-time photometric adapta- tion for dynamic environments. In CVPR, 2005. 2
work page 2005
-
[8]
Structured-light 3D surface imaging: a tutorial
Jason Geng. Structured-light 3D surface imaging: a tutorial. Advances in Optics and Photonics, 2011. 1
work page 2011
Show all 40 references
-
[9]
Making one object look like another: controlling appearance using a projector-camera system
Michael D Grossberg, Harish Peri, Shree K Nayar, and Pe- ter N Belhumeur. Making one object look like another: controlling appearance using a projector-camera system. In CVPR, 2004. 1, 2
2004
-
[10]
Robust, error-tolerant photometric projector compensation
Anselm Grundh ¨ofer and Daisuke Iwai. Robust, error-tolerant photometric projector compensation. IEEE TIP, 2015. 1, 2, 6, 7
2015
-
[11]
Recent advances in projection mapping algorithms, hardware and applications
Anselm Grundh ¨ofer and Daisuke Iwai. Recent advances in projection mapping algorithms, hardware and applications. In Computer Graphics Forum. Wiley Online Library, 2018. 1, 3
2018
-
[12]
Practical methods for geometric and photometric correction of tiled projector
Michael Harville, Bruce Culbertson, Irwin Sobel, Dan Gelb, Andrew Fitzhugh, and Donald Tanguay. Practical methods for geometric and photometric correction of tiled projector. In CVPRW, 2006. 1, 2
2006
-
[13]
Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. In ICCV, 2015. 5
2015
-
[14]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,
-
[15]
End-to-end projector pho- tometric compensation
Bingyao Huang and Haibin Ling. End-to-end projector pho- tometric compensation. In CVPR, 2019. 1, 2, 3, 4, 5, 6, 7, 8
2019
-
[16]
A single-shot-per-pose camera- projector calibration system for imperfect planar targets
Bingyao Huang, Samed Ozdemir, Ying Tang, Chunyuan Liao, and Haibin Ling. A single-shot-per-pose camera- projector calibration system for imperfect planar targets. In 2018 IEEE International Symposium on Mixed and Aug- mented Reality Adjunct (ISMAR-Adjunct) , pages 15–20. IEEE, 2018. 1
2018
-
[17]
Image-to-image translation with conditional adver- sarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. CVPR, 2017. 6, 7, 8
2017
-
[18]
Spatial transformer networks
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In NIPS,
-
[19]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6
2015
-
[20]
Practical radiometric compensation for projection display on textured surfaces using a multidi- mensional model
Yuqi Li, Aditi Majumder, Meenakshisundaram Gopi, Chong Wang, and Jieyu Zhao. Practical radiometric compensation for projection display on textured surfaces using a multidi- mensional model. In Computer Graphics Forum. Wiley On- line Library, 2018. 2
2018
-
[21]
Recti- fier nonlinearities improve neural network acoustic models
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Recti- fier nonlinearities improve neural network acoustic models. In ICML, 2013. 5
2013
-
[22]
Simple, accurate, and robust projector-camera calibration
Daniel Moreno and Gabriel Taubin. Simple, accurate, and robust projector-camera calibration. In 3DIMPVT, 2012. 1, 6
2012
-
[23]
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010. 5
2010
-
[24]
Dynamic projection mapping onto deforming non-rigid sur- face using deformable dot cluster marker.IEEE TVCG, 2017
Gaku Narita, Yoshihiro Watanabe, and Masatoshi Ishikawa. Dynamic projection mapping onto deforming non-rigid sur- face using deformable dot cluster marker.IEEE TVCG, 2017. 1, 2
2017
-
[25]
A projection system with radiometric com- pensation for screen imperfections
Shree K Nayar, Harish Peri, Michael D Grossberg, and Pe- ter N Belhumeur. A projection system with radiometric com- pensation for screen imperfections. In ICCVW PROCAMS, volume 3, 2003. 2, 5
2003
-
[26]
A threshold selection method from gray- level histograms
Nobuyuki Otsu. A threshold selection method from gray- level histograms. IEEE T-SMC, 1979. 5
1979
-
[27]
Automatic differentiation in pytorch
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017. 6
2017
-
[28]
A self-correcting pro- jector
Ramesh Raskar and Paul Beardsley. A self-correcting pro- jector. In CVPR, 2001. 1, 2
2001
-
[29]
ilamps: geo- metrically aware and self-configuring projectors
Ramesh Raskar, Jeroen Van Baar, Paul Beardsley, Thomas Willwacher, Srinivas Rao, and Clifton Forlines. ilamps: geo- metrically aware and self-configuring projectors. ACM Tran. on Graphics, 2003. 1, 2, 5, 6
2003
-
[30]
Shader lamps: Animating real objects with image-based illumination
Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay. Shader lamps: Animating real objects with image-based illumination. In Rendering Techniques. Springer, 2001. 1, 2
2001
-
[31]
Convo- lutional neural network architecture for geometric matching
Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convo- lutional neural network architecture for geometric matching. In CVPR, 2017. 4
2017
-
[32]
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI. Springer, 2015. 5
2015
-
[33]
Adict: accurate direct and inverse color transformation
Behzad Sajadi, Maxim Lazarov, and Aditi Majumder. Adict: accurate direct and inverse color transformation. In ECCV. Springer, 2010. 2
2010
-
[34]
Simultaneous geometric and ra- diometric calibration of a projector-camera pair
Marjan Shahpaski, Luis Ricardo Sapaico, Gaspard Chevas- sus, and Sabine Susstrunk. Simultaneous geometric and ra- diometric calibration of a projector-camera pair. In CVPR,
-
[35]
Adaptive stray-light compensation in dynamic multi-projection mapping
Christian Siegl, Matteo Colaianni, Marc Stamminger, and Frank Bauer. Adaptive stray-light compensation in dynamic multi-projection mapping. Computational Visual Media ,
-
[36]
Real-time pixel luminance optimization for dynamic multi-projection mapping
Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Michael Zollh ¨ofer, Shahram Izadi, Marc Stamminger, and Frank Bauer. Real-time pixel luminance optimization for dynamic multi-projection mapping. ACM Tran. on Graph- ics, 2015. 1, 2
2015
-
[37]
Inter- reflection compensation of immersive projection display by spatio-temporal screen reflectance modulation
Shoichi Takeda, Daisuke Iwai, and Kosuke Sato. Inter- reflection compensation of immersive projection display by spatio-temporal screen reflectance modulation. IEEE TVCG,
-
[38]
Multi-projectors for arbitrary surfaces without explicit cal- ibration nor reconstruction
Jean-Philippe Tardif, S ´ebastien Roy, and Martin Trudeau. Multi-projectors for arbitrary surfaces without explicit cal- ibration nor reconstruction. In 3DIM, 2003. 1, 2
2003
-
[39]
A vir- tual color reconstruction system for real heritage with light projection
Takenobu Yoshida, Chinatsu Horii, and Kosuke Sato. A vir- tual color reconstruction system for real heritage with light projection. In Proceedings of International Conference on Virtual Systems and Multimedia, volume 3, 2003. 1, 2
2003
-
[40]
Loss functions for image restoration with neural networks
Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE TCI, 2017. 4
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.