{"id":"3d071ec8-bdd0-44fb-a6d9-feb54aee9ff8","arxiv_id":"1908.06246","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CompenNet++ jointly trains a warping network and a photometric compensation network to correct projector output for nonplanar textured surfaces, outperforming two-step baselines with fewer calibration images.","lead":"This paper presents CompenNet++, a deep learning system that corrects both the geometry and color of projected images on curved, textured surfaces in one trained pipeline. Earlier methods handled geometry and color separately, which caused errors to build up; this method learns both together.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 evaluates the inverse map on camera captures (π†(~y_i) vs y_i), not the test-time compensation pipeline (π†(desired z)); the headline 'outperforms' is not directly supported. A direct project-and-capture evaluation would settle it.","rationale":"The reader's weakest assumption concerns joint-training convergence and initialization. That is a legitimate risk, especially given the explicit warning about plain-gray suboptimal solutions in Section 3.4, but it is secondary to the evaluation mismatch. Even if the network trains perfectly and the photometric loss provides full gradient signal, Table 1 as described in Section 3.8 does not measure the compensation quality produced by the Section 3.6 pipeline. The central claim in the abstract and Section 5 is that CompenNet++ outperforms two-step methods in full compensation; that claim rests on Table 1. The current protocol instead checks whether the learned inverse of T∘F can recover a projector input from its own camera capture, which is a different task. This is an internal inconsistency between the stated deployment procedure and the evaluation procedure, not a disagreement with field consensus. The paper does provide source code, a benchmark, and ablations, so the direct test is feasible. Until that direct test is reported or can be run from the released code, the strongest empirical claim should be treated as unverified rather than conditionally accepted on the basis of the current numbers.","tokens_in":1023,"tokens_out":2344,"duration_ms":179774,"concrete_test":"Re-run the Table 1 benchmark with the production pipeline: for each validation desired image y_i, compute the FOV-fitted desired image z' = A y_i, feed z' (not ~y_i) to the simplified network π'†_θ to obtain compensation z*, project z*, capture ~z*, and report PSNR/RMSE/SSIM between ~z* and z' over the same 20 setups. Apply the same direct protocol to TPS w/ SL, TPS textured w/ SL, Pix2pix w/ SL, and CompenNet w/ SL. If CompenNet++ retains its numerical advantage in this direct measurement, the concern is resolved; if the advantage shrinks, reverses, or the metric cannot be computed, the headline claim is not supported by the current experiments.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The single most load-bearing concern is an evaluation-protocol mismatch, not training convergence. Section 3.6 specifies the production pipeline: for a desired viewer image z, the compensation image z* is generated by feeding z to the (simplified) network, consistent with Eq. 6, x* = F†(T^{-1}(x);...). Section 3.8, however, reports y_i_hat = π†_θ(~y_i; ~s) and compares y_i_hat with y_i, where ~y_i is the camera capture obtained by projecting y_i. Since training supervision in Eq. 9 pairs ~x_i with x_i, the network is trained as an inverse of T∘F: given the capture ~y_i, it should recover the projector input y_i. That is inverse consistency on the range of the forward process. It is not the same as compensation, which at test time requires feeding the desired image y_i (or its FOV-fitted version) and then projecting the output. By feeding ~y_i instead of y_i, the evaluation avoids the actual deployment distribution and does not test the Fig. 5 pipeline. Even if all baselines use the same protocol, the reported PSNR/RMSE/SSIM in Table 1 may measure a proxy quantity, so the central empirical claim of outperforming two-step methods is not yet demonstrated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CompenNet++, an end-to-end neural network for full projector compensation on nonplanar textured surfaces. It extends the authors' earlier CompenNet by adding a WarpingNet subnet that learns a cascaded coarse-to-fine geometric warp directly from photometric sampling images, concatenating it with a photometric compensation subnet, and training both jointly. A benchmark with 20 setups is introduced, and experiments compare against TPS-, Pix2pix-, and CompenNet-based two-step baselines. The paper also describes task-specific weight initialization, network simplification for test-time efficiency, and reports qualitative and quantitative results claimed to outperform prior methods. The source code and benchmark are released.","tokens_in":12745,"tokens_out":10447,"duration_ms":99739,"significance":"If the empirical claims are confirmed, this would be a useful step: the first end-to-end trainable full compensation pipeline, demonstrated on a public benchmark with 20 setups, with code release. The joint-training idea is principled, and the coarse-to-fine grid refinement and test-time simplification are sensible engineering contributions. The paper explicitly ships reproducible code and a new benchmark, which are strengths. However, the evaluation protocol in Section 3.8 does not directly test the production pipeline described in Section 3.6, and the reported improvements over CompenNet with structured light are small and reported without error bars. The central comparative claim therefore needs additional evidence before the paper can be accepted.","major_comments":[{"comment":"Section 3.8 defines the reported metric as \\hat{y}_i = \\pi^\\dagger_\\theta(\\tilde{y}_i; \\tilde{s}) compared with y_i, where \\tilde{y}_i is the camera capture obtained by projecting y_i. This evaluates the inverse map on the range of the forward imaging process, i.e., reconstruction of the projector input from its capture. The production pipeline in Section 3.6 and Fig. 5 instead feeds the desired viewer image z (after affine fit z' = A z) into the simplified network and projects the output. A desired image z is not generally a camera capture \\tilde{y}_i, so it lies outside the input distribution used in Eq. (9). Table 1 therefore does not directly measure full compensation quality for arbitrary desired images; it measures inverse consistency on training-distribution captures. A direct project-and-capture experiment is required: for held-out desired images z, compute z* = \\pi'^\\dagger_\\theta(z'), project z*, capture the result, and compare it with z, for all baselines. Without that, the abstract's claim of outperforming two-step methods is not supported by the reported numbers.","section":"Section 3.8, Eq. (9), Table 1, Fig. 5"},{"comment":"Section 3.5 states that the test-time simplification is made 'without any sacrifice in performance quality,' and Section 3.6 uses the simplified \\pi'^\\dagger_\\theta at deployment, but no experiment compares \\pi'^\\dagger_\\theta with the un-simplified \\pi^\\dagger_\\theta under identical training. The rows CompenNet++ fast/faster in Table 1 use different training-set sizes and iteration counts, so they are not an ablation of the simplification. Please add an ablation that trains once and evaluates both the full and simplified networks on the same data.","section":"Section 3.5"},{"comment":"Table 1 reports averages over 20 setups but no per-setup variances or significance tests. At #Train=500, CompenNet++ attains PSNR 21.11 versus 20.96 for CompenNet w/ SL and SSIM 0.727 versus 0.712; these differences are small relative to likely cross-setup variation. Without error bars, paired tests, or per-setup results, the claim of 'clear advantages' is not quantitatively established. Reporting the standard deviation over the 20 setups, or a per-setup table or scatter plot, would address this.","section":"Table 1"}],"minor_comments":[{"comment":"There is a likely typo: 'context-aware ones typically assume pro-cam pixels one-to-one mapping and context-aware ones also consider neighborhood/global information' should read 'context-independent ones also consider neighborhood/global information.'","section":"Section 2.2"},{"comment":"The symbol x is used both for a projector input image and for a desired perceived image (e.g., Eq. (3) versus Eq. (7)). Please use separate symbols or state explicitly that the same symbol is intentionally reused.","section":"Sections 3.1 and 3.2"},{"comment":"Figure 5 is underspecified: it should state which tensor is the input to \\pi'^\\dagger_\\theta, in which coordinate frame z' lives, and how the affine transform A is applied.","section":"Fig. 5"},{"comment":"The initialization in Eq. (13) uses a synthetic surface image \\dot{s} that 'mimics' the warped surface image, but the paper does not describe how \\dot{s} is generated; a sentence or reference is needed for reproducibility.","section":"Section 3.4, Eq. (13)"},{"comment":"The statement 'we use 42 less images than two-step SL-based method' should be qualified: at #Train=500 the total is 500 versus 542, while at #Train=48 the difference is 42, so the claim depends on the training-set size.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is a reasonable incremental extension of the authors' own CompenNet, with code and a benchmark released. The main risk is the evaluation-protocol mismatch in Table 1, which does not test the production pipeline described in Section 3.6. If the authors can add a direct project-and-capture evaluation and a simplification ablation, the paper could be suitable for publication. I would not reject on novelty alone; however, the current reported numbers do not establish the headline comparative claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's genuinely new: CompenNet++ is the first end-to-end network for full projector compensation that jointly learns the geometric warp and the photometric transfer. The WarpingNet design—cascaded affine, TPS, then a learned residual grid—is a clean way to make the warp learnable without structured light, and the simplification of the trained nets into a single grid plus biases is a neat practical trick. The authors also release code and a 20-setup benchmark, which is a public good for the projection-mapping community. The qualitative examples are believable: SL-based baselines show decoding artifacts in specular regions, and the honest failure case on self-occlusion is a point in the paper's favor.\n\nThe soft spot is the evaluation protocol, and it is load-bearing. In Section 3.8, the metric is computed on \\hat{y}_i = π†(\\tilde{y}_i; \\tilde{s}) compared to y_i, where \\tilde{y}_i is the camera capture of the projector input y_i. That measures inverse consistency on the range of the forward image formation process, not the compensation pipeline of Fig. 5, which feeds the desired image z into the network and projects the output. Those are different input distributions, and the paper never does the direct project-and-capture test of the full pipeline. So the central claim that CompenNet++ outperforms two-step methods is not yet empirically demonstrated. The numerical margins over CompenNet w/ SL are small anyway (21.11 vs 20.96 PSNR at 500 samples), and none of the numbers come with error bars across the 20 setups. The Section 3.5 claim that simplification causes 'no performance drop' is also asserted rather than tested.\n\nNone of this kills the paper. The architecture is interesting, the benchmark is useful, and the evaluation gap is fixable. But a referee should ask for the direct compensation experiment, per-setup variance, and a real ablation of the simplified model. I'd send it to review.","headline":"First end-to-end full projector compensation, but the headline comparison runs on a proxy metric; worth reviewing with a demand for a direct project-and-capture test.","tokens_in":13255,"tokens_out":3267,"would_cite":true,"duration_ms":32805,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes CompenNet++, the first end-to-end network that performs full projector compensation—geometric correction plus photometric compensation—jointly, learning the geometric warp directly from photometric sampling images…","keywords":["projector compensation","geometric correction","photometric compensation","end-to-end learning","convolutional neural network","thin plate spline","structured light","projection mapping"],"falsifier":"Measure the learned geometric warp against an independent structured-light ground truth on a surface with a deep concavity or strong interreflection; if the compensation in the occluded region stays plain gray or the reprojection error exceeds that of a two-step SL baseline, the joint-training claim fails. Alternatively, ablate the grid refinement network while keeping photometric compensation fixed on a specular surface; if the end-to-end advantage disappears, the mechanism is the refinement network rather than the joint formulation.","tokens_in":12206,"feed_emoji":"📽️","tokens_out":6218,"duration_ms":53210,"temperature":0.7,"pith_summary":"This paper claims that full projector compensation—correcting where pixels land on a nonplanar textured surface and how the surface changes their colors—should be solved as one learning problem, not as two separate calibration steps. It proposes CompenNet++, a single end-to-end network that concatenates a WarpingNet for geometric correction with a CompenNet for photometric compensation. If the claim holds, projector systems can skip the structured-light scan that two-step methods require, avoid error propagation between the two stages, and run faster at test time after network simplification. The reported experiments on 20 nonplanar textured setups, including a surface behind glass with specular reflections, support the advantage over two-step baselines.","feed_headline":"One network fixes projector geometry and color together","feed_subtitle":"CompenNet++ learns the geometric warp and photometric correction from the same sampling images, skipping structured-light scans.","key_machinery":"The carrying object is WarpingNet, the geometric correction subnet: a cascaded coarse-to-fine sampling grid predictor. It stacks (1) a learnable $2\\times3$ affine matrix that maps the projector field of view toward a front view, (2) a thin-plate-spline grid with 76 learnable parameters for a smooth nonlinear warp, and (3) a grid-refinement network, built on a U-Net-like encoder-decoder, that adds residual displacements to the grid. Instead of resampling the image three times, the finer grids sample the coarser grids in coordinate space, so the input image is warped once with a single bilinear interpolation; this keeps the result sharp and reduces computation. After training, the whole geometric stage collapses to one sampling grid, and the photometric surface-feature branch collapses to additive biases.","core_discovery":"The central claim is that a deep network can learn the composite geometric warp and photometric transfer of a projector-camera-surface system directly from photometric sampling images, without building a pixel-correspondence map by structured light. The paper models a camera capture as $\\tilde{x} = \\mathcal{T}(\\mathcal{F}(x;g,s))$ and trains the network to invert both factors at once, $x^* = \\mathcal{F}^{\\dagger}(\\mathcal{T}^{-1}(x);\\mathcal{T}^{-1}(\\tilde{s}))$. The geometric part is a cascaded coarse-to-fine warp: a learnable affine transform, a thin-plate-spline grid, and a residual grid-refinement network, all composed in 2D coordinate space so the image is resampled only once. The photometric part is an improved CompenNet initialized as a linear channel-independent model to avoid degenerate solutions such as plain-gray outputs. At test time both subnets simplify to a single sampling grid plus a bias-augmented photometric backbone, with no measured performance drop.","pith_inferences":["The cascaded coordinate-space grid composition could be reused for projector-camera calibration or multi-projector blending, where geometric warps are currently estimated from markers or structured light.","The initialization recipe—starting the photometric subnet as a linear channel-independent subtractor—is a general defense against degenerate vacuous solutions in end-to-end image-to-image training, and could be tested on other compensation networks.","On surfaces with strong interreflections or global light transport, conditioning only on a single surface image may be insufficient; a testable extension would feed multiple surface images captured under different projector inputs.","A direct geometric evaluation against structured-light ground truth (for example reprojection error) would isolate how much of the end-to-end gain comes from better warping versus better photometric modeling, since the paper reports only indirect compensation quality."],"forward_implications":["Projector systems no longer need a structured-light scan: the geometric warp is learned from the same photometric sampling images used for color compensation, saving the 42 extra projected patterns used in the comparisons.","Errors stop propagating from geometric mapping into photometric compensation; joint training can tolerate specular highlights and reflections that corrupt structured-light decoding.","At test time the trained system reduces to one image resampling and one pass through the photometric backbone, which is faster and uses less memory than the training-time network.","Even with only 48 training images and shortened training, the method can keep a structural-similarity advantage over two-step baselines trained on 500 images.","The joint formulation provides a template for other projector-camera tasks where geometry and appearance are entangled and traditionally handled in sequence."],"supporting_citations":[{"why":"Supplies the photometric compensation subnet CompenNet that this work extends, together with its training loss and evaluation protocol.","marker":"[15]"},{"why":"Provides the TPS photometric compensation baseline whose plain-color sampling motivates the textured sampling strategy.","marker":"[10]"},{"why":"The structured-light method that provides geometric pixel correspondence for all two-step baselines.","marker":"[22]"},{"why":"The image-to-image translation model used as a deep learning-based two-step baseline.","marker":"[17]"},{"why":"The spatial-transformer idea that motivates learning sampling grids directly from images.","marker":"[18]"},{"why":"The encoder-decoder architecture used inside the grid refinement network.","marker":"[32]"},{"why":"The linear context-independent radiometric model that inspires the task-specific CompenNet initialization.","marker":"[25]"},{"why":"The combined L1 and SSIM loss function adopted for joint training.","marker":"[40]"}],"fun_headline_variants":["End-to-end warp and color fix for projectors","No structured light: one net compensates projectors","Joint geometric and photometric projector compensation","Learn projector warp and color from one image set","One network inverts projector geometry and shading"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that jointly training the warp and the photometric network converges to a useful solution from the proposed initialization; if the photometric loss gives insufficient gradient signal toward the correct warp, the output degrades to a plain gray image, and the paper also assumes every surface patch is directly illuminated by the projector, so self-occluded regions fail.","fun_headline_variants_meta":{"raw":{"variants":["End-to-end warp and color fix for projectors","No structured light: one net compensates projectors","Joint geometric and photometric projector compensation","Learn projector warp and color from one image set","One network inverts projector geometry and shading"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1323,"prompt_tokens":948,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":564,"tokens_out":375,"duration_ms":3369,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:51:16.379767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the learned geometric warp against an independent structured-light ground truth on a surface with a deep concavity or strong interreflection; if the compensation in the occluded region stays plain gray or the reprojection error exceeds that of a two-step SL baseline, the joint-training claim fails. Alternatively, ablate the grid refinement network while keeping photometric compensation fixed on a specular surface; if the end-to-end advantage disappears, the mechanism is the refinement network rather than the joint formulation.","supporting_citations":[{"cited_title":"End-to-end projector pho- tometric compensation","cited_arxiv_id":null,"evidence_quote":"Supplies the photometric compensation subnet CompenNet that this work extends, together with its training loss and evaluation protocol."},{"cited_title":"Robust, error-tolerant photometric projector compensation","cited_arxiv_id":null,"evidence_quote":"Provides the TPS photometric compensation baseline whose plain-color sampling motivates the textured sampling strategy."},{"cited_title":"Simple, accurate, and robust projector-camera calibration","cited_arxiv_id":null,"evidence_quote":"The structured-light method that provides geometric pixel correspondence for all two-step baselines."},{"cited_title":"Image-to-image translation with conditional adver- sarial networks","cited_arxiv_id":null,"evidence_quote":"The image-to-image translation model used as a deep learning-based two-step baseline."},{"cited_title":"Spatial transformer networks","cited_arxiv_id":null,"evidence_quote":"The spatial-transformer idea that motivates learning sampling grids directly from images."},{"cited_title":"U-net: Convolutional networks for biomedical image segmentation","cited_arxiv_id":null,"evidence_quote":"The encoder-decoder architecture used inside the grid refinement network."},{"cited_title":"A projection system with radiometric com- pensation for screen imperfections","cited_arxiv_id":null,"evidence_quote":"The linear context-independent radiometric model that inspires the task-specific CompenNet initialization."},{"cited_title":"Loss functions for image restoration with neural networks","cited_arxiv_id":null,"evidence_quote":"The combined L1 and SSIM loss function adopted for joint training."}],"review_version":1}