REVIEW 3 major objections 4 minor 30 references
CheapNVS: Real-Time On-Device Narrow-Baseline Novel View Synthesis
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A lightweight network renders slightly shifted viewpoints in real time, matching the quality of methods that are ten times slower.
desk verdict A genuinely efficient narrow-baseline NVS system with solid speed/memory measurements, but the accuracy claim overreaches because the inpainting target is a pseudo-labeler the model itself was trained to imitate. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the blending formulation of Equation 2, realized as Equation 4, which decouples warping from inpainting so that the flow decoder, mask decoder, and inpainting decoder can all read the same shared embedding and execute in parallel. The camera-extrinsics encoder maps the 12-D target transformation matrix to a 256-D embedding through a two-layer MLP; this is what lets the learned warping respond to a requested viewpoint instead of being fixed to one baseline. Multi-stage training, activating flow and mask first and the inpainting branch from the fifth epoch, lets the inpainting decoder specialise to the narrow-baseline occlusion patterns created by the learned warping. Skip connections from the RGBD encoder are kept for the mask and inpainting decoders but deliberately removed from the flow decoder, where the paper finds they degrade results.
What would settle it
Capture a set of narrow-baseline stereo pairs, run CheapNVS on the left RGB-D image with the right camera pose, and compare its output with the actual right image using LPIPS, PSNR, and SSIM; if scores against real targets are markedly worse than the paper's scores on synthetic targets, the central accuracy claim is not supported.
Extended reading notes
Core claim
CheapNVS's central claim is that the expensive, handcrafted depth-based warping of one-shot 3D photography can be replaced by a learned pipeline. The target view is generated as a convex blend $I_t = g_s(S,I_s)\cdot M + P\cdot(1-M)$, where $g_s$ is grid sampling, $S$ is a learned per-pixel shift map, $M$ is a learned occlusion mask, and $P$ is a dense inpainting prediction. A shared RGBD encoder and a two-layer MLP extrinsics encoder map image, depth, and the 12-D target camera transformation into a common latent space, and three parallel decoders produce $S$, $M$, and $P$ from that embedding. Training is multi-stage: flow and mask are first supervised against conventional 3D warping outputs, then the inpainting decoder is activated with supervision from a fine-tuned MI-GAN pseudo-labeler, using the same synthetic procedure for evaluation ground truth. On Open Images the paper reports inpainting LPIPS 0.088 versus AdaMPI's 0.100 and PSNR 29.33 versus 26.23, with 26 ms GPU inference versus 254 ms and 0.14 GB versus 0.15 GB memory, and 33 ms per frame on a tablet.
Load-bearing premise
The reported accuracy rests on assuming that images produced by conventional depth-based warping plus an inpainting network, generated identically for training and testing, faithfully represent what a real new camera viewpoint would show; the paper never checks this against real multi-view captures.
Editorial extensions
If this is right
- A single RGB-D image plus a requested camera motion can drive a synthesized viewpoint at interactive rates on a phone or tablet, without custom low-level warping kernels.
- Training on a large, diverse dataset such as Open Images gives better narrow-baseline synthesis than training on COCO for this architecture, suggesting data scale matters more than the specific benchmark.
- Because warping, mask, and inpainting are separate decoders over one shared embedding, each can be reused or fine-tuned independently, for example by swapping in a stronger inpainting block without retraining the warping branch.
- The method's accuracy is tied to the narrow-baseline distribution seen in training; larger viewpoint changes would require either a different warping mechanism or additional supervision, as the paper itself notes.
Reading between the lines
- Beyond the paper: because the ground truth is itself generated by conventional warping plus an inpainting network, the reported metrics may partly measure how well the model imitates that pipeline; an obvious test is to evaluate on real stereo pairs or multi-view bursts, which the paper does not do.
- Beyond the paper: the same conditioning trick, injecting a target camera transformation as an MLP embedding into a shared encoder, could transfer to neighbouring tasks such as frame interpolation or small-angle video stabilization, where narrow-baseline alignment is the core operation.
- Beyond the paper: a stronger inpainting teacher than the fine-tuned MI-GAN pseudo-labeler would likely raise the ceiling for the inpainting decoder, since the paper indicates that inpainting quality limits the final synthesized view.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. CheapNVS proposes a single-view narrow-baseline novel view synthesis model that replaces conventional depth-based 3D warping with lightweight learned flow and mask decoders conditioned on the target camera pose, and performs inpainting in parallel through a shared encoder. The paper introduces a multi-stage training schedule, uses Open Images for training, and reports that CheapNVS outperforms AdaMPI on LPIPS/PSNR while running roughly 10x faster and consuming 6% less memory, including real-time operation on a Samsung Tab 9+. Evaluation is performed on Open Images and COCO using LPIPS, PSNR, SSIM, and runtime/memory measurements.
Significance. If the results hold, CheapNVS offers a practical efficiency improvement: the shared-encoder architecture with parallel decoders is simple, hardware-friendly, and the measured 26 ms GPU / 33 ms mobile latency with 0.14 GB inference memory are concrete and useful. The ablations in Table II are informative about the contribution of each loss term and skip-connection choice. However, the central accuracy claim is not yet established with the same rigor: the inpainting ground truth is generated by a network that CheapNVS itself is trained to imitate, the only baseline is AdaMPI, and no significance testing or real multi-view validation is provided.
major comments (3)
- [IV-A, Table I] The accuracy comparison is self-referential. Section IV-A states that the inpainting ground truths are produced by a fine-tuned MI-GAN pseudo-labeler, and that for evaluation "We generate the ground-truths the same way as we do in training." Since the CheapNVS inpainting decoder is trained with L1 loss directly against this pseudo-labeler's outputs, the reported LPIPS/PSNR gains over AdaMPI in Table I (e.g., 0.088 vs 0.100 LPIPS on Open Images) may measure how well CheapNVS imitates MI-GAN rather than how well it synthesizes true novel views. AdaMPI is not optimized against this target. To support the "outperforms SOTA" claim, the authors should validate on real multi-view captures or on ground truths generated by a different inpainting method not used in training; at a minimum, the bias should be acknowledged and the claim softened.
- [III-B Eq. (2), III-C Eq. (4)] The mask notation is internally inconsistent. In Section III-A, M is defined as the occlusion mask "which indicates the areas to be filled in the warped image." Equation (2) then multiplies the warped image w by ϕ and the inpainting output f by (1−ϕ), and Equation (4) multiplies the warped RGB by M and the inpainting by (1−M). Both equations are the reverse of what the stated definition of an occlusion mask would imply. Either M and ϕ are valid-pixel masks and the text should say so, or the equations should be swapped (M ↔ 1−M). This is central to the formulation and must be fixed before the method can be assessed.
- [Table I, Abstract] The "outperforms the state-of-the-art" claim is too strong given the reported numbers. In Table I, CheapNVS has lower SSIM than AdaMPI on both test sets (0.883 vs 0.888 on Open Images; 0.895 vs 0.909 on COCO), and the text itself concedes "except SSIM scores." The abstract and conclusion drop this qualification. Moreover, no error bars, confidence intervals, or significance tests are provided, and the evaluation uses on-the-fly random ground truths without a stated seed, so the measured differences (e.g., LPIPS 0.071 vs 0.078 on COCO) are not statistically substantiated. Please either support the claim with proper uncertainty quantification and a summary metric across all criteria, or revise the wording.
minor comments (4)
- [Section IV-B, Table I] The row for AdaMPI retrained on Open Images reports much worse numbers than the original model (0.303 LPIPS vs 0.100 for the COCO-trained model), and the text says this training "fails to provide a tangible result." If this run is considered a failed training, it should not be presented as a competitive baseline; please clarify or remove it.
- [Section IV-A] The evaluation protocol generates test ground truths on-the-fly and does not state a fixed random seed or repeated-run statistics. Since the target transformations are randomly sampled, the numbers in Table I may not be reproducible; please provide the transformation generation details and either fix a seed or report variance across runs.
- [Section IV-B] The comparison set contains only AdaMPI. If other methods are "already inferior to AdaMPI in accuracy," a citation or brief quantitative support for that statement would help; as written, the claim is unverifiable.
- [Table I, Section IV-B] The runtime comparison does not state the input resolution used for the GPU and mobile latency measurements. Since training and ablations use 224x224 crops, please clarify the resolution for the reported 26 ms/33 ms values and confirm that the same resolution is used for AdaMPI.
Circularity Check
Inpainting accuracy is evaluated against the same MI-GAN pseudo-labels used as training targets, making the reported SOTA comparison partially self-referential.
-
fitted input called prediction
[Section IV-A (Experimental Details) and Table I; see also Section IV-D Limitations]
"For inpainting, we create our own pseudo-labeler; we finetune a pretrained MI-GAN [25] on warp-back data using OpenImages. This model is then used to generate inpainted images which are used as ground-truth for the inpainting decoder. ... For comparison with state-of-the-art, ... We generate the ground-truths the same way as we do in training."
CheapNVS's inpainting decoder is trained with an L1 loss directly against the MI-GAN pseudo-labels, and the Open Images/COCO inpainting columns of Table I are computed against targets generated 'the same way as we do in training.' The evaluation target is thus the same function whose outputs define the inpainting training loss. AdaMPI was not trained to match this particular fine-tuned MI-GAN teacher, so the reported LPIPS/PSNR inpainting advantage largely measures how well CheapNVS imitates the authors' chosen pseudo-labeler, not independently validated novel-view quality. Section IV-D's plan to 'improve synthesis quality by using a more powerful inpainting teacher' concedes that the teacher defines the current quality ceiling.
full rationale
The paper's architecture, warping approximation, runtime, and memory claims are internally consistent and externally checkable: warping is compared to conventional depth-based 3D warping, and 26/33 ms runtimes are direct measurements. The circular element is confined to the inpainting benchmark: the authors fine-tune MI-GAN to create pseudo-labels, train CheapNVS's inpainting decoder against those pseudo-labels, and then report inpainting LPIPS/PSNR against the same pseudo-labeler for both CheapNVS and AdaMPI. Because AdaMPI was not optimized to match this teacher, the head-to-head inpainting improvement is biased by construction and does not independently establish superior novel-view synthesis quality. This is not a fully circular derivation in the mathematical sense, and the runtime/memory portions of the central claim stand, hence a moderate score of 6 rather than 8-10.
Assumptions & free parameters
free parameters (5)
- Loss weights lambda_1, lambda_2, lambda_3 =
lambda_1=0, lambda_2=lambda_3=1 for epochs 0-4; lambda_1=lambda_2=lambda_3=1 from epoch 5
- Training schedule hyperparameters =
20 epochs, learning rate 1e-4, batch size 32, 224x224 crops
- Extrinsics embedding dimension =
256
- Narrow-baseline transformation distribution =
Not specified; borrowed from AdaMPI [13]
- Depth estimator per training set =
DPT for COCO, Marigold for Open Images
assumptions (5)
- domain assumption Conventional depth-based 3D warping is the correct ground-truth warping and target view generator.
- domain assumption The MI-GAN pseudo-labeler produces valid inpainting ground truth.
- domain assumption AdaMPI's random transformation generation defines the narrow-baseline distribution.
- domain assumption Input depth D is available at inference and is accurate enough.
- domain assumption The target camera pose T is known exactly.
Cite this review
Pith. "Pith review of CheapNVS: Real-Time On-Device Narrow-Baseline Novel View Synthesis." pith.science (2026). https://pith.science/paper/J52IFJKK
@misc{pith2026250114533,
author = {Pith},
title = {Pith review of: CheapNVS: Real-Time On-Device Narrow-Baseline Novel View Synthesis},
year = {2026},
howpublished = {\url{https://pith.science/paper/J52IFJKK}},
note = {Machine review of arXiv:2501.14533}
}
read the original abstract
Single-view novel view synthesis (NVS) is a notorious problem due to its ill-posed nature, and often requires large, computationally expensive approaches to produce tangible results. In this paper, we propose CheapNVS: a fully end-to-end approach for narrow baseline single-view NVS based on a novel, efficient multiple encoder/decoder design trained in a multi-stage fashion. CheapNVS first approximates the laborious 3D image warping with lightweight learnable modules that are conditioned on the camera pose embeddings of the target view, and then performs inpainting on the occluded regions in parallel to achieve significant performance gains. Once trained on a subset of Open Images dataset, CheapNVS outperforms the state-of-the-art despite being 10 times faster and consuming 6% less memory. Furthermore, CheapNVS runs comfortably in real-time on mobile devices, reaching over 30 FPS on a Samsung Tab 9+.
Figures
Reference graph
Works this paper leans on
-
[1]
Vision-only robot navigation in a neural radiance world,
Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager, “Vision-only robot navigation in a neural radiance world,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4606–4613, 2022
2022
-
[2]
Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction,
Sixu Li, Chaojian Li, Wenbo Zhu, Boyang Yu, Yang Zhao, Cheng Wan, Haoran You, Huihong Shi, and Yingyan Lin, “Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction,” in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1–13
2023
-
[3]
Finding waldo: Towards efficient exploration of nerf scene spaces,
Evangelos Skartados, Mehmet Kerim Yucel, Bruno Manganelli, Anasta- sios Drosou, and Albert Sa `a-Garriga, “Finding waldo: Towards efficient exploration of nerf scene spaces,” in Proceedings of the 15th ACM Multimedia Systems Conference , 2024, pp. 155–165
2024
-
[4]
Abril Corona-Figueroa, Jonathan Frawley, Sam Bond-Taylor, Sarath Bethapudi, Hubert PH Shum, and Chris G Willcocks, “Mednerf: Medical neural radiance fields for reconstructing 3d-aware ct-projections from a single x-ray,” in 2022 44th annual international conference of the IEEE engineering in medicine & Biology society (EMBC) . IEEE, 2022, pp. 3843–3848
work page 2022
-
[5]
Nerf: Representing scenes as neural radiance fields for view synthesis,
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021
work page 2021
-
[6]
3d gaussian splatting for real-time radiance field rendering.,
Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis, “3d gaussian splatting for real-time radiance field rendering.,” ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023
2023
-
[7]
Pose-free generalizable rendering transformer,
Zhiwen Fan, Panwang Pan, Peihao Wang, Yifan Jiang, Hanwen Jiang, Dejia Xu, Zehao Zhu, Dilin Wang, and Zhangyang Wang, “Pose-free generalizable rendering transformer,” arXiv e-prints , pp. arXiv–2310, 2023
work page 2023
-
[8]
Single-view view synthesis with self-rectified pseudo- stereo,
Yang Zhou, Hanjie Wu, Wenxi Liu, Zheng Xiong, Jing Qin, and Shengfeng He, “Single-view view synthesis with self-rectified pseudo- stereo,” International Journal of Computer Vision , vol. 131, no. 8, pp. 2032–2043, 2023
work page 2023
Show all 30 references
-
[9]
Pseudo-stereo for monocular 3d object detection in autonomous driving,
Yi-Nan Chen, Hang Dai, and Yong Ding, “Pseudo-stereo for monocular 3d object detection in autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 887–897
2022
-
[10]
A lightweight neural network for monocular view generation with occlusion handling,
Simon Evain and Christine Guillemot, “A lightweight neural network for monocular view generation with occlusion handling,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 43, no. 6, pp. 1832– 1844, 2019
2019
-
[11]
Passthrough+ real-time stereoscopic view synthesis for mobile mixed reality,
Gaurav Chaurasia, Arthur Nieuwoudt, Alexandru-Eugen Ichim, Richard Szeliski, and Alexander Sorkine-Hornung, “Passthrough+ real-time stereoscopic view synthesis for mobile mixed reality,” Proceedings of the ACM on Computer Graphics and Interactive Techniques , vol. 3, no. 1, pp...
2020
-
[12]
3d photography using context-aware layered depth inpainting,
Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang, “3d photography using context-aware layered depth inpainting,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8028–8038
2020
-
[13]
Single-view view synthesis in the wild with learned adaptive multiplane images,
Yuxuan Han, Ruicheng Wang, and Jiaolong Yang, “Single-view view synthesis in the wild with learned adaptive multiplane images,” in ACM SIGGRAPH 2022 Conference Proceedings , 2022, pp. 1–8
2022
-
[14]
Zero123++: a single image to consistent multi-view diffusion base model,
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su, “Zero123++: a single image to consistent multi-view diffusion base model,” arXiv preprint arXiv:2310.15110, 2023
-
[15]
Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction,
Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan, “Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction,” arXiv preprint arXiv:2...
2024 arXiv
-
[16]
One shot 3d photography,
Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, et al., “One shot 3d photography,” ACM Transactions on Graphics (TOG) , vol. 39, no. 4, pp. 76–1, 2020
2020
-
[17]
Structure-from- motion revisited,
Johannes Lutz Sch ¨onberger and Jan-Michael Frahm, “Structure-from- motion revisited,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016
2016
-
[18]
Incremental-ldi for multi-view coding,
Vincent Jantet, Luce Morin, and Christine Guillemot, “Incremental-ldi for multi-view coding,” in 2009 3DTV Conference: The True Vision- Capture, Transmission and Display of 3D Video . IEEE, 2009, pp. 1–4
2009
-
[19]
Deepview: View synthesis with learned gradient descent,
John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker, “Deepview: View synthesis with learned gradient descent,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. ...
2019
-
[20]
Adaptive mask-based pyramid network for realistic bokeh rendering,
Konstantinos Georgiadis, Albert Sa `a-Garriga, Mehmet Kerim Yucel, Anastasios Drosou, and Bruno Manganelli, “Adaptive mask-based pyramid network for realistic bokeh rendering,” in European Conference on Computer Vision . Springer, 2022, pp. 429–444
2022
-
[21]
Lra&ldra: Rethinking residual predictions for efficient shadow detection and removal,
Mehmet Kerim Y ¨ucel, Valia Dimaridou, Bruno Manganelli, Mete Ozay, Anastasios Drosou, and Albert Saa-Garriga, “Lra&ldra: Rethinking residual predictions for efficient shadow detection and removal,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer V...
2023
-
[22]
Mobilenetv2: Inverted residuals and linear bottlenecks,
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 4510–4520
2018
-
[23]
Adam: A method for stochastic optimization,
Diederik Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015
2015
-
[24]
The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale,
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al., “The open images dataset v4: Unified image classification, object detection, and visual relationship detection...
1956
-
[25]
Mi-gan: A simple baseline for image inpainting on mobile devices,
Andranik Sargsyan, Shant Navasardyan, Xingqian Xu, and Humphrey Shi, “Mi-gan: A simple baseline for image inpainting on mobile devices,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 7335–7345
2023
-
[26]
Focal frequency loss for image reconstruction and synthesis,
Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy, “Focal frequency loss for image reconstruction and synthesis,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 13919–13929
2021
-
[27]
The unreasonable effectiveness of deep features as a perceptual metric,
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595
2018
-
[28]
Vision transformers for dense prediction,
Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun, “Vision transformers for dense prediction,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 12179–12188
2021
-
[29]
Repurposing diffusion-based image generators for monocular depth estimation,
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Ro- drigo Caye Daudt, and Konrad Schindler, “Repurposing diffusion-based image generators for monocular depth estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9492–9502
2024
-
[30]
Slide: Single image 3d photography with soft layering and depth-aware inpainting,
Varun Jampani, Huiwen Chang, Kyle Sargent, Abhishek Kar, Richard Tucker, Michael Krainin, Dominik Kaeser, William T Freeman, David Salesin, Brian Curless, et al., “Slide: Single image 3d photography with soft layering and depth-aware inpainting,” in Proceedings of the IEEE/CVF...
2021
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.