REVIEW 3 major objections 5 minor 47 references
Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper introduces visual deprojection and claims that a conditional variational autoencoder can recover the distribution of plausible originals behind a single collapsed-dimension projection.
desk verdict The paper defines a genuinely new task and a sensible CVAE framework for it, but the central claim that samples are consistent with the input projection is neither enforced by the loss nor properly evaluated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a conditional variational autoencoder combined with a known linear averaging projection operator, $x = \sum_{k=1}^{d_p} \omega_k y_{p=k}$. A strided 3D convolutional encoder models the variational posterior $q_\psi(z|y)$, a 2D convolutional encoder models the conditional prior $p_\phi(z|x)$, and a UNet-style decoder $g_\theta(x,z)$ expands the projection along the collapsed dimension and refines the volume with 3D convolutions. The reparameterized ELBO loss ties test-time sampling to reconstruction accuracy and prior coverage, making the ambiguity of deprojection an explicit part of the model.
What would settle it
Run the trained model on projections corrupted by sensor noise or an unknown blur kernel: if the sampled originals' reprojections consistently fail to match the observed projection, or if across many test cases the ground-truth original is rarely among the samples, the learned distribution does not cover the true posterior.
Extended reading notes
Core claim
Visual deprojection is stated as the task of estimating $p(y|x)$ for signals $y$ collapsed along a dimension by a known projection $x = f_\omega(y)$; the paper proposes a conditional variational autoencoder in which a latent variable $z$ captures variability of $y$ not explainable by $x$, a prior $p_\phi(z|x)$ maps projections to latent distributions, and a decoder $g_\theta(x,z)$ synthesizes originals. Training maximizes an evidence lower bound whose loss is a $\beta$ KL divergence between posterior and conditional prior plus reconstruction error, with an optional perceptual term for faces. At test time, sampling $z$ from the learned conditional prior and decoding produces multiple plausible originals rather than a single deterministic answer. The paper reports that this sampling strategy recovers face appearances from 1D projections, gait videos from vertical spacetime projections, and digit videos from heavily motion-blurred images, with maximum-signal PSNR improving as more samples are drawn.
Load-bearing premise
The method assumes that the projection is a known, noiseless averaging of pixels along one dimension, identical at training and test time.
Editorial extensions
If this is right
- One model can handle both spatial and temporal deprojection; any dimension collapsed by a linear average becomes a candidate target.
- Sampling at test time turns a single ambiguous projection into a distribution over outcomes, so downstream tasks can reason about multiple plausible scenes rather than one estimate.
- Recovering video from a motion-blurred image is feasible when training data covers the relevant dynamics, including the arrow-of-time ambiguity.
- Drawing more samples improves the best reconstruction achievable from the model, while deterministic baselines saturate after one output.
Reading between the lines
- A natural extension the paper leaves implicit is learning the projection operator itself, so the same ELBO objective could jointly infer blur kernels and scene distributions instead of assuming a fixed average.
- The method's dependence on training data suggests a testable extension: train on paired high-speed and long-exposure real footage and measure how sample diversity and projection consistency change with dataset breadth.
- The same conditional prior structure could apply to projectional radiography, where a 3D volume is collapsed into a 2D radiograph, though medical priors and measurement noise would need their own treatment.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces 'visual deprojection,' the task of reconstructing a higher-dimensional signal (image or video) from a lower-dimensional observation obtained by averaging along one dimension. The authors formulate a conditional variational autoencoder (CVAE) in which a latent variable z captures ambiguity in the mapping from projection x to signal y, with a learned conditional prior p_phi(z|x), a variational posterior q_psi(z|y), and a deprojection network g_theta(x,z). Training minimizes the CVAE ELBO with a reconstruction term and a KL regularization term, optionally augmented with a perceptual loss. Experiments cover spatial deprojection of face images (FacePlace), spatial deprojection of walking videos, and temporal deprojection of Moving MNIST videos from motion-blurred images. The authors report PSNR curves showing that sampling multiple candidates improves maximum signal PSNR relative to deterministic baselines, and they present qualitative results for the walking-video and Moving MNIST tasks. The central claim is that sampling from the inference network at test time yields plausible candidates consistent with the input projection.
Significance. If the claims are substantiated, the paper provides a unified probabilistic framework for an interesting and understudied task, with potential applications to corner cameras, motion blur, and other collapsed-dimension problems. The CVAE derivation is standard and correctly applied, and the paper is clearly written. The experimental design includes multiple datasets and several baselines (k-NN, deterministic network, LMMSE), which is a strength. However, the paper's most distinctive claim—that samples are 'consistent with a given input projection'—is not enforced by the objective and is only partially supported by the reported projection-PSNR curves. The lack of comparison to the closest prior work on recovering video from a single motion-blurred image also weakens the novelty claim. These issues are fixable but require additional analysis and experiments.
major comments (3)
- [Sec. 4.3 and Sec. 4.4] The training loss in Eq. (6) contains only the KL divergence and the reconstruction term ||g_theta(x,z_hat) - y||^2; there is no term such as ||f_omega(g_theta(x,z_hat)) - x||^2 that enforces consistency between generated samples and the observed projection. Because p_theta(y|x,z) in Eq. (1) is a Gaussian centered at g_theta(x,z) with no dependence on the known projection operator, samples from the model are not guaranteed to lie near the fiber {y : f_omega(y) = x}. The paper's abstract and Sec. 1 claim that sampling yields candidates 'consistent with a given input projection,' but this property is not established by the objective. The reported projection-PSNR curves in Figs. 5 and 8 indicate that DET and LMMSE often achieve higher projection PSNR than the proposed method, suggesting that the consistency issue is empirically relevant. The authors should report a direct projection-consistency metric (e.g., fraction of samples within a tolerance of the input projection) and either add an explicit consistency term to the loss or discuss why the learned model is expected to satisfy the constraint approximately.
- [Sec. 4.2] The temporal deprojection experiments (Moving MNIST) are not compared against the closest prior work on recovering video from a single motion-blurred image, namely Jin et al. [18] and Purohit et al. [30], both of which are cited in the related work. Without such a comparison, the claim that this is a 'first general method' and that it outperforms existing recovery approaches is not fully supported. In addition, the walking-video experiments in Sec. 4.3 are entirely qualitative; no quantitative metrics (e.g., PSNR, SSIM, or perceptual similarity on held-out subjects) are reported. The authors should either add quantitative evaluation for the walking videos or state explicitly that these results are preliminary qualitative demonstrations.
- [Sec. 4.2] The primary signal-reconstruction metric is the maximum PSNR over k samples. This metric rewards a method for producing a diverse or broad set of candidates and does not assess whether the distribution of samples is calibrated (e.g., whether the variability across samples matches the true posterior variability). Since the paper's contribution is explicitly probabilistic, the authors should report additional metrics that capture distributional quality, such as expected PSNR over samples, diversity measures, or a calibration curve comparing sample variability to ground-truth variability. Without such metrics, the probabilistic advantage over deterministic baselines is only partially demonstrated.
minor comments (5)
- [Sec. 4.1] Equation (5) writes the log-likelihood as ||g_theta(x,z_hat) - y||^2 / (2 sigma_y^2) + const., but Eq. (6) drops the 1/(2 sigma_y^2) factor without explaining how sigma_y^2 is set. Please state the value or how it is absorbed into beta.
- [Sec. 3.2] The hyperparameter beta is selected so that the KL term lies in [5,15] on validation, and the latent dimension is fixed to 10. No sensitivity analysis is provided. A brief discussion of how these choices affect sample diversity versus reconstruction fidelity would improve reproducibility.
- [Fig. 2] The caption of Fig. 2 mentions 'N is the number of test examples in the dataset,' but N does not appear in the figure. Please clarify the notation or remove the reference.
- [References] Several references lack publication venues or years (e.g., [17], [42], [45]). Please complete the bibliographic details.
- [Sec. 4.3] The walking dataset description says 35 videos of 30 subjects, with 6 subjects held out, but does not specify the train/validation split used for hyperparameter selection. Please clarify.
Circularity Check
No significant circularity: the derivation is a standard supervised CVAE trained on held-out pairs, with no fit-to-test or self-citation carrying the central claim.
full rationale
The paper's derivation chain is self-contained against its stated task. It defines a known, noiseless linear projection x = f_omega(y) (Sec. 3), introduces a conditional VAE with prior p_phi(z|x) and decoder g_theta(x,z), and optimizes the ELBO of Eq. (6) on paired training data. At test time, samples are drawn from p_phi(z|x) without access to the ground-truth y, and evaluation is performed on held-out test projections. There is no equation in which the target result is an input by construction, and no fitted parameter is renamed as a prediction: the hyperparameters (beta, latent dimension) are selected on validation data, and the reported PSNR curves are computed on test projections. The paper's self-citations (corner cameras, arrow of time, stochastic video prediction) appear in related-work context and are not load-bearing for the proposed method; no uniqueness theorem or prior-work ansatz is invoked to force the model choice. The skeptic concern that samples are not explicitly constrained to satisfy f_omega(g_theta(x,z)) = x is a correctness and generalization issue about whether the learned posterior covers the true support, not a circularity in the derivation. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- beta (KL regularization weight) =
Not reported; tuned per dataset to keep KL in [5, 15] on validation
- Latent dimension z =
10
- Per-pixel noise variance sigma_y^2 =
Not reported
- Perceptual loss weight =
Not reported
assumptions (4)
- domain assumption The projection operator is a known linear average along one dimension: x = sum_k omega_k y_{p=k}.
- domain assumption The conditional likelihood p_theta(y | x, z) is Gaussian with fixed per-pixel variance.
- domain assumption The latent variable z captures all ambiguity in p(y|x), and the Gaussian priors p_phi(z|x), q_psi(z|y) are flexible enough to represent the needed distributions.
- domain assumption Training and test signals are drawn from the same distribution, and test projections use the same averaging operator.
Cite this review
Pith. "Pith review of Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions." pith.science (2026). https://pith.science/paper/JMLRW2EZ
@misc{pith2026190900475,
author = {Pith},
title = {Pith review of: Visual Deprojection: Probabilistic Recovery of Collapsed Dimensions},
year = {2026},
howpublished = {\url{https://pith.science/paper/JMLRW2EZ}},
note = {Machine review of arXiv:1909.00475}
}
read the original abstract
We introduce visual deprojection: the task of recovering an image or video that has been collapsed along a dimension. Projections arise in various contexts, such as long-exposure photography, where a dynamic scene is collapsed in time to produce a motion-blurred image, and corner cameras, where reflected light from a scene is collapsed along a spatial dimension because of an edge occluder to yield a 1D video. Deprojection is ill-posed-- often there are many plausible solutions for a given input. We first propose a probabilistic model capturing the ambiguity of the task. We then present a variational inference strategy using convolutional neural networks as functional approximators. Sampling from the inference network at test time yields plausible candidates from the distribution of original signals that are consistent with a given input projection. We evaluate the method on several datasets for both spatial and temporal deprojection tasks. We first demonstrate the method can recover human gait videos and face images from spatial projections, and then show that it can recover videos of moving digits from dramatically motion-blurred images obtained via temporal projection.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[18]
Learning to Extract a Video Sequence from a Single Motion-Blurred Image
Meiguang Jin, Givi Meishvili, and Paolo Favaro. Learning to extract a video sequence from a single motion-blurred image. arXiv preprint arXiv:1804.04065, 2018. 2, 7
work page Pith review arXiv 2018
-
[30]
Bring- ing alive blurred moments
Kuldeep Purohit, Anshul Shah, and AN Rajagopalan. Bring- ing alive blurred moments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 6830–6839, 2019. 2
work page 2019
-
[1]
Tensorflow: A system for large-scale machine learning
Mart ´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe- mawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementa- tion ({OSDI} 16), pages 265–283, 2016. 4
work page 2016
-
[2]
Stochastic variational video prediction
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv:1710.11252, 2017. 2
arXiv 2017
-
[3]
Depth in- formation in human gait analysis: An experimental study on gender recognition
Ricard Borr `as, `Agata Lapedriza, and Laura Igual. Depth in- formation in human gait analysis: An experimental study on gender recognition. In International Conference Image Anal- ysis and Recognition, pages 98–105. Springer, 2012. 7
work page 2012
-
[4]
Turning corners into cameras: Princi- ples and methods
Katherine L Bouman, Vickie Ye, Adam B Yedidia, Fr ´edo Durand, Gregory W Wornell, Antonio Torralba, and William T Freeman. Turning corners into cameras: Princi- ples and methods. In International Conference on Computer Vision, volume 1, page 8, 2017. 1, 2
work page 2017
-
[5]
Unsupervised pixel- level domain adaptation with generative adversarial net- works
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel- level domain adaptation with generative adversarial net- works. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, page 7, 2017. 2
work page 2017
-
[6]
Signal recov- ery from random projections
Emmanuel J Candes and Justin K Romberg. Signal recov- ery from random projections. In Computational Imaging III, volume 5674, pages 76–87. International Society for Optics and Photonics, 2005. 2
work page 2005
Show all 47 references
-
[7]
Near-optimal sig- nal recovery from random projections: Universal encod- ing strategies? IEEE transactions on information theory , 52(12):5406–5425, 2006
Emmanuel J Candes and Terence Tao. Near-optimal sig- nal recovery from random projections: Universal encod- ing strategies? IEEE transactions on information theory , 52(12):5406–5425, 2006. 2
2006
-
[8]
An introduction to compressive sampling
Emmanuel J Cand `es and Michael B Wakin. An introduction to compressive sampling. IEEE signal processing magazine, 25(2):21–30, 2008. 2
2008
-
[9]
Coherent online video style transfer
Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online video style transfer. InProc. Intl. Conf. Computer Vision (ICCV), 2017. 2
2017
-
[10]
Keras.https://keras.io, 2015
Franc ¸ois Chollet et al. Keras.https://keras.io, 2015. 4
2015
-
[11]
Motion from blur
Shengyang Dai and Ying Wu. Motion from blur. 2008. 1
2008
-
[12]
Compressed sensing
David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4):1289–1306, 2006. 2
2006
-
[13]
Generating im- ages with perceptual similarity metrics based on deep net- works
Alexey Dosovitskiy and Thomas Brox. Generating im- ages with perceptual similarity metrics based on deep net- works. In Advances in Neural Information Processing Sys- tems, pages 658–666, 2016. 3
2016
-
[14]
Removing camera shake from a single photograph
Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman. Removing camera shake from a single photograph. In ACM transactions on graphics (TOG), volume 25, pages 787–794. ACM, 2006. 2
2006
-
[15]
Example-based super-resolution
William T Freeman, Thouis R Jones, and Egon C Pasztor. Example-based super-resolution. IEEE Computer graphics and Applications, 22(2):56–65, 2002. 2
2002
-
[16]
Signal reconstruction from noisy random projections
Jarvis Haupt and Robert Nowak. Signal reconstruction from noisy random projections. IEEE Transactions on Informa- tion Theory, 52(9):4036–4048, 2006. 2
2006
-
[17]
Image-to-image translation with conditional adver- sarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. arXiv preprint, 2017. 2
2017
-
[19]
Perceptual losses for real-time style transfer and super-resolution
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pages 694–711. Springer, 2016. 3
2016
-
[20]
An introduction to variational meth- ods for graphical models
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational meth- ods for graphical models. Machine learning, 37(2):183–233,
-
[21]
Online video deblurring via dynamic tem- poral blending network
Tae Hyun Kim, Kyoung Mu Lee, Bernhard Sch ¨olkopf, and Michael Hirsch. Online video deblurring via dynamic tem- poral blending network. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 4058–4067. IEEE,
2017
-
[22]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
-
[23]
Auto-encoding varia- tional bayes
Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114, 2013. 2, 3
2013 arXiv
-
[24]
Stochastic adversarial video prediction
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523 , 2018. 2
2018 arXiv
-
[25]
Anat Levin, Peter Sand, Taeg Sang Cho, Fredo Durand, and William T. Freeman. Motion-invariant photography. ACM Transactions on Graphics (SIGGRAPH 2008). 1
2008
-
[26]
X-ray Imaging: fundamentals, industrial techniques and applications
Clint M Logan, Daniel J Schneberk, Peter J Shull, and Harry E Martz. X-ray Imaging: fundamentals, industrial techniques and applications. CRC Press, 2016. 1
2016
-
[27]
Deep multi-scale convolutional neural network for dynamic scene deblurring
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, volume 1, page 3, 2017. 7
2017
-
[28]
Motion deblurring in the wild
Mehdi Noroozi, Paramanand Chandramouli, and Paolo Favaro. Motion deblurring in the wild. In German Confer- ence on Pattern Recognition , pages 65–77. Springer, 2017. 7
2017
-
[29]
Conditional image synthesis with auxiliary classifier gans
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. arXiv preprint arXiv:1610.09585, 2016. 2
2016 arXiv
-
[31]
Recogniz- ing disguised faces
Giulia Righi, Jessie J Peissig, and Michael J Tarr. Recogniz- ing disguised faces. Visual Cognition, 20(2):143–169, 2012. 4
2012
-
[32]
U- net: Convolutional networks for biomedical image segmen- tation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In International Conference on Medical image com- puting and computer-assisted intervention , pages 234–241. Springer, 2015. 4
2015
-
[33]
Tem- poral generative adversarial nets with singular value clip- ping
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Tem- poral generative adversarial nets with singular value clip- ping. In IEEE International Conference on Computer Vision (ICCV), volume 2, page 5, 2017. 2
2017
-
[34]
Space-time super-resolution
Eli Shechtman, Yaron Caspi, and Michal Irani. Space-time super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(4):531–545, 2005. 2
2005
-
[35]
Very deep convo- lutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 3
2014 arXiv
-
[36]
Learning structured output representation using deep conditional gen- erative models
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional gen- erative models. In Advances in Neural Information Process- ing Systems, pages 3483–3491, 2015. 3
2015
-
[37]
Unsupervised learning of video representations using lstms
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudi- nov. Unsupervised learning of video representations using lstms. In International conference on machine learning , pages 843–852, 2015. 2, 4
2015
-
[38]
Unsu- pervised cross-domain image generation
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsu- pervised cross-domain image generation. arXiv preprint arXiv:1611.02200, 2016. 2
2016 arXiv
-
[39]
Mocogan: Decomposing motion and content for video generation
Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. arXiv preprint arXiv:1707.04993 , 2017. 2
2017 arXiv
-
[40]
Generating videos with scene dynamics
Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems , pages 613–621,
-
[41]
Video-to- video synthesis
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to- video synthesis. In Advances in Neural Information Pro- cessing Systems (NIPS), 2018. 2
2018
-
[42]
High-resolution image synthesis and semantic manipulation with conditional gans
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. arXiv preprint arXiv:1711.11585, 2017. 2
2017 arXiv
-
[43]
Learning and using the arrow of time
Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8052–8060, 2018. 2
2018
-
[44]
Visual dynamics: Stochastic future generation via layered cross convolutional networks
Tianfan Xue, Jiajun Wu, Katherine Bouman, and William Freeman. Visual dynamics: Stochastic future generation via layered cross convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 2018. 2
2018
-
[45]
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. arXiv preprint, 2018. 3
2018
-
[46]
Image demosaic- ing
Ruiwen Zhen and Robert L Stevenson. Image demosaic- ing. In Color image and video enhancement , pages 13–54. Springer, 2015. 2
2015
-
[47]
Unpaired image-to-image translation using cycle- consistent adversarial networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. arXiv preprint, 2017. 2
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.