REVIEW 3 major objections 5 minor 38 references
Onion-Peel Networks for Deep Video Completion
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The onion-peel network claims flow-free video completion that matches the flow-based state of the art at more than 50x the speed.
desk verdict Solid empirical video completion paper with a genuinely nice progressive-filling idea and honest reporting; the 'best on all metrics' claim leans too heavily on a synthetic benchmark, but the user study and runtime advantage carry it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the recurrence over peels of the hole, coupled with an asymmetric attention block. The peel is the set of hole pixels within Euclidean distance $p=8$ of a valid pixel; the network fills only those pixels, erodes the mask, and repeats until the hole is empty. The asymmetric attention block restricts the target side to the peel pixels and the reference side to valid (non-hole) pixels, computes cosine-similarity attention scores over all reference key features at once, and uses the softmax scores to retrieve value features that are added to the target feature map before the decoder reconstructs the peel. Key features encode appearance for matching, while value features carry the color and content to be copied. Because matching is done over all reference pixels, the spatial-temporal window is not limited to nearby frames, and no optical flow is computed. The architecture uses gated convolutions as its basic building block and a shared encoder for target and references, with the encoder run once on all reference frames.
What would settle it
One concrete test: build a video test set where a real object is removed and the true background is known, for example by filming the same scene with and without the object; if onion-peel PSNR and SSIM fall below the flow-based baseline on those natural boundaries and shadows, the synthetic-mask assumption that carries the paper's quantitative claims is the part that failed.
Extended reading notes
Core claim
The paper's central claim is that a feed-forward data-driven network can replace flow-based optimization for video completion without sacrificing quality. The onion-peel network processes the target frame recursively: at each step it defines the peel as the hole pixels within $p=8$ pixels of the boundary, encodes the target and reference frames with a shared gated-convolution encoder, matches the peel's key features to every valid key feature in all reference frames via softmax-normalized cosine similarity, and adds the retrieved value features to the target before a decoder reconstructs only that peel. This lets the hole shrink gradually so each new layer is filled with richer context, and it makes the temporal window effectively unlimited because any reference frame, near or far, can contribute. On the quantitative table the method reports the best PSNR, SSIM, and VFID against the single-image baseline [37] and the flow-based baseline [13]; in the user study its average rank is 1.95 versus 1.93 for [13], which the paper reads as comparable quality at more than 50x lower runtime. The method also extends, without modification, to reference-guided image completion, which the paper presents as a capability previous video completion methods lack.
Load-bearing premise
The load-bearing premise is that the synthetic holes used for training and evaluation — affine-warped object masks over Places2 and YouTube frames, and shuffled DAVIS video/mask pairs — are representative of real object removal with natural occlusion boundaries, shadows, and scene motion.
Editorial extensions
If this is right
- Because completion does not require optical flow, scenes with occlusions, shadows, and large holes that corrupt flow estimates remain tractable.
- The 15.8-second runtime per video, compared with 910 seconds for the flow-based baseline, moves video object removal from an offline optimization into an interactive setting.
- Matching across all reference pixels gives an effectively unlimited temporal window, so content visible only in distant frames is retrievable, unlike recurrent methods with a five-frame window.
- The same network, unchanged, completes a target image from several reference photos, covering an image-editing use case that flow-based video completion handles poorly.
- A learned temporal-consistency post-processor smooths flicker but blurs frames, making the stability-sharpness trade-off explicit.
Reading between the lines
- A likely extension is to make the peel width adaptive rather than fixed at $p=8$, trading iterations against runtime on small versus large holes.
- The asymmetric attention block is a generic masked-query retrieval module and could be applied to video object segmentation, frame interpolation, or multi-view texture transfer.
- If real object-removal scenes differ from affine-warped synthetic masks, the paper's parity with flow-based methods may not transfer; a test with filmed clean backgrounds would reveal whether it does.
- Memory scales with the number of reference pixels, so long videos will need reference-frame indexing or sparsification to keep the attention scan feasible.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Onion-Peel Networks (OPN) for video completion and reference-guided image completion. The method recursively fills a target hole one boundary peel at a time, using an encoder-decoder with gated convolutions and an asymmetric attention block that matches peel-region features in the target against valid-region features in reference frames. The authors claim flow-free operation, an unlimited spatial-temporal reference window, and quality comparable to the optimization-based state of the art at a fraction of the runtime. Evidence includes a user study on 24 DAVIS videos, quantitative results on 26 synthetic videos (PSNR, SSIM, VFID), qualitative comparisons, and an ablation of the onion-peeling schedule.
Significance. If the central claims are validated, the paper offers a practically meaningful advance: video completion at roughly 15.8 seconds per video versus 910 seconds for Huang et al., with no optical-flow computation and with an architecture that also handles reference-guided image completion. The paper is clearly written, the design choices (gated convolutions, asymmetric attention, progressive boundary filling) are well motivated, the synthetic training pipeline is described in detail, and the user study provides an external subjective benchmark. However, the load-bearing comparative evidence is currently fragile: the only real-video evidence is a close, statistically untested user-study rank difference, and the quantitative superiority is measured on synthetic holes that closely match the training mask distribution. These limitations directly affect the central claim of state-of-the-art quality for real object removal.
major comments (3)
- [Section 4.2, Table 1] The claim of quantitative superiority rests entirely on synthetic test videos constructed by shuffling DAVIS video/mask pairs, while the training data (Section 3.3) uses the same kind of object-shaped masks from segmentation/saliency datasets, randomly deformed by affine transformations over Places2/YouTube frames. The test hole distribution is therefore closely aligned with the training distribution, which can systematically favor a learned model over a generic optimization method such as Huang et al. The paper itself concedes that 'the test videos are not real.' Because the introduction frames the task as realistic object removal, I ask for evidence that the synthetic benchmark transfers to real conditions: for example, per-video quantitative results, a separation between mask sources used in training and testing, evaluation on masks with natural occlusion boundaries or shadows, and a discussion of how the observed margins would be affected by distribution shift. Without such evidence, the headline 'best on all measures' is not a secure basis for the central claim.
- [Section 4.2, User Study] The user study reports averaged ranks of 1.93 for Huang et al. and 1.95 for OPN over 24 DAVIS videos, with no significance testing, no confidence intervals, and no per-video win/loss/tie counts. With ties permitted and 100 participants per video, it is unclear whether the 0.02 rank difference is distinguishable from noise. Since this is the only subjective evidence on real videos, the conclusion that OPN is 'comparable' to Huang et al. is not yet supported. I request a paired statistical test (e.g., Wilcoxon signed-rank on per-video mean ranks), confidence intervals for the mean ranks, and a per-video score breakdown. If a significance test shows no reliable ordering, the text should state that explicitly rather than implying parity beyond the descriptive averages.
- [Section 3.3 and Section 4.3] The network is trained with a maximum of 5 recursions, but inference recurses until each hole is completely filled, which can require substantially more than 5 steps for large holes. The paper's contribution includes the statement that 'given a sufficient number of recurrences, even a large hole can be inpainted successfully,' yet there is no quantitative evaluation of behavior beyond the training-time recursion limit or of error accumulation over many peels. The ablation in Section 4.3 compares one-shot filling with peel width p=8, but only on a single qualitative example (Figure 9). I ask for a quantitative analysis across hole sizes and recursion counts, including runtime versus quality trade-offs, to substantiate the large-hole claim that is central to the onion-peel design.
minor comments (5)
- [Section 4.3, Figure 10] The temporal-consistency post-processing is shown to improve temporal smoothness but also to blur frames; please clarify in the text which numbers in Table 1 and which user-study results include post-processing, and quantify the blurring side effect if possible.
- [Section 3.1, Algorithm 1] The while-loop condition is rendered as 'while||H j i||' and is ambiguous; it should be stated as 'while the hole region is non-empty' or defined formally as a norm or cardinality condition.
- [Section 3.3, Eq. (1)] The validity loss Lvalid is applied to the raw decoder output Xhat_j over the original valid region V; please clarify why the decoder is trained to reproduce valid pixels and how V interacts with the recursively filled regions at later recursions.
- [Section 3.1] The peel width p is set to 8 without any sensitivity analysis; since p directly controls the recurrence count and the amount of context per step, a small ablation over p would help the reader understand the trade-off.
- [Section 3.3, Eq. (3)] The loss weights are said to be adopted from [20], but the mapping between the terms in Eq. (3) and the corresponding terms in [20] is not explicit; please provide a reference or a short explanation for each weight.
Circularity Check
No circularity: the method is an empirical network evaluated on external benchmarks; no claimed derivation reduces to its inputs.
full rationale
This paper makes no analytical derivation whose output is defined by its input. The onion-peel network is an empirical architecture trained on Places2/YouTube frames with synthetic object-shaped masks and evaluated on DAVIS-based benchmarks and a user study. The quantitative test set is constructed by shuffling DAVIS video/mask pairs, which is an external source not used to fit network parameters. The loss weights are adopted from prior work [20], the peel width p=8 and maximum recursion are hand-set hyperparameters, and the temporal-consistency post-processor is inspired by an external method [19]. None of these choices are fitted to the test quantity and then reported as a prediction. The only author overlap in a cited method is VINet [17], which is used as a comparison baseline rather than as load-bearing support for the paper's own claims. The synthetic test protocol may share a mask distribution with training, but that is a benchmark-validity and generalization concern, not circularity under the required definition. There is no equation in which a predicted quantity is identical by construction to a fitted input, and no uniqueness or first-principles result is imported from the authors' own prior work. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Loss weighting coefficients =
L_peel=100, L_content=0.05, L_style=120, L_tv=0.01
- Peel width p =
8
- Maximum recurrences =
5
- Reference sampling stride =
5 frames
- Training resolution =
256x256
assumptions (4)
- domain assumption Synthetic masks obtained by random affine deformation of object masks from segmentation and saliency datasets approximate real moving objects to be removed.
- domain assumption Cosine similarity between learned key features at the peel and at non-hole reference pixels is sufficient to identify correct correspondences across distant frames without optical flow.
- domain assumption A finite number of recurrences with fixed peel width will eventually fill arbitrary large holes without error accumulation.
- domain assumption Pretrained VGG-16 features provide a perceptual space suitable for completion losses.
Cite this review
Pith. "Pith review of Onion-Peel Networks for Deep Video Completion." pith.science (2026). https://pith.science/paper/ZW2DK3NN
@misc{pith2026190808718,
author = {Pith},
title = {Pith review of: Onion-Peel Networks for Deep Video Completion},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZW2DK3NN}},
note = {Machine review of arXiv:1908.08718}
}
read the original abstract
We propose the onion-peel networks for video completion. Given a set of reference images and a target image with holes, our network fills the hole by referring the contents in the reference images. Our onion-peel network progressively fills the hole from the hole boundary enabling it to exploit richer contextual information for the missing regions every step. Given a sufficient number of recurrences, even a large hole can be inpainted successfully. To attend to the missing information visible in the reference images, we propose an asymmetric attention block that computes similarities between the hole boundary pixels in the target and the non-hole pixels in the references in a non-local manner. With our attention block, our network can have an unlimited spatial-temporal window size and fill the holes with globally coherent contents. In addition, our framework is applicable to the image completion guided by the reference images without any modification, which is difficult to do with the previous methods. We validate that our method produces visually pleasing image and video inpainting results in realistic test cases.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Patchmatch: a randomized correspondence algorithm for structural image editing
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (TOG), 28(3):24, 2009. 2, 7
work page 2009
-
[2]
Patchtable: Efficient patch queries for large datasets and applications
Connelly Barnes, Fang-Lue Zhang, Liming Lou, Xian Wu, and Shi-Min Hu. Patchtable: Efficient patch queries for large datasets and applications. ACM Transactions on Graphics (TOG), 34(4):97, 2015. 7
work page 2015
-
[3]
Global contrast based salient region detection
Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 37(3):569–582, 2015. 5
work page 2015
-
[4]
Region filling and object removal by exemplar-based im- age inpainting
Antonio Criminisi, Patrick P ´erez, and Kentaro Toyama. Region filling and object removal by exemplar-based im- age inpainting. IEEE Transactions on image processing , 13(9):1200–1212, 2004. 2
work page 2004
-
[5]
Image melding: Combining in- consistent images using patch-based synthesis
Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B Goldman, and Pradeep Sen. Image melding: Combining in- consistent images using patch-based synthesis. ACM Trans- actions on Graphics (TOG), 31(4):82–1, 2012. 7
work page 2012
-
[6]
The pascal visual object classes (voc) challenge
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303–338, 2010. 5
work page 2010
-
[7]
Im- age style transfer using convolutional neural networks
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Im- age style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016. 6
2016
-
[8]
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680,
Show all 38 references
-
[9]
Image inpainting: Overview and recent advances.IEEE signal processing mag- azine, 31(1):127–144, 2014
Christine Guillemot and Olivier Le Meur. Image inpainting: Overview and recent advances.IEEE signal processing mag- azine, 31(1):127–144, 2014. 2
2014
-
[10]
Semantic contours from inverse detectors
Bharath Hariharan, Pablo Arbel ´aez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In IEEE International Conference on Computer Vision (ICCV), pages 991–998. IEEE, 2011. 5
2011
-
[11]
Scene completion using millions of photographs
James Hays and Alexei A Efros. Scene completion using millions of photographs. ACM Transactions on Graphics (TOG), 26(3):4, 2007. 2
2007
-
[12]
Image completion using planar structure guid- ance
Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Jo- hannes Kopf. Image completion using planar structure guid- ance. ACM Transactions on graphics (TOG) , 33(4):129,
-
[13]
Temporally coherent completion of dynamic video
Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Jo- hannes Kopf. Temporally coherent completion of dynamic video. ACM Transactions on Graphics (TOG), 35(6), 2016. 1, 2, 6, 7, 8
2016
-
[14]
Globally and Locally Consistent Image Completion
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and Locally Consistent Image Completion. ACM Transactions on Graphics (Proc. of SIGGRAPH 2017) , 36(4):107:1–107:14, 2017. 1, 2, 4
2017
-
[15]
A survey on data-driven video completion
Shachar Ilan and Ariel Shamir. A survey on data-driven video completion. In Computer Graphics Forum, volume 34, pages 60–85. Wiley Online Library, 2015. 2
2015
-
[16]
Perceptual losses for real-time style transfer and super-resolution
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision , pages 694–711. Springer, 2016. 6
2016
-
[17]
Deep video inpainting
Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ,
-
[18]
Adam: A method for stochastic optimization
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. 6
2015
-
[19]
Learning blind video temporal consistency
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In European Conference on Computer Vision, 2018. 6
2018
-
[20]
Reda, Kevin J
Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, and Andrew Tao Bryan Catanzaro. Image inpainting for ir- regular holes using partial convolutions. In IEEE Interna- tional Conference on Computer Vision (ICCV) , 2018. 1, 2, 6
2018
-
[21]
Exemplar-based image inpainting using multiscale graph cuts
Yunqiang Liu and Vicent Caselles. Exemplar-based image inpainting using multiscale graph cuts. IEEE transactions on image processing, 22(5):1699–1711, 2013. 2
2013
-
[22]
Patchmatch-based content completion of stereo image pairs
Bryan Morse, Joel Howard, Scott Cohen, and Brian Price. Patchmatch-based content completion of stereo image pairs. In 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, pages 555–562. IEEE, 2012. 7
2012
-
[23]
Video inpainting of complex scenes
Alasdair Newson, Andrs Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Prez. Video inpainting of complex scenes. SIAM Journal on Imaging Sciences, Society for In- dustrial and Applied Mathematics, 7(4):1993–2019, 2014. 2
1993
-
[24]
Non-local patch-based image inpainting
Alasdair Newson, Andr ´es Almansa, Yann Gousseau, and Patrick P´erez. Non-local patch-based image inpainting. Im- age Processing On Line, 7:373–385, 2017. 2
2017
-
[25]
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1, 2
2016
-
[26]
A benchmark dataset and evaluation methodology for video object segmentation
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2016. 7, 8
2016
-
[27]
Hierar- chical image saliency detection on extended cssd
Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia. Hierar- chical image saliency detection on extended cssd. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(4):717–729, 2016. 5
2016
-
[28]
Very deep convo- lutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 6
2014 arXiv
-
[29]
End- to-end memory networks
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End- to-end memory networks. In Advances in neural information processing systems, pages 2440–2448, 2015. 4
2015
-
[30]
Video inpainting by jointly learning temporal structure and spatial details
Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learning temporal structure and spatial details. In Proceedings of the 33th AAAI Conference on Artificial Intelligence, 2019. 1, 3
2019
-
[31]
Stereoscopic inpainting: Joint color and depth completion from stereo images
Liang Wang, Hailin Jin, Ruigang Yang, and Minglun Gong. Stereoscopic inpainting: Joint color and depth completion from stereo images. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008. 7
2008
-
[32]
Video-to- video synthesis
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to- video synthesis. In Advances in neural information process- ing systems, 2018. 8
2018
-
[33]
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- ing He. Non-local neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 4
2018
-
[34]
Space- time video completion
Yonatan Wexler, Eli Shechtman, and Michal Irani. Space- time video completion. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition, 2004. 2
2004
-
[35]
Youtube- vos: A large-scale video object segmentation benchmark
Ning Xu, Linjie Yang, Dingcheng Yue, Jianchao Yang, Yuchen Fan, Yuchen Liang, and Thomas Huang. Youtube- vos: A large-scale video object segmentation benchmark. In arXiv preprint arXiv:1809.03327, 2018. 6, 7
2018 arXiv
-
[36]
Free-form image inpainting with gated convolution
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. arXiv preprint arXiv:1806.03589, 2018. 1, 2, 4
2018 arXiv
-
[37]
Generative image inpainting with con- textual attention
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with con- textual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5505– 5514, 2018. 1, 2, 4, 6, 7, 8
2018
-
[38]
Places: A 10 million image database for scene recognition
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2017. 5, 7
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.