Pith. sign in

REVIEW 5 major objections 4 minor 25 references

Copy-and-Paste Networks for Deep Video Inpainting

T0 review · 5 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep video inpainting can be done by a feed-forward copy-and-paste network: learned affine alignment plus masked softmax attention matches the leading optimization method's quality at about 35 times the speed, and the same pipeline lifts…

desk verdict A practical DNN video inpainting method with a genuinely useful affine+masked-softmax design that runs ~35x faster than the optimization baseline, but the evaluation is thinner than the claims and the affine assumption is the main soft spot. read the letter →

arxiv 1908.11587 v1 pith:JG6TFHSC submitted 2019-08-30 cs.CV

Sungho Lee , Seoung Wug Oh , DaeYeun Won , Seon Joo Kim This is my paper · ORCID
classification cs.CV
keywords videoinpaintingcopy-and-pastenetworksaffinealignmentmaskedsoftmaxcontextmatchingtemporalconsistencyover-exposurerestorationlanedetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video inpainting is normally solved with heavy per-video optimization, which is slow; this paper tries to establish that a feed-forward network can do it by copying valid pixels from other frames. The proposed Copy-and-Paste Networks first estimate an affine warp between each reference frame and the target frame with a self-supervised alignment network, then use masked softmax context matching to weigh and blend reference features into the hole. On DAVIS 2017 restoration and object-removal tests the method reaches PSNR 28.37 against 28.14 for the leading optimization method and a nearly equal user-study rank, while reducing runtime from about 952 seconds to 27 seconds per video. The same framework, applied to over/under-exposed road footage, lifts lane detection accuracy from 46.69% to 83.00%, suggesting the copy-and-paste idea transfers to exposure restoration tasks.

What carries the argument

The load-bearing machinery is the pair formed by the self-supervised alignment network and the masked softmax context matching module, with the iterative reference update as the third pillar. The alignment network regresses an affine matrix (a 2D linear warp plus translation) per reference frame, trained end-to-end with a visibility-masked L1 loss, so distant frames can be brought into the target's coordinate system even when both images contain holes. The context matching module measures global similarity between each aligned reference and the target in feature space, multiplies it by a visibility map, and applies a softmax across references so that every output pixel is copied from the references that best explain it; a residual mask tells the decoder which pixels are invisible in all references and therefore must be synthesized. Completing frames sequentially, writing completed frames back into the reference set, and averaging the forward and reverse passes carries temporal consistency.

What would settle it

Record a scene with strong parallax and move the camera between target and reference frames, then fill identical holes with the affine-alignment version and a per-pixel optical-flow version; if the affine version shows visible double edges or a lower PSNR in the hole, the alignment assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that video inpainting can be decomposed into two learned operations—copying valid pixels from reference frames and pasting them into the target hole—and that this decomposition reaches the quality of the best optimization-based system while running about 35 times faster. The alignment network solves frame registration with affine matrices learned by a self-supervised loss that only supervises visible pixels, which lets the model use distant frames instead of being limited to a short temporal window. The context matching module then computes a cosine similarity between target and aligned reference features, applies masked softmax over references, and hands the decoder a weighted blend of reference features plus a mask of pixels invisible everywhere; the decoder only needs to synthesize the never-visible pixels. Sequentially updating the reference set with completed frames and averaging forward and backward passes provides temporal consistency. In the paper's experiments this yields PSNR 28.37 vs 28.14 and user rank 1.77 vs 1.74 against the optimization baseline, and the exposure-restoration application improves lane detection from 46.69% to 83.00%.

Load-bearing premise

The method assumes that a single fixed geometric warp (an affine transformation) can align every reference frame to the target frame; when objects move non-rigidly, parallax shifts, or the viewpoint changes strongly, the copied content will be misaligned and the inpainted hole will be wrong.

Editorial extensions

If this is right

  • Per-video completion time drops from about 952 seconds to 27 seconds at the tested resolution, so interactive editing and large-scale video cleanup become practical.
  • Distant reference frames become usable, so holes caused by large or slowly moving foreground objects can be filled from frames that have little overlap with the target hole.
  • No optical-flow supervision is required for alignment; the affine warps are learned from a visibility-masked reconstruction loss, simplifying training data requirements.
  • The same copy-and-paste mechanism transfers to exposure restoration: enhancing over/under-exposed frames improves a downstream lane detector from 46.69% to 83.00% accuracy on road videos.
  • Temporal consistency can be maintained without explicit flow-based temporal losses, using iterative reference replacement and forward–reverse averaging.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to replace the affine warp with a learned homography or coarse-to-fine flow inside the same copy-and-paste framework; this would test how much of the remaining quality gap comes from the alignment model rather than the copying mechanism.
  • The masked softmax context matching module is not specific to inpainting; it could be applied to video super-resolution, denoising, or frame interpolation, wherever corresponding content must be pooled across frames.
  • The exposure-restoration result suggests a preprocessing front-end for automotive perception: feeding enhanced frames to existing lane or object detectors may increase accuracy in tunnels and bright sunlight without retraining the detector, which a controlled benchmark with varied cameras could verify.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes Copy-and-Paste Networks, a feed-forward deep network for video inpainting. The system aligns reference frames to the target frame via a self-supervised affine alignment network, aggregates aligned reference features using a masked softmax context-matching module, and decodes the aggregated features to fill the target holes. Completed frames are fed back as references to improve temporal coherence. Experiments on DAVIS report PSNR and SSIM comparable to the optimization-based method of Huang et al., and a user study shows competitive quality, while the method runs about 35 times faster. The paper also extends the framework to restore over/under-exposed frames and reports a large improvement in lane detection accuracy on a private road dataset.

Significance. If the results are reproducible, the paper contributes a practical, fast alternative to optimization-based video inpainting. The self-supervised alignment loss and the masked softmax context matching are simple and potentially reusable ideas. The paper includes a direct comparison with a strong baseline and a user study, and the runtime advantage is substantial. However, the empirical support for the central 'comparable quality' claim is thin: the PSNR margin is small, SSIM is lower, no error bars or statistical tests are reported, and the lane detection validation is on a private dataset with a proprietary detector. The method's reliance on affine alignment for non-rigid scenes is not analyzed. These weaknesses should be addressed before the paper can be accepted in its current form.

major comments (5)
  1. [Section 4.1, Eq. (8)] The loss decomposition is inconsistent with the definition of Cmask in Eq. (6). Section 3.2 defines Cmask(x,y) = 1 - sum_r C^r_match(x,y) and states that Cmask marks pixels never visible in any reference frame. Then in Eq. (8), Lhole(visible) multiplies Mt by Cmask, while Lhole(invisible) multiplies Mt by (1 - Cmask). With the given definition, this assigns the copy-based loss to invisible pixels and the synthesis loss to visible pixels. This is a load-bearing training objective error; please correct the equations or clarify the intended meaning of Cmask.
  2. [Section 5.1, Table 1 and Section 5.2, Table 2] The central claim of 'comparable' quality to Huang et al. is supported by mean PSNR/SSIM over only 25 DAVIS videos and a user study on 30 videos, but no standard deviations, per-sequence scores, confidence intervals, or statistical significance tests are reported. The PSNR difference (28.37 vs 28.14) is small, SSIM is lower (0.851 vs 0.859), and the user-study ranks are very close (1.77 vs 1.74). Without error bars or a paired test, the claim that the methods are statistically indistinguishable is not established. Please provide per-sequence results and a significance test.
  3. [Section 3.1 and Eq. (1)] The alignment network restricts registration to affine transformations. While the training data include YouTube clips, no experiment quantifies alignment accuracy or analyzes failure cases for non-affine motion, parallax, or large viewpoint changes. Since the copy step depends on correct alignment, a failure analysis or a discussion of the limitations of affine alignment for general video inpainting is needed to support the claimed general-scene performance. This is not a fatal flaw, but it is a load-bearing assumption in the method design.
  4. [Section 5.3 and Table 3] The lane detection application is reported as a 46.69% to 83.00% improvement on 469 frames from a private dataset using a proprietary detector, but the paper does not describe the detector, the video content, the restoration model adaptation, or any comparison baseline (e.g., simple histogram equalization). This is a claimed contribution of the paper, but the current evidence is anecdotal and not independently verifiable. Please provide details or explicitly frame the result as a preliminary demonstration.
  5. [Section 3.3 and Fig. 11] Temporal consistency is claimed as a benefit of the reference update, but the only evidence is a single qualitative temporal profile. A quantitative temporal consistency metric (e.g., warp error between adjacent completed frames) should be reported to support this claim, especially because the paper also claims that results are temporally coherent.
minor comments (4)
  1. [Section 4.2] The synthesis of training data from YouTube clips is described as 'divided them according to the scene (7.3K scenes)', but the scene-division procedure is not specified. Please clarify how scenes are defined and how frames are sampled.
  2. [Section 3.2, Eq. (2)] The visibility map V used in the global similarity is not explicitly defined in Eq. (2); it would be clearer to state that V = Vt ⊙ Vr→t, as in Eq. (1).
  3. [Section 4.1, Eq. (9)] In the perceptual loss, the pooled features are from VGG-16; please specify which pooling layers are used (e.g., relu1_2, relu2_2, etc.), as is common in this literature.
  4. [Section 5.2] The user study description states that 40 participants evaluated videos and ties were allowed, but no information is given about how many videos each participant saw, how the order of methods was randomized, or the distribution of ranks. A brief protocol description would help reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central derivation is self-contained, with all reported quantities evaluated on held-out external benchmarks rather than derived from fitted inputs.

full rationale

The paper's derivation chain is self-contained rather than circular. The alignment network is trained with the self-supervised L1 loss in Eq. (1), and the full model is trained end-to-end with reconstruction, perceptual, style, and total variation losses in Eq. (10) on a synthesized dataset described in Section 4.2. The reported PSNR, SSIM, user-study rankings, runtime comparisons, and lane-detection accuracy are all measured on held-out DAVIS sequences and an external downstream lane-detection task, not on the training targets used to fit network parameters. No equation reduces to a fitted value that is then renamed as a prediction; the masked softmax weighting in Eqs. (2)-(6) is an architectural aggregation rule, not a re-statement of the training objective. The comparison baselines are external works [9, 13], and there is no load-bearing self-citation chain. The affine-motion assumption in Section 3.1 is a modeling limitation that could cause errors on non-affine scenes, but it is an architectural prior rather than a circularity: it is not equivalent to the training loss or to the evaluation metric. Therefore no specific circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the affine motion model, the synthetic training distribution, and standard deep network backbones. The loss weights are tuned rather than derived. No new physical entities are introduced, only network components that are part of the method.

free parameters (2)
  • Loss weights in Eq. (10) = 2, 10, 20, 6, 0.01, 24, 0.1
    Weights for L_align, L_hole(visible), L_hole(invisible), L_non-hole, L_perceptual, L_style, and L_tv. The paper states they are 'empirically determined' (Section 4.1), so they are hand-tuned hyperparameters, not derived from theory.
  • Number of reference frames m = 5 (during training)
    Section 4.3 uses five randomly selected 256x256 frames per training sample. The number of reference frames controls how much temporal context is available and affects runtime, but it is not analyzed in the ablation.
assumptions (4)
  • domain assumption Affine transformations are sufficient to align reference and target frames for inpainting
    Stated in Section 3.1; the alignment network only predicts affine matrices. This is valid for planar or slowly varying scenes but fails for complex 3D motion, which can corrupt copied content.
  • domain assumption Self-supervised alignment loss on visible pixels is sufficient for robust alignment near holes
    Eq. (1) computes L1 distance only on pixels valid in both frames; this can be satisfied even if alignment is poor inside or near the hole, and provides no constraint for regions with low texture.
  • domain assumption Synthesized training data from static images and masks transfers to real videos
    Section 4.2 creates training videos by applying random crops, shear, scale, translation, and rotation to single images, plus segmentation masks. This does not model object deformation, lighting change, or real camera motion, so sim-to-real transfer is assumed.
  • standard math Pretrained VGG-16 on ImageNet provides a valid perceptual loss for video frames
    Used in Eq. (9) for perceptual and style loss; this is a standard practice from the literature and not introduced ad hoc, but the domain shift to video inpainting is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Copy-and-Paste Networks for Deep Video Inpainting." pith.science (2026). https://pith.science/paper/JG6TFHSC

@misc{pith2026190811587,
  author       = {Pith},
  title        = {Pith review of: Copy-and-Paste Networks for Deep Video Inpainting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JG6TFHSC}},
  note         = {Machine review of arXiv:1908.11587}
}
read the original abstract

We present a novel deep learning based algorithm for video inpainting. Video inpainting is a process of completing corrupted or missing regions in videos. Video inpainting has additional challenges compared to image inpainting due to the extra temporal information as well as the need for maintaining the temporal coherency. We propose a novel DNN-based framework called the Copy-and-Paste Networks for video inpainting that takes advantage of additional information in other frames of the video. The network is trained to copy corresponding contents in reference frames and paste them to fill the holes in the target frame. Our network also includes an alignment network that computes affine matrices between frames for the alignment, enabling the network to take information from more distant frames for robustness. Our method produces visually pleasing and temporally coherent results while running faster than the state-of-the-art optimization-based method. In addition, we extend our framework for enhancing over/under exposed frames in videos. Using this enhancement technique, we were able to significantly improve the lane detection accuracy on road videos.

Figures

Figures reproduced from arXiv: 1908.11587 by the authors.

Figure 1
Figure 1. (a) We propose a DNN framework for video in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Network Overview. Our framework consists of 3 sub-networks: alignment network, copy network, and paste [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Using affine transformation for the alignment [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: An 1-D example of masked softmax. from the given masks, where 0 indicates hole pixels and 1 represents non-hole pixels. Note that the alignment network is jointly trained with other networks in an end-to-end man￾ner, not independently. 3.2. Copy-and-Paste Network After…
Figure 6
Figure 6. Figure 6: Synthesized training dataset example. We synthesize videos by compositing background im￾age sequences with object masks ( [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison of object removal results for the scenes [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: User study for video object removal results (lower value is better) [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Application of our method for the restoration of under/over-exposed images. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Ablation study for reference update. (b), (c) and [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 16 canonical work pages

  1. [1]

    Patchmatch: a randomized correspondence algorithm for structural image editing

    Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (TOG), 28(3):24, 2009. 2

    work page 2009

  2. [2]

    Mit saliency bench- mark

    Zoya Bylinskii, Tilke Judd, Ali Borji, Laurent Itti, Fr´edo Du- rand, Aude Oliva, and Antonio Torralba. Mit saliency bench- mark. 6

    work page

  3. [3]

    Real- time video super-resolution with spatio-temporal networks and motion compensation

    Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. Real- time video super-resolution with spatio-temporal networks and motion compensation. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 4778–4787, 2017. 8

    work page 2017

  4. [4]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5

    2009

  5. [5]

    Deep image homography estimation

    Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabi- novich. Deep image homography estimation. arXiv preprint arXiv:1606.03798, 2016. 2

    arXiv 2016

  6. [6]

    The pascal visual object classes challenge: A retrospective.Inter- national journal of computer vision, 111(1):98–136, 2015

    Mark Everingham, SM Ali Eslami, Luc Van Gool, Christo- pher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective.Inter- national journal of computer vision, 111(1):98–136, 2015. 6

    work page 2015

  7. [7]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680,

  8. [8]

    Background inpainting for videos with dynamic objects and a free-moving camera

    Miguel Granados, Kwang In Kim, James Tompkin, Jan Kautz, and Christian Theobalt. Background inpainting for videos with dynamic objects and a free-moving camera. In ECCV, 2012. 2

    work page 2012

Show all 25 references
  1. [9]

    Temporally coherent completion of dynamic video

    Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Jo- hannes Kopf. Temporally coherent completion of dynamic video. ACM Transactions on Graphics (TOG), 35(6), 2016. 2, 3, 6

    2016

  2. [10]

    Globally and Locally Consistent Image Completion

    Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and Locally Consistent Image Completion. ACM Transactions on Graphics (Proc. of SIGGRAPH 2017) , 36(4):107:1–107:14, 2017. 2

    2017

  3. [11]

    Spatial transformer networks

    Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Advances in neural infor- mation processing systems, pages 2017–2025, 2015. 2

    2017

  4. [12]

    Perceptual losses for real-time style transfer and super-resolution

    Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision , pages 694–711. Springer, 2016. 5

    2016

  5. [13]

    Deep video inpainting

    Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5792–5801, 2019. 1, 2, 4, 6

    2019

  6. [14]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  7. [15]

    Image inpainting for irregular holes using partial convolutions

    Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 85–100, 2018. 2

    2018

  8. [16]

    Video inpainting of complex scenes

    Alasdair Newson, Andrs Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Prez. Video inpainting of complex scenes. SIAM Journal on Imaging Sciences, Society for In- dustrial and Applied Mathematics, 7(4):1993–2019, 2014. 2

    1993

  9. [17]

    Unsupervised deep homography: A fast and robust homography estimation model

    Ty Nguyen, Steven W Chen, Shreyas S Shivakumar, Camillo Jose Taylor, and Vijay Kumar. Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robotics and Automation Letters , 3(3):2346– 2353, 2018. 2

    2018

  10. [18]

    Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 2

    2016

  11. [19]

    A benchmark dataset and evaluation methodology for video object segmentation

    Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pag...

    2016

  12. [20]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 6

    2017 arXiv

  13. [21]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 5

    2014 arXiv

  14. [22]

    Video inpainting by jointly learning temporal structure and spatial details

    Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learning temporal structure and spatial details. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5232–5239, 2019. 2

    2019

  15. [23]

    Free-form image inpainting with gated convolution

    Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. arXiv preprint arXiv:1806.03589, 2018. 2

    2018 arXiv

  16. [24]

    Generative image inpainting with con- textual attention

    Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with con- textual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5505– 5514, 2018. 2

    2018

  17. [25]

    Places: A 10 million image database for scene recognition

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017. 6

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.