REVIEW 3 major objections 5 minor 37 references
StructureFlow: Image Inpainting via Structure-aware Appearance Flow
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Two-stage inpainting that completes smooth structure before sampling texture outperforms single-stage models on public benchmarks.
desk verdict A solid two-stage inpainting architecture with a real structural idea, but the 'superior performance' claim is undercut by the paper's own Places2 FID and user-study numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the edge-preserved smooth image produced by relative total variation (RTV) smoothing. At scale $\sigma \approx 3$ it removes high-frequency texture while retaining sharp edges, giving the structure reconstructor a training target that captures global layout without texture noise. The second load-bearing component is the appearance-flow module inside the texture generator: for each missing pixel it predicts a location in the visible feature map to sample from, and the warped features are concatenated with convolutional features to synthesize detail. Gaussian sampling widens the sampling footprint, and the sampling-correctness loss keeps the flow from locking onto bad local minima during training.
What would settle it
Train the same two-stage architecture with structure labels produced by a plain Gaussian low-pass filter instead of RTV, and compare on a held-out dataset: if there is no performance drop, the edge-preserving properties of RTV are not the cause of the gain. Alternatively, vary the RTV scale $\sigma$ on a held-out set; if performance is flat across $\sigma$, the claim that $\sigma \approx 3$ is the right separation scale is a tuning artifact.
Extended reading notes
Core claim
The paper claims that an inpainting network can produce both meaningful structure and vivid texture by splitting the task into two learned stages. The first stage reconstructs an edge-preserved smooth image, rather than an edge map or the raw image, as the global structure target; the second stage uses the finished structure to guide an appearance-flow module that samples features from visible regions to synthesize texture. To stop the appearance flow from collapsing onto bad matches, the paper replaces bilinear sampling with Gaussian sampling and adds a sampling-correctness loss computed from a pretrained deep feature network. On scene, face, and street-view datasets, the paper reports higher PSNR and SSIM values than contextual-attention, partial-convolution, and edge-guided baselines, with competitive FID scores and higher human preference on structured scenes.
Load-bearing premise
The central claim rests on the assumption that a fixed scale of edge-preserving smoothing separates texture from structure uniformly across images and mask shapes, rather than being a preprocessing choice tuned to the evaluation data.
Editorial extensions
If this is right
- Inpainting models can be built modularly: a structure reconstructor followed by a texture generator, instead of one monolithic generator.
- Edge-preserved smooth images are a viable structural intermediate for inpainting, carrying layout and color without the noise of full texture.
- Appearance flow with Gaussian sampling can establish long-range correspondences between visible and missing regions, reducing blur and artifacts.
- The method enables practical editing workflows such as masking out unwanted objects or editing the reconstructed structure to add or remove content.
Reading between the lines
- Beyond the paper, this structure-first separation could be tested on video inpainting, where temporal structure makes hallucinating plausible layouts even harder.
- Beyond the paper, Gaussian sampling in appearance flow may transfer to other flow-based generation tasks such as view synthesis or image translation, with the sampling-correctness loss as a general guard against bad matches.
- Beyond the paper, if the structure prior is the real driver, replacing the fixed RTV preprocessing with a learnable per-image structure extractor could yield a fully end-to-end variant that adapts the smoothing scale automatically.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes StructureFlow, a two-stage image inpainting network. The first stage is a structure reconstructor that predicts edge-preserved smoothed images (obtained with RTV) for the missing regions, and the second stage is a texture generator that uses appearance flow, with Gaussian sampling and a sampling correctness loss, to synthesize high-frequency details. Experiments on Places2, CelebA, and Paris StreetView compare the method with Contextual Attention, Partial Convolution, and EdgeConnect, and are supplemented by ablation studies of the structure stage, the smoothing representation, and the appearance flow. The paper's abstract claims superior performance, although its own quantitative tables show mixed results on Places2, and the RTV smoothing scale is selected empirically on the Paris test set.
Significance. If the claims were fully supported, the paper would make a useful contribution: replacing edge maps with edge-preserved smooth images as the structure prior is a reasonable and potentially more informative representation, and the appearance-flow modifications address a known optimization difficulty in using flow for inpainting. The ablations do support the value of the structure reconstructor, the smoothed structure labels, and the presence of appearance flow, and the authors release source code, which strengthens reproducibility. However, the core performance claim is not consistently supported by the paper's own metrics, the key hyperparameter is tuned on a test set, and two of the three stated contributions (Gaussian sampling and sampling correctness loss) lack quantitative evaluation. These issues need to be addressed before the paper can be recommended for publication.
major comments (3)
- [Abstract; §4.2, Tables 1–2] The abstract's claim of 'superior performance' is contradicted by the paper's own quantitative results. In Table 1, for Places2 masks of 40–60%, StructureFlow's FID is 22.3803 versus EdgeConnect's 19.0003, and in Table 2, the Places2 user-study fooling rate is 23.56% for StructureFlow versus 26.36% for EdgeConnect. The conclusion also retreats to 'competitive results.' Because the central claim of the paper is the performance advantage, these internal contradictions must be resolved by either providing evidence across all metrics and mask ranges or by softening the claim.
- [§4.1, §4.3, Table 4] The choice of the RTV smoothing scale σ≈3 is made by evaluating σ ∈ {0,1,3,6,9} on the Paris test set. Since σ defines the ground-truth structure representation Sgt on which both stages are trained, selecting it on the test set means the reported improvements are partly an artifact of test-set hyperparameter fitting. Please select σ on a validation split, or alternatively show that the same σ is optimal across multiple datasets and mask ratios, and report sensitivity to this choice.
- [§4.3, Figure 6] Gaussian sampling and the sampling correctness loss are two of the three contributions listed in the paper, but their quantitative effect is not reported. The flow ablation in §4.3 only shows qualitative flow-field visualizations in Figure 6; no PSNR/SSIM/FID is given for a model with Bilinear sampling or for a model without the sampling correctness loss. Table 3 ablates the presence of flow as a whole, not these components, so the claimed improvements from these modifications are not substantiated.
minor comments (5)
- [§4.1] The sentence 'We employ edge-preserved smooth method RTV [31] to obtain the training labels of the structure reconstructor Gt' should refer to Gs, since Gt is the texture generator.
- [§4.2, Table 2] The user-study numbers are reported without statistical significance testing; the Paris gap between StructureFlow and EdgeConnect is only 0.24 percentage points, so the statement that the model achieves 'better results' there is not clearly supported.
- [§4.2, Table 1] Table 1 does not report FID for PConv (shown as '- - -'), even though FID is emphasized as a perceptual metric; this incomplete comparison should be fixed.
- [§3.2, Eq. (10)] The symbol M is first used as a binary mask matrix and then as a coordinate set; please disambiguate the notation.
- [§4.3, Figure 6] The color-coding map used to visualize appearance flow fields in Figure 6 is not explained in the caption; please add a description of what the colors and intensities encode.
Circularity Check
No circularity: the method is an empirical supervised inpainting pipeline; losses are standard supervised objectives and the sigma choice is hyperparameter tuning, not a fitted prediction.
full rationale
The paper does not present a derivation whose outputs are equivalent to its inputs by construction. The structure reconstructor is trained with an l1 loss against RTV-smoothed ground-truth images (Eq. 2) and an adversarial loss (Eq. 3); the texture generator is trained with l1, adversarial, and VGG-feature-based sampling-correctness losses (Eqs. 6, 7, 10) against ground-truth images and features. These are ordinary supervised training objectives, not circular definitions. Appearance flow is borrowed from prior work [35] and adapted with Gaussian sampling; the sampling correctness loss is explicitly supervised by ground-truth VGG features, and its role is to guide optimization, not to redefine the output. The RTV parameter sigma is selected empirically in Section 4.1 and ablated in Table 4 on the Paris dataset; this is hyperparameter tuning and is honestly described ('we empirically find the best results obtained when we set sigma approximately 3'). It is a correctness/validation concern, not a circularity, because sigma is never renamed as a prediction and no equation reduces to it. The paper also contains no load-bearing self-citations: the cited methods are external prior works. The abstract's claim of superior performance is partly contradicted by the paper's own FID and user-study numbers on Places2 (Table 1: FID 22.3803 vs EdgeConnect 19.0003; Table 2: fooling rate 23.56% vs 26.36%), but that is an evidentiary inconsistency, not circular reasoning. Overall, the derivation chain is self-contained in the sense that each component is trained against external ground-truth labels and evaluated against external benchmarks.
Assumptions & free parameters
free parameters (4)
- RTV smoothing parameter sigma =
3
- Loss weights lambda_s_l1=4, lambda_s_adv=1, lambda_t_l1=5, lambda_t_c=0.25, lambda_t_adv=1 =
as listed
- Gaussian sampling kernel size n and variance sigma =
not reported
- VGG19 feature layer used for sampling correctness loss =
not reported
assumptions (5)
- domain assumption RTV edge-preserved smooth images represent global structure without losing information needed for texture generation.
- domain assumption Image regions with similar structures have highly correlated textures, so appearance flow can copy from intact regions.
- domain assumption Gaussian sampling expands the receptive field of the sampling operation and stabilizes optimization of the appearance flow relative to bilinear sampling.
- domain assumption Cosine similarity in a pre-trained VGG19 feature space measures whether the flow sampled the correct region.
- domain assumption Adversarial training with two PatchGAN discriminators and spectral normalization produces realistic images.
Cite this review
Pith. "Pith review of StructureFlow: Image Inpainting via Structure-aware Appearance Flow." pith.science (2026). https://pith.science/paper/TVFSNYTB
@misc{pith2026190803852,
author = {Pith},
title = {Pith review of: StructureFlow: Image Inpainting via Structure-aware Appearance Flow},
year = {2026},
howpublished = {\url{https://pith.science/paper/TVFSNYTB}},
note = {Machine review of arXiv:1908.03852}
}
read the original abstract
Image inpainting techniques have shown significant improvements by using deep neural networks recently. However, most of them may either fail to reconstruct reasonable structures or restore fine-grained textures. In order to solve this problem, in this paper, we propose a two-stage model which splits the inpainting task into two parts: structure reconstruction and texture generation. In the first stage, edge-preserved smooth images are employed to train a structure reconstructor which completes the missing structures of the inputs. In the second stage, based on the reconstructed structures, a texture generator using appearance flow is designed to yield image details. Experiments on multiple publicly available datasets show the superior performance of the proposed network.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Patchmatch: A randomized correspon- dence algorithm for structural image editing
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspon- dence algorithm for structural image editing. In ACM Trans- actions on Graphics (ToG), volume 28, page 24. ACM, 2009. 1, 2
work page 2009
-
[2]
Image inpainting
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and in- teractive techniques, pages 417–424. ACM Press/Addison- Wesley Publishing Co., 2000. 2
2000
-
[3]
Image melding: Combining in- consistent images using patch-based synthesis
Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B Goldman, and Pradeep Sen. Image melding: Combining in- consistent images using patch-based synthesis. ACM Trans. Graph., 31(4):82–1, 2012. 2
work page 2012
-
[4]
What makes paris look like paris? ACM Transactions on Graphics, 31(4), 2012
Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei Efros. What makes paris look like paris? ACM Transactions on Graphics, 31(4), 2012. 5
2012
-
[5]
Flownet: Learning optical flow with convolutional networks
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Pro- ceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015. 3
work page 2015
-
[6]
Image quilting for texture synthesis and transfer
Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th an- nual conference on Computer graphics and interactive tech- niques, pages 341–346. ACM, 2001. 2
2001
-
[7]
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680,
-
[8]
Scene completion using millions of photographs
James Hays and Alexei A Efros. Scene completion using millions of photographs. ACM Transactions on Graphics (TOG), 26(3):4, 2007. 1, 2
work page 2007
Show all 37 references
-
[9]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5
2016
-
[10]
Gans trained by a two time-scale update rule converge to a local nash equilib- rium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In Advances in Neural Information Processing Sys- tems, pages 6626–6637, 2017. 5
2017
-
[11]
Determining opti- cal flow
Berthold KP Horn and Brian G Schunck. Determining opti- cal flow. Artificial intelligence, 17(1-3):185–203, 1981. 3
1981
-
[12]
Globally and Locally Consistent Image Completion
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and Locally Consistent Image Completion. ACM Transactions on Graphics (Proc. of SIGGRAPH 2017) , 36(4):107:1–107:14, 2017. 1, 2
2017
-
[13]
Flownet 2.0: Evolu- tion of optical flow estimation with deep networks
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolu- tion of optical flow estimation with deep networks. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2462–2470, 2017. 3
2017
-
[14]
Image-to-image translation with conditional adver- sarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134,
-
[15]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
-
[16]
Image inpainting for irregular holes using partial convolutions
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 85–100, 2018. 1, 3, 5
2018
-
[17]
Deep learning face attributes in the wild
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015. 5
2015
-
[18]
Video frame synthesis using deep voxel flow
Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision, pages 4463–4471, 2017. 3, 4
2017
-
[19]
Unflow: Un- supervised learning of optical flow with a bidirectional cen- sus loss
Simon Meister, Junhwa Hur, and Stefan Roth. Unflow: Un- supervised learning of optical flow with a bidirectional cen- sus loss. In Thirty-Second AAAI Conference on Artificial In- telligence, 2018. 3
2018
-
[20]
Spectral normalization for generative ad- versarial networks
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative ad- versarial networks. arXiv preprint arXiv:1802.05957, 2018. 5
2018 arXiv
-
[21]
Edgeconnect: Generative image in- painting with adversarial edge learning
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Generative image in- painting with adversarial edge learning. arXiv preprint arXiv:1901.00212, 2019. 1, 2, 3, 5, 6, 7
1901 arXiv
-
[22]
Context encoders: Feature learn- ing by inpainting
Deepak Pathak, Philipp Kr ¨ahenb¨uhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learn- ing by inpainting. 2016. 1, 2
2016
-
[23]
Optical flow estima- tion using a spatial pyramid network
Anurag Ranjan and Michael J Black. Optical flow estima- tion using a spatial pyramid network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 4161–4170, 2017. 3, 4
2017
-
[24]
Simakov, Y
D. Simakov, Y . Caspi, E. Shechtman, and M. Irani. Sum- marizing visual data using bidirectional similarity. In 2008 IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 1–8, June 2008. 2
2008
-
[25]
Two-stream con- volutional networks for action recognition in videos
Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. In Ad- vances in neural information processing systems, pages 568– 576, 2014. 3
2014
-
[26]
Spg-net: Segmentation prediction and guidance network for image inpainting
Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C-C Jay Kuo. Spg-net: Segmentation prediction and guidance network for image inpainting. arXiv preprint arXiv:1805.03356, 2018. 1, 2
2018 arXiv
-
[27]
Learning optical flow
Deqing Sun, Stefan Roth, JP Lewis, and Michael J Black. Learning optical flow. In European Conference on Computer Vision, pages 83–97. Springer, 2008. 3
2008
-
[28]
Temporal segment net- works: Towards good practices for deep action recognition
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment net- works: Towards good practices for deep action recognition. In European conference on computer vision , pages 20–36. Springer, 2016. 3
2016
-
[29]
Video-to- video synthesis
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to- video synthesis. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2018. 3
2018
-
[30]
Image smoothing via l0 gradient minimization
Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. Image smoothing via l0 gradient minimization. ACM Transactions on Graphics (SIGGRAPH Asia), 2011. 2, 3
2011
-
[31]
Structure extrac- tion from texture via relative total variation
Li Xu, Qiong Yan, Yang Xia, and Jiaya Jia. Structure extrac- tion from texture via relative total variation. ACM Transac- tions on Graphics (TOG), 31(6):139, 2012. 2, 3, 5, 7
2012
-
[32]
Free-form image inpainting with gated convolution
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. arXiv preprint arXiv:1806.03589, 2018. 1, 3
2018 arXiv
-
[33]
Generative image inpainting with con- textual attention
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with con- textual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5505– 5514, 2018. 1, 2, 4, 5, 6
2018
-
[34]
Places: A 10 million image database for scene recognition
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2018. 5
2018
-
[35]
View synthesis by appearance flow
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Ma- lik, and Alexei A Efros. View synthesis by appearance flow. In European conference on computer vision, pages 286–301. Springer, 2016. 2, 3
2016
-
[36]
To- ward multimodal image-to-image translation
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Dar- rell, Alexei A Efros, Oliver Wang, and Eli Shechtman. To- ward multimodal image-to-image translation. In Advances in Neural Information Processing Systems , pages 465–476,
-
[37]
Video frame synthesis using deep voxel flow
Xiaoou Tang Yiming Liu Ziwei Liu, Raymond Yeh and Aseem Agarwala. Video frame synthesis using deep voxel flow. In Proceedings of International Conference on Com- puter Vision (ICCV), October 2017. 3
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.