Pith. sign in

REVIEW 3 major objections 4 minor 29 references

SFSegNet: Parse Freehand Sketches using Deep Fully Convolutional Networks

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read SFSegNet parses freehand sketches into semantic part labels with 91.2% pixel accuracy, besting FCN-8s, LinkNet-34, and U-Net on a new 10,000-sketch dataset.

desk verdict Plausible engineering with a useful dataset, but the performance claims need stronger baselines and statistics before they can be taken at face value. read the letter →

arxiv 1908.05389 v1 pith:4ACAZCJE submitted 2019-08-15 cs.CV cs.LG

classification cs.CVcs.LG
keywords sketchsegmentationfreehandparsingfullyconvolutionalnetworkaffinetransformencoderreweightinglosssemanticannotateddataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that freehand sketch parsing—labeling each stroke with the object part it depicts—can be solved by a deep fully convolutional network tailored to the sparsity and shakiness of drawn lines. The authors build SFSegNet, which combines three scales of ResNet features with small affine transform encoders that canonicalize jittery strokes, and a reweighting loss that ignores the background pixels that make up about 99% of a sketch. To train and test it, they assemble a dataset of 10,000 annotated sketches across ten object categories, with a single shared set of part labels. On this dataset the network reaches 91.2% pixel accuracy and 88.6% component accuracy, ahead of standard segmentation baselines; on an earlier benchmark it also edges past a CRF-based method. If the comparisons hold, the paper shows that a single end-to-end network, without stroke-order or category priors, can segment freehand sketches at near-usable accuracy.

What carries the argument

The central mechanism is the combination of a three-stage fully convolutional ResNet-34 encoder with three targeted additions. First, intermediate outputs of the three stages are fused after upsampling, so coarse and fine stroke cues jointly determine each pixel's label. Second, an affine transform encoder—a small spatial-transformer-style module applied during each pooling step—learns a transformation matrix that aligns distorted stroke features, giving the network partial spatial invariance. Third, a reweighted cross-entropy loss sets the background class weight to zero, forcing the network to learn from the sparse stroke pixels alone. Together these pieces let the network parse sparse, jittery line drawings without relying on stroke order or a separate category classification step.

What would settle it

Re-run the evaluation on the earlier benchmark using the original fine-grained component labels, before any parts are merged or removed, and compare SFSegNet against the CRF baseline on those original labels; if SFSegNet's average pixel accuracy stops exceeding the CRF method's, the cross-dataset superiority claimed in the paper is an artifact of the re-labeling.

Watch

Extended reading notes

Core claim

SFSegNet is an end-to-end deep network that maps a raw freehand sketch directly to a per-pixel label map of object parts. The network decapitates a standard ResNet-34 classifier into three stages, adds a small scoring head after each stage, and fuses the upsampled predictions so that fine and coarse stroke structure jointly decide the label. During the three max-pooling transitions, a lightweight affine transform encoder—a mini spatial transformer—predicts a transform matrix that aligns the feature maps, making the representation partly invariant to the shaking and small rotations that make freehand strokes hard to parse. Training uses a cross-entropy loss in which the background class is weighted to zero, so the model concentrates on the roughly one percent of pixels that belong to strokes. With a new dataset of 10,000 sketches in ten categories and a single shared set of 24 part labels plus background, the authors report an average P-metric accuracy of 91.2% and C-metric of 88.6%, the best among FCN-8s/16s/32s, LinkNet-34, and U-Net baselines, and a P-metric of 74.2% on the prior benchmark, slightly above a CRF-based method.

Load-bearing premise

The claim that the network beats earlier methods assumes that re-labeling the earlier benchmark's part annotations—merging several parts into one and ignoring others—does not systematically favor the categories the network was trained to recognize.

Editorial extensions

If this is right

  • A single network trained once on the 10,000-sketch dataset can segment sketches across all ten categories with a shared label set, avoiding per-class ground truths or stroke-order information.
  • Because background pixels are ignored in the loss, the network can be trained on sketches where strokes occupy under 1% of the image, a regime where ordinary segmentation losses collapse to all-background predictions.
  • The affine transform encoder adds a measurable gain: removing it drops average P-metric accuracy from 91.2% to 89.2% and C-metric from 88.6% to 86.5%, showing that spatial alignment of stroke features helps the fully convolutional pipeline.
  • On the earlier benchmark, SFSegNet reaches 74.2% P-metric accuracy versus 73.2% for a CRF-based method, but does so without needing a separate sketch classification step or a 3D model repository, and it runs at 1–2 sketches per second.
  • The P-metric advantage over FCN-8s (2.9 points), LinkNet-34 (1.5 points), and U-Net (8.0 points) on the new dataset indicates that sketch-specific design choices—reweighting and affine alignment—matter more than generic segmentation backbone strength.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The re-labeling of the earlier benchmark removes or merges several fine-grained components; if those original parts are preserved, the comparison on that benchmark might favor methods designed for the original labels, and the claimed edge over the CRF baseline could shrink or disappear.
  • Because the affine transform encoders operate on feature maps rather than raw strokes, their benefit is likely limited to local jitter and small rotations; large viewpoint or pose changes would require more global canonicalization, a limitation that could be tested by evaluating on sketches drawn from unusual angles.
  • The single shared ground truth across all categories may understate the difficulty of segmenting object classes with very different part structures; a natural extension is to test the same architecture on a dataset with more classes or per-class part taxonomies to see whether the shared-label constraint is a help or a hindrance.
  • Since the dataset was collected by both artists and non-experts, the network's robustness to drawing skill could be measured by reporting accuracy separately for the two groups; the paper does not break out these numbers, but the claim of being 'extensively applicable' would be stronger if the model performs well on both.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SFSegNet, a fully convolutional network for semantic part segmentation of freehand sketches. The method combines a modified FCN based on ResNet34 with affine transform encoders (mini-STNs) at three stages and a background-reweighting loss that ignores the large white background. The authors introduce a 10,000-sketch dataset with 25 component labels under one ground-truth configuration, and report experiments on this dataset and on Huang et al.'s benchmark. On the authors' dataset, SFSegNet achieves 91.2% P-metric and 88.6% C-metric average accuracy, surpassing FCN-8s, LinkNet-34, and U-Net. On Huang's dataset, it reaches 74.2% P-metric, higher than Huang's method but lower than CRF on several categories.

Significance. If the reported results hold, the paper makes a useful contribution to sketch understanding: it provides a relatively large annotated sketch dataset, a simple modification of FCN with affine transform encoders, and a reweighting strategy that addresses extreme foreground/background imbalance. The idea of applying spatial-transformer-style canonicalization to stroke distortions is interesting and the experimental design, including the comparison with several FCN variants and the ablation of the affine transform encoder, is clearly presented. The dataset and code are not released, and the evaluation has weaknesses, but the central methodological proposal is plausible and the paper is likely to be of interest to the sketch-recognition community.

major comments (3)
  1. [Section IV-B and Tables III/IV] The central claim that SFSegNet outperforms state-of-the-art networks is not statistically supported. The paper reports no error bars, no significance tests, and no repeated runs with different seeds or train/test splits. The reported margins over LinkNet-34 (1.4 P-metric points) and U-Net (0.9 C-metric points) are small enough to be explained by training variance, especially since the baselines are trained with their default parameters while SFSegNet uses a tuned learning rate, weight decay, and training budget, and all models are stopped at 50 epochs. To support the claim, the authors should report results over multiple seeds, use matched training budgets and hyperparameter settings, and provide confidence intervals or significance tests.
  2. [Section IV-D and Appendix Table VI] The comparison on Huang's dataset is compromised by the manual relabeling and merging of components. The appendix shows that several components are ignored (e.g., airplane engine and propeller) or merged (e.g., multiple chair parts become 'limb'), and Section IV-D states that 'some components are annotated by mistake' and were relabeled. These edits are made by the authors without an external protocol or a demonstration that they do not favor SFSegNet's output categories. The paper should either justify each relabeling, quantify the effect of the edits, or make the relabeled ground truth publicly available so that the comparison can be independently verified. As it stands, the Huang benchmark result is not a clean external validation.
  3. [Section IV-D, Tables III and IV] The statement 'our model performs the best in each metric' is inaccurate as written. Table III shows that LinkNet-34 achieves higher P-metric accuracy than SFSegNet on Bicycle (94.0 vs 93.3) and Human (85.9 vs 85.1), and Table IV shows that U-Net has higher C-metric accuracy on Bicycle (85.9 vs 85.6). The claim should be revised to 'best average performance' or the per-class results should be discussed honestly. Misreporting per-class results, even in a summary sentence, weakens confidence in the reported averages.
minor comments (4)
  1. [Section IV-B] The text says 'All models are trained within 50 iterations,' but the context and Figure 6 indicate epochs. Please clarify whether 'iterations' means epochs or something else.
  2. [Section IV-A and Figure numbering] The labeled examples in the dataset description are referenced as 'shown in Fig. 5,' but Fig. 5 is used for qualitative segmentation results. Renumber the figures so that the dataset examples and the qualitative results are referenced consistently.
  3. [Abstract and Introduction] There are minor typos: 'segment stokes semantically' should be 'segment strokes semantically,' and 'classifications .' has an extra space before the period.
  4. [Section III-C] The reweighting loss sets the background weight to zero during training, which effectively ignores background pixels. Please clarify how the network is expected to label background pixels at inference time, e.g., whether the background class score is still produced by the final softmax.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: SFSegNet's gains come from empirical training and comparisons, not from fitting a target defined by the model or by a self-citation chain.

full rationale

The paper makes no formal derivation claim; SFSegNet is an empirical architecture. The reported improvements (Tables III–V) come from training a ResNet34-based FCN with a background-reweighting loss and affine transform encoders, compared against FCN, LinkNet-34, and U-Net. None of the reported metrics is defined in terms of the model's own outputs by construction, and no fitted parameter is renamed as a prediction. The only self-citation is reference [25] for the dataset; that dataset is described and used as a benchmark, not invoked as an axiom or theorem. The Appendix's removal/merging of components in Huang's benchmark is a manual benchmark edit and is disclosed; it could bias external comparisons but does not make the network's objective function or accuracy metric equal to its training target. Baseline training with default parameters and the absence of error bars are methodological weaknesses, not circularity. The central claim is an empirical superiority claim that is, in principle, falsifiable by re-running the comparisons, so circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim relies on several domain assumptions about the dataset, the transferability of ImageNet features, and the behavior of the affine transform encoders, plus hand-chosen hyperparameters. No equations are derived, so there are no hidden fitted constants in a mathematical sense. The largest untested assumption is that the single shared ground truth and the relabeling of Huang's dataset provide a fair benchmark.

free parameters (4)
  • background loss weight = 0
    Chosen by hand to ignore background pixels in the loss; this is the core reweighting strategy and directly shapes the training objective (Section III-C).
  • training budget = 50 epochs
    All models are trained within 50 iterations/epochs; this fixed budget may under-train baselines and affects the fairness of comparisons (Section IV-B).
  • affine encoder localization layers = 1 convolution layer
    The mini-STN uses one conv layer in the localization network; no ablation over this architectural choice is given (Section III-B).
  • component correctness threshold = 75%
    A component is considered correctly labeled if at least 75% of its pixels are correct; this threshold defines the C-metric and was chosen for evaluation (Section IV-C).
assumptions (4)
  • domain assumption ResNet34 pretrained on ImageNet transfers to binary sketch images
    The encoder is initialized with ImageNet weights and fine-tuned on sketches, but no evidence is given that these features transfer to sparse line drawings (Section III-C).
  • domain assumption A single shared ground truth per category is semantically valid
    The dataset assigns the same part labels to all sketches in a class; if part definitions are ambiguous across drawing styles, the labels are unreliable (Section IV-A).
  • domain assumption Affine transform encoders can canonicalize stroke jitter
    The paper assumes the mini-STN modules learn to align shaking strokes, but only final accuracy is shown; the learned transformation matrices are not analyzed (Section III-B).
  • domain assumption Cross-entropy with background weight zero remains a valid objective
    With background pixels completely ignored, gradients only come from stroke pixels, yet at test time the model still must predict background; the training loss gives no direct supervision for that (Section III-C).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SFSegNet: Parse Freehand Sketches using Deep Fully Convolutional Networks." pith.science (2026). https://pith.science/paper/4ACAZCJE

@misc{pith2026190805389,
  author       = {Pith},
  title        = {Pith review of: SFSegNet: Parse Freehand Sketches using Deep Fully Convolutional Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ACAZCJE}},
  note         = {Machine review of arXiv:1908.05389}
}
read the original abstract

Parsing sketches via semantic segmentation is attractive but challenging, because (i) free-hand drawings are abstract with large variances in depicting objects due to different drawing styles and skills; (ii) distorting lines drawn on the touchpad make sketches more difficult to be recognized; (iii) the high-performance image segmentation via deep learning technologies needs enormous annotated sketch datasets during the training stage. In this paper, we propose a Sketch-target deep FCN Segmentation Network(SFSegNet) for automatic free-hand sketch segmentation, labeling each sketch in a single object with multiple parts. SFSegNet has an end-to-end network process between the input sketches and the segmentation results, composed of 2 parts: (i) a modified deep Fully Convolutional Network(FCN) using a reweighting strategy to ignore background pixels and classify which part each pixel belongs to; (ii) affine transform encoders that attempt to canonicalize the shaking strokes. We train our network with the dataset that consists of 10,000 annotated sketches, to find an extensively applicable model to segment stokes semantically in one ground truth. Extensive experiments are carried out and segmentation results show that our method outperforms other state-of-the-art networks.

Figures

Figures reproduced from arXiv: 1908.05389 by the authors.

Figure 1
Figure 1. Fig.1 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Illustration of the input, output and network architecture for SFSegNet. The input sketch has been amended after preprocessing. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Sketches in one topic are drawn twice by the same volunteer. Strokes [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: The first row includes several sketches from Huang’s dataset, which are [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 4
Figure 4. Figure 4: Examples of labeled sketches in one ground truth with the maximum number of components. The first row shows sketches drawn by experts from [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: SFSegNet produces a state-of-the-art performance on our dataset, compared with FCN-8s, LinkNet-34, and U-Net. According to its outperformance [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Comparison of segmentation results TABLE V ACCURACY(%) ON OUR DATASET, COMPARED WITH THE APPLICATION OF THE AFFINE TRANSFORM ENCODER(ATE). P-metric C-metric ATE X X Airplane 92.4 93.3 86.7 87.3 Bicycle 93.1 93.3 84.1 85.6 Candelabra 92.8 94.3 94.1 95.3 Chair 87.3 90.3 …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 20 canonical work pages

  1. [1]

    Linknet: Exploiting encoder represen- tations for efficient semantic segmentation,

    A. Chaurasia and E. Culurciello, “Linknet: Exploiting encoder represen- tations for efficient semantic segmentation,” in Visual Communications and Image Processing (VCIP), 2017 IEEE . IEEE, 2017, pp. 1–4

  2. [2]

    Semantic image segmentation with deep convolutional nets and fully connected crfs,

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,” arXiv preprint arXiv:1412.7062 , 2014

  3. [3]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,

    ——, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence , vol. 40, no. 4, pp. 834–848, 2018

  4. [4]

    Sketch- based modeling,

    F. Cordier, K. Singh, E. Etem, M.-P. Cani, and Y . Gingold, “Sketch- based modeling,” in Proceedings of the 37th Annual Conference of the European Association for Computer Graphics: Tutorials. Eurographics Association, 2016, p. 7

  5. [5]

    Sketch-based image retrieval: Benchmark and bag-of-features descriptors,

    M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa, “Sketch-based image retrieval: Benchmark and bag-of-features descriptors,” IEEE transactions on visualization and computer graphics , vol. 17, no. 11, pp. 1624–1636, 2011

  6. [6]

    A neural representation of sketch drawings,

    D. Ha and D. Eck, “A neural representation of sketch drawings,” arXiv preprint arXiv:1704.03477, 2017

  7. [7]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

  8. [8]

    Data-driven segmentation and labeling of freehand sketches,

    Z. Huang, H. Fu, and R. W. Lau, “Data-driven segmentation and labeling of freehand sketches,” ACM Transactions on Graphics (TOG) , vol. 33, no. 6, p. 175, 2014

Show all 29 references
  1. [9]

    Spatial transformer networks,

    M. Jaderberg, K. Simonyan, A. Zisserman et al. , “Spatial transformer networks,” in Advances in neural information processing systems , 2015, pp. 2017–2025

  2. [10]

    A curvature estimation for pen input seg- mentation in sketch-based modeling,

    D. H. Kim and M.-J. Kim, “A curvature estimation for pen input seg- mentation in sketch-based modeling,” Computer-Aided Design, vol. 38, no. 3, pp. 238–248, 2006

  3. [11]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural infor- mation processing systems , 2012, pp. 1097–1105. TABLE VI DATASET CONFIGURATION Huang Huang(with our config) Components R G B Components...

  4. [12]

    Multi-view pairwise relationship learning for sketch based 3d shape retrieval,

    H. Li, H. Wu, X. He, S. Lin, R. Wang, and X. Luo, “Multi-view pairwise relationship learning for sketch based 3d shape retrieval,” in Multimedia and Expo (ICME), 2017 IEEE International Conference on . IEEE, 2017, pp. 1434–1439

  5. [13]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3431–3440

  6. [14]

    Automated freehand sketch segmentation using radial basis functions,

    J. Pu and D. Gur, “Automated freehand sketch segmentation using radial basis functions,” Computer-Aided Design, vol. 41, no. 12, pp. 857–864, 2009

  7. [15]

    Enhancing sketch- based image retrieval by re-ranking and relevance feedback,

    X. Qian, X. Tan, Y . Zhang, R. Hong, and M. Wang, “Enhancing sketch- based image retrieval by re-ranking and relevance feedback,” IEEE Transactions on Image Processing , vol. 25, no. 1, pp. 195–208, 2016

  8. [16]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241

  9. [17]

    Image classifica- tion with the fisher vector: Theory and practice,

    J. S ´anchez, F. Perronnin, T. Mensink, and J. Verbeek, “Image classifica- tion with the fisher vector: Theory and practice,” International journal of computer vision , vol. 105, no. 3, pp. 222–245, 2013

  10. [18]

    Example-based sketch segmentation and labeling using crfs,

    R. G. Schneider and T. Tuytelaars, “Example-based sketch segmentation and labeling using crfs,” ACM Transactions on Graphics (TOG), vol. 35, no. 5, p. 151, 2016

  11. [19]

    Sketch based interfaces: early processing for sketch understanding,

    T. M. Sezgin, T. Stahovich, and R. Davis, “Sketch based interfaces: early processing for sketch understanding,” in Proceedings of the 2001 workshop on Perceptive user interfaces . ACM, 2001, pp. 1–8

  12. [20]

    Automatic instrument segmentation in robot-assisted surgery using deep learning,

    A. A. Shvets, A. Rakhlin, A. A. Kalinin, and V . I. Iglovikov, “Automatic instrument segmentation in robot-assisted surgery using deep learning,” in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 2018, pp. 624–628

  13. [21]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

  14. [22]

    Deep spatial-semantic attention for fine-grained sketch-based image retrieval

    J. Song, Q. Yu, Y .-Z. Song, T. Xiang, and T. M. Hospedales, “Deep spatial-semantic attention for fine-grained sketch-based image retrieval.” in ICCV, 2017, pp. 5552–5561

  15. [23]

    Free hand-drawn sketch segmentation,

    Z. Sun, C. Wang, L. Zhang, and L. Zhang, “Free hand-drawn sketch segmentation,” in European Conference on Computer Vision. Springer, 2012, pp. 626–639

  16. [24]

    Going deeper with convolutions,

    C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9

  17. [25]

    Multi-column point-cnn for sketch segmentation,

    F. Wang, S. Lin, H. Li, H. Wu, J. Jiang, R. Wang, and X. Luo, “Multi-column point-cnn for sketch segmentation,” arXiv preprint arXiv:1812.11029, 2018

  18. [26]

    Data-driven method for sketch-based 3d shape retrieval based on user similar draw-style recommendation,

    F. Wang, S. Lin, H. Wu, R. Wang, and X. Luo, “Data-driven method for sketch-based 3d shape retrieval based on user similar draw-style recommendation,” in SIGGRAPH ASIA 2016 Posters . ACM, 2016, p. 34

  19. [27]

    Sketchsegnet: A rnn model for labeling sketch strokes,

    X. Wu, Y . Qi, J. Liu, and J. Yang, “Sketchsegnet: A rnn model for labeling sketch strokes,” in 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2018, pp. 1–6

  20. [28]

    Multi-scale context aggregation by dilated convolutions,

    F. Yu and V . Koltun, “Multi-scale context aggregation by dilated convolutions,” arXiv preprint arXiv:1511.07122 , 2015

  21. [29]

    Pyramid scene parsing network,

    H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2881–2890

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.