pith. sign in

arxiv: 1907.07011 · v2 · pith:U2H7XDTXnew · submitted 2019-07-16 · 💻 cs.CV

Improving Semantic Segmentation via Dilated Affinity

Boxi Wu , Shuai Zhao , Wenqing Chu , Zheng Yang , Deng Cai This is my paper

Pith reviewed 2026-05-24 20:56 UTC · model grok-4.3

classification 💻 cs.CV
keywords semantic segmentationdilated affinityauxiliary supervisionfeature refinementpropagationfully convolutional networksstructural constraints
0
0 comments X

The pith

Predicting dilated affinity as an auxiliary task improves segmentation features and enables fast refinement.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that requiring a segmentation network to predict both labels and dilated affinity, a sparse map of pixel relationships, builds structural awareness directly into the model. This joint training produces more robust features that yield finer initial segmentations. The affinity predictions then support a lightweight propagation step that further refines the output. The method adds only minor computational cost and delivers consistent gains when added to existing state-of-the-art models across benchmarks.

Core claim

By adding explicit supervision on dilated affinity alongside semantic segmentation, the network learns to capture pixel relationships at multiple scales. This dual output improves the quality of the segmentation predictions during training and supplies the information needed for a fast propagation process that corrects errors in the initial map.

What carries the argument

Dilated affinity, a sparse version of pair-wise pixel affinity predicted as an extra network output, which encodes structural relationships between pixels and supports both feature learning and post-prediction refinement.

If this is right

  • Joint training with dilated affinity produces robust feature representations that improve segmentation quality.
  • The affinity output can be used in a fast propagation process to refine the initial segmentation results.
  • The framework can be applied to existing state-of-the-art models with only minor additional expense.
  • Consistent performance gains appear on multiple benchmark datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The auxiliary affinity task could be combined with other dense-prediction objectives to improve performance further.
  • The propagation step might extend naturally to video or 3D data where structural consistency across frames or views is needed.
  • If the learned affinities capture long-range dependencies reliably, the method may reduce reliance on hand-crafted post-processing rules.

Load-bearing premise

That the affinity signal learned as an auxiliary task will transfer to meaningfully better segmentation features and that the propagation step will produce reliable refinements without introducing new errors or requiring dataset-specific tuning.

What would settle it

Running the dilated affinity branch plus propagation on a standard benchmark such as Cityscapes or PASCAL VOC and measuring no increase, or a decrease, in mean intersection-over-union compared with the base model would falsify the claim.

Figures

Figures reproduced from arXiv: 1907.07011 by Boxi Wu, Deng Cai, Shuai Zhao, Wenqing Chu, Zheng Yang.

Figure 1
Figure 1. Figure 1: (a) The original image and the segmentation results of DeepLabv3+. (b) Noisy prediction [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overall architecture of our method. y1 and y2 are the semantic labels of pixel x1 and x2. And a1,2 is their affinity: a1,2 = a2,1 =  1 if y1 = y2 0 otherwise (1) When capturing the affinity of a pair of pixels, we only consider pixels within a restricted area since distant pixels lose locality and the complexity of modeling every pair of pixels grows rapidly with the size of the feature map. On the ot… view at source ↗
Figure 3
Figure 3. Figure 3: The proportion of n0 to n8 changes with the rate of dilated affinity. Vertical axis shows the percentage of n0 to n8 with respect to all the pixels. Horizontal axis shows the corresponding rate. Image (a) and (b) is the statistics of PASCAL VOC 2012 train set and Cityscapes train set respectively. Directly using the inverse frequency based on positive neighbors may result in absurdly large weights to sampl… view at source ↗
Figure 4
Figure 4. Figure 4: shows the accuracy of dilated affinity with respect to different weighting schemes and dilation rates. The accuracies of affinity, especially those of n5 to n8, is important for our affinity propagation process. For n0 to n3, neighbor-reweight has the best performance, while for n4 to n8, sqrt-reweight and baseline achieve a better performance. (a) Affinity accuracy of sqrt￾reweight (b) Affinity accuracy o… view at source ↗
read the original abstract

Introducing explicit constraints on the structural predictions has been an effective way to improve the performance of semantic segmentation models. Existing methods are mainly based on insufficient hand-crafted rules that only partially capture the image structure, and some methods can also suffer from the efficiency issue. As a result, most of the state-of-the-art fully convolutional networks did not adopt these techniques. In this work, we propose a simple, fast yet effective method that exploits structural information through direct supervision with minor additional expense. To be specific, our method explicitly requires the network to predict semantic segmentation as well as dilated affinity, which is a sparse version of pair-wise pixel affinity. The capability of telling the relationships between pixels are directly built into the model and enhance the quality of segmentation in two stages. 1) Joint training with dilated affinity can provide robust feature representations and thus lead to finer segmentation results. 2) The extra output of affinity information can be further utilized to refine the original segmentation with a fast propagation process. Consistent improvements are observed on various benchmark datasets when applying our framework to the existing state-of-the-art model. Codes will be released soon.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes augmenting semantic segmentation networks with an auxiliary dilated affinity prediction task, where dilated affinity is a sparse form of pairwise pixel affinity. Joint supervision on this task is claimed to yield more robust features and finer initial segmentations, while the predicted affinities enable a subsequent fast propagation step to refine the output. The authors assert that applying this framework to existing state-of-the-art models produces consistent improvements across multiple benchmark datasets.

Significance. If the empirical claims hold after proper validation, the method offers a lightweight mechanism for incorporating structural pairwise information directly into the network without hand-crafted rules or expensive inference. The dual role of the affinity head (auxiliary supervision plus refinement) is conceptually appealing and could be broadly applicable if the gains are shown to be robust rather than dataset-specific.

major comments (2)
  1. [Experimental section] The central claim requires that joint training on dilated affinity produces segmentation features meaningfully superior to those from the segmentation loss alone, and that the propagation step yields net-positive refinements. However, the manuscript provides no isolated ablation separating the auxiliary loss contribution from the refinement step, nor any direct evaluation of affinity prediction accuracy against ground-truth pairwise affinities. This omission leaves both load-bearing assumptions untested.
  2. [Method and Experiments] The propagation refinement is presented as reliable and fast, yet no analysis is given of failure modes, such as when predicted affinities are only weakly correlated with semantic boundaries or when misclassifications are propagated. Without such analysis or quantitative metrics on refinement error rates, the net benefit of the second stage cannot be assessed.
minor comments (2)
  1. [Abstract and Introduction] The abstract introduces 'dilated affinity' without a formal definition or diagram in the provided text; a precise mathematical formulation (e.g., the dilation kernel size and sparsity pattern) should appear early in the method section for reproducibility.
  2. [Experiments] No mention is made of the computational overhead of the affinity head or propagation step relative to the baseline FCN; a table reporting FLOPs or runtime would clarify the 'minor additional expense' claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments highlight important gaps in experimental validation that we will address through targeted additions in the revised manuscript.

read point-by-point responses

  1. Referee: [Experimental section] The central claim requires that joint training on dilated affinity produces segmentation features meaningfully superior to those from the segmentation loss alone, and that the propagation step yields net-positive refinements. However, the manuscript provides no isolated ablation separating the auxiliary loss contribution from the refinement step, nor any direct evaluation of affinity prediction accuracy against ground-truth pairwise affinities. This omission leaves both load-bearing assumptions untested.

    Authors: We agree that isolating the auxiliary loss effect from the propagation refinement is necessary to substantiate the claims. In the revision we will add a dedicated ablation table comparing (i) the baseline segmentation network, (ii) the network trained with the additional dilated-affinity loss but without the propagation stage, and (iii) the full pipeline. We will also compute and report pixel-wise affinity prediction accuracy against ground-truth affinities derived from the semantic labels on the validation sets. These additions will directly test the two load-bearing assumptions. revision: yes

  2. Referee: [Method and Experiments] The propagation refinement is presented as reliable and fast, yet no analysis is given of failure modes, such as when predicted affinities are only weakly correlated with semantic boundaries or when misclassifications are propagated. Without such analysis or quantitative metrics on refinement error rates, the net benefit of the second stage cannot be assessed.

    Authors: We acknowledge that a quantitative characterization of refinement failure modes is currently missing. In the revised version we will include (a) a short analysis section describing conditions under which affinity predictions may be weakly correlated with boundaries, (b) per-dataset statistics on the fraction of pixels whose labels are changed by propagation together with the fraction of those changes that are correct versus incorrect relative to ground truth, and (c) selected qualitative examples illustrating both successful and unsuccessful refinements. These metrics will allow readers to evaluate the net benefit of the second stage. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical method paper with no derivations or load-bearing self-citations

full rationale

The paper presents an empirical CV method: joint training of segmentation with an auxiliary dilated affinity output, followed by a propagation refinement step. No equations, first-principles derivations, fitted parameters renamed as predictions, or uniqueness theorems appear in the provided text. The abstract and description contain no self-citations that justify core claims. Improvements are reported as observed benchmark gains when applied to existing models. The derivation chain is therefore self-contained and non-circular by the stated criteria.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities are described.

pith-pipeline@v0.9.0 · 5726 in / 939 out tokens · 19974 ms · 2026-05-24T20:56:15.067771+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    our method explicitly requires the network to predict semantic segmentation as well as dilated affinity... Joint training with dilated affinity can provide robust feature representations... refine the original segmentation with a fast propagation process

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · 6 internal anchors

  1. [1]

    TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

    Martín Abadi, Ashish Agarwal, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  2. [2]

    Loss max-pooling for semantic image segmentation

    Samuel Rota Bulò, Gerhard Neuhold, and Peter Kontschieder. Loss max-pooling for semantic image segmentation. In CVPR, pages 7082–7091, 2017

    work page 2017

  3. [3]

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2018

    work page 2018

  4. [4]

    Rethinking Atrous Convolution for Semantic Image Segmentation

    Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05587, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  5. [5]

    Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

    Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. CoRR, abs/1802.02611, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  6. [6]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016

    work page 2016

  7. [7]

    Pixellink: Detecting scene text via instance segmenta- tion

    Dan Deng, Haifeng Liu, Xuelong Li, and Deng Cai. Pixellink: Detecting scene text via instance segmenta- tion. In AAAI, 2018

    work page 2018

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009

    work page 2009

  9. [9]

    Mark Everingham, S. M. Ali Eslami, Luc J. Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98–136, 2015

    work page 2015

  10. [10]

    Golnaz Ghiasi and Charless C. Fowlkes. Laplacian pyramid reconstruction and refinement for semantic segmentation. In ECCV, 2016

    work page 2016

  11. [11]

    Bourdev, Subhransu Maji, and Jitendra Malik

    Bharath Hariharan, Pablo Arbelaez, Lubomir D. Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV

    work page

  12. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016

    work page 2016

  13. [13]

    Tsung-Wei Ke, Jyh-Jing Hwang, Ziwei Liu, and Stella X. Yu. Adaptive affinity fields for semantic segmentation. In ECCV, 2018

    work page 2018

  14. [14]

    Efficient inference in fully connected crfs with gaussian edge potentials

    Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NIPS, 2011. 9

    work page 2011

  15. [15]

    Multi-scale context intertwining for semantic segmentation

    Di Lin, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Multi-scale context intertwining for semantic segmentation. In ECCV, 2018

    work page 2018

  16. [16]

    Girshick, Kaiming He, and Piotr Dollár

    Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, 2017

    work page 2017

  17. [17]

    Learning affinity via spatial propagation networks

    Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, and Jan Kautz. Learning affinity via spatial propagation networks. In NIPS, pages 1519–1529, 2017

    work page 2017

  18. [18]

    Wei Liu, Andrew Rabinovich, and Alexander C. Berg. Parsenet: Looking wider to see better. CoRR, abs/1506.04579, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  19. [19]

    Affinity derivation and graph merge for instance segmentation

    Yiding Liu, Siyu Yang, Bin Li, Wengang Zhou, Jizheng Xu, Houqiang Li, and Yan Lu. Affinity derivation and graph merge for instance segmentation. In ECCV, 2018

    work page 2018

  20. [20]

    Semantic Image Segmentation via Deep Parsing Network

    Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, and Xiaoou Tang. Semantic image segmentation via deep parsing network. CoRR, abs/1509.02634, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  21. [21]

    Fully convolutional networks for semantic segmenta- tion

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015

    work page 2015

  22. [22]

    Michael Maire, Takuya Narihira, and Stella X. Yu. Affinity CNN: learning pixel-centric pairwise relations for figure/ground embedding. In CVPR, pages 174–182, 2016

    work page 2016

  23. [23]

    Megdet: A large mini-batch object detector

    Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: A large mini-batch object detector. In CVPR, pages 6181–6189, 2018

    work page 2018

  24. [24]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015

    work page 2015

  25. [25]

    Group normalization

    Yuxin Wu and Kaiming He. Group normalization. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII , pages 3–19, 2018

    work page 2018

  26. [26]

    Multi-Scale Context Aggregation by Dilated Convolutions

    Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. CoRR, abs/1511.07122, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  27. [27]

    Exfuse: Enhancing feature fusion for semantic segmentation

    Zhenli Zhang, Xiangyu Zhang, Chao Peng, Xiangyang Xue, and Jian Sun. Exfuse: Enhancing feature fusion for semantic segmentation. In ECCV, 2018

    work page 2018

  28. [28]

    Pyramid scene parsing network

    Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017

    work page 2017

  29. [29]

    Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip H. S. Torr. Conditional random fields as recurrent neural networks. In ICCV, 2015. 10

    work page 2015