Pith. sign in

REVIEW 5 minor 34 references

Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels

T0 review · 0 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Jointly learning an affine alignment of misaligned labels and a building segmenter yields high-quality footprints without any clean ground-truth labels.

desk verdict Clean joint unsupervised alignment+segmentation for OSM-style building labels; works without golden labels and beats the unsupervised baselines. read the letter →

arxiv 2607.10841 v1 pith:Q3UR6CD3 submitted 2026-07-12 cs.CV

Venkanna Babu Guthula , Oswin Krause , Dimitri Gominski , Hui Zhang , Johan Mottelson , Ankit Kariryaa , Nico Lang , Christian Igel This is my paper · ORCID
classification cs.CV
keywords semanticsegmentationlabelnoisespatialtransformerbuildingfootprintsremotesensingunsupervisedalignmentOpenStreetMap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Satellite and aerial images are routinely paired with building footprints from separate sources such as OpenStreetMap, but the pairings are often systematically shifted or rotated by tens of pixels. A standard segmentation network simply memorizes those wrong locations. The paper shows that a lightweight transformation network can be trained end-to-end with the segmenter so that the two modules disentangle alignment from segmentation. The decisive ingredients are a self-supervised consistency loss that forces the transformer to recover known random warps of the noisy labels, plus geometric augmentations that cancel systematic bias. On both synthetic misalignments and real OpenStreetMap data the method recovers accurate building masks and the underlying registration at the same time, without ever seeing a perfectly aligned label.

What carries the argument

The TNet–SNet pair: the TNet predicts an affine matrix that realigns the given noisy mask to the SNet’s prediction; a consistency loss that applies a known random affine warp to the noisy mask and requires the TNet to recover its inverse supplies an absolute learning signal independent of the SNet.

What would settle it

Train the identical architecture on a dataset whose misalignments contain large non-affine local warps or systematic missing-label noise; if IoU_seg and IoU_align both collapse relative to the affine-only case, the central claim fails.

Watch

Extended reading notes

Core claim

An end-to-end “align and segment” architecture that couples a spatial-transformer network to any semantic segmentation backbone can learn both a corrective affine transform and a clean building mask from misaligned labels alone, provided a self-supervised consistency loss and strong geometric augmentations are used to block shortcut learning of the original misalignment.

Load-bearing premise

All misalignment inside each image patch can be captured by a single global rotation-plus-translation; local deformations, missing buildings, or non-affine distortions are treated as residual noise that the method does not model.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper proposes Align and Segment (AnS), a model-agnostic framework that jointly learns an affine label alignment (via a lightweight TNet operating on pairs of segmentation maps) and a semantic segmentation network (SNet) from misaligned image–label pairs, without any golden labels. It combines a cross-entropy loss on the realigned masks with a self-supervised consistency loss L_Con (MSE on transformation matrices plus IoU on re-warped masks under known random affines) and geometric data augmentations (flips and 90° rotations) to prevent the SNet from simply memorizing the systematic bias. Experiments on synthetic D_uni/D_bias variants of SpaceNet-2 (Las Vegas, Paris, Khartoum), the multi-city ReBO dataset, and real OpenStreetMap labels over San Juan show that AnS recovers both high IoU_seg and IoU_align, outperforming unsupervised baselines (SC, MA, AC) while remaining competitive with supervised ones (MR, ACN).

Significance. If the claims hold, AnS removes a major practical bottleneck in remote-sensing building segmentation: the inability to train directly on abundant but misaligned open sources such as OSM or Google Open Buildings. The method is plug-and-play with any SNet, requires no golden labels at training time, ships code and derived datasets, and is stress-tested across noise magnitudes (Fig. 4), cities, and real misalignments (Table 3). These properties make the contribution immediately usable for large-scale mapping in data-scarce regions and for other multi-modal registration-plus-segmentation tasks that share the same affine-bias pathology.

minor comments (5)
  1. Table 2 and Table 3 report point estimates only; adding standard deviations over a few random seeds (or at least noting that the ranking is stable) would strengthen the robustness claims already supported by Fig. 4.
  2. Section 3.1 / Eq. (1): the three-parameter affine is clearly scoped, yet a short sentence quantifying residual non-affine error on ReBO (e.g., mean residual after optimal affine) would help readers judge how much of the remaining IoU gap is attributable to the modeling choice versus label noise.
  3. Figure 3 and the qualitative OSM examples: the predicted Îy masks are visibly blurry; a brief remark on whether a simple morphological post-process or a boundary-aware loss would tighten the edges without harming alignment would be useful.
  4. Appendix A.1 Table 4: the freezing-vs-finetuning comparison is informative; moving a one-sentence summary of the main finding (frozen DINOv3-ConvNeXt best) into the main text would improve self-containment.
  5. Minor notation: the homogeneous-coordinate convention and the exclusion of non-overlapping border pixels from the losses are stated clearly, but a single clarifying sentence near Eq. (2) would prevent any ambiguity for readers less familiar with STNs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; empirical method paper whose claims are validated against held-out golden labels never used in training or selection.

full rationale

This is a standard empirical computer-vision methods paper proposing AnS (TNet + SNet) for joint affine alignment and building segmentation from misaligned labels. The training objective is the ordinary cross-entropy L_CE between the TNet-realigned mask and the SNet prediction, regularized by an absolute-consistency loss L_Con (Eq. 3) that applies a known random affine transform to the misaligned label and measures both matrix MSE and IoU against the inverse; geometric flips/rotations further debias systematic shifts. Model selection uses only the internal IoUlearn (overlap of realigned label with SNet output). All reported claims of high-quality segmentation and alignment (IoUseg, IoUalign in Tables 2–3 and Fig. 4) are computed against held-out golden labels that are never seen during training or selection (explicitly stated in Sec. 5). No equation reduces a claimed prediction to a fitted free parameter by construction; there is no uniqueness theorem, no load-bearing self-citation chain, and no renaming of a known empirical pattern. The single-global-affine-per-patch modeling choice (Eq. 1) is an explicit limitation (Sec. 7), not a circular derivation. The paper is therefore self-contained against its external benchmarks.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The central empirical claim rests on the modeling choice that misalignments are affine, on a handful of hand-chosen hyperparameters that control the transformation range and loss balance, and on the architectural decision to freeze a DINOv3-pretrained encoder. No new physical entities are postulated; the free parameters and domain assumptions are standard for an applied ML method paper.

free parameters (4)
  • c (tanh scaling) = 0.35
    Hand-set to 0.35 to limit predicted translations to roughly ±112 px and rotations to ±20°, chosen from expected misalignment magnitude.
  • λ (MSE weight in LCon) = 100
    Hand-set to 100 so that MSE and IoU terms have similar magnitude; not cross-validated.
  • learning rate / epochs / batch size = 1e-5 / 300 / 48
    AdamW lr=1e-5, 300 epochs, batch 48; standard but still free choices that affect final numbers.
  • augmentation probability = 0.5
    Each of five geometric transforms applied with 50 % probability; chosen to cancel systematic bias.
assumptions (3)
  • domain assumption Image–label misalignment is exactly an unknown affine transform A_θ* (rotation + translation) per patch.
    Stated in Sec. 3.1 and Eq. 1; the entire TNet is designed around this three-parameter family.
  • domain assumption A frozen DINOv3-pretrained ConvNeXt-Tiny encoder supplies sufficiently general features for building segmentation.
    Architecture choice justified by ablation in appendix; freezing is required to avoid the SNet simply memorizing the bias.
  • ad hoc to paper Random geometric flips and 90° rotations of image–mask pairs remove systematic translation bias in expectation.
    Introduced in Sec. 3.2 as the second countermeasure against shortcut learning; empirically effective on Dbias but not formally proved.
invented entities (2)
  • TNet (transformation network operating on two segmentation maps)
    purpose: Predicts the affine parameters that realign a misaligned label to the SNet prediction.
    Architectural module; no independent physical existence claimed outside the method.
  • LCon consistency loss (MSE on matrices + IoU on re-warped masks)
    purpose: Provides a learning signal for TNet that is independent of SNet predictions and of the unknown true alignment.
    New loss combination relative to AutoCorrect; still a training objective, not a physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels." pith.science (2026). https://pith.science/paper/Q3UR6CD3

@misc{pith2026260710841,
  author       = {Pith},
  title        = {Pith review of: Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3UR6CD3}},
  note         = {Machine review of arXiv:2607.10841}
}
read the original abstract

Supervised learning for image segmentation typically requires spatially aligned image and label sets. When images and labels originate from different sources, the pairing may be misaligned, which can significantly deteriorate the performance of the learned models. This is especially common in remote sensing, when aerial or satellite images are co-registered with labels from another source (e.g., OpenStreetMap). In this work, we propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment. Our align and segment (AnS) approach builds on the spatial transformer module to transform the misaligned labels using an affine transformation to provide a better learning target for a canonical semantic segmentation network. We prevent shortcut learning of misaligned labels in these semantic segmentation networks through a self-supervised regularization loss and show that it is complementary to data augmentation, especially for systematically misaligned training data. A decisive characteristic of our AnS approach is that it learns without requiring any golden labels. We experimentally show on both synthetic and real-world data from different cities that our approach enables high-quality building segmentation and precise label-image alignment at the same time. Code and derived datasets are available at https://github.com/venkanna37/align-and-segment

Figures

Figures reproduced from arXiv: 2607.10841 by the authors.

Figure 1
Figure 1. Misalignment of building footprints with independent imagery. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our align and segment (AnS) methodology. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative results learned on OpenStreetMap data. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Robustness to different noise levels. Sensitivity analysis of training the model with increasing noise-levels on the random misalignment dataset Duni and sys￾tematic misalignment dataset Dbias. Robustness [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Examples of Ix (input image), Iy (golden label) and Iy′ (artificially misaligned labels). In the third column, Iy′ was generated by assigning random misalignment with magnitude of ∥Aθ − Id∥F = 0.241, tx = −48, ty = 49, and r ≈ 4.5 ◦ . In the fourth column, Iy′ was gene…
Figure 6
Figure 6. Figure 6: Distribution of area of buildings in all cities. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Example images and predictions from Duni of Las Vegas. The first column shows the RGB image (Ix) and the second column shows the golden labels (Iy). The third column presents the misalignment by comparing the misaligned labels (Iy′ ) with Iy. The fourth and fifth colum…
Figure 8
Figure 8. Figure 8: Example images and predictions from Dbias of Las Vegas. The first column shows the RGB image (Ix) and the second column shows the golden labels (Iy). The third column presents the misalignment by comparing the misaligned labels (Iy′ ) with Iy. The fourth and fifth colu…
Figure 9
Figure 9. Figure 9: Example images and predictions from Duni of Paris The first column shows the RGB image (Ix) and the second column shows the golden labels (Iy). The third column presents the misalignment by comparing the misaligned labels (Iy′ ) with Iy. The fourth and fifth columns pr…
Figure 10
Figure 10. Figure 10: Example images and predictions from Dbias of Paris. The first column shows the RGB image (Ix) and the second column shows the golden labels (Iy). The third column presents the misalignment by comparing the misaligned labels (Iy′ ) with Iy. The fourth and fifth columns…
Figure 11
Figure 11. Figure 11: Example images and predictions from Duni of Khartoum. The first column shows the RGB image (Ix) and the second column shows the golden labels (Iy). The third column presents the misalignment by comparing the misaligned labels (Iy′ ) with Iy. The fourth and fifth colum…
Figure 12
Figure 12. Figure 12: Example images and predictions from Dbias of Khartoum. The first column shows the RGB image (Ix) and the second column shows the golden labels (Iy). The third column presents the misalignment by comparing the misaligned labels (Iy′ ) with Iy. The fourth and fifth colu…
Figure 13
Figure 13. Figure 13: Imagery and OpenStreetMap building footprints of San Juan city. [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Example images and predictions from San Juan city data. [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: The predicted transformation over the entire San Juan city. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references

  1. [1]

    Nature645(8080), 399–406 (2025)

    Bettencourt, L.M.A., Marchio, N.: Infrastructure deficits and informal settlements in sub-Saharan Africa. Nature645(8080), 399–406 (2025)

    2025

  2. [2]

    figshare (2016)

    Bradbury, K., Brigman, B., Collins, L., Johnson, T., Lin, S., Newell, R., Park, S., Suresh, S., Wiesner, H., Xi, Y.: Aerial imagery object identification dataset for building and road detection, and building height estimation. figshare (2016)

    2016

  3. [3]

    In: British Machine Vision Conference (2019)

    Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: AutoCorrect: Deep inductive align- ment of noisy geometric annotations. In: British Machine Vision Conference (2019)

    2019

  4. [4]

    Corporation, M.: Microsoft global ML building footprints.https : //planetarycomputer.microsoft.com/dataset/ms-buildings(2023), accessed: 2025-11-15

    2023

  5. [5]

    In: International Conference on Learning Representations (ICLR) (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021)

    2021

  6. [6]

    Etten,A.V.,Lindenbaum,D.,Bacastow,T.M.:SpaceNet:Aremotesensingdataset and challenge series (2019)

    2019

  7. [7]

    In: ACM Conference on Computing and Sustainable Societies (SIGCAS)

    Fobi, S., Conlon, T., Taneja, J., Modi, V.: Learning to segment from misaligned and partial labels. In: ACM Conference on Computing and Sustainable Societies (SIGCAS). p. 286–290 (2020)

    2020

  8. [8]

    In: Asian Conference on Computer Vision (ACCV)

    Girard, N., Charpiat, G., Tarabalka, Y.: Aligning and updating cadaster maps with aerial images by multi-task, multi-resolution deep learning. In: Asian Conference on Computer Vision (ACCV). pp. 675–690 (2019)

    2019

Show all 34 references
  1. [9]

    In: International Geoscience and Remote Sensing Symposium (IGARSS)

    Girard,N.,Charpiat,G.,Tarabalka,Y.:Noisysupervisionforcorrectingmisaligned cadaster maps without perfect ground truth data. In: International Geoscience and Remote Sensing Symposium (IGARSS). pp. 10103–10106 (2019)

    2019

  2. [10]

    In: Computer Vision and Pattern Recognition (CVPR)

    Girard, N., Smirnov, D., Solomon, J., Tarabalka, Y.: Polygonal building extraction by frame field learning. In: Computer Vision and Pattern Recognition (CVPR). pp. 5891–5900 (2021)

    2021

  3. [11]

    Science of Remote Sensing p

    Guthula, V.B., Oehmcke, S., Chilaule, R., Zhang, H., Lang, N., Kariryaa, A., Mot- telson, J., Igel, C.: Drone imagery for roof detection, classification, and segmenta- tion to support mosquito-borne disease risk assessment: The Nacala-roof-material dataset. Science of Remote S...

    2025

  4. [12]

    In: Computer Vision and Pattern Recognition (CVPR)

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016)

    2016

  5. [13]

    Hendrycks, D., Gimpel, K.: Gaussian error linear units (GELUs) (2016)

    2016

  6. [14]

    In: Advances in Neural Information Processing Systems (NeurIPS)

    Jaderberg, M., Simonyan, K., Zisserman, A., kavukcuoglu, k.: Spatial transformer networks. In: Advances in Neural Information Processing Systems (NeurIPS). vol. 28 (2015)

    2015

  7. [15]

    In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Kun, Y., Jianxin, W.: Probabilistic end-to-end noise correction for learning with noisy labels. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    2019

  8. [16]

    Li, K., Weng, X., Deng, Y., Meng, Y., Pang, C., Xia, G.S., Zhao, X.: DragOSM: Extractbuildingroofsandfootprintsfromaerialimagesbyaligninghistoricallabels (2025)

    2025

  9. [17]

    In: Computer Vision and Pat- tern Recognition (CVPR)

    Liu, S., Liu, K., Zhu, W., Shen, Y., Fernandez-Granda, C.: Adaptive early-learning correction for segmentation from noisy annotations. In: Computer Vision and Pat- tern Recognition (CVPR). pp. 2606–2616 (2022)

    2022

  10. [18]

    In: Advances in Neural Informa- tion Processing Systems (NeurIPS)

    Liu, S., Niles-Weed, J., Razavian, N., Fernandez-Granda, C.: Early-learning regu- larization prevents memorization of noisy labels. In: Advances in Neural Informa- tion Processing Systems (NeurIPS). vol. 33 (2020)

    2020

  11. [19]

    Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A ConvNet for the2020s.In:ComputerVisionandPatternRecognition(CVPR).pp.11976–11986 (2022)

    2022

  12. [20]

    In: International Conference on Learning Representations (ICLR) (2019)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (ICLR) (2019)

    2019

  13. [21]

    In: Inter- national Geoscience and Remote Sensing Symposium (IGARSS)

    Maggiori, E., Tarabalka, Y., Charpiat, G., Alliez, P.: Can semantic labeling meth- ods generalize to any city? the inria aerial image labeling benchmark. In: Inter- national Geoscience and Remote Sensing Symposium (IGARSS). pp. 3226–3229 (2017)

    2017

  14. [22]

    ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences2, 275–282 (2022)

    Maiti, A., Oude Elberink, S., Vosselman, G.: Effect of label noise in semantic segmentation of high resolution aerial images and height data. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences2, 275–282 (2022)

    2022

  15. [23]

    org.https://www.openstreetmap.org(2017)

    OpenStreetMap contributors: Planet dump retrieved fromhttps://planet.osm. org.https://www.openstreetmap.org(2017)

    2017

  16. [24]

    In: Medical Image Computing and Computer Assisted Intervention (MICCAI)

    Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomed- ical image segmentation. In: Medical Image Computing and Computer Assisted Intervention (MICCAI). pp. 234–241. Springer (2015)

    2015

  17. [25]

    International Journal of Computer Vision (IJCV)115(3), 211–252 (2015)

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV)115(3), 211–252 (2015)

    2015

  18. [26]

    Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, ...

    2025

  19. [27]

    Sirko, W., Kashubin, S., Ritter, M., Annkah, A., Bouchareb, Y.S.E., Dauphin, Y., Keysers, D., Neumann, M., Cisse, M., Quinn, J.: Continental-scale building detection from high resolution satellite imagery (2021)

    2021

  20. [28]

    In: Advances in Neural Processing Systems (NeurIPS)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Processing Systems (NeurIPS). vol. 30 (2017) Align and Segment 17

    2017

  21. [30]

    In: International Conference on Learning Representations (ICLR) (2023)

    Yao, J., Zhang, Y., Zheng, S., Goswami, M., Prasanna, P., Chen, C.: Learning to segment from noisy annotations: A spatial correction approach. In: International Conference on Learning Representations (ICLR) (2023)

    2023

  22. [31]

    In: European Conference on Computer Vision (ECCV) (2018)

    Zampieri,A.,Charpiat,G.,Girard,N.,Tarabalka,Y.:Multimodalimagealignment through a multiscale chain of neural networks with application to remote sensing. In: European Conference on Computer Vision (ECCV) (2018)

    2018

  23. [32]

    In: Medical Image Computing and Computer Assisted Intervention (MICCAI)

    Zhang, M., Gao, J., Lyu, Z., Zhao, W., Wang, Q., Ding, W., Wang, S., Li, Z., Cui, S.: Characterizing label errors: Confident learning for noisy-labeled image segmentation. In: Medical Image Computing and Computer Assisted Intervention (MICCAI). pp. 721–730 (2020)

    2020

  24. [33]

    In: Advances in Neural Information Processing Systems (NeurIPS)

    Zhang, Z., Sabuncu, M.R.: Generalized cross entropy loss for training deep neural networks with noisy labels. In: Advances in Neural Information Processing Systems (NeurIPS). pp. 8792–8802 (2018)

    2018

  25. [34]

    In: Medical Image Computing and Computer Assisted Intervention (MICCAI)

    Zhu, H., Shi, J., Wu, J.: Pick-and-Learn: Automatic quality evaluation for noisy- labeled image segmentation. In: Medical Image Computing and Computer Assisted Intervention (MICCAI). pp. 576–584 (2019)

    2019

  26. [35]

    In: International Geoscience and Remote Sensing Symposium (IGARSS)

    Zorzi, S., Bittner, K., Fraundorfer, F.: MapRepair: Deep cadastre maps alignment and temporal inconsistencies fix in satellite images. In: International Geoscience and Remote Sensing Symposium (IGARSS). pp. 1829–1832 (2020) 18 Guthula et al. A Appendix In the following, we pro...

    2020

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.