pith. sign in

arxiv: 2606.02406 · v1 · pith:ZY37TIFGnew · submitted 2026-06-01 · 💻 cs.CV

Edge Prediction for Roof Wireframe Reconstruction with Transformers

Gustav Hanning , Ludvig Dill\'en , Jonathan Astermark , Johanna Lidholm , Viktor Larsson This is my paper

Pith reviewed 2026-06-28 15:31 UTC · model grok-4.3

classification 💻 cs.CV
keywords roof wireframe reconstructiontransformer3D edge predictionpoint cloudsemantic segmentationSfMwireframe model
0
0 comments X

The pith

A Transformer encoder-decoder reconstructs 3D roof wireframes from sparse point clouds using semantic subsampling and feature fusion.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a method to reconstruct complete 3D roof wireframe models for houses from limited input data consisting of sparse structure-from-motion point clouds along with ground-level semantic segmentations and depth maps. It proposes an end-to-end Transformer architecture that processes this data through dynamic subsampling based on semantics, augmentation with class features, and fusion with encodings from a frozen autoencoder before decoding query embeddings into edges via cross-attention. If effective, this would allow accurate 3D building models from cheaper, sparser data sources rather than requiring dense scans. A sympathetic reader would care because such reconstructions could improve applications in architecture, urban planning, and virtual reality with less expensive data collection.

Core claim

The central claim is that an end-to-end Transformer encoder-decoder inspired by DETR, combined with dynamic semantic-priority subsampling of the SfM point cloud, augmentation with Gestalt and ADE20k features, and fusion of point features with latent encodings from a frozen autoencoder, enables direct decoding of learned query embeddings into 3D wireframe edges that achieves a Hybrid Structure Score of 0.6476 on the HoHo 22k dataset, outperforming both handcrafted and learned baselines and placing second on the challenge leaderboard.

What carries the argument

The Transformer encoder-decoder with cross-attention that takes processed point features and decodes learned queries directly into 3D wireframe edges.

If this is right

  • The approach significantly outperforms handcrafted and learned baselines on the HoHo 22k dataset.
  • The model secures the second-highest position on the challenge's private leaderboard.
  • Dynamic subsampling and feature fusion allow effective use of sparse geometric and semantic data for edge prediction.
  • Direct decoding via cross-attention produces complete roof wireframe edges without intermediate steps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar Transformer processing could apply to reconstructing other linear structures like power lines or building facades from sparse data.
  • Freezing the autoencoder suggests that pre-trained feature extractors can be reused across related 3D tasks without retraining.
  • Success on this challenge implies the method may reduce reliance on dense point clouds in practical 3D reconstruction pipelines.

Load-bearing premise

The sparse SfM point cloud together with ground-level semantic segmentations and depth maps contain enough geometric and semantic information for the Transformer to accurately predict complete 3D roof wireframe edges.

What would settle it

A case where the method misses entire roof edges that are clearly supported by the input point cloud and segmentations but not prioritized in the dynamic subsampling or captured in the fused features.

Figures

Figures reproduced from arXiv: 2606.02406 by Gustav Hanning, Johanna Lidholm, Jonathan Astermark, Ludvig Dill\'en, Viktor Larsson.

Figure 1
Figure 1. Figure 1: Example scene from the HoHo 22k dataset. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The ground truth wireframe is misaligned with the SfM [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Our method predicts rooftop wireframe edges from a point cloud using a transformer encoder-decoder network. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ground truth (left) and predicted (right) roof wireframes for one scene in the HoHo 22k validation set. Our method can accurately predict longer edges but struggle with smaller details like chimneys. As the same vertex/edge can typically be seen in multiple views, the lifting is followed by a de-duplication step. The learned baseline utilizes a Perceiver-based [7] trans￾former architecture trained on fused… view at source ↗
read the original abstract

This paper presents a competitive solution to the S23DR Challenge 2026, which aims to reconstruct 3D house roof wireframe models from sparse SfM point clouds and ground-level semantic segmentations and depth maps. Our proposed method utilizes an end-to-end Transformer encoder-decoder architecture inspired by DETR. To effectively process the geometric and semantic data, the sparse SfM point cloud input is dynamically subsampled based on semantic priority and augmented with Gestalt and ADE20k class features. To further increase segmentation context, we fuse the point features with additional Gestalt feature encodings which are obtained by projecting the points into latent feature maps produced by a frozen autoencoder. Learned query embeddings are then decoded directly into 3D wireframe edges via cross-attention mechanisms. Evaluated on the "HoHo 22k" dataset, our approach significantly outperforms both handcrafted and learned baselines, achieving a Hybrid Structure Score (HSS) of 0.6476 and securing the second-highest position on the challenge's private leaderboard.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript presents a DETR-inspired Transformer encoder-decoder for 3D roof wireframe edge prediction from sparse SfM point clouds, ground-level semantic segmentations, and depth maps. The approach dynamically subsamples points by semantic priority, augments them with Gestalt/ADE20k features, fuses point features with encodings from a frozen autoencoder, and decodes learned queries into 3D edges via cross-attention. On the HoHo 22k dataset for the S23DR Challenge 2026, the method reports an HSS of 0.6476 and second place on the private leaderboard, outperforming handcrafted and learned baselines.

Significance. If the leaderboard result holds, the work provides a competitive, externally validated demonstration that Transformer architectures with semantic and learned-feature augmentation can recover complete 3D roof structures from limited inputs. The private-leaderboard evaluation supplies an independent falsification mechanism, and the method description is standard for challenge submissions. The absence of component ablations limits the ability to attribute gains to specific design choices.

minor comments (3)
  1. [Abstract] Abstract: the statement that the method 'significantly outperforms both handcrafted and learned baselines' would be strengthened by a brief quantitative comparison table or reference to the specific baseline scores on the same leaderboard.
  2. The manuscript does not report training procedure details, validation splits, or error bars; adding a short reproducibility paragraph would improve verifiability without altering the central empirical claim.
  3. Consider including a diagram of the encoder-decoder pipeline with the dynamic subsampling and frozen-autoencoder fusion steps to clarify data flow.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive review and the recommendation of minor revision. The primary observation concerns the lack of component ablations, which we address directly below. The work is a challenge submission whose main claim rests on external private-leaderboard validation rather than internal ablation studies.

read point-by-point responses

  1. Referee: The absence of component ablations limits the ability to attribute gains to specific design choices.

    Authors: We agree that ablations would strengthen attribution of performance gains. However, the manuscript is a concise challenge report whose primary evidence is the independent private-leaderboard result (0.6476 HSS, second place). The architecture follows a standard DETR-style encoder-decoder with semantic subsampling and frozen autoencoder fusion; each element was selected iteratively during development to meet the challenge constraints of sparse SfM input and ground-level imagery. Space limitations in the challenge format precluded exhaustive ablations, but the external validation already falsifies the null hypothesis that the combination does not work. If the editor requests, we can add a short paragraph summarizing the incremental contributions observed during development without new experiments. revision: no

Circularity Check

0 steps flagged

No significant circularity; empirical leaderboard result with no internal reductions

full rationale

The paper describes a standard end-to-end Transformer encoder-decoder pipeline for 3D roof wireframe edge prediction, trained on SfM point clouds augmented with semantic and Gestalt features, then evaluated via the external S23DR Challenge private leaderboard metric (HSS). No equations, predictions, or uniqueness claims are present that reduce by construction to fitted inputs or self-citations. The central result is an empirical score on held-out challenge data, falsifiable independently of any internal parameter fitting, satisfying the criteria for a self-contained non-circular derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The method rests on standard Transformer components and common feature-extraction practices; no new physical entities or ad-hoc constants are introduced beyond routine architectural hyperparameters.

axioms (1)
  • domain assumption A DETR-style cross-attention decoder can map augmented point features to complete 3D wireframe edges without additional geometric constraints.
    Invoked when the paper states that learned query embeddings are decoded directly into 3D wireframe edges via cross-attention.

pith-pipeline@v0.9.1-grok · 5714 in / 1273 out tokens · 23660 ms · 2026-06-28T15:31:25.672686+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    S23DR Competition at 3rd Workshop on Urban Scene Mod- eling @ CVPR 2026, 2026. 1

    2026

  2. [2]

    Layer Normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer Normalization.arXiv preprint arXiv:1607.06450,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    End- to-End Object Detection with Transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-End Object Detection with Transformers. InEuropean Conference on Computer Vision (ECCV), 2020. 1, 3, 7

    2020

  4. [4]

    Cambridge university press,

    Richard Hartley and Andrew Zisserman.Multiple View Ge- ometry in Computer Vision. Cambridge university press,

  5. [5]

    Metric3Dv2: A Versatile Monocular Geomet- ric Foundation Model for Zero-shot Metric Depth and Sur- face Normal Estimation.IEEE Trans

    Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3Dv2: A Versatile Monocular Geomet- ric Foundation Model for Zero-shot Metric Depth and Sur- face Normal Estimation.IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), 2024. 2

    2024

  6. [6]

    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

    Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. InInternational Conference on Machine Learning (ICML), 2015. 4

    2015

  7. [7]

    Perceiver: General Perception with Iterative Attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General Perception with Iterative Attention. InInternational Confer- ence on Machine Learning (ICML), 2021. 5

    2021

  8. [8]

    OneFormer: One Transformer to Rule Universal Image Segmentation

    Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer: One Transformer to Rule Universal Image Segmentation. InComputer Vision and Pattern Recognition (CVPR), 2023. 1

    2023

  9. [9]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations (ICLR), 2015. 4

    2015

  10. [10]

    The Hungarian Method for the Assignment Problem.Naval research logistics quarterly, 2(1-2):83–97,

    Harold W Kuhn. The Hungarian Method for the Assignment Problem.Naval research logistics quarterly, 2(1-2):83–97,

  11. [11]

    S23DR 2026 - Learned Submis- sion Baseline.https : / / huggingface

    Jack Langerman. S23DR 2026 - Learned Submis- sion Baseline.https : / / huggingface . co / jacklangerman/s23dr-2026-submission, 2026. 2, 4

    2026

  12. [12]

    Explaining Human Preferences via Met- rics for Structured 3D Reconstruction

    Jack Langerman, Denys Rozumnyi, Yuzhong Huang, and Dmytro Mishkin. Explaining Human Preferences via Met- rics for Structured 3D Reconstruction. InInternational Con- ference on Computer Vision (ICCV), 2025. 2

    2025

  13. [13]

    Decoupled Weight De- cay Regularization.International Conference on Learning Representations (ICLR), 2019

    Ilya Loshchilov and Frank Hutter. Decoupled Weight De- cay Regularization.International Conference on Learning Representations (ICLR), 2019. 4

    2019

  14. [14]

    Structured semantic 3D reconstruction (S23DR) challenge 2025 – winning solution,

    Jan Skvrna and Lukas Neumann. Structured Semantic 3D Reconstruction (S23DR) Challenge 2025 - Winning solution. arXiv preprint arXiv:2506.16421, 2025. 2

    work page arXiv 2025

  15. [15]

    Super-Convergence: Very Fast Training of Neural Networks Using Large Learn- ing Rates

    Leslie N Smith and Nicholay Topin. Super-Convergence: Very Fast Training of Neural Networks Using Large Learn- ing Rates. InArtificial intelligence and machine learning for multi-domain operations applications, pages 369–386. SPIE, 2019. 4

    2019

  16. [16]

    RoFormer: Enhanced transformer with Rotary Position Embedding.Neurocomputing, 568:127063,

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with Rotary Position Embedding.Neurocomputing, 568:127063,

  17. [17]

    Fourier Features Let Networks Learn High Frequency Functions in Low Di- mensional Domains.Neural Information Processing Systems (NeurIPS), 2020

    Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ra- mamoorthi, Jonathan Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Di- mensional Domains.Neural Information Processing Systems (NeurIPS), 2020. 3

    2020

  18. [18]

    S23DR 2026 - Handcrafted Submission Baseline.https : / / huggingface

    usm3d Team. S23DR 2026 - Handcrafted Submission Baseline.https : / / huggingface . co / usm3d / handcrafted_submission_2026, 2026. 4

    2026

  19. [19]

    Attention Is All You Need.Neural Information Processing Systems (NeurIPS), 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need.Neural Information Processing Systems (NeurIPS), 2017. 1 7

    2017

  20. [20]

    VGGT: Visual Geometry Grounded Transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual Geometry Grounded Transformer. InComputer Vi- sion and Pattern Recognition (CVPR), 2025. 1

    2025

  21. [21]

    MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details.Neural Information Processing Sys- tems (NeurIPS), 2026

    Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details.Neural Information Processing Sys- tems (NeurIPS), 2026. 1, 2

    2026

  22. [22]

    Semantic Under- standing of Scenes Through the ADE20K Dataset.Interna- tional Journal of Computer Vision (IJCV), 127(3):302–321,

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic Under- standing of Scenes Through the ADE20K Dataset.Interna- tional Journal of Computer Vision (IJCV), 127(3):302–321,