Pith. sign in

REVIEW 3 major objections 6 minor 13 references

3D Reconstruction from Sketches

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that a single sketch, translated into a photo-like image by a CycleGAN and then run through MegaDepth, yields a plausible textured 3D reconstruction for a wide variety of drawings, while stitching multiple sketches…

desk verdict Honest and clearly written, but the central "wide variety" claim rests on qualitative evidence and a test-time preprocessing trick that makes the evaluation near-circular. read the letter →

arxiv 2505.14621 v1 pith:5ABUPLV5 submitted 2025-05-20 cs.CV cs.LG

Abhimanyu Talwar , Julien Laasri This is my paper · ORCID
classification cs.CVcs.LG
keywords 3DreconstructionfromsketchesCycleGANsingle-imagedepthestimationMegasketchpreprocessingimagestitchinghistoricalscenestyletransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a 3D reconstruction can be obtained from hand-drawn sketches by chaining three off-the-shelf pieces: stitch multiple overlapping sketches when possible, translate the drawing into a photo-like image with a CycleGAN, and feed that image to MegaDepth, a neural network pretrained on real photographs for single-image depth estimation. The authors report that the stitching step works only on artificial, stylistically uniform sketches and fails on real drawings by different artists, because local features such as ORB do not match consistently across artistic styles. They also report that the remaining single-sketch pipeline produces plausible textured depth surfaces on a wide variety of drawings, including historical Field Lane sketches and unrelated test images. They contribute a generated dataset of image–sketch pairs, built by applying a dodging-based pencil-sketch procedure to a building photo database, and use it to train the CycleGAN. If the claim holds, scenes that were never photographed—such as pre-photograph-era streets—can be visualized in 3D from surviving drawings alone.

What carries the argument

The load-bearing objects are the CycleGAN generator, which performs unpaired sketch-to-photo style transfer, and the pretrained MegaDepth network, which estimates a dense depth map from a single photo. Between them sits a preprocessing step the authors call dodging: a grayscale image is blurred, inverted, and divided pixel-wise into the original to produce a pencil-sketch look, then high-pass filtered and negated to approximate the noisiness of real online sketches. Applying this same dodge at test time is what lets real drawings match the CycleGAN's training distribution, making the subsequent depth estimation behave as if it were seeing a photograph. For the stitching stage, ORB features and a homography estimate are the mechanism, and the paper's result is that this mechanism fails when drawing styles differ.

What would settle it

Take a set of unseen hand-drawn sketches, run the pipeline exactly as described, and compare either the CycleGAN output or the final depth maps against known ground truth—for instance, buildings with available photographs from the same viewpoint. If omitting the dodging preprocessing noticeably degrades the reconstructions, or if a raw sketch fed directly to the CycleGAN produces depth maps as good as the dodged version, the claim that the preprocessing is the bridge would be undermined; more directly, if most drawings from diverse artists fail to yield recognizable building structure in the depth map, the 'wide variety' claim is false.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that once a sketch is put through the same dodging-based preprocessing used to create training sketches, a CycleGAN trained on unpaired image–sketch pairs can turn it into a realistic-looking image, and that MegaDepth, applied to that image without any fine-tuning, produces depth maps that faithfully track the building structure (facades, roofs, and street layout) in most of the tested drawings. The authors are explicit that the stitching step—ORB feature matching plus homography estimation—does not generalize to real artistic sketches, even for two drawings of the same building from the same viewpoint but different styles. They therefore frame the contribution as a 'solid single-sketch-to-image pipeline' rather than a full multi-view reconstruction system.

Load-bearing premise

The load-bearing premise is that running a real test drawing through the same dodging preprocessing used to generate the training sketches makes its style match the training distribution without erasing its content; if that fails, the claimed performance on a wide variety of drawings would not transfer to raw sketches.

Editorial extensions

If this is right

  • Any single hand-drawn building sketch can be converted into a textured 3D surface without 3D supervision, because the depth network only ever sees a photo-like image.
  • Pre-photograph historical scenes can be reconstructed from one surviving drawing, which is the motivating application for Field Lane.
  • The failure of ORB-based stitching implies that multi-view sketch reconstruction requires correspondence features that are robust to artistic style, not just to scale and perspective.
  • Improving the diversity of sketch styles in the training set—more artists, more sketch-generation procedures—should directly improve the CycleGAN's translation and hence the depth quality.
  • The approach can be applied to other drawing styles immediately, since the authors already report successes on drawings unrelated to the training database.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested alternative is that the CycleGAN step is not strictly necessary: MegaDepth may already produce usable depth from normalized sketch-like inputs, so a controlled ablation that skips the CycleGAN would reveal where the style transfer actually contributes.
  • The reliance on the same dodging procedure at train and test time suggests the CycleGAN may be learning a mapping from 'dodged line art' rather than from drawings in general; a stronger evaluation would hold out drawings that were not produced by that procedure.
  • Because the pipeline is content-agnostic beyond buildings, it should transfer to other line-drawn subjects such as landscapes, objects, and anatomy, and a synthetic benchmark with known 3D models could quantify the reconstruction error that the paper only assesses qualitatively.
  • The stitching failure on same-viewpoint, different-style drawings hints that the bottleneck is not viewpoint change but style invariance of local features; developing style-robust descriptors specifically for line art would be a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an end-to-end pipeline for reconstructing 3D scenes from hand-drawn sketches. Its five stages are: (1) stitching multiple overlapping sketches via ORB feature matching and homography estimation; (2) converting a sketch into a realistic image with a CycleGAN; (3) estimating depth from that image using the pre-trained MegaDepth network; (4) plotting the depth map as a 3D surface; and (5) texture-mapping the CycleGAN output onto the surface. The authors construct a dataset of 1,005 image-sketch pairs from the Zurich Building Image Database by applying a dodging/high-pass/negative filter to photographs, and train a CycleGAN on an unpaired 600-image/600-sketch subset. The paper reports that the stitching stage does not generalize to real drawings (e.g., the Field Lane etchings) but claims that the single-sketch pipeline 'performs quite well on a wide variety of drawings' (Abstract; §4.2). Notably, every test sketch is first processed with the same dodging procedure used to generate the training sketches before being fed to the CycleGAN (§3.3.2).

Significance. If substantiated, the single-sketch claim would be a useful proof of concept that off-the-shelf style transfer (CycleGAN) and monocular depth estimation (MegaDepth) can turn architectural drawings into textured 3D surfaces. The paper deserves credit for its transparency: it explicitly documents the stitching failure (§4.1), reports imperfect style-transfer results (§4.2), specifies the dataset-construction procedure in detail, and builds the pipeline from independent pre-trained components without introducing hidden fitted parameters. The main gap is evaluation: results are qualitative only, with no metrics, baselines, or ground-truth comparison, and the test-time preprocessing confines the CycleGAN to its training-input distribution. The paper's honest self-assessment is a strength, but it also shows that the 'wide variety of drawings' claim is currently supported mainly by inspection of selected figures.

major comments (3)
  1. [§3.3.2 and §4.2] The stress-test concern lands: the central claim that the pipeline 'performs quite well on a wide variety of drawings' (Abstract; §4.2) is not established for raw drawings, because every test sketch is first passed through the same dodging/high-pass/negative operator used in §3.2 to synthesize the CycleGAN training set. The generator is therefore evaluated only on inputs that are outputs of the exact deterministic filter defining its training domain; if the CycleGAN learned to invert that filter rather than to perform general sketch-to-image translation, the Field Lane and other hand-drawn results would not transfer to unprocessed sketches. No experiment compares raw and preprocessed inputs, so the generalization claim is untested for real drawings.
  2. [§4.2] The evaluation of the single-sketch pipeline is exclusively qualitative. There are no quantitative metrics (depth error against ground truth, shape similarity, or pixel accuracy), no baseline comparisons (e.g., running MegaDepth directly on the sketch, or substituting a different sketch-to-image method), and no error analysis over a test set. The same section concedes visible failures ('drawing (c) of figure 8 still looks like a drawing after reconstruction', 'drawing (a) did not get much color from the style transfer'), so without a systematic evaluation the claimed 'wide variety' performance cannot be judged.
  3. [Steps (3)-(5) of §3] The depth estimates that determine the 3D shape are never validated against ground truth. MegaDepth runs on the CycleGAN's synthetic output, and the resulting depth maps and surface plots (Fig. 1, Fig. 8) are shown without any reference depth or 3D model of the depicted scenes. Visual plausibility of a depth map is not a substitute for reconstruction error, so even if the sketch-to-image stage were reliable, the paper does not establish that the final 3D surface is an accurate reconstruction.
minor comments (6)
  1. [§4.2, §5] The word 'descent' is used where 'decent' is intended ('pretty descent' in §4.2 and 'pretty descent results' in §5); these typos should be corrected.
  2. [Title page and References] The manuscript is dated December 2018 while the arXiv posting is 2025; the related work reflects the 2018 state of the art (no coverage of modern monocular depth models, diffusion-based image translation, or learned local features), and the authors should either update the survey or frame the paper explicitly as a report of a 2018 study.
  3. [§3.3.2] The test-time output size is 480×480 while training used 320×320 crops; the paper should state how aspect ratios are handled and whether the larger output size affects quality.
  4. [§3.3.1] The description of the training split ('randomly chose a subset of 600 images, and independently chose 600 sketches') is ambiguous for a paired dataset; it should clarify that pairing is intentionally broken for the unpaired CycleGAN setup.
  5. [Figure 5] The caption of Fig. 5 refers to 'Fig. 5(c)' twice, making it unclear which panel is the blurred-inverted mask and which is the final sketch; the step descriptions should be aligned with the panel letters.
  6. [§3.2] The sketch-generation recipe is given only in prose; specifying the blur kernel size, the scale factor, and the high-pass filter parameters (or providing pseudocode) would improve reproducibility, and releasing the generated dataset and trained model would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline uses independent pretrained components, and the test-time preprocessing is a distribution-alignment step rather than a definitional reduction.

full rationale

The paper's derivation chain is an empirical pipeline composed of independently trained components: a CycleGAN trained on synthetic sketches generated from Zurich building images, and MegaDepth used as an external pretrained depth estimator. No parameter is fitted to the test drawings, and no equation defines the output in terms of the input. The only self-referential element is the test-time preprocessing in Section 3.3.2, where real test sketches are first passed through the same dodging/high-pass/negative procedure used in Section 3.2 to create the training sketches. This aligns the test inputs with the CycleGAN's training distribution, but it is not circularity: the generator still must produce a realistic image from the processed sketch, and the resulting image and depth map are not equal to any training target by construction. The paper explicitly acknowledges limitations (stitching fails on real drawings, the CycleGAN does not always produce realistic or colored outputs, and future work would vary the sketch-generation process to improve style robustness), which are generalization and evaluation concerns rather than circular reasoning. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameter renamed as a prediction. The central qualitative claim about single-sketch reconstruction is therefore self-contained with respect to circularity, however limited its quantitative support may be.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The system relies on no new physical entities or fitted physical constants. The free parameters are hand-chosen image processing and training hyperparameters. The key assumptions are domain assumptions about feature matching, style transfer generalization, and depth transfer, all of which are plausible but not rigorously validated in the paper.

free parameters (4)
  • dodging scale factor = 256
    Hand-chosen multiplier in the sketch-generation step (Section 3.2, step 2) to make the pencil-sketch output resemble online sketches. This value directly affects the appearance of all training data.
  • CycleGAN crop size = 320x320
    Chosen in Section 3.3.1 to increase regularization by discarding ~36% of pixels. This affects the training distribution and hence the generator's output.
  • CycleGAN test output size = 480x480
    Set at test time in Section 3.3.2, larger than training crops. No systematic study justifies this choice, and it may affect depth estimation quality.
  • learning rate = 0.0002
    Default CycleGAN learning rate used for 200 epochs with linear decay after 100 epochs (Section 3.3.1). Standard hyperparameter, not fitted to the target result.
assumptions (4)
  • domain assumption ORB features can reliably find correspondences between two drawings of the same scene made by different artists.
    Used in the stitching step (Section 3.1). The authors themselves report it fails on real drawings (Section 4.1), so this assumption is empirically violated for the main use case.
  • domain assumption A CycleGAN trained on sketch-image pairs from the Zurich Building Database generalizes to drawings of other scenes and styles after test-time dodging.
    The central enabling assumption behind the single-sketch claim (Sections 3.3.2 and 4.2). If the synthetic sketch distribution is too narrow, the CycleGAN will not transfer to real drawings.
  • domain assumption MegaDepth's pre-trained single-image depth estimator transfers to CycleGAN-generated pseudo-realistic images.
    MegaDepth is used as-is (Section 3, step 3) without fine-tuning. The reliability of the depth maps for synthetic-looking images is assumed, not demonstrated.
  • domain assumption The dodging procedure creates sketches that are representative of real artists' drawings.
    The training dataset (Section 3.2) is built entirely from this procedure. If the dodged output is far from real drawing style, the CycleGAN will not generalize, which is a load-bearing premise for the single-sketch claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3D Reconstruction from Sketches." pith.science (2026). https://pith.science/paper/5ABUPLV5

@misc{pith2026250514621,
  author       = {Pith},
  title        = {Pith review of: 3D Reconstruction from Sketches},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ABUPLV5}},
  note         = {Machine review of arXiv:2505.14621}
}
read the original abstract

We consider the problem of reconstructing a 3D scene from multiple sketches. We propose a pipeline which involves (1) stitching together multiple sketches through use of correspondence points, (2) converting the stitched sketch into a realistic image using a CycleGAN, and (3) estimating that image's depth-map using a pre-trained convolutional neural network based architecture called MegaDepth. Our contribution includes constructing a dataset of image-sketch pairs, the images for which are from the Zurich Building Database, and sketches have been generated by us. We use this dataset to train a CycleGAN for our pipeline's second step. We end up with a stitching process that does not generalize well to real drawings, but the rest of the pipeline that creates a 3D reconstruction from a single sketch performs quite well on a wide variety of drawings.

Figures

Figures reproduced from arXiv: 2505.14621 by the authors.

Figure 1
Figure 1. All steps of our pipeline for different drawings. Only sketch (d) required the stitching process. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Two neighbouring viewpoints of Field Lane at [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Process to stitch multiple drawings of the same [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Steps involved in creation of our dataset of [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Feature matching completely fails on the two per [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Feature matching fails for two sketches of the [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Rest of the pipeline applied to sketches of Field [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

  1. [1]

    David G. Lowe. Distinctive image features from scale- invariant keypoints.International Journal of Computer Vi- sion, 60(2):91–110, Nov 2004

    work page 2004

  2. [2]

    Surf: Speeded up robust features

    Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In Ale ˇs Leonardis, Horst Bischof, and Axel Pinz, editors,Computer Vision – ECCV 2006, pages 404–417, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg

    work page 2006

  3. [3]

    Rublee, V

    E. Rublee, V . Rabaud, K. Konolige, and G. Bradski. Orb: An efficient alternative to sift or surf. In2011 International Con- ference on Computer Vision, pages 2564–2571, Nov 2011

    work page 2011

  4. [4]

    Image-to-image translation with conditional adver- sarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. InComputer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, 2017

    work page 2017

  5. [5]

    Unpaired image-to-image translation using cycle- consistent adversarial networks

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. InComputer Vision (ICCV), 2017 IEEE International Conference on, 2017

    work page 2017

  6. [6]

    Liebowitz, A

    D. Liebowitz, A. Criminisi, and A. Zisserman. Creating ar- chitectural models from images. InAnnual Conference of the European Association for Computer Graphics (Eurograph- ics), volume 18, pages 39–50, 1999

    work page 1999

  7. [7]

    Ashutosh Saxena, Min Sun, and Andrew Y . Ng. Make3d: Learning 3d scene structure from a single still image.IEEE Trans. Pattern Anal. Mach. Intell., 31(5):824–840, May 2009

    work page 2009

  8. [8]

    Bros- tow

    Cl ´ement Godard, Oisin Mac Aodha, and Gabriel J. Bros- tow. Unsupervised monocular depth estimation with left- right consistency.CoRR, abs/1609.03677, 2016

    arXiv 2016

Show all 13 references
  1. [9]

    Megadepth: Learning single-view depth prediction from internet photos.CoRR, abs/1804.00607, 2018

    Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos.CoRR, abs/1804.00607, 2018

    2018 arXiv

  2. [10]

    Ramachandran and G

    P. Ramachandran and G. Varoquaux. Mayavi: 3D Visualiza- tion of Scientific Data.Computing in Science & Engineering, 13(2):40–51, 2011

    2011

  3. [11]

    Springer-Verlag, Berlin, Heidelberg, 1st edition, 2010

    Richard Szeliski.Computer Vision: Algorithms and Applica- tions. Springer-Verlag, Berlin, Heidelberg, 1st edition, 2010

    2010

  4. [12]

    Hpat indexing for fast object/scene recognition based on local appearance

    Hao Shao, Tom ´aˇs Svoboda1, Tinne Tuytelaars, and Luc Van Gool. Hpat indexing for fast object/scene recognition based on local appearance. In Erwin M. Bakker, Michael S. Lew, Thomas S. Huang, Nicu Sebe, and Xiang Sean Zhou, editors,Image and Video Retrieval, pages 71–80, Berl...

    2003

  5. [13]

    How to create a beautiful pencil sketch effect with opencv and python

    Michael Beyeler. How to create a beautiful pencil sketch effect with opencv and python. http://www.askaswiss.com/2016/01/ how-to-create-pencil-sketch-opencv-python. html. Accessed: 2018-12. 6

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.