REVIEW 4 major objections 5 minor 37 references
KeystoneDepth: Visualizing History in 3D
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper converts antique stereo photographs into a 10,000-pair calibrated 3D dataset and shows a single stereo pair can drive real-time six-degree-of-freedom view synthesis on a phone.
desk verdict A genuinely useful historical stereo dataset, but the hand-set 45° FOV puts an affine distortion into every depth map and the abstract oversells the size. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the five-image GD scene representation: the left stereo view plus four rendered grayscale-plus-depth corner views, stored as texture-mapped triangle meshes, which together define a quadrilateral viewing volume inside which new viewpoints are rendered in real time. The double-reprojection technique is the training mechanism that makes this representation learnable: a depth-and-intensity mesh is rendered from a corner viewpoint and then back to the reference viewpoint, producing hole masks whose structure matches the disocclusions that real extrapolation creates, so the inpainting network needs no ground-truth novel views. A boundary mask marks foreground-side depth discontinuities and is fed to the network to keep silhouettes sharp, and the network itself is a partial-convolution encoder-decoder trained with separate intensity and depth losses.
What would settle it
Take stereographs that show objects of known physical size, recover depths with the assumed 45-degree field of view, and check whether the implied proportions and parallax motion are consistent with the known geometry across many scenes; if a different field of view is needed to make depths coherent, the central assumption fails. A complementary test is to render synthetic stereo pairs with known focal lengths and baselines and run the full pipeline, then compare recovered depth ratios and extrapolation ranges to the known values.
Extended reading notes
Core claim
The central claim is that a century of antique stereographs—uncalibrated, damaged, misaligned scans—can be converted into a large calibrated stereo dataset, and that a single rectified stereo pair is sufficient to generate new views over a continuous viewing volume at interactive rates. The pipeline filters the archive by feature matching and manual inspection, rectifies each pair with an uncalibrated rectification method under an assumed 45-degree vertical field of view, and estimates disparity from the horizontal component of dense optical flow. From about 29,480 raw scans it produced depth maps for 10,134 pairs. For rendering, the scene is stored as a reference grayscale-plus-depth image plus four synthesized corner views that define a quadrilateral viewing volume, with the missing regions filled by a partial-convolution network trained through double reprojection. The paper reports that this scheme extrapolates well beyond the input stereo baseline, avoids the depth-quantization artifacts of multiplane-image representations, and was preferred by 62.3% of participants in a blind comparison.
Load-bearing premise
The pipeline assumes a vertical field of view of 45 degrees to set each camera's focal length, and that single assumption fixes every depth map and every extrapolated view; if the true field of view differs, the recovered 3D geometry is systematically distorted.
Editorial extensions
If this is right
- The released dataset gives stereo and 3D-vision researchers more than 10,000 real historical stereo pairs with calibration, rectification, and depth, a scale and diversity no previous stereo dataset provides.
- A single antique stereo pair can drive six-degree-of-freedom view synthesis at real-time rates on a phone, so the same lightweight representation can power AR experiences beyond this collection.
- Because double reprojection generates realistic disocclusion holes from a single depth-plus-intensity image, the inpainting network can be trained on the target data itself rather than on ground-truth novel views.
- The filtering and rectification pipeline can be rerun as more of the 250,000-item archive is digitized, so the collection can grow beyond its current 10,134 depth-bearing pairs.
- The explicit-geometry-plus-inpainting approach avoids the depth-quantization 'stack of cards' artifact of learned multiplane images and supports a larger viewing volume, which is why most blind user-study participants preferred it.
Reading between the lines
- An extension the authors do not pursue is rescaling the depth maps using independently known scene dimensions (for example, buildings that still stand) to replace the hand-set field of view with measured calibration; this would turn relative depths into usable metric geometry.
- The double-reprojection training scheme is not tied to historical stereo: any single RGB-D image could be double-reprojected to create disocclusion-hole training data, so the method could transfer to modern depth cameras and indoor scenes.
- Because the recovered depths are only up to an unknown scale fixed by the assumed field of view, the physical size of the AR 'window' is not known; the experience is convincing in relative parallax but not in absolute metrics.
- A stronger validation than the paper's user study would compare views synthesized beyond the quad against real photographs of the same historical sites, but for most scenes no such photographs exist, so metric evaluation will require the known-size test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes the KeystoneDepth dataset and a view-synthesis system for antique stereographs. The authors downloaded 29,480 stereographs from the Keystone-Mast collection, automatically filtered and crowd-cropped them, rectified each surviving pair with Loop–Zhang rectification, estimated disparities with FlowNet2, and retained 10,134 processed pairs. Each dataset entry is intended to include metadata, bounding boxes, a rectified stereo pair, camera parameters, and disparity/depth maps. The second contribution is a representation built from one reference intensity-plus-depth image and four synthesized quad-corner views, with holes filled by a partial-convolution inpainting network trained using a new double-reprojection self-supervision scheme. A mobile AR application, KeystoneAR, is presented. The paper includes a manual evaluation of 711 samples (23% very few artifacts, 49% some artifacts, 28% failures) and a user study comparing the approach with Stereo Magnification.
Significance. If the calibration and depth maps are trustworthy, the dataset would be a substantial and unusual resource for historical stereo, rectification, and view-synthesis research. The paper is commendably detailed about its processing pipeline, openly reports its failure rates rather than cherry-picking successes, and includes a user study against a published baseline. The main correctness risk is the unvalidated calibration assumption in Section 3.2, which propagates into every depth map and synthesized view; in addition, the abstract overstates the processed dataset size, and the real-time claim is not backed by measurements. These issues are fixable with additional validation and by aligning the claims with the presented evidence, so the work is worthy of major revision rather than rejection.
major comments (4)
- [3.2] The vertical field of view is set by hand to 45 degrees because it produced "the most plausible depth scaling in our experiments." Since the rectification is uncalibrated and the disparity maps are obtained from optical flow, the camera intrinsics and baseline are not separately observable from the images; this assumed FOV therefore enters every calibration record, every depth map, and every view-extrapolation range in Section 4.1 and Section 6. Please provide a sensitivity analysis showing how depth maps and synthesized views change as the FOV is varied over a plausible range, or obtain an independent FOV estimate (e.g., from vanishing points or documented Keystone camera models). At a minimum, the dataset documentation should state explicitly that the depths are not metric and that the FOV is an unvalidated assumption rather than a measured quantity.
- [Abstract and Section 3] The abstract and introduction describe the collection as consisting of "tens of thousands of stereographs," but Section 3 reports that 29,480 stereographs were considered and disparity maps were computed for only 10,134 pairs. Moreover, the manual evaluation in Section 6 finds that 28% of 711 sampled entries are failures and only 23% have very few artifacts. The current wording overstates the actual processed resource and should be aligned with the 10,134 number, or the text should clearly distinguish raw scans from processed entries and present the projected expansion as a plan.
- [4 and 6] The abstract and Section 4 claim that the view-synthesis approach runs "at real-time rates on a mobile device," and Section 7 describes the KeystoneAR app, but Section 6 contains no timing measurements. Please report measured frame rates on a specific device, including the resolution and the number of GD images rendered, and if possible compare timings with the Stereo Magnification baseline. Without such measurements the real-time claim is unsupported.
- [5.3] The notation around the hole mask is inconsistent. The text defines M as "a binary mask corresponding to the holes that are to be inpainted," yet the network inputs are written as (I⊙M, D⊙M, B⊙M), and the losses are defined as L_valid = ||(Dp−Dt)⊙M||1 and L_hole = ||(Dp−Dt)⊙(1−M)||1. If M=1 marks holes, then the input multiplication zeroes out the valid region and L_valid is actually computed in the hole region; if M=1 marks valid pixels, the textual definition is wrong. Please make the mask convention explicit and consistent throughout, since this is essential for reproducing the training procedure.
minor comments (5)
- [1 and Figure 1] The introduction and Figure 1 caption say "over 10,000 rectified antique stereographs," while the abstract says "tens of thousands"; unify the terminology so that the reader can tell when raw scans versus processed entries are being described.
- [4.1] The formula rw = rh = 96b/dmax × √2/2 uses b, the camera baseline, but the paper does not explain how b is obtained from uncalibrated rectification and optical flow. Please define all variables and state whether b is in pixels, metric units, or arbitrary units.
- [5.4 and 8] There are several typographical errors: "a batch size if4" should be "a batch size of 4," "novel novel lightweight" in the conclusion repeats "novel," and the two references numbered [20] and [21] are the same paper and should be merged.
- [4.1] The abbreviation "GD image" is used in the caption of Figure 3 before it is defined in the text; define the term at its first occurrence in Section 4.1 or in the caption itself.
- [6] The user study reports only mean preference percentages (62.3% vs 27.8%). Please report the number of comparisons, the per-scene variance, and a confidence interval or statistical test so the reader can assess the strength of the preference.
Circularity Check
No circular derivation found; the hand-set 45° field of view is an openly stated calibration assumption, not a result reduced to its own inputs.
full rationale
The paper's central outputs are a dataset and a view-synthesis method, not a first-principles prediction whose derivation reuses its conclusion. Calibration uses Loop-Zhang rectification plus an explicitly declared assumption: 'we assumed a vertical field of view of 45 degrees, as that produced the most plausible depth scaling in our experiments' (Section 3.2). This is an uncontested heuristic choice, not a fitted parameter that is then renamed as a prediction; the paper presents it as an assumption and provides no claim that the depth scale is derived from other data. The disparity maps come from the external FlowNet2 model applied to the rectified images, and the view-synthesis network is trained on a subset of KeystoneDepth plus SUNCG and evaluated qualitatively and with a user study against Stereo Magnification; no loss or parameter is fitted to force a directly compared target. Self-citations such as [29] (Seitz and Dyer, view morphing) are standard related-work references and are not load-bearing justifications of the present method. The inpainting network being trained on KeystoneDepth data and then applied to other KeystoneDepth images is within-distribution supervised learning, not circular reasoning, because the training target is hole-free GD images and the evaluation is not the training loss. The hand-set FOV could be a correctness limitation because it enters every depth map and extrapolated view, but that is a robustness/validation concern, not a circularity pattern: no equation in the paper defines the claimed result in terms of, or equivalent to, its own input by construction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Vertical field of view =
45 degrees
- Quad size r_w, r_h =
96*b/d_max * sqrt(2)/2
- Depth inpainting loss weights lambda_hole, lambda_tv =
6 and 0.1
- Triangle removal depth threshold =
0.1 relative depth difference
assumptions (4)
- domain assumption Antique stereographs are captured with roughly valid stereo geometry, so uncalibrated rectification with Loop and Zhang is applicable.
- domain assumption The horizontal component of FlowNet2 optical flow provides acceptable disparity estimates on scans with noise, scratches, and exposure differences.
- ad hoc to paper The principal point of each stereo image is at the image center when the full image is available.
- standard math Standard projective geometry and light field rendering formulas hold.
Cite this review
Pith. "Pith review of KeystoneDepth: Visualizing History in 3D." pith.science (2026). https://pith.science/paper/5FMGWVR2
@misc{pith2026190807732,
author = {Pith},
title = {Pith review of: KeystoneDepth: Visualizing History in 3D},
year = {2026},
howpublished = {\url{https://pith.science/paper/5FMGWVR2}},
note = {Machine review of arXiv:1908.07732}
}
read the original abstract
This paper introduces the largest and most diverse collection of rectified stereo image pairs to the research community, KeystoneDepth, consisting of tens of thousands of stereographs of historical people, events, objects, and scenes between 1860 and 1963. Leveraging the Keystone-Mast raw scans from the California Museum of Photography, we apply multiple processing steps to produce clean stereo image pairs, complete with calibration data, rectification transforms, and depthmaps. A second contribution is a novel approach for view synthesis that runs at real-time rates on a mobile device, simulating the experience of looking through an open window into these historical scenes. We produce results for thousands of antique stereographs, capturing many important historical moments.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
3D Photos Now Rolling out on Facebook and in VR
Facebook. 3D Photos Now Rolling out on Facebook and in VR. https:// facebook360.fb.com/2018/10/11/ 3d-photos-now-rolling-out-on-facebook-and-in-vr/
work page 2018
-
[2]
New York Public Library. Stereograph Collection. https://digitalcollections.nypl.org/ collections/stereograph-collection
- [3]
-
[4]
M. Bertalmio, G. Sapiro, V . Caselles, and C. Ballester. Im- age inpainting. In Proceedings of the 27th Annual Con- ference on Computer Graphics and Interactive Techniques , SIGGRAPH ’00, 2000
work page 2000
-
[5]
G. Chaurasia, S. Duchene, O. Sorkine-Hornung, and G. Drettakis. Depth synthesis and local warps for plausi- ble image-based navigation. ACM Transactions on Graphics (TOG), 32(3):30, 2013
work page 2013
-
[6]
S. E. Chen and L. Williams. View interpolation for image synthesis. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques , SIG- GRAPH ’93, pages 279–288, New York, NY , USA, 1993. ACM
work page 1993
-
[7]
A. A. Efros and T. K. Leung. Texture synthesis by non- parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, 1999
work page 1999
- [8]
Show all 37 references
-
[9]
Hays and A
J. Hays and A. A. Efros. Scene completion using millions of photographs. ACM Transactions on Graphics (TOG), 2007
2007
-
[10]
Hedman and J
P. Hedman and J. Kopf. Instant 3d photography. ACM Trans- actions on Graphics, 37:1–12, 07 2018
2018
-
[11]
Holynski and J
A. Holynski and J. Kopf. Fast depth densification for occlusion-aware augmented reality. 2018
2018
-
[12]
Iizuka, E
S. Iizuka, E. Simo-Serra, and H. Ishikawa. Globally and Locally Consistent Image Completion. ACM Transactions on Graphics (Proc. of SIGGRAPH 2017), 2017
2017
-
[13]
E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE conference on computer vision and pattern recognition (CVPR), volume 2, page 6, 2017
2017
-
[14]
O. G. Inchang Choi, A. J. Troccoli, M. H. Kim, and J. Kautz. Extreme view synthesis. CoRR, abs/1812.04777, 2018
2018 arXiv
-
[15]
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[16]
Knapitsch, J
A. Knapitsch, J. Park, Q.-Y . Zhou, and V . Koltun. Tanks and temples: Benchmarking large-scale scene reconstruc- tion. ACM Transactions on Graphics, 36(4), 2017
2017
-
[17]
Levoy and P
M. Levoy and P. Hanrahan. Light field rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques , SIGGRAPH ’96, pages 31–42, New York, NY , USA, 1996. ACM
1996
-
[18]
Li and N
Z. Li and N. Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Computer Vision and Pattern Recognition (CVPR), 2018
2018
-
[19]
Stereograph Cards
Library of Congress. Stereograph Cards. http://www.loc.gov/pictures/collection/stereo/
-
[21]
G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro. Image inpainting for irregular holes using par- tial convolutions. arXiv preprint arXiv:1804.07723, 2018
2018 arXiv
-
[22]
Loop and Z
C. Loop and Z. Zhang. Computing rectifying homographies for stereo vision. In Computer Vision and Pattern Recog- nition, 1999. IEEE Computer Society Conference on. , vol- ume 1, pages 125–131. IEEE, 1999
1999
-
[23]
Mahajan, F
D. Mahajan, F. Huang, W. Matusik, R. Ramamoorthi, and P. N. Belhumeur. Moving gradients: a path-based method for plausible image interpolation. ACM Trans. Graph. , 28(3):42:1–42:11, 2009
2009
-
[24]
R. S. Overbeck, D. Erickson, D. Evangelakos, M. Pharr, and P. Debevec. A system for acquiring, compressing, and ren- dering panoramic light field stills for virtual reality. ACM Trans. Graph., 37, 2018
2018
-
[25]
Penner and L
E. Penner and L. Zhang. Soft 3d reconstruction for view synthesis. 36(6), 2017
2017
-
[26]
Ronneberger, P
O. Ronneberger, P. Fischer, and T. Brox. U-net: Convo- lutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234–241. Springer, 2015
2015
-
[27]
Scharstein and R
D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision, 47(1-3):7–42, Apr. 2002. 8 (a) (b) (c) (d) Figure 10: (a) Left stereo input image. (b) Depth map. (c) Reconstructed view at the following posit...
2002
-
[28]
Sch ¨ops, J
T. Sch ¨ops, J. L. Sch ¨onberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, and A. Geiger. A multi-view stereo benchmark with high-resolution images and multi- camera videos. In 2017 IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2017, Honolulu, ...
2017
-
[29]
S. M. Seitz and C. R. Dyer. View morphing. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’96, pages 21–30, New York, NY , USA, 1996. ACM
1996
-
[30]
I. B. Service. http://www.infosearchbpo.com/
-
[31]
Q. Shan, B. Curless, Y . Furukawa, C. Hernandez, and S. M. Seitz. Occluding contours for multi-view stereo. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014
2014
-
[32]
Keystone- Mast Collection
University of California Museum of Photography. Keystone- Mast Collection. http://ucr.emuseum.com/ collectionoverview/3631
-
[33]
Valentin, A
J. Valentin, A. Kowdle, J. T. Barron, N. Wadhwa, M. Dz- itsiuk, M. Schoenberg, V . Verma, A. Csaszar, E. Turner, I. Dryanovski, J. Afonso, J. Pascoal, K. Tsotsos, M. Le- ung, M. Schmidt, O. Guleryuz, S. Khamis, V . Tankovitch, S. Fanello, S. Izadi, and C. Rhemann. Depth from m...
2018
-
[34]
Zhang and T
Y . Zhang and T. Funkhouser. Deep depth completion of a sin- gle rgb-d image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 175–185, 2018
2018
-
[35]
Zhang, S
Y . Zhang, S. Song, E. Yumer, M. Savva, J.-Y . Lee, H. Jin, and T. Funkhouser. Physically-based rendering for indoor scene understanding using convolutional neural networks. The IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2017
2017
-
[36]
K. C. Zheng, A. Colburn, A. Agarwala, M. Agrawala, D. Salesin, B. Curless, and M. F. Cohen. Parallax photog- raphy: Creating 3d cinematic effects from stills. In Pro- ceedings of Graphics Interface 2009, GI ’09, pages 111–118, 2009
2009
-
[37]
T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely. Stereo magnification: Learning view synthesis using mul- tiplane images. In SIGGRAPH, 2018
2018
-
[38]
C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. High-quality video view interpolation using a layered representation. In ACM SIGGRAPH 2004 Papers , SIGGRAPH ’04, 2004. 10
2004
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.