Pith. sign in

REVIEW 3 major objections 4 minor 27 references

Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that un-posed indoor RGB images suffice for metric 3D object detection when per-image 3D boxes, rather than keypoints or point clouds, carry the matching and mapping.

desk verdict Object-centric SfM that mostly works; the metric-scale assumption is a real gap in evidence, but the reported localization numbers suggest it holds better than the stress-test fears. read the letter →

arxiv 2505.23756 v1 pith:OOCYLT62 submitted 2025-05-29 cs.CV

Justin Lazarow , Kai Kang , Afshin Dehghan This is my paper
classification cs.CV
keywords 3Dobjectdetectionun-posedimagesobject-centricmappingorientedboundingboxesmetriccameralocalizationlearnedmatchingstructurefrommotionindoorscenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rooms from Motion asks whether objects can replace points as the fundamental primitive for structure-from-motion-style reconstruction. It claims that an unordered collection of indoor RGB images is enough to produce metric camera poses and a global 3D semantic map, using only per-image 3D oriented box detections and learned object matching—no keypoints, point clouds, voxel grids, or prior poses. If this is right, scene-level 3D object detection becomes a byproduct of localization and mapping rather than a post-processing step on reconstructed geometry, and map size scales with the number of objects rather than the size of the scene. The paper further claims that when prior poses are available, optimizing global boxes against per-frame corner observations substantially improves map quality.

What carries the argument

The carrying mechanism is the metric 3D oriented box as the common representation for detection, matching, and pose estimation. A single-image 3D box detector (CuTR) predicts boxes with per-object embeddings; two learned matching networks, one operating on object embeddings with box-derived positional encodings and one on the flattened eight-corner “box cloud” with 2D image features, produce object and corner correspondences. Kabsch-Umeyama alignment on matched corners, restricted to a 4-DoF problem (yaw plus translation, since gravity is assumed known from the detector), turns matched boxes into metric relative camera poses. Global poses come from rotation and translation averaging, and the same tracks support a form of bundle adjustment that reprojects the corners of each track's representative box and optimizes its center, dimensions, and yaw against per-image observations.

What would settle it

Take an indoor sequence with known camera poses, run the RGB-only pipeline, then rescale every detected box's dimensions by a fixed factor while keeping its center fixed; if the central claim is right, the estimated metric translations and final map should shift by roughly the same factor or matching should fail, whereas recovering the original correct poses would show that metric scale is not coming from the boxes.

Watch

Extended reading notes

Core claim

The central claim is that 3D oriented boxes are a sufficient geometric primitive for both metric localization and semantic mapping in indoor scenes. Rooms from Motion replaces the 2D keypoint matcher of traditional structure-from-motion with an object-centric matcher: each image is independently processed by a detector that outputs metric 3D boxes with embeddings, a learned matching network associates boxes and their eight corners across image pairs, and Kabsch alignment on matched corners yields metric relative poses. Rotation and translation averaging over the verified pairwise poses produces global camera poses, and a union-find over matched objects forms global object tracks whose representative boxes constitute the output map. The paper reports that this un-posed pipeline outperforms point- and volume-based 3D object detectors on two large indoor datasets in several settings and gives localization accuracy comparable to or better than point-based SLAM baselines, even on RGB-only input.

Load-bearing premise

The whole pipeline assumes the single-image detector can be trusted to give true-to-scale 3D boxes with a known up direction; if the boxes are wrongly sized or the up direction is off, the relative camera poses and the final object map lose their metric accuracy.

Editorial extensions

If this is right

  • Indoor 3D object detection can be performed without prior camera poses and without constructing point clouds or voxel grids, using only RGB(-D) image collections.
  • The cost and memory of the map representation scale with the number of objects in the scene, not with scene geometry, which the paper argues enables larger scenes.
  • When camera poses are available, the object-track bundle adjustment raises mapping precision and recall substantially over the un-posed variant.
  • Object-level matching can support relocalization: new captures of the same room can be registered against the stored object map at high recall.
  • The un-posed RGB pipeline can register almost all frames of sparse, non-continuous indoor captures, where a learning-based point SLAM baseline degrades.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: the same object-centric pipeline could be re-targeted to outdoor or mixed scenes as soon as a single-image metric box detector with gravity awareness exists for those domains, since none of the matching or averaging machinery is indoor-specific.
  • Going beyond the paper, combining object-level matches with conventional keypoint matches would likely raise robustness in scenes with very few objects; the reported taxonomy and field-of-view ablations suggest sparsity of detected objects is the main failure mode.
  • If box-scale consistency held across cameras, the map could serve as a metric prior for incremental navigation: new images could relocalize against stored boxes and update tracks without re-running global averaging, a step the paper only begins with relocalization.
  • A testable extension is to use the object track's class distribution and box parameters as a lightweight semantic retrieval index over a building, enabling query-by-object across separate captures of the same room.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Rooms from Motion (RfM), a pipeline for scene-level 3D object detection from collections of unposed RGB or RGB-D images. Instead of keypoint matching and point-cloud aggregation, RfM detects metric 3D oriented boxes per image with the Cubify Transformer (CuTR), matches objects and box corners across views with learned matchers, estimates metric relative poses via Kabsch alignment on matched 3D corners, averages rotations and translations into global poses, forms object tracks, and optionally refines representative 3D boxes with bundle adjustment. Experiments on CA-1M and ScanNet++ report both mapping (AP/AR) and localization (ARE/ATE, registration rate) results, comparing against FCAF, UniDet3D, ImVoxelNet, ImGeoNet, DROID-SLAM, and CUT3R. The central claim is that object-centric 3D boxes are a sufficient primitive for metric localization and semantic mapping without a priori camera poses or point clouds.

Significance. If the main claims hold, RfM is a conceptually valuable step: it replaces keypoint-based structure-from-motion with an object-centric matcher and produces sparse, semantic, parametric maps whose size is proportional to the number of objects. The paper includes large-scale evaluation (107 CA-1M sequences and 50 ScanNet++ sequences), standard metrics, SE(3)-only alignment for unposed results, and ablations of taxonomy and field of view. A notable strength is that the mapping comparisons use independent baselines and the localization comparisons include strong point-based systems. The main weakness is that the monocular RGB pipeline's metric accuracy rests on an unverified and load-bearing assumption about CuTR's per-image box scale consistency.

major comments (3)
  1. [§3.4, §3.5, Table 3] The monocular RGB pipeline's metric accuracy rests on CuTR boxes being metrically accurate and scale-consistent across every view, but the paper provides no direct evidence for this. Section 3.4 states that box corners are 'expected to already be metrically accurate', and Section 3.5 fixes the scale parameter during translation averaging, so any per-image scale bias or drift in CuTR propagates directly into the Kabsch relative poses and the global map. I request (i) per-image predicted-versus-ground-truth box dimension/scale error statistics on held-out sequences, (ii) a scale-consistency analysis across overlapping views, and (iii) an ablation that perturbs per-image box scale (e.g., multiplicative noise of 5% and 10%) and reports the effect on ATE and AP. Without such evidence, the claim of 'metric camera poses' from pure RGB is not established.
  2. [Abstract, Table 2] The abstract claims that RfM 'produces maps of higher quality than leading point-based and multi-view 3D object detection methods on CA-1M and ScanNet++'. This is contradicted by Table 2 for the unposed RGB setting on ScanNet++: RfM (Un-posed) scores 18.6/26.1/13.6/19.5 (AP15/AR15/AP25/AR25) versus ImGeoNet's 18.8/31.3/16.9/27.8. The blanket claim is only true for the posed variant, and even there not uniformly on all metrics. Please qualify the claim by configuration or restrict it to the datasets/metrics where it holds.
  3. [§3.4, Table 3, §1] The localization comparison is confounded by the known-gravity prior. RfM uses 4-DoF alignment (yaw and translation) with gravity provided, as stated in Section 3.4 and the Table 3 note, while DROID-SLAM and CUT3R do not use this prior. The reported localization advantage may therefore be partly due to the gravity prior rather than to the object-centric representation. Please add an ablation without gravity (full 6-DoF Kabsch alignment) or provide gravity-aligned variants of the baselines. Relatedly, Section 1's claim that RfM operates without 'any metric quantities' is overstated: CuTR is trained with metric supervision, and the pipeline consumes camera intrinsics and gravity.
minor comments (4)
  1. [Table 1] The FCAF entry is cited as [19], but the correct reference for FCAF3D is [18]; [19] is ImVoxelNet.
  2. [Appendix B.2.1] There is a typo: 'Giben the thresholded detections' should read 'Given the thresholded detections'.
  3. [§1, §3.1] Please rephrase 'without ... any metric quantities' to something like 'without metric camera poses, depth, or point clouds', since the method uses camera intrinsics and gravity and relies on a network trained with metric box supervision.
  4. [Table 3] The comparison uses 100 frames for RfM and CUT3R but 250/500 frames for DROID-SLAM; please clarify how the frame count affects the comparison, or add a DROID-SLAM 100-frame result for fairness.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the pipeline composes learned detectors and geometric solvers, and the unverified metric-scale assumption on CuTR is a correctness risk, not a circular step.

full rationale

Rooms from Motion's derivation chain is: per-image CuTR 3D boxes (Section 3.1) -> Cubify Match object/corner correspondences (Sections 3.2-3.3) -> Kabsch-Umeyama relative poses (Section 3.4) -> rotation and translation averaging (Section 3.5) -> object-track formation and optional bundle adjustment (Sections 3.5-3.6). Each step is either a geometric computation or a learned module with its own supervision; no step feeds the target quantity (absolute poses or global map) back into itself. In Section 3.4, the paper derives R'_12 by Kabsch alignment of matched corners and asserts metric translation because the input boxes are 'expected to already be metrically accurate'; this is an input assumption about CuTR, not a fitted constant or a renamed prediction. The evaluation compares against FCAF3D, ImVoxelNet, ImGeoNet, UniDet3D, DROID-SLAM, and CUT3R on CA-1M and ScanNet++ using standard AP/AR/ATE/ARE metrics, so the central claims have independent empirical content. CuTR [6] is the authors' earlier work and CA-1M originates from that paper, and CuTR's monocular scale accuracy is not directly quantified here; this is a legitimate correctness/robustness risk, but it does not make the derivation circular by construction. No equation in the paper reduces to its own inputs.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new physical entity. It relies on a learned single-image 3D box detector (CuTR) and a set of hand-tuned thresholds for matching and verification; these are free parameters that affect the reported results. The gravity prior and the rigid-scene assumption are domain assumptions shared with many SLAM systems.

free parameters (7)
  • detection score threshold tau = 0.25 (CA-1M), 0.2 (ScanNet++)
    Set per dataset in Section 3.2; controls which detections enter matching.
  • object matching score threshold = 0.5 (CA-1M), 0.4 (ScanNet++)
    Set per dataset in Section 3.2; determines accepted object matches.
  • box reprojection inlier threshold = 0.75 (IoU3D >= 0.25)
    Section 3.4; used to decide if a matched object supports a relative pose.
  • sample inlier ratio = 0.5
    Section 3.4; minimum fraction of inliers for a pose sample to be accepted.
  • corner inlier distance = 10 cm
    Section 3.4; used for corner inliers in global optimization.
  • track merging thresholds = generalized IoU -0.6, affinity 0.25, suppression IoU 0.15
    Appendix B.2.2; hand-tuned for duplicate track merging.
  • number of frames per sequence = 100
    Section 4; uniform subsampling of captures, affects graph connectivity.
assumptions (4)
  • domain assumption Gravity direction (pitch/roll) is known for each image.
    Section 3.4 reduces pose estimation to 4-DoF alignment using circular regression, relying on CuTR's assumption of gravity measurements. The paper does not test the method without this prior.
  • domain assumption CuTR produces metrically accurate, scale-consistent 3D boxes per image.
    Section 3.1 adopts CuTR as base detector requiring metric scale; the whole pipeline depends on this learned prior. The paper does not independently evaluate CuTR's scale error.
  • domain assumption Scene is rigid and static over the image collection.
    Standard structure-from-motion assumption, implicit in Sections 3.5 and 3.6; no dynamic-object handling is described.
  • domain assumption Each frame contains enough detectable objects for matching.
    Registration rates are 93-97% (Table 3), so some frames fail; the method cannot localize images with no objects (stated in Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping." pith.science (2026). https://pith.science/paper/OOCYLT62

@misc{pith2026250523756,
  author       = {Pith},
  title        = {Pith review of: Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOCYLT62}},
  note         = {Machine review of arXiv:2505.23756}
}
read the original abstract

We revisit scene-level 3D object detection as the output of an object-centric framework capable of both localization and mapping using 3D oriented boxes as the underlying geometric primitive. While existing 3D object detection approaches operate globally and implicitly rely on the a priori existence of metric camera poses, our method, Rooms from Motion (RfM) operates on a collection of un-posed images. By replacing the standard 2D keypoint-based matcher of structure-from-motion with an object-centric matcher based on image-derived 3D boxes, we estimate metric camera poses, object tracks, and finally produce a global, semantic 3D object map. When a priori pose is available, we can significantly improve map quality through optimization of global 3D boxes against individual observations. RfM shows strong localization performance and subsequently produces maps of higher quality than leading point-based and multi-view 3D object detection methods on CA-1M and ScanNet++, despite these global methods relying on overparameterization through point clouds or dense volumes. Rooms from Motion achieves a general, object-centric representation which not only extends the work of Cubify Anything to full scenes but also allows for inherently sparse localization and parametric mapping proportional to the number of objects in a scene.

Figures

Figures reproduced from arXiv: 2505.23756 by the authors.

Figure 2
Figure 2. An overview of Rooms from Motion. Two-view geometry is considered first. Semantic objects as 3D boxes are independently detected in individual frames using CuTR (Section 3.1). Objects are subsequently matched at the 3D box level (Sec. 3.2) and at the implicit 3D box corner level (Sec. 3.3). Relative pose can be estimated using the matched objects and corners between the images as point sets (Sec. 3.4). Averaging ope… view at source ↗
Figure 3
Figure 3. Bundle adjustment can be extended to a global 3D object track. Each observation of a track is characterized by an image and particular 3D box detection from CuTR. Since these individual 3D boxes (shown as thin, white boxes) are usually aligned well with the RGB image, they can be used to enforce a reprojection cost function by comparing the projected corners of these detections to the projected corners of the object… view at source ↗
Figure 4
Figure 4. Qualitative comparisons from CA-1M (top half) and ScanNet++ (bottom half) of resulting [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 18 canonical work pages

  1. [1]

    Agarwal, S., Mierle, K., Team, T.C.S.: Ceres Solver (10 2023), https://github.com/ceres-solver/ ceres-solver

    work page 2023

  2. [2]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops

    DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: Self-supervised interest point detection and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 224–236 (2018)

    work page 2018

  3. [3]

    arXiv preprint arXiv:2409.19152 (2024)

    Duisterhof, B., Zust, L., Weinzaepfel, P., Leroy, V ., Cabon, Y ., Revaud, J.: Mast3r-sfm: a fully-integrated solution for unconstrained structure-from-motion. arXiv preprint arXiv:2409.19152 (2024)

    arXiv 2024

  4. [4]

    Foundations of Crystallography 32(5), 922–923 (1976)

    Kabsch, W.: A solution for the best rotation to relate two sets of vectors. Foundations of Crystallography 32(5), 922–923 (1976)

    work page 1976

  5. [5]

    arXiv preprint arXiv:2409.04234 (2024)

    Kolodiazhnyi, M., V orontsova, A., Skripkin, M., Rukhovich, D., Konushin, A.: Unidet3d: Multi-dataset indoor 3d object detection. arXiv preprint arXiv:2409.04234 (2024)

    arXiv 2024

  6. [6]

    arXiv preprint arXiv:2412.04458 (2024)

    Lazarow, J., Griffiths, D., Kohavi, G., Crespo, F., Dehghan, A.: Cubify anything: Scaling indoor 3d object detection. arXiv preprint arXiv:2412.04458 (2024)

    arXiv 2024

  7. [7]

    In: European Conference on Computer Vision

    Leroy, V ., Cabon, Y ., Revaud, J.: Grounding image matching in 3d with mast3r. In: European Conference on Computer Vision. pp. 71–91. Springer (2024)

    work page 2024

  8. [8]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Li, K., DeTone, D., Chen, Y .F.S., V o, M., Reid, I., Rezatofighi, H., Sweeney, C., Straub, J., Newcombe, R.: Odam: Object detection, association, and mapping using posed rgb video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5998–6008 (2021)

    work page 2021

Show all 27 references
  1. [9]

    In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision

    Lindenberger, P., Sarlin, P.E., Pollefeys, M.: Lightglue: Local feature matching at light speed. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision. pp. 17627–17638 (2023)

    2023

  2. [10]

    International journal of computer vision60, 91–110 (2004)

    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision60, 91–110 (2004)

    2004

  3. [11]

    IEEE transactions on robotics33(5), 1255–1262 (2017)

    Mur-Artal, R., Tardós, J.D.: Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics33(5), 1255–1262 (2017)

    2017

  4. [12]

    arXiv preprint arXiv:2412.12392 (2024)

    Murai, R., Dexheimer, E., Davison, A.J.: Mast3r-slam: Real-time dense slam with 3d reconstruction priors. arXiv preprint arXiv:2412.12392 (2024)

    2024 arXiv

  5. [13]

    IEEE Robotics and Automation Letters4(1), 1–8 (2018)

    Nicholson, L., Milford, M., Sünderhauf, N.: Quadricslam: Dual quadrics from object detections as landmarks in object-oriented slam. IEEE Robotics and Automation Letters4(1), 1–8 (2018)

    2018

  6. [14]

    In: European Conference on Computer Vision

    Pan, L., Baráth, D., Pollefeys, M., Schönberger, J.L.: Global structure-from-motion revisited. In: European Conference on Computer Vision. pp. 58–77. Springer (2024)

    2024

  7. [15]

    In: European Conference on Computer Vision

    Pan, L., Pollefeys, M., Baráth, D.: Gravity-aligned rotation averaging with circular regression. In: European Conference on Computer Vision. pp. 97–116. Springer (2024)

    2024

  8. [16]

    io, available from https://www.rerun.io/ and https://github.com/rerun-io/rerun

    Rerun Development Team: Rerun: A visualization sdk for multimodal data (2024), https://www.rerun. io, available from https://www.rerun.io/ and https://github.com/rerun-io/rerun

    2024

  9. [17]

    In: 2011 International conference on computer vision

    Rublee, E., Rabaud, V ., Konolige, K., Bradski, G.: Orb: An efficient alternative to sift or surf. In: 2011 International conference on computer vision. pp. 2564–2571. Ieee (2011)

    2011

  10. [18]

    In: European Conference on Computer Vision

    Rukhovich, D., V orontsova, A., Konushin, A.: Fcaf3d: Fully convolutional anchor-free 3d object detection. In: European Conference on Computer Vision. pp. 477–493. Springer (2022)

    2022

  11. [19]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Rukhovich, D., V orontsova, A., Konushin, A.: Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2397–2406 (2022) 10

    2022

  12. [20]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Salas-Moreno, R.F., Newcombe, R.A., Strasdat, H., Kelly, P.H., Davison, A.J.: Slam++: Simultaneous localisation and mapping at the level of objects. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1352–1359 (2013)

    2013

  13. [21]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: Learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4938–4947 (2020)

    2020

  14. [22]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Sun, J., Shen, Z., Wang, Y ., Bao, H., Zhou, X.: Loftr: Detector-free local feature matching with transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8922–8931 (2021)

    2021

  15. [23]

    Advances in neural information processing systems34, 16558–16569 (2021)

    Teed, Z., Deng, J.: Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems34, 16558–16569 (2021)

    2021

  16. [24]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Tu, T., Chuang, S.P., Liu, Y .L., Sun, C., Zhang, K., Roy, D., Kuo, C.H., Sun, M.: Imgeonet: Image-induced geometry-aware voxel representation for multi-view 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6996–7007 (2023)

    2023

  17. [25]

    In: CVPR (2025)

    Wang*, Q., Zhang*, Y ., Holynski, A., Efros, A.A., Kanazawa, A.: Continuous 3d perception model with persistent state. In: CVPR (2025)

    2025

  18. [26]

    IEEE Transactions on Robotics39(4), 2912–2932 (2023)

    Wu, Y ., Zhang, Y ., Zhu, D., Deng, Z., Sun, W., Chen, X., Zhang, J.: An object slam framework for association, mapping, and high-level tasks. IEEE Transactions on Robotics39(4), 2912–2932 (2023)

    2023

  19. [27]

    unlabeled

    Yeshwanth, C., Liu, Y .C., Nießner, M., Dai, A.: Scannet++: A high-fidelity dataset of 3d indoor scenes. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12–22 (2023) 11 AAppendix A.1 Additional Experiments A.1.1 Ablations We explicitly ablate t...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.