REVIEW 3 major objections 7 minor 32 references
Locate n' Rotate: Two-stage Openable Part Detection with Foundation Model Priors
T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that a two-stage Transformer framework with separate decoders for part detection and motion prediction, each fused with a foundation-model prior (perceptual grouping from a lightweight SAM, geometric surface norms from a…
desk verdict Useful incremental OPD paper; the detection gain is credible, but the 1.2-point motion gain is fragile due to single-run results and checkpoint selection on the reporting split. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument hinges on two structural choices. First, the model splits detection and motion prediction into two Transformer decoder stacks (built on Mask2Former): a semantic segmentation decoder that fuses the OPD query features with perceptual grouping features from an EfficientSAM encoder via cross-attention, and a motion decoder that fuses the same query features with geometric features (surface normals) from a DSINE encoder. Second, training uses an optimal-transport assignment whose cost augments the usual class/mask cost with two motion terms: an origin match cost that penalizes drift of the predicted origin along the ground-truth axis, scaled by object diagonal length, and an axis match cost that penalizes the angle between predicted and ground-truth axes. These costs are what carry the claimed accuracy gain in motion prediction.
What would settle it
Run the same training protocol (OPDReal pretraining, OPDMulti fine-tuning, 60000 steps, checkpoint selection on validation +MAO) at least ten times with different random seeds, and evaluate on a held-out test split never used for checkpoint selection. If the mean detection mAP gain over OPDMulti falls below 4.9 minus its standard error, or if the +MAO gain is within one standard deviation of zero, the central superiority claim would fail. A simpler check: re-running the released code on the same split should reproduce the reported numbers within a small tolerance; otherwise the claim is not reproducible as stated.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that openable part detection improves when perceptual and geometric reasoning are separated into two decoder stages, each conditioned on a distinct foundation-model prior, and when the training-time assignment of predictions to ground-truth parts includes motion-aware costs. The perceptual grouping encoder (EfficientSAM) supplies segmentation priors that reduce confusion between distinct objects, while the geometric understanding encoder (DSINE) supplies surface-normal priors that pull predicted motion origins and axes closer to the true articulation geometry. Combined with an optimal-transport matching cost that penalizes origin drift along the ground-truth axis and axis angular error, this lifts part detection mAP from 32.9 to 37.8 and the strict +MAO motion metric from 16.0 to 17.2 on OPDMulti, with qualitative gains on in-the-wild images.
Load-bearing premise
The reported gains rest on the assumption that the OPDMulti validation split, used to pick the best checkpoint via the +MAO metric, is a stable and unbiased estimate of true performance; all results are single runs with no error bars, so the 1–5 percentage point margins could shrink or vanish if the split is unrepresentative or the runs are not reproducible.
Editorial extensions
If this is right
- A single RGB image suffices to tell a robot where a drawer is and along which axis to pull it, removing the need for depth sensors in routine articulated-object manipulation.
- Separating detection from motion prediction in the decoder prevents the two prediction heads from interfering, a design pattern that transfers to other joint detection-and-regression tasks.
- The motion-aware optimal-transport cost can be dropped into any detector that regresses axes or origins, not just openable parts.
- Because the perceptual and geometric encoders are swappable (SAM, EfficientSAM, Omnidata, DSINE), the framework is a general receptacle for foundation-model priors.
- Higher precision at low recall means a robot can run at a high confidence threshold and act on few false positives.
Reading between the lines
- The paper reports single-run results on a validation split used for checkpoint selection; a natural next test is multi-seed evaluation on a truly held-out split, since the claimed margins (1–5 points) may be optimistic under selection bias.
- The authors observe that geometric features also help part detection because pixels sharing a surface normal tend to belong to the same part; this suggests deeper fusion of normal maps into the detection branch could yield further gains than the current cross-attention layer.
- The same two-stage recipe—grouping prior for detection, geometric prior for regression—could transfer to other affordance tasks such as graspable-part detection or handle detection in cluttered scenes.
- The origin-match cost only penalizes origin drift along the ground-truth axis; adding a cost on the prismatic translation direction could improve +MAO further on sliding parts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MOPD, a Mask2Former-based two-stage framework for openable part detection from RGB images. It augments the standard OPD encoder with a perceptual grouping encoder (EfficientSAM) and a geometric understanding encoder (DSINE), uses separate semantic-segmentation and motion decoders, and introduces an optimal-transport assignment cost that includes origin and axis motion terms. Experiments on the OPDMulti dataset report that MOPD improves over OPDMulti by 4.9 points in part-detection mAP and 1.2 points in +MAO motion-parameter accuracy, with additional ablations on encoder choices, optimal transport, and efficiency.
Significance. If the reported gains are stable, MOPD is a useful step for openable part detection: it demonstrates that foundation-model priors can be injected into both detection and motion branches, and the two-decoder design cleanly separates mask/type prediction from motion regression. The paper makes code and models publicly available, reports efficiency numbers, and includes several controlled ablations. The main significance is conditional on the robustness of the numbers: the detection gain is substantial, but the motion-parameter gain is small and the current evidence does not rule out checkpoint-selection or run-to-run noise.
major comments (3)
- [Section 4.2, Table 1, Fig. 3] The headline +MAO gain of 1.2 points (16.0 to 17.2) is not robust as reported. Section 4.2 states that the best checkpoint is selected on validation-set performance using the +MAO metric, Fig. 3 labels results on the 'val split', and the paper does not state that Table 1 uses a separate test split or multiple seeds. The spread of +MAO values across Tables 2 and 3 (15.9, 16.1, 16.5, 16.6, 16.7) is comparable to the claimed 1.2-point improvement, so the motion-parameter portion of the central claim may be within run-to-run variation. Please report results on a held-out test split, provide multiple-seed means with standard deviations, and clarify which split underlies Table 1.
- [Table 3, Section 4.3] The ablation results are internally inconsistent with the claim that the perceptual grouping encoder contributes to motion accuracy. Removing the perceptual encoder yields +MAO of 16.7, which is higher than the full model's 16.6, while the +M metric drops by only 0.7 points (36.1 to 35.4). Table 2 also shows +MAO of 16.5 when the geometric encoder is replaced with Omnidata. These numbers do not clearly support the conclusion in Section 4.3 that both feature encoders improve motion parameter prediction; the paper should reconcile them or temper the component-level claims.
- [Section 4.4, Table 1] The claim that the proposed optimal-transport motion assignment 'significantly' improves motion accuracy is not well supported by the reported single-run results. The OT variant improves +MAO from 16.6 to 17.2 (0.6 points) but decreases +MA from 20.7 to 20.1, and no error bars are given. Given the small magnitude and the conflicting +MA direction, the contribution of the motion match cost should be demonstrated with repeated runs or a larger margin before it is presented as a key contribution.
minor comments (7)
- [Section 3.5] The segmentation loss equation contains a typo: 'λcls + Lcls' should presumably be 'λclsLcls', since a bare λcls term is not a loss.
- [Section 3.5] In the motion loss equation, Lo is used for both the motion-origin loss and the pose loss, making the formula ambiguous; use distinct symbols such as Lorg and Lpose.
- [Section 3.6] The origin-match-cost equation has a vector-valued right-hand side while a cost should be scalar; it should be written with a norm, e.g., ||(Opred - Ogt) × Igt|| / Ldiag, and Ldiag should be defined as the ground-truth object diagonal.
- [Table 1] The row for MOPD(Optimal Transport) is missing a separator between the +M and +MA values ('37.720.1'); this should be corrected.
- [Table 2 caption] The caption text is garbled: 'In the third row. We frozen EfficientSAM and DSINE.' should be a complete sentence describing the frozen-weights variant.
- [Section 3.2, Section 3.4] The text uses 'FNN layers' in Section 3.2 but 'FFN' in Section 3.4; the acronym should be consistent (FFN is the standard feedforward network abbreviation).
- [Fig. 3 caption] The phrase 'valid dataset' should be 'validation dataset', and it would help to state explicitly whether the quantitative table uses the same split as the qualitative figure.
Circularity Check
No circularity found: the reported gains are empirical, supervised by external ground truth, and the pretrained encoders come from independent datasets; validation-based checkpoint selection is a statistical concern, not a derivation-from-inputs.
full rationale
The central claim is an empirical performance comparison on the OPDMulti benchmark, not a derived result. Training is end-to-end supervised by external ground-truth masks and motion parameters, and the two auxiliary encoders (EfficientSAM and DSINE) are pretrained on independent datasets (SA-1B for segmentation, surface-normal data for geometric estimation), so the perceptual and geometric priors are not constructed from the OPD labels being predicted. The optimal-transport matching cost Corigin and Caxis uses ground-truth axis and origin for assignment during training; this is metric-aligned with the +MA/+MAO evaluation, but it does not define the reported mAP values, which are computed by comparing predicted detections to held-out ground truth. The self-citations in the related-work sections are background references and are not load-bearing: no central premise or architectural choice is justified solely by an author-overlapping citation, and no uniqueness theorem or ansatz is imported from the authors' prior work. The only caveat, that the best checkpoint is selected on the validation split using +MAO and results are reported as single runs without error bars, is an experimental-validity and generalization concern, not a circularity: the numbers are not equal to the training objective by construction. Therefore no load-bearing step reduces, by definition or by construction, to its own inputs.
Assumptions & free parameters
free parameters (3)
- Loss weights λcd, λdice, λcls, λc, λa, λo, λpose
- Confidence threshold for valid predictions =
0.8
- Number of decoder layers L1 and L2
assumptions (3)
- domain assumption Ground truth masks, motion axes, origins, and part types in OPDMulti and OPDReal are accurate.
- domain assumption EfficientSAM and DSINE produce features that, after fine-tuning, improve the OPD task beyond the OPD encoder's own features.
- domain assumption The OPDMulti validation split is a representative test set and the reported single-run numbers are stable.
Cite this review
Pith. "Pith review of Locate n' Rotate: Two-stage Openable Part Detection with Foundation Model Priors." pith.science (2026). https://pith.science/paper/A3CTYP56
@misc{pith2026241213173,
author = {Pith},
title = {Pith review of: Locate n' Rotate: Two-stage Openable Part Detection with Foundation Model Priors},
year = {2026},
howpublished = {\url{https://pith.science/paper/A3CTYP56}},
note = {Machine review of arXiv:2412.13173}
}
read the original abstract
Detecting the openable parts of articulated objects is crucial for downstream applications in intelligent robotics, such as pulling a drawer. This task poses a multitasking challenge due to the necessity of understanding object categories and motion. Most existing methods are either category-specific or trained on specific datasets, lacking generalization to unseen environments and objects. In this paper, we propose a Transformer-based Openable Part Detection (OPD) framework named Multi-feature Openable Part Detection (MOPD) that incorporates perceptual grouping and geometric priors, outperforming previous methods in performance. In the first stage of the framework, we introduce a perceptual grouping feature model that provides perceptual grouping feature priors for openable part detection, enhancing detection results through a cross-attention mechanism. In the second stage, a geometric understanding feature model offers geometric feature priors for predicting motion parameters. Compared to existing methods, our proposed approach shows better performance in both detection and motion parameter prediction. Codes and models are publicly available at https://github.com/lisiqi-zju/MOPD
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the 3rd Conference on Robot Learning (2019)
Abbatematteo, B., Tellex, S., Konidaris, G.: Learning to generalize kinematic mod- els to novel objects. In: Proceedings of the 3rd Conference on Robot Learning (2019)
work page 2019
-
[2]
In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
Bae, G., Davison, A.J.: Rethinking inductive biases for surface normal estimation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
work page 2024
-
[3]
In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition
Chen, X., Liu, T., Zhao, H., Zhou, G., Zhang, Y.Q.: Cerberus transformer: Joint semantic, affordance and attribute parsing. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 19649–19658 (2022)
work page 2022
-
[4]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)
2022
-
[5]
In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition
Deitke, M., Han, W., Herrasti, A., Kembhavi, A., Kolve, E., Mottaghi, R., Sal- vador, J., Schwenk, D., VanderBilt, E., Wallingford, M., et al.: Robothor: An open simulation-to-real embodied ai platform. In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 3164–3174 (2020)
work page 2020
-
[6]
arXiv preprint arXiv:2404.03634 (2024)
Ding, K., Chen, B., Wu, R., Li, Y., Zhang, Z., Gao, H.a., Li, S., Zhu, Y., Zhou, G., Dong, H., et al.: Preafford: Universal affordance-based pre-grasping for diverse objects and environments. arXiv preprint arXiv:2404.03634 (2024)
arXiv 2024
-
[7]
arXiv preprint arXiv:2205.04382 (2022)
Eisner, B., Zhang, H., Held, D.: Flowbot3d: Learning 3d articulation flow to ma- nipulate articulated objects. arXiv preprint arXiv:2205.04382 (2022)
arXiv 2022
-
[8]
Hu, R., Savva, M., van Kaick, O.: Functionality representations and applications for shape analysis. In: Computer Graphics Forum. vol. 37, pp. 603–624. Wiley Online Library (2018)
work page 2018
Show all 32 references
-
[9]
In: European Conference on Computer Vision
Jiang, H., Mao, Y., Savva, M., Chang, A.X.: Opd: Single-view 3d openable part detection. In: European Conference on Computer Vision. pp. 410–426. Springer (2022)
2022
-
[10]
In: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition
Jiang, Z., Hsu, C.C., Zhu, Y.: Ditto: Building digital twins of articulated objects from interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition. pp. 5616–5626 (2022)
2022
-
[11]
In: 2008 IEEE International Conference on Robotics and Automation
Katz, D., Brock, O.: Manipulating articulated objects with interactive perception. In: 2008 IEEE International Conference on Robotics and Automation. pp. 272–277. IEEE (2008)
2008
-
[12]
arXiv preprint arXiv:2108.03272 (2021)
Li, C., Xia, F., Martín-Martín, R., Lingelbach, M., Srivastava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al.: igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.03272 (2021)
2021 arXiv
-
[13]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Li, X., Wang, H., Yi, L., Guibas, L.J., Abbott, A.L., Song, S.: Category-level articulated object pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3706–3715 (2020)
2020
-
[14]
In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755...
2014
-
[15]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Liu, L., Xu, W., Fu, H., Qian, S., Yu, Q., Han, Y., Lu, C.: Akb-48: A real-world articulated object knowledge base. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14809–14818 (2022)
2022
-
[16]
IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
Long,X.,Zheng,Y.,Zheng,Y.,Tian,B.,Lin,C.,Liu,L.,Zhao,H.,Zhou,G.,Wang, W.: Adaptive surface normal constraint for geometric estimation from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
2024
-
[17]
The International Journal of Robotics Research38(9), 1013–1019 (2019)
Martín-Martín, R., Eppner, C., Brock, O.: The rbo dataset of articulated objects and interactions. The International Journal of Robotics Research38(9), 1013–1019 (2019)
2019
-
[18]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Mo, K., Guibas, L.J., Mukadam, M., Gupta, A., Tulsiani, S.: Where2act: From pixels to actions for articulated 3d objects. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6813–6823 (2021)
2021
-
[19]
In: Proceedings of the IEEE/CVF international conference on computer vision
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al.: Habitat: A platform for embodied ai research. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9339–9347 (2019)
2019
-
[20]
In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Shen,B.,Xia,F.,Li,C.,Martín-Martín,R.,Fan,L.,Wang,G.,Pérez-D’Arpino,C., Buch, S., Srivastava, S., Tchapmi, L., et al.: igibson 1.0: a simulation environment for interactive tasks in large realistic scenes. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Sy...
2021
-
[21]
arXiv preprint arXiv:2303.14087 (2023)
Sun, X., Jiang, H., Savva, M., Chang, A.X.: Opdmulti: Openable part detection for multiple objects. arXiv preprint arXiv:2303.14087 (2023)
2023 arXiv
-
[22]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Wang, H., Sridhar, S., Huang, J., Valentin, J., Song, S., Guibas, L.J.: Normal- ized object coordinate space for category-level 6d object pose and size estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2642–2651 (2019)
2019
-
[23]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Wang, X., Zhou, B., Shi, Y., Chen, X., Zhao, Q., Xu, K.: Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8876– 8884 (2019)
2019
-
[24]
In: European conference on computer vision
Wang, Y., Wu, R., Mo, K., Ke, J., Fan, Q., Guibas, L.J., Dong, H.: Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In: European conference on computer vision. pp. 90–107. Springer (2022)
2022
-
[25]
arXiv preprint arXiv:2312.00863 (2023)
Xiong, Y., Varadarajan, B., Wu, L., Xiang, X., Xiao, F., Zhu, C., Dai, X., Wang, D., Sun, F., Iandola, F., et al.: Efficientsam: Leveraged masked image pretraining for efficient segment anything. arXiv preprint arXiv:2312.00863 (2023)
2023 arXiv
-
[26]
arXiv preprint arXiv:2006.14865 (2020)
Yan, Z., Hu, R., Yan, X., Chen, L., Van Kaick, O., Zhang, H., Huang, H.: Rpm- net: recurrent prediction of motion and parts from point cloud. arXiv preprint arXiv:2006.14865 (2020)
2020 arXiv
-
[27]
In: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition
Yang, G., Sun, D., Jampani, V., Vlasic, D., Cole, F., Chang, H., Ramanan, D., Freeman, W.T., Liu, C.: Lasr: Learning articulated shape reconstruction from a monocular video. In: Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition. pp. 15980–15989 (2021)
2021
-
[28]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Yang, G., Vo, M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H.: Banmo: Build- ing animatable 3d neural models from many casual videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2863– 2873 (2022)
2022
-
[29]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Zhao, H., Lu, M., Yao, A., Guo, Y., Chen, Y., Zhang, L.: Physics inspired op- timization on semantic transfer features: An alternative method for room layout estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 10–18 (2017)
2017
-
[30]
Zheng, Y., Li, X., Li, P., Zheng, Y., Jin, B., Zhong, C., Long, X., Zhao, H., Zhang,Q.:Monoocc:Diggingintomonocularsemanticoccupancyprediction.arXiv preprint arXiv:2403.08766 (2024)
2024 arXiv
-
[31]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Zhong, C., Zheng, Y., Zheng, Y., Zhao, H., Yi, L., Mu, X., Wang, L., Li, P., Zhou, G., Yang, C., et al.: 3d implicit transporter for temporally consistent keypoint dis- covery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3869–3880 (2023)
2023
-
[32]
IEEE Robotics and Automation Letters 5(4), 5159–5166 (2020)
Zhong, L., Zhang, Y., Zhao, H., Chang, A., Xiang, W., Zhang, S., Zhang, L.: Seeing through the occluders: Robust monocular 6-dof object pose tracking via model-guided video object segmentation. IEEE Robotics and Automation Letters 5(4), 5159–5166 (2020)
2020
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.