Pith. sign in

REVIEW 2 major objections 4 minor 39 references

RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RoboPEPP claims that pre-training an encoder to predict the embeddings of masked robot joints makes it robustly estimate pose and joint angles from a single image, even under occlusion.

desk verdict A genuinely useful JEPA-for-robot-pose paper with a real core result on synthetic and occlusion data, but the real-world headline numbers are inflated by a protocol that gives RoboPEPP test-sequence fine-tuning and test-tuned hyperparameters the baselines never get. read the letter →

arxiv 2411.17662 v2 pith:A3HLPEAC submitted 2024-11-26 cs.RO cs.CV

Raktim Gautam Goswami , Prashanth Krishnamurthy , Yann LeCun , Farshad Khorrami This is my paper · ORCID
classification cs.ROcs.CV
keywords robotposeestimationjointangleself-supervisedlearningembeddingpredictivearchitectureocclusionrobustnessvisiontransformerkeypointdetectionsim-to-realtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a vision system can estimate an articulated robot's pose and joint angles from a single RGB image far more reliably under occlusion if the encoder is pre-trained to predict the appearance of masked joints from the surrounding image context. The proposed method, RoboPEPP, masks regions around four randomly chosen joints and trains an encoder-predictor pair to reconstruct the masked joints' embeddings, forcing the encoder to internalize the robot's physical structure and kinematic constraints. After this self-supervised pre-training, the same encoder-predictor is fine-tuned with a joint-angle network and a keypoint heatmap network, and at evaluation a confidence filter drops out-of-view keypoints before an EPnP solver recovers the pose. On the DREAM benchmark it reports the best area-under-curve of the ADD metric among methods that, like it, assume unknown joint angles and unknown bounding boxes, and it degrades least under synthetic occlusions (AUC 35.1 at 40% occlusion versus 28.2 for the nearest baseline). The practical stake is that collaborative robots and human-robot interaction settings, where joints report unreliable or unknown states and parts of the robot are often blocked from view, would get accurate pose estimates without markers or known joint angles.

What carries the argument

The load-bearing mechanism is a joint-masking embedding-predictive pre-training step, an adaptation of the Joint-Embedding Predictive Architecture (JEPA) to robot images. Four randomly chosen joint regions, each covering 15–20% of the image, are masked; a Vision Transformer encoder processes only the unmasked patches, and a predictor must output embeddings for all patches that match, in L1 distance, the embeddings of a target backbone running on the unmasked image. The target backbone is updated by exponential moving average of the encoder, avoiding trivial solutions. After pre-training, the encoder-predictor is fine-tuned with a Joint Net (iterative MLP refining joint angles over four steps) and a Keypoint Net (transposed-convolution upsampling to per-joint heatmaps), with random masks of up to 20% of the image applied during fine-tuning. At evaluation, a confidence threshold filters keypoints likely outside the field of view, and the remaining 2D keypoints, matched to 3D joint positions from forward kinematics and predicted angles, feed an EPnP solver. The joint-masked pre-training is what encodes the robot's physical structure; the random fine-tuning masks and keypoint filtering are what make the system degrade gracefully under occlusion and truncation.

What would settle it

Run the same self-supervised fine-tuning protocol that RoboPEPP receives (10 epochs on the test sequence, with per-sequence keypoint and joint-network learning rates) on the HPE baseline, and compare AUC-ADD on Panda RS and ORB; if HPE matches or exceeds RoboPEPP's 80.5 and 77.5, the claimed real-world advantage is an artifact of the adaptation protocol rather than the joint-masking pre-training. A simpler check is to re-evaluate RoboPEPP on those sequences without the per-sequence hyperparameter choices (the supplementary states a 1e-10 keypoint learning rate and a 100-pixel bounding-box expansion were selected after observing test AUC move from 70.4 to 80.5 on Panda RS).

Watch

Extended reading notes

Core claim

RoboPEPP's central discovery is that joint-targeted embedding predictive pre-training — masking the joints of a robot in an image and training a predictor to infer their latent embeddings from the unmasked context — gives a Vision Transformer encoder an understanding of the robot's physical model that transfers to pose and joint estimation. The paper argues that this pre-training is what makes the method simultaneously accurate and resilient under occlusion: a model pre-trained with random masking, or not pre-trained at all, degrades far more quickly when part of the robot is hidden (AUC 30 and 22.6 at 40% occlusion, versus 35.1 for joint-masked pre-training). Fine-tuning with random input masks and filtering low-confidence keypoints at inference add further tolerance to truncation, and the whole pipeline runs in 23 ms per image on an RTX A4000. On the DREAM benchmark, RoboPEPP reports AUC-ADD of 84.1 on Panda Photo and 80.5 on the hardest real-world sequence (Panda RS) without known joint angles or ground-truth bounding boxes, outperforming the render-and-compare baseline RoboPose and the feed-forward baseline HPE even when HPE is given the ground-truth bounding box.

Load-bearing premise

The real-world performance claims assume that fine-tuning RoboPEPP for 10 epochs on the actual test sequences (Panda AK, XK, RS, ORB), while the HPE and RoboPose baselines receive no equivalent test-time adaptation, is a fair comparison; if that test-distribution access is counted as leakage, the reported real-world advantages shrink or disappear, though the synthetic Panda Photo and Kuka Photo results stand independently.

Editorial extensions

If this is right

  • If joint-masked pre-training is what confers resilience to occlusion, then the same encoder should transfer to other robot-embodied tasks that require knowing where the robot is, such as dynamic prediction and imitation learning.
  • Because RoboPEPP needs no fiducial markers, known joint angles, or ground-truth bounding boxes at inference, it can be deployed in collaborative cells where the robot's state is unreliable or unavailable.
  • The 23 ms execution time with top accuracy means the method can run at interactive rates on a single GPU, suitable for closed-loop human-robot interaction.
  • The confidence-based keypoint filtering makes the pipeline degrade gracefully under truncation: when part of the robot leaves the frame, the system drops those keypoints instead of trusting false peaks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the joint-masking idea is not robot-specific: the same context-informed masking could be applied to the articulation points of animals, hands, or deformable objects, since the pre-training objective is to infer hidden parts from visible context.
  • A testable extension is to replace the synthetic black-rectangle occlusions with real-world clutter (tools, other robot parts) to see whether the learned physical-model understanding survives realistic textures and shadows.
  • The reported sensitivity of HPE to bounding-box width (up to 50% accuracy drop with 10-pixel wider boxes) hints that RoboPEPP's edge may come partly from its RoI-expansion training curriculum rather than from pre-training; ablating the curriculum against the pre-training would separate the two contributors.
  • If the encoder indeed encodes forward-kinematic structure, then fine-tuning on a new robot with a different kinematic chain should require far fewer labeled images than training from scratch; this is a direct, testable prediction of the paper's central claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. RoboPEPP proposes a two-stage robot pose and joint angle estimation method. In the first stage, a ViT encoder-predictor is pre-trained with an embedding-predictive objective (Eq. 1) in which randomly selected joint regions are masked and the predictor must reconstruct the target embeddings of masked patches from context. In the second stage, the encoder-predictor is fine-tuned together with a Joint Net and a Keypoint Net, using random masking as augmentation, a confidence-based keypoint filter at inference, and EPnP for pose. A self-supervised sim-to-real fine-tuning step (Eq. 4) using a differentiable PnP solver is applied on real-world sequences. Experiments on the DREAM dataset report AUC-ADD, joint-angle MAE, occlusion robustness, PCK, and runtime. The paper claims the best results among methods that do not assume known joint angles or bounding boxes, lower sensitivity to occlusions, and the shortest execution time.

Significance. The core idea is attractive: robot-specific masking of joints during embedding-predictive pre-training may inject a physical-model prior into the encoder, and the ablations (Fig. 8a) show that this pre-training improves occlusion robustness (AUC 22.6 without pre-training versus 35.1 with joint masking at 40% occlusion). The synthetic results (Panda Photo, Kuka Photo) are obtained under a fair protocol with respect to bounding boxes, and the code is released. If the claims are confirmed, the work would be a useful step for markerless robot pose estimation in occluded scenes. However, the real-world comparisons in Table 2 rest on a protocol in which RoboPEPP is fine-tuned on the test sequences themselves while baselines are not; that asymmetry, combined with test-set-tuned hyperparameters reported in the supplementary, limits the evidence for the unconditional 'best results' claim. The synthetic and occlusion evidence does not suffer this confound.

major comments (2)
  1. [Sec. 4.2.1 / Table 2 / Sec. 3.3 / Supp. A2] The real-world rows of Table 2 (Panda AK, XK, RS, ORB) compare RoboPEPP after 10 epochs of per-sequence self-supervised fine-tuning on those test sequences with HPE and RoboPose evaluated without any analogous adaptation. Supplementary A2 further shows that the keypoint learning rate (1e-10), the joint learning rate (1e-5), and the 100-pixel bounding-box expansion were selected by observing the ADD AUC on Panda RS move from 70.4 to 80.5. Thus the real-world advantages reported in Table 2 conflate the proposed architecture with transductive test-distribution access and test-set-tuned hyperparameters. This is load-bearing because the abstract's 'best results' claim rests on these rows. The authors should either give the competing methods the same self-supervised fine-tuning protocol, report the real-world numbers without fine-tuning (as in Fig. 8c), or clearly state that the comparison is transductive and temper the abstract accordingly.
  2. [Abstract and Sec. 5] The abstract states without qualification that RoboPEPP 'achieves the best results in robot pose and joint angle estimation while being the least sensitive to occlusions and requiring the lowest execution time.' Table 2 shows that on Kuka DR, RoboPose attains AUC 80.2 versus RoboPEPP's 76.2, and on Baxter DR, HPE with a known bounding box attains 58.8 versus RoboPEPP's 34.4. The claim can be defended only within the specific comparison class of methods that use unknown joint angles and unknown bounding boxes, and even there the real-world evidence is subject to the protocol asymmetry described above. The abstract and conclusion should be revised to state the precise comparison class and to separate the synthetic/occlusion results from the real-world results.
minor comments (4)
  1. [Sec. 3.2.2] In the sentence describing the reshaping of V, 'contianing' should be 'containing'.
  2. [References] References [34] and [35] are the same paper (Tian et al., 'Robot structure prior guided temporal attention for camera-to-robot pose estimation from image sequence'); one of the duplicated entries should be removed.
  3. [Sec. 4.2.3 / Fig. 6] The text reports only the AUC values at 40% occlusion (35.1, 28.2, 14.5) and leaves the relative percentage drops to the figure, where they are difficult to read; please report the relative drops for each method in the text.
  4. [Supp. A3] The sentence in the supplementary stating that 'some fine-tuning of this parameter may be necessary' refers to the 100-pixel bounding-box expansion, which was tuned on the test set. This should be disclosed in the main text as a limitation rather than appearing only in the supplementary.

Circularity Check

1 steps flagged · score 3.0 of 10

Real-world Table 2 results are partly fitted on test-set AUC; the core pre-training derivation is not circular.

  1. fitted input called prediction [Supplementary A2-A3; Table 2 real-world rows (Panda AK/XK/RS/ORB)]
    "Models are fine-tuned separately for each real-world dataset for 10 epochs with a batch size of 64. ... On the Panda RS dataset, for example, using a learning rate of 10−10 for the Keypoint Network and 10−5 for the Joint Network improved ADD AUC from 70.4 to 80.5. ... Increasing all the bounding box sizes by 100 pixels on all sides generally yields robust robot pose estimation results. However, some fine-tuning of this parameter may be necessary for optimal performance depending on the specific dataset."

    The Table 2 real-world rows are the objective used to select hyperparameters: the Keypoint/Joint learning rates were chosen because they moved the Panda RS test AUC from 70.4 to 80.5, and the 100-pixel RoI expansion is described as needing dataset-specific tuning. In addition, each real-world test sequence is itself used for 10 epochs of self-supervised fine-tuning before the reported evaluation. Thus the reported real-world gains over HPE/RoboPose are not out-of-sample predictions of the pretraining method; they are partly fitted values (transductive adaptation plus test-set-tuned hyperparameters). This is a statistical/protocol circularity, not an equation-level reduction, and it does not affect the synthetic Panda Photo/Kuka Photo or occlusion experiments.

full rationale

The core derivation chain is not circular: Eq. (1) defines an embedding-predictive pretraining loss on masked joint patches, the downstream losses (Eqs. 2-3) and PnP-based pose estimation (Sec. 3.3) are evaluated on held-out synthetic Photo/DR sets and a synthetic occlusion suite, none of which are used to set the pretraining or fine-tuning hyperparameters. The self-citations to JEPA/V-JEPA [3,7] are standard architectural references, not uniqueness claims, and they do not substitute for the experiments. The one concrete circularity concern is in the real-world Table 2 rows: the supplementary reports that the Keypoint/Joint learning rates were chosen by watching Panda RS test AUC move from 70.4 to 80.5, and that each real-world test sequence receives 10 epochs of self-supervised fine-tuning before evaluation, while HPE and RoboPose are not given the same adaptation. That makes the real-world 'best results' partly fitted evaluation numbers rather than independent predictions. It is a protocol/statistical circularity localized to the real-world columns; the synthetic Panda Photo, Kuka Photo, and occlusion results are unaffected and still provide independent support for the pretraining mechanism. I therefore score the circularity as mild (3/10).

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method introduces no invented physical entities and relies on standard machinery (EPnP, forward kinematics, ViT). Its novelty lives in the pre-training objective and training schedules, which contribute the listed free parameters. The main unevaluated premises are sim-to-real transfer, the synthetic occlusion proxy, and detector reliability; the joint-masking mechanism itself is an ad hoc design validated only by the paper's own ablations.

free parameters (6)
  • Sim-to-real Keypoint Net learning rate = 1e-10 (Joint Net 1e-5)
    Supplementary A2 explicitly chose these rates after observing ADD AUC rise from 70.4 to 80.5 on the Panda RS test set, so the reported real-world gain is partly a tuned result.
  • Inference bounding-box expansion = 100 pixels on all sides for real datasets
    Supplementary A3: expansion size is hand-set and the text admits per-dataset fine-tuning may be needed, yet it directly affects the cropped inputs behind the Table 2 real-world numbers.
  • Loss balance schedule alpha(t) = 0.0001 to 0.01, 0.1, 1.0 at epochs 5, 10, 40
    Sec. 3.2.3: hand-chosen curriculum balancing the joint and keypoint losses; not ablated.
  • Pre-training mask configuration = 4 masked joints, 15-20% image area, aspect ratio 0.75-1.5
    Sec. 3.1: chosen by hand; only the random-versus-joint masking choice is ablated, not the mask count or size.
  • Keypoint filtering threshold epsilon = default threshold, decremented by 0.025 until at least 4 keypoints remain
    Sec. 3.3: iterative scheme with a hand-set decrement; the filtering benefit is ablated but the threshold value itself is not.
  • RoI expansion curriculum lambda = 0 to 120 pixels across epochs 0 to 110
    Sec. 3.3: hand-scheduled bounding-box augmentation during training; together with the 100-pixel inference expansion it defines the effective input distribution seen at test time.
assumptions (6)
  • standard math EPnP returns the correct camera-to-robot transform given at least four accurate 2D-3D correspondences.
    Sec. 3.3 relies on EPnP as the pose solver; treated as a black box with no error analysis.
  • domain assumption GroundingDINO detects the robot sufficiently well for the RoI crop, and the 100-pixel expansion fixes any residual error.
    Sec. 3.3 and Supplementary A3: detector accuracy is never quantified on the test sequences; only qualitative statements are given.
  • domain assumption The domain-randomized synthetic training data transfers to photo-realistic and real cameras well enough for pose-level accuracy.
    Sec. 4.1: all models are trained on DR sequences and tested on Photo and real sequences; sim-to-real fidelity is a load-bearing premise of the benchmark protocol.
  • domain assumption Random black rectangular and circular overlays are a representative proxy for real occlusions encountered by collaborative robots.
    Sec. 4.2.3: the entire occlusion-robustness claim rests on this synthetic protocol.
  • ad hoc to paper Masking joint regions and predicting their embeddings instills a useful physical-model prior in the encoder.
    Sec. 3.1 states the mechanism; Fig. 8a shows joint masking helps, but mostly on real datasets, and the paper admits random masking is competitive on synthetic sets.
  • domain assumption Forward kinematics and camera intrinsics are exactly known for every test image.
    Sec. 3 problem description; standard for the DREAM benchmark and assumed for all PnP computations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training." pith.science (2026). https://pith.science/paper/A3HLPEAC

@misc{pith2026241117662,
  author       = {Pith},
  title        = {Pith review of: RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A3HLPEAC}},
  note         = {Machine review of arXiv:2411.17662}
}
read the original abstract

Vision-based pose estimation of articulated robots with unknown joint angles has applications in collaborative robotics and human-robot interaction tasks. Current frameworks use neural network encoders to extract image features and downstream layers to predict joint angles and robot pose. While images of robots inherently contain rich information about the robot's physical structures, existing methods often fail to leverage it fully; therefore, limiting performance under occlusions and truncations. To address this, we introduce RoboPEPP, a method that fuses information about the robot's physical model into the encoder using a masking-based self-supervised embedding-predictive architecture. Specifically, we mask the robot's joints and pre-train an encoder-predictor model to infer the joints' embeddings from surrounding unmasked regions, enhancing the encoder's understanding of the robot's physical model. The pre-trained encoder-predictor pair, along with joint angle and keypoint prediction networks, is then fine-tuned for pose and joint angle estimation. Random masking of input during fine-tuning and keypoint filtering during evaluation further improves robustness. Our method, evaluated on several datasets, achieves the best results in robot pose and joint angle estimation while being the least sensitive to occlusions and requiring the lowest execution time.

Figures

Figures reproduced from arXiv: 2411.17662 by the authors.

Figure 1
Figure 1. Comparison of an existing robot pose estimation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the RoboPEPP framework for robot pose and joint angle estimation. (a) Joint regions are masked to pre-train an [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Joint Net: A global average pooling layer aggregates the patch embeddings, v1, . . . , vM, into vg, which is then iteratively refined using an MLP to estimate the joint angles. deduce missing information about physical structures. 3.2. Keypoint Detection and Joint Angle Estimation The pre-trained encoder and predictor are then fine-tuned, where they extract embeddings v_i \in \mathbb {R}^d for i \in \{ 1, 2, \dots ,… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The examples show predicted heatmaps for Joint 5 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative Comparison on Panda Photo (Example 1) and Occlusion (Example 2 and 3) datasets: Predicted poses and joint angles are used to generate a mesh overlaid on the original image, where closer alignment indicates greater accuracy. Highlighted rectangles indicate r…
Figure 6
Figure 6. Figure 6: AUC comparison of the distance metric under varying [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Execution time and computation analysis on the Panda [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Ablation studies on (a) Pre-Training, (b) Keypoint Filtering, and (c) Sim-to-Real fine-tuning on the Panda test datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 32 canonical work pages

  1. [1]

    Asano, Christian Rupprecht, and Andrea Vedaldi

    Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In Proceedings of the International Conference on Learning Representations, 2020. 2

    work page 2020

  2. [2]

    Masked siamese networks for label-efficient learning

    Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bo- janowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In Proceedings of the European Conference on Computer Vision , pages 456–473. Springer, 2022

    work page 2022

  3. [3]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619–15629, Vancouver, Canada, 2023. 2, 3, 11

    work page 2023

  4. [4]

    Layer normalization

    Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 11

    arXiv 2016

  5. [5]

    Real-time holistic robot pose es- timation with unknown states

    Shikun Ban, Juling Fan, Xiaoxuan Ma, Wentao Zhu, Yu Qiao, and Yizhou Wang. Real-time holistic robot pose es- timation with unknown states. In European Conference on Computer Vision, pages 1–17, Malmo, Sweden, 2024. 1, 2, 4, 5, 6, 7, 8, 12, 13

    work page 2024

  6. [6]

    Vi- creg: Variance-invariance-covariance regularization for self- supervised learning

    Adrien Bardes, Jean Ponce, and Yann LeCun. Vi- creg: Variance-invariance-covariance regularization for self- supervised learning. In Proceedings of the International Conference on Learning Representations, 2022. 2

    work page 2022

  7. [7]

    V-JEPA: Latent video prediction for visual repre- sentation learning, 2024

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-JEPA: Latent video prediction for visual repre- sentation learning, 2024. 2, 3

    work page 2024

  8. [8]

    Unsupervised learning of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Ad- vances in neural information processing systems , 33:9912– 9924, 2020. 2

    2020

Show all 39 references
  1. [9]

    End-to-end learnable geometric vision by backprop- agating PnP optimization

    Bo Chen, Alvaro Parra, Jiewei Cao, Nan Li, and Tat-Jun Chin. End-to-end learnable geometric vision by backprop- agating PnP optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8100–8109, 2020. 2, 5

    2020

  2. [10]

    Learning human-to-robot handovers from point clouds

    Sammy Christen, Wei Yang, Claudia P ´erez-D’Arpino, Ot- mar Hilliges, Dieter Fox, and Yu-Wei Chao. Learning human-to-robot handovers from point clouds. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9654–9664, 2023. 1

    2023

  3. [11]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In Proceedings o...

  4. [12]

    ARTag, a fiducial marker system using digi- tal techniques

    Mark Fiala. ARTag, a fiducial marker system using digi- tal techniques. In Proceedings of the IEEE Computer Soci- ety Conference on Computer Vision and Pattern Recognition, pages 590–596, San Diego, CA, 2005. 1, 2

    2005

  5. [13]

    Automatic generation and detection of highly reliable fiducial markers under occlusion

    Sergio Garrido-Jurado, Rafael Mu ˜noz-Salinas, Fran- cisco Jos ´e Madrid-Cuevas, and Manuel Jes ´us Mar ´ın- Jim´enez. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recogni- tion, 47(6):2280–2292, 2014. 1, 2

    2014

  6. [14]

    PoseFusion: Multi-scale keypoint correspondence for monocular camera-to-robot pose estimation in robotic ma- nipulation

    Xujun Han, Shaochen Wang, Xiucai Huang, and Zhen Kan. PoseFusion: Multi-scale keypoint correspondence for monocular camera-to-robot pose estimation in robotic ma- nipulation. In Proceedings of the IEEE International Con- ference on Robotics and Automation, pages 795–801, Yoko-...

    2024

  7. [15]

    Robust robot-camera calibra- tion

    Jarmo Ilonen and Ville Kyrki. Robust robot-camera calibra- tion. In Proceedings of the International Conference on Ad- vanced Robotics, pages 67–74. IEEE, 2011. 2

    2011

  8. [16]

    Single-view robot pose and joint angle estimation via render & compare

    Yann Labb ´e, Justin Carpentier, Mathieu Aubry, and Josef Sivic. Single-view robot pose and joint angle estimation via render & compare. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1654–1663, 2021. 1, 2, 5, 6, 7, 12, 13

    2021

  9. [17]

    Fractalnet: Ultra-deep neural networks without residuals

    Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In Proceedings of the International Conference on Learning Representations, 2022. 11

    2022

  10. [18]

    Cornernet: Detecting objects as paired keypoints

    Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European confer- ence on computer vision (ECCV), pages 734–750, 2018. 4

    2018

  11. [19]

    Camera-to-robot pose estimation from a single im- age

    Timothy E Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, and Stan Birch- field. Camera-to-robot pose estimation from a single im- age. In Proceedings of the IEEE International Conference on Robotics and Automation , pages 9426–9432, Pari...

  12. [20]

    EPnP: An accurate O(n) solution to the p n p problem.Inter- national journal of computer vision, 81:155–166, 2009

    Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. EPnP: An accurate O(n) solution to the p n p problem.Inter- national journal of computer vision, 81:155–166, 2009. 2, 4, 5

    2009

  13. [21]

    Self-supervised monocular multi-robot relative localization with efficient deep neural networks

    Shushuai Li, Christophe De Wagter, and Guido CHE De Croon. Self-supervised monocular multi-robot relative localization with efficient deep neural networks. In Proceed- ings of the International Conference on Robotics and Au- tomation, pages 9689–9695. IEEE, 2022. 1

    2022

  14. [22]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In Pro- ceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, Venice, Italy, 2017. 4

    2017

  15. [23]

    Grounding Dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding Dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 5, 11

    2023 arXiv

  16. [24]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5, 11

    2017 arXiv

  17. [25]

    Jingpei Lu, Florian Richter, and Michael C. Yip. Markerless camera-to-robot pose estimation via self-supervised sim-to- real transfer. In Proceedings of the IEEE/CVF Conference 9 on Computer Vision and Pattern Recognition, pages 21296– 21306, Vancouver, Canada, 2023. 1, 2, 5

    2023

  18. [26]

    AprilTag: A robust and flexible visual fiducial system

    Edwin Olson. AprilTag: A robust and flexible visual fiducial system. In Proceedings of the IEEE International Confer- ence on Robotics and Automation, pages 3400–3407, Shang- hai, China, 2011. 1, 2

    2011

  19. [27]

    Range-aided ego-centric collaborative pose estimation for multiple robots

    Andreas Papadimitriou, Sina Sharif Mansouri, and George Nikolakopoulos. Range-aided ego-centric collaborative pose estimation for multiple robots. Expert Systems with Applica- tions, 202:117052, 2022. 1

    2022

  20. [28]

    Robot sensor calibration: solving AX= XB on the euclidean group.IEEE Transactions on Robotics and Automation, 10(5):717–721, 1994

    Frank C Park and Bryan J Martin. Robot sensor calibration: solving AX= XB on the euclidean group.IEEE Transactions on Robotics and Automation, 10(5):717–721, 1994. 2

    1994

  21. [29]

    Accelerating 3d deep learning with pytorch3d

    Nikhila Ravi, Jeremy Reizenstein, David Novotny, Tay- lor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501, 2020. 2

    2007 arXiv

  22. [30]

    Coop- erative heterogeneous multi-robot systems: A survey

    Yara Rizk, Mariette Awad, and Edward W Tunstel. Coop- erative heterogeneous multi-robot systems: A survey. ACM Computing Surveys (CSUR), 52(2):1–31, 2019. 1

    2019

  23. [31]

    Semi-perspective decoupled heatmaps for 3D robot pose estimation from depth maps

    Alessandro Simoni, Stefano Pini, Guido Borghi, and Roberto Vezzani. Semi-perspective decoupled heatmaps for 3D robot pose estimation from depth maps. IEEE Robotics and Au- tomation Letters, 7(4):11569–11576, 2022. 2

    2022

  24. [32]

    Robot pose nowcasting: Fore- cast the future to improve the present

    Alessandro Simoni, Francesco Marchetti, Guido Borghi, Federico Becattini, Lorenzo Seidenari, Roberto Vezzani, and Alberto Del Bimbo. Robot pose nowcasting: Fore- cast the future to improve the present. arXiv preprint arXiv:2308.12914, 2023. 2

    2023 arXiv

  25. [33]

    Pose estimation and adaptive robot be- haviour for human-robot interaction

    Mikael Svenstrup, Soren Tranberg, Hans Jorgen Andersen, and Thomas Bak. Pose estimation and adaptive robot be- haviour for human-robot interaction. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 3571–3576. IEEE, 2009. 1

    2009

  26. [34]

    Robot structure prior guided temporal attention for camera-to-robot pose estimation from image sequence

    Yang Tian, Jiyao Zhang, Zekai Yin, and Hao Dong. Robot structure prior guided temporal attention for camera-to-robot pose estimation from image sequence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8917–8926, Vancouver, Canada, 2023. 2

    2023

  27. [35]

    Robot structure prior guided temporal attention for camera-to-robot pose estimation from image sequence

    Yang Tian, Jiyao Zhang, Zekai Yin, and Hao Dong. Robot structure prior guided temporal attention for camera-to-robot pose estimation from image sequence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8917–8926, Vancouver, Canada, 2023. 1

    2023

  28. [36]

    Robokeygen: robot pose and joint angles estimation via diffusion-based 3d key- point generation

    Yang Tian, Jiyao Zhang, Guowei Huang, Bin Wang, Ping Wang, Jiangmiao Pang, and Hao Dong. Robokeygen: robot pose and joint angles estimation via diffusion-based 3d key- point generation. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 5375–...

    2024

  29. [37]

    Multi-view human pose estimation in human-robot interac- tion

    Chengjun Xu, Xinyi Yu, Zhengan Wang, and Linlin Ou. Multi-view human pose estimation in human-robot interac- tion. In Proceedings of the Annual Conference of the IEEE Industrial Electronics Society , pages 4769–4775. IEEE,

  30. [38]

    Calibrating a robot cam- era

    Dekun Yang and John Illingworth. Calibrating a robot cam- era. In Proceedings of the British Machine Vision Confer- ence, pages 1–10, York, UK, 1994. 2

    1994

  31. [39]

    robotic arm

    Xiaopin Zhong, Wenxuan Zhu, Weixiang Liu, Jianye Yi, Chengxiang Liu, and Zongze Wu. G-SAM: A robust one- shot keypoint detection framework for PnP based robot pose estimation. Journal of Intelligent & Robotic Systems, 109(2): 28, 2023. 2 10 Supplementary Material A1. Encoder a...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.