Pith. sign in

REVIEW 4 major objections 7 minor 4 cited by

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces a model trained on two million rendered views of 3D objects that estimates an object's front-facing 3D orientation from a single real image, and reports large accuracy gains over both expert methods and…

desk verdict A genuinely new single-image orientation estimator with a clever data pipeline, but the headline real-image SOTA claim is unverifiable until the paper specifies how each benchmark's native pose labels were converted to its spherical convention. read the letter →

arxiv 2412.18605 v1 pith:YKK344BO submitted 2024-12-24 cs.CV

Zehan Wang , Ziang Zhang , Tianyu Pang , Chao Du , Hengshuang Zhao , Zhou Zhao This is my paper · ORCID
classification cs.CV
keywords objectorientationestimationsingle-image3Dposefront-faceannotationprobabilitydistributionfittingrenderedimagetrainingsynthetic-to-realtransferzero-shotevaluationspatialreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Object orientation—which way an object's meaningful front face points in 3D—has been a missing perceptual primitive in vision models, largely because orientation labels are scarce. This paper tries to establish that the missing primitive can be learned at scale by generating the labels rather than collecting them: it renders two million images from random viewpoints of 3D models whose front faces were chosen by a large vision-language model aided by symmetry checks, and trains a single-image estimator on those renders. The estimator predicts three angles (azimuth, polar, and in-plane rotation) as probability distributions over one-degree bins, a target the paper shows is far more trainable than direct regression. The authors report large accuracy gains over existing expert methods and vision-language baselines on both rendered and real images, and they show that handing the estimated angles to an LLM as text lifts spatial question-answering accuracy. If the central claim holds, orientation becomes an available input for object-centric reasoning, generation, and 3D asset alignment.

What carries the argument

The load-bearing mechanism is orientation-as-distribution fitting. Ground-truth azimuth, polar, and rotation angles are converted into target probability masses over 1-degree bins: a Gaussian for the polar angle and a circular (wrapped) Gaussian for azimuth and rotation, with variances fixed at 2 degrees, 20 degrees, and 1 degree respectively. The network is trained with cross-entropy to reproduce those masses, and at inference the highest-probability bin is the prediction. This reformulation turns an unstable continuous regression into a smooth, well-behaved classification-adjacent objective that captures the near-equivalence of adjacent angles such as 29, 30, and 31 degrees. The second load-bearing component is the data pipeline: canonical-pose filtering, vision-language-model front-face annotation from orthogonal views with symmetry checks, and free-view rendering, which together remove the manual-labeling bottleneck and make training set scale a matter of rendering time.

What would settle it

Annotate a random subset of the evaluation objects with several independent human labelers, asking them to pick the front face from the same orthogonal views, and measure inter-labeler agreement; if agreement on the annotator model's choices drops well below the 86 to 92 percent range the paper reports for its own annotation ablation, or if the model's accuracy falls sharply when evaluated against human-chosen fronts, the claim of robust, convention-independent orientation estimation is not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that object orientation can be estimated from a single free-view image by a model trained entirely on synthetic renders. Its model, Orient Anything, takes an image and outputs a confidence that the object has a meaningful front face, plus three angles specifying that front face relative to the camera: azimuth, polar, and in-plane rotation. The training labels are generated by a pipeline that filters 3D assets to canonical upright poses, asks a vision-language model to pick the front face from four orthogonal views with symmetry information, and then renders 40 random views per object to create two million annotated images. On top of this data, the paper argues, the decisive design is to predict each angle as a discretized probability distribution rather than a single number, which stabilizes training and yields transfers to real photos. Evaluated on five real-world benchmarks it has never seen, the model reports substantially lower angle errors than an expert 3D detection baseline, and when its outputs are converted to text for an LLM, spatial question accuracy on the paper's Ori-Bench rises from about 33 percent for the best baseline to 51.5 percent.

Load-bearing premise

The load-bearing premise is that the front-face labels used for training and evaluation are trustworthy: every label comes from one large vision-language model's choice among four orthogonal views, filtered by symmetry, so any systematic bias in that model's notion of 'front' is inherited by both the training signal and the accuracy numbers.

Editorial extensions

If this is right

  • Single-image orientation estimation becomes practical for arbitrary object categories without CAD models, reference views, or depth information as prerequisites.
  • Adding Orient Anything's angle estimates as text to an LLM lifts performance on orientation-based spatial questions, including part reasoning (left versus right paw or hand) and perspective-taking relations, far above what current vision-language models achieve directly.
  • The same estimates can score whether generated images obey a requested orientation or perspective, giving text-to-image systems a spatial reward signal they currently lack.
  • Because multi-view predictions are consistent, the model can vote across views to recover and re-align the orientation of 3D assets, which can then be used to expand the rendering pipeline.
  • The confidence head handles objects with no meaningful front face, so the method applies uniformly to symmetric or front-less objects instead of crashing or returning arbitrary angles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The definition of 'front' is a labeling convention, not a geometric invariant. A useful stress test would compare the model's accuracy when the same benchmark objects are re-annotated by several independent human labelers, or by a different vision-language model, to see how much the reported numbers depend on the annotator's notion of front.
  • Because the target distributions are fixed-width Gaussians, the loss assumes all angular errors of the same size are equally bad; predicting per-instance variance or learning the distribution shape could improve calibration on ambiguous, partly occluded, or symmetric objects.
  • The architecture is category-agnostic, so coupling it with an open-world detector and tracker should extend it to video, where temporal smoothing across frames could further reduce angle errors; the paper evaluates only independent still images.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces Orient Anything, a single-image object orientation estimation model trained on 2M rendered images from Objaverse. The training labels are produced by a pipeline that first filters 3D assets in canonical poses, then uses Gemini-1.5-Pro to select the object's front face from four orthographic views (with symmetry analysis as a filter), and finally renders 40 random viewpoints per object with spherical angle annotations (polar theta, azimuth phi, camera rotation delta). The model uses a DINOv2 visual encoder and multiple prediction heads that fit discretized Gaussian distributions over the three angles, plus a confidence head for whether a meaningful front face exists. The authors evaluate on a manually annotated rendered test set, a COCO-based 8-direction real-image benchmark, five real-world datasets (SUN RGB-D, KITTI, nuScenes, Objectron, ARKitScenes), and a new VQA benchmark (Ori-Bench), reporting state-of-the-art results against Cube R-CNN, GPT-4o, and Gemini-1.5-Pro. They also demonstrate applications in spatial VQA, generation scoring, and 3D model orientation voting.

Significance. If the results are validated, this is a potentially significant contribution: it is the first large-scale, category-agnostic single-image orientation estimator that works on free-view images, and the probability-distribution-fitting objective is a simple and effective alternative to direct regression. The paper's strengths include a clear data-generation pipeline, extensive ablations (learning objective, number of views, initialization, augmentation), a new benchmark (Ori-Bench), and reproducible methodology in principle. However, the central 'state-of-the-art in real images' claim depends on an unspecified annotation-conversion protocol for the five real datasets, and the definition of 'meaningful front face' is inherited from a VLM rather than from an independent human-consensus protocol. These issues are load-bearing for the paper's main claims and need to be addressed before the results can be fully trusted.

major comments (4)
  1. [Sec. 6.3, Table 3] The zero-shot real-image evaluation on SUN RGB-D, KITTI, nuScenes, Objectron, and ARKitScenes requires converting each dataset's native orientation annotations (3D bounding boxes, yaw angles, 9-DoF poses) into the paper's spherical convention (polar theta, azimuth phi, rotation delta relative to a front axis). The paper does not specify this conversion for any dataset, nor how Cube R-CNN's raw outputs are mapped to the same convention. Without this protocol, the numeric errors in Table 3 are not verifiable, and the comparison against Cube R-CNN is not well-defined. The appendix template (Appendix E) only describes converting predicted azimuth to 8-direction text, not the reverse mapping of ground truth into spherical angles. The authors should provide a detailed, per-dataset conversion description, including the definition of the object front axis for each object category and how camera-up alignment is handled.
  2. [Sec. 4 (Step 2), Table 4, Sec. 6.2] The training labels are produced by Gemini-1.5-Pro's judgment of the front face from four orthographic views, and the paper's own ablation (Table 4) reports only 86–92% agreement between this VLM and manual annotations on 300 objects. Since Gemini is also a baseline in the experiments, any systematic bias in Gemini's notion of 'front' (e.g., for keyboards, remote controls, shoes) is baked into the trained model and into the comparison. The manual annotation protocol for the test sets (Sec. 6.2 and Sec. 6.3) is not described: no annotation instructions, inter-annotator agreement, or criteria for 'meaningful front face' are given. This is load-bearing because the paper defines the task as estimating the 'meaningful front face' and claims the model is the first to align with it. The authors need to specify the manual annotation protocol and show that it is independent of the VLM's notion, or explicitly discuss the subjectivity of the front-face definition and its impact on results.
  3. [Sec. 6.2, Sec. 6.3, Tables 2 and 3] No error bars, confidence intervals, or statistical significance tests are reported for any of the main quantitative results. For example, the COCO 8-direction benchmark has 1,600 samples, and the difference between Orient Anything (72.44%) and GPT-4o (45.78%) is large, but the difference between ViT-B (70.19%) and ViT-L (72.44%) may not be meaningful without variance estimates. Similarly, the rendered test set has only 300 objects (4,800 images), and the manual annotation may introduce additional variance. The authors should report standard deviations over multiple runs or bootstrap confidence intervals for the key tables, and at least state the number of independent training runs.
  4. [Sec. 3, Table 1 (Ori-Bench)] Ori-Bench consists of only 200 base images (100 COCO and 100 DALL-E 3), with each mirrored pair counted only if both versions are answered correctly. This yields small effective sample sizes, especially for the Spatial Part Reasoning task (39+39 samples). The paper reports overall accuracy of 51.50% for Orient Anything+LLM versus 33.00% for Gemini, but does not report per-subset results (COCO vs. DALL-E 3) or any confidence interval. Additionally, the LLM used in the Orient Anything+LLM pipeline appears to be the same GPT-4o model used as a baseline, so the comparison partly measures the benefit of additional spatial-context prompting rather than a new reasoning capability. The authors should report per-task and per-source breakdowns, state the statistical significance of the differences, and clarify the role of the LLM in the pipeline.
minor comments (7)
  1. [Sec. 5.1, after Eq. (2)] The text says "sigma_phi is the variance for polar distribution" but the context refers to the azimuth distribution; this is a typo that should be corrected.
  2. [Sec. 5.2 heading] The heading "Sythetic-to-Real Transferring" contains a typo; it should be "Synthetic-to-Real Transferring".
  3. [Sec. 6.4, Table 5] In the Training Initialization block, "DIONv2" is a typo for "DINOv2".
  4. [Sec. 5.1, Eq. (2) and surrounding text] The rotation angle is denoted as both gamma (in the text before Eq. (2)) and delta (in Eq. (2) and elsewhere); please use a single symbol consistently.
  5. [Sec. 4, Step 2] The symmetry threshold is described as "similarity exceeds the threshold" without specifying the threshold value or how SIFT, SSIM, and color similarity are combined; please provide the exact criterion for reproducibility.
  6. [Sec. 6.3] For the five real-world datasets, the paper states that "1,000 objects with 3D orientation annotations are randomly selected and cropped" but does not specify whether ground-truth boxes or an external detector is used for cropping, and whether the same crops are used for all baselines; this should be clarified.
  7. [Appendix D] The appendix says all Ori-Bench samples are in the attached file, but the main text does not state whether this file will be released publicly; please clarify availability of the benchmark and annotations.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper's headline results rest on independent manual test annotations and external benchmarks; the Gemini-derived training labels create a data-bias concern, not a circular derivation.

full rationale

The derivation chain is: Objaverse models are filtered for canonical pose (Sec. 4 Step 1), front faces are annotated by Gemini-1.5-Pro with symmetry filtering (Step 2), 2M images are rendered with spherical theta/phi/delta labels, and the model is trained to fit discretized Gaussian target distributions (Eqs. 1-2) under cross-entropy loss (Eq. 4). Nothing in the training objective or the inference argmax (Sec. 5.1) is defined in terms of the model's own outputs or of the quantities it claims to predict: the target distributions are constructed from ground-truth angles with stated variance hyperparameters, and the rendered test set (Sec. 6.2) uses 300 manually annotated objects. The real-image results in Table 3 are compared against ground-truth annotations of SUN RGB-D, KITTI, nuScenes, Objectron, and ARKitScenes, not against Orient Anything's own predictions, so there is no fit-then-predict loop. The strongest concern is that Gemini-1.5-Pro both supplies the training front-face labels and appears as a baseline in Tables 2-3, so the model can inherit Gemini's notion of 'front'; however, the annotation ablation in Table 4 is measured against 300 manually annotated objects, and the COCO real-image benchmark is manually labeled, which keeps the headline evaluations independent of the labeling model. The unspecified conversion of native benchmark annotations into the paper's spherical convention in Table 3 is a reproducibility and correctness risk, not a circularity: the paper gives no equation showing the benchmark error is algebraically identical to the training target. No self-citation is load-bearing; initialization from DINOv2 is an independent, externally validated choice. The score of 1 reflects the training-label/baseline overlap as a data-bias concern rather than a circular derivation.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim depends on six manually chosen scalars and five domain assumptions, most notably that a VLM can reliably define 'front face' for a large 3D asset library. No new physical entities are introduced.

free parameters (6)
  • sigma_theta = 2.0 degrees
    Variance of Gaussian target for polar angle distribution; manually set in Sec. 6.1 and ablated in Fig. 5, shown to be insensitive.
  • sigma_phi = 20.0 degrees
    Variance of circular Gaussian target for azimuth distribution; manually set in Sec. 6.1.
  • sigma_delta = 1.0 degree
    Variance of circular Gaussian target for rotation distribution; manually set in Sec. 6.1.
  • lambda = 1.0
    Loss coefficient for orientation judgment term in Eq. 4; set without ablation.
  • confidence_threshold = 0.5
    Threshold for deciding an object has a meaningful front face at inference; chosen without reported tuning.
  • views_per_object = 40
    Number of rendered views per object; chosen by ablation in Tab. 5.
assumptions (5)
  • domain assumption Objaverse objects in canonical poses can be identified by PCA on edge maps of orthographic views with 2-degree tolerance.
    Step 1 of data pipeline; if filtering misclassifies tilted objects, the front-face classification task becomes ill-posed.
  • domain assumption Gemini-1.5-Pro, given four orthographic views plus symmetry hints, identifies the meaningful front face of an object with acceptable accuracy.
    Step 2 of pipeline; the training labels are generated by this VLM, and the paper reports 86-92% agreement with manual annotation on a 300-object subset.
  • domain assumption The front face of an object is a well-defined, single, consistent property for objects that are not centrally symmetric.
    Used for both annotation and evaluation; the paper acknowledges ambiguity for remote, keyboard, toaster, etc., and some categories score at or near chance.
  • domain assumption Rendering 3D models from random views with the camera aimed at the object center produces images whose orientation labels are exactly the spherical angles (theta, phi, delta) of the camera relative to the annotated front direction.
    Step 3 of pipeline; errors here directly corrupt the training signal.
  • domain assumption DINOv2 features trained on real images transfer to rendered images and back to real images sufficiently for orientation estimation.
    Sec. 5.2: the synthetic-to-real strategy rests on initialization and augmentation; no explicit theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models." pith.science (2026). https://pith.science/paper/YKK344BO

@misc{pith2026241218605,
  author       = {Pith},
  title        = {Pith review of: Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKK344BO}},
  note         = {Machine review of arXiv:2412.18605}
}
read the original abstract

Orientation is a key attribute of objects, crucial for understanding their spatial pose and arrangement in images. However, practical solutions for accurate orientation estimation from a single image remain underexplored. In this work, we introduce Orient Anything, the first expert and foundational model designed to estimate object orientation in a single- and free-view image. Due to the scarcity of labeled data, we propose extracting knowledge from the 3D world. By developing a pipeline to annotate the front face of 3D objects and render images from random views, we collect 2M images with precise orientation annotations. To fully leverage the dataset, we design a robust training objective that models the 3D orientation as probability distributions of three angles and predicts the object orientation by fitting these distributions. Besides, we employ several strategies to improve synthetic-to-real transfer. Our model achieves state-of-the-art orientation estimation accuracy in both rendered and real images and exhibits impressive zero-shot ability in various scenarios. More importantly, our model enhances many applications, such as comprehension and generation of complex spatial concepts and 3D object pose adjustment.

Figures

Figures reproduced from arXiv: 2412.18605 by the authors.

Figure 1
Figure 1. We introduce a novel method for estimating the object orientation in images, represented by the red axis, while the blue and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Understanding object orientation is essential for spatial [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The orientation data collection pipeline is composed of three steps: [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Orient Anything consists of a simple visual encoder and multiple prediction heads. It is trained to judge if the object in the input [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Ablation study for hyper-parameter [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Generated images with given textual prompt (left two from DALL-E 3 [ [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Qualitative results on SUN RGB-D. Ground Truth Cube RCNN Orient Anything [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Qualitative results on KITTI and nuScenes. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Qualitative results on Objectron. Ground Truth Cube RCNN Orient Anything [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 12
Figure 12. Figure 12: More visualization of images in the wild. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 11
Figure 11. Figure 11: Qualitative results on ARKitScenes [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 13
Figure 13. Figure 13: Visualization and qualitative comparison on the [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Visualization and qualitative comparison on the [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Visualization and qualitative comparison on the [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GenSpace: Benchmarking Spatially-Aware Image Generation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    GenSpace benchmarks spatial awareness in image generation with a 3D reconstruction-based evaluator, showing models struggle with allocentric relations and metric measurements.

  2. Disentangling 3D Modeling from Spatial Reasoning

    cs.LG 2026-08 conditional novelty 6.0 of 10

    DiSR splits spatial reasoning into 3D perception by frozen expert models and language-based reasoning by a LoRA-tuned LLM, reporting state-of-the-art results on 3DSRBench and SPAR-Bench with 0.33M training samples.

  3. MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    On MarineEVT, an event-centric 20K-pair marine video QA benchmark, EVT-R1 with tool-integrated RL scores 48.89 average accuracy, 5.22 points above the best untuned open-source VLM and 8.54 above the best tool-using co...

  4. UniPose9D: Universal Category-Agnostic Object Pose Estimation

    cs.CV 2026-07 conditional novelty 5.5 of 10

    A single category-agnostic model recovers metric 9D object pose from one masked RGB-D observation via point-pair NOCS prediction, flow matching, and adaptive N-hop Kabsch–Umeyama.

Reference graph

Works this paper leans on

51 extracted references · 21 canonical work pages · cited by 4 Pith papers

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision–ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 422–440. Springer, 2020. 2

    work page 2020

  2. [2]

    Objectron: A large scale dataset of object-centric videos in the wild with pose an- notations

    Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jian- ing Wei, and Matthias Grundmann. Objectron: A large scale dataset of object-centric videos in the wild with pose an- notations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7822–7831,

  3. [3]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19129– 19139, 2022. 2

    2022

  4. [4]

    Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897, 2021. 7

    arXiv 2021

  5. [5]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 3, 8

    work page 2023

  6. [6]

    Omni3d: A large benchmark and model for 3d object detection in the wild

    Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 13154–13164, 2023. 1, 3, 6

    work page 2023

  7. [7]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. 6

    2020

  8. [8]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202–221. Springer, 2020. 2

    2020

Show all 51 references
  1. [9]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2023

  2. [10]

    Pope: 6-dof promptable pose estimation of any object in any scene with one reference

    Zhiwen Fan, Panwang Pan, Peihao Wang, Yifan Jiang, De- jia Xu, and Zhangyang Wang. Pope: 6-dof promptable pose estimation of any object in any scene with one reference. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 7771–7781, 2024. 3 9

    2024

  3. [11]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pat- tern recognition, pages 3354–3361. IEEE, 2012. 6

    2012

  4. [12]

    Zero-shot category-level object pose estimation

    Walter Goodwin, Sagar Vaze, Ioannis Havoutis, and Ingmar Posner. Zero-shot category-level object pose estimation. In European Conference on Computer Vision, pages 516–532. Springer, 2022. 3

    2022

  5. [13]

    Seeing through their eyes: Evaluating visual per- spective taking in vision language models

    Gracjan G ´oral, Alicja Ziarko, Michal Nauman, and Maciej Wołczyk. Seeing through their eyes: Evaluating visual per- spective taking in vision language models. arXiv preprint arXiv:2409.12969, 2024. 2

    2024 arXiv

  6. [14]

    A survey of 6dof object pose estimation methods for different application scenarios

    Jian Guan, Yingming Hao, Qingxiao Wu, Sicong Li, and Yingjian Fang. A survey of 6dof object pose estimation methods for different application scenarios. Sensors, 24(4): 1076, 2024. 3

    2024

  7. [15]

    Align deep features for oriented object detection

    Jiaming Han, Jian Ding, Jie Li, and Gui-Song Xia. Align deep features for oriented object detection. IEEE transac- tions on geoscience and remote sensing, 60:1–11, 2021. 3

    2021

  8. [16]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 6

    2022

  9. [17]

    Orientdream: Streamlining text-to-3d generation with explicit orientation control

    Yuzhong Huang, Zhong Li, Zhang Chen, Zhiyuan Ren, Gu- osheng Lin, Fred Morstatter, and Yi Xu. Orientdream: Streamlining text-to-3d generation with explicit orientation control. arXiv preprint arXiv:2406.10000, 2024. 2

    2024 arXiv

  10. [18]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2, 6, 9

    2024 arXiv

  11. [19]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9492...

    2024

  12. [20]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 1, 3

    2023

  13. [21]

    Flux, 2024

    Black Forest Labs. Flux, 2024. 8

    2024

  14. [22]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2022

  15. [23]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1

    2024

  17. [25]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 1

    2023 arXiv

  18. [26]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  19. [27]

    Distinctive image features from scale- invariant keypoints

    David G Lowe. Distinctive image features from scale- invariant keypoints. International journal of computer vi- sion, 60:91–110, 2004. 4

    2004

  20. [28]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yi- tao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 2

    2022 arXiv

  21. [29]

    Nope: Novel object pose estimation from a sin- gle image

    Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Yinlin Hu, Renaud Marlet, Mathieu Salzmann, and Vincent Lepetit. Nope: Novel object pose estimation from a sin- gle image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17923–...

    2024

  22. [30]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 6

    2023 arXiv

  23. [31]

    Pvnet: Pixel-wise voting network for 6dof pose estimation

    Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hu- jun Bao. Pvnet: Pixel-wise voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4561–4570,

  24. [32]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  25. [33]

    Segment anything meets point tracking

    Frano Raji ˇc, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Mar- tin Danelljan, and Fisher Yu. Segment anything meets point tracking. arXiv preprint arXiv:2307.01197, 2023. 1

    2023 arXiv

  26. [34]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 1

    2024 arXiv

  27. [35]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,

  28. [36]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 3 10

    2021 arXiv

  29. [37]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  30. [38]

    Mvdream: Multi-view diffusion for 3d gen- eration

    Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d gen- eration. arXiv preprint arXiv:2308.16512, 2023. 2

    2023 arXiv

  31. [39]

    Sun rgb-d: A rgb-d scene understanding benchmark suite

    Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567–576, 2015. 6

    2015

  32. [40]

    Implicit 3d orientation learning for 6d object detection from rgb images

    Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proceedings of the european conference on computer vi- sion (ECCV), pages 699–715, 2018. 3

    2018

  33. [41]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  34. [42]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 2, 6

    2024 arXiv

  35. [43]

    Tracking everything everywhere all at once

    Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 19795–19806, 2023. 1

    2023

  36. [44]

    Embodiedscan: A holistic multi- modal 3d perception suite towards embodied ai

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi- modal 3d perception suite towards embodied ai. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  37. [45]

    Lego-net: Learning regular rearrangements of ob- jects in rooms

    Qiuhong Anna Wei, Sijie Ding, Jeong Joon Park, Rahul Sajnani, Adrien Poulenard, Srinath Sridhar, and Leonidas Guibas. Lego-net: Learning regular rearrangements of ob- jects in rooms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19...

    2023

  38. [46]

    Human-object interaction from human-level instructions

    Zhen Wu, Jiaman Li, and C Karen Liu. Human-object interaction from human-level instructions. arXiv preprint arXiv:2406.17840, 2024. 2

    2024 arXiv

  39. [47]

    Oriented r-cnn for object detection

    Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, and Junwei Han. Oriented r-cnn for object detection. InProceed- ings of the IEEE/CVF international conference on computer vision, pages 3520–3529, 2021. 3

    2021

  40. [48]

    Depth any- thing v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv preprint arXiv:2406.09414, 2024. 6

    2024 arXiv

  41. [49]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 17...

    2024

  42. [50]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633–641,

  43. [2017]

    From the perspective of viewer<OBJ1> is on the left and <OBJ2> is on the right of the view

    3 11 Category Cube RCNN Gemini GPT-4o Orient Anything (ViT-L) bed 75% 15% 40% 100%(+25%) monitor 35% 50% 50% 100%(+50%) oven 50% 10% 65% 100%(+35%) teddy bear 20% 40% 45% 100%(+55%) motorbike 5% 20% 40% 95%(+55%) parking meter 40% 55% 65% 95%(+30%) laptop 65% 45% 50% 95%(+30%)...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.