Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ZeroKey detects named 3D keypoints on arbitrary object categories with no 3D keypoint annotations, by asking a point-aware multimodal language model to localize parts in rendered views and then aggregating the back-projected points.

desk verdict First credible zero-shot 3D keypoint detector built on Molmo's point-level MLLM abilities, but the evaluation on three categories is too thin to support all the claims; still, the core idea is reproducible and worth reviewing. read the letter →

arxiv 2412.06292 v1 pith:XBMYKIXX submitted 2024-12-09 cs.CV

Bingchen Gong , Diego Gomez , Abdullah Hamdi , Abdelrahman Eldesokey , Ahmed Abdelreheem , Peter Wonka , Maks Ovsjanikov This is my paper · ORCID
classification cs.CV
keywords zero-shot3Dkeypointdetectionmultimodallargelanguagemodelspoint-levelreasoningmulti-viewaggregationback-projectionHDBSCANclusteringnamingMolmo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that 3D keypoint detection can be done without any 3D keypoint annotations or training. It shows that the point-level localization ability of a multimodal language model (Molmo), trained on pixel-level annotations, transfers to rendered views of 3D shapes: ask it to 'point to the nose', back-project its 2D answers, and cluster them across views. On the KeypointNet benchmark the method matches or beats several supervised and few-shot detectors, and it works on arbitrary categories with no per-category adaptation. If the claim holds, 3D shape understanding no longer needs a supervised keypoint pipeline: any named part becomes an instant query.

What carries the argument

The load-bearing component is Molmo, an open multimodal language model trained with a dedicated pixel-level annotation task that lets it answer 'Point to {query} in this image' with a 2D coordinate. The method wraps this with three mechanisms: GPT-4o generates the candidate keypoint names; each of 26 rendered views is prompted with the name and the resulting 2D points are back-projected along cached depth rays using a small patch to stabilize sharp intersections; HDBSCAN then clusters the multi-view 3D candidates, returning only consistent keypoints and filtering outliers.

What would settle it

Test ZeroKey on a shape category that does not appear in Molmo's pixel-annotation dataset (for example, a watering can) and measure its geodesic IoU against a small human-annotated keypoint set; if at a 0.05 threshold the IoU is no better than the RedCircle random baseline, the localization signal is not genuinely zero-shot.

Watch

Extended reading notes

Core claim

The central discovery is that pixel-level annotation data used to train recent MLLMs carries usable 3D keypoint knowledge: without any ground-truth 3D labels, ZeroKey extracts and names salient keypoints on a 3D model by prompting Molmo to point at the keypoint in each rendered view, back-projecting the 2D points into 3D via cached per-pixel depth, and clustering across views with HDBSCAN to suppress noise. On KeypointNet (airplane, chair, table), the zero-shot method achieves an IoU of 54.64% at a geodesic threshold of 0.10, clearly above the text-to-point baselines (CLIP-DINOiser: 25.56, RedCircle: 18.50, GPT-4o: 20.73) and within the range of supervised and few-shot methods.

Load-bearing premise

The pipeline's accuracy is inherited wholesale from Molmo: if its 2D point answers are wrong or inconsistent on rendered views of an unseen shape, the back-projection and clustering cannot recover, and the zero-shot result fails.

Editorial extensions

If this is right

  • Keypoint detection becomes a text query: any part that can be named can be localized on any shape, with no per-category training set.
  • The same multi-view lifting can produce pseudo-labels to bootstrap supervised 3D keypoint datasets on new categories.
  • The Schelling points study shows the recovered points align with human-chosen salient points, linking language describability to annotation agreement.
  • The method establishes a baseline for zero-shot 3D keypoint detection that future vision-language approaches can be measured against.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'zero-shot' claim is scoped to 3D annotations: Molmo's 2D localization was trained on pixel-annotated images, so the true test of generalization is whether that 2D knowledge transfers to synthetic renderings and categories absent from its annotation corpus.
  • The cluster density from HDBSCAN could be reused as a confidence measure for whether a named part exists on a given shape, giving the pipeline a built-in 'not present' signal that the paper does not exploit.
  • The method could be inverted: instead of asking GPT-4o for names, let Molmo describe a marked point and then re-find it, a self-supervised loop that the paper's describability experiments hint at but do not develop into a full consistency score.
  • Because candidate names come from a language model, the ceiling of performance is set by the naming step; using a part taxonomy or image-derived description rather than free-form names would likely reduce ambiguous prompts like 'corner of the table'.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ZeroKey, a zero-shot 3D keypoint detection pipeline that first uses GPT-4o to generate a list of salient keypoint names from rendered views, then prompts Molmo to localize each named keypoint in multiple rendered views, back-projects these 2D detections onto the mesh, and aggregates them via patch refinement and HDBSCAN clustering. The method is evaluated on three KeypointNet categories (airplane, chair, table) with IoU at varying geodesic thresholds, along with ablations over prompt specificity, number of views, and clustering, plus additional applications to Schelling points and point describability.

Significance. If the full pipeline works as claimed, this would be an interesting demonstration that pixel-level pretraining in MLLMs transfers to 3D keypoint detection without 3D annotations. The paper's strengths include a simple, reproducible pipeline built on frozen models, a clear ablation showing the importance of point-specific prompts and HDBSCAN aggregation, and public code. The performance gap over the CLIP-DINOiser and GPT-4o baselines is substantial. However, the evaluation as it stands does not yet support the full 'extracting and naming' generality claim: the reported numbers use manually supplied names derived from ground-truth keypoint IDs, and only three classes are tested.

major comments (4)
  1. [Section 6.1, Supplementary D] The evaluation protocol manually attaches text descriptions to each ground-truth keypoint ID in KeypointNet and uses those descriptions as prompts to Molmo. This measures localization of externally specified keypoints, not the automatic extraction and naming step proposed in Section 4.1. The abstract's claim of 'extracting and naming salient keypoints' is therefore not supported by the quantitative results. A proper evaluation would run the full pipeline (GPT-4o candidate generation followed by Molmo localization) and measure how well the resulting keypoint set matches the ground truth, without constructing prompts from the ground-truth labels. This is load-bearing for the zero-shot claim.
  2. [Section 6.1, Table 1, Supplementary Table 2] The paper reports results only for airplane, chair, and table, three of the sixteen KeypointNet categories, and only as aggregate IoU curves with no per-category breakdown or variance. Given that the paper claims arbitrary-category generality and dedicates Supplementary D to all sixteen categories, the absence of per-category numbers is a significant omission; categories with many ambiguous or duplicated descriptions (e.g., bathtub, vessel) may behave very differently. At minimum, report per-category IoU and standard deviations across shapes, and ideally include at least one difficult category. The abstract's 'competitive performance compared to supervised methods' should also be qualified, since the supplementary table shows the method below the best few-shot methods (B2-3D, FSKD) at all thresholds.
  3. [Section 4.3, Eq. (7)] The clustering and matching details needed to interpret the IoU numbers are underspecified. The paper does not state how HDBSCAN's minPts k and the patch size h are chosen, whether these are fixed across categories, or how predicted clusters are matched to ground-truth keypoints. Figure 4 shows that the number of detected points for a single prompt ('corner of the table') grows with the number of views, so the matching rule is not trivial; without it, the reported IoU cannot be reproduced. Please specify the exact matching protocol and report sensitivity to h, k, and M.
  4. [Section 4.2, Eq. (1), Section 4.3] The entire pipeline's upper bound is Molmo's 2D point localization accuracy on rendered synthetic views of unseen categories, but this quantity is never isolated. A back-projected 2D error propagates directly into the 3D estimate; clustering cannot correct a consistently wrong point. The paper should report per-view 2D localization error against projected ground-truth keypoints for the evaluated categories, and ideally on unseen categories, to establish where errors originate and whether the zero-shot claim holds under less favorable renderings.
minor comments (4)
  1. [Section 6.3] The sentence 'Qualitative results are shown in Table 1' should be 'Quantitative results are shown in Table 1', since Table 1 contains numeric IoU values.
  2. [Supplementary C] The sentence 'Qualitative results are provided in Table 2' should likewise be 'Quantitative results are provided in Table 2'.
  3. [Section 6.4] The claim that 6 views achieve 80% of the full performance is not backed by axes or exact values in Figure 8; please state the number in the text and label the figure axes.
  4. [Figure 4] The caption says that as the number of views increases, ZeroKey detects more keypoints for 'corner of the table'; this ambiguity is central to the evaluation, so clarify whether the evaluation prompts also yield multiple clusters and how these are counted in the IoU metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ZeroKey's predicted keypoints come from external MLLMs and geometric aggregation, with no fitting to the benchmark.

full rationale

No circularity found. ZeroKey's derivation chain is: GPT-4o proposes candidate keypoint names (Sec. 4.1); Molmo returns 2D point locations per view (Eq. 1); those locations are back-projected along cached depth rays with patch averaging (Eqs. 2-6); and HDBSCAN clusters the multi-view back-projections (Eq. 7). No parameter is fitted to KeypointNet ground truth, and the IoU numbers in Tables 1-2 are evaluated against an external benchmark. The manually written text prompts in Supplementary D are input queries that name the target keypoints; they do not determine the predicted 3D coordinates, which are produced by Molmo's localization plus geometric back-projection. The same-authors citations ([1], [2], [5], [45]) appear in related-work or label-provenance contexts and are not load-bearing for the central zero-shot claim; no uniqueness theorem or fitted parameter is imported from them. Molmo [11], GPT-4o [32], CLIP-DINOiser [46], and KeypointNet [50] are external to this paper. The skeptic concern that Molmo may degrade on unseen categories or ambiguous prompts is a generalization and correctness risk, not a circularity: it does not make the output equivalent to the input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central pipeline rests on the reliability of an external MLLM (Molmo), standard camera geometry, and the validity of the manually annotated keypoint labels used for evaluation. No entities are invented and no parameters are fitted to the benchmark, but several implementation choices (view count, patch size, cluster size) are hand-set and unreported.

free parameters (3)
  • Patch size h for back-projection refinement
    Used in Eq. 6 to stabilize back-projection by averaging rays in an h by h patch around the detected 2D point; h is never reported.
  • HDBSCAN minPts k
    Sec 4.3 says 'setting correct minPts k' but the value and selection procedure are not given; this controls which clusters are kept as keypoints.
  • Number of rendered views M = 26
    Main experiments use 26 views; the ablation shows 6 views reach about 80% of full performance. The choice is manual and affects the cost and accuracy trade-off.
assumptions (4)
  • domain assumption Molmo provides reliable 2D point localization for language queries on rendered images of 3D shapes.
    Eq. 1 in Sec 4.2 depends on Molmo's point-level reasoning transferring from natural images to synthetic renderings of unseen categories.
  • standard math Pinhole camera model with cached depth enables correct ray-mesh back-projection.
    Eqs. 2-4; standard computer graphics assumption; the paper uses the depth buffer to avoid explicit ray intersection.
  • domain assumption KeypointNet ground-truth annotations and the manually assigned text labels are accurate.
    Sec 6.1: the text prompts used for evaluation are manually annotated for each keypoint ID, so evaluation quality depends on these labels matching human consensus.
  • domain assumption HDBSCAN clustering isolates the consistent keypoint cluster from noisy back-projections.
    Sec 4.3; assumes true keypoint projections form a dense cluster across views while spurious detections are scattered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models." pith.science (2026). https://pith.science/paper/XBMYKIXX

@misc{pith2026241206292,
  author       = {Pith},
  title        = {Pith review of: ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBMYKIXX}},
  note         = {Machine review of arXiv:2412.06292}
}
read the original abstract

We propose a novel zero-shot approach for keypoint detection on 3D shapes. Point-level reasoning on visual data is challenging as it requires precise localization capability, posing problems even for powerful models like DINO or CLIP. Traditional methods for 3D keypoint detection rely heavily on annotated 3D datasets and extensive supervised training, limiting their scalability and applicability to new categories or domains. In contrast, our method utilizes the rich knowledge embedded within Multi-Modal Large Language Models (MLLMs). Specifically, we demonstrate, for the first time, that pixel-level annotations used to train recent MLLMs can be exploited for both extracting and naming salient keypoints on 3D models without any ground truth labels or supervision. Experimental evaluations demonstrate that our approach achieves competitive performance on standard benchmarks compared to supervised methods, despite not requiring any 3D keypoint annotations during training. Our results highlight the potential of integrating language models for localized 3D shape understanding. This work opens new avenues for cross-modal learning and underscores the effectiveness of MLLMs in contributing to 3D computer vision challenges.

Figures

Figures reproduced from arXiv: 2412.06292 by the authors.

Figure 1
Figure 1. Zero-shot 3D Keypoint Detection. Without any ground truth labels or supervised training, our method leverages the point-level reasoning embedded within MLLMs to extract and name salient keypoints on 3D models. The figure illustrates how our approach achieves competitive performance compared to CLIP-DINOiser [46] baselines, highlighting the potential of integrating language models with vision tasks for enhanced 3D sh… view at source ↗
Figure 2
Figure 2. ZeroKey Pipeline. Our proposed ZeroKey employs MLLM Molmo for zero-shot keypoint detection on 3D objects by 1) rendering multiple views for a given shape, 2) leveraging MLLM reasoning in each view using point-specific prompts, and 3) aggregating the results through clustering, eliminating the need for annotated training data for 3D keypoint detection. detailed image captioning to complex visual reasoning. These mode… view at source ↗
Figure 3
Figure 3. Comparing the ground truth KeypointNet dataset an [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The number of rendered views versus the detected key [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: We ask Molmo to describe the green point, and using this as a prompt ZeroKey predicts the blue point. We show that salient [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: We compare against baselines CLIP-DINOiser and Red [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 6
Figure 6. Figure 6: We show through a quantitative study that “salient” [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Renderings from the ablation study where the MLLM [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Superposition of the keypoints predicted by ZeroKey [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation

    cs.CV 2025-08 conditional novelty 4.0 of 10

    Giving surgical-tool keypoint detection to Qwen2.5-VL via LoRA fine-tuning reaches MPJPE 0.0627 on SurgeoNet, comparable with or better than dedicated YOLOv8-Pose and SurgeoNet baselines.

Reference graph

Works this paper leans on

54 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    Zero-shot 3d shape correspon- dence

    Ahmed Abdelreheem, Abdelrahman Eldesokey, Maks Ovs- janikov, and Peter Wonka. Zero-shot 3d shape correspon- dence. In SIGGRAPH Asia 2023 Conference Papers, pages 1–11, 2023. 3

    2023

  2. [2]

    Satr: Zero-shot semantic segmentation of 3d shapes

    Ahmed Abdelreheem, Ivan Skorokhodov, Maks Ovsjanikov, and Peter Wonka. Satr: Zero-shot semantic segmentation of 3d shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15166–15179, 2023. 3

    work page 2023

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  4. [4]

    Claude 3.5 sonnet model card addendum

    AI Anthropic. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 2024. 2

    work page 2024

  5. [5]

    Ncp: Neural cor- respondence prior for effective unsupervised shape match- ing

    Souhaib Attaiki and Maks Ovsjanikov. Ncp: Neural cor- respondence prior for effective unsupervised shape match- ing. Advances in Neural Information Processing Systems , 35:28842–28857, 2022. 2

    work page 2022

  6. [6]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 2

    arXiv 2005

  7. [7]

    Understanding the limits of vision language mod- els through the lens of the binding problem

    Declan Campbell, Sunayana Rane, Tyler Giallanza, Nicol `o De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M Frankland, Thomas L Griffiths, Jonathan D Co- hen, et al. Understanding the limits of vision language mod- els through the lens of the binding problem. arXiv preprint arXiv:2411.00238, 2024. 2

    arXiv 2024

  8. [8]

    Unsuper- vised learning of intrinsic structural representation points

    Nenglun Chen, Lingjie Liu, Zhiming Cui, Runnan Chen, Duygu Ceylan, Changhe Tu, and Wenping Wang. Unsuper- vised learning of intrinsic structural representation points. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 9121–9130, 2020. 3

    work page 2020

Show all 54 references
  1. [9]

    Schelling points on 3d surface meshes

    Xiaobai Chen, Abulhair Saparov, Bill Pang, and Thomas Funkhouser. Schelling points on 3d surface meshes. ACM Transactions on Graphics (TOG) , 31(4):1–12, 2012. 3, 6, 12, 13

    2012

  2. [10]

    3dmv: Joint 3d-multi- view prediction for 3d semantic scene segmentation

    Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi- view prediction for 3d semantic scene segmentation. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 452–468, 2018. 3

    2018

  3. [11]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2...

    2024 arXiv

  4. [12]

    Unsupervised learning of category-specific symmetric 3d keypoints from point sets

    Clara Fernandez-Labrador, Ajad Chhatkuli, Danda Pani Paudel, Jose J Guerrero, C ´edric Demonceaux, and Luc Van Gool. Unsupervised learning of category-specific symmetric 3d keypoints from point sets. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 2...

    2020

  5. [13]

    Mvtn: Multi-view transformation network for 3d shape recognition

    Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–11, 2021. 3

    2021

  6. [14]

    V oint cloud: Multi-view point cloud representation for 3d understanding

    Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. V oint cloud: Multi-view point cloud representation for 3d understanding. In The Eleventh International Conference on Learning Representations, 2023. 3

    2023

  7. [15]

    Mvtn: Learning multi-view transforma- tions for 3d understanding

    Abdullah Hamdi, Faisal AlZahrani, Silvio Giancola, and Bernard Ghanem. Mvtn: Learning multi-view transforma- tions for 3d understanding. International Journal of Com- puter Vision, 2024. 3

    2024

  8. [16]

    Unsupervised keypoints from pretrained diffusion models

    Eric Hedlin, Gopal Sharma, Shweta Mahajan, Xingzhe He, Hossam Isack, Abhishek Kar, Helge Rhodin, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised keypoints from pretrained diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  9. [17]

    3d-llm: In- jecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: In- jecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494,

  10. [18]

    3d-sis: 3d se- mantic instance segmentation of rgb-d scans

    Ji Hou, Angela Dai, and Matthias Nießner. 3d-sis: 3d se- mantic instance segmentation of rgb-d scans. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4421–4430, 2019. 3

    2019

  11. [19]

    Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels

    Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engel- mann. Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. European Conference on Computer Vision (ECCV), 2024. 3

    2024

  12. [20]

    Keypointdeformer: 9 Unsupervised 3d keypoint discovery for shape control

    Tomas Jakab, Richard Tucker, Ameesh Makadia, Jiajun Wu, Noah Snavely, and Angjoo Kanazawa. Keypointdeformer: 9 Unsupervised 3d keypoint discovery for shape control. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 12783–12792, 2021. 3

    2021

  13. [21]

    Multi-view pointnet for 3d scene understanding

    Maximilian Jaritz, Jiayuan Gu, and Hao Su. Multi-view pointnet for 3d scene understanding. In Proceedings of the IEEE International Conference on Computer Vision Work- shops, pages 0–0, 2019. 3

    2019

  14. [22]

    3d shape segmentation with projective convolutional networks

    Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3d shape segmentation with projective convolutional networks. In proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 3779–3788, 2017. 3

    2017

  15. [23]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything, 2023. 3

    2023

  16. [24]

    Virtual multi-view fusion for 3d semantic segmentation

    Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, and Caroline Pantofaru. Virtual multi-view fusion for 3d semantic segmentation. In European Conference on Computer Vision (ECCV) , pages 518–535. Springer, 2020. 3

    2020

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1, 2

    2023

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 2

    2023

  19. [27]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2

    2024

  20. [28]

    3d-to-2d distillation for indoor scene parsing

    Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 3d-to-2d distillation for indoor scene parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4464–4474, 2021. 3

    2021

  21. [29]

    Learning to segment 3d point clouds in 2d image space

    Yecheng Lyu, Xinming Huang, and Ziming Zhang. Learning to segment 3d point clouds in 2d image space. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12255–12264, 2020. 3

    2020

  22. [30]

    Egoloc: Revisiting 3d object localiza- tion from egocentric videos with visual queries

    Jinjie Mai, Abdullah Hamdi, Silvio Giancola, Chen Zhao, and Bernard Ghanem. Egoloc: Revisiting 3d object localiza- tion from egocentric videos with visual queries. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 45–57, 2023. 3

    2023

  23. [31]

    Tracknerf: Bundle adjusting nerf from sparse and noisy views via feature tracks, 2024

    Jinjie Mai, Wenxuan Zhu, Sara Rojas, Jesus Zarzar, Abdul- lah Hamdi, Guocheng Qian, Bing Li, Silvio Giancola, and Bernard Ghanem. Tracknerf: Bundle adjusting nerf from sparse and noisy views via feature tracks, 2024. 3

    2024

  24. [32]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1, 2, 4

    2023

  25. [33]

    Synthesize diagnose and optimize: Towards fine- grained vision-language understanding

    Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zux- uan Wu. Synthesize diagnose and optimize: Towards fine- grained vision-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13279–13288, 2024. 2

    2024

  26. [34]

    Shapellm: Universal 3d object understanding for embodied interaction

    Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In Computer Vision – ECCV 2024: 18th European Confer- ence, Milan, Italy, September 29–October 4, 2024, Proc...

    2024

  27. [35]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 2, 3

    2021

  28. [36]

    Vision language models are blind

    Pooyan Rahmanzadehgervi, Logan Bolton, Moham- mad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. arXiv preprint arXiv:2407.06581, 2024. 2

    2024 arXiv

  29. [37]

    The Strategy of Conflict: with a new Preface by the Author

    Thomas C Schelling. The Strategy of Conflict: with a new Preface by the Author. Harvard university press, 1980. 6

    1980

  30. [38]

    Mask3D: Mask Trans- former for 3D Semantic Instance Segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Trans- former for 3D Semantic Instance Segmentation. In Inter- national Conference on Robotics and Automation (ICRA) ,

  31. [39]

    Skele- ton merger: an unsupervised aligned keypoint detector

    Ruoxi Shi, Zhengrong Xue, Yang You, and Cewu Lu. Skele- ton merger: an unsupervised aligned keypoint detector. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 43–52, 2021. 3

    2021

  32. [40]

    What does clip know about a red circle? vi- sual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? vi- sual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11987–11997, 2023. 3, 7, 8, 13

    2023

  33. [41]

    Discovery of latent 3d key- points via end-to-end geometric reasoning

    Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tomp- son, and Mohammad Norouzi. Discovery of latent 3d key- points via end-to-end geometric reasoning. Advances in neu- ral information processing systems, 31, 2018. 3

    2018

  34. [42]

    Ldls: 3- d object segmentation through label diffusion from 2-d im- ages

    Brian H Wang, Wei-Lun Chao, Yan Wang, Bharath Hariha- ran, Kilian Q Weinberger, and Mark Campbell. Ldls: 3- d object segmentation through label diffusion from 2-d im- ages. IEEE Robotics and Automation Letters , 4(3):2902– 2909, 2019. 3

    2019

  35. [43]

    Learning 3d keypoint descriptors for non-rigid shape matching

    Hanyu Wang, Jianwei Guo, Dong-Ming Yan, Weize Quan, and Xiaopeng Zhang. Learning 3d keypoint descriptors for non-rigid shape matching. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018. 3

    2018

  36. [44]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International conference on machine learn- i...

    2022

  37. [45]

    Back to 3d: Few-shot 3d keypoint detection with back-projected 2d features

    Thomas Wimmer, Peter Wonka, and Maks Ovsjanikov. Back to 3d: Few-shot 3d keypoint detection with back-projected 2d features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4154– 4164, 2024. 2, 4, 13

    2024

  38. [46]

    Clip-dinoiser: Teaching clip a few dino tricks for open- 10 vocabulary semantic segmentation

    Monika Wysoczanska, Oriane Sim ´eoni, Micha¨el Ramamon- jisoa, Andrei Bursuc, Tomasz Trzcinski, and Patrick P ´erez. Clip-dinoiser: Teaching clip a few dino tricks for open- 10 vocabulary semantic segmentation. In Proc. ECCV, 2024. 1, 7, 8, 13

    2024

  39. [47]

    Pointllm: Empowering large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiang- miao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision , pages 131–147. Springer,

  40. [48]

    Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding

    Le Xue, Mingfei Gao, Chen Xing, Roberto Mart ´ın-Mart´ın, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF conference o...

    2023

  41. [49]

    3dfeat-net: Weakly su- pervised local 3d features for point cloud registration

    Zi Jian Yew and Gim Hee Lee. 3dfeat-net: Weakly su- pervised local 3d features for point cloud registration. In Proceedings of the European conference on computer vision (ECCV), pages 607–623, 2018. 3

    2018

  42. [50]

    Keypointnet: A large-scale 3d keypoint dataset aggre- gated from numerous human annotations

    Yang You, Yujing Lou, Chengkun Li, Zhoujun Cheng, Liangwei Li, Lizhuang Ma, Cewu Lu, and Weiming Wang. Keypointnet: A large-scale 3d keypoint dataset aggre- gated from numerous human annotations. arXiv preprint arXiv:2002.12687, 2020. 3, 4, 6

    2002 arXiv

  43. [51]

    Ukpgan: A general self-supervised keypoint detector

    Yang You, Wenhai Liu, Yanjie Ze, Yong-Lu Li, Weiming Wang, and Cewu Lu. Ukpgan: A general self-supervised keypoint detector. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2022. 2, 7

    2022

  44. [52]

    Good at captioning bad at counting: Benchmarking gpt-4v on earth observation data

    Chenhui Zhang and Sherrie Wang. Good at captioning bad at counting: Benchmarking gpt-4v on earth observation data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 7839–7849, 2024. 2

    2024

  45. [53]

    Uni3d: Exploring unified 3d representation at scale

    Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.06773,

    arXiv

  46. [54]

    List possible salient keypoints (in text)

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2 11 ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from...

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.